10 Jan 2012
JohnMozilla
In my trip to/from Cambodia, I specifically set aside some time to visit Hong Kong. (As I write, I’m watching the brilliant haze of clouds over Deep Water Bay, and am on my 3rd coffee before heading out to explore the city!).
On Thursday 26rd Jan 2012, in Hong Kong , I’ll meet Haggen and some community members here in Hong Kong. While it is really close to Chinese New Year, if you are in Hong Kong, please do come join us, say hi, learn a little about how Release Engineering at Mozilla works, and get to meet some other Mozilla community members. Even in this day and age of hi-tech, its still great to say “hello” in person.
Haggen has setup this facebook page to track logistics for the event. HKCommons has donated use of their venue at HKCommons, 9/F, 633 King’s Road, Quarry Bay, Hong Kong. They’ve got plenty of space, and enough internet connectivity for us all. If you have questions about the event, or the location, you can always email Haggen or myself directly.
Hope to see you soon!
[UPDATED 18jan2012 to include address of venue.]
10 Jan 2012
JohnMozilla
We recently added Khmer localized builds of Firefox to Aurora. (At the same time, we also added ach, ff, lij, wo locales, but that is a story for another blog).
If you can read/write Khmer, please download the Khmer version of Firefox’s Aurora build from here, use it as your daily browser, and file bugs for anything you see that you think needs fixing. All bugs welcome! As bugs are fixed, new Aurora nightly releases will be published, and you’ll automatically update to the newest Aurora build with the latest fixes and latest localization changes.
Its important to keep in mind that this is an Aurora build, not a Beta or Release build. This means there are bugs – some strings are still being localized; some code bugs are still being fixed, etc, etc. There’s still more work to be done but getting onto Aurora is a significant milestone on the way to having Khmer included in the official Firefox release.
To celebrate this big milestone, at 2pm, Monday 23rd Jan 2012, I’ll meet Vannak and some of the Khmer localizers who helped make all this happen, as well as with people from other interesting software projects in Cambodia, so this is already looking to be an exciting event. We’ll be meeting at East-West Management Institute, House #43, Street 208, Phnom Penh, Cambodia. Events formally go from 2pm-5pm Monday 23rd January, 2012; at 7pm, we’ll head over to Romdeng, House# 74, Street 174, Phnom Penh for dinner+drinks+chat.
Its been 5 years since my last visit to Cambodia, but I’m eager to be coming back to celebrate this big milestone, and to say hello to some new (and some familiar) faces. If you are near Phnom Penh, Cambodia, on Mon 23rd, please come by, say hi, attend some of the sessions and then join the celebrations. It would be great to meet up in person, talk about Mozilla, and help make plans for improving Khmer Firefox until it is ready to be on Beta and then on to Release.
If you have any questions, please do email Vannak, Mark West or myself.
[UPDATED 19jan2012 with location addresses and exact times]
03 Jan 2012
JohnMozilla
Note: This is the first “infrastructure load” report I’ve done since we switched to the new rapid release model. Its worth noting that because of the rapid release model, we’ve jumped to 30+ active project branches. Plotting all these branches would make the charts too noisy, and hard to interpret. Therefore, while I still look at the metrics for all branches, I now only chart the busiest or the most strategically important branches.
Having said all that, now lets get to the interesting stuff!
- #checkins-per-month: We finally broke the 3,000 checkins-per-month barrier. August set a new record with 3,089 checkins. Then November set a new record with 3,209 checkins, and finally December set another new record with 3,262 checkins!
- #checkins-per-day: We set a new record of 169 checkins-per-day on 06-nov-2011, only to then match or exceed that new record 4 times in December (184 on 08-dec, 169 on 15-dec, 171 on 19-dec, 184 on 21-dec).
- I find these records more impressive because they were set in November and December – when we expected to see *low* checkin volumes. Both months have major vacations in them, so we historically see checkin volume decrease as people take vacations. Further, both months had multiple prolonged tree closures, caused by various colo outages, db server outages, etc. (The trees were able to remain open during the email outage because of the proactive work by RelEng and WebDev to refactor tbpl.mozilla.org earlier this year! Topic for another blog post.) Between the vacations, and the outages, we expected to see significantly decreased checkins in both months, so setting new records like this was unexpected. My current hypothesis is that while the outages caused a backlog of pending fixes, we were able to quickly handle the load spikes when infrastructure came back online, and developers all resumed work… until the next outage. This “full-go, full-stop, full-go” oscillation still managed to set a new record. With some of these issues now resolved, and with developers coming back from vacations, I’m eager to see what numbers we are capable of handling in January.
mozilla-inbound, fx-team:
Its very cool to see how developers have taken to using mozilla-inbound (and more recently fx-team) as integration branches, instead of having everyone landing directly into mozilla-central.
- In the chart above, note that the number of mozilla-central checkins has decreased significantly, as number of checkins on mozilla-inbound increased.
-
Another interesting effect of this was while unwinding backlog of pending checkins after the outages. In the past whenever we had to unwind a large backlog of pending checkins, we’d keep the trees closed while developers did metered checkins to work through the backlog. However, after each of these recent outages, developer usage of mozilla-inbound and fx-team as integration branches meant that backlogs were cleared out more quickly, and with less manual metering of checkins on mozilla-central.
mozilla-aurora, mozilla-beta:
I note that ~2% of our monthly checkins land into mozilla-aurora, and in turn, about half of those ~1% then also land onto mozilla-beta. Part of me feels this is a healthy low number of fixes landing on aurora, and a healthy even lower number of fixes landing on beta. And this was part of the plan for the rapid release model. At the same time, part of me also feels like too many fixes are still needed on mozilla-beta, and I fear that this is a sign too many bugs are making it through mozilla-central / mozilla-aurora to mozilla-beta before being detected. In which case, what could we do differently in order to change this? Honestly, I cant tell for certain, and I’d be curious what others think. (Oh, and for the record, I’m always glad whenever we catch a problem *before* we ship a release; it avoids us having to do a chemspill release and its better for Firefox users too!)
misc other details:
Pushes per day
Pushes by hour of day
29 Dec 2011
JohnMozilla
Background:
At the Mozilla AllHands in Sept2011, I was surprised to find that my proposed session on git was accepted, and even more surprised at how well this session was attended! The room was full of people from all different groups across Mozilla. And boy, were people passionate. The slides don’t capture the lively back-forth discussions, but this is obviously a topic that people care deeply about, hence my blog post.
Here’s a quick summary of the main speaking points:
- Some projects in Mozilla are now basing their code in github.com, not on Mozilla’s existing release infrastructure. They do this despite the fact that github in one sense has reduced functionality (doesnt do all builds/test/automated-regression alerts/etc), and despite that using github like this causes extra manual headaches of periodically importing/exporting code, and complicates branch mechanics of releases. So why do this?
- Theory#1: git vs hg
- Some people prefer to use the git commandline tool. At same time, some people prefer to use the hg commandline tool. Each Distributed Version Control System has various technical merits, and drawbacks, so I put this git-vs-hg debate into the same category as an emacs-vs-vi debate. (I do *not* mean that as a put down – I mean this in the nicest possible way – the reality of life is that people have their own unique hard-earned preferences)
- Theory#2: github vs hg+bugzilla+graphserver+tbpl
- Instead of this debate being about the mechanical differences of the command line tools, I instead believe the debate is actually about Developer Workflow. This is not just theoretical; it makes a day-to-day difference to how developers get their job done.
- Every mozilla developer ends up using hg+bugzilla+graphserver+tbpl. These are organically grown, over the years, and each went from hacky-experiments-to-mission-critical once people started to rely on them. However, they are not smoothly cross-integrated.
- Mozilla saw a jump in checkins-per-day when tbpl.mozilla.org went live; tree closures were shorter, in part because people could more easily figure out who broke the build, and who/what to back out. (Of course, there’s more to it then *just* tbpl, but the usability issue here is the key point).
- While different people have talked about this, in different companies over the years, I think github.com is a really compelling proof point that good developer workflow makes a difference. If you can make it easier for a developer to find a bug, get the bugfix reviewed and landed, and be able to see if it made a difference, developers will WANT to use your production systems.
- Apart from workflow, there’s also security and autonomy concerns. How valuable are your bug discussions, review history, regression tracking? Who else would you entrust this valuable data to? If you find someone you trust, how do you know they’ll be around as long/longer then Mozilla?
Hal Wine has now got up-to-speed in this area because of his work on git-staging. Next, Hal is starting to see how feasible would it be to support both git command line tools and hg command line tools in our production RelEng + WebDev + IT systems. And meanwhile, Hal, LauraT, myself and some others are gathering to see what we can do to improve developer workflow. As Mozilla grows, its important to make it easier to do coding work here – after all, this is one way to encourage more people to contribute, and to help Mozilla scale.
If you have suggestions, or want to help, we’d love to hear from you – or if you prefer, you can just follow the tip of the iceberg in bug#713782.
27 Dec 2011
JohnMozilla, Soapbox
Well, here’s a nice surprise – another album from General Fuzz. Click on the album cover to jump over to the music and listen for yourself.
Tomorrow, I’ll load this new album up and see what I can crank out in the office with headphones on. Thanks, James!
19 Dec 2011
JohnMozilla
I’m really excited to welcome Hal Wine to Release Engineering.
Hal has lots of experience in the trenches of Release Engineering, with a rare combination of experience working in distributed groups (not just solo!), working on small/embedded systems and working at scale. To make him even more unique, he’s a really nice guy *and* he even wears an occasional Hawaiian shirt! The only down side is that we’re now rethinking our previously lenient policy on bad puns 
He’s reporting to me, and already helping bring more tegras online to help close out the “android as tier 1″ project, while he’s also got his teeth sunk into a large interesting skunkworks project (more details coming soon in a separate post).
Hal’s based here in the San Francisco office, but you can find him on irc as “hwine”. Please do stop by and say hi – he’s already getting quite settled in as part of the family.
[UPDATE: Hal's blog is now up and running here. joduinn 03-jan-2012 ]
07 Dec 2011
JohnMozilla
I’d like to welcome John Hopkins to Release Engineering. I apologize that this welcome is a very belated welcome; John started in RelEng on 19th Sept and coop already posted a welcome here.
From one perspective, its not really any change here, John had already been coming to the RelEng group meetings since earlier this year when Mozilla Messaging folded back into Mozilla Corporation. Even before that, John was a familiar face to us, working on lots of the same buildbot and release automation code as the rest of RelEng.
From another perspective, there is potential for significant change here. Are there economies of scale if we merge the existing Thunderbird release engineering systems and the existing Firefox release engineering systems together? Now that we are in rapid-release cycles for both products, could we sim-ship Thunderbird with Firefox? How would we handle long term support issues for both products? Some things seem obvious, like how the tool chains for compiling-and-linking are identical for both products, so sharing machines and setup would help, and how we can improve Mozilla’s busfactor – jhopkins is the only person in RelEng who has shipped Thunderbird release since the old TB2 automation. Some things are less obvious, like what to do with the different spec hardware which was being used by MoMo. All this will play out over the coming months. Suggestions, comments, ideas all welcome!
As a good sign of how well everyone already works well together, the same week jhopkins officially joined RelEng, we discovered a serious problem: the production MoMo machines for Thunderbird were sitting on a shelf in the old Vancouver office, and these production machines were a few days away from being powered off as part of the move to a new Vancouver office. As the new office wasnt ready, so Vancouver people had to move to a temp-workspace for some weeks, and the Thunderbird production machines couldnt come to the temp-workspace offices. This meant the Thunderbird production machines would be offline, and the trees closed for the duration. No-one liked the idea of closing the Thunderbird trees for weeks, so we all piled in to help.
An immense volume of behind-the-scenes work took place, the Thunderbird continuous integration processes were migrated to our existing RelEng colos, toolchains setup on slightly different spec machines, and then production builds+tests enabled in the RelEng colo – all before the old Vancouver office was vacated, and all without closing the Thunderbird trees at all.
Bug#688230 has all the details. As I said in the “Friends of the Tree” section of the Mozilla Foundation weekly call at the time, this was an impressive volume of behind the scenes work, and developers using production systems never hit any problems!
Welcome John!
27 Nov 2011
JohnMozilla, Travel
(long overdue post; found these photos while writing another travel post for my website.)
Immediately after I returned from Japan last year, I was really happy to find these fruit flavoured, gummy chews also available in SF. Weirdly addictive, and I’ve made specific trips out to buy more from the only shop I know that sells them in SF. 
In the same week that I returned to California, someone else (who shall remain nameless by request!) brought the following back from his trip.
These tasted awful, and the flavour lingered for hours, despite vigorous toothbrushing 
11 Nov 2011
JohnMozilla
About the same time that Tristan was at the OpenWeb conference in Dublin, Ireland, I was also in Dublin, for a slightly different reason.
The record heavy rain complicated logistics a bit, but it was great to meet with people at Dublin City University and also Dublin Institute of Technology about opportunities in open source projects. There was lots of curiosity and interest about Mozilla, the work that goes into Firefox and even the very idea of working at a company that was all about open source. The impromptu corridor discussions about increasing involving open source were exciting. It was a jam-packed day, but I was really grateful it all came together.
Big thanks to Mike Scott, Markus Helfert, David Sinclair, Martin Crane, Maeve Long, Geraldine Farrell, Mary Regan and Damhait Harvey for making this possible.
(ps: when I say rain, I mean *rain*. (“More than one month’s rain fell on Dublin in 24 hours…”). Click photo for full story. 
12 Oct 2011
JohnMozilla
There’s been a lot of discussion in the Enterprise Working Group emails, and on the monthly EWG calls, about how Mozilla’s change to rapid release cadence is impacting enterprises. While brainstorming in a recent EWG call, I asked:
“how do enterprises handle rapid releases of other software products?”
After a few minutes discussion, we agreed to continue the discussion afterards. I’ve since emailed this same question to some lists, but am now cross-posting to my blog, in the hope that even more will see this question.
Does anyone have examples of other software products that do “rapid releases” and are being successfully handled by enterprises? If so, can you share some details? (If you prefer to email me privately, that is totally cool, please use my joduinn [at] mozilla [dot] com address). I ask because maybe we don’t need to reinvent the wheel here. If there is a tried-and-tested approach that already works well for other applications, that same approach might also makes work for Mozilla’s Firefox.
Some obvious examples of “rapid release software” in enterprises are OS-patch-updates updates, but what about Microsoft Office? Google Chrome? Flash? Java? Anything else? (In my mind updating anti-virus signature files is different scenario, but I could be misreading this scenario).
Thanks.
John.
Older Entries
RSS
