Behind the scene mechanics of Firefox5.0 / Fennec5.0

By now everyone knows that Firefox5.0 shipped, for desktop and for mobile, on 21June2011. That has already been covered elsewhere in great detail. However, now that the dust has settled, there are some behind-the-scenes details that I felt were important to draw attention to.

1) The Firefox5.0 and Fennec5.0 releases were both were based on the *same* identical changeset.
That wasn’t a coincidence – since 5.0beta2, every beta leading up to the Firefox5.0 release and Fennec 5.0 release was built from the same *identical* changeset. This was a major new milestone, made possible by streamlined infrastructures, and has important consequences for Mozilla’s ability to quickly find/fix/deliver security releases to protect our users.

2) We shipped Firefox5.0, Fennec5.0 and Firefox 3.6.18 all on the same day.
Shipping a major “new feature” release is tricky business – there’s a lot of fiddly details, and it’s typically “all hands on deck”. Because of this, we used to make sure *nothing* else was scheduled anywhere near a major release day. However, shipping a major release and announcing the security fixes in it, without also shipping a fix for older branches can be seen as a mixed message to our users on the older branches. Simultaneously shipping the same security fixes in security releases for the older supported branches is whats best for our entire set of users. However, this is tricky to do, and requires a lot of extra planning.

The first time we felt organized enough to safely ship a major “new feature” release was when we shipped Firefox4.0 and Firefox3.6.x and Firefox3.5.x on the one day. It was a very long, hectic 14 hour day (from 7am – ~9pm PDT), but it meant we could protect all Firefox users at the same time from a late breaking security exploit. Fennec4.0 had to wait and ship a week later. By contrast, when we shipped Firefox5.0, Fennec5.0 and Firefox3.6.18, we were all done in a calm orderly 4 hours (from 7am – ~11am PDT).

3) In the days leading up to the Firefox5.0/Fennec5.0/3.6.18 release, we did nine last-minute betas/releases in quick succession.
This fast turnaround was only possible because RelEng’s ongoing automation improvements has reduced our build times from 45hours down to 8hours. It was only because of this fast turnaround that Mozilla could accommodate some last minute fixes without impacting the release schedule. I feel its important to point out that, while there were tight deadlines, and a lot of very precise fast moving footwork, this was all done without burning out the humans in RelEng working those releases.

As Gary pointed out in a brief celebration speech Tuesday, its a big deal to change Mozilla’s development culture from “ship one big bang product when its ready” to “ship lots of smaller incremental products much more frequently”. Doing that transition in such a short timeframe is super impressive, and I believe was made possible by the infrastructure work from RelEng. Switching from the one-track model (trunk in cvs) to the multiple-concurrent-tracks model (currently ~35 active project branches in mercurial). Stabilizing the new mobile product infrastructure, and integrating it with the stable desktop product automation. Relentlessly improving our automation. “baby steps… relentless baby steps”. The list goes on and on…

I’m immensely proud of the quiet behind-the-scenes work that RelEng has done over the last 4 years to make this faster-release-cadence environment possible here at Mozilla. Thank you aki, armen, bear, bhearsum, catlee, coop, dustin, jhford, lsblakk, rail, nthomas.

3 thoughts on “Behind the scene mechanics of Firefox5.0 / Fennec5.0

  1. What you guys achieved is really great. I’m happy I got insight into this and could be right there when we were first plotting how to do hg-based release automation.You and your team did awesome work there, be proud of it!