Firefox by the (wall-clock) numbers

Mozilla released Firefox on Thursday 07-feb-2008, at 16:45pm PST. From “Dev says go” to “release is now available to public” was almost 11 days (10d 21h 30m) wall-clock time, of which  Build&Release took almost 4 days (95h20m). Sadly, not all important milestones were recorded, so if anyone have info for the times marked ??:?? below, please let me know and I’ll update.

19:10 28jan: Dev says “go” for rc1
19:45 28jan: builds started
22:45 28jan: linux builds handed to QA
02:05 29jan: mac builds handed to QA
04:00 29jan: win32 signed builds handed to QA
12:00 29jan: update snippets available on betatest update channel
13:00 30nov: bug#414856 declared showstopper, Dev says “go” for rc2.
14:50 30jan: builds started
17:00 30jan: linux builds handed to QA
18:55 30jan: mac builds handed to QA
??:?? ??jan: unsigned-win32 builds waiting to be signed when QA discovered that bug#413250 has another showstopper exploit path
14:40 31jan: Dev says “go” for rc3
14:41 31jan: builds started
16:45 31jan: linux builds handed to QA
18:35 31jan: mac builds handed to QA
10:05 01feb: signed-win32 signed builds handed to QA
18:05 01feb: bug#415292 declared showstopper….
20:00 01feb: Dev says “go” for rc4
20:40 01feb: builds started
01:25 02feb: linux & mac builds handed to QA
04:35 02feb: signed-win32 signed builds handed to QA
08:20 02feb: update snippets available on betatest update channel
10:30 04feb: QA says “go to beta”.
11:25 04feb: update snippets on beta update channel
09:00 07feb: Dev & QA says “go” for Release; Build already completed final signing, bouncer entries
09:20 07feb: mirror replication started
11:45 07feb: mirror absorption good for testing to start on releasetest channel
14:40 07feb: QA completes testing releasetest.
??:?? 07feb: website changes finalized and visible. Build given “go” to make updates snippets live.
??:?? 07feb: update snippets available on live update channel
16:45 07feb: release announced


1) The Build Automation used in FF2.0.0.12  included a bunch of fixes landed after FF2.0.0.11, which helped make things smoother. Despite the number of respins, it felt like all the housekeeping of the last few weeks paid off.

2) For better or worse, we are putting all our blow-by-blow scribbles public, so the curious can read about it, warts and all, here. Some highlights were:

  • win32 builds failed out first time because buildbot slave was not restarted correctly. These need to be configured to auto-reboot cleanly.
  • mac build failed out first time and had to be restarted. This has happened to us intermittently before, and we dont know yet how to reproduce or fix this.
  • Long standing, but previously unknown, bug#414966 was fixed. This involved portions of tag respin code using UTC, while other portions used PST. If the initial rc1 build was started after 4pm, this bug would cause some parts of automation to expect one date, while other portions would expect another date, and show up as a problem if we did a respin. As far as we can tell, this bug has always been present, and we have just been “lucky” until now.
  • Long standing bug#388524 was finally fixed. This caused users on “beta” channel to be offered complete updates, even when partial updates were available for them. We now serve partial updates to beta channel users whenever possible.

3) Like before, we waited until morning to start pushing to mirrors. This was done so mirror absorption completed as QA were arriving in the office to start testing update channels. We did this because we wanted to reduce the time files were on the mirrors untested; in the past, overly excited people have post the locations of the files as “released” on public forums, even though they are not finished the last of the sanity checks. Coordinating the mirror push like this reduced that likelihood just a bit.

4) Mirror absorption took 2.5 hours to reach all values >= 50%. However, this felt a little low, so we’ll go back to a 60% threshold for future releases.

take care