Mozilla released Firefox 2.0.0.12 on Thursday 07-feb-2008, at 16:45pm PST. From “Dev says go” to “release is now available to public” was almost 11 days (10d 21h 30m) wall-clock time, of which  Build&Release took almost 4 days (95h20m). Sadly, not all important milestones were recorded, so if anyone have info for the times marked ??:?? below, please let me know and I’ll update.

19:10 28jan: Dev says “go” for rc1
19:45 28jan: 2.0.0.12rc1 builds started
22:45 28jan: 2.0.0.12rc1 linux builds handed to QA
02:05 29jan: 2.0.0.12rc1 mac builds handed to QA
04:00 29jan: 2.0.0.12rc1 win32 signed builds handed to QA
12:00 29jan: update snippets available on betatest update channel
13:00 30nov: bug#414856 declared showstopper, Dev says “go” for rc2.
14:50 30jan: 2.0.0.12rc2 builds started
17:00 30jan: 2.0.0.12rc2 linux builds handed to QA
18:55 30jan: 2.0.0.12rc2 mac builds handed to QA
??:?? ??jan: 2.0.0.12rc2 unsigned-win32 builds waiting to be signed when QA discovered that bug#413250 has another showstopper exploit path
14:40 31jan: Dev says “go” for rc3
14:41 31jan: 2.0.0.12rc3 builds started
16:45 31jan: 2.0.0.12rc3 linux builds handed to QA
18:35 31jan: 2.0.0.12rc3 mac builds handed to QA
10:05 01feb: 2.0.0.12rc3 signed-win32 signed builds handed to QA
18:05 01feb: bug#415292 declared showstopper….
20:00 01feb: Dev says “go” for rc4
20:40 01feb: 2.0.0.12rc4 builds started
01:25 02feb: 2.0.0.12rc4 linux & mac builds handed to QA
04:35 02feb: 2.0.0.12rc4 signed-win32 signed builds handed to QA
08:20 02feb: 2.0.0.12rc4 update snippets available on betatest update channel
10:30 04feb: QA says “go to beta”.
11:25 04feb: update snippets on beta update channel
09:00 07feb: Dev & QA says “go” for Release; Build already completed final signing, bouncer entries
09:20 07feb: mirror replication started
11:45 07feb: mirror absorption good for testing to start on releasetest channel
14:40 07feb: QA completes testing releasetest.
??:?? 07feb: website changes finalized and visible. Build given “go” to make updates snippets live.
??:?? 07feb: update snippets available on live update channel
16:45 07feb: release announced

Notes:

1) The Build Automation used in FF2.0.0.12  included a bunch of fixes landed after FF2.0.0.11, which helped make things smoother. Despite the number of respins, it felt like all the housekeeping of the last few weeks paid off.

2) For better or worse, we are putting all our blow-by-blow scribbles public, so the curious can read about it, warts and all, here. Some highlights were:

• win32 builds failed out first time because buildbot slave was not restarted correctly. These need to be configured to auto-reboot cleanly.
• mac build failed out first time and had to be restarted. This has happened to us intermittently before, and we dont know yet how to reproduce or fix this.
• Long standing, but previously unknown, bug#414966 was fixed. This involved portions of tag respin code using UTC, while other portions used PST. If the initial rc1 build was started after 4pm, this bug would cause some parts of automation to expect one date, while other portions would expect another date, and show up as a problem if we did a respin. As far as we can tell, this bug has always been present, and we have just been “lucky” until now.
• Long standing bug#388524 was finally fixed. This caused users on “beta” channel to be offered complete updates, even when partial updates were available for them. We now serve partial updates to beta channel users whenever possible.

3) Like before, we waited until morning to start pushing to mirrors. This was done so mirror absorption completed as QA were arriving in the office to start testing update channels. We did this because we wanted to reduce the time files were on the mirrors untested; in the past, overly excited people have post the locations of the files as “released” on public forums, even though they are not finished the last of the sanity checks. Coordinating the mirror push like this reduced that likelihood just a bit.

4) Mirror absorption took 2.5 hours to reach all values >= 50%. However, this felt a little low, so we’ll go back to a 60% threshold for future releases.

take care

John.