Mozilla released Firefox3.0beta3 on Tuesday 12-feb-2008, at 17:45pm PST. From “Dev says go” to “release is now available to public” was 8 days (8d 0h 25m) wall-clock time, of which Build&Release took 2 days 9 hours.

17:20 04feb: Dev says “go” for rc1
18:30 04feb: 3.0b3rc1 builds started
10:10 05feb: Build declared rc1 bad because of mozconfig mismatch after bug#407794.
10:55 05feb: 3.0b3rc2 builds started
13:05 05feb: 3.0b3rc2 mac builds handed to QA
22:35 05feb: 3.0b3rc2 linux and signed-win32 builds handed to QA
23:15 05feb: Tier1 locales discovered broken at time of “Dev go to Build”.
15:20 05feb: figured out how to do rc3 for these locales without invalidating completed testing on rc2 builds
15:20 06feb: 3.0b3rc3 builds started
17:35 06feb: 3.0b3rc3 mac & linux builds handed to QA
21:20 06feb: 3.0b3rc3 signed-win32 signed builds handed to QA
16:05 07feb: 3.0b3rc3 update snippets available on betatest update channel
17:00 11feb: Dev & QA says “go” for Release; Build already completed final signing, bouncer entries
07:10 12feb: mirror replication started
09:55 12feb: mirror absorption good for testing to start on releasetest channel
12:50 12feb: QA completes testing releasetest.
14:00 12feb: website changes finalized and visible. Build given “go” to make updates snippets live.
14:45 12feb: update snippets available on live update channel
17:45 12feb: QA completes testing beta channel. Release announced

Notes:

1) The Build Automation used in FF3.0b3  included a bunch of fixes landed after FF3.0b2, which helped make things smoother. Despite the number of respins, all the housekeeping of the last few weeks paid off.

2) For better or worse, we are putting all our blow-by-blow scribbles public, so the curious can read about it, warts and all, here. Those Build Notes also link to our tracking bug#409880.

3) Build declared rc1 invalid because the release mozconfig used did not include a set of changes made for the nightly mozconfig. The changes to the nightly mozconfig were landed 13-dec-2007 as part of bug#407794, but corresponding changes were not made to the release mozconfigs. Its the first time we’ve had mozconfig changes land since we started using automation, and this uncovered a hole in our process: neither the automation, nor any humans verified these two mozconfigs before starting builds. We will now manually diff these before starting, and are working to automate this in bug#386338.

4) At the time Build was given a “go” to start builds, the locales es-ES, jp, jp-mac and pa-IN were all broken. As some of these were Tier1 locales, there was quite some effort put into figuring how to rebuild just those locales without invalidating the test work already completed in all the other locales. Once we figured out how to do that, we agreed to respin to uptake these locales. For clarity, *all* locales were called rc3, even though most were identical to rc2, and the only difference between rc2 and rc3 was the addition of the 4 locales.

5) Like before, we waited until morning to start pushing to mirrors. This was done so mirror absorption completed as QA were arriving in the office to start testing update channels. We did this because we wanted to reduce the time files were on the mirrors untested; in the past, overly excited people have post the locations of the files as “released” on public forums, even though they are not finished the last of the sanity checks. Coordinating the mirror push like this reduced that likelihood just a bit.

6) Mirror absorption took just under 3 hours to reach all values >= 60%, our usual threshold.

take care

John.