Mozilla released Firefox3.0beta4 on Monday 10-mar-2008, at 17:25pm PST. From “Dev says go” to “release is now available to public” was just over 7 days (7d 6h 10m) wall-clock time, of which Build&Release took just over 3 days (3d 2h 05m).

11:15 03mar: Dev says “go” for rc1
16:10 03mar: 3.0b4rc1 builds started
23:15 03mar: 3.0b4rc1 mac builds handed to QA
00:05 04mar: 3.0b4rc1 linux builds handed to QA
06:00 04mar: 3.0b4rc1 signed-win32 builds handed to QA
11:15 04mar: 3.0b4rc1 three missing linux locales were resolved and handed to QA. See bug#419771 and bug#407796 for details.
15:35 04mar: 3.0b4rc1 update snippets available on betatest update channel
08:35 07mar: 3.0b4rc1 showstopper: discovered win32 was compiled without PGO. Need to respin win32 builds. Mac and linux confirmed ok.
11:50 07mar: 3.0b4rc2 win32 builds started
00:05 08mar: 3.0b4rc2 signed-pgo-win32 builds handed to QA
14:00 08mar: 3.0b4rc2 update snippets available on betatest update channel
20:00 09mar: Dev & QA says “go” for Beta; Build already completed final signing, bouncer entries
07:00 10mar: mirror replication started
09:15 10mar: mirror absorption good for testing to start on releasetest channel
13:15 10mar: QA completes testing releasetest.
14:45 10mar: website changes finalized and visible. Build given “go” to make updates snippets live.
15:50 10mar: update snippets available on live beta update channel
17:25 10mar: QA completes testing beta channel. Release announced

Notes:

1) The Build Automation used in FF3.0b4 included a bunch of fixes landed after FF3.0b3, which helped make things smoother. Despite the respin, yet again, all the housekeeping of the last few weeks paid off.

2) For better or worse, we are putting all our blow-by-blow scribbles public, so the curious can read about it, warts and all, here. Those Build Notes also link to our tracking bug#418926.

3) It took us much longer then usual to start the builds.We had been distracted on other projects during the prior week, and not done *any* of the prerequesite setup work in advance of this release.
4) We hit bug#419771 and bug#407796 as fallout from the recent kernel updates on this machine, which delayed announcing win32 builds by a few hours.

5) In 3.0b4rc1, the win32 builds were confirmed to be compiled *without* the PGO compiler optimizer. This was a problem caused by how the new PGO compiler was being enabled in tinderbox, and was completely a Build snafu. The same changes were required to two copies of an identical config file, but we only updated one, and forgot about the other. We had to completely rebuild the win32 builds from the beginning, and verified the bits as they were being produced. Note that mac and linux builds did not have to be rebuilt, but to avoid confusion, we symlinked linux-rc1 -> linux-rc2 and mac-rc1 -> mac-rc2.

6) Like before, we waited until morning to start pushing to mirrors. This was done so mirror absorption completed as QA were arriving in the office to start testing update channels. We did this because we wanted to reduce the time files were on the mirrors untested; in the past, overly excited people have post the locations of the files as “released” on public forums, even though they are not finished the last of the sanity checks. Coordinating the mirror push like this reduced that likelihood just a bit.

7) Mirror absorption took 2 hours 15mins to reach all values >= 60%, slightly faster then our usual threshold.

take care

John.