Mozilla released Firefox 22.214.171.124 on Thursday 07-feb-2008, at 16:45pm PST. From “Dev says go” to “release is now available to public” was almost 11 days (10d 21h 30m) wall-clock time, of whichÂ Build&Release took almost 4 days (95h20m). Sadly, not all important milestones were recorded, so if anyone have info for the times marked ??:?? below, please let me know and I’ll update.
19:10 28jan: Dev says “go” for rc1
19:45 28jan: 126.96.36.199rc1 builds started
22:45 28jan: 188.8.131.52rc1 linux builds handed to QA
02:05 29jan: 184.108.40.206rc1 mac builds handed to QA
04:00 29jan: 220.127.116.11rc1 win32 signed builds handed to QA
12:00 29jan: update snippets available on betatest update channel
13:00 30nov: bug#414856 declared showstopper, Dev says “go” for rc2.
14:50 30jan: 18.104.22.168rc2 builds started
17:00 30jan: 22.214.171.124rc2 linux builds handed to QA
18:55 30jan: 126.96.36.199rc2 mac builds handed to QA
??:?? ??jan: 188.8.131.52rc2 unsigned-win32 builds waiting to be signed when QA discovered that bug#413250 has another showstopper exploit path
14:40 31jan: Dev says “go” for rc3
14:41 31jan: 184.108.40.206rc3 builds started
16:45 31jan: 220.127.116.11rc3 linux builds handed to QA
18:35 31jan: 18.104.22.168rc3 mac builds handed to QA
10:05 01feb: 22.214.171.124rc3 signed-win32 signed builds handed to QA
18:05 01feb: bug#415292 declared showstopper….
20:00 01feb: Dev says “go” for rc4
20:40 01feb: 126.96.36.199rc4 builds started
01:25 02feb: 188.8.131.52rc4 linux & mac builds handed to QA
04:35 02feb: 184.108.40.206rc4 signed-win32 signed builds handed to QA
08:20 02feb: 220.127.116.11rc4 update snippets available on betatest update channel
10:30 04feb: QA says “go to beta”.
11:25 04feb: update snippets on beta update channel
09:00 07feb: Dev & QA says “go” for Release; Build already completed final signing, bouncer entries
09:20 07feb: mirror replication started
11:45 07feb: mirror absorption good for testing to start on releasetest channel
14:40 07feb: QA completes testing releasetest.
??:?? 07feb: website changes finalized and visible. Build given “go” to make updates snippets live.
??:?? 07feb: update snippets available on live update channel
16:45 07feb: release announced
1) The Build Automation used in FF18.104.22.168Â included a bunch of fixes landed after FF22.214.171.124, which helped make things smoother. Despite the number of respins, it felt like all the housekeeping of the last few weeks paid off.
2) For better or worse, we are putting all our blow-by-blow scribbles public, so the curious can read about it, warts and all, here. Some highlights were:
- win32 builds failed out first time because buildbot slave was not restarted correctly. These need to be configured to auto-reboot cleanly.
- mac build failed out first time and had to be restarted. This has happened to us intermittently before, and we dont know yet how to reproduce or fix this.
- Long standing, but previously unknown, bug#414966 was fixed. This involved portions of tag respin code using UTC, while other portions used PST. If the initial rc1 build was started after 4pm, this bug would cause some parts of automation to expect one date, while other portions would expect another date, and show up as a problem if we did a respin. As far as we can tell, this bug has always been present, and we have just been “lucky” until now.
- Long standing bug#388524 was finally fixed. This caused users on “beta” channel to be offered complete updates, even when partial updates were available for them. We now serve partial updates to beta channel users whenever possible.
3) Like before, we waited until morning to start pushing to mirrors. This was done so mirror absorption completed as QA were arriving in the office to start testing update channels. We did this because we wanted to reduce the time files were on the mirrors untested; in the past, overly excited people have post the locations of the files as “released” on public forums, even though they are not finished the last of the sanity checks. Coordinating the mirror push like this reduced that likelihood just a bit.
4) Mirror absorption took 2.5 hours to reach all values >= 50%. However, this felt a little low, so we’ll go back to a 60% threshold for future releases.