Firefox 3.0beta1 by the (wall-clock) numbers

Mozilla released Firefox 3.0 beta1 on Monday 19-nov-2007, at 11.10pm PST.

From “code freeze” to “release is now available to public” was 19 days 23 hours wall-clock time, of which Build&Release took 9 days and 1 hour.

23:59 31oct: code freeze for 3.0beta1
15:00 06nov: Dev says “go” to Build
18:25 06nov: rc1 builds started
20:30 06nov: win32 builds failed out (bug#346214)
22:30 06nov: win32 builds restarted after bug#346214 fixed on release branch
14:30 07nov: linux, mac and unsigned win32 builds handed to QA
17:25 07nov: rc1 abandoned (see details below)
17:25 07nov: rc2 builds started
17:20 08nov: rc2 builds abandoned (bug#402999)
19:05 08nov: rc3 builds started after bug#402999 fixed on release branch
17:30 09nov: linux & mac builds handed to QA
14:55 12nov: win32 signed builds handed to QA
18:15 16nov: Dev & QA says “go” for Release; Build starts final signing, bouncer entries
14:45 19nov: final signing, bouncer entries done; mirror replication started
23:10 19nov: announced

1) There is no Build automation running on trunk, so this release was done manually.
2) The rc1 builds were abandoned because of a manual error in how cvs was tagged. Two Build engineers were working in parallel to speed things up: one engineer typed PDT timezone into one machine, while the other engineer typed PST into another computer, so the two machines had an hour difference in what source timestamp to use for the builds. That one hour difference meant the generated builds missed one important last minute showstopper bugfix. This was totally a manual snafu within the Build team, and would have been avoided if automation was in place on trunk. (Ironically, daylight savings time only changed this same week; a week earlier this same snafu would have passed blissfully unnoticed!)
3) During rc1, there was a 4h20m delay while the Build team investigated a regxpcom test error at the end of the win32 build. Turns out the build was actually fine. The error was caused by a collision between the hourly build and production build processes running on the same machine at the same instant. Killing the hourly build and rerunning production worked fine.
4) By prior agreement, we did not create update snippets for this beta. Any users on earlier Alpha releases would not get updates refreshing them forward to beta1; instead Alpha users would have to manually download and install beta1. We do plan on creating update snippets for beta2 and beta3.
5) Because this was a Beta release, we did not do any “beta period” before releasing the beta! 🙂
6) Mirror absorption took 8 hours to reach the 70% threshold, not the usual 3 hours. In a random coincidence, there was a problem with one of the central rsync hubs in the mirror farm, and also one of the major mirrors, further compounded by problems when switching to backup servers. Dave Miller has all the drama details of late night pagers, and various mirror owners jumping to help (shout out to Shane!).

take care

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.