Firefox by the (wall-clock) numbers

Mozilla released Firefox on Thursday 01-nov-2007, at 5.40pm PST.

From “do we need a release” to “release is now available to public” was 11 days 2 hours wall-clock time, of which the Beta period took 2.75 days, and Build&Release took 37 hours.

15:35 22oct: decide regressions introduced in FF2008 justify producing a quick FF2009 to address
12:30 25oct: Dev says “go”
14:40 25oct: 2009rc1 builds started
20:00 25oct: linux builds handed to QA
22:00 25oct: mac builds handed to QA
01:00 26oct: win32 signed builds handed to QA
19:40 26oct: update snippets on betatest update channel
16:30 29oct: QA says “go” for Beta
16:50 29oct: update snippets on beta update channel
10:40 01nov: Dev & QA says “go” for Release; Build starts final signing, bouncer entries
14:15 01nov: final signing, bouncer entries done; mirror replication started
17:15 01nov: update snippets on live update channel; announced

While Build Automation in FF2009 was much smoother than FF2008, this was not yet a “human free” release:
1) The talkback server had been renamed after the FF2.0.0.8 release shipped and before FF2.0.0.9 started, so our first automation run timed out at the end of the build, waiting for humans to answer the RSA “are you sure you want to connect to this machine” login question?! 🙁 We didnt detect this until the build overran the estimated completion time, but then after a quick fix, we were forced to rerun the entire build again. This would have been caught if our nightlies were part of the same build automation (see bug#401936)
2) We still manually do signing, adding bouncer entries, starting mirror replication and monitoring mirror replication, pushing snippets to beta channel, pushing snippets to release channel. Combined, these took 6.5 hours of the Build time, and are worthy of automation attention. Pushing updates snippets to betatest channel has been automated since the FF2008 release.
3) Mirror absorption took 3 hours to reach 72-80%. The mac DMG files always straggle much lower then everything else for mirror absorption, apparently a known problem with how webservers handle that file type, but new details are emerging in bug#402141. Experiments continue, but every time we do a release, we always give thanks to morgamic for giving us the tools to measure with!

take care

One thought on “Firefox by the (wall-clock) numbers

Leave a Reply