Mozilla released Firefox 2.0.0.7 on Tuesday 18-sep-2007, at 3pm PST. For background on this security firedrill, see here.

This was our first production run using the new automation, so I thought the following wall-clock numbers might be interesting. From “initial report” to “fix available to public” was 6.25 day wall-clock time. Of that, Build&Release took just under 2 days (45 hours).

09:00 Wed: bug reported 9am (or 8.30am?). Dev start working on fix
13:40 Fri: fix landed on 1.8 branch
14:30 Fri: build started
18:30 Fri: linux builds handed to QA
22:30 Fri: mac builds handed to QA
22:30 Fri: win32 unsigned builds handed to QA
11:58 Sat: win32 signed builds handed to QA (1st time)
01:30 Sun: win32 signed builds handed to QA (2nd time, rebuilt on old
machine)
12:10 Sun: update snippets pushed to beta update channel
15:00 Tue: update snippets pushed to live update channel; announced

Full disclaimer, while this fast turnaround kept Mike Shaver happy, it was not yet a “human free” release. We hit 4 issues, which required manual intervention:

1) last minute question about possible CVS-cross-branch tagging problem in automation scripts. Problem unconfirmed, but decided to manually tag anyway, just to be safe. Problem still unconfirmed, but test case now designed to clarify for future releases (see bug#396290)

2) l10n builds on win32 had the wrong cr-lf settings in README, EULA. This root cause of this was an internal communications snafu within the Build&Release group. Historically, we build l10n win32 on different machines to win32 en-US machines. As part of automation rollout, some folks thought the l10n win32 builds were now being done on same machines as en-US for 2005+2006, some thought l10n win32 was still being built on different machines. Because these different machines have different cygwin cr-lf settings, this problem first surfaced as a problem where text files like README, EULA had the wrong cr-lf settings. It was caught by a recently added test. Rather the debug/fix the problem, we just built on the old l10n machine and shipped that for win32. This miscommunication has been clarified. Still checking if there’s anything else here we missed.

3) signing still done manually. This is known and expected. Note: as the step-before-signing finished late at night, the automation waited overnight until human woke up and did the signing the next morning.

4) manually copying bits from stage to build-console after each step completed. This was a known issue that we expected to have fixed for the scheduled 2007 release, but was not yet in place when this Firefox2.0.0.7 firedrill started. After each step finished, we had to manually copy files between “stage” and “build-console”, so that the next step would find the files it was expecting. Was intrusive and annoying. On track to be completed before end sept. (see bug#396438)

tc
John.
=====
ps: After the release, we’ve heard a few questions about the new GPG key. The previous key had expired sept2006, and was still being used, until this new key was available in August2007. We used the new key in Firefox3a7, and also in Firefox2007. After the Firefox2007 release, some questions about how to confirm the new public signing key on key servers. We’ve reviewed the keys on key servers, and they seem ok, but are still investigating. (see bug#377781).