23:59 04dec: Dev closes the tree, allowing only blockers from now on
23:59 07dec: Dev declares code freeze for 3.0beta2
11:15 10dec: Dev says “go” to Build
12:05 10dec: rc1 builds started
16:20 10dec: mac builds handed to QA
17:05 10dec: linux builds handed to QA
02:50 12dec: signed win32 builds handed to QA
18:30 12dec: linux & mac updates available on betatest; win32 updates still being debugged
08:10 13dec: linux & mac & win32 updates available on betatest
17:30 17dec: Dev & QA says “go” for Release; Build starts final signing, bouncer entries
10:00 18dec: final signing, bouncer entries done; mirror replication started
12:10 18dec: mirror replication completed
19:40 18dec: website updates finished, announced

1) This was our first attempt using Build automation on trunk, so we had some teething problems keep us on our toes. The curious can look through: 407077, 404340, 405042, 405153, 406601, 406613, 406640, 406602 407177 407275 407333 407351 407670 407825 407962 407988 408058 408610 408928 408934.
2) One of our automation VMs failed out with NetApp I/O error during the builds. This was after tagging+branching completed, so we were able to reopen trunk while debugging this. Problem recurred after a reboot, but went away after a 3rd reboot. We’ve seen this intermittently in the past with a few VMs, but so far its still unresolved. Current theory is that we need a kernel update on our VM, being applied now during this downtime after the release. See details in bug#407796.
3) The tree closure work by Dev in the previous week really helped with stability. For Beta1, we had to create rc1, rc2, rc3. However, for Beta2, we only had to create rc1. Nice! Thank you!!
4) From the time Dev said “go” at 11:15, until Build reopened trunk at 14.10pm was approx 3 hours. This speedy reopening of the tree meant that Dev didnt have a huge backlog of pending changes that needed to land in a carefully coordinated manner, like what happened after the week-long closure during the Beta1 release. This was good news for Dev, and a direct consequence of using automation.
5) Having the builds done early meant that QA had time to test before we went into TestDay. It was a nice bonus to go into a TestDay with bits that QA had already tried out, and to get a full TestDay of testing during this release cycle!
6) As part of beta2, we generated updates to bring forward beta1 users to beta2. As automation was going ok, we decided to also bring forward users from every previous alpha and beta (gecko1.9a1, 1.9a2, 1.9a3…1.9a8, 3.0b1) up to 3.0b2. Testing found problems when updating from 1.9a2, 1.9a3, 1.9a4. Updating from all other alphas, and from beta1, were all confirmed to be ok, so this was deemed not a showstopper. We deleted the generated updates for 1.9a2, 1.9a3, 1.9a4, and pushed out updates for all the other versions. After some debugging, it turned out to be a problem in Firefox3.0 places code introduced between alpha1-2 and fixed between alpha4-5. We expect this to be fixed in time for Beta3, for details see bug#408443.
7) Kubla, our new website Content Management System, was used for the first time to get firstrun, release notes and announcements visible on our websites. We had a couple of teething problems with this, but overall this looks like a nice step forward for us. Wil posted some details about Kubla here; its well worth a read.
8) An interesting side effect of speeding the release up this much was having to wait for ReleaseNotes and website changes to catch up! A good problem to have! 🙂 Seriously, the real fix here is to make sure that schedule changes on the release are widely advertised across all people working on the release; somehow only certain subsets of people were informed of the earlier ship date. This meant some people who were ahead of schedule for the Friday release date suddenly had to play rush-catch-up for the Tuesday release date. It was one of the topics covered in our postmortem.
9) Because this was a Beta release, we did not do any “beta period” before releasing the beta! 🙂
10) Mirror absorption took just over 2 hours to reach the > 60% threshold, faster than usual.

Overall, the Build team had a hectic week chasing down various automation teething problems & bugs, but the automation worked quite nicely. While shipping Beta2 still took longer then setting up the China Data Center (!), Beta2 was much smoother and quicker than Beta1!
take care
John.