Busy morning yesterday

4 Comments

Yesterday morning, we did a 4-way sim-ship – we simultaneously shipped four different products: Firefox4.0beta8, Fennec4.0beta3, FirefoxHome1.1, Sync Addon for Firefox 1.6.

The cool new features in each of those releases are already covered elsewhere, so I’ll just focus on the mechanics and processes we went through to make this 4-way sim-ship happen.

  1. We’ve become used to our new ability to sim-ship different versions of Firefox smoothly and quickly (for example, shipping Firefox 3.5.x and 3.6.x security releases within 17 hours). However, yesterday was a very different experience for us. We did four releases instead of two. And more importantly, we did different products, not different versions of the same product – which meant different release processes for each of the four releases had to be cross-coordinated.
  2. Firefox 4.0beta8 was bumpy because of bugs we hit in some new RelEng automation code. Sadly all respins for Firefox 4.0b8 were caused by bugs in our RelEng automation. (More details to come soon in separate blogpost, after our postmortem.)
  3. While debugging one of these Firefox 4.0beta8 respins, we were distracted by a real fire alarm – the building fire alarms went off and we all had to evacuate while the fire department went running in looking for the fire. Luckily, while we were waiting outside, Rail discovered he was still within wifi range, so he was able to continue work on fixing the blocker problem. (kudos to Dustin for his impromptu extra support!)
  4. Fennec 4.0beta3 went really smoothly, until a late breaking problem discovered as we uploaded Fennec to the Google Marketplace. Fixing this caused Aki to do *two* complete rebuilds of Fennec, and then some further late night hacking afterward… all in ~10 hours. This super-fast turnaround was only possible because of months of preparation by Aki. Amazing work, Aki, truly amazing.

This was the first 4-way sim-ship we’ve done, which is impressive by itself, and we’ve also learned lots. In addition to the usual release mechanics, there was a lot of additional cross-project coordination to keep us on our toes. Its easy to ignore all the things that went smoothly, and focus on what we need to do better next time, but we should remember it *all*. I know there will be a next time, and I know we will do even better. As Murphy’s Law would predict, all these releases happened while several people were out sick, flying off to family vacations, while the Mozilla AllHands was in full swing and just before the Christmas vacations. For me, I was most impressed by how all the different people across Mozilla pitched in to help, all trusted each others professionalism and all worked together to get these releases out to our users. This was a great experience, so thank you to everyone!

4 Comments (+add yours?)

  1. Anonymous
    24 Dec 2010 @ 02:30:16

    So, what caused the false alarm?

    Reply

    • John
      24 Dec 2010 @ 11:46:23

      The false alarm was caused a water pressure surge from the Mountain View city water lines sent water into the building sprinkler pipes. No water actually came *out* of the sprinklers, but the pressure change was enough to fool the sensors into thinking that the sprinklers had been activated, thus set off the fire alarms.

      Of course, figuring this out took the MVFD and building maintenance people some investigation, while MVFD also searched the building to see if there was a real fire somewhere people hadn’t seen yet.

      (On a positive note, we now know that from the time the alarm started ringing, to the time MVFD were on scene was literally only a couple of minutes. They were *fast*, which is great to know in case there ever is a real fire!)

      Reply

  2. Dan
    24 Dec 2010 @ 05:33:41

    Checking for updates in the Addons manager doesn’t find the new version of Sync for me. It’s still 1.5.1.

    Reply

    • John
      24 Dec 2010 @ 11:53:22

      hi Dan;

      Sorry to hear that. If it helps, I’m running Firefox4.0b8 (of course!) on OSX10.6.5, and I upgraded to Firefox Sync v1.6 without any problems. If you are still having problems upgrading to Sync v1.6, can you let us know what version of Firefox you are running, and what OS you are on?

      Reply

Leave a Reply