Background:
At the Mozilla AllHands in Sept2011, I was surprised to find that my proposed session on git was accepted, and even more surprised at how well this session was attended! The room was full of people from all different groups across Mozilla. And boy, were people passionate. The slides don’t capture the lively back-forth discussions, but this is obviously a topic that people care deeply about, hence my blog post.

Here’s a quick summary of the main speaking points:

• Some projects in Mozilla are now basing their code in github.com, not on Mozilla’s existing release infrastructure. They do this despite the fact that github in one sense has reduced functionality (doesnt do all builds/test/automated-regression alerts/etc), and despite that using github like this causes extra manual headaches of periodically importing/exporting code, and complicates branch mechanics of releases. So why do this?
• Theory#1: git vs hg
  • Some people prefer to use the git commandline tool. At same time, some people prefer to use the hg commandline tool. Each Distributed Version Control System has various technical merits, and drawbacks, so I put this git-vs-hg debate into the same category as an emacs-vs-vi debate. (I do *not* mean that as a put down – I mean this in the nicest possible way – the reality of life is that people have their own unique hard-earned preferences)
• Theory#2: github vs hg+bugzilla+graphserver+tbpl
  • Instead of this debate being about the mechanical differences of the command line tools, I instead believe the debate is actually about Developer Workflow. This is not just theoretical; it makes a day-to-day difference to how developers get their job done.
  • Every mozilla developer ends up using hg+bugzilla+graphserver+tbpl. These are organically grown, over the years, and each went from hacky-experiments-to-mission-critical once people started to rely on them. However, they are not smoothly cross-integrated.
  • Mozilla saw a jump in checkins-per-day when tbpl.mozilla.org went live; tree closures were shorter, in part because people could more easily figure out who broke the build, and who/what to back out. (Of course, there’s more to it then *just* tbpl, but the usability issue here is the key point).
  • While different people have talked about this, in different companies over the years, I think github.com is a really compelling proof point that good developer workflow makes a difference. If you can make it easier for a developer to find a bug, get the bugfix reviewed and landed, and be able to see if it made a difference, developers will WANT to use your production systems.
  • Apart from workflow, there’s also security and autonomy concerns. How valuable are your bug discussions, review history, regression tracking? Who else would you entrust this valuable data to? If you find someone you trust, how do you know they’ll be around as long/longer then Mozilla?

  Hal Wine has now got up-to-speed in this area because of his work on git-staging. Next, Hal is starting to see how feasible would it be to support both git command line tools and hg command line tools in our production RelEng + WebDev + IT systems. And meanwhile, Hal, LauraT, myself and some others are gathering to see what we can do to improve developer workflow. As Mozilla grows, its important to make it easier to do coding work here – after all, this is one way to encourage more people to contribute, and to help Mozilla scale.

  If you have suggestions, or want to help, we’d love to hear from you – or if you prefer, you can just follow the tip of the iceberg in bug#713782.

I’m really excited to welcome Hal Wine to Release Engineering.

Hal has lots of experience in the trenches of Release Engineering, with a rare combination of experience working in distributed groups (not just solo!), working on small/embedded systems and working at scale. To make him even more unique, he’s a really nice guy *and* he even wears an occasional Hawaiian shirt! The only down side is that we’re now rethinking our previously lenient policy on bad puns 🙂

He’s reporting to me, and already helping bring more tegras online to help close out the “android as tier 1” project, while he’s also got his teeth sunk into a large interesting skunkworks project (more details coming soon in a separate post).

Hal’s based here in the San Francisco office, but you can find him on irc as “hwine”. Please do stop by and say hi – he’s already getting quite settled in as part of the family.

[UPDATE: Hal’s blog is now up and running here. joduinn 03-jan-2012 ]

I’d like to welcome John Hopkins to Release Engineering. I apologize that this welcome is a very belated welcome; John started in RelEng on 19th Sept and coop already posted a welcome here.

From one perspective, its not really any change here, John had already been coming to the RelEng group meetings since earlier this year when Mozilla Messaging folded back into Mozilla Corporation. Even before that, John was a familiar face to us, working on lots of the same buildbot and release automation code as the rest of RelEng.

From another perspective, there is potential for significant change here. Are there economies of scale if we merge the existing Thunderbird release engineering systems and the existing Firefox release engineering systems together? Now that we are in rapid-release cycles for both products, could we sim-ship Thunderbird with Firefox? How would we handle long term support issues for both products? Some things seem obvious, like how the tool chains for compiling-and-linking are identical for both products, so sharing machines and setup would help, and how we can improve Mozilla’s busfactor – jhopkins is the only person in RelEng who has shipped Thunderbird release since the old TB2 automation. Some things are less obvious, like what to do with the different spec hardware which was being used by MoMo. All this will play out over the coming months. Suggestions, comments, ideas all welcome!

As a good sign of how well everyone already works well together, the same week jhopkins officially joined RelEng, we discovered a serious problem: the production MoMo machines for Thunderbird were sitting on a shelf in the old Vancouver office, and these production machines were a few days away from being powered off as part of the move to a new Vancouver office. As the new office wasnt ready, so Vancouver people had to move to a temp-workspace for some weeks, and the Thunderbird production machines couldnt come to the temp-workspace offices. This meant the Thunderbird production machines would be offline, and the trees closed for the duration. No-one liked the idea of closing the Thunderbird trees for weeks, so we all piled in to help.

An immense volume of behind-the-scenes work took place, the Thunderbird continuous integration processes were migrated to our existing RelEng colos, toolchains setup on slightly different spec machines, and then production builds+tests enabled in the RelEng colo – all before the old Vancouver office was vacated, and all without closing the Thunderbird trees at all.

Bug#688230 has all the details. As I said in the “Friends of the Tree” section of the Mozilla Foundation weekly call at the time, this was an impressive volume of behind the scenes work, and developers using production systems never hit any problems!

Welcome John!