The RelEng story for 2010

…and here is our 2010 story. For perspective, click to see our 2009 story and our 2008 story.

Let us know what you think! Does this feel like the right focus? Does this address what feels most important to you?

This story should help us make sense of our goals across the four quarters of 2010. As usual, we focused on 4 areas during the year:

1) Continued streamlining of release infrastructure:
We now ship Major releases ever six months, all with partner builds, major updates etc. To keep up with this faster major release cycle, we continue to showcase our streamlined automation. For security releases, we now routinely provide 4-way, 5-way and occasionally 6-way “simultaneous-ship” releases.

This fast-paced major release cycle was possible because of our
continuing behind-the-scenes work on release automation, full featured project-branches, and our scalable machine infrastructure spread across multiple locations. In numbers, our growth path was:

  • May2007: 3 OS on 2 code lines on 89 machines
  • Dec2008: 7 OS on 5 code lines on 253 machines
  • Dec2009: 12 OS on 11 code lines on 550 machines
  • Dec2010: 19 OS on 20 code lines on 1000 machines

2) Continue to refine and simplify:
(…or “reducing our drag coefficient” so we can move faster.)
During 2010, we finally turned off the legacy Thunderbird 2 systems, after supporting those users on MoMo’s behalf since early 2008. We also turned off the Firefox3.0 systems. Both of these were the last of our cvs-based releases. Dropping both of those, combined with some enhancements we upstreamed to buildbot, meant we could continue to make improvements and reduce complexity in our automation. While the frantic nature of this work has reduced a bit since 2009, there’s still plenty of room for improvement that repays us back every time we do releases.

3) Outreach:
In addition to doing builds/tests/perf and releases, there are other ways we can use our infrastructure to help Mozilla. In 2009, we ran weekly code coverage jobs as the first jobs run on our machines outside of the traditional Firefox build/test/perf jobs. In 2010, we extended that further by running fuzzer jobs, and other code hygiene tools during idle times. We also helped Labs, and some other xulrunner partner projects, quickly scale and support users by running their jobs on our infrastructure – thus helping them avoid reinventing the wheel.

4) continued to improve Quality of Life:
The larger team has settled in together and continues to work together well under stress. Our shared skills keeps our bus factor good, and our quality-of-life healthy. We all did good work we were proud of, learned new things at conferences, taught each other new things, took vacations and improved our lives.

One thought on “The RelEng story for 2010

Leave a Reply