Month: February 2013

“We are all remoties” at Haas MBA, U.C.Berkeley

joduinn6 Comments

[UPDATE: The newest version of this presentation is here. joduinn 09nov2014]

Last weekend, I was super-honored to be a guest speaker at the Haas Berkeley MBA program. My session was part of their “Global Teams” module, where they cover the theory and practice of effective teamwork, managing in global companies, and managing in fluid/rapidly changing environments.

My host, Homa Bahrami, invited me to show how Mozilla’s Release Engineering group has pushed the envelope, and had developed a well-tested concrete set of tips+tricks which allow a geo-distributed group to work highly effectively.

People’s attention was caught right at the start by my summary and graphic showing just how distributed Mozilla’s RelEng group actually is:

    * 16 people
    * 15 locations
    * 4 non-adjacent timezones
    * 0 in “headquarters”


 

By comparison, most people think of remoties as either:
…or…

The fact that any group could work together this effectively while being so geo-distributed was startling to them. Add to that, the fact that this group has been able to create strategic-level improvements to Mozilla’s software development abilities, hence increasing Mozilla’s options in the marketplace, generated even more interest.

Overall, the entire session was lively and interactive, with great questions, and discussions back-forth across the room. Everyone was fully engaged all the way… even after the lunch food arrived, we continued the discussions in the corridor outside the room.

I was delighted by the insightful questions, and very interested to hear the different perspectives that everyone brought from their varied backgrounds outside the MBA program.

For me, personally, I found it re-affirming to hear that the tips+tricks that we’ve built up within RelEng over the years are applicable to other groups, and other organizations.

It was a thoroughly wonderful experience. Big thanks to Homa for the invite, and to everyone for their full-on engagement.

[For a PDF copy of the entire presentation, click here or on the smiley faces! For the sake of my poor blogsite, the much, much, larger keynote files are available on request.]

Infrastructure load for January 2013

joduinnLeave a comment
  • #checkins-per-month: We had 6,247 checkins in January 2013. This exceeds our previous all-time record of 5,893 in October2012.

    As usual, we handled this load with >95% of all builds consistently being started within 15mins. Sadly, our test pools continue to have a hard time, both with the increased rate of checkins, and the ever-increasing number of test suites being run per checkin. Some test jobs are now runnable on AWS, and are now running there. Some *should* run on AWS, but fail for some reason – work continues. And some tests *need* hardware, so we’re continuing work to buy and power up more test machines to build out capacity there; please continue to bear with us. Oh, and of course, if you know of any test suites that no longer need to be run per-checkin, please let us know so we can immediately reduce the load a little. Every little helps put scarce test CPU to better use.

  • #checkins-per-day: During January, 18-of-31 days had over 200 checkins-per-day, and 2-of-31 days had over 300 checkins-per-day (06jan had 307 checkins; 10jan had 302 checkins). The pattern is to be expected as both of these days were during our first full week back after holidays – typically our busiest week of the year…and this year coincided with a B2G workweek!
  • #checkins-per-hour: Checkins are still mostly mid-day PT/afternoon ET, but the load has increased across the day. For 33% of every day (8 of every 24 hours), we sustained over 10 checkins per hour. Heaviest load times this month were 1-2pm PT (13 checkins-per-hour) and 2-3pm PT (13.36 checkins-per-hour – which matched our previous record of 13.36checkins-per-hour set in November2012!).

mozilla-inbound, mozilla-central, fx-team:
Ratios of checkins across these branches remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch, with 29% of all checkins, consistently far more then the other integration branches combined (fx-team has 1% of checkins, mozilla-central has 2.3% of checkins). As usual, very few people land directly on mozilla-central these days.

mozilla-aurora, mozilla-beta, mozilla-b2g18:

  • 3.1% of our total monthly checkins landed into mozilla-aurora. This is back down to normal aurora levels. This is expected since b2g changes are no longer being landed into aurora and beta.
  • 1.6% of our total monthly checkins landed into mozilla-beta. This is back down to normal beta levels (maybe even slightly lower). This is expected since b2g changes are no longer being landed into aurora and beta.
  • 4.6% of our total monthly checkins landed into mozilla-b2g18. These are all fixes *only* for the B2G releases, so important enough to be worth calling out here, like the aurora and beta branches.

misc other details:

  • Pushes per day
    • You can clearly see weekends through the month… and which week was the “first-week-after-holidays-combined-with-B2G-workweek”.

  • Pushes by hour of day
      Mid-morning PT is consistently the biggest spike of checkins, although this month the checkin load stayed high throughout the entire PT working day.