“we are all remoties” at MozCamp Singapore

[UPDATE: The newest version of this presentation is here. joduinn 12feb2014, 09nov2014]

In November, I was asked to present “we are all remoties” at MozCamp Singapore. In the end, I ended up presenting twice! The second time was on the main stage, in the largest room, where the keynote was held.

Giving a presentation in a room that big is always daunting, but during the presentation, it was encouraging to see the people that had been hovering at the back near the coffee machines + snacks gradually move to the remaining empty seats near the front and start taking notes. After the talk, I spent the rest of the day answering lots of questions and getting encouraging feedback.

Interesting that most of the people who came looking to attend “the remoties talk” had either heard an early version of it at Mozilla Summit in 2010, or heard about it from someone who was there. The people who heard it before thought it a good refresher; the people who were hearing it for the first time found it immediately useful to their day-to-day working lives! All self-confessed that this was a talk they never thought they’d find interesting so they almost skipped… but now thought it was essential, and wanted to know if I would I give the same presentation with their group?!?

Humbling. And encouraging. All at the same time.

Every time I get to talk about “remoties”, whether in a formal setting like MozCamp, or in discussions with people in other companies, I have two strong feelings:

  1. Passionate: I feel more and more convinced this topic is super important to the Mozilla community. In the changing face of the software industry, I feel this is becoming important to an increasing proportion of workplaces outside of Mozilla. Given Mozilla’s origins, we have a long standing reputation for successfully working with people in different physical locations. As we grow, we need to learn how to scale this part of our DNA. I feel if any organization can do this right, and show the way for other organizations to do it right, Mozilla can. The impact on the industry cannot be overstated.
  2. Embarrassed: In preparation for each talk, I pour over the slides, fix typos, rehearse and generally try to make it better. Every time, I fix lots of errors. And literally every time on stage, I find even more errors. Feedback and questions afterwards make me tweak the presentation every time. After my recent presentation at Netflix, I completely rewrote most of the presentation. Each time I do this, I feel better about the revised version, and embarrassed by the earlier versions.

Therefore, I’ve posted a PDF of the slides here.

Please do ask questions and/or give feedback/corrections/suggestions – either in comments below, or by emailing me. I’ll do my best to work them all into a revised presentation before the next talk which is already scheduled for outside Mozilla (more news soon!).


Infrastructure load for December 2012

  • #checkins-per-month: We had 5,333 checkins in December 2012. This is down from our all-time record of 5,893 in October, but still the 5th highest number of checkins in 2012. As you’ll see below, we were on track to set another record for the month when checkins declined because the “holiday effect”.
  • As usual, we handled this load with >95% of all builds consistently being started within 15mins. Sadly, our test pools continue to have a hard time, both with the increased rate of checkins, and the ever-increasing number of test suites being run per checkin. We’re continuing to work on buying and powering up more test machines, so please continue to bear with us. Meanwhile, if you know of any test suites that no longer need to be run per-checkin, please let us know so we can put scare test CPU to better use.
  • #checkins-per-day: During December, 14-of-30 days had over 200 checkins-per-day, and we peaked on 11dec with 296 checkins. Of interest here is the “holiday effect”: the significant drop in checkins which started on 21st (the Friday of the weekend before Christmas Eve), and continued to New Years Eve at the end of the month. If the rate of checkins in the first 20days of the month had continued, we’d have easily exceeded 6,000checkins per month and set a new all-time record.
  • #checkins-per-hour: Checkins are still mostly mid-day PT/afternoon ET, although they seem to be flattening out a bit. Instead of one specific hour spiking load above others on average 5 hours of every day sustained over 10 checkins-per-hour, and an additional 3 hours of every day sustained 9 checkins-per-hour.

mozilla-inbound, mozilla-central, fx-team:
Ratios of checkins across these branches remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch, with 27.8% of all checkins, consistently far more then the other integration branches combined (fx-team has 0.7% of checkins, mozilla-central has 2.1% of checkins). As usual, very few people land directly on mozilla-central – in fact more people go through approval process to land on mozilla-aurora.

mozilla-aurora, mozilla-beta:

  • 4.6% of our total monthly checkins landed into mozilla-aurora. This is an decrease from last month, but still higher then is typical. I believe this is caused by the number of b2g changes being landed into aurora and beta.
  • 3.2% of our total monthly checkins landed into mozilla-beta. As predicted last month, the recent transition of b2g to beta in late November caused increased checkins on beta for December.

(Standard disclaimer: I’m always glad whenever we catch a problem *before* we ship a release; it avoids us having to do a chemspill release and also we ship better code to our Firefox users in the first place.)

misc other details:

  • Pushes per day
    • You can clearly see weekends through the month.

  • Pushes by hour of day
      It is interesting that mid-morning PT is consistently the biggest spike of checkins during the day. I wonder if this is caused by ET developers doing checkins immediately after lunch, at the same time as PT developers have just settled into the office after coffee and initial emails?

“we are all remoties” at Netflix

[UPDATE: The newest version of this presentation is here. joduinn 12feb2014, 09nov2014]

Netflix asked me to present about how Mozilla handles distributed work groups – “we are all remoties” – in October. This invitation came about because Netflix RelEng team were impressed by the scale and efficiency of Mozilla’s RelEng group – and then totally impressed when they found out that Mozilla’s RelEng group was physically all remoties. Unheard of in Netflix.

Exciting, and a little daunting, all at the same time. Oh, and by the way, could it be recorded as part of their Netflix University training series?

To set context, its worth noting that Netflix has an explicit zero-remoties hiring policy, so this presentation generated quite some debate beforehand and during the Q+A sessions and afterwards.

Big thanks to everyone from Netflix who attended – the genuine curiosity and very direct, honest questions, with me and with each other, were great. After 5.75 years (and counting) in Mozilla’s very-distributed RelEng, I still forget that what feels “normal” for me is atypical for a lot of other companies. All the discussions helped me identify a bunch of assumptions that need to be called out in the presentation. Every time I have a chance to talk about remoties like this, I end up restructuring the presentation yet again to highlight missed assumptions. Thanks to all the Q+A here, the “remoties” presentation at MozCamp Singapore a month later was quite different and I hope much better (separate blog post coming).

Its still surprising to me how much I care about group organization. Done badly, its a big impediment to people getting their work done. Done well, it helps people be more effective. And, as noted by several people at Netflix, many aspects of our we-are-all-remoties group organization practices help even zero-remotie groups be more effective.

Many thanks to Curt Patrick, Gareth Bowles, Carl Quinn and Adrian Cockroft for helping make this happen, as well as for all the lively discussions before and since.