Infrastructure load for January 2012

Let’s get straight to the interesting stuff!

  • #checkins-per-month: January set another new record with 3,962 checkins for the month. Another new record volume of checkins, in our continuing series of new records. The previous record was December2011, with 3,262 checkins, and before that November2011 held previous record with 3,209 checkins.
  • #checkins-per-day: We set a new record of 210 checkins-per-day on 10-jan-2012. This wasnt just a one-day spike; 15 of the 22 working days of January had over 150 checkins-per-day.
  • Historically, we have always seen a increase in checkins in January, and within a week of returning from vacation is typically the highest load. The record 210 checkins-per-day happened on the 10th January. (I personally suspect this is the backlog from people working over the holidays? but dont know how to confirm this.)

mozilla-inbound, fx-team:
Its very cool to see how developers have taken to using mozilla-inbound (and more recently fx-team) as integration branches, instead of having everyone landing directly into mozilla-central.

  • In the chart above, note that the number of mozilla-central checkins (189) continues to decrease, while the number of checkins on mozilla-inbound (1050) has increased.
  • In the past whenever we had to unwind a large backlog of pending checkins, or back out a complicated bustage, we’d keep the mozilla-central tree closed for all checkins for the duration… which blocked all landings. However, now with mozilla-inbound and fx-team as integration branches in addition to mozilla-central, it means that developers have the option to continue landing their unrelated patches on an open integration branch while the cleanup work continues on the closed branch. In theory, if you give smart humans a easy way to route around a blockage, they’ll quickly start to use it so they can continue to get things done. In reality, very cool to see it actually happening.

mozilla-aurora, mozilla-beta:
I note that ~3.5% of our total monthly checkins landed into mozilla-aurora, and ~1% of our total monthly checkins landed into mozilla-beta. Put another way, a third of aurora checkins also landed onto mozilla-beta.

Part of me feels this is a healthy low number of fixes landing on aurora, and a healthy even lower number of fixes landing on beta. And this was part of the plan for the rapid release model.

At the same time, part of me also feels like too many fixes are still needed on mozilla-beta, and I fear that this is a sign too many bugs are not being detected earlier in the development cycle when on mozilla-central or mozilla-aurora. In which case, what could we do differently in order to change this? Honestly, I cant tell for certain, and I’d be curious what others think.

(Oh, and for the record, I’m always glad whenever we catch a problem *before* we ship a release; it avoids us having to do a chemspill release and we ship better code to our Firefox users.)

misc other details:
Pushes per day

Pushes by hour of day

3 thoughts on “Infrastructure load for January 2012