Infrastructure load for November 2012

  • #checkins-per-month: We had 5,646 checkins in November 2012. This is slightly down from last month’s record of 5,893, but still the 3rd highest number of checkins in 2012, after October (5,893 checkins) and August (5,803 checkins). Again, we handled this record load with >95% of all builds consistently being started within 15mins. Sadly, our test pools continue to have a hard time, both with the increased rate of checkins, and the increased number of test suites being run per checkin. We’re continuing to work on buying and powering up more test machines, please continue to bear with us. Meanwhile, if you know of any test suites that no longer need to be run per-checkin, please let us know.
  • #checkins-per-day: During November, 19-of-30 days had over 200 checkins-per-day. Worth noting was spikes of 295 checkins on 13nov and 301 checkins on 27nov.
  • #checkins-per-hour: The peak across this month was a new record 13.36 checkins per hour between 10-11am. Of interest, I note that for 25% of every day (6 of every 24 hours), we sustained over 10 checkins per hour.

mozilla-inbound, mozilla-central, fx-team:
Ratios of checkins across these branches remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch, with 26.9% of all checkins, consistently far more then the other integration branches combined (fx-team has 0.8% of checkins, mozilla-central has 2.2% of checkins). As usual, very few people land directly on mozilla-central – in fact more people go through approval process to land on mozilla-aurora.

mozilla-aurora, mozilla-beta:

  • 4.9% of our total monthly checkins landed into mozilla-aurora. This is an decrease from last month’s high, but still higher then is typical. I believe this is caused by the number of b2g changes being landed into aurora, which then stopped ~20th November, with the migration of b2g from aurora to beta.
  • 2.2% of our total monthly checkins landed into mozilla-beta. This is typical for beta. I suspect the recent transition of b2g to beta just before holiday season here in the US, means we didnt see many b2g-related checkins on beta for November and I do expect beta to show increased checkins for December.

(Standard disclaimer: I’m always glad whenever we catch a problem *before* we ship a release; it avoids us having to do a chemspill release and also we ship better code to our Firefox users in the first place.)

misc other details:

  • Pushes per day
    • You can clearly see weekends through the month.

  • Pushes by hour of day
      It is interesting that mid-morning PT is consistently the biggest spike of checkins during the day. I wonder if this is caused by ET developers doing checkins immediately after lunch, at the same time as PT developers have just settled into the office after coffee and initial emails?

2 thoughts on “Infrastructure load for November 2012