Infrastructure load for December 2012

1 Comment

  • #checkins-per-month: We had 5,333 checkins in December 2012. This is down from our all-time record of 5,893 in October, but still the 5th highest number of checkins in 2012. As you’ll see below, we were on track to set another record for the month when checkins declined because the “holiday effect”.
  • As usual, we handled this load with >95% of all builds consistently being started within 15mins. Sadly, our test pools continue to have a hard time, both with the increased rate of checkins, and the ever-increasing number of test suites being run per checkin. We’re continuing to work on buying and powering up more test machines, so please continue to bear with us. Meanwhile, if you know of any test suites that no longer need to be run per-checkin, please let us know so we can put scare test CPU to better use.
  • #checkins-per-day: During December, 14-of-30 days had over 200 checkins-per-day, and we peaked on 11dec with 296 checkins. Of interest here is the “holiday effect”: the significant drop in checkins which started on 21st (the Friday of the weekend before Christmas Eve), and continued to New Years Eve at the end of the month. If the rate of checkins in the first 20days of the month had continued, we’d have easily exceeded 6,000checkins per month and set a new all-time record.
  • #checkins-per-hour: Checkins are still mostly mid-day PT/afternoon ET, although they seem to be flattening out a bit. Instead of one specific hour spiking load above others on average 5 hours of every day sustained over 10 checkins-per-hour, and an additional 3 hours of every day sustained 9 checkins-per-hour.

Overall load since Jan 2009

mozilla-inbound, mozilla-central, fx-team:
Ratios of checkins across these branches remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch, with 27.8% of all checkins, consistently far more then the other integration branches combined (fx-team has 0.7% of checkins, mozilla-central has 2.1% of checkins). As usual, very few people land directly on mozilla-central – in fact more people go through approval process to land on mozilla-aurora.

Infrastructure load by branch

mozilla-aurora, mozilla-beta:

  • 4.6% of our total monthly checkins landed into mozilla-aurora. This is an decrease from last month, but still higher then is typical. I believe this is caused by the number of b2g changes being landed into aurora and beta.
  • 3.2% of our total monthly checkins landed into mozilla-beta. As predicted last month, the recent transition of b2g to beta in late November caused increased checkins on beta for December.

(Standard disclaimer: I’m always glad whenever we catch a problem *before* we ship a release; it avoids us having to do a chemspill release and also we ship better code to our Firefox users in the first place.)

misc other details:

  • Pushes per day
    • You can clearly see weekends through the month.

    #Pushes this month

  • Pushes by hour of day
      It is interesting that mid-morning PT is consistently the biggest spike of checkins during the day. I wonder if this is caused by ET developers doing checkins immediately after lunch, at the same time as PT developers have just settled into the office after coffee and initial emails?

    #Pushes per hour

One Comment (+add yours?)

  1. Grant
    Jan 21, 2013 @ 10:43:31

    Keep up the good work guys, very incredible!

    Reply

Leave a Reply