Infrastructure load for May 2013

  • #checkins-per-month: We had 6,041 checkins in May 2013. This is down from last month’s 6,364 checkins, and down from our record of 6,433 in Mar2013.

    Overall load since Jan 2009

  • #checkins-per-day: We had 293 checkins checkins on 21may. During May, 21-of-31 days had over 200 checkins-per-day – thats almost every working day. Only 7-of-31 days had over 250 checkins-per-day, much lower than usual. Unusually, we never exceeded 300 checkins-per-day any day this month – first time in many months.
  • #checkins-per-hour: Checkins are still mostly mid-day PT/afternoon ET. For 5 of every 24 hours, we sustained over 10 checkins per hour. Heaviest load time this month was 1pm-2pm PT (12.5 checkins-per-hour).
  • As usual, our build pool handled the load well, with >95% of all builds consistently being started within 15mins. Our test pool continues to improve. All the hard work by RelEng, ATeam and IT is paying off, we’re seeing more test jobs being handled with better response times. The peak for May was 54,195 test jobs on 23may. Still more work to be done here, but encouraging.

    As always, if you know of any test suites that no longer need to be run per-checkin, please let us know so we can immediately reduce the load a little. Also, if you know of any test suites which are perma-orange, and hidden on tbpl.m.o, please let us know – thats the worst of both worlds – using up scarce CPU time and not being displayed. We’ll make sure to file bugs to get tests fixed, or disabled – every little bit helps put scarce test CPU to better use.

mozilla-inbound, birch/b2g-inbound, mozilla-central, fx-team:
Ratios of checkins across these branches remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch, dropping slightly to 24.2% of all checkins, yet still consistently far more then all other integration branches combined. mozilla-central increased slightly to 2.3% of checkins.

The “birch as b2g-inbound” experiment continued to be a great success, and with 4.2% of this month’s checkins landing here, birch has already become the 4th busiest branch (after try, mozilla-inbound and gaia-central). Birch is also helping reduce pain of any mozilla-inbound closures, and further proving the lure of sheriff-assisted-landings to developers. Given the success of this experiment, bug#875989 tracks setting up b2g-inbound on a permanent basis.

The lure of sheriff assistance continues to be consistently popular, and as usual, very few people land directly on mozilla-central these days. The fx-team branch saw an increase to 2.1% of checkins, which I believe can be attributed to the increased help with landings from sheriffs (and sometimes gps) this month.

Infrastructure load by branch

mozilla-aurora, mozilla-beta, mozilla-b2g18, gaia-central:
Of our total monthly checkins:

  • 1.7% landed into mozilla-aurora, slightly lower than last month.
  • 1.2% landed into mozilla-beta, slightly lower than last month.
  • 1.7% landed into mozilla-b2g18, slightly higher then last month.
  • 5.9% landed into gaia-central, same as last month. gaia-central continues to be the third busiest branch overall, after try and mozilla-inbound.

misc other details:

  • Pushes per day
    • You can clearly see weekends through the month. Its worth noting that we had >200 checkins-per-day almost every working day in May. This
      has been true for a few months now, so it is starting to feel like 200 checkins-per-day is the new “normal” for Mozilla.

    • Pushes by hour of day
        Mid-morning PT is consistently the biggest volume of checkins, specifically between 1pm-2pm PT, with 12.5 checkins-per-hour.

Leave a Reply