• September was special. Our previous record was to run 52,000 test jobs in a 24 hour day on 27aug… impressive by any standards. But in September, we blew past that record twice: we handled 66,456 test jobs on 11sep, and then we handled 73,453 test jobs in a 24 hour day on 17sep. Stunning, simply stunning.
• #checkins-per-month: We had 7,580 checkins in September 2013. This is ~2% below last month’s record 7,771 checkins.
  
  

• #checkins-per-day: We hit 416 checkins on 03sep; impressive, yet still below our previous single-day record of 443 checkins on 26aug. During September, yet again all working days were over 200 checkins per day… In fact, if you exclude Friday 08sep and Monday 15sep when people were traveling for the b2g workweek, our weekday load throughout the month was 285 checkins per day, or higher. 19-of-30 days had over 250 checkins-per-day, 13-of-30 days had over 300 checkins-per-day. 2-of-30 days had over 400 checkins-per-day.
  
• #checkins-per-hour: Checkins are still mostly mid-day PT/afternoon ET. For 8 of every 24 hours, we sustained over 12 checkins per hour. Our heaviest load time this month was 10am-11am PT 15.73 checkins-per-hour (a checkin every 3.8 min – a new record.
  

mozilla-inbound, b2g-inbound, fx-team:

• mozilla-inbound continues to be heavily used as an integration branch. As developers start to use other *-inbound branches, we saw use of mozilla-inbound at 17.4% of all checkins is still much reduced from typical, yet only slightly higher then last month which was the lowest ever usage of mozilla-inbound. The use of multiple *-inbounds is clearly helping improve bottlenecks (see pie chart below) and the congestion on mozilla-inbound is being reduced significantly as people use switch to using other *-inbound branches instead. This also reduces stress and backlog headaches on sheriffs, which is good. All very cool to see and a definite part of the reason we continue to hit new records this month.
• b2g-inbound continues to be a great success, with 8.8% of this month’s checkins landing here, a slight increase over last month’s 8.2% and further evidence that use of this branch is stabilizing.
• With sheriff coverage, fx-team is clearly a very active third place for developers, with 5.5% of checkins this month, This is a slight drop from last month, but use also appears to be stabilizing. Having sheriff coverage clearly made a difference.
• The combined total of these 3 integration branches is 31.7%, which is fairly consistent. Put another way, sheriff moderated branches consistently handle approx 1/3 of all checkins.

  

mozilla-aurora, mozilla-beta, mozilla-b2g18, gaia-central:
Of our total monthly checkins:

• 2.3% landed into mozilla-central, slightly higher than last month. As usual, very few people land directly on mozilla-central these days, when there are sheriff-assisted branches available instead.
• 1.7% landed into mozilla-aurora, about the same as last month.
• 0.7% landed into mozilla-beta, slightly lower than last month.
• 0.3% landed into mozilla-b2g18, slightly lower then last month. This should quickly drop to zero as we move to gecko26.
• Note: gaia-central, and all other gaia-* branches, are not counted here anymore. For details, see here.

misc other details:
As usual, our build pool handled the load well, with >95% of all builds consistently being started within 15mins. Our test pool is getting up to par and we’re seeing more test jobs being handled with better response times. The peak per-day test load for September was insane: our previous record was 52,000 test jobs on 27aug… which we blew right past when we handled 66,456 test jobs on 11sep, and then again when we handled 73,453 test jobs a week later on 17sep. Still more work to be done here, but very encouraging progress.

As always, if you know of any test suites that no longer need to be run per-checkin, please let us know so we can immediately reduce the load a little. Also, if you know of any test suites which are perma-orange, and hidden on tbpl.m.o, please let us know – thats the worst of both worlds – using up scarce CPU time and not being displayed for people to make use of. We’ll make sure to file bugs to get tests fixed – or disabled – every little bit helps put scarce test CPU to better use.