- #checkins-per-month: We had 7,051 checkins in July 2013. This is 20% above last month’s 5,893 checkins, and 10% above our previous all-time-record of 6,433 in Mar2013.
- #checkins-per-day: We had 370 checkins checkins on 02jul. During July, 20-of-31 days had over 200 checkins-per-day, 18-of-31 days had over 250 checkins-per-day. Of note, 10-of-31 days had over 300 checkins-per-day – a huge jump in load.
- #checkins-per-hour: Checkins are still mostly mid-day PT/afternoon ET. For 10 of every 24 hours, we sustained over 10 checkins per hour. Heaviest load time this month was 1pm-2pm PT (14.7 checkins-per-hour) – a new record.
As usual, our build pool handled the load well, with >95% of all builds consistently being started within 15mins. The use of multiple inbounds is really helping improve bottlenecks. Our test pool continues to improve. All the hard work by RelEng, ATeam and IT is paying off, we’re seeing more test jobs being handled with better response times. The work on fixing/disabling any tests that are hidden-yet-still-being-run is also improving our test situation. The peak for July was 55,983 test jobs on 18jul. Still more work to be done here, but very encouraging.
As always, if you know of any test suites that no longer need to be run per-checkin, please let us know so we can immediately reduce the load a little. Also, if you know of any test suites which are perma-orange, and hidden on tbpl.m.o, please let us know – thats the worst of both worlds – using up scarce CPU time and not being displayed for people to make use of. We’ll make sure to file bugs to get tests fixed – or disabled – every little bit helps put scarce test CPU to better use.
mozilla-inbound, birch/b2g-inbound, fx-team:
mozilla-inbound continues to be heavily used as an integration branch. Its noteworthy that as developers start to use other -inbound branches, we saw mozilla-inbound reduce significantly to 21.3% of all checkins. Its still consistently far more then all other integration branches combined, but you can see the congestion reduced as people use other *-inbound branches.
The “birch as b2g-inbound” experiment is officially a great success, with 7.9% of this month’s checkins landing here, birch has now become the 3rd busiest branch (after try, and mozilla-inbound). Birch is also helping reduce pain of any mozilla-inbound closures, and further proving the lure of sheriff-assisted-landings to developers. As of 01aug, the official “b2g-inbound” branch is officially open, on a permanent basis, to use instead of birch. I expect the percentage on this branch to stabilize in the coming weeks.
The fx-team branch increased slightly to 2.4% of checkins this month, as sheriffs coverage started late in the month. I expect the percentage on this branch to grow over August, as more people rely on sheriff support here.
The combined total of these 3 integration branches is 31.6%, showing just how much our sheriffs are helping.
mozilla-aurora, mozilla-beta, mozilla-b2g18, gaia-central:
Of our total monthly checkins:
- 1.8% landed into mozilla-central, slightly lower than last month. As usual, very few people land directly on mozilla-central these days, when there are sheriff-assisted branches available instead.
- 2.1% landed into mozilla-aurora, slightly higher than last month.
- 1.1% landed into mozilla-beta, slightly higher than last month.
- 1.2% landed into mozilla-b2g18, slightly lower then last month.
- Note: gaia-central, and all other gaia-* branches, are not counted here anymore. For details, see here.
misc other details:
- Pushes per day
- You can clearly see weekends through the month. Its worth noting that we had >200 checkins-per-day almost every working day in July. This has been true for a few months now, so it is starting to feel like 200 checkins-per-day is the new “normal” for a working day at Mozilla. Having 10-of-31 days over 300 checkins-per-day is a big deal.
- Pushes by hour of day
-
Mid-day PT is consistently the biggest volume of checkins, specifically between 1pm-2pm PT, with 14.77 checkins-per-hour, and 2pm-3pm PT, with 12.63 checkins-per-hour. Its interesting to see load spreading out across the day, with 10-of-every-24 hours sustaining over 10 checkins per hour. Heaviest load time this month was 1pm-2pm PT (14.7 checkins-per-hour) – a new record.
- You can clearly see weekends through the month. Its worth noting that we had >200 checkins-per-day almost every working day in July. This has been true for a few months now, so it is starting to feel like 200 checkins-per-day is the new “normal” for a working day at Mozilla. Having 10-of-31 days over 300 checkins-per-day is a big deal.
John O’Duinn: Infrastructure load for July 2013 – http://t.co/zirPZAKGcg #mozilla #mozillians mozilla