• #checkins-per-month: We had 5,893 checkins in June 2013. This is down from last month’s 6,041 checkins, and down from our record of 6,433 in Mar2013. Some of this reduction is because of changes made to how we count checkins on gaia-* branches (see details below), and some of this reduction feels possibly like summer-vacation-after-big-1.0-deadline.
  
  
  

• #checkins-per-day: We had 309 checkins checkins on 18jun. During June, 20-of-30 days had over 200 checkins-per-day – thats every working day, if we include Monday 03jun which had 199! Only 11-of-30 days had over 250 checkins-per-day, which is lower than usual.
• #checkins-per-hour: Checkins are still mostly mid-day PT/afternoon ET. For 7 of every 24 hours, we sustained over 10 checkins per hour. Heaviest load time this month was 2pm-3pm PT (12.6 checkins-per-hour), and 1pm-2pm PT (12.5 checkins-per-hour).

As usual, our build pool handled the load well, with >95% of all builds consistently being started within 15mins. Our test pool continues to improve. All the hard work by RelEng, ATeam and IT is paying off, we’re seeing more test jobs being handled with better response times. The peak for June was 47,154 test jobs on 26jun. Still more work to be done here, but very encouraging.


As always, if you know of any test suites that no longer need to be run per-checkin, please let us know so we can immediately reduce the load a little. Also, if you know of any test suites which are perma-orange, and hidden on tbpl.m.o, please let us know – thats the worst of both worlds – using up scarce CPU time and not being displayed for people to make use of. We’ll make sure to file bugs to get tests fixed – or disabled – every little bit helps put scarce test CPU to better use.

mozilla-inbound, birch/b2g-inbound, mozilla-central, fx-team:
Ratios of checkins across these branches remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch, increasing slightly to 24.9% of all checkins, yet still consistently far more then all other integration branches combined. mozilla-central increased slightly to 2.4% of checkins.

The “birch as b2g-inbound” experiment continued to be a great success, and with 7.2% of this month’s checkins landing here, birch has now become the 3rd busiest branch (after try, and mozilla-inbound). Birch is also helping reduce pain of any mozilla-inbound closures, and further proving the lure of sheriff-assisted-landings to developers. With other interrupts now handled, RelEng has now resumed work on bug#875989 to set up “b2g-inbound” on a permanent basis.

The lure of sheriff assistance continues to be consistently popular, and as usual, very few people land directly on mozilla-central these days.

The fx-team branch remained constant at 2.1% of checkins this month.

mozilla-aurora, mozilla-beta, mozilla-b2g18, gaia-central:
Of our total monthly checkins:

• 1.6% landed into mozilla-aurora, about the same as last month.
• 1.0% landed into mozilla-beta, slightly lower than last month.
• 1.3% landed into mozilla-b2g18, slightly lower then last month.

Note: gaia-central, and all other gaia-* branches, will not be counted here anymore. We made some changes in late May / early June to all gaia branches; one disabled running gaia-ui tests, another allowed us to generate gecko+gaia every time a more frequent gecko change landed, and also display on tbpl the exact gaia changeset used. This simplified developer+sheriff regression hunting, and also reduced confusion amongst developers. It also allowed us to stop generating unneeded builds on gaia-* branches, which in turn reduced our infrastructure load. Because of this, it doesnt make sense to continue counting checkins on gaia-* in the context of this blogpost, just like we don’t count checkins on l10n repos. I also note that these changes for gaia-* since 28may explains why the May checkin load was a little lower then originally expected. (thanks to catlee, hwine, nthomas and zeller for their help on this.)

misc other details:

• Pushes per day
  • You can clearly see weekends through the month. Its worth noting that we had >200 checkins-per-day almost every working day in June. This has been true for a few months now, so it is starting to feel like 200 checkins-per-day is the new “normal” for a working day at Mozilla.
  • Pushes by hour of day
    Mid-day PT is consistently the biggest volume of checkins, specifically between 1pm-2pm PT, with 12.5 checkins-per-hour, and 2pm-3pm PT, with 12.63 checkins-per-hour. Its interesting to see load spreading out across the day, with 7-of-24 hours (8am-2pm PT) having over 10 checkins-per-hour.

    Share this:

    • Twitter
    • Facebook

    Like this:

    Like Loading...

    Related