Mozilla

Infrastructure load for April 2013

joduinn1 Comment
  • #checkins-per-month: We had 6,364 checkins in April 2013. This is only slightly below our record of 6,433 in Mar2013. Every working day was consistently busy (>200 checkins per working day) and load-per-day was busy across longer periods of each day.

  • #checkins-per-day: On 09apr, we had 311 checkins – our second-busiest day on record (the record remains 323 checkins-per-day on 18mar2013). During April, 22-of-30 days had over 200 checkins-per-day – thats every working day. 12-of-30 days had over 250 checkins-per-day (2-of-30 days had over 300 checkins-per-day!).
  • #checkins-per-hour: Checkins are still mostly mid-day PT/afternoon ET, but the load has increased across the day. For 11 of every 24 hours, we sustained over 10 checkins per hour. Heaviest load times this month were 11am-noon PT (12.77 checkins-per-hour). Its interesting to note we had an atypical spike in load at 5am – possibly from CET based contributors or the B2G workweek.

    • As usual, our build pool handled the load well, with >95% of all builds consistently being started within 15mins. Our test pool is handling this load much better too. In an encouraging sign that all the hard work by RelEng, ATeam and IT is paying off, we’re seeing more test jobs being handled with better response times… The peak for April was 52,118 test jobs on 23apr – our first time handling over 50,000 test jobs in a 24-hour-day. Still more work to be done here, but very encouraging progress.

    As always, if you know of any test suites that no longer need to be run per-checkin, please let us know so we can immediately reduce the load a little. Also, if you know of any test suites which are perma-orange, and hidden on tbpl.m.o, please let us know – thats the worst of both worlds – using up scarce CPU time and not being displayed. We’ll make sure to file bugs to get tests fixed, or disabled – every little bit helps put scarce test CPU to better use.

mozilla-inbound, mozilla-central, fx-team:
Ratios of checkins across these branches remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch,
with 26.1% of all checkins, consistently far more then the other integration branches combined.

In mid-April, we started using birch as a b2g-inbound for the B2G workweek. This experiment was a great success, and has been continued. We had 1.7% of checkins landed on birch, which is impressive considering it was only in use for half the month!

As usual, fx-team has ~1% of checkins, mozilla-central has 1.8% of checkins.

The lure of sheriff assistance on mozilla-inbound (and now birch/b2g-inbound) continues to be consistently popular, and as usual, very few people land directly on mozilla-central these days.

mozilla-aurora, mozilla-beta, mozilla-b2g18, gaia-central:
Of our total monthly checkins:

  • 2.0% landed into mozilla-aurora, very similar to last month.
  • 1.4% landed into mozilla-beta, very similar to last month.
  • 2.1% landed into mozilla-b2g18, slightly higher then last month.
  • 5.9% landed into gaia-central, slightly higher then last month. gaia-central continues to be the third busiest branch overall, after try and mozilla-inbound. Obviously, these checkins are *only* for the B2G releases, so worth calling out here.

misc other details:

  • Pushes per day
    • You can clearly see weekends through the month. Its worth noting that we had >200 checkins-per-day every working day in April.

    • Pushes by hour of day
        Mid-morning PT is consistently the biggest volume of checkins, although this month the checkin load stayed high throughout the entire PT working day, and particularly spiked between 11am-noon PT, with 12.77 checkins-per-hour. I found the spike in activity at 5am to be interesting, as it is unusual. My theory is that this shows an increase in checkins from people in Europe / CET timezone; partially explained by the B2G workweek in Madrid.

One thought on “Infrastructure load for April 2013”