Mozilla

Infrastructure load for January 2013

joduinnLeave a comment
  • #checkins-per-month: We had 6,247 checkins in January 2013. This exceeds our previous all-time record of 5,893 in October2012.

    As usual, we handled this load with >95% of all builds consistently being started within 15mins. Sadly, our test pools continue to have a hard time, both with the increased rate of checkins, and the ever-increasing number of test suites being run per checkin. Some test jobs are now runnable on AWS, and are now running there. Some *should* run on AWS, but fail for some reason – work continues. And some tests *need* hardware, so we’re continuing work to buy and power up more test machines to build out capacity there; please continue to bear with us. Oh, and of course, if you know of any test suites that no longer need to be run per-checkin, please let us know so we can immediately reduce the load a little. Every little helps put scarce test CPU to better use.

  • #checkins-per-day: During January, 18-of-31 days had over 200 checkins-per-day, and 2-of-31 days had over 300 checkins-per-day (06jan had 307 checkins; 10jan had 302 checkins). The pattern is to be expected as both of these days were during our first full week back after holidays – typically our busiest week of the year…and this year coincided with a B2G workweek!
  • #checkins-per-hour: Checkins are still mostly mid-day PT/afternoon ET, but the load has increased across the day. For 33% of every day (8 of every 24 hours), we sustained over 10 checkins per hour. Heaviest load times this month were 1-2pm PT (13 checkins-per-hour) and 2-3pm PT (13.36 checkins-per-hour – which matched our previous record of 13.36checkins-per-hour set in November2012!).

mozilla-inbound, mozilla-central, fx-team:
Ratios of checkins across these branches remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch, with 29% of all checkins, consistently far more then the other integration branches combined (fx-team has 1% of checkins, mozilla-central has 2.3% of checkins). As usual, very few people land directly on mozilla-central these days.

mozilla-aurora, mozilla-beta, mozilla-b2g18:

  • 3.1% of our total monthly checkins landed into mozilla-aurora. This is back down to normal aurora levels. This is expected since b2g changes are no longer being landed into aurora and beta.
  • 1.6% of our total monthly checkins landed into mozilla-beta. This is back down to normal beta levels (maybe even slightly lower). This is expected since b2g changes are no longer being landed into aurora and beta.
  • 4.6% of our total monthly checkins landed into mozilla-b2g18. These are all fixes *only* for the B2G releases, so important enough to be worth calling out here, like the aurora and beta branches.

misc other details:

  • Pushes per day
    • You can clearly see weekends through the month… and which week was the “first-week-after-holidays-combined-with-B2G-workweek”.

  • Pushes by hour of day
      Mid-morning PT is consistently the biggest spike of checkins, although this month the checkin load stayed high throughout the entire PT working day.