Infrastructure load for February 2013

#checkins-per-month: We had 5,382 checkins in February 2013. This drop from last month surprised me. Maybe January was abnormally high because of the first-week-back-after-holidays rush, combined with the B2G workweek? Maybe February was abnormally low because it was a short month, combined with restrictions to checkins as we approached B2Gv1.0.0, B2Gv1.0.1 and Mobile World Congress? Next month’s numbers will help show the trend here, but meanwhile, if you have opinions, I’d be curious to hear them.

As usual, our build pool handled the load well, with >95% of all builds consistently being started within 15mins.

Our test pool situation continues to improve, but is not yet as great as the situation with our builders. We’re making good progress, but the rate of checkins, the improved capacity of the build machines to generate more builds that need testing, the ever-increasing number of test suites to run on each build and the hardware specific nature of some test suites make this test capacity problem harder to solve. New hardware is still (slowly) coming. Meanwhile, RelEng, ATeam and devs continue the work of finding test suites which should (in theory!) be able to run on AWS, then fixing them to make them run green. Once a test suite runs green on AWS, RelEng stops scheduling that test suite on physical machines. This means double goodness: the AWS-based test suites have great wait times on AWS, and the remaining physical-hardware-based test suites have slightly improved wait times because fewer jobs are being scheduled on our scarce hardware.

Of course, some tests *need* hardware, so we’re continuing work to buy and power up more test machines to increase test capacity anyways; please continue to bear with us while this happens. Oh, and of course, if you know of any test suites that no longer need to be run per-checkin, please let us know so we can immediately reduce the load a little. Every little helps put scarce test CPU to better use.
#checkins-per-day: During February, 18-of-28 days had over 200 checkins-per-day, and 8-of-28 days had over 250 checkins-per-day (the high-water-mark for the month was 20feb with 270 checkins).
#checkins-per-hour: Checkins are still mostly mid-day PT/afternoon ET, but the load has increased across the day. For almost 33% of every day (7 of every 24 hours), we sustained over 10 checkins per hour. Heaviest load times this month were 10-11am PT (14 checkins-per-hour – a new record, exceeding our previous record of 13.36 checkins-per-hour set in November2012!).

mozilla-inbound, mozilla-central, fx-team:
Ratios of checkins across these branches remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch, with 28.3% of all checkins, consistently far more then the other integration branches combined. As usual, fx-team has ~1% of checkins, mozilla-central has 2.2% of checkins. The lure of sheriff assistance on mozilla-inbound continues to be consistently popular, and as usual, very few people land directly on mozilla-central these days.

mozilla-aurora, mozilla-beta, mozilla-b2g18, gaia-central:
Of our total monthly checkins:

2.5% landed into mozilla-aurora. This is slightly lower then normal
aurora levels, and expected since b2g changes are no longer being landed
into aurora and beta.
1.3% landed into mozilla-beta. This is slightly lower then normal
beta levels, and expected since b2g changes are no longer being landed
into aurora and beta.
1.8% landed into mozilla-b2g18. These checkins are *only* for the
B2G releases, so worth calling out here.
3.1% landed into gaia-central, making gaia-central the third
busiest branch overall, after try and mozilla-inbound. Obviously, these
checkins are *only* for the B2G releases, so worth calling out here.

misc other details:

Pushes per day
- You can clearly see weekends through the month.
Pushes by hour of day
Share this:
Like this:
Like Loading...

Related

Share this:

Like this:

Related