Infrastructure load for November 2012

  • #checkins-per-month: We had 5,646 checkins in November 2012. This is slightly down from last month’s record of 5,893, but still the 3rd highest number of checkins in 2012, after October (5,893 checkins) and August (5,803 checkins). Again, we handled this record load with >95% of all builds consistently being started within 15mins. Sadly, our test pools continue to have a hard time, both with the increased rate of checkins, and the increased number of test suites being run per checkin. We’re continuing to work on buying and powering up more test machines, please continue to bear with us. Meanwhile, if you know of any test suites that no longer need to be run per-checkin, please let us know.
  • #checkins-per-day: During November, 19-of-30 days had over 200 checkins-per-day. Worth noting was spikes of 295 checkins on 13nov and 301 checkins on 27nov.
  • #checkins-per-hour: The peak across this month was a new record 13.36 checkins per hour between 10-11am. Of interest, I note that for 25% of every day (6 of every 24 hours), we sustained over 10 checkins per hour.

mozilla-inbound, mozilla-central, fx-team:
Ratios of checkins across these branches remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch, with 26.9% of all checkins, consistently far more then the other integration branches combined (fx-team has 0.8% of checkins, mozilla-central has 2.2% of checkins). As usual, very few people land directly on mozilla-central – in fact more people go through approval process to land on mozilla-aurora.

mozilla-aurora, mozilla-beta:

  • 4.9% of our total monthly checkins landed into mozilla-aurora. This is an decrease from last month’s high, but still higher then is typical. I believe this is caused by the number of b2g changes being landed into aurora, which then stopped ~20th November, with the migration of b2g from aurora to beta.
  • 2.2% of our total monthly checkins landed into mozilla-beta. This is typical for beta. I suspect the recent transition of b2g to beta just before holiday season here in the US, means we didnt see many b2g-related checkins on beta for November and I do expect beta to show increased checkins for December.

(Standard disclaimer: I’m always glad whenever we catch a problem *before* we ship a release; it avoids us having to do a chemspill release and also we ship better code to our Firefox users in the first place.)

misc other details:

  • Pushes per day
    • You can clearly see weekends through the month.

  • Pushes by hour of day
      It is interesting that mid-morning PT is consistently the biggest spike of checkins during the day. I wonder if this is caused by ET developers doing checkins immediately after lunch, at the same time as PT developers have just settled into the office after coffee and initial emails?

Android x86 builds now on

This week, kmoir quietly enabled Android x86 builds on tbpl.m.o. This was already announced in this week’s Mozilla developer platform meeting but is important enough to repeat here too.

These builds are being run per-checkin, as well as nightly. There are updates for nightly builds. Obviously, there’s a lot of details to still work through (crash symbols, release builds, signing, etc…), but this is a major milestone worth noting. You can follow progress, and offer help to kmoir, on the remaining work in bug#750366.

As always with standing up new platforms in production, if you see problems on tbpl.m.o, please do not just hide the breakages; instead, please file bugs in and we’ll be happy to investigate.

Big thanks to kmoir, callek, jmaher, blassey and others for making this happen.

multi-locale b2g builds now on

(This is so important to FirefoxOS localization that its worth cross-posting.)

In case you missed it, bhearsum just announced that b2g builds on tbpl.m.o are now multi-locale.

There’s two sets of locales here, so to be precise:

  • existing desktop and unagi, otoro builds: ar, en-US, fr, es, pt-BR, zh-TW. These locales are specifically chosen because they help developers debug various font-specific issues (right-to-left, etc).
  • new desktop builds: ar, as, bn-BD, ca, cs, cy, de, el, en-US, eo, es, et, eu, ff, fr, fy-NL, ga-IE, gd, gl, he, hi-IN, ht, hu, id, it, ja, ko, lij, ml, nl, or, pa, pl, pt-BR, ro, ru, sl, sq, sv-SE, te, tr, ur, zh-CN, zh-TW. These are the list of locales being actively worked on. We expect this list to continue growing. For progress, track this file:

For more info, check out bhearsum’s blogpost1 or blogpost2 or bug#766962.

Big thanks to bhearsum, stas, aki and others for making this happen. And hey, if you want to help with localizing boot2gecko / FirefoxOS into your locale, please contact the friendly folks in dev.l10n newsgroups – they’d be delighted with your help.