Infrastructure load for December 2009


  • Numbers were down significantly compared to the previous few months. I’d guess an indicator that Mozilla entered the end phase of the Firefox 3.6.0 and Fennec 1.0 releases, both around the same time. The holidays in the last half of the month also reduced the load.
  • The numbers for this month are:
    • 1,268 code changes to our mercurial-based repos, which triggered:
    • 15,345 build jobs, or ~65 jobs per hour.
    • 33,263 unittest jobs, or ~44 jobs per hour.
    • 31,372 talos jobs, or ~42 talos jobs per hour.
  • Yet again, we enabled new Talos suites, and also disabled some unittest suites at different times during the month. For simplicity, I’ve ignored those changes for now, and will include in next month’s data.
  • Interesting to note that the trend of “what time of day has most checkins” changed again this month. Not sure what this means, but worth pointing out that each month seems to be different. This makes finding a “good” time for a downtime almost impossible.
  • We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.

Here’s how it looks compared to the year so far:

Detailed breakdown is :

Here’s how the math works out:

The types of build, unittest and performance jobs triggered by each individual push are best described here.

One thought on “Infrastructure load for December 2009

  1. In the analyses “per day of month”, I notice that around days 5-6, 12-13, 19-20, 26-27 are there are “slacks” lower than on surrounding days, a periodicity of period 7. I guess it might be useful to have a “day of week” line added. — A look at last year’s calendar shows that these were Saturdays and Sundays; in addition, the gap starts early and last longer around Christmas. Not really a news-breaking discovery of course. What is more surprising is the relatively high activity (for a weekend) on the second weekend (12-13): was there a “software release” at or just after that date? In fact, the fortnight from Tuesday 8 to Friday 18 almost looks like a double week with no weekend break in the middle.

    The last graph (by time of day) is labeled according to “Mountain View clocks” I suppose, i.e., add 3 for Massachusetts universities, 8 for Britain (and Ireland and Portugal), 9 for most of continental Europe (from Spain to Poland), etc. Not sure what can be deduced from it without additional knowledge about e.g. how much devs “do” or “don’t” work on Mozilla as part of their paid jobs or student curricula (and therefore do it “during” or “after” office hours, respectively) in which time zones etc.