Infrastructure load for February 2010


The number of pushes started increasing finally, after Firefox 3.6.0 and Fennec 1.0 releases. Try Server usage surpassed all other branches this month.

  • The numbers for this month are:
    • 1,264 code changes to our mercurial-based repos, which triggered 133,897 jobs:
    • 14,003 build jobs, or ~107 jobs per hour.
    • 58,110 unittest jobs, or ~86 jobs per hour.
    • 61,694 talos jobs, or ~92 talos jobs per hour.
  • Our Unittest and Talos load continues high, like last month, and we expect this to jump further as more OS are still being added to Talos.
  • Once we start running Unittests on all the Talos OS, we expect load to jump again. In advance of that, we’re spinning up more machines to handle this future spike in load.
  • The trend of “what time of day is busiest” changed again this month. Not sure what this means, but worth pointing out that each month seems to be different. This makes finding a “good” time for a downtime almost impossible.
  • The entire series of these infrastructure load blogposts can be found here.
  • We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.

Detailed breakdown is :

Here’s how the math works out (Descriptions of build, unittest and performance jobs triggered by each individual push are here:

2 thoughts on “Infrastructure load for February 2010

  1. No comm-central or comm-1.9.1 statistics?
    Maybe the activity per time-of-day is unstable, but it isn’t the same by day-of-week: weekends exhibit consistently low activity, particularly “special” weekends like Valentine’s Day in February and New Year’s in January.
    What about graphing the pushes against the local time-of-day for the time zone of the guy doing the pushing? Or maybe that time zone isn’t always known?