Infrastructure load for March 2010


March 2010 sets a new record for the number of pushes since we started recording load in Jan 2009. Try Server usage continues to mount, surpassing all other branches again.

Overall load since Jan 2009The numbers for this month are:

  • 1,971 code changes to our mercurial-based repos, which triggered 214,066 jobs:
  • 23,787 build jobs, or ~32 jobs per hour.
  • 95,493 unittest jobs, or ~128 jobs per hour.
  • 94,786 talos jobs, or ~128 talos jobs per hour.
  • It is interesting to note that for several months now, our load is roughly broken into 3 parts: 1/3 TryServer, 1/3 mozilla-central, and 1/3 all-other-branches-combined.

Infrastructure load by branch


  • Our Unittest and Talos load continues high, like last month, and we expect this to jump further as more OS are still being added to Talos.
  • Once we start running Unittests on all the Talos OS, we expect load to jump again. In advance of that, we’re spinning up more machines to handle this future spike in load.
  • The trend of “what time of day is busiest” changed again this month. Not sure what this means, but worth pointing out that each month seems to be different. This makes finding a “good” time for a downtime almost impossible.
  • The entire series of these infrastructure load blogposts can be found here.
  • We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.

Detailed breakdown is :
#Pushes this month

#Pushes per hour

Here’s how the math works out (Descriptions of build, unittest and performance jobs triggered by each individual push are here:
the math behind the graphs

UPDATED: thanks to jesse for spotting a math typo, now fixed. joduinn 05may2010.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.