Summary:
March 2010 sets a new record for the number of pushes since we started recording load in Jan 2009. Try Server usage continues to mount, surpassing all other branches again.
- 1,971 code changes to our mercurial-based repos, which triggered 214,066 jobs:
- 23,787 build jobs, or ~32 jobs per hour.
- 95,493 unittest jobs, or ~128 jobs per hour.
- 94,786 talos jobs, or ~128 talos jobs per hour.
- It is interesting to note that for several months now, our load is roughly broken into 3 parts: 1/3 TryServer, 1/3 mozilla-central, and 1/3 all-other-branches-combined.
Details:
- Our Unittest and Talos load continues high, like last month, and we expect this to jump further as more OS are still being added to Talos.
- Once we start running Unittests on all the Talos OS, we expect load to jump again. In advance of that, we’re spinning up more machines to handle this future spike in load.
- The trend of “what time of day is busiest” changed again this month. Not sure what this means, but worth pointing out that each month seems to be different. This makes finding a “good” time for a downtime almost impossible.
- The entire series of these infrastructure load blogposts can be found here.
- We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.
Detailed breakdown is :
Here’s how the math works out (Descriptions of build, unittest and performance jobs triggered by each individual push are here:
UPDATED: thanks to jesse for spotting a math typo, now fixed. joduinn 05may2010.