Summary:

May 2010 had lower load compared to recent months, Try Server usage continues to grow quickly, jumping from 33% to 44% of overall load in one month.

The numbers for this month are:

• 1,584 code changes to our mercurial-based repos, which triggered 164,380 jobs:
• 18,510 build jobs, or ~25 jobs per hour.
• 69,702 unittest jobs, or ~94 jobs per hour.
• 76,168 talos jobs, or ~102 talos jobs per hour.

Details:

• The number of builds we generate per checkin changed throughout the month: linux64, 10.6 64bit, win64 android came online at different dates. To be conservative, I’m omitting these from this month, and I’ll track all these new OS in June.
• Our Unittest and Talos load continues high, like last month, and is increasing as we continue to add more OS to Talos. Again, being conservative, those part-month numbers are excluded from this month.
• Running Unittests on all the Talos OS is still progressing. Getting unittests running green on the new Talos OS was tough. However, its proven harder then we’d like to get the unittest-on-builders turned off, as we keep finding new surprise dependencies that need reworking. Until these are all resolved, we continue to be in the worst-case situation of running unittest-on-builders AND unittest-on-test-minis. Once we finally disable unittest-on-builders, I’ll update the math here. We added 160 more machines to help with this load, but this continues to cause us significant load issues daily.
• The trend of “what time of day is busiest” changed again this month. Not sure what this means, but worth pointing out that each month seems to be different. This makes finding a “good” time for a downtime almost impossible.
• The entire series of these infrastructure load blogposts can be found here.
• We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.

Detailed breakdown is :

Here’s how the math works out (Descriptions of build, unittest and performance jobs triggered by each individual push are here: