Summary:

There were 2,707 pushes in August 2010. This is well above our previous record of 1,971 in January, and 50% above the 1,838 jobs we handled last month. TryServer continues to be the busiest branch of the entire infrastructure, and its worth noting that we did more pushes to TryServer during this month then we did to the entire RelEng infrastructure, combined across all branches, in any given month during first half of 2009.

The numbers for this month are:

• 2,707 code changes to our mercurial-based repos, which triggered 336,910 jobs:
• 51,217 build jobs, or ~69 jobs per hour.
• 162,909 unittest jobs, or ~219 jobs per hour.
• 122,784 talos jobs, or ~117 talos jobs per hour.

Details:

• You can clearly see the drop in load over the last few days in August – caused by a US national holiday, and a Canadian national holiday, on the same long weekend.
• The trend of “what time of day is busiest” changed again this month. Not sure what this means, but worth pointing out that each month seems to be different. This makes finding a “good” time for a downtime almost impossible.
• We are still double-running unittests for some OS; running unittest-on-builder and also unittest-on-tester. This continues while developers and QA work through the issues. Whenever unittest-on-test-machine is live and green, we disable unittest-on-builders to reduce wait times for builds.
• The entire series of these infrastructure load blogposts can be found here.
• We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.
• Anamaria is getting closer to having dashboard reports like this generated automatically – something I’ll rejoice!

Detailed breakdown is :

Here’s how the math works out (Descriptions of build, unittest and performance jobs triggered by each individual push are here: