Summary:

There were 2,322 pushes in November 2010. This is a continued drop from September (2,436 pushes) and October (2,360 pushes). This continued drop in number of checkins is expected, considering the prolonged lockdown for FF4.0beta7, immediately followed by the lockdown for FF4.0beta8.

The numbers for this month are:

• 2,322 code changes to our mercurial-based repos, which triggered 292,035 jobs:
• 43,738 build jobs, or ~61 jobs per hour.
• 138,585 unittest jobs, or ~192 jobs per hour.
• 109,712 talos jobs, or ~152 talos jobs per hour.

Interesting side effect of these lockdowns is the significant increase in TryServer usage. This is the first time that TryServer has become significantly more then half the overall load for the entire RelEng infrastructure. It feels like developers who were blocked from landing were continuing to work by developing and testing patches using TryServer, but thats just conjecture.

Details:

• The long-running lockdown for FF4.0beta7, and then for FF4.0beta8 definitely took their hit on who was able to checkin, and where/when.
• We are still double-running unittests for some OS; running unittest-on-builder and also unittest-on-tester. This continues while developers and QA work through the issues. Whenever unittest-on-test-machine is live and green, we disable unittest-on-builders to reduce wait times for builds. Any help with these tests would be great!
• The entire series of these infrastructure load blogposts can be found here.
• We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.

Detailed breakdown is :

Here’s how the math works out (Descriptions of build, unittest and performance jobs triggered by each individual push are here: