Mozilla

Infrastructure load for November 2010

joduinn2 Comments

Summary:

There were 2,322 pushes in November 2010. This is a continued drop from September (2,436 pushes) and October (2,360 pushes). This continued drop in number of checkins is expected, considering the prolonged lockdown for FF4.0beta7, immediately followed by the lockdown for FF4.0beta8.

The numbers for this month are:

  • 2,322 code changes to our mercurial-based repos, which triggered 292,035 jobs:
  • 43,738 build jobs, or ~61 jobs per hour.
  • 138,585 unittest jobs, or ~192 jobs per hour.
  • 109,712 talos jobs, or ~152 talos jobs per hour.

Interesting side effect of these lockdowns is the significant increase in TryServer usage. This is the first time that TryServer has become significantly more then half the overall load for the entire RelEng infrastructure. It feels like developers who were blocked from landing were continuing to work by developing and testing patches using TryServer, but thats just conjecture.

Details:

  • The long-running lockdown for FF4.0beta7, and then for FF4.0beta8 definitely took their hit on who was able to checkin, and where/when.
  • We are still double-running unittests for some OS; running unittest-on-builder and also unittest-on-tester. This continues while developers and QA work through the issues. Whenever unittest-on-test-machine is live and green, we disable unittest-on-builders to reduce wait times for builds. Any help with these tests would be great!
  • The entire series of these infrastructure load blogposts can be found here.
  • We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.

Detailed breakdown is :

Here’s how the math works out (Descriptions of build, unittest and performance jobs triggered by each individual push are here:

2 thoughts on “Infrastructure load for November 2010”

  1. Actually, we’re double-running tests on the totally-incapable Win2K3 slaves and on Win7 because we need to have WinXP up before we can get rid of Win2K3, which is blocked not on developers and QA, but on Armen getting enough time and resources and help to get WinXP running. As you say, any help with that would be great!

  2. It looks like I will have XP optimized unit tests running this week.
    In the next 2 weeks we should have w2k3 disabled.

    More than time and resources I was hitting a problem that I didn’t figure out until yesterday afternoon.