Last week was the first time that the entire RelEng group was together in the one place at the one time. We’ve grown a lot over the last couple of years, but the few times we met, we’ve always had some people who couldn’t make it. This time, we *all* were in one place at one time!

Armen managed to snag this photo before we scattered for various airports. In case you missed the introductions in last Monday’s Mozilla Foundation call, here they are again! From left to right:

John (jhford), Chris (catlee), Lukas (lsblakk), Aki (aki), Armen (armenzg), John (joduinn), Nick (nthomas), Ben (bhearsum), Alice (alice), Chris (coop).

Several large projects wrapped up at the end of Q2, we were already well into Q3 work, so this was a perfect time for us to step back, and figure out what next big projects to tackle in Q4, Q1. The group has grown rapidly. The scale of the infrastructure is growing fast (409 machines as of today, with more still being powered up). The complexity of the infrastructure is growing also, with a new mobile platform, and 3 new project branches, all added just in July. And a bunch of new o.s. requests that came in today.

All very exciting.

In addition to the scheduled topics, there was lots of impromptu discussions, and whiteboard drawing going on. Oh, alright, yes, there was also late night food, drinks and rockband! 🙂

For other photos of the week, and the creative-commons licence for this photo, check out Armen’s flickr stream here.

Summary:

• We pushed 1,018 code changes to our mercurial-based repos. Over the month, this translates into:
• 11,279 build/unittest jobs, or ~15.7 jobs per hour.
• 5,837 talos jobs, or ~8 talos jobs per hour.
• TryServer now does linux-arm and WinMo builds, which creates extra load per push.
• The mozilla-1.9.1 branch has a few bursts of activity, but you can see attention moving to mozilla-central, as post-3.5 work starts to open up.
• Aki just pointed out that we’re not yet tracking mobile-browser branch or the l10n repacks. Also, in July we are spinning up new places branch, electrolysis branch, mozilla-1.9.2 branch, so expect to see them in next month’s report.

Details:

As each of these pushes triggers multiple different types of builds/unittest jobs, the math is:

• mozilla-central: 12 jobs per push (L/M/W opt, L/M/W leaktest, L/M/W unittest, linux64 opt, linux-arm, WinMo)
• mozilla-1.9.1: 12 jobs per push (L/M/W opt, L/M/W leaktest, L/M/W unittest, linux64 opt, linux-arm, WinMo)
• tracemonkey: 7 jobs per push (L/M/W opt, L/M/W unittest, linux64 opt)
• try: 11 jobs per push (L/M/W opt, L/M/W leaktest, L/M/W unittest, linux-arm, WinMo)
• theoretical total: (515 x 12) + (130 x 12) + (141 x 7) + (232 x 11) = 11,279 jobs per month = 15.7 jobs per hour.

Now that we have pooled all the talos slaves, we can count triggered talos jobs for every opt build as follows:

• mozilla-central: 6 jobs per push (ubuntu, osx10.4, osx10.5, WinVista, WinXP, linux-arm)
• mozilla-1.9.1: 5 jobs per push (ubuntu, osx10.4, osx10.5, WinVista, WinXP)
• tracemonkey: 5 jobs per push (ubuntu, osx10.4, osx10.5, WinVista, WinXP)
• try: 5 jobs per push (ubuntu, osx1.4, osx10.5, WinVista, WinXP)
• theoretical total: (515 x 6) + (130 x 5) + (141 x 5) + (232 x 6) = 5837 jobs per month = 8 jobs per hour.

Sorry for the delay in getting this posted.

Summary:

• We pushed 1,134 code changes to our mercurial-based repos. This translates into 12,345 build/unittest jobs, or ~16.6 jobs per hour, over the month.
• We hit 108 pushes on May 19th. This is a new high-water-mark, by far the biggest load in any one single day since we start measuring. This was the rush of checkins to beat the mid-May code freeze date.
• Mozilla-191 and TryServer now does linux-arm and WinMo builds, which creates extra load per push.
• The mozilla-1.9.1 branch was more active then usual, as expected in the lead up to FF3.5.0 release.
• The chart below shows no data for TryServer for the week of 1st – 7th May. This was because of our resetting of the repo, and is expected. The TryServer was up and being used, but this means TryServer numbers are too low.
• We’re still not measuring load on Talos yet.

Details:

As each of these pushes triggers multiple different types of builds/unittest jobs, the *theoretical* total amount of work done by the pool-of-slaves in May was 12,345 jobs. For each push, we do:

• mozilla-central: 12 jobs per push (L/M/W opt, L/M/W leaktest, L/M/W unittest, linux64 opt, linux-arm, WinMo)
• mozilla-1.9.1: 12 jobs per push (L/M/W opt, L/M/W leaktest, L/M/W unittest, linux64 opt, linux-arm, WinMo)
• tracemonkey: 7 jobs per push (L/M/W opt, L/M/W unittest, linux64 opt)
• try: 9 jobs per push (L/M/W opt, L/M/W leaktest, L/M/W unittest)
• theoretical total: (523 x 12) + (282 x 12) + (138 x 7) + (191 x 9) = 12,345 jobs per month = 16.6 jobs per hour.

UPDATE: TryServer supported for linux-arm and WinMo were enabled in the last few days of the month, so I’ve excluded them from the math, and recalculated number-of-jobs.  John. 17July2009

Looking at the people who moved to FF3.5.0 yesterday:

• 65% downloaded the FF3.5.0 installer and installed from it
• 35% manually did Help->CheckForUpdates

That is a large percentage of people doing CheckForUpdates.

Considering this was our first time having Major Update available on release day, and considering there was no user prompting of this new major update ability, I found these percentages quite delightfully stunning.