Late Wednesday night, we completed the project to migrate these repos:
http://hg.mozilla.org/mozilla-aurora –> http://hg.mozilla.org/releases/mozilla-aurora
http://hg.mozilla.org/mozilla-beta –> http://hg.mozilla.org/releases/mozilla-beta

This project started several weeks ago when we moved the repos and put symlinks in place. We did this so anyone using the repo in either location would see things working just fine during the transition phase. With all infrastructure pointing to the new repo locations, the last few weeks have been mostly wait-and-see, waiting for a quiet time between “the mozilla-centra –> aurora merge” on Tuesday, and the “enabling of updates on aurora” this morning. Late Wed night we removed the symlinks, and cleaned up http://hg.m.o to remove the duplicate entries.

If you see *any* problems, please reopen and comment in bug#652229. Special thanks to NoahM in IT for all his help in this project.

Other info:
* We added HTTP redirects to hg.m.o, and will leave these in place, in order to maintain existing URLs in any bugs that refer to the old http://hg.mozilla.org/mozilla-aurora and http://hg.mozilla.org/mozilla-beta locations.

* People starting new patches for mozilla-aurora, mozilla-beta should reclone using the new http://hg.mozilla.org/releases/mozilla-aurora or http://hg.mozilla.org/releases/mozilla-beta locations. Alternatively, more fearless hackers can update their .hg/hgrc file.

For more details see the original post to dev.planning.

Summary:

We had 1,751 pushes in March 2011 – a continued significant drop from the last few months. I believe this is because of the continued checkin restrictions during March as we got closer to the Firefox4.0 release. However, I have no way to prove that, it is just my interpretation. If you have other suggestions, please let me know.

Details:

• Load on TryServer is exactly the same (52% vs 52% in previous month) of our overall load.
• The numbers for this month are:
  • 1,751 code changes to our mercurial-based repos, which triggered 214,670 jobs:
  • 32,798 build jobs, or ~44 jobs per hour.
  • 103,608 unittest jobs, or ~139 jobs per hour.
  • 78,264 talos jobs, or ~105 talos jobs per hour.
• We are still double-running unittests for some OS; running unittest-on-builder and also unittest-on-tester. This continues while developers and QA work through the issues. Whenever unittest-on-test-machine is live and green, we disable unittest-on-builders to reduce wait times for builds. Any help with these tests would be great!
• The entire series of these infrastructure load blogposts can be found here.
• We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.

Even more details:

Here’s how the math works out (Descriptions of build, unittest and performance jobs triggered by each individual push are here:

On Friday, we shipped Firefox 5.0beta2 and Fennec 5.0beta2. The cool new features have already been covered elsewhere, but there were two details that I thought were important:

1. Firefox and Fennec both used the same changeset http://hg.mozilla.org/releases/mozilla-beta/rev/2b3275216413.
2. Firefox and Fennec shipped at the same time. Its true we’ve sim-shipped these products before, but this was the first time planned as part of the new rapid release cadence.

This is an important milestone for us, as we bring Fennec up to parity with Firefox.

Aki led some great behind-the-scenes cleanup work to make this happen. And even with that work in place, we still needed some careful workarounds in place for last week’s builds (details in https://www.releasemechanix.com/2011/05/05/branch-mechanics-for-fennec-5-0).

We’ll continue to consolidate the product code bases, and consolidate the release automation behind these two products, so we can do this again more easily next time. Meanwhile, it was really great to see both these releases ship, from the same changeset, on Friday morning.

Summary:

(Sorry for the long-delay in posting these, things are starting to calm down again, so I’ll catch up.)

We had 2,060 pushes in February 2011 – a significant drop from the last few months. I believe this is because of the checkin restrictions during February (for Firefox4.0beta11, Firefox4.0beta12, Fennec4.0beta4, Fennec4.0beta5) as we got closer to the Firefox4.0 release, but honestly, that is just my interpretation. I have no way to prove that. If you have other suggestions, please let me know.

Details:

• Load on TryServer is about the same (52% vs 53% in previous month). of our overall load. People continue to send their patches to TryServer before landing, so patches that do land are less-risky, and the tree stays green more often!
• The numbers for this month are:
  • 2,060 code changes to our mercurial-based repos, which triggered 262,340 jobs:
  • 39,365 build jobs, or ~58 jobs per hour.
  • 123,603 unittest jobs, or ~184 jobs per hour.
  • 99,372 talos jobs, or ~148 talos jobs per hour.
• We are still double-running unittests for some OS; running unittest-on-builder and also unittest-on-tester. This continues while developers and QA work through the issues. Whenever unittest-on-test-machine is live and green, we disable unittest-on-builders to reduce wait times for builds. Any help with these tests would be great!
• The entire series of these infrastructure load blogposts can be found here.
• We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.

Detailed breakdown is :

Here’s how the math works out (Descriptions of build, unittest and performance jobs triggered by each individual push are here:

Please join me in welcoming Marc as an intern in RelEng.

Marc comes from UWaterloo, and started last Monday! He’s started working on some release automation around Android signing (details in bug#603826, and will then be helping Lukas with the buildbot-bugzilla-tryserver integration project.

Once Marc has finished fixing up his website / blog, you can follow along on his website! Of course, if you are in 650Castro, or in #build, please do stop by and say hello to “mjessome”.

In case people missed it earlier, Lukas setup 3 new disposable branches recently: Alder, Holly and Larch, in addition to the existing Birch, Cedar and Maple.

As we move to the faster release cadence, it seems like more and more people are working in their own project branches, and then landing everything when its all done. Very cool.

All these builds and tests take place on the shared pool of machines, so the results are the same. But keep in mind, whether a project branch makes sense for your specific project really depends on how many people are involved in the shared work, and how long that shared work is expected to take to complete.

Here’s the guidelines we’ve seen make most sense so far:

• If it is one person, working by themself, then a user repo and Tryserver is typically enough.
• If it is a group of people, working on a project for days/weeks/months, then a disposable branch makes most sense. Just edit this wiki page to claim a disposable project branch, and you’re ready within a few hours.
• If it is a group of people, working on a project for quarters/years, then a new dedicated project branch makes most sense. File a bug with details from here, and you’ll have your shiny new dedicated project branch in 2 weeks.

Thanks Lukas!

Summary: To deliver Fennec5.0 betas starting 17may, the new plan is to backport Fennec code changes from mozilla-beta back to split repos and and ship Fennec5.0 betas from the split repos using existing release automation.

Details for the curious:
To ship Fennec5.0 on the same release cadence as Firefox5.0, on the consolidated repos on mozilla-beta, requires:
a) changes to release automation to handle consolidated repos
b) remainder of buildbot0.7 -> buildbot0.8 migration – to streamline release day mechanics.

These changes are on track for Fennec6, along with Firefox6, but we cannot get these changes into production before 17may, in time for Fennec5/Firefox5.

To ship a Fennec5.0 beta on the 17may date, the new proposed plan is to:

1. setup new mobile-5.0, mozilla-5.0 repos
2. for l10n, RelEng thinks we can point automation to the l10n/mozilla-beta repos.
3. for sanity check, RelEng will build an early Fennec5.0beta1 before 17may, to make sure this infrastructure works.
4. wait for the planned drop from mozilla-aurora –> mozilla-beta on 17may
5. have mfinkle/blassey merge code from mozilla-beta to new mobile-5.0 & mozilla-5.0 repos
6. have aki/lsblakk create Fennec5.0beta2…5.0betaN using existing release automation.

RelEng will make a staging Fennec5.0beta build next week to verify all this before 17may.

We will repeat this for every Fennec5.0beta until we get these automation changes live in production. Current estimate is early June. Whether we ship Fennec5.0 in this way depends on exact date the new automation is ready – to be revisited.

NOTE: While there are currently no Fennec4.0.x releases scheduled, it is important that we remain able to ship Fennec4.0.x chemspill releases if needed. This plan does not impact our ability to do Fennec4.0.1 chemspills.

Hope all that makes sense – let me know if I missed anything or you have questions. Also, see mfinkle’s related post to mozilla.dev.planning.
John.