Summary:
We produce nightly builds for each of the release branches (1.9.0, 1.9.1, 1.9.2), along with updates to keep users on the latest mozilla-central nightly.

Project branches are different. We create nightly builds, but creating updates to bring users on a project branch to the next nightly build on the project branch turned out to be hard.

This is now fixed. 🙂

Lorentz nightly users have been getting nightly updates for a couple of weeks now, and that worked fine. We’re now enabling nightly updates for other project branches that have enough daily changes to make it worthwhile.

This is a big milestone in Mozilla’s transition to concurrent branch development, and kudos to nthomas and coop for making this happen.

More details for the curious:
The problem sounds easy at first: we already have updates for mozilla-central, mozilla191, mozilla192… how hard can it be?

Well, like a lot of RelEng, the devil is in the details.

What version number do you call a particular project branch? The same as the branch its based from? Something unique? Oh, and some gotchas:

• pretending that a project branch was a really old version doesnt work. A safety feature of the updater logic is that you cannot update to an older version. Users on “1.9.4” would hit problems if they tried to “upgrade” to released “1.9.3”. For example, existing Firefox “1.9.2” users could not upgrade to “1.9.0.999”, or “-1.1.1.1”. And we’d have to be careful to not accidently bump into some old, valid, updates still
  live for 1.5.0.12 for example.
• pretending that tracemonkey was a newer future number wasnt perfect either. When we do have a *real* mozilla-1.9.4 in the future, those builds will get updated to the fake-number-for-projectbranch builds and be broken. Planting landmines for ourselves in the future seemed a bad idea. I call this polluting our update-namespace.
• the updater systems uses numbers, and its unknown what happens if you try something like “1.9.2tracemonkey” or “x.y.z”. Investigating this required digging in our release automation code, as well as AUS server logic, as well as the Firefox client updater code. Several attempts at this ended up in scary, hard to prove correct code, and caused us to drop this goal in earlier quarters.

The breakthrough was when we realized we could make updates for project branches have the same version number as their parent, yet be on a different update channel. This is similar to how we do partner updates. With some small tweaks, around how we handle fallbacks, we might be able to do similar here. For example, the parent of lorentz is mozilla-192, so we gave lorentz the same *version* as 192, but a different lorentz-nightly channel.

Bingo. This worked first time.

After quite some testing to make sure we weren’t missing a horrible pothole, we used this approach. Once the updates-on-slaves project was live in production, we had the spare CPU cycles to generate these updates every night. Lorentz nightly users were highest priority, so we tried it there first, and all were happy.

The Lorentz branch is going away, but we know we can do this for Electrolysis and Tracemonkey branches now. We’re not going to enable these updates for every project branch; if there’s not at least a change/two per day, its really just not worthwhile. Its not free to generate the update snippets, and preserving them on AUS takes quite some space and time in perpetuity. These same systems still offer updates to any slow-upgrading FF1.5.0.12 users, so we don’t want to put anything on here unless it provides real value.

Having these updates available for developers on Tracemonkey and Electrolysis to dogfood their own work, gives us the usual goodness of faster development, faster bugfixing and faster landing of project branch work into new releases. This is a big milestone in our abililty to transition from single-track-development to concurrent branch development, and is great news… for developers, QA, users and Firefox.

Further details at https://bugzilla.mozilla.org/show_bug.cgi?id=534954.
and the various linked bugs from there.