It give me great pleasure to formally announce that both Chris AtLee and Chris Cooper have been promoted to Engineering Managers. This formalizes what we’ve been doing in RelEng for multiple quarters already, so in one sense, there is nothing new happening here. However, it is great to recognize the awesome hard work that both Chris and Chris have been doing, and formalize it going forward.
Congrats on the well deserved promotions, catlee and coop!
John.
Its easy to skip a blog post about how to run meetings – yawn – and skip on to the
more exciting posts about some new shiny tech topic. Don’t make the same
mistake I did. This is a quick read and will change your working
life.
Deb did a blogpost a while ago about how to run a more efficient 1×1
meeting. To be honest, I saw the post and skipped over it “Dont have
time to read that, and anyway, I’ve done lots of 1x1s – each unique to
needs of each individual, and they go just fine, thanks anyway”.
Then Coop, in his own polite understated way, told me we were going to
try this format. It worked great for Coop’s 1×1 with Armen, and he
thought it might improve Coop’s 1×1 with me. Our first meeting took
longer then usual, but that was each of us getting used to the
changeover. The second meeting, and all meetings since then, have been
much shorter than usual, and far more productive for both people!
Maybe its just something unique to coop and myself?
But it felt worth trying with a few other people, which I did over the
next couple of weeks. At that point, I was totally convinced. We now use
this for all my 1x1s in RelEng.
Why does it work so so well?
The brilliance of Deb’s approach is that it is super low-tech and super
easy to use. As engineers, we’re always tempted to look for technical
solution to any problem, but the few attempts I’ve seen so far have all
added complexity and got in the way. By contrast, Deb stepped back and
revisited the essence of the original problem from a completely different
perspective and I love what she came up with.
Try her suggestion. If it doesnt work for you, go back to what you did
before, no harm done. But maybe, just maybe, you will love it, and find
yourself giving a silent “Thank you, Deb.” after every 1×1, just like I do.
[UPDATE: Ben Horowitz just blogged about this also. joduinn 04-sep-2012]
The flight disruptions in Europe complicated the RelEng gathering in Toronto last week. Rail’s flights were canceled, and it took a while to find alternate flights that worked – he finally made it to Toronto, and it was great for everyone to meet in person. All united at last.
The week together was awesome. The advance planning is a bit of a headache, but the time spent together and the brainstorming of knotty problems make it all well worthwhile. With so many “remoties”, we’re used to being very a distributed group, yet there were a bunch of problems that we worked though in just the few days we were together. As always, I find myself leaving these group gatherings feeling excited by the things we’ve done, the major projects we’re working on next, and proud of the wide range of smart unique people in the group.
(from L->R; standing: coop, bhearsum, rail, joduinn, catlee, bear, lsblakk, jhford, nthomas, aki; sitting: armenzg, alice. Photo thanks to Aki!)
ps: Rail’s going to stay in Toronto for next week also, to work with catlee and bhearsum, so if you see him in the Mozilla office, please do take a moment to say hi!
The Icelandic volcano eruption is still causing significant travel disruptions in Europe, and looking to get worse. The news is covered with stories of entire countries closing their airspace for the first time, photos of stranded travelers in airports, stories of people taking taxis from England to Switzerland – all sounds bad. Even RelEng is impacted by these flight disruptions: we’re all meeting in Toronto this week, but sadly Rail is stuck in Moscow.
This picture from flightradar24.com posted a more understandable summary of the scale of the disruption. The combination of flight data with maps summed up the situation in a very intuitive way, and I really liked how they did this. Nice job, flightradar24.com .
(Oh, and before you ask why close entire country airspace for “some dust”, you should check out the stories about BritishAirways Flight#9 and KLM Flight#867 during other volcanic eruptions. Both ended well, but still…)
Watch this space – at some point focus of news will shift from the flight and economic disruption of this eruption to how this will change weather patterns.
Summary:
We produce nightly builds for each of the release branches (1.9.0, 1.9.1, 1.9.2), along with updates to keep users on the latest mozilla-central nightly.
Project branches are different. We create nightly builds, but creating updates to bring users on a project branch to the next nightly build on the project branch turned out to be hard.
This is now fixed. 🙂
Lorentz nightly users have been getting nightly updates for a couple of weeks now, and that worked fine. We’re now enabling nightly updates for other project branches that have enough daily changes to make it worthwhile.
This is a big milestone in Mozilla’s transition to concurrent branch development, and kudos to nthomas and coop for making this happen.
More details for the curious:
The problem sounds easy at first: we already have updates for mozilla-central, mozilla191, mozilla192… how hard can it be?
Well, like a lot of RelEng, the devil is in the details.
What version number do you call a particular project branch? The same as the branch its based from? Something unique? Oh, and some gotchas:
The breakthrough was when we realized we could make updates for project branches have the same version number as their parent, yet be on a different update channel. This is similar to how we do partner updates. With some small tweaks, around how we handle fallbacks, we might be able to do similar here. For example, the parent of lorentz is mozilla-192, so we gave lorentz the same *version* as 192, but a different lorentz-nightly channel.
Bingo. This worked first time.
After quite some testing to make sure we weren’t missing a horrible pothole, we used this approach. Once the updates-on-slaves project was live in production, we had the spare CPU cycles to generate these updates every night. Lorentz nightly users were highest priority, so we tried it there first, and all were happy.
The Lorentz branch is going away, but we know we can do this for Electrolysis and Tracemonkey branches now. We’re not going to enable these updates for every project branch; if there’s not at least a change/two per day, its really just not worthwhile. Its not free to generate the update snippets, and preserving them on AUS takes quite some space and time in perpetuity. These same systems still offer updates to any slow-upgrading FF1.5.0.12 users, so we don’t want to put anything on here unless it provides real value.
Having these updates available for developers on Tracemonkey and Electrolysis to dogfood their own work, gives us the usual goodness of faster development, faster bugfixing and faster landing of project branch work into new releases. This is a big milestone in our abililty to transition from single-track-development to concurrent branch development, and is great news… for developers, QA, users and Firefox.
Further details at https://bugzilla.mozilla.org/show_bug.cgi?id=534954.
and the various linked bugs from there.
Summary:
We used to generate all updates on one dedicated machine (prometheus-vm). We now generate updates as jobs queued to the pool of slaves. This makes our current work faster, and unblocks us to do some awesome stuff. /me bows with gratitude to coop.
More details for the curious:
Why bother refactoring how we create updates? We have one old machine doing night build updates for years, and creating updates are quick, they take just 15mins a night:
2.5 minutes per update x 3 OS x 1 en-US locale x 2 branches = 15 minutes
Having the one machine doing this for the two active code branches was trivial, so why not just leave it alone, there’s plenty of other things to fix, right?
The problem is that 15mins is for en-US nightly updates only. We wanted to treat l10n as equals, so we figured out how to produce nightly updates for l10n builds as well. This had never been done in Mozilla before, and was great for the l10n community. However, when we turned on l10n nightly updates in production, it changed the math significantly. What used to take 15 minutes now took:
2.5 minutes per update x 3 OS x 75 locales x 2 branches = 1,125 minutes = 18.75hours.
Compounding the situation was RelEng being asked to support 3 active fully localized releases (FF3.0, FF3.5, FF3.6) and also 5 project branches that all wanted nightly updates. And increase from 3 OS to 7 OS on most of those branches.
Clearly one machine couldnt do all this in a 24 hour day.
One approach we considered was just to clone this machine, do half the work on one, half on another identical machine. This *might* have worked, but wasnt risk free as the old system isnt documented anywhere, and we’d have to verify the two systems could not trip each other up, corrupt the updates and break users.
Whatever we changed here had to be so well understood that we would be confident it was not breaking any user updates. And it had to scale. And solve the “single point of failure” problem to be reliable for our needs.
Coop figured out how the old system worked, how it could be broken into independent concurrent chunks, and how it could be integrated into buildbot, so these could be run after each nightly. Details for the curious are here. Its been tricky, because the code is fiddly, there are many sharp edges, and high risk – any bugs would generate bad updates that complete break a user – so confidence in the accuracy of the updates has to be rock solid. And it has to not break other users of the same patch generation code, like Seamonkey, Camino, etc.
Coop rolled this into production a week ago, after months of testing in staging. From the outside looking in, no-one noticed any difference, except that 18 hours of update generation was now being done in < 5 hours. For such a huge change, having nobody notice any problems is a great accomplishment.
Within RelEng, this change means that nightly updates for a branch are now done in under 1/3rd of the time. It also means we can now do multiple sets of updates concurrently, spread across the pool, which scales our ability to generate updates. Because of this, we can now generate updates for the new linux64 and osx10.6 64bit builds, like announced here. Because of this, in a 24 hour day, we can now generate updates for 3 fully localized release branches and 5 non-localized project branches… but I’m getting ahead of myself! There’s more in the next post…
Yesterday, TinderBoxPushLog started showing little green squares under “Mac OSX64” and “Linux64”. Guess what – those are OSX10.6 64bit and linux64 bit builds!
There are opt and debug builds, with incremental builds triggered by checkin during the day, full clobber builds run every night and all builds available on ftp.m.o. All the usual goodness.
If you are running on 64bit OS, and want to help, you can download the nightly builds from here and try them out. We tested like crazy in staging, so think it all works, but if you hit any problems, we’d love to hear about them, particularly with the nightly updating, or crash reporting.
As I type, the rest of the mechanics are still being rolled out to production in digestible chunks – unittests, talos, and release-build-automation are some highlights – but it is exciting to see this new-desktop-OS work rollout to production. The curious can follow the 10.6 64bit work here and the linux64 work here.
There’s been a ton of behind the scenes work to make this happen. Please send chocolate/beer/kudos to Bear, Armen, Coop, Josh. Also a special shout out to jlazaro, jdow for all the work imaging up 90 machines so these new OS came online with enough slaves in the pool-of-slaves to keep wait times healthy.
Lots of big exciting stuff came online in RelEng land last week, and even more this week.
We’ve been so busy rolling all these into production, we’ve not had time to talk about them much. Time to fix that – but where to start… ?
(Note: this is a *great* problem to have!)
From a RelEng point of view, Firefox 3.6.3 was our fastest release yet.
While this is great to see, this was not a perfect release for us, and we know we can do better:
We’ll go through this in more detail in our postmortem later today, and file bugs for any other gotchas… all to try and improve the *next* release.
Note: as usual, I do not include the time it takes to *develop* and *test* a fix for a bug. I’m totally happy with developers taking the time needed to develop a good fix, and QA taking the time needed to verify the fix works as expected. If you start rushing any of that, its easy to accidentally compromise the quality of the fix. Which is bad. However, the time taken from “go to build” to “release is out” is a measure of the efficiency of the release automation process; something we’re constantly working on improving *between* releases. The faster we can safely release, the better.
ps: The speed at which Mozilla got this bug fixed, tested and distributed to users was noticed here
– very cool!
UPDATE: fixed typo in URL. joduinn 05apr2010
You must be logged in to post a comment.