13 Dec 2010
JohnMozilla
Summary: Mozilla nightly builds were originally not setup to do nightly builds for a branch using the same code revision across all OS. This complicated any attempt to use nightly builds to track down a platform specific bug. This has now been fixed. Send beer and chocolate to catlee.
If you are curious for details, read on!
Because of how Mozilla originally setup the nightly build system, there was three little known quirks from the very beginning:
1) The nightly build was of the tip of the code at the time build started.
This sounds good, but this meant that anyone who checked in late at night ran the risk of breaking the build, and then not being able to back out the change quickly enough, causing the nightly builds to fail out.
2) The machines for each OS started builds at different times
A nightly was started whenever the machine finished building the previous build, and it was the first build started after 3am. The first build after 3am would build using the tip of the code, and be published as a nightly build. However “first job after 3am” when there was only one build machine per OS meant starting the nightly build at different times for different OS; this window of possible changes was ~3hours (longest build time minus 1min). Anyone who checked in during that period would get their change included in the nightly build for that OS, but not in the nightly build for any OS already started.
3) Nightly builds on different OS went into different directories
Because the nightly builds start at different times, the generated builds got different BuildIDs, and are posted into different directories on ftp.m.o. This complicated regression hunting work.
All confusing.
Catlee landed some changes recently which mean that now:
1) The nightly build is of the most recent “good” code.
The nightly build now does not automatically build “tip”. This sounds counterintuitive at first, but actually makes sense – read on. The nightly build now starts by attempting to find a changeset that is newer then the previous nightly, and which is also known to be a good changeset. Right now the definition of “good changeset” is “compiles+links”. Eventually, as there are fewer intermittent tests, the plan is to change that definition to “compiles+links+passes-tests”. Worst case, if *no* changeset has successfully built since the previous nightly, then we’ll fall back to current behavior, and attempt to build tip even though we expect it to fail.
2) Each nightly build is told its BuildID and changeset
The buildbot master tells the build slave which BuildID and changeset to use for the nightly builds. This means the nightly builds are created with the same BuildID for each OS – which means that the nightly builds for each OS show up in the *same* directory on ftp.m.o. No more finding last night’s mozilla-central nightly for linux, mac and win32 in different directories!
All obvious goodness!
“Why wasn’t this fixed years ago???” I hear you ask. It has only become possible after all the other recent changes and scaling done in RelEng, as well as detangling what “build start time” and “BuildID” mean in Tinderbox, Makefile and MozillaBuildSystem. Fixing this very long standing annoyance should help developers and QA triage problems with nightly builds, and also makes me happy. For the curious, further details are in bug#570814.
02 Dec 2010
JohnMozilla, Travel
No visit to Japan would be complete without some strange (to me) food experiments.
Last time I was here, I was surprised to find I actually liked Strawberry KitKat, and RedBean KitKat, so was lulled into a false sense of confidence when I bought these.
Turns out they are white-chocolate-combined-with-cheddar-cheese flavor. Vote: Instant Yuk. Tried a couple of times over the next few days, but there was no way I could make these palatable, so had to toss the rest of the pack out.
01 Dec 2010
JohnMozilla, Travel
Found this parked outside the bus depot in Nagasaki last week.
‘nuf said! 
UPDATE: More details on this car from Daihatsu here.
29 Nov 2010
JohnMozilla, Travel
This morning’s brunch was interrupted by the building gently quietly swaying, without any warning. It started gently enough, and because I’ve been on enough trains recently, I didn’t think there was anything wrong with the room doing that. At first. Then I wondered maybe it was a train/truck driving by, until I remembered that despite all the nearby traffic in this large busy city, I’ve never felt any vibrations/rattles in the hotel at all.
Oh, that’s right. They get earthquakes here too.
Um. Wait. I’m on the 12th floor. In a city where I don’t speak the language. For lack of any better plan, I quickly finish getting dressed and put on my shoes. By then its all over. So I sit back down and finish my coffee.
(ps: usgs.gov reports it as 6.6, while the Washington Post reports it as 6.9, which would make it equal to the 6.9 earthquake that hit Loma Prieta in 1989.)
29 Nov 2010
JohnMozilla, Travel
While in Tokyo, Gen and a bunch of his friends in Tokyo organized a “monjya boat” dinner around Tokyo bay after work. (If you don’t know what Monjayaki, click here and here!)
Quite an unusual experience. Small little boat, with one low cabin of all glass windows. Freezing cold outside. Roasting hot inside. Crowded, with groups of 4-6 seated on the floor around a small low table, all packed close to each other. There was a cooking surface in the middle of the table. The cooking surface was gas powered (you could see the flames), with a plastic hosepipe heading under the mats – it looked like we were all sitting on stockpiles of fuel. The crew politely warned us to not put anything under the table as it got hot and would burn. Once everyone started cooking, the cabin heated up *fast*.
You cooked the meal yourself with bowls of ingredients that the crew on the boat gave you. The joke was that no matter how you cooked it, or what ingredients you were given, it all tasted the same! It was all the food you can eat, and all the beer you can drink – but you dont get any more until you have proven that you have finished what they gave you already. It was oddly yummy, and lots of fun. At our table, most of the cooking was done by Gen and his wife, but I managed to cook one without burning it – anyone who knows my cooking skills understands the enormity of that!
The conversations were great, and it was really interesting to meet others in the Tokyo software business, including some expats. The background of the water-level views of Tokyo made it even more wonderful.
Randomly while we waited for our boat, there was a guy standing there blindfolded, with his hands tied behind his back – in a giant chicken costume! Turns out he was on our boat. Something weird about watching a guy nervously being walked down the gangwalk to a boat at night tied up in a chicken suit. His group were taking good care of him but we never figured out if it was a birthday party, some work initiation/promotion party, or the beginning of some bachelor party.
(Here’s a quick photo Gen took of us all heading back to the train afterward – with Gen invisible behind the camera!)
Great great evening, and many thanks to Gen for making this happen.
27 Nov 2010
JohnMozilla, Travel
No trip to Tokyo would be complete without a few hours wandering lost inside a DonQuijote store. Best done late at night, after food and a bar or two. Their website doesn’t give you any idea of the madness that awaits you – and there’s no real way to describe it! You have to see it to believe it!
My personal favorite this year was the brightly colored stuffed-bean-bag-pigs cascading down from the ceiling. Sorta like what some travelers bring as neck pillows on airplanes, only…worse? better?
Last year while I was in the same store, my favorites were
- the AA battery that you could recharge by plugging into a USB port
- the USB plug-in fragrance dispenser
- the savings bank with motion-sensor activated mouth
- the full-size R2D2 robot-trashcan – press either pedal to open its head!
23 Nov 2010
JohnCars & Driving, Mozilla, Travel
Walking the back streets near Shibuya, in Tokyo, I found this immaculate Morgan +4 parked on the side of the narrow street, outside a crowded barber shop.
Really spectacular, especially in this setting of narrow streets where even the delivery vans are super-compact. For anyone curious for more details about this Morgan +4 car click here
19 Nov 2010
JohnMozilla
Summary: Mozharness is a project to refactor internal ReleaseAutomation code. This includes moving code out of buildbot custom libraries and into standalone scripts/programs/Makefiles.
This is important, exciting, work because:
1) upgrading buildbot will be easier:
Today, upgrading buildbot in production is complicated because of all our code in buildbot custom. Every upgrade requires significant testing of these custom libraries, and some recoding if buildbot internals change. Moving this logic out of buildbot custom to external scripts makes it easier for us to upgrade buildbot in production going forward.
2) refactoring is needed for Fennec:
Fennec Release Automation still runs on buildbot 0.7.x, while Firefox Release Automation runs on (incompatible) buildbot 0.8.2. Refactoring and consolidating these gets us improved Fennec automation, and one code base for RelEng to maintain going forward.
3) others can use the *same* scripts as RelEng:
Moving the consolidated logic out of buildbot custom means that others outside of RelEng can run the *exact* same scripts and Makefile targets that RelEng uses. This is a big help to developers debugging build/test differences between something running on their machine vs a RelEng system.
Obviously, anyone using undocumented internal buildbot custom code will have to modify their code as we refactor. Going forward, we recommend that people do not rely on undocumented internals. Instead, we recommend using the JSON feed available and already being used by TBPL. If this “supported API” isn’t sufficient, please file bugs to have us fix the supported API – its much better then using undocumented internals and having to rewrite your code every time we fix something internally.
For anyone who wants LOTS more details, you should start by reading Aki’s posts here and here before joining the discussions in dev.planning.
16 Nov 2010
JohnMozilla
Summary:
There were 2,436 pushes in September 2010. This is below the record of 2,707 in August. I note Sept 15th set a new record of 160 pushes in one day.
The numbers for this month are:
- 2,707 code changes to our mercurial-based repos, which triggered 336,910 jobs:
- 46,175 build jobs, or ~64 jobs per hour.
- 145,509 unittest jobs, or ~202 jobs per hour.
- 115,608 talos jobs, or ~160 talos jobs per hour.
Yet again, TryServer continues to be almost half the load of all branches combined on the entire infrastructure.
Details:
- September 15th set a new record high of 160 pushes in a single day!
- We are still double-running unittests for some OS; running unittest-on-builder and also unittest-on-tester. This continues while developers and QA work through the issues. Whenever unittest-on-test-machine is live and green, we disable unittest-on-builders to reduce wait times for builds. Any help with these tests would be great!
- The entire series of these infrastructure load blogposts can be found here.
- We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.
Detailed breakdown is :
Here’s how the math works out (Descriptions of build, unittest and performance jobs triggered by each individual push are here:
15 Nov 2010
JohnMozilla
On Saturday 20th November, Mozilla is hosting the Firefox Developer Conference here in Tokyo.
Its a one day event, but will be jam packed with interesting discussions about Firefox4 for desktop, Fennec for mobile, Jetpack, and HTML5. There might even be something about Release Engineering and our developer infrastructure! 
If you’re nearby, please do stop by – it looks like it will be informative and fun!
Older Entries Newer Entries
RSS
