John O’Duinn’s Soapbox

A quick introduction.

Rail has joined Release Engineering, and will be based in Moscow.

Some of you may already know him from his automation work in the Turkish and Russian locales for various Mozilla projects including Firefox. Or maybe you know him from his work in openoffice.org, or debian, or pootle. Anyway, he’s been using his skills in automation to make each of those better, so we’re delighted to have another like-minded automation person joining RelEng.

On irc you’ll find him as “Rail”, which is pronounced “ray-eel”.

(Oh, for the curious - he’ll be based in Moscow, as Toronto is “not cold enough” for him right now!)

I recently found this clip of Clifford Stoll at TEDTalk.

I’d never seen him live, so had no idea what it would be like. His continuously exploring curious mind was refreshing to me, and served as a wake up call to me to not get stuck in my ways. Further, the way he brought practical everyday logic to early education was something I found personally moving. His approach of explaining complicated things to kids when they’re curious and eager to learn, and before they start to believe “its supposed to be hard”, really resonated with me. The experiment he put together for his 8th grade class to calculate the speed of sound was straightfoward, understandable and quick. His attitude reminded me of the people, both in and out of school, who had the most influence during my education, and I found myself thanking them yet again for their inspiration. His closing quote from the bell inscription really resonated with me (no pun intended).

Watching this inspired me to dust off my old 2nd edition copy of his book “The Cuckoo’s Egg”, and reread that.

Its now of course an old story from the late ’80s about Clifford, when he was a university undergrad student in Berkeley tracking down a bug in some auditing software. He eventually discovers that the auditing errors are because of unauthorized breaches in the university computers - being used as a conduit to attack classified military research computers. As he untangles the giant ball of interwoven strands of the problem, he just wrote everything down like it was a personal journal, interweaving details of home cooked meals with his new girlfriend, emails and hone calls with sysadmins at the targetted systems, invented hacks to detect when intruders returned to their computer systems, even his own self-questioning of his hippie anti-establishment roots as he cycled across campus to meet with the NSA suits who took interest once evidence of KGB involvement started to emerge.

I found it a fascinating readable story, just like as if a close friend was telling me about some strange thing that happened that day at work… made only slightly more surreal when you keep in mind that its all true.

If you use our xulrunner builds, you know we provide builds for xulrunner on the 1.9.0, 1.9.1, 1.9.2, mozilla-central and lorentz branches. Almost all of these xulrunner builds are produced through the same pool-of-slaves that are used for Firefox - except for the oldest xulrunner1.9.0 which are still being produced the old way - on dedicated unique machines.

This blog post, as well as mfinkle’s blog post here, the dev.platform newsgroup posting here and the discussion in the Platform meeting on 27jan2010, are to let people know we’re planning to stop xulrunner builds and releases on the 1.9.0.x branch. This would only affect Mozilla building and releasing xulrunner 1.9.0.x. Mozilla will continue to build and release xulrunner from the 1.9.1, 1.9.2, mozilla-central and lorentz branches. Obviously, the source code to xulrunner 1.9.0.x would remain available and developers could always build and release their own xulrunner 1.9.0 packages from the source code if they wish.

While considering this, we realized that we don’t know how many people are using which branch of xulrunner; further, we have no idea if people are using pre-built xulrunner or building their own. If you are using xulrunner1.9.0, please let us know in the newsgroups if this will impact you either way. Meanwhile, unless there are objections, we’ll continue preparations to stop xulrunner 1.9.0 builds in bug#544678.

As you all know by now, on Friday we released Fennec1.0. The new features and improvements are already being covered by other people in great detail, so I thought I’d focus on something totally different that happened behind the scenes for this release.

Lots of things were unique about Friday’s fennec1.0 release, but here’s the three that stuck out in my mind.

1. Automation on mobile phones is much trickier then automation on desktops. See blogposts by jhford, aki and occaisionally myself on all the fun with imaging, setup, automation, scaling, calibration of all these machines. There’s still more work underway with new phones and new platforms - stay tuned!

2. Fennec1.0 included a multi-locale version, which we believe is a first for Mozilla.

   Until now, whenever time we shipped localized versions of a product, each separate download contained only one localization. If you wanted a different locale, you downloaded a different installer - or if you were savvy enough, you could tinker with some add-ons to switch xpi files. Historically this kinda makes sense on a desktop computer, and allowed us to save on download size, and improve first-time out-of-the-box startup time. However, an application installed on a phone could be shipped to a range of locales, so “multi-locale” was born. The basic idea is to detect the locale of the host OS on first run, and if that locale is one of the locales supplies in multi-locale installation, use that. There are tons of edge-cases though, which made for lots of fun, but I think it was well worth the effort in the end. The multi-locale version of fennec1.0 included 12 locales (cs, da, de, es-ES, fi, fr, it, nb-NO, nl, pl, pt-PT, ru). There are also 38 different single-locale versions of Fennec available for individual download.

   With this now proven to work in Fennec1.0, if you feel we should do something similar for Firefox, please comment below, or here, or in bug#485861

3. As part of all this, we had to learn how to get deb files generated at the right step in release automation, and how/where to host these deb files in order to make them useful. We will be reusing this knowledge.

Enjoy Fennec, and let us know what you think!

ps: the cake was great!

Looking at the approx 4.5million people who moved to FF3.6.0 since yesterday morning:

• ~66% (3million) downloaded the FF3.6.0 installer and installed from it
• ~33% (1.5million) manually did Help->CheckForUpdates

This is very similar to what we saw with the FF3.5.0 release. As far as I can tell, we did not gain 3million *new* users yesterday. My belief, without data to prove it, is that most/all of those 3million people were *existing* Firefox users who chose to do a download install, instead of doing CheckForUpdate. I guess there are cases where a Firefox user might want to do a fresh install, but I would have thought that most people would prefer the easier in-place upgrade of Help->CheckForUpdates.

To me, I was hoping to see:

• ~20% downloaded the FF3.6.0 installer and installed from it
• ~80% manually did Help->CheckForUpdates

Is there something we could do differently, so that next time, most existing Firefox users do Help->CheckForUpdates, instead of a fresh download? Is it possible that people hear about the new release, but do not know about Help->CheckForUpdates? What do you think?

As you all know by now, Firefox 3.6.0 shipped today. The new features and improvements are already being covered by other people in great detail, so I thought I’d focus on something totally different that happened behind the scenes for this release.

Firefox 3.6.0 is the first release where we ran all our automated unittests and talos performance suites on the officially branded release builds (you can see the test results for yourself by clicking on the image!).

As you know, we run unittests and performance tests on unbranded builds for each checkin, and also on the unbranded nightly builds. However, releases are branded builds - and until now, the best we could do was run unittests and performance suites on unbranded builds, and if all looked good, then branch from that same source code, rebuild with branding enabled, and hand those branded builds to QA for testing and release.

While this approach has worked “good enough” for years now, we had to carefully track the possible differences between unbranded and branded builds, and hope that compile changes caused by branding was covered by manual testing. To fix this required some fiddly reworking of unittests and talos performance suites. For details, the curious can read through bug#383136, bug#457753, bug#517212 and all of their linked bugs.

Big tip-o-the-hat to catlee for making this happen, and hope you enjoy your new Firefox!.

This 8minute video shows the winner of “Ukraine’s Got Talent” drawing pictures in sand with light from underneath shining up to a projector. Sounds kinda strange at first, but don’t be fooled. I found it mesmerizing and quite touching.

Before you click the video link, you need to read this explanation first.

Kseniya Simonova, 24, uses sand and music to show how ordinary Ukrainian people were affected by the German invasion during World War II. She begins by creating a scene showing a couple sitting holding hands on a bench under a starry sky, but then warplanes appear and the happy scene is obliterated. It is replaced by a woman’s face crying, but then a baby arrives and the woman smiles again. Once again war returns and Miss Simonova throws the sand into chaos from which a young woman’s face appears. She quickly becomes an old widow, her face wrinkled and sad, before the image turns into a monument to an Unknown Soldier. This outdoor scene becomes framed by a window as if the viewer is looking out on the monument from within a house. In the final scene, a mother and child appear inside and a man standing outside, with his hands pressed against the glass, saying goodbye.

The Great Patriotic War, as it is called in Ukraine, resulted in one in four of the population being killed with 8-11 million deaths out of a population of 42 million, and you can see the very personal impact of her work on the live audience.

If you don’t care about Talos performance results, or Talos hardware, stop reading now!

Its been a hectic couple of weeks on the Talos front since my last post. Here’s a quick summary of whats been going on:

In RelEng, we’re using this recalibration as a chance to cleanup a few long-standing details of how Talos slaves are configured. These changes to the Talos ref images include:

• setting up these new machines in the Build network, with the rest of the build machines, not in the QA network, where Talos has been running since its inception. This will allow us to clean up some VPN and firewall configurations within the colo.
• changing accounts on Talos machines to be consistent with all the other build machines
• making sure configuration management software is installed.
• bumping up OS versions, which have been intentionally been unchanged since 2007!:
  • OSX10.4: we’re leaving on old machines for now. From initial glance, this doesnt work on the new minis, and is already de-supported on mozilla-central, so we might end up just leaving it. More on this as it emerges.

  • OSX10.5: upgraded from 10.5.2 to 10.5.8.
  • WinXP: no OS change
  • WinVista: replaced with Win7
  • ubuntu7: replaced with fedora12 (32bit)
  • ubuntu 64bit: replaced with fedora12 (64bit)

In IT:

• The first batch of 100 minis arrived just before New Years. Last week, Matthew and Phong (with a little help from aki, jhford and myself) spent a day unboxing, removing wrappers, sorting power cables, putting on asset tags, scanning serial numbers, etc, etc, etc. Scroll down for photos!

• the racks were delivered, installed and cabled. Also power upgraded and air conditioning prep’d by middle of last week.
• as ref images were completed, and tested in staging, use them to image a set of minis.

As of Friday evening, 60 of the new slaves are imaged using the new OSX10.5.8, WinXP and Win7 reference images, racked and powered.

In the coming week, we’ll:

• have the remaining 40 imaged with linux32, linux64.

• schedule a downtime to have all these new slaves enabled in production, along side the existing production slaves.

  If all goes well, after about 2 weeks, we’ll take the old systems out of production, and declare that first phase done. Stay tuned for more details.

  (If you’ve read this far, and have questions about anything that I’ve missed, please let me know.)

  
  
  
  

Summary:

• Numbers were down significantly compared to the previous few months. I’d guess an indicator that Mozilla entered the end phase of the Firefox 3.6.0 and Fennec 1.0 releases, both around the same time. The holidays in the last half of the month also reduced the load.
• The numbers for this month are:
• 1,268 code changes to our mercurial-based repos, which triggered:
• 15,345 build jobs, or ~65 jobs per hour.
• 33,263 unittest jobs, or ~44 jobs per hour.
• 31,372 talos jobs, or ~42 talos jobs per hour.
• Yet again, we enabled new Talos suites, and also disabled some unittest suites at different times during the month. For simplicity, I’ve ignored those changes for now, and will include in next month’s data.
• Interesting to note that the trend of “what time of day has most checkins” changed again this month. Not sure what this means, but worth pointing out that each month seems to be different. This makes finding a “good” time for a downtime almost impossible.
• We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.

Here’s how it looks compared to the year so far:

Detailed breakdown is :

Here’s how the math works out:

The types of build, unittest and performance jobs triggered by each individual push are best described here.

Most “management” books focus on working in for-profit companies. By contrast, this was the first book I’ve read that specifically addressed non-profit organizations.

There was some of the usual stuff about working with managers and employees. But there were also sections on working with volunteers (as opposed to employees), working in mixed employee+volunteer settings, fund-raising, public relations, outreach, political lobbying and the use of metrics to improve how you work. In an era when so many for-profit companies are focused on short term quarterly profits, and not on longer term health of their organization, or their people, I found it refreshing to see these self-evident truths written down:

“…[it is your] responsibility to ensure that the association is still exists in 10 years time to do the good work”

“…[it is your] responsibility to leave organization in better shape then [you] found it.”

“keep everyone informed”. (Something I struggle with constantly, given the very distributed nature of RelEng here in Mozilla.)

At 99pages, its a quick read compared to all the weighty tomes you usually find in that section of the bookstore. Later, after some thought, I went back and read it again, skipping some sections, and carefully re-reading others slowly. He also referred to several other interesting books that I’d never heard of, making my amazon.com wishlist even longer! I found his writing style terse, informative, down-to-earth, and occasionally a little repetitive. I since discovered the author is a Mensan, former US Marine and former Massachusetts State Senator, which explains a lot!

Overall, it was repetitive in a few places, not relevant to Mozilla in a few places, but I still found it well worth the read, and the re-read!