“The answers you seek are in Norway”

huh???

Suddenly, I realize that *making* fortune cookies could be a lot of
fun! 🙂

Heading home Friday evening, I stopped by Aki’s desk and took these quick photos. At first glance, these might not look all that exciting:

…but look closer… closer… see the little bits of paper taped on the machines??

This is actually really really big! This is the beginning of a pool of mobile devices slaves to automatically run unittest and talos on mobile builds.

Aki’s been using these two machines to figure out linux mobile hardware setup problems around the hardware setup, toolchain setup, memory limitations as well as how to get a buildbot slave running on the device, communicating to a buildbot master.
Since end of Sept, we’ve been producing linux-arm builds of fennec automatically on checkin and every night. However, humans were still needed to manually download specific builds onto the physical devices when they had time, and manually run some tests. There’s still no automated unittests, talos, graphserver for mobile.

The build automation was done by updating all the linux slaves in our existing pool of slaves, and then doing cross-compile builds. That allowed us to re-use a bunch of existing infrastruture. However, we need to use actual devices for testing and performance… which opens a whole set of problems we have to figure out for the first time. For example:

• if the device hangs during a test, how do you powercycle it? (Most people first suggest removing battery and running on remote switchable a/c; however, these devices refuse to boot unless a battery is present, and it looks like they need a physical button pushed on after a power reset, so we’re still looking for a solution – all suggestions welcome!)
• How do these devices communicate to other machines; how do we get builds onto, and test data off from, these devices? The buildbot master needs to tell slaves when and what to do. The slaves need to post results to buildbot master as well as ftp.m.o and graphserver. Something (nagios?) needs to monitor to see if devices are ok. If we use wifi, does that cause noisy deviations in our test results? If we use ethernet, what s/w drivers and cable converters do we need?
• If we setup devices in office, for easier manual reboots, what ssh/firewall changes do we need to get access to other build systems in secured area in colo?
• What toolchain do we need to install on these devices? Both of these devices were “polluted” when we got them, meaning no-one was exactly sure whats on them, and we’re fairly sure they’re different from each other. We’ve got some more machines on order, so will have to figure out clean-setup-instructions as part of this setup.
• Is there enough memory in the device to run o.s. + buildbot + talos + talos-tools + fennec? Can far can we increase memory? Do we have to rewrite buildbot/talos to fit into low memory situations?
• We can’t build on these devices, but unittests currently requires building first before testing.
• What can we do about turnaround time? These devices are *SLOW*… Break testsuites into chunks that can be run in parallel?

And all that is before you get to the the fun stuff… the usually expected problems, like how many tests fail because of different display properties, memory availability, slower CPUs causing test timeouts, environment differences, etc. etc. etc.

While there’s still lots to solve along the way, its exciting to watch Aki making methodical, persistent and rapid progress. Very very cool.

Last Wednesday, we setup the new Firefox3.1 project branch. (Details on the machine-setup work coming in a separate blog post.) Meanwhile, I thought people might find this diagram useful to help understand the repository cloning/branching work that happened.

Note:

• To keep terminology consistent, last week we created a “project branch”. We’ll be creating “release branches” on this project branch for each milestone.
• Before we allow checkins to the new mozilla-1.9.1 project branch, we will need to carefully cross-check all the changes landed on mozilla-central since the mozilla-1.9.1 branch was created (The area in the dotted circle) and verify that those changes were also landed on mozilla1.9.1.
• We want the mozilla-1.9.1 branch to start from FF3.1beta2. To speedup opening the mozilla-1.9.1 branch, we branched early. This means that before we allow checkins to the new mozilla-1.9.1 project branch, we will need to carefully cross-check all the changes that landed on mozilla-central since the mozilla-1.9.1 branch was created (The area in the dotted circle), up to the branch for FF3.1beta2, and verify that those changes were also landed on mozilla1.9.1. Any later FF3.1beta2 respin would require patches to also be landed on mozilla-1.9.1.
• There’s been discussions recently on whether we’d have a FF3.1b3 or not. I included FF3.1b3 in the diagram so that, if we do decide to do Beta3, people could see where it would be branched from. This is not meant to state or imply that we are doing Beta3! 🙂
• The diagram above shows logical active-branches-of-code. At the SCM repository level, some of these active-branches-of-code are implemented as in-repo-branches, and some are implemented as separate-cloned-repos. For the sake of clarity, I’ve omitted those details in this post, but would be happy to go into further detail, if anyone is interested.

Hope that helps?

tc
John.

UPDATE#1: added extra text in 2nd bullet point, describing the cross-branch synchronization, to clarify how we branched and why.

UPDATE#2: Sam Sidler and and Gavin Sharp both pointed out that the branching diagram had an error when describing how FF1.5 was branched from FF2.0. They are correct, and I thank them. The branch diagram above has been updated, and is now correct. I since found the same error in a presentation we gave here at the 2008 Mozilla Summit (diagram had same error), and a brownbag in Mountain View office (diagram had same error).

…tweaking their usual shopping cart:

…and for the serious shoppers, there is even:

While I like their sense of humour, I still can’t believe its “Christmas shopping season” already?!?

Found this article and had to share. It struck me as a great experiment, yet somehow I had never even considered this idea before.

An artist wanted to see just how dedicated the letter sorting people were in the Royal Mail in England the UK. So, she mailed letters addressed using puzzles instead of the usual name-and-address format. Of the 130 letters she mailed, only 10 never arrived. Really good delivery rate, imho, especially considering at least one of them required the postal workers to complete the crossword!!

For the full story, and other examples, click here.

I’m delighted that voting today was really really busy.

I live a short walk from my polling station, so walked over before breakfast, hoping to beat the rush. No luck, it was mobbed. By 8.30, lines were shorter, but still… At this point, I just patiently waited in the line, and ended up way late for work. But very very proud to have voted and proud of how good humored everyone else in line was.

While buying supplies for the election coverage tonight at the office, I found Joe Six Pack:

Its real beer, from a local microbrewery. I just love that someone thought of doing this… and also that they used their “Mavericks” beers for this. Lets see which one people prefer!

😀
tc
John.
=====

ps: Continuing in the election theme – here’s a local prankster who tried to do some street renaming: