Firefox 2.0.0.8 by the (wall-clock) numbers

Mozilla released Firefox 2.0.0.8 on Tuesday 18-oct-2007, at 5.30pm PST.

From “code freeze” to “fix available to public” was 14 days 2 hours wall-clock time, which included a 7day Beta period (this was a non-firedrill release). Build&Release took 68 hours.

15:00 04oct: Dev says “go”
15:33 04oct: 2008rc1 builds started
18:20 04oct: linux builds handed to QA
19:45 04oct: mac builds handed to QA
12:45 05oct: win32 signed builds handed to QA
20:05 05oct: update snippets on betatest update channel
11:30 08oct: 2008rc1 halted. Respin declared for bugs 398422 and 398837
15:20 08oct: Dev says “go”
16:05 08oct: 2008rc2 builds started
19:50 08oct: linux builds handed to QA
22:05 08oct: mac builds handed to QA
00:45 08oct: win32 signed builds handed to QA
01:00 10oct: update snippets on betatest update channel
15:05 10oct: QA says “go” for Beta
16:05 10oct: update snippets on beta update channel
11:55 18oct: Dev & QA says “go” for Release; Build starts final signing, bouncer entries
14:25 18oct: final signing, bouncer entries done; mirror replication started
17:30 18oct: update snippets on live update channel; announced

While Build Automation in FF2008 was much smoother than FF2007, this was not yet a “human free” release:
1) signing still done manually in two places. This is known and expected.

2) As the initial build steps get automated, the steps near the end of the process become more visible. Steps like pushing-updates-snippets-to-channels, adding bouncer entries, starting mirror replication and monitoring mirror replication are now worthy of automation attention. Combined, these took 6.5 hours of the Build time, and were all manual.

3) It was interesting to note that we needed only 3 hours of mirror replication time to reach 65-72% mirror absorption. There’s been quite a lot of folklore around how long it takes for mirror replication, but as mirrors have changed, we’ve been measuring to get concrete data. Even for a mirror replication in daytime, like in this release, we saw quick absorption around 60% within the first 2hours. We are still experimenting with IT to find out how much absorption is “enough”, so decided to wait until absorption hit around 70%, just to play safe. This is definitely not a science, we will continue experimenting with this in future releases… any comments/feedback very very welcome!

take care
John

The Tipping Point by Malcolm Gladwell

It felt to me like he was covering a bunch of different topics, or short essays, all in the same book. Some resonated with me much more then others. In particular, these two:

Chapter1: Epidemics:
To me, I always thought of epidemics in the medical sense, flu outbreaks, avian flu, etc. However, I was fascinated by how the same study of epidemics could be applied to other completely unrelated fields. Human fashion. Graffiti. Litter. Teenage smoking. One example he detailed was a gonorrhea outbreak in Colorado Springs, Colorado (population 100,000+), which tipped over from statistically insignificant background noise, to epidemic, because of the activity of 168 people in 6 local bars in 4 small neighborhoods of the town. A statically insignificant small group of people. Another epidemic example he detailed was how “The Broken Window Theory” was applied to the New York City subway, dealing with graffiti, and fare-evaders. I particularly like the two cultural insights behind how they dealt with fare-evaders.

The first culture changes was with the cops. Seem the cops preferred to chase bigger fish, instead of wasting the afternoon doing paperwork on one trivial misdemeanor fare-evader arrest. However, a few simple ideas changed things dramatically. Instead of doing onesy-twosy arrests, they had 10+ plain clothes cops handcuff fare evaders to each other on the platform like a large daisy chain, and then only come up from the subway station with a “full” daisy chain. Instead of driving each suspect through traffic to the police station, they converted a bus into a mobile police station so that paperwork, fingerprints, background checks could done on site without a slow trip to the police station. Instead of just fining someone for fare-evading, and letting them go, they always ran a full background criminal check on each fare evader – and found something interesting: 1 out of 7 fare-evaders had outstanding arrest warrants; 1 out of 20 had illegal weapons. Suddenly, the cops on the street felt it was not “just a fare-evader”… now a daisy-chain of 20 fare-evaders was a really interesting surprise bonanza box, and the easiest way in the world to catch “real” bad guys.

The second cultural change was with the subway riders. Seems that the general public attitude had deteriorated to “why should I pay, if everyone else is evading”. Even people who would not normally break the rules, who would never consider themselves as criminals, were avoiding paying fares “because everyone else did it too”. However, the daisy-chain of handcuffed fare-evaders was a clearly visible deterrent, a reminder of what the rules were, and how society expected people to behave. Quickly, the number of people trying to fare-evade dropped. Which meant more people paid. It also created the perception of the subway being safer, so more people felt safe to choose to travel on subway. So even more people paid. All this meant they had more money to fix other problems, like old rolling stock, tracks, ticketing systems, etc.
Of course, the NYC subway is not perfect, then and now. However, a handful of small, carefully chosen physical changes, triggered a couple of critical cultural changes, which turned around a problem that had previously been almost given up for lost. I realized its really easy to trick yourself into thinking that a big reward requires a big effort project, and then with those mental blinkers on, only allow yourself to consider big ticket items. “How little things can make a big difference” is a really good subtitle for this book.

Chapter 5: Human group size:
Gladwell contends that human groups, and intra-group human loyalties, only scale up to about 150 people. His suggests its a function of the limits of the human brain to handle all the combinations of relationships between everyone in the group. For groups under 150 people, the inter-personal relationships, friendships, peer pressure, of everyone knowing each other tends to keep people focused and working together towards a common group goal. However, groups that grow over 150 quickly lose internal cohesion, internal focus, because they are just too big for everyone to really know everyone else. Once that cohesion breaks down, people instead start forming smaller subgroups, trusting their own subgroup, questioning motives of those other subgroups, and focusing on their own personal agendas. In a Western-business culture setting, it would be called internal-company-rivalry.

I once worked in a company as it grew from 42people to 180people, and experienced that change of internal cohesion myself. At the time, I just knew “things had changed”, but only later, looking back, I figured out it was to do with how many people were in the company, and not feeling like we were all working together anymore. Personally, I’d always thought the change from “we tight knit small band of brothers in a small company” to “I’m just a nameless cog in a faceless bureaucracy” started to happen somewhere around 100 people, but that was just a gut guess. However, Gladwell shows examples of hunter gatherer tribes, ranging from Australia to Greenland, all averaging just under 150 people per village. The Hutterites (a religion similar roots to the Amish) have a strict policy that once a community approaches 150 people, its splits into two equal separate communities. In business, the same principle is followed by Gore Associates (the manufacturer of GoreTex)… and they believe that contributes to why they have employee turnover 1/3 of the industry average, are profitable for 35 years in a row (and counting), and are constantly successfully innovating new products and markets… all without formal management structures. In groups under 150, he suggests that “personal loyalties and direct man-to-man contacts” keep everyone focused on doing the right thing for the organisation.

There’s quite a lot of the book given to describing Mavens, Connectors and Salesmen. I’m still thinking that part over, not convinced yet.

Would I recommend this book? Yes, especially these two chapters.

Firefox 3alpha8 by the (wall-clock) numbers

Mozilla released Firefox3a8 on Thursday, 20-sep-2007, at 08:30am PST.

This was a manual build run (not automated on trunk yet), and an alpha release (not a high-priority security release), so the numbers are quite different to the earlier Firefox2.0.0.7 release. Even as an apples-to-baseballs comparison, I thought the numbers were interesting and worth sharing. From “code freeze” to “available for public download” was 14.33 days wall-clock time. Of that time Build&Release took 2.25 days (55 hours including the respin).

00:01 06-sept: M8 code freeze, tree closed
18:48 11-sept: Dev verifies last fix landed, and gives “go” to build
20:46 11-sept: Build starts building
01:26 12-sept: blocker bug#395862 filed
08:40 12-sept: blocker patch landed
09:22 12-sept: Build restarts building
13:49 12-sept: linux & mac builds handed to QA
18:01 12-sept: signed-win32 build handed to QA
11:17 18-sept: QA signed off on all builds
00:01 19-sept: Build supposed to finish signing and publish builds externally
02:58 20-sept: files available externally for download
08:31 20-sept: mirror absorption completed and release announced

There were a few interesting point about this release
1) There was a 5.75 day delay between when the code freeze started, and when the tree was first deemed ready for builds to start.
2) After builds started, a last minute blocker bug caused those builds to be abandoned and new builds started. This respin cost Build 12 hours.
3) Between 13-17sept inclusive, both Build and QA switched to work on FF2.0.0.7 (a higher priority security firedrill release). This caused wall-clock delays.
4) After QA signoff, we delayed releasing Firefox3a8 from 18sept to 19sept, to avoid traffic load of releasing Firefox3a8 on the same day as Firefox2.0.0.7.
5) There was a 1 day delay between when QA signed off on the builds and when Build group ran the remaining manual steps (signing installer, pushing bits externally, etc). These remaining Build steps only took a handful of hours to complete. However, the person doing those remaining manual steps (ie me!), was sidetracked with other non-release work.

Firefox 2.0.0.7 by the (wall-clock) numbers

Mozilla released Firefox 2.0.0.7 on Tuesday 18-sep-2007, at 3pm PST. For background on this security firedrill, see here.

This was our first production run using the new automation, so I thought the following wall-clock numbers might be interesting. From “initial report” to “fix available to public” was 6.25 day wall-clock time. Of that, Build&Release took just under 2 days (45 hours).

09:00 Wed: bug reported 9am (or 8.30am?). Dev start working on fix
13:40 Fri: fix landed on 1.8 branch
14:30 Fri: build started
18:30 Fri: linux builds handed to QA
22:30 Fri: mac builds handed to QA
22:30 Fri: win32 unsigned builds handed to QA
11:58 Sat: win32 signed builds handed to QA (1st time)
01:30 Sun: win32 signed builds handed to QA (2nd time, rebuilt on old
machine)
12:10 Sun: update snippets pushed to beta update channel
15:00 Tue: update snippets pushed to live update channel; announced

Full disclaimer, while this fast turnaround kept Mike Shaver happy, it was not yet a “human free” release. We hit 4 issues, which required manual intervention:

1) last minute question about possible CVS-cross-branch tagging problem in automation scripts. Problem unconfirmed, but decided to manually tag anyway, just to be safe. Problem still unconfirmed, but test case now designed to clarify for future releases (see bug#396290)

2) l10n builds on win32 had the wrong cr-lf settings in README, EULA. This root cause of this was an internal communications snafu within the Build&Release group. Historically, we build l10n win32 on different machines to win32 en-US machines. As part of automation rollout, some folks thought the l10n win32 builds were now being done on same machines as en-US for 2005+2006, some thought l10n win32 was still being built on different machines. Because these different machines have different cygwin cr-lf settings, this problem first surfaced as a problem where text files like README, EULA had the wrong cr-lf settings. It was caught by a recently added test. Rather the debug/fix the problem, we just built on the old l10n machine and shipped that for win32. This miscommunication has been clarified. Still checking if there’s anything else here we missed.

3) signing still done manually. This is known and expected. Note: as the step-before-signing finished late at night, the automation waited overnight until human woke up and did the signing the next morning.

4) manually copying bits from stage to build-console after each step completed. This was a known issue that we expected to have fixed for the scheduled 2007 release, but was not yet in place when this Firefox2.0.0.7 firedrill started. After each step finished, we had to manually copy files between “stage” and “build-console”, so that the next step would find the files it was expecting. Was intrusive and annoying. On track to be completed before end sept. (see bug#396438)

tc
John.
=====
ps: After the release, we’ve heard a few questions about the new GPG key. The previous key had expired sept2006, and was still being used, until this new key was available in August2007. We used the new key in Firefox3a7, and also in Firefox2007. After the Firefox2007 release, some questions about how to confirm the new public signing key on key servers. We’ve reviewed the keys on key servers, and they seem ok, but are still investigating. (see bug#377781).

A taste of Burning Man 2007

Burning Man 2007 happened in the Nevada desert again this year. While a bunch of my friends, and some work colleagues did go, I ended up not going this year. Instead I stayed at home, working, and living vicariously through video snippets and the occasional news headline. It was interesting to note how easy it is to find street parking, and also commute to work, while Burning Man is going on… and how for days after it ends, the city is packed with absolutely *filthy* cars!

In the news, Arsonist burns The Man early; organisers rebuild on-the-scene

For mechanical inventiveness, there’s a trebuchet for launching burning pianos, Dance Dance Immolation, Spider-walking Transporter, Synchronized flame throwers, and the DMV showing off some art cars.

For large scale, there’s the Oil Derrick being installed, being enjoyed up close, being burnt up close and being burnt from 1 mile away.

…and of course, for spiritual, there is The Last Temple.

[sigh]

BBC TopGear in a Bugatti Veyron

The folks in BBC TopGear do it again. Bugatti claims the Veyron is “the fastest production car in the world”. The basic stats of the car:

  • fastest car in production in the world
  • 1,001 horsepower
  • 8.0L w16 engine (they stuck two v8 engines together!)
  • 4 turbochargers
  • top speed 253mph / 407kph
  • and 10 radiators, to stop it all overheating!
  • more specs on wikipedia

So, of course, the folks from TopGear ask the real question: Does it actually go up to that theoretical top speed? To find the answer, watch this! (its ~9mins video clip). Amazing well put together snippet, good camera work, music. Even if you dont care too much about cars, you should watch this because of the production quality. And, well, if you’re interested in cars, its even better! Just dont try this at home!

UPDATE:TopGear and YouTube are playing “whack-a-mole”, taking down different copies of this video clip, but its being reposted by fans as fast as they take them down. Search for “Top Gear Bugatti Veyron”. Its 7min40sec long and starts with a black Veyron on a rainy road with the voiceover saying “Vlad the Impaler is back”! Dont look at any shorter video, as it will ruin the ending!

UPDATE: BBC’s TopGear stopped playing “whack-a-mole” and now have topgear.com, as well as an official TopGear channel on youtube.com, each with great clips from various shows.

(john; 14nov2009)

Le Dome du Marais, Paris

Even though its mentioned in the latest lonely planet, its an upmarket place, with good food and wine. Make sure to reserve a table, even if you just call ahead that same day. They have enough tourists & non-locals come through that they speak reasonable english. Ask for a table in the dome room, against the wall. Nice scenery and the people watching can be good too.
Like all french restaurants that care about their cooking, they do “extra mini courses” between the official 3 courses on the menu, so when you add up the little snack courses before and between the other courses, it turns into a 6-7 course meal. All very very tasty, and very filling! They have a full time wine somalier who is quite a personality, loves his job, and is good at it. Unless you *really* know what you are doing with wines, just tell him what you do/dont like about wine, and how much you’re willing to spend, enjoy the interaction and follow his advice.
While the food and wine was all good, the desert left me speechless. Something orange-and-lemon, as best as my french understanding could take me. At first glance, it looked like a tumbler glass containing orange segments in foamy orange juice, and a small baked cake with a small dollop of cream on top. However, it turned out to be *way* better then that.

The orange foam had the same consistency you would get by pouring Coke into a glass too quickly… except the foam was orange coloured, not Coke-light-brown. And the instead of the foam collapsing down, like Coke foam does, this foam remained in unchanged shape all the time. If you closed your eyes, and moved a spoon through it, the spoon would meet *no* resisitance and you wouldnt be able to tell when you were moving the spoon through foam or not. It was that light! Then I noticed that the orange segments were not just any old broken chunks of orange. Instead, they were all perfectly formed complete segments, each individually peeled of their skin. And the orange juice was mixed with some sort of orange liquor. The dolop of cream on the side cake also had some orange zest added, so had a mild orange flavour to it. The cake was some variation of a lemon pound cake, and it was sitting in a little white baking dish, and the cake sitting in a few mm of lemon liquor. The transition from light orange foam, to stronger orange to strongest orange juice, to light orange cream, to light lemon cake, to stronger lemon liquor…. was all just awesome. Dont know how else to describe it.

Even now, 2 months later, I still think about that desert… and need to go back again!

Le Dome du Marais
53 bis, rue des Francs Bourgeois
75004 Paris
ph 01-42 74 54 17
email ledomedumarais@hotmail.com
open tues-sat

Paris Car Culture

Obviously local environment, and local culture, influence what people look for in car design and car usage. After living in California for 9+ years now, and then arriving in Paris for a couple of weeks, I found the car culture differences between Californian cars and Parisian cars quite amazing. More then I realized from previous trips.

  1. Tiny streets, tiny parking spaces. By tiny, I mean that villages in West of Ireland have bigger roads. Simply fitting your car down these narrow one-way streets, with parked cars lining each side of the road is a challenge in a small car. VW Golf counts as big. Very common here are Smart Cars, small, fast, practical city cars, and the fact that there are some made in France is probably helping too.
  2. Expensive petrol/gasoline – hence an interest in hybrids. So what new, thats all the rage in California too, right? Well, then I spotted this. Turns out that electric cars have reserved spaces throughout Paris, and the parking meter includes a recharging plug for the car!