Firefox 2.0.0.8 by the (wall-clock) numbers

Mozilla released Firefox 2.0.0.8 on Tuesday 18-oct-2007, at 5.30pm PST.

From “code freeze” to “fix available to public” was 14 days 2 hours wall-clock time, which included a 7day Beta period (this was a non-firedrill release). Build&Release took 68 hours.

15:00 04oct: Dev says “go”
15:33 04oct: 2008rc1 builds started
18:20 04oct: linux builds handed to QA
19:45 04oct: mac builds handed to QA
12:45 05oct: win32 signed builds handed to QA
20:05 05oct: update snippets on betatest update channel
11:30 08oct: 2008rc1 halted. Respin declared for bugs 398422 and 398837
15:20 08oct: Dev says “go”
16:05 08oct: 2008rc2 builds started
19:50 08oct: linux builds handed to QA
22:05 08oct: mac builds handed to QA
00:45 08oct: win32 signed builds handed to QA
01:00 10oct: update snippets on betatest update channel
15:05 10oct: QA says “go” for Beta
16:05 10oct: update snippets on beta update channel
11:55 18oct: Dev & QA says “go” for Release; Build starts final signing, bouncer entries
14:25 18oct: final signing, bouncer entries done; mirror replication started
17:30 18oct: update snippets on live update channel; announced

While Build Automation in FF2008 was much smoother than FF2007, this was not yet a “human free” release:
1) signing still done manually in two places. This is known and expected.

2) As the initial build steps get automated, the steps near the end of the process become more visible. Steps like pushing-updates-snippets-to-channels, adding bouncer entries, starting mirror replication and monitoring mirror replication are now worthy of automation attention. Combined, these took 6.5 hours of the Build time, and were all manual.

3) It was interesting to note that we needed only 3 hours of mirror replication time to reach 65-72% mirror absorption. There’s been quite a lot of folklore around how long it takes for mirror replication, but as mirrors have changed, we’ve been measuring to get concrete data. Even for a mirror replication in daytime, like in this release, we saw quick absorption around 60% within the first 2hours. We are still experimenting with IT to find out how much absorption is “enough”, so decided to wait until absorption hit around 70%, just to play safe. This is definitely not a science, we will continue experimenting with this in future releases… any comments/feedback very very welcome!

take care
John

The Tipping Point by Malcolm Gladwell

It felt to me like he was covering a bunch of different topics, or short essays, all in the same book. Some resonated with me much more then others. In particular, these two:

Chapter1: Epidemics:
To me, I always thought of epidemics in the medical sense, flu outbreaks, avian flu, etc. However, I was fascinated by how the same study of epidemics could be applied to other completely unrelated fields. Human fashion. Graffiti. Litter. Teenage smoking. One example he detailed was a gonorrhea outbreak in Colorado Springs, Colorado (population 100,000+), which tipped over from statistically insignificant background noise, to epidemic, because of the activity of 168 people in 6 local bars in 4 small neighborhoods of the town. A statically insignificant small group of people. Another epidemic example he detailed was how “The Broken Window Theory” was applied to the New York City subway, dealing with graffiti, and fare-evaders. I particularly like the two cultural insights behind how they dealt with fare-evaders.

The first culture changes was with the cops. Seem the cops preferred to chase bigger fish, instead of wasting the afternoon doing paperwork on one trivial misdemeanor fare-evader arrest. However, a few simple ideas changed things dramatically. Instead of doing onesy-twosy arrests, they had 10+ plain clothes cops handcuff fare evaders to each other on the platform like a large daisy chain, and then only come up from the subway station with a “full” daisy chain. Instead of driving each suspect through traffic to the police station, they converted a bus into a mobile police station so that paperwork, fingerprints, background checks could done on site without a slow trip to the police station. Instead of just fining someone for fare-evading, and letting them go, they always ran a full background criminal check on each fare evader – and found something interesting: 1 out of 7 fare-evaders had outstanding arrest warrants; 1 out of 20 had illegal weapons. Suddenly, the cops on the street felt it was not “just a fare-evader”… now a daisy-chain of 20 fare-evaders was a really interesting surprise bonanza box, and the easiest way in the world to catch “real” bad guys.

The second cultural change was with the subway riders. Seems that the general public attitude had deteriorated to “why should I pay, if everyone else is evading”. Even people who would not normally break the rules, who would never consider themselves as criminals, were avoiding paying fares “because everyone else did it too”. However, the daisy-chain of handcuffed fare-evaders was a clearly visible deterrent, a reminder of what the rules were, and how society expected people to behave. Quickly, the number of people trying to fare-evade dropped. Which meant more people paid. It also created the perception of the subway being safer, so more people felt safe to choose to travel on subway. So even more people paid. All this meant they had more money to fix other problems, like old rolling stock, tracks, ticketing systems, etc.
Of course, the NYC subway is not perfect, then and now. However, a handful of small, carefully chosen physical changes, triggered a couple of critical cultural changes, which turned around a problem that had previously been almost given up for lost. I realized its really easy to trick yourself into thinking that a big reward requires a big effort project, and then with those mental blinkers on, only allow yourself to consider big ticket items. “How little things can make a big difference” is a really good subtitle for this book.

Chapter 5: Human group size:
Gladwell contends that human groups, and intra-group human loyalties, only scale up to about 150 people. His suggests its a function of the limits of the human brain to handle all the combinations of relationships between everyone in the group. For groups under 150 people, the inter-personal relationships, friendships, peer pressure, of everyone knowing each other tends to keep people focused and working together towards a common group goal. However, groups that grow over 150 quickly lose internal cohesion, internal focus, because they are just too big for everyone to really know everyone else. Once that cohesion breaks down, people instead start forming smaller subgroups, trusting their own subgroup, questioning motives of those other subgroups, and focusing on their own personal agendas. In a Western-business culture setting, it would be called internal-company-rivalry.

I once worked in a company as it grew from 42people to 180people, and experienced that change of internal cohesion myself. At the time, I just knew “things had changed”, but only later, looking back, I figured out it was to do with how many people were in the company, and not feeling like we were all working together anymore. Personally, I’d always thought the change from “we tight knit small band of brothers in a small company” to “I’m just a nameless cog in a faceless bureaucracy” started to happen somewhere around 100 people, but that was just a gut guess. However, Gladwell shows examples of hunter gatherer tribes, ranging from Australia to Greenland, all averaging just under 150 people per village. The Hutterites (a religion similar roots to the Amish) have a strict policy that once a community approaches 150 people, its splits into two equal separate communities. In business, the same principle is followed by Gore Associates (the manufacturer of GoreTex)… and they believe that contributes to why they have employee turnover 1/3 of the industry average, are profitable for 35 years in a row (and counting), and are constantly successfully innovating new products and markets… all without formal management structures. In groups under 150, he suggests that “personal loyalties and direct man-to-man contacts” keep everyone focused on doing the right thing for the organisation.

There’s quite a lot of the book given to describing Mavens, Connectors and Salesmen. I’m still thinking that part over, not convinced yet.

Would I recommend this book? Yes, especially these two chapters.

Firefox 3alpha8 by the (wall-clock) numbers

Mozilla released Firefox3a8 on Thursday, 20-sep-2007, at 08:30am PST.

This was a manual build run (not automated on trunk yet), and an alpha release (not a high-priority security release), so the numbers are quite different to the earlier Firefox2.0.0.7 release. Even as an apples-to-baseballs comparison, I thought the numbers were interesting and worth sharing. From “code freeze” to “available for public download” was 14.33 days wall-clock time. Of that time Build&Release took 2.25 days (55 hours including the respin).

00:01 06-sept: M8 code freeze, tree closed
18:48 11-sept: Dev verifies last fix landed, and gives “go” to build
20:46 11-sept: Build starts building
01:26 12-sept: blocker bug#395862 filed
08:40 12-sept: blocker patch landed
09:22 12-sept: Build restarts building
13:49 12-sept: linux & mac builds handed to QA
18:01 12-sept: signed-win32 build handed to QA
11:17 18-sept: QA signed off on all builds
00:01 19-sept: Build supposed to finish signing and publish builds externally
02:58 20-sept: files available externally for download
08:31 20-sept: mirror absorption completed and release announced

There were a few interesting point about this release
1) There was a 5.75 day delay between when the code freeze started, and when the tree was first deemed ready for builds to start.
2) After builds started, a last minute blocker bug caused those builds to be abandoned and new builds started. This respin cost Build 12 hours.
3) Between 13-17sept inclusive, both Build and QA switched to work on FF2.0.0.7 (a higher priority security firedrill release). This caused wall-clock delays.
4) After QA signoff, we delayed releasing Firefox3a8 from 18sept to 19sept, to avoid traffic load of releasing Firefox3a8 on the same day as Firefox2.0.0.7.
5) There was a 1 day delay between when QA signed off on the builds and when Build group ran the remaining manual steps (signing installer, pushing bits externally, etc). These remaining Build steps only took a handful of hours to complete. However, the person doing those remaining manual steps (ie me!), was sidetracked with other non-release work.