There’s been a lot of discussion in the Enterprise Working Group emails, and on the monthly EWG calls, about how Mozilla’s change to rapid release cadence is impacting enterprises. While brainstorming in a recent EWG call, I asked:

“how do enterprises handle rapid releases of other software products?”

After a few minutes discussion, we agreed to continue the discussion afterards. I’ve since emailed this same question to some lists, but am now cross-posting to my blog, in the hope that even more will see this question.

Does anyone have examples of other software products that do “rapid releases” and are being successfully handled by enterprises? If so, can you share some details? (If you prefer to email me privately, that is totally cool, please use my joduinn [at] mozilla [dot] com address). I ask because maybe we don’t need to reinvent the wheel here. If there is a tried-and-tested approach that already works well for other applications, that same approach might also makes work for Mozilla’s Firefox.

Some obvious examples of “rapid release software” in enterprises are OS-patch-updates updates, but what about Microsoft Office? Google Chrome? Flash? Java? Anything else? (In my mind updating anti-virus signature files is different scenario, but I could be misreading this scenario).

Thanks.
John.

(I raised this in the Enterprise Working Group (EWG) call this week, and it resonated strongly with some people. Therefore, I’m posting this out more widely to hopefully get more feedback.)

After all the recent discussions about “what enterprise users wanted”, I found myself wondering if we were all even attempting to solve the same problem, so I stepped back, and re-read *lots* of posts from different enterprises over the last few months.

I now believe Mozilla, and the enterprises in the Enterprise Working Group, are working to solve three overlapping but orthogonal problems.

1) Cost of verifying that a new version of Firefox is safe to deploy.
Some enterprises verify with a quick running of an ACID test. Some SaaS vendors verify by doing wider testing, and deploying bugfixes to their products. One complication for SaaS vendors is that end users may be running on newer versions of Firefox anyway, on non-enterprise machines. This can cause problems that make both the SaaS vendor, and Mozilla, look bad. We havent spent much time on this so far.

(I still wonder if we could design a testsuite compatibility test suites, in the same mindset as HTML5, JavaCompatibilityKit, etc that might help speed up this verification step?)

2) Cost of deploying a new version of Firefox to all supported users
Once an enterprise has verified a specific version of Firefox, how much effort does it take to deploy that new version onto all their machines/users. This discussion typically quickly focuses on MSI and similar technologies for doing widespread deployments, although there are some other options like an inhouse AUS or equiv. Regardless of the technology used, the idea here is to have a centralized way to move forward all users to a newer version of Firefox, without having to walk/drive/fly a human to every computer in order to manually do a new install. Sometimes this also includes discussions about silent updates.

3) Frequency of doing this all over again
The frequency of the Firefox release cadence directly impacts how often enterprises have to go back to do (1) and (2) all over again.

The verify+deploy work is typically so painful that most enterprises only do this for “new feature” releases, and not for “security only dot-releases”. For most enterprises, it seems that Mozilla’s cadence of “new feature” releases every 12-18-24 months was infrequent enough that the verify+deploy work was tolerable. However, Mozilla’s more frequent feature releases means more frequent cost of verification+deploying, which can become a significant business problem.

The ESR proposal is attempting to address this increased recurring cost and this is where most of the discussions have been taking place so far.

(It’s worth noting that everyone involved from Mozilla and different enterprises understands and agrees that Mozilla’s faster cadence of “new feature” releases is important for Mozilla to remain relevant in the browser marketplace.)

Just my thoughts, but I’d be curious to hear what others think.

John.

As some of you reading this may already know, there’s an proposal under discussion for how Mozilla could support releases for longer durations as requested by some enterprises.

I’d like to modify the latest Extended Support Release (ESR) proposal as follows:
1) Mozilla would not generate overlapping Extended Support Release (ESR) builds
2) Change the timing of when enterprises start to deploy new versions of Firefox.

The details of this are subtle, so please bear with me, while I try to explain with an brief example:

1) Mozilla anoints a specific Firefox release to be supported for a total of 42 weeks.
For the purposes of discussion, lets say this is Firefox 8.0. This means Firefox 8.0 users would be guaranteed to receive seven scheduled security-only dot-releases (plus, of course, any unplanned security chemspills that came up in that 42 weeks timeframe). Before the end of the 42 weeks, Mozilla would anoint another release to be supported for 42 weeks. To continue this example, after Firefox 8.0, the next release to be anointed would be Firefox 15.

Schedule-wise, this means:
** 8.0.1 would sim-ship with 9.0.
** 8.0.2 would sim-ship with 10.0.
…
** 8.0.7 would sim-ship with 15.0
** 15.0.1 would sim-ship with 16.0
…

2) Enterprises would start to verify/certify with the 8.0.0 release.
However, enterprises would *not* deploy 8.0.0. Specifically, enterprises would only start deployments of 8.0.1 at the time that 9.0 is released. (This is important for mechanical details about how updates are served – see more below.). (To be precise, enterprises deploy the latest 8.0.x available at the time 9.0 is released; if there are no chemspills, this would be 8.0.1, but if there are chemspills, it is always the latest latest 8.0.x available at the time of the 9.0 release).

3) When doing releases, RelEng makes a small change to how we publish updates between releases:

* 8.0.1 would sim-ship with 9.0.
** Mozilla would NOT enable updates from 8.0.0 -> 8.0.1
** Mozilla would enable updates from 8.0.0 -> 9.0.0
* 8.0.2 would sim-ship with 10.0.
** Mozilla would enable updates from 8.0.1 -> 8.0.2
** Mozilla would enable updates from 9.0.0 -> 10.0.0
…
* 8.0.7 would sim-ship with 15.0
** Mozilla would enable updates from 8.0.6 -> 8.0.7
** Mozilla would enable updates from 14.0.0 -> 15.0.0
* 15.0.1 would sim-ship with 16.0
** Mozilla would NOT enable updates from 15.0.0 -> 15.0.1
** Mozilla would enable updates from 8.0.7 -> 15.0.1
** Mozilla would enable updates from 15.0.0 -> 16.0.0
* 16.0.1 would sim-ship with 17.0
** Mozilla would enable updates from 15.0.1 -> 15.0.2
** Mozilla would enable updates from 16.0.0 -> 17.0.0
…

Thats it.

There are a few reasons why I recommend these modifications to the proposal:

1) Minimal changes to RelEng release automation or our update infrastructure. This means mechanically, we can put this new proposal into action more easily.

2) No need for any metrics infrastructure changes – all current infrastructure should just work as-is.

3) No need for Mozilla to generate overlapping concurrent ESR releases. This is significant because:
3a) the original proposal would have Mozilla sim-ship Firefox13.0, Firefox8.0.5esr and Firefox13.0esr at the same time as we also migrate aurora->beta and central->aurora. This is a significant increase in the mechanical work for RelEng in a *very* tight timeframe.
3b) this reduces the number of landings developers have to do for security fixes for 12-weeks-in-every-42 weeks. This also reduces the number of release builds to be generated if we have a chemspill in that 12-weeks-in-every-42-weeks window.

I believe these modifications still meet all the same objectives of the original proposal, yet are mechanically easier to implement. Therefore, I believe Mozilla could put this modified ESR proposal into action more easily then the existing ESR proposal.

Let me know if I missed anything, or if you have any concerns.

Thanks
John.