There’s been a lot of discussion in the Enterprise Working Group emails, and on the monthly EWG calls, about how Mozilla’s change to rapid release cadence is impacting enterprises. While brainstorming in a recent EWG call, I asked:
“how do enterprises handle rapid releases of other software products?”
After a few minutes discussion, we agreed to continue the discussion afterards. I’ve since emailed this same question to some lists, but am now cross-posting to my blog, in the hope that even more will see this question.
Does anyone have examples of other software products that do “rapid releases” and are being successfully handled by enterprises? If so, can you share some details? (If you prefer to email me privately, that is totally cool, please use my joduinn [at] mozilla [dot] com address). I ask because maybe we don’t need to reinvent the wheel here. If there is a tried-and-tested approach that already works well for other applications, that same approach might also makes work for Mozilla’s Firefox.
Some obvious examples of “rapid release software” in enterprises are OS-patch-updates updates, but what about Microsoft Office? Google Chrome? Flash? Java? Anything else? (In my mind updating anti-virus signature files is different scenario, but I could be misreading this scenario).
Thanks.
John.
Ubuntu releases roughly every six months with every fourth release being a long term support release. Obviously these timelines are different to Firefox’s but maybe a similar approach could be made to work.
I think you need to look at the topic in multiple dimensions. First off, you need to decide whether you are viewing it from the publisher of the release or the consumer of the release.
From the publisher of the release I have seen three very common models applied:
(1) Release when you have achieved the level of functionality you set out to achieve. Note that can be a single (minor) feature enhancement. So in theory if this were repeated often you can theoretically have releases daily.
(2) Release when a critical bug (or bugs) have been fixed. This may or may not be combined with feature enhancements coming from #1 above.
(3) Time bound releases. This mode says “we release a rev on the Nth day of the month each month” – or quarter or week.
All of the above are the most common practices I have seen. However, this is not the full picture. Clearly the essential thing release managers should be sure of is that the release be run with long regression analysis to avoid introducing new bugs. This assumes a heavy emphasis and commitment to testing! Sadly, this is not always the case. For me personally, I don’t trust my code – I trust my tests. And this is why we also seek to do code coverage to help ensure that we are really testing the thing.
Turning to the recipient of new releases. I think basically the answer is it depends. Some considerations from that angle:
(1) Does the new release address a critical bug in their system?
(2) Does the new release address a bug in our system?
(3) Am I on a supported release of their software?
(4) How long has it been since I’ve rev’ed my release to theirs? Am I falling behind?
Once again, though, the key factor comes down to testing. I need to make sure that before I rev my release I pass all (and I mean all) known regression tests on my end. If something fails in regression testing then I will not uptake the new changes.
Maven’s approach makes these considerations (on the publisher and consumer end) fairly straightforward (even though I am not a great fan on HOW they implemented the tool).
Well as far as deployment for Flash and Oracle Java goes, I think we’re all agreed that the status quo is thoroughly inadequate and that if we take anything from those particularly examples it’s how not to handle the situation.
I still can’t get past the irony of Mozilla only asking this and similar questions well AFTER the fact. LOL. What a stupid lot we humans are.
I think you’re answer will be: they don’t.
My product at Sophos (the Sophos Email Appliance) does rapid releases, but it’s a bit (a lot) easier since we just have a single appliance at the gateway that we update automatically. It’s a tough problem, to be sure.
For Chrome we give IT admins complete control over update frequency and rollout via group policy controls for the Google Updater. At the same time, our silent update process that makes updating consumer machines easy also makes it easy for enterprises to go ahead and deploy updates when they do wish to. We also explicitly advise them to keep Chrome updated on some sort of regular schedule even if it doesn’t match the speed of our public releases — e.g. quarterly.
Enterprises seem to have responded well to this so far.
Poorly, would be the short answer. And that’s not just because of the enterprises themselves, but also the vendors of software they use. For example, the company I work for provides a Java applet-based UI for our product – we’re trying to get away from that in favour of pure-web, but that’s beside the point…
So, over the years, we’ve found a variety of issues relating to certain combinations of browsers, and Java plugin versions. For example, when IE8 came out, we found it crashed on several screens with issues relating to the Java plugin version.
Short version then, is that we maintain a very specific techstack that we support for each version of our product – partly because we don’t want to deal with all the possible combinations of versions, and partly because our clients want specific statements on what we support and don’t support. We don’t stop them from using unsupported combinations, but it’s made clear that we’re unlikely to fix bugs that don’t occur on the supported techstack, and they’ll have to pay us if they want new versions certified.
And to put it clearly, that means that “rapid release” is a no-go. For reasons relating to the above, our customers simply don’t do automatic updates of anything – browsers, Java, Flash – unless forced to by either major known security holes, or the end-of-life of that version. And so they tend to be running with quite old versions of Java (1.4, 1.5) and Flash plugins, because they want to be supported by their vendors.
An observation on mindset – these organisations are unbelievably risk-averse. Part of the problem is vendor certification, but it’s also that they regard any kind of change as a potential disaster. To you, it’s a minor bug fix release – to them it’s a possible call-center outage costing millions of dollars and bad publicity.
Just to give a view on timescales
I work for a global IT company. Many of our users are still on Windows XP, for all the reasons explained above. We only provide support for Windows 7 in some geographies, meaning that XP is still the default OS in some places.