Fun and Games with Major Updates (followup)

There were lots of comments and questions raised from the last two blog posts [1] [2]. As some people were asking variations of the same questions, I thought it would be less confusing to summarize the questions and answers all here instead of replying in comment threads. Please let me know if I missed any of the comments, or misunderstood the questions, ok?

Q) We make minor updates to all previous dot releases, why not make major updates available to all previous dot releases also?

A) Every time we do a minor release, we make updates available for *all* previous dot releases in that product release train. For user efficiency, we make partial updates (smaller, faster downloads) available for the previous dot release. For our own sanity, we make full/complete updates (larger, slower downloads) available for all other older dot releases. To give a concrete example, when we released FF3.0.5, we made partial updates available to users on FF3.0.4, and we make full/complete updates available to users on FF3.0.3, FF3.0.2, FF3.0.1 and FF3.0.0. In the diagram, the partial updates are noted by dotted-lines, the full/complete updates are noted by dashed lines. (For larger image, click here). This works on the assumptions that:

  1. the release-drivers team only allows highly restricted set of changes into a security release that do not impact user migration, and
  2. that the physical bits users get for “full/complete update” are identical.

This means that QA can test partial update and then test some of the full/complete upgrade combinations, but not have to test *all* of them. This is a very significant time saving.

When we released the major update from FF2.0.0.20->FF3.0.5 in December2008, we took the same identical existing full/complete updates already produced as part of the FF3.0.5 release, and made those full/complete updates visible to FF2.0.0.20 users.

If we wanted to, we could certainly make those same full/complete updates available to migrate users from FF2.0.0.19->FF3.0.5, FF2.0.0.18->FF3.0.5, FF2.0.0.17->FF3.0.5…. However, major updates *do* contain changes that impact user migration, which makes QA correctly more concerned about thorough testing each possible migration path, on each o.s., in each locale, which implies a *lot* more manual work. To keep this combination-explosion practical, we only make major update offers available to one specific dot release at a time. By convention, we’ve always targeted the latest dot release, because:

  1. that is where most of the users are, and
  2. those are the users who’ve already shown interest in keeping up to date. This does also mean that someone on an older dot release has to upgrade first to latest, and *then* get to see the MU offer. For those laggard users, it seems like a suboptimal two-upgrades-in-a-row, which would be totally avoidable if the user upgrades as soon as we make each offer available, more importantly this is safer because the user is moving forward along tested migration paths.

Q) Can we somehow tell users who are prevented from getting major update (“no admin privs”, or “unsupported o.s.”), exactly what is preventing them from getting the updates, and maybe advise them what to do?

A) For users with “no admin privs”, the “check for updates” menu is grayed out, and the background idle checking is disabled. This means that we never hear from them, and therefore cant send any messages back. It might be possible to have something on the client side which notifies the users that they are running on locked-systems, but it might quickly frustrate the users, as they cant do anything about it! Any and all suggestions are really welcome here.
For users on “unsupported o.s.”, they do contact us, we just do not have any FF3.x updates available for them. We could, in theory, generate some fake major update which was only a message dialog, telling people to upgrade to a secure, supported o.s. However, historically, we’ve been cautious about doing that, because users on old computer might not have a choice, or the spare cash to do this, and they’d then get bothered by the recurring notices.
Hopefully, that explains how we got to where we are today. However, both of these areas are excellent problems that we’re still wrestling with. If you’ve any ideas/suggestions, please do let us know – we’re all ears!!

Q) Currently, whenever we offer a minor update, it cuts off users from seeing the major update. Shouldn’t it be the other way around – and the major upgrade be more important then the minor one?

A) No. As much as we love our shiny new release and would love people to major update to it, doing a major upgrade is more disruptive to users. Profiles migrate. Awesome bars get added. Icons move/change. All this needs some getting used to, and some people may / maynot like the changes. However, the minor changes only contains critical security fixes, few (if any) user visible changes and the sooner we get that to users, the safer they are.
Q) Could Mozilla always leave a major update option available for users on latest FF1.5.0.x and FF2.0.0.x?

A) Yes, we could, and I believe we should always leave a major update offer standing on the last dot release of a product. There are some limitations, but personally, I really like that idea. Its something that we were never great at in the past because there was just too many things in the air at a time. However, now that we’re stabilizing the infrastructure, we’re able to start paying better attention to things like this. Part of the reason for all these posts about major updates was because I noticed that for the recent FF2->FF3.0 major updates, we had long periods of time where there were *no* major updates available to users. And this was when we were supporting both release trains, were actively trying to move people from FF2->FF3.0, and were trying to figure out why so many people were not migrating!?!?!? Now that producing these major updates is streamlined, we can consider some options that were not possible before.

Answering this question properly is a little more complex, and involves some proposals in my next blog post, so I’ll defer the rest of the answer to there, ok?

Hope all that makes sense.
John.

4 thoughts on “Fun and Games with Major Updates (followup)

  1. Q) Could Mozilla always leave a major update option available for users on latest FF1.5.0.x and FF2.0.0.x?

    Your answer to that question is misleading. There is always a major update on the hook for the latest Firefox 1.5.0.x and 2.0.0.x. The question is really if we can make the major update go to the latest release, which we aren’t currently doing. The answer isn’t just about build work but also about QA work, see your first question/answer.

  2. Q) Could Mozilla always leave a major update option available for users on latest FF1.5.0.x and FF2.0.0.x?

    Your answer to that question is misleading. There is always a major update on the hook for the latest Firefox
    1.5.0.x and 2.0.0.x. The question is really if we can make the major update go to the latest release,
    which we aren’t currently doing.

    Sam, you raised three points:

    1) Actually, nope, the question and answer are accurate – I think you misunderstood. The question remains about making sure that a user on the latest dot-release of the older release line could always see a major update offer to the newer release line.

    2) The assertion that “There is always a major update on the hook for the latest Firefox 1.5.0.x and 2.0.0.x.” is factually incorrect. There has not always a major update on the hook for the latest Firefox 1.5.0.x and 2.0.0.x. We have not done that for FF2. While I don’t have complete records for FF1.5, I don’t believe our major updates situation was any better for FF1.5 either.

    Here is the history of FF2 releases, listing each security releases and major update offer that we did:

    17jun: FF3.0.0 released, no major update available
    25aug: major update available Firefox 2.0.0.16->3.0.1
    23sep: major update disabled by Firefox 2.0.0.17 release
    04dec: major update available Firefox 2.0.0.18->3.0.4
    16dec: major update disabled by Firefox 2.0.0.19 release
    17dec: theoretical EOL date for FF2
    08jan: major update available Firefox 2.0.0.20->3.0.5

    There are 26 weeks between the release of FF3.0 and the theoretical EOL of FF2. Major update offers were only available 6 of the 26 weeks. This means that a fully motivated FF2 user who eagerly wanted to major update to FF3.0 could only see a major update offer for 6 of the 26 weeks.

    The proposal here was to change how we do releases, so that users could always see a major update offer. For all 26 of the 26 weeks.

    True, it would be nice if that major update offer was to the newest dot-release on the newer release line, and I think we can do that, but I’m focusing first on making sure users can always see some major update.

    Obviously, for FF1.5.0.14, now that we are no longer producing dot releases, the last major update offer we made available is still valid and visible. But that continuously visible offer only happens after we already EOL’d the older FF1.5 release line, and only after the critical 26 week initial period is already over. The same will be true for FF2 once we know we’re never producing any new FF2 releases.

    3) “The answer isn’t just about build work but also about QA work, see your first question/answer.” Yes, I totally agree, this is multiple areas within the Mozilla project; RelEng and QA for sure, but additionally, this also impacts IT, release-drivers and lots of other areas.

    Hope that clarifies
    John.