Mozilla / Release Engineering

“Release Engineering as a Force Multiplier” keynote at RelEngCon 2013

joduinn16 Comments

The world’s first ever Release Engineering conference was held in San Francisco on 20may2013, as part of ICSE 2013.

It was a great honor for me to be invited to give the opening keynote. This was a rare opportunity to outline some of the industry-changing RelEng-at-scale work being done at Mozilla. It also allowed me to describe some very important non-technical tactics that we used to turnaround Mozilla’s situation from “company-threatening-and-human-burnout” to “company-enabling-and-human-sustainable“. All stuff that is applicable to other software companies. Presenting the first session at the first RelEng conference helped set the tone for the event, so being down-to-earth and practical felt important.

My presentation focused on how effective RelEng helped Mozilla compete against larger, better funded, companies. This is what I mean by “Release Engineering as a Force Multiplier”. To give some perspective of scale, I showed:

  • company logos scaled by headcount for each of Apple, Google, Microsoft and Mozilla. I noted that using revenue/profits instead of headcount would be just as disproportionately out of scale.

  • company logos scaled by browser market share for each of Apple, Google, Microsoft and Mozilla, based on publicly available market share data

(The full set of slides are available here, or by clicking on either of the two thumbnails above. If you want the original 25MB keynote file, let me know.)

Anyone who has ever talked with me about RelEng knows I feel very strongly that:

  • Release Engineering is important to the success of every software project. Writing a popular v1.0 product is just the first step. If you want to retain your initial users by shipping v1.0.1 fixes, or grow your user base by shipping new v2.0 features to your existing users, you need a reproducible pipeline for accurately delivering software. Smaller projects may not have someone with formal RelEng title, but there’s always someone doing RelEng work. Otherwise, your product, and your organization, will not survive.
  • Building an effective RelEng pipeline requires a different mindset to writing a great software product. Both are non-trivial, and understanding this difference in mindset enables you to hire wisely.
  • The importance of Release Engineering for the sustainability of a software company is only beginning to be recognized. Release Engineering is not taught as a discipline in any CompSci course that I know of. This has the unfortunate side effect that most Release Engineers only learn on-the-job, usually as a side-effect of helping out during an organizational emergency. This limits the effectiveness of Release Engineering to what can be learned on the job, and because information isn’t widely shared, its hard to learn from other people’s mistakes or successes. By contrast, when I was studying CompSci for my undergrad and postgrad degrees, there were all sorts of course modules, books and published papers detailing which sorting algorithm was best suited for which types of data, which graphics algorithms were best suited for which types of images, etc… but nothing about Release Engineering. Nothing about how to build a pipeline to deliver that software to users… even though every software company lives-or-dies by the efficiency of their delivery-pipeline.

This conference was a great start to helping raise the understanding of the industry on these three points, and many more.

Big hat-tip to the organizers (Bram, Christian, Foutse, Kim) – they did an awesome job putting together a unique and special event. Other industry speakers included Google, LinkedIn, MicrosoftResearch, Netflix, PuppetLabs as well as a wide range of academic speakers – see full program details here and here. In the past, I’ve attended plenty of industry conferences which have a sales-touch to them (“our technology is great, you should buy our stuff” or “our technology is great, you should come work here”), or academic conferences which have academic-accuracy-but-can-lack-urgent-practicality. By contrast, this conference was very different. All the speakers spoke candidly, humbly and objectively about the technology and tactical successes that worked for them at their companies, and equally candidly about their failures as “teaching moments” for everyone to learn from. The level of raw honestly by all these speakers, from all these different companies and academia was super-honest and very collaborative. Additionally, the sheer volume, and the quality, of attendees blew everyone away… the discussions during breaks were just electric. Very very refreshing. I dont know if this magic was because of the carefully chosen mix of industry-vs-academia speakers, or if this was because the organizers were also a mix of industry and academia, but regardless – the end result was simply fantastic.

There’s already plans underway to have this RelEngConference again next year… if you build software delivery pipelines for your company, or if you work in a software company that has software delivery needs, I recommend you follow @relengcon and plan on attending next year. You’ll be very glad you did!

16 thoughts on ““Release Engineering as a Force Multiplier” keynote at RelEngCon 2013”

    • hi Roc;

      It looks to me like you scaled both the X and Y dimensions of the logo
      by the headcount ratio? It would make more sense to scale them by the
      square root of the headcount ratio so the areas of the logos are
      proportional to the headcount.

      I actually scaled the area of the logos in the same ratio as the headcount ratio. (apple and mozilla logos are square-ish, but google and microsoft logos are very rectangular). I’m traveling right now, but can get you the math notes when I return home next week.

      Also, can you respond to
      http://soberbuildengineer.com/blog/2013/06/the-dark-ages-that-never-were/ in
      some way?
      Thanks for the pointer – I didnt see that. Will do.

      tc
      John.

  2. Seems like scaling by total corporate headcount or profits is somewhat misleading considering how many more things Google/Microsoft/Apple do than Mozilla. It would be more useful to scale by the size of the browser team at each company if that data were public (which it unfortunately isn’t). In the absence of that, these images don’t really show anything that one can draw conclusions from.

  3. The company logos by headcount is kind of deceiving. No doubt Apple, Microsoft, and Google have more employees working on their respective browsers than Mozilla, but I doubt it’s as different as you portray. It’d be better if you used employee numbers based on those that actually work on the browser, or at least are within the same division.

    For example, Google has an Android, Chrome, & Apps division, which maps fairly well to Mozilla. But comparing Google’s 53,000 employees (16,000 of which are at Motorola) to Mozilla’s 900 isn’t even close to a fair comparison. Google has a lot of other, non-browser, non-OS things they work on. (Like, a search engine, for example.)

    The same is true for Microsoft and Apple, even if you include their entire OS groups because Mozilla is now working on Firefox OS.

    In any case, your graphic feels very disingenuous when comparing headcount and with a bit more research, you could make it a lot more accurate.

    • hi ss, Peter;

      You both raised similar points, so I’m responding to both of you here together.

      These slides show that Mozilla is much smaller then Apple/Google/Microsoft, and yet, as I described in my presentation we are able to do large scale production to compete in the marketplace on a par with Apple/Google/Microsoft.

      1) Mozilla is a couple of orders of magnitude smaller then Apple/Google/Microsoft. As I said in the presentation, regardless of whether the logos were scaled by total headcount, or by a completely different measure like annual revenue, the sheer difference in scale is worth noting. Its one thing to vaguely say “Mozilla is smaller then Apple/Google/Microsoft” – the point was to show just how much smaller, and for that I felt it was important to use publicly verifiable data.

      2) Yes, the slides do show ratios of size-of-logos-by-area based the headcount numbers for all employees in each of the 4 organizations. I agree that Apple, Google and Microsoft do have people working on non-browser products – and I said so in the presentation. I remind you that Mozilla also has people working on non-browser products (gaia-for-FirefoxOS, Sync servers, persona, legal, business-development, WebFWD, metrics, … to mention just a few of the bigger projects!), so this felt like a fair apples-to-apples comparison.

      3) Using publicly verifiable data felt important, and as Peter noted, headcount-by-product-within-company data is not published. Further, even if that data was published, attempting to use headcount-by-product-within-company would be complicated by how to count shared-service-groups-within-company, like Release Engineering, which spend time across browser and non-browser projects. Hence, I felt this approach was impractical.

      Given that total headcount, or total revenue, are equally suitable for the apples-to-apples purpose of this slide and are using public available data, I feel the slide portrays the sheer difference in scale between these four organizations quite clearly and accurately.

      Hope that clarifies – and thanks for the comments!

      John.

  6. @SS

    John addressed the “total employees” point in his actual talk, which I saw live at RelEng conference. It counts everyone at the company, and while Apple Microsoft and Google do other stuff, it still matters.

    John gave a repeat of the same talk the following week at Google, where he was recorded live. I invite you to watch it yourself at http://www.youtube.com/watch?v=7j0NDGJVROI.

  9. Hi,

    I’m student from germany and I have to write a scientific paper at my university about release engineering. My topic in this context is “A behind-the-scene look: How do the big companies do it”. Now I’m looking for informations and found your talk: Release Engineering as a “force-multiplier”. I would like to reflect your ideas in my paper. Therefore it would be greate if you could send me some more informations like a paper or some slides.

    Thanks for your help.