A Release Engineer’s view on rapid releases of books

No Comments

As a release engineer, I’ve designed and built infrastructure for companies that used the waterfall-model and for companies that used a rapid release model. I’m now writing a book, so recently I have been looking at a very different industry through this same RelEng lens.

In days of old, here’s a simplistic description of what typically happened. Book authors would write their great masterpiece on their computer, in MSWord/Scrivener/emacs/typewriter/etc. Months (or years!) later, when the entire book was written, the author would deliver the “complete” draft masterpiece to the book company. Editors at the book company would then start reading from page1 and wade through the entire draft, making changes, fixing typos, looking for plot holes, etc. Sometimes they could make these corrections themselves, sometimes the changes required sending the manuscript back to the author to rewrite portions. Later, external reviewers would also provide feedback, causing even more changes. If all this took a long time, sometimes the author had to update the book with new developments to make sure the book would be up-to-date when it finally shipped. Tracking all these changes was detailed, time-pressured and chaotic work. Eventually, a book would finally go to press. You could reduce risk as well as simplify some printing and shipping logistics by having bookshops commit to pre-buy bulk quantities – but this required hiring a staff of sales reps selling the unwritten books in advance to bookstores before the books were fully written. Even so, you could still over/under estimate the future demand. And lets not forget that once people start reading the book, and deciding if they like it, you’ll get word-of-mouth, book reviews and bestseller-lists changing the demand unpredictably.

If people don’t like the book, you might have lots of copies of an unsellable book on your hands, which represents wasted paper and sunk costs. The book company is out-of-pocket for all the salaries and expenses from the outset, as well as the unwanted printed books and would hope to recover those expenses later from other more successful books. Similar to the Venture Capital business model, the profits from the successful books help recoup the losses from the other unprofitable books.

If people do like the book, and you have a runaway success, you can recoup the losses of the other books so long as you avoid being out-of-stock, which could cause people to lose interest in the book because of back order delays. Also, any errors missed in copy-editing and proofreading could potentially lead to legal exposure and/or forced destruction of all copies of printed books, causing further back order delays and unexpected costs. Two great examples are The Sinner’s Bible and the Penguin cook book.

This feels like the classic software development “waterfall” model.

By contrast, one advantage with the rapid release model is that you get quick feedback on the portions you have shipped so far, allowing you to revisit and quickly adjust/fix as needed… even while you are still working on completing and shipping the remaining portions. By the time you ship the last portion of the project, you’ve already adjusted and fixed a bunch of surprise gotchas found in the earlier chunks. By the time you ship your last portion, your overall project is much healthier and more usable. After all, for the already shipped portions, any problems discovered in real-world-use have already been fixed up. By comparison, a project where you write the same code for 18-24 months and then publish it all at the same time will have you dealing with all those adjustments/fixes for all the different chunks *at the same time*. Delivering smaller chunks helps keep you honest about whether you are still on schedule, or let you quickly see whether you are starting to slip the schedule.

The catch is that this rapid release requires sophisticated automation to make the act of shipping each chunk as quick, speedy and reliable as possible. It also requires dividing a project into chunks that can be shipped separately – which is not as easy to do as you might hope. Doing this well requires planning out the project carefully in small shippable chunks, so you can ship when each chunk is written, tested and ready for public consumption. APIs and contractual interfaces need to be understood and formalized. Dependencies between each chunk needs to figured out. Work estimates for writing and testing each module need to be guessed. A calendar schedule is put together. The list goes on and on… Aside: You probably should do this same level of planning also for waterfall model releases, but its easy to miss hidden dependencies or impact of schedule slips until the release date when everything was supposed to come together on the day.

So far, nothing new here to any release engineer reading this.

One of the (many) reasons I went with O’Reilly as the publisher for this book was their decision to invest in their infrastructure and process. O’Reilly borrowed a page from Release Engineers and invested in automation to switch their business from a waterfall model to a rapid release model. This change helps keep schedules realistic, as schedule slips can be spotted early. It helps get early feedback which helps ship a better final product, so the ratio of successful book vs unprofitable books should improve. It helps judge demand, which helps the final printing production planning to reduce cost wasting with less successful books, and to improve profits and timeliness for successful books. This capability is a real competitive advantage in a very competitive business market. This is an industry game changer.

When you know you want “rapid release for books”, you discover a lot of tools were already there, with a different name and slightly-different-use-cases. Recent technologies advances help make the rest possible. The overall project (“table of contents”). The breaking up of the overall project into chunks (“book chapters”). Delivering usable portions as they are ready (print on demand, electronic books + readers, online updates). Users (“readers”) who get the electronic versions will get update notices when newer versions become available. At O’Reilly, the process now looks like this:

Book authors and editors agree on a table of contents and an approximate ship date (write a project plan with scope) before signing contracts.

Book authors “write their text” (commit their changes) into a shared hosted repository, as they are writing throughout the writing creative process, not just at the end. This means that O’Reilly editors can see the state of the project as changes happen, not just when the author has finished the entire draft manuscript. It also means that O’Reilly reduces risk of catastrophic failure if an author’s laptop crashes, or is stolen without a backup.

A hosted automation system reads from the repository, validates the contents and then converts that source material into every electronic version of the book that will be shipped, including what is sent to the print-on-demand systems for generating the ink-on-paper versions. This is similar to how one source code revision is used to generate binaries for different deliverables – OSX, Windows, linux, iOS, Android,…

O’Reilly has all the automation and infrastructure you would expect from a professional-grade Release Engineering team. Access controls, hosted repos, hosted automation, status dashboards, tools to help you debug error messages, teams to help answer any support questions as well as maintain and improve the tools. Even a button that says “Build!”!!
The only difference is that the product shipped from the automation is a binary that you view in an e-reader (or send to a print-on-demand printing press), instead of a binary that you invoke to run as an application on your phone/desktop/server. With this mindset, and all this automation, it is no surprise that O’Reilly also does rapid releases of books, for the same reasons software companies do rapid releases. Very cool to see.

I’ve been thinking about this a lot recently because I’m now putting together my first “early release” of my book-in-progress. More info on the early release in the coming days when it is available. As a release engineer and first time author, I’ve found the entire process both totally natural and self-evident and at the same time a little odd and scary. I’d be very curious to hear what people think… both of the content of the actual book-in-progress, and also of this “rapid release of books” approach.

Meanwhile, it’s time to refill my coffee mug and get back to typing.


The USENIX Release Engineering Summit, Nov 2015

No Comments

The USENIX Release Engineering Summit 2015 (“URES15”) is quickly approaching – this time it will be held in November in Washington DC along with LISA2015. To register to attend URES15, click here or on the logo.

Given the two (!) great URES conferences were last year, I expect URES15 to be fun and informative and very-down-to-earth practical all at the same time. One of the great things about these RelEng conferences is that you get to hear what did, and did not, work for others working the same front lines – all in a factual constructive way. Sharing important ideas help raise the bar for everyone, so every one of these feel priceless. It is also a great way to meet many other seasoned people in this niche part of the computer business,

If you are a Release Engineer, Site Reliability Engineer, Production Operations, deal with DevOps culture or are someone who keeps wanting to find a better way to reliably ship better quality code, you should attend! You’ll be glad you did!

Note: If you have a project that you worked on which others would find informative, you can submit a proposal here. The deadline for proposals is Friday 04sep2015.

See you there!

“we are all remote” at Cultivate NYC

No Comments

It’s official!!

I’ll be speaking about remoties at the O’Reilly Cultivate conference in NYC!

Cultivate logoCultivate is being held on 28-29 Sept 2015, in the Javits conference center, in New York City. This is intentionally the same week, and same location, as the O’Reilly Strata+Hadoop World conference, so if you lead others in your organization, and are coming to Strata anyways, you should come a couple of days early to focus on cultivate-ing (!) your leadership skills. For more background on O’Reilly’s series of Cultivate conferences, check out this great post by Mike Loukides. I attended the Cultivate Portland conference last month, when it was co-located with OSCON, and found it insightful edge-of-my-seat stuff. I expect Cultivate NYC to be just as exciting.

Meanwhile, of course, I’m still writing like crazy on my book (and writing code when no-one is looking!), so have to run. As always, if you work remotely, or are part of a distributed team, I’d love to hear what does/doesn’t work for you and any wishes you have for topics to include in the book – just let me know.

Hope to see you in NYC next month.


“An Illustrated Book of Bad Arguments” by Ali Almossawi

No Comments

Most engineers I know are good at writing software. Some engineers I know are good at writing software for extraordinarily complex projects. Some have a brain that naturally, instinctively, just worked this way while some took all sorts of courses on algorithms and structured programming in university. Regardless, when you are coding by yourself, these important coding and problem solving skills are probably good enough. However, when you transition from coding-by-yourself to coding-in-a-team, there is the need for an additional skill.

Now you need the skills to explain to other engineers why your code, your approach, is best. Maybe even why your approach is better then an alternate approach being proposed by someone else. It is a given that you can write good code – that is how you got on the team in the first place. The question is: can you explain why your code is better then any other alternatives available?

This skill might sound deceptively simple, but in reality it has a few super complex twists to it. Someone once explained it to me as follows: smart people, given the same start point, the same goal, and the same restrictions/limitations, would come up with similar-enough solutions. Maybe not identical, because all humans are slightly different, but at least similar in approach – and would solve the same problem. Given that, if I was to find myself in a situation where very different solutions are being discussed, maybe we have different understanding of what the starting point is? Or what the goal is? Or what restrictions/assumptions/limitations to work within? Double checking all those can quickly uncover missing (or invalid) problem statements, restrictions/assumptions/limitations or goals. If it turns out that we’re trying to solve different problems, then of course the proposed solutions would be very different!

If your skills in logic/reasoning/conflict are weak, and if you skip the verification step, then it is easy to get frustrated by your inability to “convince others” of the greatness of your code or design. Maybe you cannot explain it correctly. Maybe the others don’t want to listen to you – because they feel equally about their proposals. Maybe both. If you are passionate about your code, but cannot have others understand its greatness, the situation can easily turn to frustration (“I give up – why does no-one understand how great my code is?”) or even anger (“This code is too important to give up on – what do they know anyway – they’re just idiots”). This shift from working together to find the best code/design (something which can be objectively measured or stress-tested, after everyone agrees on what they all think is “best”.) to destructively debating the other humans (using hard-to-measure qualities) is dangerous. Even if the “right” code is actually chosen in the end, the manner in which the discussion was carried out can be toxic to the group.

Like many other engineers, I never had any formal training in logic/reasoning/conflict at any of my universities, so I had to learn these skills on-the-job, or at workshops I found over the years. Sometimes it went well. Sometimes it did not. I’m still working on this. Being able to talk through differences of opinions, in a constructive way, is a crucial skill set to develop as you work with others during your engineering career. And essential for *everyone* in a team to have, in order for the *team* to work together effectively.

I stumbled across this book in 2014, loved it, and was then delighted and surprised to discover that it was written by someone I knew from when I worked at Mozilla!! Ali’s book is focused on describing the different types of arguments used in logic debates, with very easy-to-understand examples. This will help you keep your logic and reasoning honest when you are a discussion. It will also make it easy to notice when someone else switches to using “Ad Hominem” or “No True Scotsman” or “Genetic Fallacy” or “Straw Man” or any other of these types of arguments on you.

I think many many people, including myself, find the idea of a book about arguments to be daunting, almost off-putting, so I really like how the simple one-page description and the full page cartoon images make the whole topic area more approachable, appealing and in turn, very easy to understand in a non-threatening way. At 56 pages, it is a super short book, and almost 50% of those pages are cartoons, so you could expect to be done in a few minutes. However, each page found me sitting rethinking different conflicts over the years, so this took a lot longer to read then I expected. And I re-read it often. In addition to the printed version of the book, there is also a fun https://bookofbadarguments.com/.

Try it, I think you will like it.

ps: This is one of two great books that I really like in this area – both of which I think should be required reading for engineers. Both of which I just discovered I had not yet blogged about, hence this post. Watch for another post coming soon.

The “we are all remoties” book!?!


I’ve been working in distributed teams, as well as talking, presenting, coaching and blogging about “remoties”‚ in one form or another for 8?9? years now. So, I’m excited to announce that I recently signed a contract with O’Reilly to write a book about how to successfully work in, and manage in, a geo-distributed world. Yes, I’m writing a “we are all remoties” book. If you’ve been in one of my ever-evolving “we are all remoties” sessions, you have an idea of what will be included.

If you’ve ever talked with me about the pros (and cons!) of working as a remote employee or of working in a distributed team, you already know how passionate I am about this topic. I care deeply about people being able to work well together, and having meaningful careers, while being physically or somehow otherwise remote from each other. Done incorrectly, this situation can be frustrating and risky to your career, as well as risky to employers. Done correctly, however, this could be a global change for good, raising the financial, technical and economic standards across all sorts of far flung places around the globe. Heady game-changing stuff indeed.

There are many “advocacy books” out there, explaining why working remote is a good / reasonable thing to do – typically written from the perspective of the solo person who is already remote. There are also many different tools becoming available to help people working in distributed teams – exciting to see. However, I found very few books, or blogposts, talking about the practical mechanics of *how* to use a combination of these tools and some human habits to allow humans to work together effectively in distributed teams, especially at any scale or over a sustained amount of time. Hence, my presentations, and now, this upcoming book.


  • if you are physically geo-distributed from the people you work with, I’d like to hear what does or doesn’t work for you. If you know someone who is in this situation, please share this post with them.
  • If you have experience working in distributed teams, is there something that you wish was already explained in a book? Something that you had to learn the hard way, but which you wish was clearly signposted to make it easier for others following to start working in distributed teams? Do you have any ideas that did / didn’t work for you?
  • If you have published something on the internet about remoties, please be tolerant of any questions I might ask. If you saw any of my “we are all remoties” presentations, is there anything that you would like to see covered in more/less detail? Anything that you wish was written up in a book to help make the “remote” path easier for those following behind?

…you can reach me on twitter (“@joduinn”) or on email (john at my-domain-name – and be sure to include “remoties” in the subject, to get past spam filters.)

Now, time to brew some coffee and get back to typing.

(updated 31jul2015 to add twitter + email address.)

“Hot Seat: The CEO Guidebook” by Dan Shapiro

No Comments

This book just came out and I loved it. If you are starting a company, or thinking of it, you need to read this book. Period.

Dan covered a whole range of topics very succinctly, and in easy-to-follow language. When and how to raise funds. What all those terms mean. Who should (and should not!) be on your board, and why. How to allocate shares and ownership between co-founders. Where to incorporate your company (Dan has strong opinions on this!). How to create (and then also maintain) company culture. A great section on decision making. A section on “Hiring” in the context of the Manhattan project vs the moon shot Apollo project that I think every engineering hiring manager should read before building a team. Several true stories about startups where co-founders mismatches caused company threatening problems (trivia: 6 of 10 startups lose a co-founder in early days). And some good (and bad!) stories of how important trust was.

Some great quotes that resonated with me:

“You have limited resources of time and money. When they run out, you go bankrupt. The important thing is not cost/benefit: it’s opportunity cost.”

(in the context of how much travel was needed for all the in-person meetings with investors when raising funding) “…Alaska Airlines gave me MVP status for my efforts. In January.”

“Entrepreneurship is the pursuit of opportunity without regard to the resources currently controlled”. Prof Stevenson, Harvard.

In a variation of the “fail fast” mantra in developer circles, Dan notes that “…while it might seem like cold comfort now, the sooner you fail, the sooner you can try again.” Oh, and he’s not just saying it – that was the ending of a chapter where he detailed the failure of one of his startups.

His tolerance for large volumes of coffee and pointer to suggested reading “Coffee, CYP1A2 Genotype, and Risk of Myocardial Infarction” was a great and unexpected tangent for me personally. (More info here Journal of American Medical Association)

“Startups don’t out think their competitors; they out-execute them.”

“If leadership is the forest, then management is the trees. Day to day, it’s what consumes your time, and its imperative that you get it right.”

It takes skill and seasoned-experience-in-the-field to have one person cover all these different topics. Even more skill to do so clearly, and concisely. Putting them all together in a way that makes sense was great. Just great. If you are starting a company, or thinking of it, you need to read this book. Period.

Aside: Having this on my kindle app, on my trusty nexus5 phone was quite a good reading experience. The book was written in short, digestible chapters, which I could quickly complete standing a store line, or in the back of a taxi between meetings. It also encouraged me to think more about the chapter I just finished in the time before I got to stop and read some more. A nice way to digest the many lessons in here. I’m still experimenting with what books I find do work best on phone+kindle vs ink-on-paper, but at least for this book, reading on kindle worked for me.

(Disclaimer: I bought this book because I’m starting my own company, and that is the basis of the above review. As this book is published by O’Reilly Press, it feels important to disclose that I am also currently doing some work with O’Reilly… which did not influence anything I wrote here.)

“We are ALL Remoties” (Jun2015 edition)


Since my last post on “remoties”, I’ve done several more presentations, some more consulting work for private companies, and even started writing this down more explicitly (exciting news coming here soon!). While I am always refining these slides, this latest version is the first major “refactor” of this presentation in a long time. I think this restructuring makes the slides even easier to follow – there’s a lot of material to cover here, so this is always high on my mind.

Without further ado – you can get the latest version of these slides, in handout PDF format, by clicking on the thumbnail image.

Certainly, the great responses and enthusiastic discussions every time I go through this encourages me to keep working on this. As always, if you have any questions, suggestions or good/bad stories about working remotely or as part of a geo-distributed teams, please let me know (either by email or in the comments below) – I’d love to hear them.


Mozilla’s Release Engineering now on Dr Dobbs!

No Comments

book coverLong time readers of this blog will remember when The Architecture of Open Source Applications (vol2) was published, containing a chapter describing the tools and mindsets used when re-building Mozilla’s Release Engineering infrastructure. (More details about the book, about the kindle and nook versions, and about the Russian version(!).

Dr Dobbs recently posted an article here which is an edited version of the Mozilla Release Engineering chapter. As a long time fan of Dr Dobbs, seeing this was quite an honor, even with the sad news here.

Obviously, Mozilla’s release automation continues to evolve, as new product requirements arise, or new tools help further streamline things. There is still lots of interesting work being done here – for me, top of mind is Task Cluster, and ScriptHarness (v0.1.0 and v0.2.0). Release Engineering at scale is both complex, and yet very interesting – so you should keep watching these sites for more details, and consider if they would also help in your current environment. As they are all open source, you can of course join in and help!

For today, I just re-read the Dr. Dobbs article with a fresh cup of coffee, and remembered the various different struggles we went through as we scaled Mozilla’s infrastructure up so we could quickly grow the company, and the community. And then in the middle of it all, found time with armenzg, catlee and lsblakk to write about it all. While some of the technical tools have changed since the chapter was written, and some will doubtless change again in the future, the needs of the business, the company and the community still resonate.

For anyone doing Release Engineering at scale, the article is well worth a quiet read.

The canary in the coal mine

1 Comment

After my recent “We are ALL Remoties” presentation at Wikimedia, I had some really great followup conversations with Arthur Richards at WikiMedia Foundation. Arthur has been paying a lot of attention to scrum and agile methodologies – both in the wider industry and also specifically in the context of his work at Wikimedia Foundation, which has people in different locations. As you can imagine, we had some great fun conversations – about remoties, about creating culture change, and about all-things-scrum – especially the rituals and mechanics of doing daily standups with a distributed team.

Next time you see a group of people standing together looking at a wall, and moving postit notes around, ask yourself: “how do remote people stay involved and contribute?” Taking photographs of the wall of postit notes, or putting the remote person on a computer-with-camera-on-wheeled-cart feels like a duct-tape workaround; a MacGyver fix done quickly, with the best of intentions, genuinely wanting to help the remote person be involved, but still not-a-great experience for remoties.

There has to be a better way.

We both strongly agree that having people in different locations is just a way to uncover the internal communication problems you didn’t know you already have… the remote person is the canary in the coal mine. Having a “we are all remoties” mindset helps everyone become more organized in their communications, which helps remote people *and* also the people sitting near each other in the office.

Arthur talked about this idea in his recent (and lively and very well attended!) presentation at the Annual Scrum Alliance “Global Scrum Gathering” event in Phoenix, Arizona. His slides are now visible here and here.

If you work in an agile / scrum style environment, especially with a geo-distributed team of humans, it’s well worth your time to read Arthur’s presentation! Thought provoking stuff, and nice slides too!

A new focus

1 Comment

I’ve just added two new categories (“Release Engineering” and “Startup”) to my blog. This reflects the new reality of my life.

Obviously, many of my existing posts are already about Release Engineering, an area I care deeply about, yet somehow I just never flagged them correctly – I’ll fix that. The bigger news is about “Startup”. A few months ago, I decided to take the plunge and actually do what I’ve been talking about for years – start my own startup.

Since then, every day has been really busy, exciting, scary and fun – sometimes even all on the same day! Finding a bug in some AWS API documentation. Reading legal contracts with a highlighter and having to stop to Google some of the terms. Getting phone calls from a stranger that start with “you don’t know me, but I got your name from xxx and I hope you can help…”. Saying “what could possibly go wrong” multiple times a day. Joking about “pay no attention to the man behind the green curtain” while preparing a demo. Politely declining a job offer from a cold call recruiter, hanging up, taking a deep breath, calmly reminding myself that 9 out of 10 startup fail, and then jumping back into the fray. Oddly enough, I find I’m sleeping more, and feeling less stressed!? So far. Oh, and I’m drinking even more coffee then usual (yes, that is possible!).

Things are still under wraps, but as soon as there’s something worthwhile to show or talk about, I’ll post here on my blog.

In addition to the PRODUCT of the startup, I’ll also be blogging about the PROCESS of creating the startup. Technical, business, human aspects… warts and all. I’ve found it really helpful, and encouraging, to read posts from other founders and investors who’ve gone before me, on what they learned while building a startup – not just airbrushed niceties but also the genuine good/bad/funny/horror/irreverent/snafu stories that people have posted about life while building a startup. Some I’ve nodded along with, say “that was obvious”. Some I’ve re-read multiple times carefully and made mental notes. All are honest and helpful – to me and I’m sure many many others also. In that “pay it forward” spirit, I’ll make time to blog about this, and hopefully others starting their own entrepreneurial path will find these posts helpful – in a “oh, that is clever, I should make sure to do that” way… or “oh boy, I need to make sure to *never* do that” way… or somewhere in between!

I have to say I feel incredibly lucky with the support and encouragement of friends, family, former-co-workers and others I’ve bumped into over the years. Not just generic “don’t worry – it will be fine” support. Even with best of intentions, telling people what you think they want to hear, even when you think it may not be a good idea, is not good – it can set someone up to fail. Instead, I’ve been getting really helpful, informed, constructive support and advice like “maybe if you change it to…” or “have you asked xxx, she might have an insight…” or “that was good, don’t change that” or “… ok, that didn’t go well, so how will you do better next time?” Sometimes hard to hear, but always true from the heart and totally honest. This support is priceless, and means a great deal to me, so I find myself listening very carefully and humbly thanking people a LOT.

Older Entries