“Distributed” ER#3 now available!

No Comments

Book Cover for DistributedEarlier this week, just before the US Thanksgiving holidays, we shipped Early Release #3 for my “Distributed” book-in-progress.

Early Release #3 (ER#3) adds two new chapters: Ch.1 remoties trends, Ch.2 the real cost of an office, and many tweaks/fixes to the previous Chapters. There are now a total of 9 chapters available (1,2,4,6,7,8,10,13,15) arranged into three sections. These chapters were the inspiration for recent presentations and blog posts here, here and here.)

ER#3 comes one month after ER#2. You can buy ER#3 by clicking here, or clicking on the thumbnail of the book cover. Anyone who already has ER#1 or ER#2 should get prompted with a free update to ER#3. (If you don’t please let me know!). And yes, you’ll get updated when ER#4 comes out next month.

Please let me know what you think of the book so far. Your feedback get to help shape/scope the book! Is there anything I should add/edit/change? Anything you found worked for you, as a “remotie” or person in a distributed team, which you wish you knew when you were starting? If you were going to setup a distributed team today, what would you like to know before you started?

Thank you to everyone who’s already sent me feedback/opinions/corrections – all changes that are making the book better. I’m merging changes/fixes as fast as I can – some days are fixup days, some days are new writing days. All great to see coming together. To make sure that any feedback doesn’t get lost or caught in spam filters, it’s best to email a special email address (feedback at oduinn dot com) although feedback via twitter and linkedin works also. Thanks again to everyone for their encouragement, proof-reading help and feedback so far.

Now, it’s time to get back to typing. ER#4 is coming soon!


“Distributed” Early Release #2 now available!

No Comments

Book Cover for DistributedLast week, we rolled out Early Release #2 for my “Distributed” book-in-progress.

EarlyRelease#2 (ER#2) adds two new chapters (Ch.12 one-on-ones and reviews; Ch.14 group socials and work-weeks). There are also a bunch of tweaks and fixes to the previous Chapters 1,5,6,7,9, including grouping related chapters into three sections.

This is one month after ER#1. You can buy ER#2 by clicking here, or clicking on the thumbnail of the book cover. Anyone who already bought ER#1 should get prompted to update to ER#2. (If you don’t please let me know!). And yes, you’ll get updated when ER#3 comes out. For added goodness with an Early Release, your feedback get to help shape/scope the book!

Please let me know what you think of the book so far. Is there anything I should add/edit/change? Anything you found worked for you, as a “remotie” or person in a distributed team, which you wish you knew when you were starting? If you were going to setup a distributed team today, what would you like to know before you started?

Thank you to everyone who’s already sent me feedback/opinions/corrections – all changes that I hope make the book better. To make sure that any feedback doesn’t get lost or caught in spam filters, it’s best to email a special email address (feedback at oduinn dot com) although feedback via twitter and linkedin works also. Thanks again to everyone for their encouragement, proof-reading help and feedback so far.

Now, it’s time to brew more coffee and get back to typing. ER#3 is only a few weeks away!


“Distributed” presentation at Cultivate NYC

No Comments

I recently presented about distributed teams and remoties at Cultivate, NYC. The conference was held in the Javits center, in Manhattan, in the midst of the Strata/Hadoop NYC conference and was quite a crowd. The place is so large there were multiple pigeons flying around *inside* the building all week long!

After blogging and presenting about “remoties” and distributed teams for years, this was very different to anything I’ve presented before. I covered a lot of remotie / distributed team topics, including some new research I’m writing in my book.

The slides are here (or click the thumbnail image). I’ll post a copy of the recording once I have it.

Liza Daly, CTO at Safari already posted a great writeup about her presentation, and also on points in my presentation about meetings, so no need for me to repeat those here. Instead, you should read her post. I was encouraged to see that she and I are both tackling some of the same problems, in similar ways.

I’ll touch on three other points that got a lot of followup questions and discussions afterwards:

1) Remote/distributed teams are not a new phenomenon.

I believe humans have been able to work in distributed teams since we learned how to communicate. However, documentation *that* old is hard to find. Examples I *could* find were:

Almost 500 years ago (1519-1522), Magellan’s expedition sailed around the world. This project (“expedition to find spices”) started with “the boss” (The King of Spain) delegated full management authority to Magellan – empowering Magellan to execute staff if he saw fit (He did!) and even start wars on behalf of the King without first checking back to the King asking for approval (He did, which is how he died in battle in the Philippines). The King was being practical, not reckless. After all, it took them 3 years to sail around the world, discovering a new route as they went – so it was simply not practical to stop and wait while a message went back to HQ asking for the King’s permission to do something. To replace dead employees, Magellan “recruited” locals found on the way, training them on the job. Only a handful of the original 270 staff made it back home alive. Only 1 of the 5 original ships made it back home. By contrast, today’s navies are in constant contact with HQ, can request new supplies and trained staff to be quickly delivered anywhere in the world – and they need to ask permission from the boss before starting a war. This change in how they work is because of how they can communicate. Modern day submarines operate under different rules, again because of how they communicate.

Exactly 200 years ago was the Battle of Waterloo. On one day, 18 June 1815, the armies of France, England and Prussia converged for battle. These 191,000 men fought, and in one 24hour day, there were 65,000 dead/wounded/MIA. Using the best communication technologies available at the time, the generals would stand on hilltops overlooking the flat-ish battlefield, and send orders to soldiers across the battlefield by sending foot-messengers, waving flags and playing prearranged coded sequences of notes on their bugles. This scaled better than having all soldiers standing within earshot of the generals, but still had lots of limitations, not least of which was needing soldiers to hold battles in daytime, and difficulty coordinating work out of sight of each other. By contrast, modern soldiers have night vision equipment, as well as personal encrypted radios so they can communicate quickly and safely with coworkers, even if they are out of sight from each other, as well as “back to HQ” for supplies and reinforcements.

In each of these examples, the use of new communication tools radically changed how these organizations operate.

“Changing how we communicate lets us change how we work”(click to tweet)

Similarly, we have new communication tools in business. The ability to talk with people far away (telephone), the ability to write on a portable device (portable typewriter), and the ability to create moving pictures (reel-to-reel movie cameras) have gone from “expensive new technologies” to routinely included for free in each laptop and cellphone. Given the possibilities these new technologies allow, and their widespread availability, why do so many companies still arrange their organizations like this when their customers are global and look like this?

2) Hiring:

Literally everyone attending Cultivate NYC was hiring. Not one exception. And everyone was having a hard time hiring in today’s super-competitive environment. Yet, as best as I could see in the large hall, very few said “remote welcome” on the job description.

This is important because recent changes to society are also impacting how companies should recruit.

There are no more “hired for life” jobs, which makes people less willing to relocate for each job as they progress through their career. This year (2015) in the USA, there is a generational change, where the biggest segment of the workforce is now Millennials – people who grew up thinking streaming video on cellphones is “normal”! There is widespread broadband availability, making it possible to afford internet connectivity at home as-good-as-or-better than your connectivity at the office. Forrester estimated that in 2009, there were ~34million work-from-home people, and that by next year (2016), it will be ~63million people working from home… with the rate continuing to rise as broadband rollout continues.

To make matters worse for hiring managers, some qualified people who have the right skills won’t even apply for the job, simply because they know that if they got the job, they cannot accept it. That’s worth repeating. They have the skills, but they won’t even give you the chance to interview them, simply because if they got the job, they cannot reliably get to/from the office… so they don’t even apply. So, if you are a hiring manager, ask yourself:

Are you “Hiring the best person for the job”? Or are you “Hiring the best person for the job who is willing to relocate”?

If you want to “hire the best person for the job (regardless of where they live)”, then explicitly put “remote welcome” in your job description. This will let you hire better candidates, hire faster and give you a business advantage over your all-in-one-location competitors.

3) Disaster Planning

The week before Cultivate NYC, the Pope visited NYC. This caused NYPD to plan widespread street shutdowns for security during various site visits, prompting NYC Mayor de Blasio to encourage people to work from home for the week.

“If someone has the option to stay home, work from home, that’s a great choice…”. de Blasio, Mayor, NYC.

Similarly, while Hurricane Sandy in NYC caused death and destruction, many people were fine, simply stuck at home, unable to get to the office because subways was flooded. Their home was fine. Their office was fine. They just could not *get* to the office. And because their work habits assumed being in the office, they were unable to work effectively from home.

It’s one thing to work-from-home, on a part-time basis, choosing to specifically only do solo-work / focus-work. Instead, I asked people could they do their “normal job” (meetings, conference calls, pair-programming, escalations, interacting with co-workers, etc) while working from home? Most people actually cannot do this – because they are self-trained to go to office to do this in person. By contrast, experienced remote / distributed teams do this just fine – they are using cheap, familiar tools and crisp human processes which they practice daily.

I left everyone with a one day experiment: If you left this conference today, and went home with just the contents of your bag, could you do your normal job tomorrow (for just one day) if the office was unexpectedly closed tomorrow? If not, don’t worry – its just a one day experiment. Learn why, come into the office the next day, fix it, and try again the following week.

“We build fault tolerant systems. We need fault tolerant companies.”(click to tweet)

There were lots of note-taking throughout, and it was very exciting to see the wide-spread interest – at the conference as well as in the followup back-to-back meetings throughout the rest of the week.

ps: I also want to give a shout out of thanks to Jason and Laurel for the thousand-and-one logistical details they worked through, literally all the time. Amazing to see.

The “Distributed” book-in-progress: Early Release#1 now available!

No Comments

My previous post described how O’Reilly does rapid releases, instead of waterfall-model releases, for book publishing. Since then, I’ve been working with the folks at O’Reilly to get the first milestone of my book ready.

As this is the first public deliverable of my first book, I had to learn a bunch of mechanics, asking questions and working through many, many details. Very time consuming, and all new-to-me, hence my recent silence. The level of detailed coordination is quite something – especially when you consider how many *other* books O’Reilly has in progress at the same time.

Book Cover for DistributedOne evening, while in the car to a social event with friends, I looked up the “not-yet-live” page to show to friends in the car – only to discover it was live. Eeeeek! People could now buy the 1st milestone drop of my book. Exciting, and scary, all at the same time. Hopefully, people like it, but what if they don’t? What if I missed an important typo in all the various proof-reading sessions? I barely slept at all that night.

In O’Reilly language, this drop is called “Early Release #1 (ER#1)”. Now that ER#1 is out, and I have learned a bunch about the release mechanics involved, the next milestone drop should be more routine. Which is good, because we’re doing these every month. Oh, and like software: anyone who buys ER#1 will be prompted to update when ER#2 is available later in Oct, and prompted again when ER#3 is available in Nov, and so on.

You can buy the book-in-progress by clicking here, or clicking on the thumbnail of the book cover. And please, do let me know what you think – Is there anything I should add/edit/change? Anything you found worked for you, as a “remotie” or person in a distributed team, which you wish you knew when you were starting? If you were going to setup a distributed team today, what would you like to know before you started?

To make sure that any feedback doesn’t get lost or caught in spam filters, I’ve setup a special email address (feedback at oduinn dot com) although I’ve already been surprised by feedback via twitter and linkedin. Thanks again to everyone for their encouragement, proof-reading help and feedback so far.

Now, it’s time to brew more coffee and get back to typing.


A Release Engineer’s view on rapid releases of books

No Comments

As a release engineer, I’ve designed and built infrastructure for companies that used the waterfall-model and for companies that used a rapid release model. I’m now writing a book, so recently I have been looking at a very different industry through this same RelEng lens.

In days of old, here’s a simplistic description of what typically happened. Book authors would write their great masterpiece on their computer, in MSWord/Scrivener/emacs/typewriter/etc. Months (or years!) later, when the entire book was written, the author would deliver the “complete” draft masterpiece to the book company. Editors at the book company would then start reading from page1 and wade through the entire draft, making changes, fixing typos, looking for plot holes, etc. Sometimes they could make these corrections themselves, sometimes the changes required sending the manuscript back to the author to rewrite portions. Later, external reviewers would also provide feedback, causing even more changes. If all this took a long time, sometimes the author had to update the book with new developments to make sure the book would be up-to-date when it finally shipped. Tracking all these changes was detailed, time-pressured and chaotic work. Eventually, a book would finally go to press. You could reduce risk as well as simplify some printing and shipping logistics by having bookshops commit to pre-buy bulk quantities – but this required hiring a staff of sales reps selling the unwritten books in advance to bookstores before the books were fully written. Even so, you could still over/under estimate the future demand. And lets not forget that once people start reading the book, and deciding if they like it, you’ll get word-of-mouth, book reviews and bestseller-lists changing the demand unpredictably.

If people don’t like the book, you might have lots of copies of an unsellable book on your hands, which represents wasted paper and sunk costs. The book company is out-of-pocket for all the salaries and expenses from the outset, as well as the unwanted printed books and would hope to recover those expenses later from other more successful books. Similar to the Venture Capital business model, the profits from the successful books help recoup the losses from the other unprofitable books.

If people do like the book, and you have a runaway success, you can recoup the losses of the other books so long as you avoid being out-of-stock, which could cause people to lose interest in the book because of back order delays. Also, any errors missed in copy-editing and proofreading could potentially lead to legal exposure and/or forced destruction of all copies of printed books, causing further back order delays and unexpected costs. Two great examples are The Sinner’s Bible and the Penguin cook book.

This feels like the classic software development “waterfall” model.

By contrast, one advantage with the rapid release model is that you get quick feedback on the portions you have shipped so far, allowing you to revisit and quickly adjust/fix as needed… even while you are still working on completing and shipping the remaining portions. By the time you ship the last portion of the project, you’ve already adjusted and fixed a bunch of surprise gotchas found in the earlier chunks. By the time you ship your last portion, your overall project is much healthier and more usable. After all, for the already shipped portions, any problems discovered in real-world-use have already been fixed up. By comparison, a project where you write the same code for 18-24 months and then publish it all at the same time will have you dealing with all those adjustments/fixes for all the different chunks *at the same time*. Delivering smaller chunks helps keep you honest about whether you are still on schedule, or let you quickly see whether you are starting to slip the schedule.

The catch is that this rapid release requires sophisticated automation to make the act of shipping each chunk as quick, speedy and reliable as possible. It also requires dividing a project into chunks that can be shipped separately – which is not as easy to do as you might hope. Doing this well requires planning out the project carefully in small shippable chunks, so you can ship when each chunk is written, tested and ready for public consumption. APIs and contractual interfaces need to be understood and formalized. Dependencies between each chunk needs to figured out. Work estimates for writing and testing each module need to be guessed. A calendar schedule is put together. The list goes on and on… Aside: You probably should do this same level of planning also for waterfall model releases, but its easy to miss hidden dependencies or impact of schedule slips until the release date when everything was supposed to come together on the day.

So far, nothing new here to any release engineer reading this.

One of the (many) reasons I went with O’Reilly as the publisher for this book was their decision to invest in their infrastructure and process. O’Reilly borrowed a page from Release Engineers and invested in automation to switch their business from a waterfall model to a rapid release model. This change helps keep schedules realistic, as schedule slips can be spotted early. It helps get early feedback which helps ship a better final product, so the ratio of successful book vs unprofitable books should improve. It helps judge demand, which helps the final printing production planning to reduce cost wasting with less successful books, and to improve profits and timeliness for successful books. This capability is a real competitive advantage in a very competitive business market. This is an industry game changer.

When you know you want “rapid release for books”, you discover a lot of tools were already there, with a different name and slightly-different-use-cases. Recent technologies advances help make the rest possible. The overall project (“table of contents”). The breaking up of the overall project into chunks (“book chapters”). Delivering usable portions as they are ready (print on demand, electronic books + readers, online updates). Users (“readers”) who get the electronic versions will get update notices when newer versions become available. At O’Reilly, the process now looks like this:

Book authors and editors agree on a table of contents and an approximate ship date (write a project plan with scope) before signing contracts.

Book authors “write their text” (commit their changes) into a shared hosted repository, as they are writing throughout the writing creative process, not just at the end. This means that O’Reilly editors can see the state of the project as changes happen, not just when the author has finished the entire draft manuscript. It also means that O’Reilly reduces risk of catastrophic failure if an author’s laptop crashes, or is stolen without a backup.

A hosted automation system reads from the repository, validates the contents and then converts that source material into every electronic version of the book that will be shipped, including what is sent to the print-on-demand systems for generating the ink-on-paper versions. This is similar to how one source code revision is used to generate binaries for different deliverables – OSX, Windows, linux, iOS, Android,…

O’Reilly has all the automation and infrastructure you would expect from a professional-grade Release Engineering team. Access controls, hosted repos, hosted automation, status dashboards, tools to help you debug error messages, teams to help answer any support questions as well as maintain and improve the tools. Even a button that says “Build!”!!
The only difference is that the product shipped from the automation is a binary that you view in an e-reader (or send to a print-on-demand printing press), instead of a binary that you invoke to run as an application on your phone/desktop/server. With this mindset, and all this automation, it is no surprise that O’Reilly also does rapid releases of books, for the same reasons software companies do rapid releases. Very cool to see.

I’ve been thinking about this a lot recently because I’m now putting together my first “early release” of my book-in-progress. More info on the early release in the coming days when it is available. As a release engineer and first time author, I’ve found the entire process both totally natural and self-evident and at the same time a little odd and scary. I’d be very curious to hear what people think… both of the content of the actual book-in-progress, and also of this “rapid release of books” approach.

Meanwhile, it’s time to refill my coffee mug and get back to typing.


“An Illustrated Book of Bad Arguments” by Ali Almossawi

No Comments

Most engineers I know are good at writing software. Some engineers I know are good at writing software for extraordinarily complex projects. Some have a brain that naturally, instinctively, just worked this way while some took all sorts of courses on algorithms and structured programming in university. Regardless, when you are coding by yourself, these important coding and problem solving skills are probably good enough. However, when you transition from coding-by-yourself to coding-in-a-team, there is the need for an additional skill.

Now you need the skills to explain to other engineers why your code, your approach, is best. Maybe even why your approach is better then an alternate approach being proposed by someone else. It is a given that you can write good code – that is how you got on the team in the first place. The question is: can you explain why your code is better then any other alternatives available?

This skill might sound deceptively simple, but in reality it has a few super complex twists to it. Someone once explained it to me as follows: smart people, given the same start point, the same goal, and the same restrictions/limitations, would come up with similar-enough solutions. Maybe not identical, because all humans are slightly different, but at least similar in approach – and would solve the same problem. Given that, if I was to find myself in a situation where very different solutions are being discussed, maybe we have different understanding of what the starting point is? Or what the goal is? Or what restrictions/assumptions/limitations to work within? Double checking all those can quickly uncover missing (or invalid) problem statements, restrictions/assumptions/limitations or goals. If it turns out that we’re trying to solve different problems, then of course the proposed solutions would be very different!

If your skills in logic/reasoning/conflict are weak, and if you skip the verification step, then it is easy to get frustrated by your inability to “convince others” of the greatness of your code or design. Maybe you cannot explain it correctly. Maybe the others don’t want to listen to you – because they feel equally about their proposals. Maybe both. If you are passionate about your code, but cannot have others understand its greatness, the situation can easily turn to frustration (“I give up – why does no-one understand how great my code is?”) or even anger (“This code is too important to give up on – what do they know anyway – they’re just idiots”). This shift from working together to find the best code/design (something which can be objectively measured or stress-tested, after everyone agrees on what they all think is “best”.) to destructively debating the other humans (using hard-to-measure qualities) is dangerous. Even if the “right” code is actually chosen in the end, the manner in which the discussion was carried out can be toxic to the group.

Like many other engineers, I never had any formal training in logic/reasoning/conflict at any of my universities, so I had to learn these skills on-the-job, or at workshops I found over the years. Sometimes it went well. Sometimes it did not. I’m still working on this. Being able to talk through differences of opinions, in a constructive way, is a crucial skill set to develop as you work with others during your engineering career. And essential for *everyone* in a team to have, in order for the *team* to work together effectively.

I stumbled across this book in 2014, loved it, and was then delighted and surprised to discover that it was written by someone I knew from when I worked at Mozilla!! Ali’s book is focused on describing the different types of arguments used in logic debates, with very easy-to-understand examples. This will help you keep your logic and reasoning honest when you are a discussion. It will also make it easy to notice when someone else switches to using “Ad Hominem” or “No True Scotsman” or “Genetic Fallacy” or “Straw Man” or any other of these types of arguments on you.

I think many many people, including myself, find the idea of a book about arguments to be daunting, almost off-putting, so I really like how the simple one-page description and the full page cartoon images make the whole topic area more approachable, appealing and in turn, very easy to understand in a non-threatening way. At 56 pages, it is a super short book, and almost 50% of those pages are cartoons, so you could expect to be done in a few minutes. However, each page found me sitting rethinking different conflicts over the years, so this took a lot longer to read then I expected. And I re-read it often. In addition to the printed version of the book, there is also a fun https://bookofbadarguments.com/.

Try it, I think you will like it.

ps: This is one of two great books that I really like in this area – both of which I think should be required reading for engineers. Both of which I just discovered I had not yet blogged about, hence this post. Watch for another post coming soon.

The “we are all remoties” book!?!


I’ve been working in distributed teams, as well as talking, presenting, coaching and blogging about “remoties”‚ in one form or another for 8?9? years now. So, I’m excited to announce that I recently signed a contract with O’Reilly to write a book about how to successfully work in, and manage in, a geo-distributed world. Yes, I’m writing a “we are all remoties” book. If you’ve been in one of my ever-evolving “we are all remoties” sessions, you have an idea of what will be included.

If you’ve ever talked with me about the pros (and cons!) of working as a remote employee or of working in a distributed team, you already know how passionate I am about this topic. I care deeply about people being able to work well together, and having meaningful careers, while being physically or somehow otherwise remote from each other. Done incorrectly, this situation can be frustrating and risky to your career, as well as risky to employers. Done correctly, however, this could be a global change for good, raising the financial, technical and economic standards across all sorts of far flung places around the globe. Heady game-changing stuff indeed.

There are many “advocacy books” out there, explaining why working remote is a good / reasonable thing to do – typically written from the perspective of the solo person who is already remote. There are also many different tools becoming available to help people working in distributed teams – exciting to see. However, I found very few books, or blogposts, talking about the practical mechanics of *how* to use a combination of these tools and some human habits to allow humans to work together effectively in distributed teams, especially at any scale or over a sustained amount of time. Hence, my presentations, and now, this upcoming book.


  • if you are physically geo-distributed from the people you work with, I’d like to hear what does or doesn’t work for you. If you know someone who is in this situation, please share this post with them.
  • If you have experience working in distributed teams, is there something that you wish was already explained in a book? Something that you had to learn the hard way, but which you wish was clearly signposted to make it easier for others following to start working in distributed teams? Do you have any ideas that did / didn’t work for you?
  • If you have published something on the internet about remoties, please be tolerant of any questions I might ask. If you saw any of my “we are all remoties” presentations, is there anything that you would like to see covered in more/less detail? Anything that you wish was written up in a book to help make the “remote” path easier for those following behind?

…you can reach me on twitter (“@joduinn”) or on email (john at my-domain-name – and be sure to include “remoties” in the subject, to get past spam filters.)

Now, time to brew some coffee and get back to typing.

(updated 31jul2015 to add twitter + email address.)

“Hot Seat: The CEO Guidebook” by Dan Shapiro

No Comments

This book just came out and I loved it. If you are starting a company, or thinking of it, you need to read this book. Period.

Dan covered a whole range of topics very succinctly, and in easy-to-follow language. When and how to raise funds. What all those terms mean. Who should (and should not!) be on your board, and why. How to allocate shares and ownership between co-founders. Where to incorporate your company (Dan has strong opinions on this!). How to create (and then also maintain) company culture. A great section on decision making. A section on “Hiring” in the context of the Manhattan project vs the moon shot Apollo project that I think every engineering hiring manager should read before building a team. Several true stories about startups where co-founders mismatches caused company threatening problems (trivia: 6 of 10 startups lose a co-founder in early days). And some good (and bad!) stories of how important trust was.

Some great quotes that resonated with me:

“You have limited resources of time and money. When they run out, you go bankrupt. The important thing is not cost/benefit: it’s opportunity cost.”

(in the context of how much travel was needed for all the in-person meetings with investors when raising funding) “…Alaska Airlines gave me MVP status for my efforts. In January.”

“Entrepreneurship is the pursuit of opportunity without regard to the resources currently controlled”. Prof Stevenson, Harvard.

In a variation of the “fail fast” mantra in developer circles, Dan notes that “…while it might seem like cold comfort now, the sooner you fail, the sooner you can try again.” Oh, and he’s not just saying it – that was the ending of a chapter where he detailed the failure of one of his startups.

His tolerance for large volumes of coffee and pointer to suggested reading “Coffee, CYP1A2 Genotype, and Risk of Myocardial Infarction” was a great and unexpected tangent for me personally. (More info here Journal of American Medical Association)

“Startups don’t out think their competitors; they out-execute them.”

“If leadership is the forest, then management is the trees. Day to day, it’s what consumes your time, and its imperative that you get it right.”

It takes skill and seasoned-experience-in-the-field to have one person cover all these different topics. Even more skill to do so clearly, and concisely. Putting them all together in a way that makes sense was great. Just great. If you are starting a company, or thinking of it, you need to read this book. Period.

Aside: Having this on my kindle app, on my trusty nexus5 phone was quite a good reading experience. The book was written in short, digestible chapters, which I could quickly complete standing a store line, or in the back of a taxi between meetings. It also encouraged me to think more about the chapter I just finished in the time before I got to stop and read some more. A nice way to digest the many lessons in here. I’m still experimenting with what books I find do work best on phone+kindle vs ink-on-paper, but at least for this book, reading on kindle worked for me.

(Disclaimer: I bought this book because I’m starting my own company, and that is the basis of the above review. As this book is published by O’Reilly Press, it feels important to disclose that I am also currently doing some work with O’Reilly… which did not influence anything I wrote here.)

Mozilla’s Release Engineering now on Dr Dobbs!

No Comments

book coverLong time readers of this blog will remember when The Architecture of Open Source Applications (vol2) was published, containing a chapter describing the tools and mindsets used when re-building Mozilla’s Release Engineering infrastructure. (More details about the book, about the kindle and nook versions, and about the Russian version(!).

Dr Dobbs recently posted an article here which is an edited version of the Mozilla Release Engineering chapter. As a long time fan of Dr Dobbs, seeing this was quite an honor, even with the sad news here.

Obviously, Mozilla’s release automation continues to evolve, as new product requirements arise, or new tools help further streamline things. There is still lots of interesting work being done here – for me, top of mind is Task Cluster, and ScriptHarness (v0.1.0 and v0.2.0). Release Engineering at scale is both complex, and yet very interesting – so you should keep watching these sites for more details, and consider if they would also help in your current environment. As they are all open source, you can of course join in and help!

For today, I just re-read the Dr. Dobbs article with a fresh cup of coffee, and remembered the various different struggles we went through as we scaled Mozilla’s infrastructure up so we could quickly grow the company, and the community. And then in the middle of it all, found time with armenzg, catlee and lsblakk to write about it all. While some of the technical tools have changed since the chapter was written, and some will doubtless change again in the future, the needs of the business, the company and the community still resonate.

For anyone doing Release Engineering at scale, the article is well worth a quiet read.

“REAMDE” by Neal Stephenson


(At 1,044 pages, this book looks daunting. I’ve enjoyed other Neal Stephenson books, especially Cryptonomicon, so I didn’t let the size of the book deter me when I was buying it. But I find reading a long book with complex intertwined plots needs continuity – no point in picking it up and trying to resume after leaving it unopened for weeks! Even though I bought this book over a year ago, I only finally had time to read it in the last couple of weeks. Aside, in this day-and-age-of-laptops-and-kindles, I was amused by the odd sidelook I got whenever I settled into a nearby coffee shop and produced this weighty hardback ink-on-paper tomb!)

Wikipedia has a great summary here, but obviously be warned that it has lots of plot spoilers. Without giving too much plot away, I liked the book. From my perspective, I really enjoyed how Neal can interweave different stories. While there were many different interwoven stories here, the ones that are top of mind for me were:

  • the hacker-and-former-girlfriend-get-kidnapped story
  • the spy-tracking-jihadists story
  • the massive on-line game business story

All very different stories, yet the detailed coverage of each make me think Neal has a great understanding of hackers, encryption, different-business-market-economies-of-massive-on-line-games, Soviet-veterans-of-the-confict-in-Afganistan, internet-cafes-in-developing-worlds… the list goes on and on. I even found the way computer issues were covered to be accurately describes (typically a pet peeve for me!). In the midst of all the other drama, I was greatly amused by the image of a super-important invulnerable character (Egdod) walking in unattended mode back to home base, while various other T’Rain players were attacking him / defending him / rubber-necking the impossible sight of Egdod moving through their world. And somehow, someday, I need to find a way to use the throwaway joke about “Your org chart?”, “No, orc chart”.

The book was a great read, and I’d recommend it.

Older Entries