A few years ago, I first put my toes into the book publishing world by co-writing a portion of AOSAv2 about Mozilla’s RelEng infrastructure. Having never written part of a published book before, I had no idea what I was really getting into. It was a lot of work, in the midst of an already-busy-day-time-job and yet, I found it was strangely quite rewarding. Not financially rewarding – all proceeds from the book went to Amnesty – but rewarding in terms of getting us all to organize our thoughts to write down in a clear, easy to read way, explaining the million-and-one details that “we just knew instinctively”, and hopefully helping spread the word to other software companies on what we did when changing Mozilla’s release cadence.
While working on AOSAv2, clearly explaining the technology was hard work, as expected, but I was surprised by how much work went into “simple” mechanics – merging back reviewer feedback, tracking revisions, dealing with formatting of tables and diagrams, publishing in different formats… and remember, this was a situation where book publisher contracts, revenue and other “messy stuff” was already taken care of by others. I “just” had to write. I was super happy to have the great guidance and support of Greg Wilson and Amy Brown who had been-there-done-that, helped work through all those details, and kept us all on track.
Ever since then, I’ve been considering more writing, but daunted by all the various details above and beyond “just writing”. These blog posts help scratch that itch, in between my own real-life-work-deadlines, but the idea of writing a full book, by myself, still lingered. A while ago, I grabbed “APE: Author, Publisher, Entrepreneur-How to Publish a Book” by Guy Kawasaki. It looked like a good HowTo manual, I’ve enjoyed some of his other books, and he’s always a great presenter, so I was looking forward to some quiet time to read this book cover-to-cover.
This was well worth the time, and I re-read it a few times!
For me, some of the highlights were:
“why are you writing a book?” I really like how Guy turned this question around to “why would someone else want to read your book”. Excellent mind-flip. I’ve met a few people who want to write a book, and even a few published authors, and I’ve talked with them about my own ideas about writing. But no-one, not one, ever reversed the question like this. It was instantly self-evident to me – it takes time to read a book, and we’re all busy. So, even if someone gave me a book for free, why would I want to skip work and/or social plans to read a book by someone I don’t know. Making it clear, immediately, why someone would find it worthwhile reading your book is a crucial step that I think many people skip past. As the author, keeping this in mind at all times while writing, will help keep you focused on the straight-and-narrow path to writing a book that people would actually want to read.
Money: Most publishers are super-secret about their contracts/terms/conditions, which can make a new time author feel like they’re going to be taken (The only exception I know of is Apress, who publish all their terms on their website, with a “no haggling” clause). To help educate potential authors, I respect how much full detail Guy & Shawn gave in small, easy to follow, words.
“Tell the world you’re writing a book – not that you’re thinking of writing a book.” Again, an excellent mind-flip to help keep you motivated and writing, every single day, whether you want to or not. Also, they provided many links to writer’s clubs (writer support groups!?) who would help you keep motivated.
print-on-demand vs print-big-batch: This reminded me of how software release cycles are changing the software industry from old monolith release cadence to rapid-release cadence. “Old way”: a big-bang-release every unpredictable 18months, with a costly big print run, and lots of ways to handle financial risk of under/over selling; any corrections are postponed until the next big-bang-release if it looks like there is enough interest. “New way”: build infrastructure to enable print-on-demand. Do smaller, more frequent, releases, each with small print runs, (almost) no risk of under/over selling, corrections handled frequently and easily. Yes, at first glance, each printed book might seem more expensive this way, but when you factor in the lack-of-under/over selling, removed financial risk, and benefits of frequent updates to the almost-free electronic readers, it actually feels cheaper, more efficient and more appealing to me.
In addition to printed books, there’s a good description of pros/cons of the different popular electronic formats (PDF, MOBI, EPUB, DAISY, APK…) as well as related DRM.
The differences between ebook publishers (Amazon, Apple, Barnes & Noble, Google, Kobo, …), Author Publisher Services (Lulu, Blurb, Author Solutions, …) and Print-on-Demand (Walkerville Publishing, Lightning Source, …) was detailed and very helpful. Complex chapter, with lots of data, and ending with the reassuring “Don’t obsess about making the wrong choice, however, because most distribution decisions are changeable.”!
translations, audiobooks: normally, these are handled as edge cases. Guy & Shawn walk through some of the options (Audible/Amazon, Books-on-Tape/RandomHouse), as well as financial & legal realities.
Some fun examples of rejection responses by agents/publishers. My personal favorite was a rejection sent to George Orwell about Animal Farm “It is impossible to sell animal stories in the USA”.
All in all, I found the writing style personal, helpful, direct and super honest. Even the way they ended the book… “Thank you. Now go write a book! —Guy and Shawn”
Its been a while since I last blogged about “remoties”, but it continues to be a very popular topic! In addition to Twilio in February, I’ve given presentations at Automattic (best known for WordPress), RiotGames (twice) and Haas, UCBerkeley (twice), as well as smaller private discussions with several other companies.
You can get the slides in PDF format by clicking on the thumbnail of the first slide. (I’m happy to share the original very large keynote file, just let me know and we’ll figure out a way to share without hammering my poor website.)
Remoties are clearly something that people care deeply about. Geo-distributed teams are becoming more commonplace, and yet the challenges continue to be very real. The interest before each presentation is cautiously high, while the Q+A discussions during/afterwards are very engaged and lively. Every time, I find myself tweaking, honing and refining the presentation again and again… yet, the core principles remain the same:
remoties / geo-distributed teams can be very effective, and can be sustained over time.
remoties != compromise. In fact, a geo-distributed team means you can hire best-available, not “just” best-willing-to-relocate.
easy to use, cheap, technologies work just fine if used correctly (maybe even better then expensive systems?)
crisp, careful organization of human processes is essential
in a geo-distributed team, *everyone* is a remotie, even people who happen to sit in an office. If you are remote from someone else, that makes you *both* remoties. Hence the working title “we are ALL remoties”.
Given how this topic impacts people’s jobs, and their lives, I’m not surprised by the passionate responses, and each time, the lively discussions encourage me to keep talking about this. As always, if you have any questions, suggestions or good/bad stories about working in a remote or geo-distributed teams, please let me know – I’d love to hear them.
ps: I noticed in my website logs that a lot of people were still downloading my original remoties slides, first posted in apr2012, even though I’d posted multiple revisions of the slides since. So, I’ve gone back and updated my earlier “remoties” blog posts to all point to these latest-and-greatest slides.
I was honored to give the opening keynote for USENIX URES14 East in Philadelphia in June 2014.
“The Value of Release Engineering as a Force Multiplier” keynote built on top of the “RelEng as a Force Multiplier” presentation I gave at RelEngConf 2013 and as then as a Google Tech Talk. (To get the slides in PDF format, click on thumbnail. Happy to share the original 25MB keynote file, just let me know and we’ll figure out a way to share without hammering my poor website.)
Anyone who has ever talked with me about RelEng knows I feel very strongly that Release Engineering is important to the success of every software project. Writing a popular v1.0 product is just the first step. If you want to keep your initial early-adopter users by shipping v1.0.1 fixes, or grow your user base by shipping new v2.0 features to your existing users, you need a reproducible pipeline for accurately delivering software in a repeatable manner. Otherwise, you are “only” delivering a short-lived flash-in-the-pan one-off project. In my opinion, this pipeline is another product that software companies need to develop, alongside their own unique product, if they want to stay in the marketplace, and scale.
Its typical for Release Engineers to talk about the value of RelEng in terms that Release Engineers value – timely delivery, accurate builds, turnaround time, etc. I believe its important to also describe Release Engineering in terms that people across an organization can understand. In my keynote, I specifically talked about the value of RelEng in terms that people-who-run-companies value – unique business opportunities, market / competitive advantages, new business models, reduced legal risk, etc.
Examples included: Mozilla’s infrastructure improvements which reduced turnaround time for delivering security fixes as well as helped deter future attacks… Hortonwork’s business ability to provide enterprise-grade support SLAs to customers running mission critical production “big data” systems on 100% open source Apache Hadoop… and even NASA’s remote software update of the Mars Rover.
People seemed to enjoy the presentation, with lively questions during, afterwards… and even into the end-of-day panel session.
Big thanks to the organizers (especially Dinah McNutt (RelEng at Google), Gareth Bowles) – they did an awesome job putting together a unique and special event.
Oh, and one more thing! Next week, USENIX URES14 West will start on Monday 10nov2014 in Seattle. If you are in the area, or can get there for Monday, you should attend! And make sure to see Kmoir’s presentation “Scaling Capacity While Saving Cash” – if you follow her blog, you know you can expect it to be well worth attending.
[Updated to include links to usenix recordings. joduinn 08nov2014]
The same great people who brought RelEng Conf 2013 did it again earlier this year with the sold-out-wait-listed RelEng Conf 2014. Hosted at Google’s HQ campus, it was a great combination of academic and battle-hardened down-to-earth no-holds-barred industry presentations and panel sessions.
Well, well, well… I’m very happy to discover that General Fuzz just put out yet another album. Tonight was a great first-listen, and I know this will make a happy addition to tomorrow’s quiet Sunday morning first-coffee. Thanks, James!
All in all, a great fun read, and I found the “extra” sidebar cartoons equally fun… especially the yakshaver! If you like xkcd, and don’t already have this book, go get it.
ps: He’s got a new book coming out in a few days, a book tour in progress, and a really subtle turtles-all-the-way-down comic which nudges about the new book… if you look *really* closely! I’m looking forward to getting my hands on it!
“Lost Cat” tells the true story of how an urban cat owner (one of the authors) loses her cat, then has the cat casually walk back in the door weeks later healthy and well. The book details various experiments the authors did using GPS trackers, and tiny “CatCam” cameras to figure out where her cat actually went. Overlaying that data onto google maps surprised them both – they never knew their cats roamed so far and wide across the city. The detective work they did to track down and then meeting with “Cat StealerA” and “Cat Stealer B” made for a fun read… Just like “Meanwhile in San Francisco”, the illustrations are all paintings. Literally. My all-time favorite painting of any cat ever is on page7.
A fun read… and a great gift to any urban cat owners you know.
The Hortonworks Release Engineering team is growing, so we’re hiring!
We’re passionate about open source, and ensure that all 100% of code in a Hortonworks HDP release is open sourced in the Apache Software Foundation Hadoop project. We work with other large organizations to help them upstream their contributions to the Apache project, which helps accelerate the general Hadoop community. Its so important to us, it is part of the Hortonworks Manifesto.
We’re proud of our HDP releases. Our clients rely on HDP in production environments where phrases like “petabytes per day” and “zettabytes” are common. We sim-ship on centos5, centos6, ubuntu, debian, suse and windows – all from the same changeset. Building and testing at this scale has its own special forms of challenges, and is exciting. In the rare case where customers hit production issues, we are able to deliver supported fixes super-quickly.
The Hortonworks Release Engineering team works hard behind the scenes to design, build and maintain the infrastructure-at-scale needed to make this possible. For more details, and to apply, click here.
Note: The current team is spread across 3 cities, so remoties are welcome, even encouraged! Hardly a surprise if you read the other remoties posts on my blog, but worth stating explicitly!
This was the first significant feature release shipped since I joined Hortonworks at the start of the year. There’s lots of interesting new features, and functionality in this HDP2.1 release – already well covered by others in great detail here. Oh, and of course, you can download it from here.
In this post, I’ll instead focus on some of the behind-the-scenes mechanics. There were lots of major accomplishments in this release, but the ones that really stood out to me were:
1) sim-ship windows and linux.
This was the first HDP release where all OS were built from the same changeset and shipped at the same time. Making this happen was a hectic first priority in January. As well as the plumbing/mechanics within RelEng, it also took lots of coordination changes across different groups within Hortonworks to make this happen. The payoff on this was great. We sim-shipped, which is great and massively important for HWX as a company. Even more importantly, we set things up so we could sim-ship for every HDP2.1-and-above release going forward… and we proved it by sim-shipping the quick followup HDP184.108.40.206 release on 02may2014.
2) adding 5 new components.
HDP2.1 contained 17 components, compared to HDP 2.0 (with 12 components) and HDP 1.3 (with 10 components), making HDP2.1 the largest growth of components ever?!? Oh, and in addition to the new components, every one of the 12 pre-existing components were also significantly updated to newer versions. That meant that each required significant new integration work, new installers on all supported OS (…remember the “sim-ship” goal?). Oh, and we were to ship all this new functionality at the fastest cadence yet.
3) improving support for other trains.
In January, we were learning how to support 3 active trains of code: supporting 1.3 and 2.0 maintenance work, while also building out infrastructure for 2.1 new-product-development-work… even while the 2.1 development work was in progress, which obviously complicated things for developers. Today, we’re supporting 4 active trains: maintenance work for 1.3, 2.0 and 2.1, as well as the 2.2 new-product-development-work. This time, the 2.2 infrastructure was built out and live before developers finished working on 2.1… enabling the developers! Things are not perfect yet, by any means, but today (with 4 trains) feels calmer and more organized then earlier this year (with â€œonlyâ€ 3 trains).
All great improvements to see up close, and all important to us as we scale. Big thanks to everyone for their help… and do stay tuned for even more improvements already underway.