01 Jan 2016
A few days ago, my Skype stopped working. Specifically, every time I started Skype, it would hang while logging into the Skype network. It took a while to debug this, so here are the details of how I fixed it, in case it helps others – or I have to do this again!
What am I running:
- MacBookPro running OSX 10.10.5
- Skype-for-Mac v7.18.342
When starting Skype, I saw the usual blue Skype window with the white blob moving in a circular pattern to show progress… and after literally one second or maybe two seconds, that blue window would disappear, leaving the Skype icon grayed out on the Mac taskbar. If I tab-switched to the Skype application, the mouse cursor would instantly pinwheel and the Skype menus would not respond to any mouse clicks. If I pressed command-option-escape to view all running applications, the Skype application was listed as “Not responding”. At first, I dismissed it as some odd intermittent problem, then killed and restarted Skype. Only to have it hang again, exactly the same way. This hang happened 100% of the time.
It had been almost a week since I last used Skype, so I wasn’t *certain* what I had changed since my last Skype call. I knew there were no hardware changes, OS patch changes. If anything changed at all, it is possible that Skype auto-updated or that Adobe Flash prompted me to manually update since my last Skype call, but I am not 100% certain.
Here’s some things I tried first, unsuccessfully:
- Force-quit Skype. Double-click on Skype to restart it. Hang.
- Force-quite Skype. Reboot mac. Double-click on Skype to restart it. Hang.
- To verify that my Skype account was ok, the Skype servers were up-and-running, and my wifi router was not misbehaving, I started Skype on my phone and also on my tablet. Both logged in fine, using wifi and both worked first time.
- Download the latest version of Skype (which was the same 7.18.342 version) from Skype.com download page. Installed new download of Skype 7.18.342 over existing installation of Skype 7.18.342. Reboot mac. Double-click on Skype to restart it. Hang.
- All to no success.
Frustrated, I spent some time googling the internet, tried lots of variations on the above steps – all without success. Finding old unanswered pleas for help in customer forums from others with the same problem is never encouraging. Now it was serious. I finally stumbled across this post about Skype crashing a computer which helped – even though my skype was not crashing, and certainly not crashing my computer!
Here’s the steps that fixed it:
- Kill Skype. Out of paranoia, I rebooted my mac.
- Using Chooser, rename the directory “/Users/[your-user-name]/Library/Application Support/Skype” to “/Users/[your-user-name]/Library/Application Support/Skype.broken_yyyy_mm_dd” (I find this helps me later figure out when I last had to fix this problem!)
- Change skype on Dock to not auto-start on login.
- Download new installation Skype dmg from Skype.com download page. At time of writing, the latest version was
v7.18.342 v7.19.407. v7.21.350
- Install new Skype 7.19.407 over existing installation of Skype.
- Reboot mac.
- Double-click on Skype icon on Mac Dock or in Application folder. (Do *not* double-click on skype.app located in the “Application Support” folder – that will not work.)
- On first login, my contacts and previous messages were all missing. However, over the next minute or so, all my previous contacts as well as all previous messages showed up again. When all settled down, the only thing that was missing all was my “starred contacts” list, which was easy to manually re-create.
Hope that helps.
(Updated to include new version of Skype, and to note that after several weeks of Skype running just fine, Skype hung again this morning. If this happens again with Skype v7.19.407, I’ll update this post. joduinn 27jan2016)
(Updated to include new version of Skype, and to note that after several weeks of Skype running just fine, Skype hung again just now after reboot. Unclear if caused by reboot or by upgrade to xcode, which caused me to reboot. Lets see if this new version v7.21.350 works better. joduinn 27feb2016)
03 Dec 2015
In my recent blog post about the one-time-setup and recurring-costs of an office, I mostly focused on financial costs, human distraction costs, and the cost of increased barriers to hiring. This post talks about another important scenario: when your physical office limits potential company revenue.
Pivigo.com is a company in London, England, which connects companies that need help with data science problems with Ph.D data science graduates who are leaving academia looking for real-world problems to solve. This 2.5 year old company was founded by Dr Kim Nilsson, (ex-astronomer and MBA!), and as of today employs 4 people.
For Pivigo to be viable, Kim needed:
- a pipeline of companies looking for help with their real-world Data Science problems. No shortage there.
- a pipeline of Ph.D graduates looking for their “first non-academic” project. No shortage there.
- a carefully curated staff of people who understand both Academic and Commercial worlds are essential to help keep things on track, and make sure the event is a success for everyone. Kim has been quietly, diligently working on growing a world-class team at Pivigo for years. Tricky, but Pivigo’s hiring has been going great – although they are always interested to meet outstanding people!
- a physical place where everyone could meet and work together.
Physical space turned out to be the biggest barrier to Pivigo’s growth and was also the root cause of some organizational problems:
1) Venue: The venue Pivigo had guaranteed access to could only be used once a year, so they could only do one “event” each year. Alternate venues they could find were unworkable because of financial costs, or the commute logistics in London. Given they could only have one course per year, it was in Pivigo’s interest to have these classes be as large as possible. However, because of the importance of creating a strong network bonding between the participants, physical size of venue, and limits on skilled human staffing, the biggest they could do was ~80 people in this once-a-year event. These limits on the once-a-year event puts a financial cap on the company’s potential revenue.
2) Staffing: These big once-a-year events were super-disruptive event to all the staff at Pivigo. Between the courses, there was administrative work to do – planning materials, interviewing candidates and companies, arranging venue and hotel logistics, etc. However, the “peak load time” during the course clearly outscaled the “low load time” in between courses. Hiring for the “peak load times” of the courses meant that there would be a lot of expensive “low load / idle time” between each peak. The situation is very similar to building capacity in fixed cost physical data centers compared to AWS variable-by-demand costs. To add to the complexity, finding and hiring people with these very specialised skills took a long time, so it was simply not practical to “hire by the hour/day” a la gig-economy. Smoothing out the peaks-and-troughs of human workload was essential for Pivigo’s growth and sustainability. If they could hold courses more frequently, they could hold smaller, more frequent, courses and reduce the “peak load” spike. Also, changing to a faster cadence of smaller spikes would make Pivigo operationally more sustainable and scalable.
3) Revenue: Relying on one big event each year gives a big spike of revenue, which the company then slowly spends out over the year – until the next big event. Each and every event has to be successful, in order for the company to survive the next year. This makes each event a high-risk event for the company. This financial unpredictability limits company long term planning and hiring. Changing to smaller, more frequent, courses make Pivigo’s financial revenue stream healthier, safer and more predictable.
4) Pipeline of applicants: Interested candidates and companies had a once-a-year chance to apply. If they missed the deadline or were turned away because the class was already full, they had to wait an entire year for the next course. Obviously, many did not wait – waiting a year is simply too long. Holding these courses more frequently make it more likely that candidates – and companies – might wait until the next course. Finding a way to increase the cadence of these courses would improve the pipeline for Pivigo.
If Pivigo could find a way to hold these courses more frequently, instead of just once-a-year, then they could accelerate growth of their company. To do this, they had to fix the bottleneck caused by the physical location.
Three weeks ago, Pivigo completed their first ever fully-distributed “virtual” course. It used no physical venue. And it was a resounding success. Just like the typical “in-person” events, teams formed and bonded, good work was done, and complex problems were solved. Pivigo staff, course participants and project sponsors were all happy. Just like usual.
This maps shows everyone’s physical location.
To make this first-ever fully-distributed “virtual” S2DS event successful, we focused on some ideas outlined in my previous presentations here, here and also in my book. Some things I specifically thought were worth highlighting:
1) Keep tools simple Helping people focus on the job-at-hand required removing unnecessary and complex tools. The simpler the tools better. We used zoom, slack and email. After all, people were here to work together on a real-world data science problem, not to learn how to use complex tools.
2) Very crisply organized human processes. None of these people were seasoned “remoties”, so this was all new to all of them. They first met as part of this course. They had to learn how to work together as a team, professionally and as social humans, at the same time as they worked on their project which had to be completed by a fixed deadline.
3) As this was Pivigo’s first time doing this, Kim made a smart decision to explicitly limit size, so there were only 15 people. This gave Kim, Jason and the rest of the staff extra time and space to carefully check in with each of the remote participants and gave everyone best chance of success. Future events will experiment with cohort sizes.
4) Each participant said that they only applied because they could attend “remotely” – even though *none* of them had prior experience working remotely like this. Pivigo were able to interview, and recruit participants who would normally not even apply for the London-based event. The most common reason I heard for not being able to travel to London was disruption to parents with new children – successful applicants worked from their homes on real-world problems, while still being able to take care of their family. The cost of travel to/from England, and the cost of living in London were also mentioned. The need and demand was clearly there. As was their willingness to try something they’d never done before.
5) I note the diversity impact of this new approach. This cohort had a ratio of 26% female / 74% male, while prior in-person S2DS classes typically had a ratio of 35% female / 65% male. This is only one data point, so we’ll watch this with the next S2DS event, and see if there is a trend.
The Virtual S2DS programme was a success. The project outcomes were of similar quality to the campus based events, the participants felt they got a great experience that will help their careers going forward, and, most importantly, the group bonded more strongly than was expected. In a post-event survey, the participants said they would reach out to each other in the future if they had a question or a problem that the network could help with. Interestingly, several of them also expressed an interest in continuing remote working, something they had not considered before.
For Kim and the Pivigo team, this newly-learned ability to hold fully distributed events is game-changing stuff. Physical space is no longer a limiting factor. Now, they can hold more frequent, smaller courses – smoothing down the peaks and troughs of “load”, while also improving the pipelines by making their schedule more timely for applicants. Pivigo are investigating if they could even arrange to run some of these courses concurrently, which would be even more exciting – stay tuned.
Congratulations to Kim and the rest of Pivigo staff. And a big thank you to Adrienne, Aldo, Christine, Prakash, Nina, Lauren, Gordon, Lee, Christien, Rogelio, Sergio, Tiziana, Felipe, Fabio and Mark for quietly helping prove that this approach worked just fine.
John & Kim.
ps: Pivigo are now accepting applications for their next “virtual” event and their next inperson event. If you are an M.Sc./Ph.D. graduate, with a good internet connection, and looking for your first real-world project, apply here: http://www.s2ds.org/. Companies looking for help with data science problems can get in touch with Kim and the rest of the Pivigo team at email@example.com.
28 Nov 2015
Books, Mozilla, Remoties
Earlier this week, just before the US Thanksgiving holidays, we shipped Early Release #3 for my “Distributed” book-in-progress.
Early Release #3 (ER#3) adds two new chapters: Ch.1 remoties trends, Ch.2 the real cost of an office, and many tweaks/fixes to the previous Chapters. There are now a total of 9 chapters available (1,2,4,6,7,8,10,13,15) arranged into three sections. These chapters were the inspiration for recent presentations and blog posts here, here and here.)
ER#3 comes one month after ER#2. You can buy ER#3 by clicking here, or clicking on the thumbnail of the book cover. Anyone who already has ER#1 or ER#2 should get prompted with a free update to ER#3. (If you don’t please let me know!). And yes, you’ll get updated when ER#4 comes out next month.
Please let me know what you think of the book so far. Your feedback get to help shape/scope the book! Is there anything I should add/edit/change? Anything you found worked for you, as a “remotie” or person in a distributed team, which you wish you knew when you were starting? If you were going to setup a distributed team today, what would you like to know before you started?
Thank you to everyone who’s already sent me feedback/opinions/corrections – all changes that are making the book better. I’m merging changes/fixes as fast as I can – some days are fixup days, some days are new writing days. All great to see coming together. To make sure that any feedback doesn’t get lost or caught in spam filters, it’s best to email a special email address (feedback at oduinn dot com) although feedback via twitter and linkedin works also. Thanks again to everyone for their encouragement, proof-reading help and feedback so far.
Now, it’s time to get back to typing. ER#4 is coming soon!
20 Nov 2015
The shift from “building your own datacenter” to “using the cloud” revolutionized how companies viewed internal infrastructure, and significantly reduced the barrier to starting your own fast-growth, global-scale company. Suddenly, you could have instant, reliable, global-scale infrastructure.
(Personally, I dislike the term “cloud” but it’s the easiest vendor-neutral term I know for describing essential infrastructure running on rent-by-the-hour Amazon AWS, Google GCE, Microsoft Azure and others…)
Like any new major change, “the cloud” went through an uphill acceptance curve with resistance from established nay-sayers. Meanwhile, smaller companies with no practical alternatives jumped in with both feet and found that “the cloud” worked just fine. And scaled better. And was cheaper to run. And was faster to setup, so the opportunity-cost was significantly reduced.
Today, of course, “using the cloud” for your infrastructure has crossed the chasm. It is the default. Today, if you were starting a new company, and went looking for funding to build your own custom datacenter, you’d need to explain why you were not “using the cloud”. Deciding to have your own physical data center involves one-time-setup costs as well as ongoing recurring operational costs. Similarly, deciding to have a physical office involves one-time-setup costs as well as ongoing recurring operational costs.
Rethinking infrastructure from the fixed costs of servers and datacenters to rented by the hour “in the cloud” is an industry game changer. Similarly, rethinking the other expensive part of a company’s infrastructure — the physical office — is an industry game changer.
Just like physical datacenters, deciding to setup an office is an expensive decision which complicates, not liberates, the ongoing day-to-day life of your company.
The reality of having an office
It is easy to skip past the “Do we really need an office?” question – and plunge into the mechanics, without first thinking through some company-threatening questions.
What city, and which neighborhood in the city, is the best location for your company office? Sometimes the answer is “near to where the CEO lives”, or “near the offices of our lead VCs”. However, this should include answers to questions like “where will we find most of the talent (people) we plan to hire?” and “where will most of our customers be?”.
What size should your office be? This requires thinking through your hiring plans — not just for today, but also for the duration of the lease — typically 3–5–10 years. The consequences of this decision may be even longer, given how some people do not like relocating! When starting a company, it is very tricky to accurately predict the answers to these questions for multiple years into the future.
Business plans change. Technologies change. Market needs and finances change. Product scope changes. Companies pivot. Brick-and-mortar buildings (usually) stay where they are.
If you convince yourself that your company does need a physical office, setting up and running an office is “non-trivial”. You quickly get distracted by the expensive logistics and operational mechanics of a physical building – instead of keeping focus on people and the shipping product.
You need to negotiate, sign and pay leases. Debate offices-with-doors vs open-plan — and if open-plan, do you want library-quiet, or bull-pen with cross-chatter and music? Negotiate seating arrangements — including the who-gets-a-window-view debate. Construct the actual office-space, bathrooms and kitchens. Pick, buy and install desks, chairs, ping-pong tables and fridges. Set up wifi, security doorbadge systems, printers, phones. Hire staff who are focused on running the physical office, not focused on your product. The list goes on and on. All of these take time, money and most importantly focus. This distracts humans away from the entire point of the company — hiring humans to create and ship product to earn money. And the distraction does not end once the office is built — maintaining and running a physical office takes ongoing time, money and focus.
After your office is up-and-running, you discover the impact this new office has on hiring. You pay to relocate people who would be great additions to your company, but do not live near your new office. You are disappointed by good people turning down job offers because of the location. You have debates about “hiring the best person for the job” vs “hiring the best person for the job who is willing to relocate”. You have to limit hiring because you don’t have a spare desk available. You need to sublease a part of your new office space, because growth plans changed because revenue didn’t go as well as hoped – and now you have unused idle office space costing you money every month.
The benefits of no office
You dedicate more time, money and focus on the people, and the shipping product — simply by avoiding the financial costs, lead-time-delays and focus-distractions of setting up a physical office.
Phrased another way: Distributed teams let you focus the company time and money where it is most important — on the people and the product. After all, it doesn’t matter how fancy your office is unless you have a product that people want to use.
Having no office lets you sidestep a few potentially serious and distracting ongoing problems:
You don’t need to worry about signing a lease for a space that is too small (or too large) for the planned growth of the company. You avoid adding a large recurring cost (a lease) to the company books, which impacts your company’s financial burn rate.
You don’t need to worry if the location of the office helps or hinders future hiring plans. You don’t need to worry about good people turn down your job offers simply because of the office location. You can hire from a significantly larger pool of candidates, so you can hire better and faster then all-in-one-location competitors. For more on this, see .
Even larger companies like Aetna, with established offices, have been encouraging work-from-home since 2005 – because they can hire more people and also because of the money savings from real estate. Last I’ve heard, Aetna was saving $78 million a year by having people work from home. Each year. No wonder Dell and others are now doing the same.
You sidestep human distractions about office layout.
You don’t need to worry about business continuity if the office is closed for a while.
Sidestepping all these distractions helps you (and everyone else in the company) focus attention and money on the people and the product you are building and shipping. This is a competitive advantage over all-in-one-office companies. Important stuff to keep in mind when you ask yourself “Do we really need an office?”
(Versions of this post are on medium.com and also in the latest early release of my “Distributed” book.)
(Photo credit: Woodwards building Vancouver demolition 2 by Tannoy | CC BY-SA 3.0 via Wikimedia Commons)
27 Oct 2015
Books, Mozilla, Remoties
Last week, we rolled out Early Release #2 for my “Distributed” book-in-progress.
EarlyRelease#2 (ER#2) adds two new chapters (Ch.12 one-on-ones and reviews; Ch.14 group socials and work-weeks). There are also a bunch of tweaks and fixes to the previous Chapters 1,5,6,7,9, including grouping related chapters into three sections.
This is one month after ER#1. You can buy ER#2 by clicking here, or clicking on the thumbnail of the book cover. Anyone who already bought ER#1 should get prompted to update to ER#2. (If you don’t please let me know!). And yes, you’ll get updated when ER#3 comes out. For added goodness with an Early Release, your feedback get to help shape/scope the book!
Please let me know what you think of the book so far. Is there anything I should add/edit/change? Anything you found worked for you, as a “remotie” or person in a distributed team, which you wish you knew when you were starting? If you were going to setup a distributed team today, what would you like to know before you started?
Thank you to everyone who’s already sent me feedback/opinions/corrections – all changes that I hope make the book better. To make sure that any feedback doesn’t get lost or caught in spam filters, it’s best to email a special email address (feedback at oduinn dot com) although feedback via twitter and linkedin works also. Thanks again to everyone for their encouragement, proof-reading help and feedback so far.
Now, it’s time to brew more coffee and get back to typing. ER#3 is only a few weeks away!
27 Oct 2015
Last week, I was looped into a discussion around “what companies are remote-friendly”, “how to find a job when you are a remotie” and “what companies have a good remote culture?”. These questions are trickier than one might expect – after all, everyone will have their own definition of “good”.
My metric for deciding if a company has a good remote culture is a variation of the Net Promoter Score:
“if you work remotely for a company, would you recommend the company to a friend?”
If you don’t find a remote job by word-of-mouth referral from a friend, how can you find a job in a remote friendly company? I have been accumulating a few useful sites for remote job seekers, avoiding all the “make $$$ working from home” spam out there. After last week’s discussion, I realized I haven’t seen this list posted in any one place, so thought it worthwhile to post here. Without further ado, in no particular order, here goes:
Some sites seemed more focused on freelancer / short term positions, which were less of interest to me at this time, so I’ve skipped most of those, and only listed a couple here for completeness:
Some of these sites are better then others in terms of how you search for “remote” jobs – for example, I think this needs a little UX love. But something is better then nothing. As I said in my presentation at Cultivate NYC, for job-listings posted on your own company website, simply having “remote welcome” in the job description is a great quick-and-easy start.
Hopefully, people find this list useful. Of course, if there’s any forums / job boards listing remote jobs that you see missing from here, please let me know.
(Thanks to @laurelatoreilly, @jessicard, @qethanm and @hwine for last week’s discussion.)
(Updated to add powertofly.com joduinn 05jan2016)
(Updated to add teleworksusa.com joduinn 05jan2016)
20 Oct 2015
I recently presented about distributed teams and remoties at Cultivate, NYC. The conference was held in the Javits center, in Manhattan, in the midst of the Strata/Hadoop NYC conference and was quite a crowd. The place is so large there were multiple pigeons flying around *inside* the building all week long!
After blogging and presenting about “remoties” and distributed teams for years, this was very different to anything I’ve presented before. I covered a lot of remotie / distributed team topics, including some new research I’m writing in my book.
The slides are here (or click the thumbnail image). I’ll post a copy of the recording once I have it.
Liza Daly, CTO at Safari already posted a great writeup about her presentation, and also on points in my presentation about meetings, so no need for me to repeat those here. Instead, you should read her post. I was encouraged to see that she and I are both tackling some of the same problems, in similar ways.
I’ll touch on three other points that got a lot of followup questions and discussions afterwards:
1) Remote/distributed teams are not a new phenomenon.
I believe humans have been able to work in distributed teams since we learned how to communicate. However, documentation *that* old is hard to find. Examples I *could* find were:
Almost 500 years ago (1519-1522), Magellan’s expedition sailed around the world. This project (“expedition to find spices”) started with “the boss” (The King of Spain) delegated full management authority to Magellan – empowering Magellan to execute staff if he saw fit (He did!) and even start wars on behalf of the King without first checking back to the King asking for approval (He did, which is how he died in battle in the Philippines). The King was being practical, not reckless. After all, it took them 3 years to sail around the world, discovering a new route as they went – so it was simply not practical to stop and wait while a message went back to HQ asking for the King’s permission to do something. To replace dead employees, Magellan “recruited” locals found on the way, training them on the job. Only a handful of the original 270 staff made it back home alive. Only 1 of the 5 original ships made it back home. By contrast, today’s navies are in constant contact with HQ, can request new supplies and trained staff to be quickly delivered anywhere in the world – and they need to ask permission from the boss before starting a war. This change in how they work is because of how they can communicate. Modern day submarines operate under different rules, again because of how they communicate.
Exactly 200 years ago was the Battle of Waterloo. On one day, 18 June 1815, the armies of France, England and Prussia converged for battle. These 191,000 men fought, and in one 24hour day, there were 65,000 dead/wounded/MIA. Using the best communication technologies available at the time, the generals would stand on hilltops overlooking the flat-ish battlefield, and send orders to soldiers across the battlefield by sending foot-messengers, waving flags and playing prearranged coded sequences of notes on their bugles. This scaled better than having all soldiers standing within earshot of the generals, but still had lots of limitations, not least of which was needing soldiers to hold battles in daytime, and difficulty coordinating work out of sight of each other. By contrast, modern soldiers have night vision equipment, as well as personal encrypted radios so they can communicate quickly and safely with coworkers, even if they are out of sight from each other, as well as “back to HQ” for supplies and reinforcements.
In each of these examples, the use of new communication tools radically changed how these organizations operate.
“Changing how we communicate lets us change how we work”(click to tweet)
Similarly, we have new communication tools in business. The ability to talk with people far away (telephone), the ability to write on a portable device (portable typewriter), and the ability to create moving pictures (reel-to-reel movie cameras) have gone from “expensive new technologies” to routinely included for free in each laptop and cellphone. Given the possibilities these new technologies allow, and their widespread availability, why do so many companies still arrange their organizations like this when their customers are global and look like this?
Literally everyone attending Cultivate NYC was hiring. Not one exception. And everyone was having a hard time hiring in today’s super-competitive environment. Yet, as best as I could see in the large hall, very few said “remote welcome” on the job description.
This is important because recent changes to society are also impacting how companies should recruit.
There are no more “hired for life” jobs, which makes people less willing to relocate for each job as they progress through their career. This year (2015) in the USA, there is a generational change, where the biggest segment of the workforce is now Millennials – people who grew up thinking streaming video on cellphones is “normal”! There is widespread broadband availability, making it possible to afford internet connectivity at home as-good-as-or-better than your connectivity at the office. Forrester estimated that in 2009, there were ~34million work-from-home people, and that by next year (2016), it will be ~63million people working from home… with the rate continuing to rise as broadband rollout continues.
To make matters worse for hiring managers, some qualified people who have the right skills won’t even apply for the job, simply because they know that if they got the job, they cannot accept it. That’s worth repeating. They have the skills, but they won’t even give you the chance to interview them, simply because if they got the job, they cannot reliably get to/from the office… so they don’t even apply. So, if you are a hiring manager, ask yourself:
Are you “Hiring the best person for the job”? Or are you “Hiring the best person for the job who is willing to relocate”?
If you want to “hire the best person for the job (regardless of where they live)”, then explicitly put “remote welcome” in your job description. This will let you hire better candidates, hire faster and give you a business advantage over your all-in-one-location competitors.
3) Disaster Planning
The week before Cultivate NYC, the Pope visited NYC. This caused NYPD to plan widespread street shutdowns for security during various site visits, prompting NYC Mayor de Blasio to encourage people to work from home for the week.
“If someone has the option to stay home, work from home, that’s a great choice…”. de Blasio, Mayor, NYC.
Similarly, while Hurricane Sandy in NYC caused death and destruction, many people were fine, simply stuck at home, unable to get to the office because subways was flooded. Their home was fine. Their office was fine. They just could not *get* to the office. And because their work habits assumed being in the office, they were unable to work effectively from home.
It’s one thing to work-from-home, on a part-time basis, choosing to specifically only do solo-work / focus-work. Instead, I asked people could they do their “normal job” (meetings, conference calls, pair-programming, escalations, interacting with co-workers, etc) while working from home? Most people actually cannot do this – because they are self-trained to go to office to do this in person. By contrast, experienced remote / distributed teams do this just fine – they are using cheap, familiar tools and crisp human processes which they practice daily.
I left everyone with a one day experiment: If you left this conference today, and went home with just the contents of your bag, could you do your normal job tomorrow (for just one day) if the office was unexpectedly closed tomorrow? If not, don’t worry – its just a one day experiment. Learn why, come into the office the next day, fix it, and try again the following week.
“We build fault tolerant systems. We need fault tolerant companies.”(click to tweet)
There were lots of note-taking throughout, and it was very exciting to see the wide-spread interest – at the conference as well as in the followup back-to-back meetings throughout the rest of the week.
ps: I also want to give a shout out of thanks to Jason and Laurel for the thousand-and-one logistical details they worked through, literally all the time. Amazing to see.
08 Oct 2015
Books, Mozilla, Release Engineering, Remoties
My previous post described how O’Reilly does rapid releases, instead of waterfall-model releases, for book publishing. Since then, I’ve been working with the folks at O’Reilly to get the first milestone of my book ready.
As this is the first public deliverable of my first book, I had to learn a bunch of mechanics, asking questions and working through many, many details. Very time consuming, and all new-to-me, hence my recent silence. The level of detailed coordination is quite something – especially when you consider how many *other* books O’Reilly has in progress at the same time.
One evening, while in the car to a social event with friends, I looked up the “not-yet-live” page to show to friends in the car – only to discover it was live. Eeeeek! People could now buy the 1st milestone drop of my book. Exciting, and scary, all at the same time. Hopefully, people like it, but what if they don’t? What if I missed an important typo in all the various proof-reading sessions? I barely slept at all that night.
In O’Reilly language, this drop is called “Early Release #1 (ER#1)”. Now that ER#1 is out, and I have learned a bunch about the release mechanics involved, the next milestone drop should be more routine. Which is good, because we’re doing these every month. Oh, and like software: anyone who buys ER#1 will be prompted to update when ER#2 is available later in Oct, and prompted again when ER#3 is available in Nov, and so on.
You can buy the book-in-progress by clicking here, or clicking on the thumbnail of the book cover. And please, do let me know what you think – Is there anything I should add/edit/change? Anything you found worked for you, as a “remotie” or person in a distributed team, which you wish you knew when you were starting? If you were going to setup a distributed team today, what would you like to know before you started?
To make sure that any feedback doesn’t get lost or caught in spam filters, I’ve setup a special email address (feedback at oduinn dot com) although I’ve already been surprised by feedback via twitter and linkedin. Thanks again to everyone for their encouragement, proof-reading help and feedback so far.
Now, it’s time to brew more coffee and get back to typing.
15 Sep 2015
Books, Mozilla, Release Engineering, Remoties
As a release engineer, I’ve designed and built infrastructure for companies that used the waterfall-model and for companies that used a rapid release model. I’m now writing a book, so recently I have been looking at a very different industry through this same RelEng lens.
In days of old, here’s a simplistic description of what typically happened. Book authors would write their great masterpiece on their computer, in MSWord/Scrivener/emacs/typewriter/etc. Months (or years!) later, when the entire book was written, the author would deliver the “complete” draft masterpiece to the book company. Editors at the book company would then start reading from page1 and wade through the entire draft, making changes, fixing typos, looking for plot holes, etc. Sometimes they could make these corrections themselves, sometimes the changes required sending the manuscript back to the author to rewrite portions. Later, external reviewers would also provide feedback, causing even more changes. If all this took a long time, sometimes the author had to update the book with new developments to make sure the book would be up-to-date when it finally shipped. Tracking all these changes was detailed, time-pressured and chaotic work. Eventually, a book would finally go to press. You could reduce risk as well as simplify some printing and shipping logistics by having bookshops commit to pre-buy bulk quantities – but this required hiring a staff of sales reps selling the unwritten books in advance to bookstores before the books were fully written. Even so, you could still over/under estimate the future demand. And lets not forget that once people start reading the book, and deciding if they like it, you’ll get word-of-mouth, book reviews and bestseller-lists changing the demand unpredictably.
If people don’t like the book, you might have lots of copies of an unsellable book on your hands, which represents wasted paper and sunk costs. The book company is out-of-pocket for all the salaries and expenses from the outset, as well as the unwanted printed books and would hope to recover those expenses later from other more successful books. Similar to the Venture Capital business model, the profits from the successful books help recoup the losses from the other unprofitable books.
If people do like the book, and you have a runaway success, you can recoup the losses of the other books so long as you avoid being out-of-stock, which could cause people to lose interest in the book because of back order delays. Also, any errors missed in copy-editing and proofreading could potentially lead to legal exposure and/or forced destruction of all copies of printed books, causing further back order delays and unexpected costs. Two great examples are The Sinner’s Bible and the Penguin cook book.
This feels like the classic software development “waterfall” model.
By contrast, one advantage with the rapid release model is that you get quick feedback on the portions you have shipped so far, allowing you to revisit and quickly adjust/fix as needed… even while you are still working on completing and shipping the remaining portions. By the time you ship the last portion of the project, you’ve already adjusted and fixed a bunch of surprise gotchas found in the earlier chunks. By the time you ship your last portion, your overall project is much healthier and more usable. After all, for the already shipped portions, any problems discovered in real-world-use have already been fixed up. By comparison, a project where you write the same code for 18-24 months and then publish it all at the same time will have you dealing with all those adjustments/fixes for all the different chunks *at the same time*. Delivering smaller chunks helps keep you honest about whether you are still on schedule, or let you quickly see whether you are starting to slip the schedule.
The catch is that this rapid release requires sophisticated automation to make the act of shipping each chunk as quick, speedy and reliable as possible. It also requires dividing a project into chunks that can be shipped separately – which is not as easy to do as you might hope. Doing this well requires planning out the project carefully in small shippable chunks, so you can ship when each chunk is written, tested and ready for public consumption. APIs and contractual interfaces need to be understood and formalized. Dependencies between each chunk needs to figured out. Work estimates for writing and testing each module need to be guessed. A calendar schedule is put together. The list goes on and on… Aside: You probably should do this same level of planning also for waterfall model releases, but its easy to miss hidden dependencies or impact of schedule slips until the release date when everything was supposed to come together on the day.
So far, nothing new here to any release engineer reading this.
One of the (many) reasons I went with O’Reilly as the publisher for this book was their decision to invest in their infrastructure and process. O’Reilly borrowed a page from Release Engineers and invested in automation to switch their business from a waterfall model to a rapid release model. This change helps keep schedules realistic, as schedule slips can be spotted early. It helps get early feedback which helps ship a better final product, so the ratio of successful book vs unprofitable books should improve. It helps judge demand, which helps the final printing production planning to reduce cost wasting with less successful books, and to improve profits and timeliness for successful books. This capability is a real competitive advantage in a very competitive business market. This is an industry game changer.
When you know you want “rapid release for books”, you discover a lot of tools were already there, with a different name and slightly-different-use-cases. Recent technologies advances help make the rest possible. The overall project (“table of contents”). The breaking up of the overall project into chunks (“book chapters”). Delivering usable portions as they are ready (print on demand, electronic books + readers, online updates). Users (“readers”) who get the electronic versions will get update notices when newer versions become available. At O’Reilly, the process now looks like this:
Book authors and editors agree on a table of contents and an approximate ship date (write a project plan with scope) before signing contracts.
Book authors “write their text” (commit their changes) into a shared hosted repository, as they are writing throughout the writing creative process, not just at the end. This means that O’Reilly editors can see the state of the project as changes happen, not just when the author has finished the entire draft manuscript. It also means that O’Reilly reduces risk of catastrophic failure if an author’s laptop crashes, or is stolen without a backup.
A hosted automation system reads from the repository, validates the contents and then converts that source material into every electronic version of the book that will be shipped, including what is sent to the print-on-demand systems for generating the ink-on-paper versions. This is similar to how one source code revision is used to generate binaries for different deliverables – OSX, Windows, linux, iOS, Android,…
O’Reilly has all the automation and infrastructure you would expect from a professional-grade Release Engineering team. Access controls, hosted repos, hosted automation, status dashboards, tools to help you debug error messages, teams to help answer any support questions as well as maintain and improve the tools. Even a button that says “Build!”!!
The only difference is that the product shipped from the automation is a binary that you view in an e-reader (or send to a print-on-demand printing press), instead of a binary that you invoke to run as an application on your phone/desktop/server. With this mindset, and all this automation, it is no surprise that O’Reilly also does rapid releases of books, for the same reasons software companies do rapid releases. Very cool to see.
I’ve been thinking about this a lot recently because I’m now putting together my first “early release” of my book-in-progress. More info on the early release in the coming days when it is available. As a release engineer and first time author, I’ve found the entire process both totally natural and self-evident and at the same time a little odd and scary. I’d be very curious to hear what people think… both of the content of the actual book-in-progress, and also of this “rapid release of books” approach.
Meanwhile, it’s time to refill my coffee mug and get back to typing.
02 Sep 2015
Mozilla, Release Engineering
The USENIX Release Engineering Summit 2015 (“URES15”) is quickly approaching – this time it will be held in November in Washington DC along with LISA2015. To register to attend URES15, click here or on the logo.
Given the two (!) great URES conferences were last year, I expect URES15 to be fun and informative and very-down-to-earth practical all at the same time. One of the great things about these RelEng conferences is that you get to hear what did, and did not, work for others working the same front lines – all in a factual constructive way. Sharing important ideas help raise the bar for everyone, so every one of these feel priceless. It is also a great way to meet many other seasoned people in this niche part of the computer business,
If you are a Release Engineer, Site Reliability Engineer, Production Operations, deal with DevOps culture or are someone who keeps wanting to find a better way to reliably ship better quality code, you should attend! You’ll be glad you did!
Note: If you have a project that you worked on which others would find informative, you can submit a proposal here. The deadline for proposals is Friday 04sep2015.
See you there!