HDP2.1 shipped on 22apr2014.

This was the first significant feature release shipped since I joined Hortonworks at the start of the year. There’s lots of interesting new features, and functionality in this HDP2.1 release – already well covered by others in great detail here. Oh, and of course, you can
download it from here.

In this post, I’ll instead focus on some of the behind-the-scenes mechanics. There were lots of major accomplishments in this release, but the ones that really stood out to me were:

1) sim-ship windows and linux.
This was the first HDP release where all OS were built from the same changeset and shipped at the same time. Making this happen was a hectic first priority in January. As well as the plumbing/mechanics within RelEng, it also took lots of coordination changes across different groups within Hortonworks to make this happen. The payoff on this was great. We sim-shipped, which is great and massively important for HWX as a company. Even more importantly, we set things up so we could sim-ship for every HDP2.1-and-above release going forward… and we proved it by sim-shipping the quick followup HDP2.1.2.0 release on 02may2014.

2) adding 5 new components.
HDP2.1 contained 17 components, compared to HDP 2.0 (with 12 components) and HDP 1.3 (with 10 components), making HDP2.1 the largest growth of components ever?!? Oh, and in addition to the new components, every one of the 12 pre-existing components were also significantly updated to newer versions. That meant that each required significant new integration work, new installers on all supported OS (…remember the “sim-ship” goal?). Oh, and we were to ship all this new functionality at the fastest cadence yet.

3) improving support for other trains.
In January, we were learning how to support 3 active trains of code: supporting 1.3 and 2.0 maintenance work, while also building out infrastructure for 2.1 new-product-development-work… even while the 2.1 development work was in progress, which obviously complicated things for developers. Today, we’re supporting 4 active trains: maintenance work for 1.3, 2.0 and 2.1, as well as the 2.2 new-product-development-work. This time, the 2.2 infrastructure was built out and live before developers finished working on 2.1… enabling the developers! Things are not perfect yet, by any means, but today (with 4 trains) feels calmer and more organized then earlier this year (with “only” 3 trains).

All great improvements to see up close, and all important to us as we scale. Big thanks to everyone for their help… and do stay tuned for even more improvements already underway.

John.