If you don’t care about Talos performance results, or Talos hardware, stop reading now!

In the last dev platform meeting, I mentioned we’d be doing a lot of Talos recalibration work in early January 2010, and promised updates as we figured out the plan. We’re still working out some details, but here’s what we know so far.

Talos runs performance tests on mac mini hardware. The 1.83GHz minis we use today were all spec’d 2+ years ago, when we migrated from the previous 1.66GHz minis. During 2009, we kept buying and powering up more 1.83GHz minis to keep up with increasing demand of:

• having more developers doing more frequent checkins
• adding more talos suites to run per checkin and
• running Talos on additional new OS.

However, these 1.83GHz minis have long since been discontinued by Apple, so we can’t buy any more. The “trade-in” program we ran in November got us another ~42 minis, bringing us up to 159 minis – barely enough to support the extra load needed through the FF3.6 release cycle.

At this point, our only option is to buy a whole larger set of newer spec minis, and recalibrate Talos performance results on those newer spec minis. The new spec minis are: 2.26GHz minis, with 2GB RAM and 160GB disk. We’ll recalibrate the Talos results by running the new minis concurrently with the old minis for a week or two. After we verify there are no inconsistencies in the results, we can then power off the old Talos minis, and recycle them for use elsewhere.

To avoid possible confusion about performance results, this is being done soon after FF3.6 ships and before FF3.7 works ramps up. In the meanwhile, we’re now starting the behind-the-scenes unboxing, racking, networking and configuration work.

This means a few things:

• changing Talos hardware means re-calibrating all Talos results, including results for a list of selected important historical milestones. We’re still figuring that list out.
• its important that all talos machines, across all branches, all OS, are identical, so developers can do like-with-like comparison of results across the board. This means we need to replace all the minis at one time.
• for OSX, we’ll have to change from using OSX 10.5.2 to using OSX 10.5.8. This is because 10.5.2 does not work on the new minis. This OS change *might* modify performance results so we held off doing it until this time.
• these new minis will be able to support OSX 10.6 when we add 10.6 support to Talos.
• we’re working with IT to figure out how to rack,power,network these machines here in 650castro, as the MPT colo is already full.
• we’re also using this time to move these machines from the QA network (where Talos was originally created) over to the Build network along with the rest of the build and unittest systems. We’ll also bring the logins, pswds, etc in sync to make them all easier to maintain.

Obviously, there are lots of details in this project to be careful of. If you think we’ve missed something please let us know. I’m cross-posting this to a couple of newsgroups to make sure this important upcoming change is noticed. However, please post followups in dev.planning or in bug#537065.

Thanks for reading this far, and take care
John.

(Maybe this is old news to everyone else, but I stumbled across this article recently, and found it quite interesting.)
http://money.cnn.com/magazines/business2/business2_archive/2006/07/01/8380230/index.htm

The term “mighty micro multinational” was new to me, but after reading this article from 2006, I think the description fits Mozilla fairly well. Basically, as I understand it, the writer makes the distinction between:

• traditional companies which are based in one location until forced to react to change. Examples include ExxonMobile and IBM. This change can be because of acquisition (buying a company in a different city/state/country), offshoring (moving “less critical” work to cheaper locations, while keeping “critical” work in HQ), real estate (outgrow physical HQ, and cant find adjacent office space in the neighborhood), or a whole range of other external factors.
• a “new” category of companies which are multi-location, and multi-national, from their very inception. Literally they are location agnostic. And they do this by choice. Examples include Skype, MySQL and VistaPrint; interestingly enough, Mozilla was not mentioned in the article at all.

The difference between these two types of companies is important in internal coordination, communication, attitudes to different cultures, and most importantly hiring. Being location agnostic means able to hire “the best person for the job”, not just “the best person for the job who lives nearby, or is willing to relocate for the job”. Quote: “…it doesn’t matter much where [the developer] is physically as long as he has a broadband connection.” To handle the extra complexity, the company is held together by having people who are naturally able to bridge the different cultures, keep all the communication flowing across time zones, and are skilled in detangling communication snafus whenever they arise. He calls that role “the magic expatriate”, and as far as I can tell, Mozilla has an amazing collection of magic expatriates!

Software engineers scattered around the world each with a laptop, VOIP headset and IM/IRC sounds slightly futuristic in the business-focused article, yet its normal life in Mozilla for years now. And its very cool.

(As an exercise to the reader, just try saying “Mozilla the mighty micro multinational” 3 times in a row quickly!)

The development environment at Mozilla is fairly complex. As a new Mozilla developer, it can be tricky just learning where to land code changes, and figuring out if your changes landed ok and worked as you hoped! If you think back to your first day contributing code changes at Mozilla – is there anything you wish was explained to you on the first day, which would have helped you get up to speed quicker and made your life easier?

Until recently we’ve been relying on MDC, some scattered wiki docs, and lots of word-of-mouth – seasoned veterans mentoring newcomers until the newcomers can, in turn, mentor other newcomers. But this is tough to scale as more and more people start contributing. Also, as we modify, streamline and scale up RelEng infrastructure, we occasionally see veterans teach newcomers how things *used* to be, not how things are *now* !

Here’s a brown bag that we hope will help with all that.

We did a trial run of this during the AllHands, with some pre-existing Mozilla developers, and tweaked it a bit based on comments. Once we’ve been through a few iterations, we’ll put together a video clip but for now, you’ll have to imagine my voice doing the talkover voice. Now its time to find some real newcomers and see if it makes sense… so, what do you think?

update: we’ve done this brownbag a few times now since the AllHands, and each time, we tweak it further based on comments and questions asked. There might still be further changes, but at this point, it was worthwhile updating the PDF in this blogpost. joduinn 15feb2010, again 14jun2010 and again 17dec2010.

Summary:

• Overall, total numbers approx the same as last month. However, it is interesting to see fewer checkins on mozilla-central, but a significant increase in mozilla-191, mozilla-192 checkins. I suspect this is caused by increased focus on security releases and the upcoming FF3.6 release. The increased load on try is probably also related to this.
• The numbers for this month are:
• 1,675 code changes to our mercurial-based repos, which triggered:
• 20,029 build jobs, or ~84 jobs per hour.
• 40,421 unittest jobs, or ~56 jobs per hour.
• 41,467 talos jobs, or ~58 talos jobs per hour.
• 18th November was our third busiest day of the year, with 103 checkins. For comparison, this high level was only exceeded on 22nd Sept (116 checkins) and 20th May (108 checkins).
• We enabled new Talos suites, and also disabled some unittest suites at different times during the month. For simplicity, I’ve ignored those changes for now, and will include in next month’s data.
• We are still not tracking down any l10n repacks, nightly builds, release builds or any “idle-timer” builds.

Here’s how it looks compared to the year so far:

Detailed breakdown is :

Here’s how the math works out:

The types of build, unittest and performance jobs triggered by each individual push are best described here.