Recalibrating Talos performance results on newer hardware

If you don’t care about Talos performance results, or Talos hardware, stop reading now!

In the last dev platform meeting, I mentioned we’d be doing a lot of Talos recalibration work in early January 2010, and promised updates as we figured out the plan. We’re still working out some details, but here’s what we know so far.

Talos runs performance tests on mac mini hardware. The 1.83GHz minis we use today were all spec’d 2+ years ago, when we migrated from the previous 1.66GHz minis. During 2009, we kept buying and powering up more 1.83GHz minis to keep up with increasing demand of:

  • having more developers doing more frequent checkins
  • adding more talos suites to run per checkin and
  • running Talos on additional new OS.

However, these 1.83GHz minis have long since been discontinued by Apple, so we can’t buy any more. The “trade-in” program we ran in November got us another ~42 minis, bringing us up to 159 minis – barely enough to support the extra load needed through the FF3.6 release cycle.

At this point, our only option is to buy a whole larger set of newer spec minis, and recalibrate Talos performance results on those newer spec minis. The new spec minis are: 2.26GHz minis, with 2GB RAM and 160GB disk. We’ll recalibrate the Talos results by running the new minis concurrently with the old minis for a week or two. After we verify there are no inconsistencies in the results, we can then power off the old Talos minis, and recycle them for use elsewhere.

To avoid possible confusion about performance results, this is being done soon after FF3.6 ships and before FF3.7 works ramps up. In the meanwhile, we’re now starting the behind-the-scenes unboxing, racking, networking and configuration work.

This means a few things:

  • changing Talos hardware means re-calibrating all Talos results, including results for a list of selected important historical milestones. We’re still figuring that list out.
  • its important that all talos machines, across all branches, all OS, are identical, so developers can do like-with-like comparison of results across the board. This means we need to replace all the minis at one time.
  • for OSX, we’ll have to change from using OSX 10.5.2 to using OSX 10.5.8. This is because 10.5.2 does not work on the new minis. This OS change *might* modify performance results so we held off doing it until this time.
  • these new minis will be able to support OSX 10.6 when we add 10.6 support to Talos.
  • we’re working with IT to figure out how to rack,power,network these machines here in 650castro, as the MPT colo is already full.
  • we’re also using this time to move these machines from the QA network (where Talos was originally created) over to the Build network along with the rest of the build and unittest systems. We’ll also bring the logins, pswds, etc in sync to make them all easier to maintain.

Obviously, there are lots of details in this project to be careful of. If you think we’ve missed something please let us know. I’m cross-posting this to a couple of newsgroups to make sure this important upcoming change is noticed. However, please post followups in dev.planning or in bug#537065.

Thanks for reading this far, and take care

One thought on “Recalibrating Talos performance results on newer hardware