Updating Mozilla’s Update server

(Followup from Tuesday’s Platform meeting as well as bhearsum’s blog post#1 and blog post#2.)

Updating our users is something we do very carefully.

Updating *how* we update our users is something we do very *very* VERY carefully.

On Monday 30sept, nightly users on mozilla-central will be served updates from the new AUS server. Users don’t need to do anything different – it should “just work”. (Of course, if you see any problems with updating, we’d like to hear about it… please file a bug!)

Users on aurora, beta and release channels are *not* being switched over yet. All in good time.


Additional context:
Update servers have to work accurately, securely, consistently and at scale. One of the big scary things that any software company has to do is update the system by which all of the company’s users are served updates. The same is true here at Mozilla. After all, if anything goes wrong, we can’t physically go around to each user’s house/office, everywhere around the world, to fix the problem caused by a bad update to their Firefox or Thunderbird installation. (aside: Mobile-app-only software companies avoid this, by manually uploading to Apple/Google stores, and relying on Apple/Google to do the update distribution for them. This helps small companies that *only* ship mobile apps keep their users up-to-date, but is not without risk. Offtopic, so watch for separate blog post.)

Since before 2007, Mozilla is currently serving all updates using AUS (Application Update Service), an update server that was originally written by @Morgamic. Even when I joined Mozilla in 2007, there were ongoing jaded discussions about a repeatedly deferred project called “AUS2 – The next big rewrite”. All to say, the current production AUS code has served (all of Mozilla’s users and Mozilla’s RelEng!) well, even so many years after it was originally written and put into production. Big hat tip to @Morgamic.

As we scaled up the rest of RelEng infrastructure, some early design decisions that used to be fine became trickier for us. Manually updating the live code on live production server with different version numbers was ok-to-do when Mozilla was back on the old-traditional-slower-release-cadence… but became a fragile concern when we moved to rapid-release-cadence. New requirements like being able to throttle specific operating systems at different update rates. New requirements like custom updates users for users of specific custom builds. New requirements like dropping support for specific sub-variants of an OS…

Finally, we found time last year and again earlier this year to dedicate people in RelEng to make concrete progress on this. As we got interrupted by other big externally-facing projects, we’d suspend work until later. Then resume work a few months later. Then suspend again. Then resume again. In so doing, the name and requirements kept evolving AUS2…AUS3…AUS4…BALROG. More details in bug#832454 and bug#583244. RelEng (and some brave volunteers) have been dogfooding for a while, using Balrog to get updates for nightly mozilla-central builds. We’re ready to roll this out to the next most adventurous population – nightly users on mozilla-central.

This has been the result of years of planning, and quiet focused coding by bhearsum, rail, catlee, nthomas over multiple quarters. This is a ReallyBigDeal ™ – for RelEng, for all Mozilla developers, and for Mozilla’s users – so it is exciting to see come into production.

254 compute hours per checkin

The rate of checkins continues to rise rapidly. The volume of builds+tests run per checkin is also increasing rapidly. The combination of these two increases multiplied together is an impressive rate of growth.

How many hours of builds and tests do we run per commit?

  • 254 compute hours = ~10.5 compute *days* (sep2013)
  • 137 compute hours = ~5.7 compute *days* (aug2012)
  • 110 compute hours = ~4.6 compute *days*(jan2012)
  • ~40 compute hours = ~1.6 compute *days*(2009)

10.5 compute *days* of builds and tests per checkin is a lot – especially when we average one checkin every ~6 minutes last month, and handle a checkin every 4minutes during peak times. Put another way, that’s an average of ~6.9 compute years of builds and tests being run per day, or 213 compute *years* for the month of August. Mind boggling.

If you look at the “wait times” posts in dev.tree-management, you’ll see that systems we run on AWS are scaling nicely to handle the load, and we continue to move production load over to AWS for nativefennec / B2G desktop builds / linux desktop …. and then recycle existing inhouse hardware to help with inhouse capacity for win32 and win64.

Some of this growth is good (new builds, new test suites).
But some of growth is bad (consistently failing tests; obsolete tests; builds that are no longer needed)… and this slows us all down.

You can help.

  1. When you push to tryserver, choose wisely. If you need “all”, do it. But ask yourself… do you really need all?
  2. If you know of any build/test that is no longer needed, let us know.
  3. If you know of any build/test being hidden on tbpl.m.o because it constantly fails let us know or file a bug. We’ll disable the job to stop wasting CPU time, which will help everyone else’s jobs get processed more quickly. As needed, we’ll also file bugs to track fixing the problem and then reenable once the build/test is running green. While any one given build or test suite might not seem like a lot of time to save, we run all these jobs literally thousands of times per day.

    Every little saving helps makes things even faster for everyone!

    Thank you!
    John.
    ===========================================
    List of builds and tests run (with duration of job in seconds)
    Android 2.2 Armv6 Tegra mozilla-central opt test crashtest 1205.62037037
    Android 2.2 Armv6 Tegra mozilla-central opt test jsreftest-1 1961.08333333
    Android 2.2 Armv6 Tegra mozilla-central opt test jsreftest-2 1996.19626168
    Android 2.2 Armv6 Tegra mozilla-central opt test jsreftest-3 2043.25233645
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-1 1039.10909091
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-2 912.092592593
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-3 1291.40740741
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-4 1066.77570093
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-5 979.252336449
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-6 1103.2
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-7 975.355140187
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-8 1050.5
    Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-1 2538.24299065
    Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-2 2849.25
    Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-3 2632.22429907
    Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-4 2589.43925234
    Android 2.2 Armv6 Tegra mozilla-central opt test robocop-1 1494.53271028
    Android 2.2 Armv6 Tegra mozilla-central opt test robocop-2 2251.78504673
    Android 2.2 Armv6 Tegra mozilla-central opt test xpcshell 2289.25688073
    Android 2.2 Armv6 mozilla-central build 4112.26470588
    Android 2.2 Debug mozilla-central build 4612.8627451
    Android 2.2 Tegra mozilla-central opt test crashtest 1198.16666667
    Android 2.2 Tegra mozilla-central opt test jsreftest-1 1905.94545455
    Android 2.2 Tegra mozilla-central opt test jsreftest-2 1798.31192661
    Android 2.2 Tegra mozilla-central opt test jsreftest-3 2102.10091743
    Android 2.2 Tegra mozilla-central opt test mochitest-1 1014.01834862
    Android 2.2 Tegra mozilla-central opt test mochitest-2 893.449541284
    Android 2.2 Tegra mozilla-central opt test mochitest-3 1270.65454545
    Android 2.2 Tegra mozilla-central opt test mochitest-4 981.158878505
    Android 2.2 Tegra mozilla-central opt test mochitest-5 938.247706422
    Android 2.2 Tegra mozilla-central opt test mochitest-6 1128.26126126
    Android 2.2 Tegra mozilla-central opt test mochitest-7 988.91588785
    Android 2.2 Tegra mozilla-central opt test mochitest-8 997.898148148
    Android 2.2 Tegra mozilla-central opt test plain-reftest-1 2527.41121495
    Android 2.2 Tegra mozilla-central opt test plain-reftest-2 2815.85185185
    Android 2.2 Tegra mozilla-central opt test plain-reftest-3 2623.35514019
    Android 2.2 Tegra mozilla-central opt test plain-reftest-4 2582.18691589
    Android 2.2 Tegra mozilla-central opt test robocop-1 1542.2293578
    Android 2.2 Tegra mozilla-central opt test robocop-2 2233.76576577
    Android 2.2 Tegra mozilla-central opt test xpcshell 2308.20720721
    Android 2.2 Tegra mozilla-central talos remote-tcanvasmark 886.648148148
    Android 2.2 Tegra mozilla-central talos remote-tp4m_nochrome 1147.54205607
    Android 2.2 Tegra mozilla-central talos remote-trobocheck2 983.117117117
    Android 2.2 Tegra mozilla-central talos remote-trobopan 1006.74074074
    Android 2.2 Tegra mozilla-central talos remote-troboprovider 1280.57009346
    Android 2.2 Tegra mozilla-central talos remote-ts 705.117647059
    Android 2.2 Tegra mozilla-central talos remote-tspaint 688.58974359
    Android 2.2 Tegra mozilla-central talos remote-tsvg 751.75
    Android 2.2 Tegra mozilla-central talos remote-tsvgx 1428.88073394
    Android 2.2 mozilla-central build 4251.46078431
    Android 2.2 no-ionmonkey Tegra mozilla-central opt test jsreftest-1 2015.42045455
    Android 2.2 no-ionmonkey Tegra mozilla-central opt test jsreftest-2 1812.60227273
    Android 2.2 no-ionmonkey Tegra mozilla-central opt test jsreftest-3 2138.88636364
    Android 2.2 no-ionmonkey mozilla-central build 3993.67647059
    Android 4.0 Panda mozilla-central opt test crashtest 1413.98843931
    Android 4.0 Panda mozilla-central opt test jsreftest-1 1584.75842697
    Android 4.0 Panda mozilla-central opt test jsreftest-2 1582.75147929
    Android 4.0 Panda mozilla-central opt test jsreftest-3 1566.4566474
    Android 4.0 Panda mozilla-central opt test mochitest-1 1216.17679558
    Android 4.0 Panda mozilla-central opt test mochitest-2 1098.2754491
    Android 4.0 Panda mozilla-central opt test mochitest-3 1410.21387283
    Android 4.0 Panda mozilla-central opt test mochitest-4 1197.27906977
    Android 4.0 Panda mozilla-central opt test mochitest-5 1212.03888889
    Android 4.0 Panda mozilla-central opt test mochitest-6 1166.41489362
    Android 4.0 Panda mozilla-central opt test mochitest-7 1209.33532934
    Android 4.0 Panda mozilla-central opt test mochitest-8 1140.14754098
    Android 4.0 Panda mozilla-central opt test mochitest-gl 1116.70414201
    Android 4.0 Panda mozilla-central opt test robocop-1 1288.49710983
    Android 4.0 Panda mozilla-central opt test robocop-2 1739.20833333
    Android 4.0 Panda mozilla-central opt test robocop-3 1587.77192982
    Android 4.0 Panda mozilla-central opt test xpcshell 1989.28021978
    Android 4.0 Panda mozilla-central talos remote-tcanvasmark 2836.71018277
    Android 4.0 Panda mozilla-central talos remote-tp4m_nochrome 1484.12068966
    Android 4.0 Panda mozilla-central talos remote-trobocheck2 1168.93292683
    Android 4.0 Panda mozilla-central talos remote-trobopan 1077.8969697
    Android 4.0 Panda mozilla-central talos remote-troboprovider 1370.75903614
    Android 4.0 Panda mozilla-central talos remote-ts 850.294573643
    Android 4.0 Panda mozilla-central talos remote-tspaint 933.782608696
    Android 4.0 Panda mozilla-central talos remote-tsvg 960.408333333
    Android 4.0 Panda mozilla-central talos remote-tsvgx 1608.24
    Android 4.2 x86 Emulator mozilla-central opt test androidx86-set-1 3556.85074627
    Android 4.2 x86 Emulator mozilla-central opt test androidx86-set-2 2033.921875
    Android 4.2 x86 mozilla-central build 4230.7254902
    Android Armv6 Tegra 250 mozilla-central opt test crashtest 1199.5
    Android Armv6 Tegra 250 mozilla-central opt test jsreftest-1 1950.375
    Android Armv6 Tegra 250 mozilla-central opt test jsreftest-2 1819.0625
    Android Armv6 Tegra 250 mozilla-central opt test jsreftest-3 2357.21875
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-1 1026.875
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-2 893.21875
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-3 1279.72727273
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-4 1037.84375
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-5 986.28125
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-6 1083.5625
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-7 1004.09090909
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-8 1019.36363636
    Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-1 2711.9375
    Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-2 3252.09375
    Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-3 2679.6875
    Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-4 2923.84375
    Android Armv6 Tegra 250 mozilla-central opt test robocop-1 1455.78125
    Android Armv6 Tegra 250 mozilla-central opt test robocop-2 2155.1875
    Android Armv6 Tegra 250 mozilla-central opt test xpcshell 2191.09375
    Android Armv6 mozilla-central build 4099.38709677
    Android Debug mozilla-central build 4822.32258065
    Android Tegra 250 mozilla-central opt test crashtest 1161.53125
    Android Tegra 250 mozilla-central opt test jsreftest-1 1902.25
    Android Tegra 250 mozilla-central opt test jsreftest-2 1873.1875
    Android Tegra 250 mozilla-central opt test jsreftest-3 2000.6875
    Android Tegra 250 mozilla-central opt test mochitest-1 978.96875
    Android Tegra 250 mozilla-central opt test mochitest-2 841.90625
    Android Tegra 250 mozilla-central opt test mochitest-3 1245.6875
    Android Tegra 250 mozilla-central opt test mochitest-4 983.1875
    Android Tegra 250 mozilla-central opt test mochitest-5 956.84375
    Android Tegra 250 mozilla-central opt test mochitest-6 1055.65625
    Android Tegra 250 mozilla-central opt test mochitest-7 960.6875
    Android Tegra 250 mozilla-central opt test mochitest-8 942.242424242
    Android Tegra 250 mozilla-central opt test plain-reftest-1 2731.84375
    Android Tegra 250 mozilla-central opt test plain-reftest-2 3308.125
    Android Tegra 250 mozilla-central opt test plain-reftest-3 2618.5
    Android Tegra 250 mozilla-central opt test plain-reftest-4 2881.09375
    Android Tegra 250 mozilla-central opt test robocop-1 1491.9375
    Android Tegra 250 mozilla-central opt test robocop-2 2182.28125
    Android Tegra 250 mozilla-central opt test xpcshell 2204.3125
    Android Tegra 250 mozilla-central talos remote-tcanvasmark 869.59375
    Android Tegra 250 mozilla-central talos remote-tp4m_nochrome 2693.90625
    Android Tegra 250 mozilla-central talos remote-trobocheck2 959.571428571
    Android Tegra 250 mozilla-central talos remote-trobopan 982.6875
    Android Tegra 250 mozilla-central talos remote-troboprovider 1245.3125
    Android Tegra 250 mozilla-central talos remote-ts 689.40625
    Android Tegra 250 mozilla-central talos remote-tsvg 748.375
    Android Tegra 250 mozilla-central talos remote-tsvgx 1422.21875
    Android X86 mozilla-central build 4151.77419355
    Android mozilla-central build 4090.67741935
    Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-1 1847.66666667
    Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-2 1916.77777778
    Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-3 1846.33333333
    Android no-ionmonkey mozilla-central build 4080.74193548
    Linux mozilla-central build 6547.11278195
    Linux mozilla-central leak test build 8033.2556391
    Linux x86-64 mozilla-central asan build 7470.8880597
    Linux x86-64 mozilla-central build 6484.28571429
    Linux x86-64 mozilla-central debug asan build 12863.7744361
    Linux x86-64 mozilla-central debug static analysis build 5324.42105263
    Linux x86-64 mozilla-central hsts preload update 861.0
    Linux x86-64 mozilla-central leak test build 7858.93233083
    Linux x86-64 mozilla-central leak test spidermonkey_tier_1-generational build 2355.7826087
    Linux x86-64 mozilla-central leak test spidermonkey_tier_1-rootanalysis build 3213.34782609
    OS X 10.7 64-bit mozilla-central leak test build 5344.62406015
    OS X 10.7 mozilla-central build 6902.86666667
    Rev3 Fedora 12 mozilla-central debug test mochitest-browser-chrome 6873.0
    Rev3 Fedora 12×64 mozilla-central debug test mochitest-browser-chrome 5713.30952381
    Rev4 MacOSX Lion 10.7 mozilla-central debug test crashtest 912.038167939
    Rev4 MacOSX Lion 10.7 mozilla-central debug test jetpack 2180.04511278
    Rev4 MacOSX Lion 10.7 mozilla-central debug test jsreftest 1628.6969697
    Rev4 MacOSX Lion 10.7 mozilla-central debug test marionette 402.893129771
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-1 2839.85714286
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-2 1178.44274809
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-3 2682.72932331
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-4 700.62406015
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-5 1240.26865672
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-browser-chrome 8036.58823529
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-other 2266.85925926
    Rev4 MacOSX Lion 10.7 mozilla-central debug test reftest 3225.32824427
    Rev4 MacOSX Lion 10.7 mozilla-central debug test xpcshell 1349.66412214
    Rev4 MacOSX Lion 10.7 mozilla-central opt test crashtest 554.69005848
    Rev4 MacOSX Lion 10.7 mozilla-central opt test jetpack 704.111111111
    Rev4 MacOSX Lion 10.7 mozilla-central opt test jsreftest 724.851612903
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-1 1441.04545455
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-2 555.107594937
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-3 1205.51298701
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-4 378.549019608
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-5 588.419354839
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-browser-chrome 2229.45454545
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-other 966.287581699
    Rev4 MacOSX Lion 10.7 mozilla-central opt test reftest 1754.82051282
    Rev4 MacOSX Lion 10.7 mozilla-central opt test xpcshell 792.214285714
    Rev4 MacOSX Lion 10.7 mozilla-central talos chromez 989.668831169
    Rev4 MacOSX Lion 10.7 mozilla-central talos dirtypaint 780.272727273
    Rev4 MacOSX Lion 10.7 mozilla-central talos dromaeojs 1594.50649351
    Rev4 MacOSX Lion 10.7 mozilla-central talos other 630.180645161
    Rev4 MacOSX Lion 10.7 mozilla-central talos rafx 2532.88888889
    Rev4 MacOSX Lion 10.7 mozilla-central talos svgr 1694.70414201
    Rev4 MacOSX Lion 10.7 mozilla-central talos tp5o 1107.99363057
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test crashtest 846.484848485
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jetpack 1911.14503817
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jsreftest 1486.49618321
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test marionette 350.969465649
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-1 2757.54961832
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-2 1085.3030303
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-3 2481.8778626
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-4 697.496183206
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-5 1191.85496183
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-browser-chrome 7223.29850746
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-other 2115.2406015
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test reftest 3124.41666667
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test xpcshell 1666.88986784
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test crashtest 542.647058824
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jetpack 713.529411765
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jsreftest 689.921568627
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-1 1469.20779221
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-2 527.124183007
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-3 1205.6474359
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-4 378.908496732
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-5 610.248366013
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-browser-chrome 2237.68387097
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-other 965.201298701
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test reftest 1799.96732026
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test xpcshell 771.849673203
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos chromez 970.058441558
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dirtypaint 795.846846847
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dromaeojs 1586.25806452
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos other 602.039215686
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos rafx 2507.92592593
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos svgr 1713.57831325
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos tp5o 1030.33766234
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test crashtest 732.175572519
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test jetpack 1565.57251908
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test jsreftest 1365.38167939
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test marionette 370.757575758
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-1 2291.09848485
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-2 955.816793893
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-3 2098.70992366
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-4 588.038167939
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-5 964.083969466
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-browser-chrome 6020.40740741
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-other 1762.40909091
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test reftest 2338.97709924
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test xpcshell 803.992366412
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test crashtest 470.823529412
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test jetpack 578.065359477
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test jsreftest 701.444444444
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-1 1303.61290323
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-2 529.376623377
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-3 1114.85897436
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-4 369.169934641
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-5 533.967320261
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-browser-chrome 1726.04575163
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-other 892.810457516
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test reftest 1137.37662338
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test xpcshell 598.797385621
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos chromez 1026.55555556
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos dirtypaint 810.513513514
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos dromaeojs 1576.92810458
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos other 613.409090909
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos rafx 2286.77777778
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos svgr 1237.26829268
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos tp5o 869.503267974
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test crashtest 961.595744681
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test jetpack 1461.6993007
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test jsreftest 2009.38297872
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-1 2748.66901408
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-2 1187.30496454
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-3 2160.24647887
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-4 1040.14084507
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-5 1176.88652482
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-browser-chrome 4652.4295302
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-other 2144.56737589
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test reftest 3649.06382979
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test xpcshell 2459.59859155
    Ubuntu HW 12.04 mozilla-central talos chromez 952.193277311
    Ubuntu HW 12.04 mozilla-central talos dirtypaint 738.035294118
    Ubuntu HW 12.04 mozilla-central talos dromaeojs 1516.75833333
    Ubuntu HW 12.04 mozilla-central talos other 589.775
    Ubuntu HW 12.04 mozilla-central talos rafx 2280.65217391
    Ubuntu HW 12.04 mozilla-central talos svgr 1377.856
    Ubuntu HW 12.04 mozilla-central talos tp5o 893.647058824
    Ubuntu HW 12.04 x64 mozilla-central talos chromez 964.446280992
    Ubuntu HW 12.04 x64 mozilla-central talos dirtypaint 730.454545455
    Ubuntu HW 12.04 x64 mozilla-central talos dromaeojs 1521.45
    Ubuntu HW 12.04 x64 mozilla-central talos other 560.719008264
    Ubuntu HW 12.04 x64 mozilla-central talos rafx 2237.69565217
    Ubuntu HW 12.04 x64 mozilla-central talos svgr 1266.752
    Ubuntu HW 12.04 x64 mozilla-central talos tp5o 882.860655738
    Ubuntu VM 12.04 mozilla-central debug test crashtest 1458.65546218
    Ubuntu VM 12.04 mozilla-central debug test jetpack 1894.76470588
    Ubuntu VM 12.04 mozilla-central debug test jsreftest 2499.36134454
    Ubuntu VM 12.04 mozilla-central debug test marionette 411.090909091
    Ubuntu VM 12.04 mozilla-central debug test mochitest-1 4014.19327731
    Ubuntu VM 12.04 mozilla-central debug test mochitest-2 1359.38655462
    Ubuntu VM 12.04 mozilla-central debug test mochitest-3 2995.5210084
    Ubuntu VM 12.04 mozilla-central debug test mochitest-4 1133.36666667
    Ubuntu VM 12.04 mozilla-central debug test mochitest-5 1570.45378151
    Ubuntu VM 12.04 mozilla-central debug test mochitest-other 2739.95798319
    Ubuntu VM 12.04 mozilla-central debug test reftest 6202.39516129
    Ubuntu VM 12.04 mozilla-central debug test xpcshell 3091.74380165
    Ubuntu VM 12.04 mozilla-central opt test crashtest 584.541666667
    Ubuntu VM 12.04 mozilla-central opt test crashtest-ipc 648.475
    Ubuntu VM 12.04 mozilla-central opt test jetpack 778.710743802
    Ubuntu VM 12.04 mozilla-central opt test jsreftest 1359.88235294
    Ubuntu VM 12.04 mozilla-central opt test mochitest-1 2331.575
    Ubuntu VM 12.04 mozilla-central opt test mochitest-2 680.358333333
    Ubuntu VM 12.04 mozilla-central opt test mochitest-3 1485.38655462
    Ubuntu VM 12.04 mozilla-central opt test mochitest-4 682.487394958
    Ubuntu VM 12.04 mozilla-central opt test mochitest-5 939.621848739
    Ubuntu VM 12.04 mozilla-central opt test mochitest-browser-chrome 3086.47058824
    Ubuntu VM 12.04 mozilla-central opt test mochitest-other 1400.20168067
    Ubuntu VM 12.04 mozilla-central opt test reftest 4367.49166667
    Ubuntu VM 12.04 mozilla-central opt test reftest-ipc 239.924369748
    Ubuntu VM 12.04 mozilla-central opt test reftest-no-accel 4395.63636364
    Ubuntu VM 12.04 mozilla-central opt test xpcshell 1769.68067227
    Ubuntu VM 12.04 x64 mozilla-central debug test crashtest 995.388429752
    Ubuntu VM 12.04 x64 mozilla-central debug test jetpack 1542.07438017
    Ubuntu VM 12.04 x64 mozilla-central debug test jsreftest 1878.79166667
    Ubuntu VM 12.04 x64 mozilla-central debug test marionette 419.858333333
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-1 3084.53333333
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-2 1104.36666667
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-3 2377.53333333
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-4 934.818181818
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-5 1111.96666667
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-other 2087.90833333
    Ubuntu VM 12.04 x64 mozilla-central debug test reftest 3202.85833333
    Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell 2222.9338843
    Ubuntu VM 12.04 x64 mozilla-central opt test crashtest 499.308333333
    Ubuntu VM 12.04 x64 mozilla-central opt test jetpack 743.258333333
    Ubuntu VM 12.04 x64 mozilla-central opt test jsreftest 1009.11570248
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-1 1914.46666667
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-2 604.758333333
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-3 1359.05
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-4 619.157024793
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-5 658.2
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-browser-chrome 2530.58333333
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-other 1124.975
    Ubuntu VM 12.04 x64 mozilla-central opt test reftest 2600.90909091
    Ubuntu VM 12.04 x64 mozilla-central opt test xpcshell 1315.70833333
    WINNT 5.2 mozilla-central build 8063.63013699
    WINNT 5.2 mozilla-central leak test build 9790.52054795
    WINNT 6.1 x86-64 mozilla-central build 8946.48201439
    WINNT 6.1 x86-64 mozilla-central leak test build 8435.95192308
    WINNT 6.2 mozilla-central debug test crashtest 751.873134328
    WINNT 6.2 mozilla-central debug test jetpack 1364.60583942
    WINNT 6.2 mozilla-central debug test jsreftest 1395.36486486
    WINNT 6.2 mozilla-central debug test mochitest-1 2228.87407407
    WINNT 6.2 mozilla-central debug test mochitest-2 1000.60869565
    WINNT 6.2 mozilla-central debug test mochitest-3 2069.69343066
    WINNT 6.2 mozilla-central debug test mochitest-4 603.706766917
    WINNT 6.2 mozilla-central debug test mochitest-5 896.619402985
    WINNT 6.2 mozilla-central debug test mochitest-browser-chrome 5021.21830986
    WINNT 6.2 mozilla-central debug test mochitest-other 1730.02255639
    WINNT 6.2 mozilla-central debug test reftest 2305.29411765
    WINNT 6.2 mozilla-central debug test xpcshell 2103.74814815
    WINNT 6.2 mozilla-central opt test crashtest 460.167883212
    WINNT 6.2 mozilla-central opt test jetpack 831.781021898
    WINNT 6.2 mozilla-central opt test jsreftest 716.274074074
    WINNT 6.2 mozilla-central opt test mochitest-1 1417.82962963
    WINNT 6.2 mozilla-central opt test mochitest-2 563.420382166
    WINNT 6.2 mozilla-central opt test mochitest-3 1308.9084507
    WINNT 6.2 mozilla-central opt test mochitest-4 400.184397163
    WINNT 6.2 mozilla-central opt test mochitest-5 448.872727273
    WINNT 6.2 mozilla-central opt test mochitest-browser-chrome 1979.98518519
    WINNT 6.2 mozilla-central opt test mochitest-metro-chrome 367.215189873
    WINNT 6.2 mozilla-central opt test mochitest-other 997.897810219
    WINNT 6.2 mozilla-central opt test reftest 1064.02941176
    WINNT 6.2 mozilla-central opt test reftest-no-accel 1067.54676259
    WINNT 6.2 mozilla-central opt test xpcshell 1983.23913043
    WINNT 6.2 mozilla-central talos chromez 1286.47101449
    WINNT 6.2 mozilla-central talos dirtypaint 1333.3047619
    WINNT 6.2 mozilla-central talos dromaeojs 1799.29411765
    WINNT 6.2 mozilla-central talos other 882.548611111
    WINNT 6.2 mozilla-central talos rafx 2325.04347826
    WINNT 6.2 mozilla-central talos svgr 1289.19858156
    WINNT 6.2 mozilla-central talos tp5o 992.410071942
    Windows 7 32-bit mozilla-central debug test crashtest 797.466165414
    Windows 7 32-bit mozilla-central debug test jetpack 1354.84210526
    Windows 7 32-bit mozilla-central debug test jsreftest 1578.18045113
    Windows 7 32-bit mozilla-central debug test mochitest-1 2346.2761194
    Windows 7 32-bit mozilla-central debug test mochitest-2 1016.79411765
    Windows 7 32-bit mozilla-central debug test mochitest-3 2117.04511278
    Windows 7 32-bit mozilla-central debug test mochitest-4 638.593984962
    Windows 7 32-bit mozilla-central debug test mochitest-5 907.218045113
    Windows 7 32-bit mozilla-central debug test mochitest-browser-chrome 5023.84172662
    Windows 7 32-bit mozilla-central debug test mochitest-other 1675.0
    Windows 7 32-bit mozilla-central debug test reftest 2529.97744361
    Windows 7 32-bit mozilla-central debug test xpcshell 1346.7443609
    Windows 7 32-bit mozilla-central opt test crashtest 527.288888889
    Windows 7 32-bit mozilla-central opt test jetpack 769.192592593
    Windows 7 32-bit mozilla-central opt test jsreftest 703.711111111
    Windows 7 32-bit mozilla-central opt test mochitest-1 1476.86666667
    Windows 7 32-bit mozilla-central opt test mochitest-2 607.866666667
    Windows 7 32-bit mozilla-central opt test mochitest-3 1322.02205882
    Windows 7 32-bit mozilla-central opt test mochitest-4 426.385185185
    Windows 7 32-bit mozilla-central opt test mochitest-5 538.296296296
    Windows 7 32-bit mozilla-central opt test mochitest-browser-chrome 1784.14074074
    Windows 7 32-bit mozilla-central opt test mochitest-other 962.288888889
    Windows 7 32-bit mozilla-central opt test reftest 1180.63703704
    Windows 7 32-bit mozilla-central opt test reftest-no-accel 1137.61029412
    Windows 7 32-bit mozilla-central opt test xpcshell 1166.88888889
    Windows 7 32-bit mozilla-central talos chromez 1223.25925926
    Windows 7 32-bit mozilla-central talos dirtypaint 1424.21
    Windows 7 32-bit mozilla-central talos dromaeojs 1809.20740741
    Windows 7 32-bit mozilla-central talos other 898.432624113
    Windows 7 32-bit mozilla-central talos rafx 2261.52173913
    Windows 7 32-bit mozilla-central talos svgr 1244.19148936
    Windows 7 32-bit mozilla-central talos tp5o 1070.30147059
    Windows 7 32-bit mozilla-central talos xperf 614.072992701
    Windows XP 32-bit mozilla-central debug test crashtest 764.954887218
    Windows XP 32-bit mozilla-central debug test jetpack 1190.70895522
    Windows XP 32-bit mozilla-central debug test jsreftest 1575.59398496
    Windows XP 32-bit mozilla-central debug test mochitest-1 2158.66917293
    Windows XP 32-bit mozilla-central debug test mochitest-2 939.481203008
    Windows XP 32-bit mozilla-central debug test mochitest-3 1893.02255639
    Windows XP 32-bit mozilla-central debug test mochitest-4 578.879699248
    Windows XP 32-bit mozilla-central debug test mochitest-5 835.45112782
    Windows XP 32-bit mozilla-central debug test mochitest-browser-chrome 4572.21393035
    Windows XP 32-bit mozilla-central debug test mochitest-other 1603.60447761
    Windows XP 32-bit mozilla-central debug test reftest 2385.20895522
    Windows XP 32-bit mozilla-central debug test xpcshell 753.683823529
    Windows XP 32-bit mozilla-central opt test crashtest 413.133333333
    Windows XP 32-bit mozilla-central opt test jetpack 527.794117647
    Windows XP 32-bit mozilla-central opt test jsreftest 659.433823529
    Windows XP 32-bit mozilla-central opt test mochitest-1 1336.01481481
    Windows XP 32-bit mozilla-central opt test mochitest-2 487.474074074
    Windows XP 32-bit mozilla-central opt test mochitest-3 1094.94814815
    Windows XP 32-bit mozilla-central opt test mochitest-4 366.851851852
    Windows XP 32-bit mozilla-central opt test mochitest-5 482.804347826
    Windows XP 32-bit mozilla-central opt test mochitest-browser-chrome 1612.69852941
    Windows XP 32-bit mozilla-central opt test mochitest-other 880.814814815
    Windows XP 32-bit mozilla-central opt test reftest 1002.15441176
    Windows XP 32-bit mozilla-central opt test xpcshell 582.841726619
    Windows XP 32-bit mozilla-central talos chromez 1002.6962963
    Windows XP 32-bit mozilla-central talos dirtypaint 1017.12
    Windows XP 32-bit mozilla-central talos other 646.133802817
    Windows XP 32-bit mozilla-central talos rafx 2196.86956522
    Windows XP 32-bit mozilla-central talos svgr 1235.08965517
    Windows XP 32-bit mozilla-central talos tp5o 957.955555556
    b2g_emulator mozilla-central opt test marionette-webapi 1320.44
    b2g_emulator mozilla-central opt test reftest-1 1709.15254237
    b2g_emulator mozilla-central opt test reftest-10 1719.45299145
    b2g_emulator mozilla-central opt test reftest-2 1692.43103448
    b2g_emulator mozilla-central opt test reftest-3 1709.35042735
    b2g_emulator mozilla-central opt test reftest-4 1841.12931034
    b2g_emulator mozilla-central opt test reftest-5 2088.30172414
    b2g_emulator mozilla-central opt test reftest-6 2040.93965517
    b2g_emulator mozilla-central opt test reftest-7 3393.53448276
    b2g_emulator mozilla-central opt test reftest-8 1516.5
    b2g_emulator mozilla-central opt test reftest-9 1861.65517241
    b2g_emulator_vm mozilla-central opt test crashtest-1 2871.608
    b2g_emulator_vm mozilla-central opt test crashtest-2 2733.39655172
    b2g_emulator_vm mozilla-central opt test crashtest-3 2594.16101695
    b2g_emulator_vm mozilla-central opt test mochitest-1 1827.7394958
    b2g_emulator_vm mozilla-central opt test mochitest-2 1675.73275862
    b2g_emulator_vm mozilla-central opt test mochitest-3 2587.41322314
    b2g_emulator_vm mozilla-central opt test mochitest-4 1699.68067227
    b2g_emulator_vm mozilla-central opt test mochitest-5 1441.98275862
    b2g_emulator_vm mozilla-central opt test mochitest-6 1321.13675214
    b2g_emulator_vm mozilla-central opt test mochitest-7 1246.63247863
    b2g_emulator_vm mozilla-central opt test mochitest-8 1409.75213675
    b2g_emulator_vm mozilla-central opt test mochitest-9 2525.18803419
    b2g_emulator_vm mozilla-central opt test xpcshell 584.495726496
    b2g_mozilla-central_emulator-debug_dep 7271.58208955
    b2g_mozilla-central_emulator-jb-debug_dep 10738.1278195
    b2g_mozilla-central_emulator-jb_dep 10171.2706767
    b2g_mozilla-central_emulator_dep 6716.56716418
    b2g_mozilla-central_hamachi_dep 6306.73134328
    b2g_mozilla-central_hamachi_eng_dep 6204.47692308
    b2g_mozilla-central_helix_dep 6333.37037037
    b2g_mozilla-central_inari_dep 6202.20149254
    b2g_mozilla-central_leo_dep 6319.72180451
    b2g_mozilla-central_leo_eng_dep 6388.43609023
    b2g_mozilla-central_linux32_gecko build 6104.19548872
    b2g_mozilla-central_linux64_gecko build 6063.90225564
    b2g_mozilla-central_macosx64_gecko build 4039.61654135
    b2g_mozilla-central_nexus-4_dep 8721.24358974
    b2g_mozilla-central_unagi_dep 6141.65413534
    b2g_mozilla-central_win32_gecko build 7563.01709402
    b2g_ubuntu64_vm mozilla-central opt test gaia-ui-test 687.007042254
    b2g_ubuntu64_vm mozilla-central opt test gaia-unit 524.288732394

    ============================================

6,674 builds and 66,456 test jobs in a 24 hour day

On 13sep2013, we processed 6,674 builds and 66,456 test jobs. That is yet another new record. In one 24hour day. It blew past our previous record just the week before where we processed over 5,000 builds-a-day for the first time ever. September has been quite the month… and its not over yet!

To put that data in perspective, almost 5 months ago, on 23apr2013, was the first time we ever did over 52,118 test jobs in a 24 hour day… and it was almost 17 months ago, on 03may2012, when we first broke 40,000 test jobs in a 24 hour day. To put these numbers in perspective:

  • 66,456 (13sep2013)
  • 52,118 (23apr2013)
  • 40,207 (03may2012)
  • 30,145 (08dec2011)
  • 20,575 (06oct2011)
  • 10,597 (08oct2010)

We’re continuing to build out systems to get ahead of this increasing load, but its worth pausing to note the sheer scale and rate of growth here.

The growth is impressive.

The acceleration of the *rate* of growth is even more impressive.

Infrastructure load for August 2013

  • #checkins-per-month: We had 7,771 checkins in August 2013. This is 10% above last month’s record 7,051 checkins.

    Overall load since Jan 2009

  • #checkins-per-day: We had 443 checkins on 26aug; this new record blew past our previous single-day record of 370 checkins on 02jul. During August, yet again all working days were over 200 checkins per day… in fact our lowest weekday load was 243 checkins on 08aug. 21-of-31 days had over 250 checkins-per-day, 11-of-31 days had over 300 checkins-per-day. 5-of-31 days exceeded our previous record of 370 checkins. Our first time over the 400 checkins-per-day threshold was 443 checkins on 26aug. Wow.
  • #checkins-per-hour: Checkins are still mostly mid-day PT/afternoon ET. For 11 of every 24 hours, we sustained over 12 checkins per hour. This month was the first month we went over 15 checkins-per-hour, and we did that for 3 of every 24 hours! Our heaviest load time this month was 1pm-2pm PT (15.4 checkins-per-hour – a new record), with close followup at 10am-11am (15.2 checkins-per-hour) and 2pm-3pm (15.0 checkins-per-hour).

mozilla-inbound, b2g-inbound, fx-team:

  • mozilla-inbound continues to be heavily used as an integration branch. Its noteworthy that as developers start to use other *-inbound branches, we saw mozilla-inbound reduce significantly to 16.3% of all checkins, the lowest ever usage of mozilla-inbound. The use of multiple *-inbounds is clearly helping improve bottlenecks (see pie chart below) and the congestion on mozilla-inbound is being reduced significantly as people use switch to using other *-inbound branches instead. All very cool to see and a definite part of the reason we hit new records this month.
  • b2g-inbound continues to be a great success, with 8.2% of this month’s checkins landing here, and as predicted last month, the percentage of checkins on this branch is stabilizing.
  • As predicted, usage of fx-team branch jumped for August, as it now had official full-time sheriffs coverage, just like mozilla-inbound and b2g-inbound. At 6.2% of checkins this month, its clearly a very active third place for developers to land. Usage of “fx-team” might grow a little more, over next month or two, before stabilizing. Having sheriff coverage clearly made a difference.
  • The combined total of these 3 integration branches is 30.7%, showing just how much our sheriffs are helping across the board.

    Infrastructure load by branch

mozilla-aurora, mozilla-beta, mozilla-b2g18, gaia-central:
Of our total monthly checkins:

  • 2.0% landed into mozilla-central, slightly higher than last month. As usual, very few people land directly on mozilla-central these days, when there are sheriff-assisted branches available instead.
  • 1.6% landed into mozilla-aurora, slightly lower than last month.
  • 1.4% landed into mozilla-beta, slightly higher than last month.
  • 0.5% landed into mozilla-b2g18, slightly lower then last month.
  • Note: gaia-central, and all other gaia-* branches, are not counted here anymore. For details, see here.

misc other details:
As usual, our build pool handled the load well, with >95% of all builds consistently being started within 15mins. Our test pool continues to improve. All the hard work by RelEng, ATeam and IT is paying off, we’re seeing more test jobs being handled with better response times. The work on fixing/disabling any tests that are hidden-yet-still-being-run is also improving our test situation. The peak per-day test load for August was 52,000 test jobs on 27aug. Still more work to be done here, but very encouraging progress.

As always, if you know of any test suites that no longer need to be run per-checkin, please let us know so we can immediately reduce the load a little. Also, if you know of any test suites which are perma-orange, and hidden on tbpl.m.o, please let us know – thats the worst of both worlds – using up scarce CPU time and not being displayed for people to make use of. We’ll make sure to file bugs to get tests fixed – or disabled – every little bit helps put scarce test CPU to better use.