254 compute hours per checkin

The rate of checkins continues to rise rapidly. The volume of builds+tests run per checkin is also increasing rapidly. The combination of these two increases multiplied together is an impressive rate of growth.

How many hours of builds and tests do we run per commit?

  • 254 compute hours = ~10.5 compute *days* (sep2013)
  • 137 compute hours = ~5.7 compute *days* (aug2012)
  • 110 compute hours = ~4.6 compute *days*(jan2012)
  • ~40 compute hours = ~1.6 compute *days*(2009)

#test jobs in a day

10.5 compute *days* of builds and tests per checkin is a lot – especially when we average one checkin every ~6 minutes last month, and handle a checkin every 4minutes during peak times. Put another way, that’s an average of ~6.9 compute years of builds and tests being run per day, or 213 compute *years* for the month of August. Mind boggling.

If you look at the “wait times” posts in dev.tree-management, you’ll see that systems we run on AWS are scaling nicely to handle the load, and we continue to move production load over to AWS for nativefennec / B2G desktop builds / linux desktop …. and then recycle existing inhouse hardware to help with inhouse capacity for win32 and win64.

Some of this growth is good (new builds, new test suites).
But some of growth is bad (consistently failing tests; obsolete tests; builds that are no longer needed)… and this slows us all down.

You can help.

  1. When you push to tryserver, choose wisely. If you need “all”, do it. But ask yourself… do you really need all?
  2. If you know of any build/test that is no longer needed, let us know.
  3. If you know of any build/test being hidden on tbpl.m.o because it constantly fails let us know or file a bug. We’ll disable the job to stop wasting CPU time, which will help everyone else’s jobs get processed more quickly. As needed, we’ll also file bugs to track fixing the problem and then reenable once the build/test is running green. While any one given build or test suite might not seem like a lot of time to save, we run all these jobs literally thousands of times per day.

    Every little saving helps makes things even faster for everyone!

    Thank you!
    John.
    ===========================================
    List of builds and tests run (with duration of job in seconds)
    Android 2.2 Armv6 Tegra mozilla-central opt test crashtest 1205.62037037
    Android 2.2 Armv6 Tegra mozilla-central opt test jsreftest-1 1961.08333333
    Android 2.2 Armv6 Tegra mozilla-central opt test jsreftest-2 1996.19626168
    Android 2.2 Armv6 Tegra mozilla-central opt test jsreftest-3 2043.25233645
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-1 1039.10909091
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-2 912.092592593
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-3 1291.40740741
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-4 1066.77570093
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-5 979.252336449
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-6 1103.2
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-7 975.355140187
    Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-8 1050.5
    Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-1 2538.24299065
    Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-2 2849.25
    Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-3 2632.22429907
    Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-4 2589.43925234
    Android 2.2 Armv6 Tegra mozilla-central opt test robocop-1 1494.53271028
    Android 2.2 Armv6 Tegra mozilla-central opt test robocop-2 2251.78504673
    Android 2.2 Armv6 Tegra mozilla-central opt test xpcshell 2289.25688073
    Android 2.2 Armv6 mozilla-central build 4112.26470588
    Android 2.2 Debug mozilla-central build 4612.8627451
    Android 2.2 Tegra mozilla-central opt test crashtest 1198.16666667
    Android 2.2 Tegra mozilla-central opt test jsreftest-1 1905.94545455
    Android 2.2 Tegra mozilla-central opt test jsreftest-2 1798.31192661
    Android 2.2 Tegra mozilla-central opt test jsreftest-3 2102.10091743
    Android 2.2 Tegra mozilla-central opt test mochitest-1 1014.01834862
    Android 2.2 Tegra mozilla-central opt test mochitest-2 893.449541284
    Android 2.2 Tegra mozilla-central opt test mochitest-3 1270.65454545
    Android 2.2 Tegra mozilla-central opt test mochitest-4 981.158878505
    Android 2.2 Tegra mozilla-central opt test mochitest-5 938.247706422
    Android 2.2 Tegra mozilla-central opt test mochitest-6 1128.26126126
    Android 2.2 Tegra mozilla-central opt test mochitest-7 988.91588785
    Android 2.2 Tegra mozilla-central opt test mochitest-8 997.898148148
    Android 2.2 Tegra mozilla-central opt test plain-reftest-1 2527.41121495
    Android 2.2 Tegra mozilla-central opt test plain-reftest-2 2815.85185185
    Android 2.2 Tegra mozilla-central opt test plain-reftest-3 2623.35514019
    Android 2.2 Tegra mozilla-central opt test plain-reftest-4 2582.18691589
    Android 2.2 Tegra mozilla-central opt test robocop-1 1542.2293578
    Android 2.2 Tegra mozilla-central opt test robocop-2 2233.76576577
    Android 2.2 Tegra mozilla-central opt test xpcshell 2308.20720721
    Android 2.2 Tegra mozilla-central talos remote-tcanvasmark 886.648148148
    Android 2.2 Tegra mozilla-central talos remote-tp4m_nochrome 1147.54205607
    Android 2.2 Tegra mozilla-central talos remote-trobocheck2 983.117117117
    Android 2.2 Tegra mozilla-central talos remote-trobopan 1006.74074074
    Android 2.2 Tegra mozilla-central talos remote-troboprovider 1280.57009346
    Android 2.2 Tegra mozilla-central talos remote-ts 705.117647059
    Android 2.2 Tegra mozilla-central talos remote-tspaint 688.58974359
    Android 2.2 Tegra mozilla-central talos remote-tsvg 751.75
    Android 2.2 Tegra mozilla-central talos remote-tsvgx 1428.88073394
    Android 2.2 mozilla-central build 4251.46078431
    Android 2.2 no-ionmonkey Tegra mozilla-central opt test jsreftest-1 2015.42045455
    Android 2.2 no-ionmonkey Tegra mozilla-central opt test jsreftest-2 1812.60227273
    Android 2.2 no-ionmonkey Tegra mozilla-central opt test jsreftest-3 2138.88636364
    Android 2.2 no-ionmonkey mozilla-central build 3993.67647059
    Android 4.0 Panda mozilla-central opt test crashtest 1413.98843931
    Android 4.0 Panda mozilla-central opt test jsreftest-1 1584.75842697
    Android 4.0 Panda mozilla-central opt test jsreftest-2 1582.75147929
    Android 4.0 Panda mozilla-central opt test jsreftest-3 1566.4566474
    Android 4.0 Panda mozilla-central opt test mochitest-1 1216.17679558
    Android 4.0 Panda mozilla-central opt test mochitest-2 1098.2754491
    Android 4.0 Panda mozilla-central opt test mochitest-3 1410.21387283
    Android 4.0 Panda mozilla-central opt test mochitest-4 1197.27906977
    Android 4.0 Panda mozilla-central opt test mochitest-5 1212.03888889
    Android 4.0 Panda mozilla-central opt test mochitest-6 1166.41489362
    Android 4.0 Panda mozilla-central opt test mochitest-7 1209.33532934
    Android 4.0 Panda mozilla-central opt test mochitest-8 1140.14754098
    Android 4.0 Panda mozilla-central opt test mochitest-gl 1116.70414201
    Android 4.0 Panda mozilla-central opt test robocop-1 1288.49710983
    Android 4.0 Panda mozilla-central opt test robocop-2 1739.20833333
    Android 4.0 Panda mozilla-central opt test robocop-3 1587.77192982
    Android 4.0 Panda mozilla-central opt test xpcshell 1989.28021978
    Android 4.0 Panda mozilla-central talos remote-tcanvasmark 2836.71018277
    Android 4.0 Panda mozilla-central talos remote-tp4m_nochrome 1484.12068966
    Android 4.0 Panda mozilla-central talos remote-trobocheck2 1168.93292683
    Android 4.0 Panda mozilla-central talos remote-trobopan 1077.8969697
    Android 4.0 Panda mozilla-central talos remote-troboprovider 1370.75903614
    Android 4.0 Panda mozilla-central talos remote-ts 850.294573643
    Android 4.0 Panda mozilla-central talos remote-tspaint 933.782608696
    Android 4.0 Panda mozilla-central talos remote-tsvg 960.408333333
    Android 4.0 Panda mozilla-central talos remote-tsvgx 1608.24
    Android 4.2 x86 Emulator mozilla-central opt test androidx86-set-1 3556.85074627
    Android 4.2 x86 Emulator mozilla-central opt test androidx86-set-2 2033.921875
    Android 4.2 x86 mozilla-central build 4230.7254902
    Android Armv6 Tegra 250 mozilla-central opt test crashtest 1199.5
    Android Armv6 Tegra 250 mozilla-central opt test jsreftest-1 1950.375
    Android Armv6 Tegra 250 mozilla-central opt test jsreftest-2 1819.0625
    Android Armv6 Tegra 250 mozilla-central opt test jsreftest-3 2357.21875
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-1 1026.875
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-2 893.21875
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-3 1279.72727273
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-4 1037.84375
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-5 986.28125
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-6 1083.5625
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-7 1004.09090909
    Android Armv6 Tegra 250 mozilla-central opt test mochitest-8 1019.36363636
    Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-1 2711.9375
    Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-2 3252.09375
    Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-3 2679.6875
    Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-4 2923.84375
    Android Armv6 Tegra 250 mozilla-central opt test robocop-1 1455.78125
    Android Armv6 Tegra 250 mozilla-central opt test robocop-2 2155.1875
    Android Armv6 Tegra 250 mozilla-central opt test xpcshell 2191.09375
    Android Armv6 mozilla-central build 4099.38709677
    Android Debug mozilla-central build 4822.32258065
    Android Tegra 250 mozilla-central opt test crashtest 1161.53125
    Android Tegra 250 mozilla-central opt test jsreftest-1 1902.25
    Android Tegra 250 mozilla-central opt test jsreftest-2 1873.1875
    Android Tegra 250 mozilla-central opt test jsreftest-3 2000.6875
    Android Tegra 250 mozilla-central opt test mochitest-1 978.96875
    Android Tegra 250 mozilla-central opt test mochitest-2 841.90625
    Android Tegra 250 mozilla-central opt test mochitest-3 1245.6875
    Android Tegra 250 mozilla-central opt test mochitest-4 983.1875
    Android Tegra 250 mozilla-central opt test mochitest-5 956.84375
    Android Tegra 250 mozilla-central opt test mochitest-6 1055.65625
    Android Tegra 250 mozilla-central opt test mochitest-7 960.6875
    Android Tegra 250 mozilla-central opt test mochitest-8 942.242424242
    Android Tegra 250 mozilla-central opt test plain-reftest-1 2731.84375
    Android Tegra 250 mozilla-central opt test plain-reftest-2 3308.125
    Android Tegra 250 mozilla-central opt test plain-reftest-3 2618.5
    Android Tegra 250 mozilla-central opt test plain-reftest-4 2881.09375
    Android Tegra 250 mozilla-central opt test robocop-1 1491.9375
    Android Tegra 250 mozilla-central opt test robocop-2 2182.28125
    Android Tegra 250 mozilla-central opt test xpcshell 2204.3125
    Android Tegra 250 mozilla-central talos remote-tcanvasmark 869.59375
    Android Tegra 250 mozilla-central talos remote-tp4m_nochrome 2693.90625
    Android Tegra 250 mozilla-central talos remote-trobocheck2 959.571428571
    Android Tegra 250 mozilla-central talos remote-trobopan 982.6875
    Android Tegra 250 mozilla-central talos remote-troboprovider 1245.3125
    Android Tegra 250 mozilla-central talos remote-ts 689.40625
    Android Tegra 250 mozilla-central talos remote-tsvg 748.375
    Android Tegra 250 mozilla-central talos remote-tsvgx 1422.21875
    Android X86 mozilla-central build 4151.77419355
    Android mozilla-central build 4090.67741935
    Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-1 1847.66666667
    Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-2 1916.77777778
    Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-3 1846.33333333
    Android no-ionmonkey mozilla-central build 4080.74193548
    Linux mozilla-central build 6547.11278195
    Linux mozilla-central leak test build 8033.2556391
    Linux x86-64 mozilla-central asan build 7470.8880597
    Linux x86-64 mozilla-central build 6484.28571429
    Linux x86-64 mozilla-central debug asan build 12863.7744361
    Linux x86-64 mozilla-central debug static analysis build 5324.42105263
    Linux x86-64 mozilla-central hsts preload update 861.0
    Linux x86-64 mozilla-central leak test build 7858.93233083
    Linux x86-64 mozilla-central leak test spidermonkey_tier_1-generational build 2355.7826087
    Linux x86-64 mozilla-central leak test spidermonkey_tier_1-rootanalysis build 3213.34782609
    OS X 10.7 64-bit mozilla-central leak test build 5344.62406015
    OS X 10.7 mozilla-central build 6902.86666667
    Rev3 Fedora 12 mozilla-central debug test mochitest-browser-chrome 6873.0
    Rev3 Fedora 12×64 mozilla-central debug test mochitest-browser-chrome 5713.30952381
    Rev4 MacOSX Lion 10.7 mozilla-central debug test crashtest 912.038167939
    Rev4 MacOSX Lion 10.7 mozilla-central debug test jetpack 2180.04511278
    Rev4 MacOSX Lion 10.7 mozilla-central debug test jsreftest 1628.6969697
    Rev4 MacOSX Lion 10.7 mozilla-central debug test marionette 402.893129771
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-1 2839.85714286
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-2 1178.44274809
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-3 2682.72932331
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-4 700.62406015
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-5 1240.26865672
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-browser-chrome 8036.58823529
    Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-other 2266.85925926
    Rev4 MacOSX Lion 10.7 mozilla-central debug test reftest 3225.32824427
    Rev4 MacOSX Lion 10.7 mozilla-central debug test xpcshell 1349.66412214
    Rev4 MacOSX Lion 10.7 mozilla-central opt test crashtest 554.69005848
    Rev4 MacOSX Lion 10.7 mozilla-central opt test jetpack 704.111111111
    Rev4 MacOSX Lion 10.7 mozilla-central opt test jsreftest 724.851612903
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-1 1441.04545455
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-2 555.107594937
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-3 1205.51298701
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-4 378.549019608
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-5 588.419354839
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-browser-chrome 2229.45454545
    Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-other 966.287581699
    Rev4 MacOSX Lion 10.7 mozilla-central opt test reftest 1754.82051282
    Rev4 MacOSX Lion 10.7 mozilla-central opt test xpcshell 792.214285714
    Rev4 MacOSX Lion 10.7 mozilla-central talos chromez 989.668831169
    Rev4 MacOSX Lion 10.7 mozilla-central talos dirtypaint 780.272727273
    Rev4 MacOSX Lion 10.7 mozilla-central talos dromaeojs 1594.50649351
    Rev4 MacOSX Lion 10.7 mozilla-central talos other 630.180645161
    Rev4 MacOSX Lion 10.7 mozilla-central talos rafx 2532.88888889
    Rev4 MacOSX Lion 10.7 mozilla-central talos svgr 1694.70414201
    Rev4 MacOSX Lion 10.7 mozilla-central talos tp5o 1107.99363057
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test crashtest 846.484848485
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jetpack 1911.14503817
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jsreftest 1486.49618321
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test marionette 350.969465649
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-1 2757.54961832
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-2 1085.3030303
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-3 2481.8778626
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-4 697.496183206
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-5 1191.85496183
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-browser-chrome 7223.29850746
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-other 2115.2406015
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test reftest 3124.41666667
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test xpcshell 1666.88986784
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test crashtest 542.647058824
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jetpack 713.529411765
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jsreftest 689.921568627
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-1 1469.20779221
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-2 527.124183007
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-3 1205.6474359
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-4 378.908496732
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-5 610.248366013
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-browser-chrome 2237.68387097
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-other 965.201298701
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test reftest 1799.96732026
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test xpcshell 771.849673203
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos chromez 970.058441558
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dirtypaint 795.846846847
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dromaeojs 1586.25806452
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos other 602.039215686
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos rafx 2507.92592593
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos svgr 1713.57831325
    Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos tp5o 1030.33766234
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test crashtest 732.175572519
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test jetpack 1565.57251908
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test jsreftest 1365.38167939
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test marionette 370.757575758
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-1 2291.09848485
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-2 955.816793893
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-3 2098.70992366
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-4 588.038167939
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-5 964.083969466
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-browser-chrome 6020.40740741
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-other 1762.40909091
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test reftest 2338.97709924
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test xpcshell 803.992366412
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test crashtest 470.823529412
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test jetpack 578.065359477
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test jsreftest 701.444444444
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-1 1303.61290323
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-2 529.376623377
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-3 1114.85897436
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-4 369.169934641
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-5 533.967320261
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-browser-chrome 1726.04575163
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-other 892.810457516
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test reftest 1137.37662338
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test xpcshell 598.797385621
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos chromez 1026.55555556
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos dirtypaint 810.513513514
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos dromaeojs 1576.92810458
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos other 613.409090909
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos rafx 2286.77777778
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos svgr 1237.26829268
    Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos tp5o 869.503267974
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test crashtest 961.595744681
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test jetpack 1461.6993007
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test jsreftest 2009.38297872
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-1 2748.66901408
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-2 1187.30496454
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-3 2160.24647887
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-4 1040.14084507
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-5 1176.88652482
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-browser-chrome 4652.4295302
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-other 2144.56737589
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test reftest 3649.06382979
    Ubuntu ASAN VM 12.04 x64 mozilla-central opt test xpcshell 2459.59859155
    Ubuntu HW 12.04 mozilla-central talos chromez 952.193277311
    Ubuntu HW 12.04 mozilla-central talos dirtypaint 738.035294118
    Ubuntu HW 12.04 mozilla-central talos dromaeojs 1516.75833333
    Ubuntu HW 12.04 mozilla-central talos other 589.775
    Ubuntu HW 12.04 mozilla-central talos rafx 2280.65217391
    Ubuntu HW 12.04 mozilla-central talos svgr 1377.856
    Ubuntu HW 12.04 mozilla-central talos tp5o 893.647058824
    Ubuntu HW 12.04 x64 mozilla-central talos chromez 964.446280992
    Ubuntu HW 12.04 x64 mozilla-central talos dirtypaint 730.454545455
    Ubuntu HW 12.04 x64 mozilla-central talos dromaeojs 1521.45
    Ubuntu HW 12.04 x64 mozilla-central talos other 560.719008264
    Ubuntu HW 12.04 x64 mozilla-central talos rafx 2237.69565217
    Ubuntu HW 12.04 x64 mozilla-central talos svgr 1266.752
    Ubuntu HW 12.04 x64 mozilla-central talos tp5o 882.860655738
    Ubuntu VM 12.04 mozilla-central debug test crashtest 1458.65546218
    Ubuntu VM 12.04 mozilla-central debug test jetpack 1894.76470588
    Ubuntu VM 12.04 mozilla-central debug test jsreftest 2499.36134454
    Ubuntu VM 12.04 mozilla-central debug test marionette 411.090909091
    Ubuntu VM 12.04 mozilla-central debug test mochitest-1 4014.19327731
    Ubuntu VM 12.04 mozilla-central debug test mochitest-2 1359.38655462
    Ubuntu VM 12.04 mozilla-central debug test mochitest-3 2995.5210084
    Ubuntu VM 12.04 mozilla-central debug test mochitest-4 1133.36666667
    Ubuntu VM 12.04 mozilla-central debug test mochitest-5 1570.45378151
    Ubuntu VM 12.04 mozilla-central debug test mochitest-other 2739.95798319
    Ubuntu VM 12.04 mozilla-central debug test reftest 6202.39516129
    Ubuntu VM 12.04 mozilla-central debug test xpcshell 3091.74380165
    Ubuntu VM 12.04 mozilla-central opt test crashtest 584.541666667
    Ubuntu VM 12.04 mozilla-central opt test crashtest-ipc 648.475
    Ubuntu VM 12.04 mozilla-central opt test jetpack 778.710743802
    Ubuntu VM 12.04 mozilla-central opt test jsreftest 1359.88235294
    Ubuntu VM 12.04 mozilla-central opt test mochitest-1 2331.575
    Ubuntu VM 12.04 mozilla-central opt test mochitest-2 680.358333333
    Ubuntu VM 12.04 mozilla-central opt test mochitest-3 1485.38655462
    Ubuntu VM 12.04 mozilla-central opt test mochitest-4 682.487394958
    Ubuntu VM 12.04 mozilla-central opt test mochitest-5 939.621848739
    Ubuntu VM 12.04 mozilla-central opt test mochitest-browser-chrome 3086.47058824
    Ubuntu VM 12.04 mozilla-central opt test mochitest-other 1400.20168067
    Ubuntu VM 12.04 mozilla-central opt test reftest 4367.49166667
    Ubuntu VM 12.04 mozilla-central opt test reftest-ipc 239.924369748
    Ubuntu VM 12.04 mozilla-central opt test reftest-no-accel 4395.63636364
    Ubuntu VM 12.04 mozilla-central opt test xpcshell 1769.68067227
    Ubuntu VM 12.04 x64 mozilla-central debug test crashtest 995.388429752
    Ubuntu VM 12.04 x64 mozilla-central debug test jetpack 1542.07438017
    Ubuntu VM 12.04 x64 mozilla-central debug test jsreftest 1878.79166667
    Ubuntu VM 12.04 x64 mozilla-central debug test marionette 419.858333333
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-1 3084.53333333
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-2 1104.36666667
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-3 2377.53333333
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-4 934.818181818
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-5 1111.96666667
    Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-other 2087.90833333
    Ubuntu VM 12.04 x64 mozilla-central debug test reftest 3202.85833333
    Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell 2222.9338843
    Ubuntu VM 12.04 x64 mozilla-central opt test crashtest 499.308333333
    Ubuntu VM 12.04 x64 mozilla-central opt test jetpack 743.258333333
    Ubuntu VM 12.04 x64 mozilla-central opt test jsreftest 1009.11570248
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-1 1914.46666667
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-2 604.758333333
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-3 1359.05
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-4 619.157024793
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-5 658.2
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-browser-chrome 2530.58333333
    Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-other 1124.975
    Ubuntu VM 12.04 x64 mozilla-central opt test reftest 2600.90909091
    Ubuntu VM 12.04 x64 mozilla-central opt test xpcshell 1315.70833333
    WINNT 5.2 mozilla-central build 8063.63013699
    WINNT 5.2 mozilla-central leak test build 9790.52054795
    WINNT 6.1 x86-64 mozilla-central build 8946.48201439
    WINNT 6.1 x86-64 mozilla-central leak test build 8435.95192308
    WINNT 6.2 mozilla-central debug test crashtest 751.873134328
    WINNT 6.2 mozilla-central debug test jetpack 1364.60583942
    WINNT 6.2 mozilla-central debug test jsreftest 1395.36486486
    WINNT 6.2 mozilla-central debug test mochitest-1 2228.87407407
    WINNT 6.2 mozilla-central debug test mochitest-2 1000.60869565
    WINNT 6.2 mozilla-central debug test mochitest-3 2069.69343066
    WINNT 6.2 mozilla-central debug test mochitest-4 603.706766917
    WINNT 6.2 mozilla-central debug test mochitest-5 896.619402985
    WINNT 6.2 mozilla-central debug test mochitest-browser-chrome 5021.21830986
    WINNT 6.2 mozilla-central debug test mochitest-other 1730.02255639
    WINNT 6.2 mozilla-central debug test reftest 2305.29411765
    WINNT 6.2 mozilla-central debug test xpcshell 2103.74814815
    WINNT 6.2 mozilla-central opt test crashtest 460.167883212
    WINNT 6.2 mozilla-central opt test jetpack 831.781021898
    WINNT 6.2 mozilla-central opt test jsreftest 716.274074074
    WINNT 6.2 mozilla-central opt test mochitest-1 1417.82962963
    WINNT 6.2 mozilla-central opt test mochitest-2 563.420382166
    WINNT 6.2 mozilla-central opt test mochitest-3 1308.9084507
    WINNT 6.2 mozilla-central opt test mochitest-4 400.184397163
    WINNT 6.2 mozilla-central opt test mochitest-5 448.872727273
    WINNT 6.2 mozilla-central opt test mochitest-browser-chrome 1979.98518519
    WINNT 6.2 mozilla-central opt test mochitest-metro-chrome 367.215189873
    WINNT 6.2 mozilla-central opt test mochitest-other 997.897810219
    WINNT 6.2 mozilla-central opt test reftest 1064.02941176
    WINNT 6.2 mozilla-central opt test reftest-no-accel 1067.54676259
    WINNT 6.2 mozilla-central opt test xpcshell 1983.23913043
    WINNT 6.2 mozilla-central talos chromez 1286.47101449
    WINNT 6.2 mozilla-central talos dirtypaint 1333.3047619
    WINNT 6.2 mozilla-central talos dromaeojs 1799.29411765
    WINNT 6.2 mozilla-central talos other 882.548611111
    WINNT 6.2 mozilla-central talos rafx 2325.04347826
    WINNT 6.2 mozilla-central talos svgr 1289.19858156
    WINNT 6.2 mozilla-central talos tp5o 992.410071942
    Windows 7 32-bit mozilla-central debug test crashtest 797.466165414
    Windows 7 32-bit mozilla-central debug test jetpack 1354.84210526
    Windows 7 32-bit mozilla-central debug test jsreftest 1578.18045113
    Windows 7 32-bit mozilla-central debug test mochitest-1 2346.2761194
    Windows 7 32-bit mozilla-central debug test mochitest-2 1016.79411765
    Windows 7 32-bit mozilla-central debug test mochitest-3 2117.04511278
    Windows 7 32-bit mozilla-central debug test mochitest-4 638.593984962
    Windows 7 32-bit mozilla-central debug test mochitest-5 907.218045113
    Windows 7 32-bit mozilla-central debug test mochitest-browser-chrome 5023.84172662
    Windows 7 32-bit mozilla-central debug test mochitest-other 1675.0
    Windows 7 32-bit mozilla-central debug test reftest 2529.97744361
    Windows 7 32-bit mozilla-central debug test xpcshell 1346.7443609
    Windows 7 32-bit mozilla-central opt test crashtest 527.288888889
    Windows 7 32-bit mozilla-central opt test jetpack 769.192592593
    Windows 7 32-bit mozilla-central opt test jsreftest 703.711111111
    Windows 7 32-bit mozilla-central opt test mochitest-1 1476.86666667
    Windows 7 32-bit mozilla-central opt test mochitest-2 607.866666667
    Windows 7 32-bit mozilla-central opt test mochitest-3 1322.02205882
    Windows 7 32-bit mozilla-central opt test mochitest-4 426.385185185
    Windows 7 32-bit mozilla-central opt test mochitest-5 538.296296296
    Windows 7 32-bit mozilla-central opt test mochitest-browser-chrome 1784.14074074
    Windows 7 32-bit mozilla-central opt test mochitest-other 962.288888889
    Windows 7 32-bit mozilla-central opt test reftest 1180.63703704
    Windows 7 32-bit mozilla-central opt test reftest-no-accel 1137.61029412
    Windows 7 32-bit mozilla-central opt test xpcshell 1166.88888889
    Windows 7 32-bit mozilla-central talos chromez 1223.25925926
    Windows 7 32-bit mozilla-central talos dirtypaint 1424.21
    Windows 7 32-bit mozilla-central talos dromaeojs 1809.20740741
    Windows 7 32-bit mozilla-central talos other 898.432624113
    Windows 7 32-bit mozilla-central talos rafx 2261.52173913
    Windows 7 32-bit mozilla-central talos svgr 1244.19148936
    Windows 7 32-bit mozilla-central talos tp5o 1070.30147059
    Windows 7 32-bit mozilla-central talos xperf 614.072992701
    Windows XP 32-bit mozilla-central debug test crashtest 764.954887218
    Windows XP 32-bit mozilla-central debug test jetpack 1190.70895522
    Windows XP 32-bit mozilla-central debug test jsreftest 1575.59398496
    Windows XP 32-bit mozilla-central debug test mochitest-1 2158.66917293
    Windows XP 32-bit mozilla-central debug test mochitest-2 939.481203008
    Windows XP 32-bit mozilla-central debug test mochitest-3 1893.02255639
    Windows XP 32-bit mozilla-central debug test mochitest-4 578.879699248
    Windows XP 32-bit mozilla-central debug test mochitest-5 835.45112782
    Windows XP 32-bit mozilla-central debug test mochitest-browser-chrome 4572.21393035
    Windows XP 32-bit mozilla-central debug test mochitest-other 1603.60447761
    Windows XP 32-bit mozilla-central debug test reftest 2385.20895522
    Windows XP 32-bit mozilla-central debug test xpcshell 753.683823529
    Windows XP 32-bit mozilla-central opt test crashtest 413.133333333
    Windows XP 32-bit mozilla-central opt test jetpack 527.794117647
    Windows XP 32-bit mozilla-central opt test jsreftest 659.433823529
    Windows XP 32-bit mozilla-central opt test mochitest-1 1336.01481481
    Windows XP 32-bit mozilla-central opt test mochitest-2 487.474074074
    Windows XP 32-bit mozilla-central opt test mochitest-3 1094.94814815
    Windows XP 32-bit mozilla-central opt test mochitest-4 366.851851852
    Windows XP 32-bit mozilla-central opt test mochitest-5 482.804347826
    Windows XP 32-bit mozilla-central opt test mochitest-browser-chrome 1612.69852941
    Windows XP 32-bit mozilla-central opt test mochitest-other 880.814814815
    Windows XP 32-bit mozilla-central opt test reftest 1002.15441176
    Windows XP 32-bit mozilla-central opt test xpcshell 582.841726619
    Windows XP 32-bit mozilla-central talos chromez 1002.6962963
    Windows XP 32-bit mozilla-central talos dirtypaint 1017.12
    Windows XP 32-bit mozilla-central talos other 646.133802817
    Windows XP 32-bit mozilla-central talos rafx 2196.86956522
    Windows XP 32-bit mozilla-central talos svgr 1235.08965517
    Windows XP 32-bit mozilla-central talos tp5o 957.955555556
    b2g_emulator mozilla-central opt test marionette-webapi 1320.44
    b2g_emulator mozilla-central opt test reftest-1 1709.15254237
    b2g_emulator mozilla-central opt test reftest-10 1719.45299145
    b2g_emulator mozilla-central opt test reftest-2 1692.43103448
    b2g_emulator mozilla-central opt test reftest-3 1709.35042735
    b2g_emulator mozilla-central opt test reftest-4 1841.12931034
    b2g_emulator mozilla-central opt test reftest-5 2088.30172414
    b2g_emulator mozilla-central opt test reftest-6 2040.93965517
    b2g_emulator mozilla-central opt test reftest-7 3393.53448276
    b2g_emulator mozilla-central opt test reftest-8 1516.5
    b2g_emulator mozilla-central opt test reftest-9 1861.65517241
    b2g_emulator_vm mozilla-central opt test crashtest-1 2871.608
    b2g_emulator_vm mozilla-central opt test crashtest-2 2733.39655172
    b2g_emulator_vm mozilla-central opt test crashtest-3 2594.16101695
    b2g_emulator_vm mozilla-central opt test mochitest-1 1827.7394958
    b2g_emulator_vm mozilla-central opt test mochitest-2 1675.73275862
    b2g_emulator_vm mozilla-central opt test mochitest-3 2587.41322314
    b2g_emulator_vm mozilla-central opt test mochitest-4 1699.68067227
    b2g_emulator_vm mozilla-central opt test mochitest-5 1441.98275862
    b2g_emulator_vm mozilla-central opt test mochitest-6 1321.13675214
    b2g_emulator_vm mozilla-central opt test mochitest-7 1246.63247863
    b2g_emulator_vm mozilla-central opt test mochitest-8 1409.75213675
    b2g_emulator_vm mozilla-central opt test mochitest-9 2525.18803419
    b2g_emulator_vm mozilla-central opt test xpcshell 584.495726496
    b2g_mozilla-central_emulator-debug_dep 7271.58208955
    b2g_mozilla-central_emulator-jb-debug_dep 10738.1278195
    b2g_mozilla-central_emulator-jb_dep 10171.2706767
    b2g_mozilla-central_emulator_dep 6716.56716418
    b2g_mozilla-central_hamachi_dep 6306.73134328
    b2g_mozilla-central_hamachi_eng_dep 6204.47692308
    b2g_mozilla-central_helix_dep 6333.37037037
    b2g_mozilla-central_inari_dep 6202.20149254
    b2g_mozilla-central_leo_dep 6319.72180451
    b2g_mozilla-central_leo_eng_dep 6388.43609023
    b2g_mozilla-central_linux32_gecko build 6104.19548872
    b2g_mozilla-central_linux64_gecko build 6063.90225564
    b2g_mozilla-central_macosx64_gecko build 4039.61654135
    b2g_mozilla-central_nexus-4_dep 8721.24358974
    b2g_mozilla-central_unagi_dep 6141.65413534
    b2g_mozilla-central_win32_gecko build 7563.01709402
    b2g_ubuntu64_vm mozilla-central opt test gaia-ui-test 687.007042254
    b2g_ubuntu64_vm mozilla-central opt test gaia-unit 524.288732394

    ============================================

10 thoughts on “254 compute hours per checkin”

  1. Based on a lot of guesses and assumptions, the numbers could be put a bit more readable:
    http://jsbin.com/EmoDAz/1
    It seems that the “rafx” tests, that take about 5 hours, have been removed two weeks ago in bug 914228 (perhaps on another branch).

    1. hi And;

      That grid display is *awesome*. I havent yet sat down to do detailed cross-matching of list with the grid display, but at first skim, it looks fairly accurate… and way more useful then what I had… My giant sequential list was sorted by OS to give some scale, but this grid is *great*. More as I work through the details.

      You literally made my day, thank you.

      tc
      John.

      1. Thanks for the kind reply.
        The script do have bugs (e.g. that “ff-desktop” for “Win64” is shown to be build, but not tested), but it was written last year and had bit-rotted (I think I emailed you at the time). I also hacked one up, that crawled the buildlogs to show a task breakdown. Would be interesting with a breakdown showing if the task i cpu/io/net-bound og just sleeping.

        But at least I think it might be helpful to quickly be able to spot, that e.g. “talos dromaeojs” is not being run for the “opt” version of “ff-desktop” on “windows xp”, that it seems the ubuntu VM is slower than the other machines, or which test-suites are using most time.

        1. hi And;

          I sent you private email – in case you didnt receive it, can you email me at john at oduinn dot com ? I’ve some tweaks to suggest and would like to put together a blog post just about your grid… its super helpful!

          Thanks
          John.

  2. Can we convert that to an AWS cost, to put a dollar figure on the amount of money a missed DONTBUILD costs?

    I don’t know what instances you are using, and whether we have a volume discount, but if they were Medium Standard at $0.12 an hour, that’s $30.48…

    Gerv

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.