The rate of checkins continues to rise rapidly. The volume of builds+tests run per checkin is also increasing rapidly. The combination of these two increases multiplied together is an impressive rate of growth.
How many hours of builds and tests do we run per commit?
- 254 compute hours = ~10.5 compute *days* (sep2013)
- 137 compute hours = ~5.7 compute *days* (aug2012)
- 110 compute hours = ~4.6 compute *days*(jan2012)
- ~40 compute hours = ~1.6 compute *days*(2009)
10.5 compute *days* of builds and tests per checkin is a lot – especially when we average one checkin every ~6 minutes last month, and handle a checkin every 4minutes during peak times. Put another way, that’s an average of ~6.9 compute years of builds and tests being run per day, or 213 compute *years* for the month of August. Mind boggling.
If you look at the “wait times” posts in dev.tree-management, you’ll see that systems we run on AWS are scaling nicely to handle the load, and we continue to move production load over to AWS for nativefennec / B2G desktop builds / linux desktop …. and then recycle existing inhouse hardware to help with inhouse capacity for win32 and win64.
Some of this growth is good (new builds, new test suites).
But some of growth is bad (consistently failing tests; obsolete tests; builds that are no longer needed)… and this slows us all down.
You can help.
- When you push to tryserver, choose wisely. If you need “all”, do it. But ask yourself… do you really need all?
- If you know of any build/test that is no longer needed, let us know.
- If you know of any build/test being hidden on tbpl.m.o because it constantly fails let us know or file a bug. We’ll disable the job to stop wasting CPU time, which will help everyone else’s jobs get processed more quickly. As needed, we’ll also file bugs to track fixing the problem and then reenable once the build/test is running green. While any one given build or test suite might not seem like a lot of time to save, we run all these jobs literally thousands of times per day.
Every little saving helps makes things even faster for everyone!
Thank you!
John.
===========================================
List of builds and tests run (with duration of job in seconds)
Android 2.2 Armv6 Tegra mozilla-central opt test crashtest 1205.62037037
Android 2.2 Armv6 Tegra mozilla-central opt test jsreftest-1 1961.08333333
Android 2.2 Armv6 Tegra mozilla-central opt test jsreftest-2 1996.19626168
Android 2.2 Armv6 Tegra mozilla-central opt test jsreftest-3 2043.25233645
Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-1 1039.10909091
Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-2 912.092592593
Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-3 1291.40740741
Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-4 1066.77570093
Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-5 979.252336449
Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-6 1103.2
Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-7 975.355140187
Android 2.2 Armv6 Tegra mozilla-central opt test mochitest-8 1050.5
Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-1 2538.24299065
Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-2 2849.25
Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-3 2632.22429907
Android 2.2 Armv6 Tegra mozilla-central opt test plain-reftest-4 2589.43925234
Android 2.2 Armv6 Tegra mozilla-central opt test robocop-1 1494.53271028
Android 2.2 Armv6 Tegra mozilla-central opt test robocop-2 2251.78504673
Android 2.2 Armv6 Tegra mozilla-central opt test xpcshell 2289.25688073
Android 2.2 Armv6 mozilla-central build 4112.26470588
Android 2.2 Debug mozilla-central build 4612.8627451
Android 2.2 Tegra mozilla-central opt test crashtest 1198.16666667
Android 2.2 Tegra mozilla-central opt test jsreftest-1 1905.94545455
Android 2.2 Tegra mozilla-central opt test jsreftest-2 1798.31192661
Android 2.2 Tegra mozilla-central opt test jsreftest-3 2102.10091743
Android 2.2 Tegra mozilla-central opt test mochitest-1 1014.01834862
Android 2.2 Tegra mozilla-central opt test mochitest-2 893.449541284
Android 2.2 Tegra mozilla-central opt test mochitest-3 1270.65454545
Android 2.2 Tegra mozilla-central opt test mochitest-4 981.158878505
Android 2.2 Tegra mozilla-central opt test mochitest-5 938.247706422
Android 2.2 Tegra mozilla-central opt test mochitest-6 1128.26126126
Android 2.2 Tegra mozilla-central opt test mochitest-7 988.91588785
Android 2.2 Tegra mozilla-central opt test mochitest-8 997.898148148
Android 2.2 Tegra mozilla-central opt test plain-reftest-1 2527.41121495
Android 2.2 Tegra mozilla-central opt test plain-reftest-2 2815.85185185
Android 2.2 Tegra mozilla-central opt test plain-reftest-3 2623.35514019
Android 2.2 Tegra mozilla-central opt test plain-reftest-4 2582.18691589
Android 2.2 Tegra mozilla-central opt test robocop-1 1542.2293578
Android 2.2 Tegra mozilla-central opt test robocop-2 2233.76576577
Android 2.2 Tegra mozilla-central opt test xpcshell 2308.20720721
Android 2.2 Tegra mozilla-central talos remote-tcanvasmark 886.648148148
Android 2.2 Tegra mozilla-central talos remote-tp4m_nochrome 1147.54205607
Android 2.2 Tegra mozilla-central talos remote-trobocheck2 983.117117117
Android 2.2 Tegra mozilla-central talos remote-trobopan 1006.74074074
Android 2.2 Tegra mozilla-central talos remote-troboprovider 1280.57009346
Android 2.2 Tegra mozilla-central talos remote-ts 705.117647059
Android 2.2 Tegra mozilla-central talos remote-tspaint 688.58974359
Android 2.2 Tegra mozilla-central talos remote-tsvg 751.75
Android 2.2 Tegra mozilla-central talos remote-tsvgx 1428.88073394
Android 2.2 mozilla-central build 4251.46078431
Android 2.2 no-ionmonkey Tegra mozilla-central opt test jsreftest-1 2015.42045455
Android 2.2 no-ionmonkey Tegra mozilla-central opt test jsreftest-2 1812.60227273
Android 2.2 no-ionmonkey Tegra mozilla-central opt test jsreftest-3 2138.88636364
Android 2.2 no-ionmonkey mozilla-central build 3993.67647059
Android 4.0 Panda mozilla-central opt test crashtest 1413.98843931
Android 4.0 Panda mozilla-central opt test jsreftest-1 1584.75842697
Android 4.0 Panda mozilla-central opt test jsreftest-2 1582.75147929
Android 4.0 Panda mozilla-central opt test jsreftest-3 1566.4566474
Android 4.0 Panda mozilla-central opt test mochitest-1 1216.17679558
Android 4.0 Panda mozilla-central opt test mochitest-2 1098.2754491
Android 4.0 Panda mozilla-central opt test mochitest-3 1410.21387283
Android 4.0 Panda mozilla-central opt test mochitest-4 1197.27906977
Android 4.0 Panda mozilla-central opt test mochitest-5 1212.03888889
Android 4.0 Panda mozilla-central opt test mochitest-6 1166.41489362
Android 4.0 Panda mozilla-central opt test mochitest-7 1209.33532934
Android 4.0 Panda mozilla-central opt test mochitest-8 1140.14754098
Android 4.0 Panda mozilla-central opt test mochitest-gl 1116.70414201
Android 4.0 Panda mozilla-central opt test robocop-1 1288.49710983
Android 4.0 Panda mozilla-central opt test robocop-2 1739.20833333
Android 4.0 Panda mozilla-central opt test robocop-3 1587.77192982
Android 4.0 Panda mozilla-central opt test xpcshell 1989.28021978
Android 4.0 Panda mozilla-central talos remote-tcanvasmark 2836.71018277
Android 4.0 Panda mozilla-central talos remote-tp4m_nochrome 1484.12068966
Android 4.0 Panda mozilla-central talos remote-trobocheck2 1168.93292683
Android 4.0 Panda mozilla-central talos remote-trobopan 1077.8969697
Android 4.0 Panda mozilla-central talos remote-troboprovider 1370.75903614
Android 4.0 Panda mozilla-central talos remote-ts 850.294573643
Android 4.0 Panda mozilla-central talos remote-tspaint 933.782608696
Android 4.0 Panda mozilla-central talos remote-tsvg 960.408333333
Android 4.0 Panda mozilla-central talos remote-tsvgx 1608.24
Android 4.2 x86 Emulator mozilla-central opt test androidx86-set-1 3556.85074627
Android 4.2 x86 Emulator mozilla-central opt test androidx86-set-2 2033.921875
Android 4.2 x86 mozilla-central build 4230.7254902
Android Armv6 Tegra 250 mozilla-central opt test crashtest 1199.5
Android Armv6 Tegra 250 mozilla-central opt test jsreftest-1 1950.375
Android Armv6 Tegra 250 mozilla-central opt test jsreftest-2 1819.0625
Android Armv6 Tegra 250 mozilla-central opt test jsreftest-3 2357.21875
Android Armv6 Tegra 250 mozilla-central opt test mochitest-1 1026.875
Android Armv6 Tegra 250 mozilla-central opt test mochitest-2 893.21875
Android Armv6 Tegra 250 mozilla-central opt test mochitest-3 1279.72727273
Android Armv6 Tegra 250 mozilla-central opt test mochitest-4 1037.84375
Android Armv6 Tegra 250 mozilla-central opt test mochitest-5 986.28125
Android Armv6 Tegra 250 mozilla-central opt test mochitest-6 1083.5625
Android Armv6 Tegra 250 mozilla-central opt test mochitest-7 1004.09090909
Android Armv6 Tegra 250 mozilla-central opt test mochitest-8 1019.36363636
Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-1 2711.9375
Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-2 3252.09375
Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-3 2679.6875
Android Armv6 Tegra 250 mozilla-central opt test plain-reftest-4 2923.84375
Android Armv6 Tegra 250 mozilla-central opt test robocop-1 1455.78125
Android Armv6 Tegra 250 mozilla-central opt test robocop-2 2155.1875
Android Armv6 Tegra 250 mozilla-central opt test xpcshell 2191.09375
Android Armv6 mozilla-central build 4099.38709677
Android Debug mozilla-central build 4822.32258065
Android Tegra 250 mozilla-central opt test crashtest 1161.53125
Android Tegra 250 mozilla-central opt test jsreftest-1 1902.25
Android Tegra 250 mozilla-central opt test jsreftest-2 1873.1875
Android Tegra 250 mozilla-central opt test jsreftest-3 2000.6875
Android Tegra 250 mozilla-central opt test mochitest-1 978.96875
Android Tegra 250 mozilla-central opt test mochitest-2 841.90625
Android Tegra 250 mozilla-central opt test mochitest-3 1245.6875
Android Tegra 250 mozilla-central opt test mochitest-4 983.1875
Android Tegra 250 mozilla-central opt test mochitest-5 956.84375
Android Tegra 250 mozilla-central opt test mochitest-6 1055.65625
Android Tegra 250 mozilla-central opt test mochitest-7 960.6875
Android Tegra 250 mozilla-central opt test mochitest-8 942.242424242
Android Tegra 250 mozilla-central opt test plain-reftest-1 2731.84375
Android Tegra 250 mozilla-central opt test plain-reftest-2 3308.125
Android Tegra 250 mozilla-central opt test plain-reftest-3 2618.5
Android Tegra 250 mozilla-central opt test plain-reftest-4 2881.09375
Android Tegra 250 mozilla-central opt test robocop-1 1491.9375
Android Tegra 250 mozilla-central opt test robocop-2 2182.28125
Android Tegra 250 mozilla-central opt test xpcshell 2204.3125
Android Tegra 250 mozilla-central talos remote-tcanvasmark 869.59375
Android Tegra 250 mozilla-central talos remote-tp4m_nochrome 2693.90625
Android Tegra 250 mozilla-central talos remote-trobocheck2 959.571428571
Android Tegra 250 mozilla-central talos remote-trobopan 982.6875
Android Tegra 250 mozilla-central talos remote-troboprovider 1245.3125
Android Tegra 250 mozilla-central talos remote-ts 689.40625
Android Tegra 250 mozilla-central talos remote-tsvg 748.375
Android Tegra 250 mozilla-central talos remote-tsvgx 1422.21875
Android X86 mozilla-central build 4151.77419355
Android mozilla-central build 4090.67741935
Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-1 1847.66666667
Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-2 1916.77777778
Android no-ionmonkey Tegra 250 mozilla-central opt test jsreftest-3 1846.33333333
Android no-ionmonkey mozilla-central build 4080.74193548
Linux mozilla-central build 6547.11278195
Linux mozilla-central leak test build 8033.2556391
Linux x86-64 mozilla-central asan build 7470.8880597
Linux x86-64 mozilla-central build 6484.28571429
Linux x86-64 mozilla-central debug asan build 12863.7744361
Linux x86-64 mozilla-central debug static analysis build 5324.42105263
Linux x86-64 mozilla-central hsts preload update 861.0
Linux x86-64 mozilla-central leak test build 7858.93233083
Linux x86-64 mozilla-central leak test spidermonkey_tier_1-generational build 2355.7826087
Linux x86-64 mozilla-central leak test spidermonkey_tier_1-rootanalysis build 3213.34782609
OS X 10.7 64-bit mozilla-central leak test build 5344.62406015
OS X 10.7 mozilla-central build 6902.86666667
Rev3 Fedora 12 mozilla-central debug test mochitest-browser-chrome 6873.0
Rev3 Fedora 12×64 mozilla-central debug test mochitest-browser-chrome 5713.30952381
Rev4 MacOSX Lion 10.7 mozilla-central debug test crashtest 912.038167939
Rev4 MacOSX Lion 10.7 mozilla-central debug test jetpack 2180.04511278
Rev4 MacOSX Lion 10.7 mozilla-central debug test jsreftest 1628.6969697
Rev4 MacOSX Lion 10.7 mozilla-central debug test marionette 402.893129771
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-1 2839.85714286
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-2 1178.44274809
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-3 2682.72932331
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-4 700.62406015
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-5 1240.26865672
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-browser-chrome 8036.58823529
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-other 2266.85925926
Rev4 MacOSX Lion 10.7 mozilla-central debug test reftest 3225.32824427
Rev4 MacOSX Lion 10.7 mozilla-central debug test xpcshell 1349.66412214
Rev4 MacOSX Lion 10.7 mozilla-central opt test crashtest 554.69005848
Rev4 MacOSX Lion 10.7 mozilla-central opt test jetpack 704.111111111
Rev4 MacOSX Lion 10.7 mozilla-central opt test jsreftest 724.851612903
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-1 1441.04545455
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-2 555.107594937
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-3 1205.51298701
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-4 378.549019608
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-5 588.419354839
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-browser-chrome 2229.45454545
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-other 966.287581699
Rev4 MacOSX Lion 10.7 mozilla-central opt test reftest 1754.82051282
Rev4 MacOSX Lion 10.7 mozilla-central opt test xpcshell 792.214285714
Rev4 MacOSX Lion 10.7 mozilla-central talos chromez 989.668831169
Rev4 MacOSX Lion 10.7 mozilla-central talos dirtypaint 780.272727273
Rev4 MacOSX Lion 10.7 mozilla-central talos dromaeojs 1594.50649351
Rev4 MacOSX Lion 10.7 mozilla-central talos other 630.180645161
Rev4 MacOSX Lion 10.7 mozilla-central talos rafx 2532.88888889
Rev4 MacOSX Lion 10.7 mozilla-central talos svgr 1694.70414201
Rev4 MacOSX Lion 10.7 mozilla-central talos tp5o 1107.99363057
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test crashtest 846.484848485
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jetpack 1911.14503817
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jsreftest 1486.49618321
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test marionette 350.969465649
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-1 2757.54961832
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-2 1085.3030303
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-3 2481.8778626
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-4 697.496183206
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-5 1191.85496183
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-browser-chrome 7223.29850746
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-other 2115.2406015
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test reftest 3124.41666667
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test xpcshell 1666.88986784
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test crashtest 542.647058824
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jetpack 713.529411765
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jsreftest 689.921568627
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-1 1469.20779221
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-2 527.124183007
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-3 1205.6474359
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-4 378.908496732
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-5 610.248366013
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-browser-chrome 2237.68387097
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-other 965.201298701
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test reftest 1799.96732026
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test xpcshell 771.849673203
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos chromez 970.058441558
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dirtypaint 795.846846847
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dromaeojs 1586.25806452
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos other 602.039215686
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos rafx 2507.92592593
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos svgr 1713.57831325
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos tp5o 1030.33766234
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test crashtest 732.175572519
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test jetpack 1565.57251908
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test jsreftest 1365.38167939
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test marionette 370.757575758
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-1 2291.09848485
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-2 955.816793893
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-3 2098.70992366
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-4 588.038167939
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-5 964.083969466
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-browser-chrome 6020.40740741
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-other 1762.40909091
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test reftest 2338.97709924
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test xpcshell 803.992366412
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test crashtest 470.823529412
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test jetpack 578.065359477
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test jsreftest 701.444444444
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-1 1303.61290323
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-2 529.376623377
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-3 1114.85897436
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-4 369.169934641
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-5 533.967320261
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-browser-chrome 1726.04575163
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test mochitest-other 892.810457516
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test reftest 1137.37662338
Rev5 MacOSX Mountain Lion 10.8 mozilla-central opt test xpcshell 598.797385621
Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos chromez 1026.55555556
Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos dirtypaint 810.513513514
Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos dromaeojs 1576.92810458
Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos other 613.409090909
Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos rafx 2286.77777778
Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos svgr 1237.26829268
Rev5 MacOSX Mountain Lion 10.8 mozilla-central talos tp5o 869.503267974
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test crashtest 961.595744681
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test jetpack 1461.6993007
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test jsreftest 2009.38297872
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-1 2748.66901408
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-2 1187.30496454
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-3 2160.24647887
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-4 1040.14084507
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-5 1176.88652482
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-browser-chrome 4652.4295302
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-other 2144.56737589
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test reftest 3649.06382979
Ubuntu ASAN VM 12.04 x64 mozilla-central opt test xpcshell 2459.59859155
Ubuntu HW 12.04 mozilla-central talos chromez 952.193277311
Ubuntu HW 12.04 mozilla-central talos dirtypaint 738.035294118
Ubuntu HW 12.04 mozilla-central talos dromaeojs 1516.75833333
Ubuntu HW 12.04 mozilla-central talos other 589.775
Ubuntu HW 12.04 mozilla-central talos rafx 2280.65217391
Ubuntu HW 12.04 mozilla-central talos svgr 1377.856
Ubuntu HW 12.04 mozilla-central talos tp5o 893.647058824
Ubuntu HW 12.04 x64 mozilla-central talos chromez 964.446280992
Ubuntu HW 12.04 x64 mozilla-central talos dirtypaint 730.454545455
Ubuntu HW 12.04 x64 mozilla-central talos dromaeojs 1521.45
Ubuntu HW 12.04 x64 mozilla-central talos other 560.719008264
Ubuntu HW 12.04 x64 mozilla-central talos rafx 2237.69565217
Ubuntu HW 12.04 x64 mozilla-central talos svgr 1266.752
Ubuntu HW 12.04 x64 mozilla-central talos tp5o 882.860655738
Ubuntu VM 12.04 mozilla-central debug test crashtest 1458.65546218
Ubuntu VM 12.04 mozilla-central debug test jetpack 1894.76470588
Ubuntu VM 12.04 mozilla-central debug test jsreftest 2499.36134454
Ubuntu VM 12.04 mozilla-central debug test marionette 411.090909091
Ubuntu VM 12.04 mozilla-central debug test mochitest-1 4014.19327731
Ubuntu VM 12.04 mozilla-central debug test mochitest-2 1359.38655462
Ubuntu VM 12.04 mozilla-central debug test mochitest-3 2995.5210084
Ubuntu VM 12.04 mozilla-central debug test mochitest-4 1133.36666667
Ubuntu VM 12.04 mozilla-central debug test mochitest-5 1570.45378151
Ubuntu VM 12.04 mozilla-central debug test mochitest-other 2739.95798319
Ubuntu VM 12.04 mozilla-central debug test reftest 6202.39516129
Ubuntu VM 12.04 mozilla-central debug test xpcshell 3091.74380165
Ubuntu VM 12.04 mozilla-central opt test crashtest 584.541666667
Ubuntu VM 12.04 mozilla-central opt test crashtest-ipc 648.475
Ubuntu VM 12.04 mozilla-central opt test jetpack 778.710743802
Ubuntu VM 12.04 mozilla-central opt test jsreftest 1359.88235294
Ubuntu VM 12.04 mozilla-central opt test mochitest-1 2331.575
Ubuntu VM 12.04 mozilla-central opt test mochitest-2 680.358333333
Ubuntu VM 12.04 mozilla-central opt test mochitest-3 1485.38655462
Ubuntu VM 12.04 mozilla-central opt test mochitest-4 682.487394958
Ubuntu VM 12.04 mozilla-central opt test mochitest-5 939.621848739
Ubuntu VM 12.04 mozilla-central opt test mochitest-browser-chrome 3086.47058824
Ubuntu VM 12.04 mozilla-central opt test mochitest-other 1400.20168067
Ubuntu VM 12.04 mozilla-central opt test reftest 4367.49166667
Ubuntu VM 12.04 mozilla-central opt test reftest-ipc 239.924369748
Ubuntu VM 12.04 mozilla-central opt test reftest-no-accel 4395.63636364
Ubuntu VM 12.04 mozilla-central opt test xpcshell 1769.68067227
Ubuntu VM 12.04 x64 mozilla-central debug test crashtest 995.388429752
Ubuntu VM 12.04 x64 mozilla-central debug test jetpack 1542.07438017
Ubuntu VM 12.04 x64 mozilla-central debug test jsreftest 1878.79166667
Ubuntu VM 12.04 x64 mozilla-central debug test marionette 419.858333333
Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-1 3084.53333333
Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-2 1104.36666667
Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-3 2377.53333333
Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-4 934.818181818
Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-5 1111.96666667
Ubuntu VM 12.04 x64 mozilla-central debug test mochitest-other 2087.90833333
Ubuntu VM 12.04 x64 mozilla-central debug test reftest 3202.85833333
Ubuntu VM 12.04 x64 mozilla-central debug test xpcshell 2222.9338843
Ubuntu VM 12.04 x64 mozilla-central opt test crashtest 499.308333333
Ubuntu VM 12.04 x64 mozilla-central opt test jetpack 743.258333333
Ubuntu VM 12.04 x64 mozilla-central opt test jsreftest 1009.11570248
Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-1 1914.46666667
Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-2 604.758333333
Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-3 1359.05
Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-4 619.157024793
Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-5 658.2
Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-browser-chrome 2530.58333333
Ubuntu VM 12.04 x64 mozilla-central opt test mochitest-other 1124.975
Ubuntu VM 12.04 x64 mozilla-central opt test reftest 2600.90909091
Ubuntu VM 12.04 x64 mozilla-central opt test xpcshell 1315.70833333
WINNT 5.2 mozilla-central build 8063.63013699
WINNT 5.2 mozilla-central leak test build 9790.52054795
WINNT 6.1 x86-64 mozilla-central build 8946.48201439
WINNT 6.1 x86-64 mozilla-central leak test build 8435.95192308
WINNT 6.2 mozilla-central debug test crashtest 751.873134328
WINNT 6.2 mozilla-central debug test jetpack 1364.60583942
WINNT 6.2 mozilla-central debug test jsreftest 1395.36486486
WINNT 6.2 mozilla-central debug test mochitest-1 2228.87407407
WINNT 6.2 mozilla-central debug test mochitest-2 1000.60869565
WINNT 6.2 mozilla-central debug test mochitest-3 2069.69343066
WINNT 6.2 mozilla-central debug test mochitest-4 603.706766917
WINNT 6.2 mozilla-central debug test mochitest-5 896.619402985
WINNT 6.2 mozilla-central debug test mochitest-browser-chrome 5021.21830986
WINNT 6.2 mozilla-central debug test mochitest-other 1730.02255639
WINNT 6.2 mozilla-central debug test reftest 2305.29411765
WINNT 6.2 mozilla-central debug test xpcshell 2103.74814815
WINNT 6.2 mozilla-central opt test crashtest 460.167883212
WINNT 6.2 mozilla-central opt test jetpack 831.781021898
WINNT 6.2 mozilla-central opt test jsreftest 716.274074074
WINNT 6.2 mozilla-central opt test mochitest-1 1417.82962963
WINNT 6.2 mozilla-central opt test mochitest-2 563.420382166
WINNT 6.2 mozilla-central opt test mochitest-3 1308.9084507
WINNT 6.2 mozilla-central opt test mochitest-4 400.184397163
WINNT 6.2 mozilla-central opt test mochitest-5 448.872727273
WINNT 6.2 mozilla-central opt test mochitest-browser-chrome 1979.98518519
WINNT 6.2 mozilla-central opt test mochitest-metro-chrome 367.215189873
WINNT 6.2 mozilla-central opt test mochitest-other 997.897810219
WINNT 6.2 mozilla-central opt test reftest 1064.02941176
WINNT 6.2 mozilla-central opt test reftest-no-accel 1067.54676259
WINNT 6.2 mozilla-central opt test xpcshell 1983.23913043
WINNT 6.2 mozilla-central talos chromez 1286.47101449
WINNT 6.2 mozilla-central talos dirtypaint 1333.3047619
WINNT 6.2 mozilla-central talos dromaeojs 1799.29411765
WINNT 6.2 mozilla-central talos other 882.548611111
WINNT 6.2 mozilla-central talos rafx 2325.04347826
WINNT 6.2 mozilla-central talos svgr 1289.19858156
WINNT 6.2 mozilla-central talos tp5o 992.410071942
Windows 7 32-bit mozilla-central debug test crashtest 797.466165414
Windows 7 32-bit mozilla-central debug test jetpack 1354.84210526
Windows 7 32-bit mozilla-central debug test jsreftest 1578.18045113
Windows 7 32-bit mozilla-central debug test mochitest-1 2346.2761194
Windows 7 32-bit mozilla-central debug test mochitest-2 1016.79411765
Windows 7 32-bit mozilla-central debug test mochitest-3 2117.04511278
Windows 7 32-bit mozilla-central debug test mochitest-4 638.593984962
Windows 7 32-bit mozilla-central debug test mochitest-5 907.218045113
Windows 7 32-bit mozilla-central debug test mochitest-browser-chrome 5023.84172662
Windows 7 32-bit mozilla-central debug test mochitest-other 1675.0
Windows 7 32-bit mozilla-central debug test reftest 2529.97744361
Windows 7 32-bit mozilla-central debug test xpcshell 1346.7443609
Windows 7 32-bit mozilla-central opt test crashtest 527.288888889
Windows 7 32-bit mozilla-central opt test jetpack 769.192592593
Windows 7 32-bit mozilla-central opt test jsreftest 703.711111111
Windows 7 32-bit mozilla-central opt test mochitest-1 1476.86666667
Windows 7 32-bit mozilla-central opt test mochitest-2 607.866666667
Windows 7 32-bit mozilla-central opt test mochitest-3 1322.02205882
Windows 7 32-bit mozilla-central opt test mochitest-4 426.385185185
Windows 7 32-bit mozilla-central opt test mochitest-5 538.296296296
Windows 7 32-bit mozilla-central opt test mochitest-browser-chrome 1784.14074074
Windows 7 32-bit mozilla-central opt test mochitest-other 962.288888889
Windows 7 32-bit mozilla-central opt test reftest 1180.63703704
Windows 7 32-bit mozilla-central opt test reftest-no-accel 1137.61029412
Windows 7 32-bit mozilla-central opt test xpcshell 1166.88888889
Windows 7 32-bit mozilla-central talos chromez 1223.25925926
Windows 7 32-bit mozilla-central talos dirtypaint 1424.21
Windows 7 32-bit mozilla-central talos dromaeojs 1809.20740741
Windows 7 32-bit mozilla-central talos other 898.432624113
Windows 7 32-bit mozilla-central talos rafx 2261.52173913
Windows 7 32-bit mozilla-central talos svgr 1244.19148936
Windows 7 32-bit mozilla-central talos tp5o 1070.30147059
Windows 7 32-bit mozilla-central talos xperf 614.072992701
Windows XP 32-bit mozilla-central debug test crashtest 764.954887218
Windows XP 32-bit mozilla-central debug test jetpack 1190.70895522
Windows XP 32-bit mozilla-central debug test jsreftest 1575.59398496
Windows XP 32-bit mozilla-central debug test mochitest-1 2158.66917293
Windows XP 32-bit mozilla-central debug test mochitest-2 939.481203008
Windows XP 32-bit mozilla-central debug test mochitest-3 1893.02255639
Windows XP 32-bit mozilla-central debug test mochitest-4 578.879699248
Windows XP 32-bit mozilla-central debug test mochitest-5 835.45112782
Windows XP 32-bit mozilla-central debug test mochitest-browser-chrome 4572.21393035
Windows XP 32-bit mozilla-central debug test mochitest-other 1603.60447761
Windows XP 32-bit mozilla-central debug test reftest 2385.20895522
Windows XP 32-bit mozilla-central debug test xpcshell 753.683823529
Windows XP 32-bit mozilla-central opt test crashtest 413.133333333
Windows XP 32-bit mozilla-central opt test jetpack 527.794117647
Windows XP 32-bit mozilla-central opt test jsreftest 659.433823529
Windows XP 32-bit mozilla-central opt test mochitest-1 1336.01481481
Windows XP 32-bit mozilla-central opt test mochitest-2 487.474074074
Windows XP 32-bit mozilla-central opt test mochitest-3 1094.94814815
Windows XP 32-bit mozilla-central opt test mochitest-4 366.851851852
Windows XP 32-bit mozilla-central opt test mochitest-5 482.804347826
Windows XP 32-bit mozilla-central opt test mochitest-browser-chrome 1612.69852941
Windows XP 32-bit mozilla-central opt test mochitest-other 880.814814815
Windows XP 32-bit mozilla-central opt test reftest 1002.15441176
Windows XP 32-bit mozilla-central opt test xpcshell 582.841726619
Windows XP 32-bit mozilla-central talos chromez 1002.6962963
Windows XP 32-bit mozilla-central talos dirtypaint 1017.12
Windows XP 32-bit mozilla-central talos other 646.133802817
Windows XP 32-bit mozilla-central talos rafx 2196.86956522
Windows XP 32-bit mozilla-central talos svgr 1235.08965517
Windows XP 32-bit mozilla-central talos tp5o 957.955555556
b2g_emulator mozilla-central opt test marionette-webapi 1320.44
b2g_emulator mozilla-central opt test reftest-1 1709.15254237
b2g_emulator mozilla-central opt test reftest-10 1719.45299145
b2g_emulator mozilla-central opt test reftest-2 1692.43103448
b2g_emulator mozilla-central opt test reftest-3 1709.35042735
b2g_emulator mozilla-central opt test reftest-4 1841.12931034
b2g_emulator mozilla-central opt test reftest-5 2088.30172414
b2g_emulator mozilla-central opt test reftest-6 2040.93965517
b2g_emulator mozilla-central opt test reftest-7 3393.53448276
b2g_emulator mozilla-central opt test reftest-8 1516.5
b2g_emulator mozilla-central opt test reftest-9 1861.65517241
b2g_emulator_vm mozilla-central opt test crashtest-1 2871.608
b2g_emulator_vm mozilla-central opt test crashtest-2 2733.39655172
b2g_emulator_vm mozilla-central opt test crashtest-3 2594.16101695
b2g_emulator_vm mozilla-central opt test mochitest-1 1827.7394958
b2g_emulator_vm mozilla-central opt test mochitest-2 1675.73275862
b2g_emulator_vm mozilla-central opt test mochitest-3 2587.41322314
b2g_emulator_vm mozilla-central opt test mochitest-4 1699.68067227
b2g_emulator_vm mozilla-central opt test mochitest-5 1441.98275862
b2g_emulator_vm mozilla-central opt test mochitest-6 1321.13675214
b2g_emulator_vm mozilla-central opt test mochitest-7 1246.63247863
b2g_emulator_vm mozilla-central opt test mochitest-8 1409.75213675
b2g_emulator_vm mozilla-central opt test mochitest-9 2525.18803419
b2g_emulator_vm mozilla-central opt test xpcshell 584.495726496
b2g_mozilla-central_emulator-debug_dep 7271.58208955
b2g_mozilla-central_emulator-jb-debug_dep 10738.1278195
b2g_mozilla-central_emulator-jb_dep 10171.2706767
b2g_mozilla-central_emulator_dep 6716.56716418
b2g_mozilla-central_hamachi_dep 6306.73134328
b2g_mozilla-central_hamachi_eng_dep 6204.47692308
b2g_mozilla-central_helix_dep 6333.37037037
b2g_mozilla-central_inari_dep 6202.20149254
b2g_mozilla-central_leo_dep 6319.72180451
b2g_mozilla-central_leo_eng_dep 6388.43609023
b2g_mozilla-central_linux32_gecko build 6104.19548872
b2g_mozilla-central_linux64_gecko build 6063.90225564
b2g_mozilla-central_macosx64_gecko build 4039.61654135
b2g_mozilla-central_nexus-4_dep 8721.24358974
b2g_mozilla-central_unagi_dep 6141.65413534
b2g_mozilla-central_win32_gecko build 7563.01709402
b2g_ubuntu64_vm mozilla-central opt test gaia-ui-test 687.007042254
b2g_ubuntu64_vm mozilla-central opt test gaia-unit 524.288732394============================================
Every 6 mins someone adds new code to Firefox, each consumes 10.5 days of compute time running tests. http://t.co/YPZhCKiJgn
drop xp
Toutes les 6 minutes, quelqu’un ajoute du code à Firefox, déclenchant 254 heures-machine de tests http://t.co/z2SkaLgDHO
It’s easy to guess that Mozilla’s repo gets a large number of checkins per day. But one checkin every 6 minutes? Wow. http://t.co/iwAwfZSB1D
Based on a lot of guesses and assumptions, the numbers could be put a bit more readable:
http://jsbin.com/EmoDAz/1
It seems that the “rafx” tests, that take about 5 hours, have been removed two weeks ago in bug 914228 (perhaps on another branch).
hi And;
That grid display is awesome. I havent yet sat down to do detailed cross-matching of list with the grid display, but at first skim, it looks fairly accurate… and way more useful then what I had… My giant sequential list was sorted by OS to give some scale, but this grid is great. More as I work through the details.
You literally made my day, thank you.
tc
John.
Thanks for the kind reply.
The script do have bugs (e.g. that “ff-desktop” for “Win64” is shown to be build, but not tested), but it was written last year and had bit-rotted (I think I emailed you at the time). I also hacked one up, that crawled the buildlogs to show a task breakdown. Would be interesting with a breakdown showing if the task i cpu/io/net-bound og just sleeping.
But at least I think it might be helpful to quickly be able to spot, that e.g. “talos dromaeojs” is not being run for the “opt” version of “ff-desktop” on “windows xp”, that it seems the ubuntu VM is slower than the other machines, or which test-suites are using most time.
hi And;
I sent you private email – in case you didnt receive it, can you email me at john at oduinn dot com ? I’ve some tweaks to suggest and would like to put together a blog post just about your grid… its super helpful!
Thanks
John.
Can we convert that to an AWS cost, to put a dollar figure on the amount of money a missed DONTBUILD costs?
I don’t know what instances you are using, and whether we have a volume discount, but if they were Medium Standard at $0.12 an hour, that’s $30.48…
Gerv
[…] my last post about our compute-load-per-checkin, I received a email that made me sit up and smile. Andershol had “a quick script” that quickly […]