137 hours compute hours every ~6 minutes

10 Comments

Followup from today’s Dev platform meeting.

We talk a lot about how the number of checkins continue to rise – its always great to see developers being able to do more work – and the “Infrastructure load” series of posts is one way to measure that. However, its worth noting that the amount of work we trigger per checkin also continues to rise.

Some of this is good (new builds, new test suites), but some of this is bad (consistently failing tests; obsolete tests).

How many hours of builds and tests do we run per commit?

  • In 2009, each commit triggered ~40 compute hours
  • In Jan 2012, each commit triggered 110 compute hours
  • In Aug 2012, each commit triggered 137 compute hours

137 hours of builds and tests per checkin is a lot – especially when we do a checkin every ~6 minutes. Put another way, that’s ~3.5 years of builds and tests being run per day.

If you look at the “wait times” posts in dev.tree-management, you’ll see that we’re having a tough time keeping up. We’re busily venting excess production load over to AWS for nativefennec / B2G desktop builds / linux desktop …. and are then recycling existing hardware to help grow inhouse capacity for win32 and win64.

But you can help.

Please file a bug if you know of:

  • any build or test that is being run but no longer needed OR

  • any build or test being hidden on tbpl.m.o because it constantly fails

We’ll disable the job to stop wasting CPU time, which will help everyone’s wait times. As needed, we’ll also file bugs to track fixing the problem and then reenable once the build/test is running green.

While any one given test suite might not seem like a lot of time to save, we run all these suites literally thousands of times per day. Every little saving helps makes things faster for everyone!

Thank you!
===========================================
Full list (with duration of job in seconds)
Android Armv6 mozilla-central build 1523.29213483
Android Debug mozilla-central build 1461.34090909
Android Tegra 250 mozilla-central opt test crashtest-2 1509.39215686
Android Tegra 250 mozilla-central opt test crashtest-3 1627.30769231
Android Tegra 250 mozilla-central opt test jsreftest-1 1635.7047619
Android Tegra 250 mozilla-central opt test jsreftest-2 2023.08181818
Android Tegra 250 mozilla-central opt test jsreftest-3 1498.89215686
Android Tegra 250 mozilla-central opt test mochitest-1 1723.55445545
Android Tegra 250 mozilla-central opt test mochitest-2 1433.60909091
Android Tegra 250 mozilla-central opt test mochitest-3 1533.75247525
Android Tegra 250 mozilla-central opt test mochitest-4 1311.97222222
Android Tegra 250 mozilla-central opt test mochitest-5 1332.30392157
Android Tegra 250 mozilla-central opt test mochitest-6 1373.30188679
Android Tegra 250 mozilla-central opt test mochitest-7 1373.56190476
Android Tegra 250 mozilla-central opt test mochitest-8 1691.24528302
Android Tegra 250 mozilla-central opt test reftest-1 1481.94166667
Android Tegra 250 mozilla-central opt test reftest-2 1585.47572816
Android Tegra 250 mozilla-central opt test reftest-3 1606.37864078
Android Tegra 250 mozilla-central opt test robocop 1813.01941748
Android Tegra 250 mozilla-central talos remote-tdhtml 1506.13861386
Android Tegra 250 mozilla-central talos remote-tp4m_nochrome 1592.23529412
Android Tegra 250 mozilla-central talos remote-trobocheck 1398.76146789
Android Tegra 250 mozilla-central talos remote-trobocheck2 1584.97142857
Android Tegra 250 mozilla-central talos remote-trobocheck3 1460.77669903
Android Tegra 250 mozilla-central talos remote-trobopan 1567.03883495
Android Tegra 250 mozilla-central talos remote-troboprovider 1592.09345794
Android Tegra 250 mozilla-central talos remote-ts 1761.95145631
Android Tegra 250 mozilla-central talos remote-tsvg 1490.09803922
Android XUL Tegra 250 mozilla-central opt test crashtest-2 1647.91428571
Android XUL Tegra 250 mozilla-central opt test crashtest-3 1430.54368932
Android XUL Tegra 250 mozilla-central opt test jsreftest-1 1550.04761905
Android XUL Tegra 250 mozilla-central opt test jsreftest-2 1593.39805825
Android XUL Tegra 250 mozilla-central opt test jsreftest-3 1754.47058824
Android XUL Tegra 250 mozilla-central opt test mochitest-1 1409.15686275
Android XUL Tegra 250 mozilla-central opt test mochitest-2 1521.00980392
Android XUL Tegra 250 mozilla-central opt test mochitest-3 1230.08737864
Android XUL Tegra 250 mozilla-central opt test mochitest-4 1525.06796117
Android XUL Tegra 250 mozilla-central opt test mochitest-5 1431.87254902
Android XUL Tegra 250 mozilla-central opt test mochitest-6 1476.26923077
Android XUL Tegra 250 mozilla-central opt test mochitest-7 1272.8490566
Android XUL Tegra 250 mozilla-central opt test mochitest-8 1799.10576923
Android XUL Tegra 250 mozilla-central opt test reftest-1 1569.52427184
Android XUL Tegra 250 mozilla-central opt test reftest-2 1658.66666667
Android XUL Tegra 250 mozilla-central opt test reftest-3 1551.54205607
Android XUL mozilla-central build 1277.25
Android mozilla-central build 1486.5
B2G gb_armv7a_gecko mozilla-central build 1264.95652174
B2G gb_armv7a_gecko-debug mozilla-central build 1539.45977011
B2G ics_armv7a_gecko mozilla-central build 1498.29850746
B2G ics_armv7a_gecko-debug mozilla-central build 1439.40298507
Linux mozilla-central build 1429.60227273
Linux mozilla-central leak test build 1413.39772727
Linux x86-64 mozilla-central build 1362.26136364
Linux x86-64 mozilla-central leak test build 1859.20454545
OS X 10.7 64-bit mozilla-central leak test build 1418.80681818
OS X 10.7 mozilla-central build 1324.79545455
Rev3 Fedora 12 mozilla-central debug test crashtest 1793.67857143
Rev3 Fedora 12 mozilla-central debug test jetpack 1527.98809524
Rev3 Fedora 12 mozilla-central debug test jsreftest 1477.39285714
Rev3 Fedora 12 mozilla-central debug test mochitest-other 1443.39285714
Rev3 Fedora 12 mozilla-central debug test mochitests-1/5 1704.18823529
Rev3 Fedora 12 mozilla-central debug test mochitests-2/5 1576.8452381
Rev3 Fedora 12 mozilla-central debug test mochitests-3/5 1615.10714286
Rev3 Fedora 12 mozilla-central debug test mochitests-4/5 1448.51190476
Rev3 Fedora 12 mozilla-central debug test mochitests-5/5 1751.01190476
Rev3 Fedora 12 mozilla-central debug test reftest 1603.91666667
Rev3 Fedora 12 mozilla-central debug test xpcshell 1416.51190476
Rev3 Fedora 12 mozilla-central opt test crashtest 1610.98809524
Rev3 Fedora 12 mozilla-central opt test crashtest-ipc 1559.86904762
Rev3 Fedora 12 mozilla-central opt test jetpack 1546.75
Rev3 Fedora 12 mozilla-central opt test jsreftest 1443.92857143
Rev3 Fedora 12 mozilla-central opt test mochitest-other 1358.32142857
Rev3 Fedora 12 mozilla-central opt test mochitests-1/5 1444.32941176
Rev3 Fedora 12 mozilla-central opt test mochitests-2/5 1247.67857143
Rev3 Fedora 12 mozilla-central opt test mochitests-3/5 1574.0952381
Rev3 Fedora 12 mozilla-central opt test mochitests-4/5 1452.3452381
Rev3 Fedora 12 mozilla-central opt test mochitests-5/5 1345.52380952
Rev3 Fedora 12 mozilla-central opt test peptest 1388.17857143
Rev3 Fedora 12 mozilla-central opt test reftest 1365.30952381
Rev3 Fedora 12 mozilla-central opt test reftest-ipc 1832.04761905
Rev3 Fedora 12 mozilla-central opt test reftest-no-accel 1419.16666667
Rev3 Fedora 12 mozilla-central opt test xpcshell 1645.51764706
Rev3 Fedora 12 mozilla-central talos chromer 1427.22619048
Rev3 Fedora 12 mozilla-central talos dirty 1282.8
Rev3 Fedora 12 mozilla-central talos dirtypaint 1629.80952381
Rev3 Fedora 12 mozilla-central talos dromaeo 1365.82222222
Rev3 Fedora 12 mozilla-central talos dromaeojs 1465.46153846
Rev3 Fedora 12 mozilla-central talos jstests 1725.84444444
Rev3 Fedora 12 mozilla-central talos nochromer 1409.21428571
Rev3 Fedora 12 mozilla-central talos other 1230.77380952
Rev3 Fedora 12 mozilla-central talos svgr 1486.88095238
Rev3 Fedora 12 mozilla-central talos tpn 1460.95238095
Rev3 Fedora 12×64 mozilla-central debug test crashtest 1460.81176471
Rev3 Fedora 12×64 mozilla-central debug test jetpack 1299.35294118
Rev3 Fedora 12×64 mozilla-central debug test jsreftest 1620.12790698
Rev3 Fedora 12×64 mozilla-central debug test mochitest-other 1441.22727273
Rev3 Fedora 12×64 mozilla-central debug test mochitests-1/5 1636.98888889
Rev3 Fedora 12×64 mozilla-central debug test mochitests-2/5 1292.28235294
Rev3 Fedora 12×64 mozilla-central debug test mochitests-3/5 1520.34117647
Rev3 Fedora 12×64 mozilla-central debug test mochitests-4/5 1539.4
Rev3 Fedora 12×64 mozilla-central debug test mochitests-5/5 1563.67058824
Rev3 Fedora 12×64 mozilla-central debug test reftest 1460.58823529
Rev3 Fedora 12×64 mozilla-central debug test xpcshell 1292.23529412
Rev3 Fedora 12×64 mozilla-central opt test crashtest 1539.57647059
Rev3 Fedora 12×64 mozilla-central opt test jetpack 1716.28235294
Rev3 Fedora 12×64 mozilla-central opt test jsreftest 1619.31764706
Rev3 Fedora 12×64 mozilla-central opt test mochitest-other 1523.48837209
Rev3 Fedora 12×64 mozilla-central opt test mochitests-1/5 1461.10588235
Rev3 Fedora 12×64 mozilla-central opt test mochitests-2/5 1548.6744186
Rev3 Fedora 12×64 mozilla-central opt test mochitests-3/5 1630.16470588
Rev3 Fedora 12×64 mozilla-central opt test mochitests-4/5 1540.01176471
Rev3 Fedora 12×64 mozilla-central opt test mochitests-5/5 1639.5047619
Rev3 Fedora 12×64 mozilla-central opt test peptest 1845.95294118
Rev3 Fedora 12×64 mozilla-central opt test reftest 1786.08235294
Rev3 Fedora 12×64 mozilla-central opt test xpcshell 1321.78823529
Rev3 Fedora 12×64 mozilla-central talos chromer 1399.71764706
Rev3 Fedora 12×64 mozilla-central talos dirty 1582.13333333
Rev3 Fedora 12×64 mozilla-central talos dirtypaint 1609.03529412
Rev3 Fedora 12×64 mozilla-central talos dromaeo 1793.82222222
Rev3 Fedora 12×64 mozilla-central talos dromaeojs 1266.35
Rev3 Fedora 12×64 mozilla-central talos jstests 1830.17777778
Rev3 Fedora 12×64 mozilla-central talos nochromer 1310.87058824
Rev3 Fedora 12×64 mozilla-central talos other 1684.15294118
Rev3 Fedora 12×64 mozilla-central talos svgr 1864.89411765
Rev3 Fedora 12×64 mozilla-central talos tpn 1607.82352941
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test crashtest 1567.03960396
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test jetpack 1587.17821782
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test jsreftest 1711.85148515
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitest-other 1638.87128713
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-1/5 1424.64356436
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-2/5 1589.06930693
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-3/5 1592.65346535
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-4/5 1477.56435644
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-5/5 1662.66336634
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test peptest 1594.99009901
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test reftest 1704.13861386
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test xpcshell 1524.73267327
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos chromer 1540.58415842
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty 1663.2037037
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirtypaint 1493.07920792
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dromaeo 1467.05555556
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dromaeojs 1631.04255319
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos jstests 1470.88888889
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos nochromer 1586.46534653
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos other 1580.38613861
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos svgr 1610.73267327
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos tpn 1768.98019802
Rev3 WINNT 5.1 mozilla-central debug test crashtest 1346.4
Rev3 WINNT 5.1 mozilla-central debug test jetpack 1568.725
Rev3 WINNT 5.1 mozilla-central debug test jsreftest 1465.05
Rev3 WINNT 5.1 mozilla-central debug test mochitest-other 1893.2625
Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5 1669.0875
Rev3 WINNT 5.1 mozilla-central debug test mochitests-2/5 1697.8875
Rev3 WINNT 5.1 mozilla-central debug test mochitests-3/5 1615.8375
Rev3 WINNT 5.1 mozilla-central debug test mochitests-4/5 1627.75
Rev3 WINNT 5.1 mozilla-central debug test mochitests-5/5 1597.425
Rev3 WINNT 5.1 mozilla-central debug test reftest 1757.05
Rev3 WINNT 5.1 mozilla-central debug test xpcshell 1720.375
Rev3 WINNT 5.1 mozilla-central opt test crashtest 1719.84146341
Rev3 WINNT 5.1 mozilla-central opt test jetpack 1640.7804878
Rev3 WINNT 5.1 mozilla-central opt test jsreftest 1725.46341463
Rev3 WINNT 5.1 mozilla-central opt test mochitest-other 1554.07317073
Rev3 WINNT 5.1 mozilla-central opt test mochitests-1/5 1405.34146341
Rev3 WINNT 5.1 mozilla-central opt test mochitests-2/5 1378.73170732
Rev3 WINNT 5.1 mozilla-central opt test mochitests-3/5 1467.54878049
Rev3 WINNT 5.1 mozilla-central opt test mochitests-4/5 1545.7195122
Rev3 WINNT 5.1 mozilla-central opt test mochitests-5/5 1382.70731707
Rev3 WINNT 5.1 mozilla-central opt test peptest 1382.92682927
Rev3 WINNT 5.1 mozilla-central opt test reftest 1650.84146341
Rev3 WINNT 5.1 mozilla-central opt test xpcshell 1608.17073171
Rev3 WINNT 5.1 mozilla-central talos chromer 1694.86585366
Rev3 WINNT 5.1 mozilla-central talos dirty 1556.30952381
Rev3 WINNT 5.1 mozilla-central talos dirtypaint 1651.80487805
Rev3 WINNT 5.1 mozilla-central talos dromaeo 1415.38095238
Rev3 WINNT 5.1 mozilla-central talos dromaeojs 1341.225
Rev3 WINNT 5.1 mozilla-central talos jstests 1612.71428571
Rev3 WINNT 5.1 mozilla-central talos nochromer 1556.0
Rev3 WINNT 5.1 mozilla-central talos other 1326.64634146
Rev3 WINNT 5.1 mozilla-central talos svgr 1524.90243902
Rev3 WINNT 5.1 mozilla-central talos tpn 1711.64634146
Rev3 WINNT 6.1 mozilla-central debug test crashtest 1308.125
Rev3 WINNT 6.1 mozilla-central debug test jetpack 1633.7875
Rev3 WINNT 6.1 mozilla-central debug test jsreftest 1833.8625
Rev3 WINNT 6.1 mozilla-central debug test mochitest-other 1391.5375
Rev3 WINNT 6.1 mozilla-central debug test mochitests-1/5 1398.2625
Rev3 WINNT 6.1 mozilla-central debug test mochitests-2/5 1269.4875
Rev3 WINNT 6.1 mozilla-central debug test mochitests-3/5 1545.325
Rev3 WINNT 6.1 mozilla-central debug test mochitests-4/5 1451.45
Rev3 WINNT 6.1 mozilla-central debug test mochitests-5/5 1265.275
Rev3 WINNT 6.1 mozilla-central debug test reftest 1292.25
Rev3 WINNT 6.1 mozilla-central debug test xpcshell 1563.7875
Rev3 WINNT 6.1 mozilla-central opt test crashtest 1645.51219512
Rev3 WINNT 6.1 mozilla-central opt test jetpack 1549.59756098
Rev3 WINNT 6.1 mozilla-central opt test jsreftest 1926.76829268
Rev3 WINNT 6.1 mozilla-central opt test mochitest-other 1845.15853659
Rev3 WINNT 6.1 mozilla-central opt test mochitests-1/5 1392.75609756
Rev3 WINNT 6.1 mozilla-central opt test mochitests-2/5 1516.2804878
Rev3 WINNT 6.1 mozilla-central opt test mochitests-3/5 1643.31707317
Rev3 WINNT 6.1 mozilla-central opt test mochitests-4/5 1658.19512195
Rev3 WINNT 6.1 mozilla-central opt test mochitests-5/5 1658.73170732
Rev3 WINNT 6.1 mozilla-central opt test peptest 1412.69512195
Rev3 WINNT 6.1 mozilla-central opt test reftest 1557.63529412
Rev3 WINNT 6.1 mozilla-central opt test reftest-no-accel 1773.42682927
Rev3 WINNT 6.1 mozilla-central opt test xpcshell 1849.8902439
Rev3 WINNT 6.1 mozilla-central talos chromer 1764.8902439
Rev3 WINNT 6.1 mozilla-central talos dirty 1440.97619048
Rev3 WINNT 6.1 mozilla-central talos dirtypaint 1697.02439024
Rev3 WINNT 6.1 mozilla-central talos dromaeo 1511.4047619
Rev3 WINNT 6.1 mozilla-central talos dromaeojs 1442.2
Rev3 WINNT 6.1 mozilla-central talos jstests 1709.88095238
Rev3 WINNT 6.1 mozilla-central talos nochromer 1299.13414634
Rev3 WINNT 6.1 mozilla-central talos other 1595.7195122
Rev3 WINNT 6.1 mozilla-central talos svgr 1865.98780488
Rev3 WINNT 6.1 mozilla-central talos tpn 1861.63414634
Rev3 WINNT 6.1 mozilla-central talos xperf 1551.48809524
Rev3 WINNT 6.1 x64 mozilla-central opt test crashtest 1598.8375
Rev3 WINNT 6.1 x64 mozilla-central opt test jetpack 1258.225
Rev3 WINNT 6.1 x64 mozilla-central opt test jsreftest 1472.35
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitest-other 1490.725
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-1/5 1404.075
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-2/5 1376.5875
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-3/5 1592.975
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-4/5 1463.95
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-5/5 1449.9
Rev3 WINNT 6.1 x64 mozilla-central opt test peptest 1529.0
Rev3 WINNT 6.1 x64 mozilla-central opt test reftest 1672.975
Rev3 WINNT 6.1 x64 mozilla-central opt test xpcshell 1316.7625
Rev3 WINNT 6.1 x64 mozilla-central talos chromer 1588.0875
Rev3 WINNT 6.1 x64 mozilla-central talos dirty 1083.07142857
Rev3 WINNT 6.1 x64 mozilla-central talos dirtypaint 1317.36842105
Rev3 WINNT 6.1 x64 mozilla-central talos dromaeo 1551.97619048
Rev3 WINNT 6.1 x64 mozilla-central talos dromaeojs 1061.57894737
Rev3 WINNT 6.1 x64 mozilla-central talos nochromer 1117.725
Rev3 WINNT 6.1 x64 mozilla-central talos other 1464.9625
Rev3 WINNT 6.1 x64 mozilla-central talos svgr 1294.5875
Rev3 WINNT 6.1 x64 mozilla-central talos tpn 1587.9875
Rev4 MacOSX Lion 10.7 mozilla-central debug test crashtest 1413.51190476
Rev4 MacOSX Lion 10.7 mozilla-central debug test jetpack 1886.20238095
Rev4 MacOSX Lion 10.7 mozilla-central debug test jsreftest 1689.70238095
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-other 1419.03529412
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-1/5 1367.14285714
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-2/5 1447.96428571
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-3/5 1306.36904762
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-4/5 1560.03571429
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-5/5 1346.22619048
Rev4 MacOSX Lion 10.7 mozilla-central debug test reftest 1160.25
Rev4 MacOSX Lion 10.7 mozilla-central debug test xpcshell 1641.98809524
Rev4 MacOSX Lion 10.7 mozilla-central opt test crashtest 1567.85148515
Rev4 MacOSX Lion 10.7 mozilla-central opt test jetpack 1488.48514851
Rev4 MacOSX Lion 10.7 mozilla-central opt test jsreftest 1610.36633663
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-other 1559.95049505
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-1/5 1551.06930693
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-2/5 1439.67326733
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-3/5 1611.44554455
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-4/5 1471.16831683
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-5/5 1648.7029703
Rev4 MacOSX Lion 10.7 mozilla-central opt test peptest 1444.3960396
Rev4 MacOSX Lion 10.7 mozilla-central opt test reftest 1564.81553398
Rev4 MacOSX Lion 10.7 mozilla-central opt test xpcshell 1543.82178218
Rev4 MacOSX Lion 10.7 mozilla-central talos chromer 1400.03960396
Rev4 MacOSX Lion 10.7 mozilla-central talos dirty 1479.14814815
Rev4 MacOSX Lion 10.7 mozilla-central talos dirtypaint 1492.04950495
Rev4 MacOSX Lion 10.7 mozilla-central talos dromaeo 1390.87037037
Rev4 MacOSX Lion 10.7 mozilla-central talos dromaeojs 1378.08510638
Rev4 MacOSX Lion 10.7 mozilla-central talos jstests 1653.78181818
Rev4 MacOSX Lion 10.7 mozilla-central talos nochromer 1320.79207921
Rev4 MacOSX Lion 10.7 mozilla-central talos other 1492.05940594
Rev4 MacOSX Lion 10.7 mozilla-central talos svgr 1500.45544554
Rev4 MacOSX Lion 10.7 mozilla-central talos tpn 1516.94059406
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test crashtest 1336.52380952
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jetpack 1568.20238095
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jsreftest 1532.78823529
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-other 1355.49425287
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-1/5 1452.45238095
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-2/5 1497.10714286
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-3/5 1482.0952381
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-4/5 1443.01190476
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-5/5 1237.79761905
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test reftest 1343.89411765
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test xpcshell 1545.88372093
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test crashtest 1669.87128713
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jetpack 1679.46534653
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jsreftest 1621.41584158
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-other 1568.45098039
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-1/5 1644.94059406
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-2/5 1851.26732673
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-3/5 1618.50485437
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-4/5 1634.09803922
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-5/5 1557.76153846
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test peptest 1636.86138614
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test reftest 1602.41176471
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test xpcshell 1590.94059406
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos chromer 1665.55882353
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dirty 1248.79245283
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dirtypaint 1670.63366337
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dromaeo 1626.81481481
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dromaeojs 1625.02083333
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos jstests 1739.57407407
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos nochromer 1436.52941176
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos other 1602.79207921
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos svgr 1676.52475248
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos tpn 1481.43564356
WINNT 5.2 mozilla-central build 1420.93258427
WINNT 5.2 mozilla-central leak test build 1364.53932584
WINNT 6.1 x86-64 mozilla-central build 1437.09090909
jetpack-mozilla-central-fedora-debug 2145.93333333
jetpack-mozilla-central-fedora-opt 1462.06896552
jetpack-mozilla-central-fedora64-debug 1806.75862069
jetpack-mozilla-central-fedora64-opt 1603.75862069
jetpack-mozilla-central-leopard-debug 1972.31034483
jetpack-mozilla-central-leopard-opt 2482.82758621
jetpack-mozilla-central-lion-debug 1446.65517241
jetpack-mozilla-central-lion-opt 1509.27586207
jetpack-mozilla-central-snowleopard-debug 1982.5862069
jetpack-mozilla-central-snowleopard-opt 1654.34482759
jetpack-mozilla-central-w764-debug 2471.62068966
jetpack-mozilla-central-w764-opt 1800.65517241
jetpack-mozilla-central-win7-debug 1794.37931034
jetpack-mozilla-central-win7-opt 1567.93103448
jetpack-mozilla-central-xp-debug 2147.37931034
jetpack-mozilla-central-xp-opt 1786.24137931
============================================

10 Comments (+add yours?)

  1. Ben Hearsum
    21 Aug 2012 @ 18:46:59

    Johnath calculated this to be a millenia per year: https://twitter.com/johnath/status/237980920102137856

    Reply

  2. Havvy
    21 Aug 2012 @ 19:28:14

    1370 years per year to be more accurate. But that assumes a constant checkin rate. Maybe some calculus could be involved to find the total for this year + estimate?

    Reply

  3. Benoit Jacob
    21 Aug 2012 @ 20:34:17

    Have you considered other ways of reducing test load, such as:
    – running certain tests only once a day, instead on on every checkin?
    – optionally triggering per-checking runs of such tests when a certain directory is touched?

    For example, in mochitest-1, probably the most expensive mochitest is the WebGL one. But it’s useless to run it on every checkin. It’s useful to run it on every checkin that touches either content/canvas, gfx/gl, or gfx/layers; and it’s useful to run it once a day for integration testing, but it’s not useful to run it on every checkin. In this way, the load incurred by this test could be reduced by over 90% without losing coverage and with only very little loss of immediacy of detecting regressions.

    Reply

  4. Justin Lebar
    21 Aug 2012 @ 20:35:47

    Like I said in the newsgroup, I think we need clear reporting of wait times (similar to the monthly infra load reports).

    Then we can evaluate whether we’re turning tests around within an acceptable amount of time. At the moment, I don’t even know whether tests have actually gotten slower (maybe it’s just my imagination), and if they have gotten slower, how much faster they’d need to get to match our speed from a few months ago.

    Reply

    • John
      27 Aug 2012 @ 19:25:36

      hi jlebar:

      We already send daily “wait times” posts to mozilla.dev.tree-mangement. There’s 3 different pools where we need to measure “waittimes”, so there’s 3 posts per day.

      We do need a way to get that data into something that makes trend-graph-over-time easier to see. However, meanwhile, if you just look at the subject field of the posts over time, you can see the number of jobs per day increasing, and the wait times for *some* specific OS getting worse.

      tc
      John.

      Reply

  5. Phil Ringnalda
    22 Aug 2012 @ 00:09:25

    It doesn’t make a significant difference to the total, but those jetpack-mozilla-central-* jobs at the end of the list are not m-c on-push builds (the ones ending in Jetpack above are) – jetpack-* are the jetpack repo on-push builds on the separate Addon-SDK tree.

    Some of those number seem wildly unlikely, too: “WINNT 5.2 mozilla-central build 1420.93258427″ would be 23 minutes for a Windows build, when every Windows build for the last two days took between 115 and 120 minutes. 10.6 opt mochitest-5 I noticed the other day takes between 6 and 9 minutes, but you’ve got it at nearly 26 minutes.

    Reply

  6. @clochix
    22 Aug 2012 @ 00:26:10

    Chaque commit (±240 par jour) dans le dépôt de Mozilla déclenche 137h machine de compil et tests http://t.co/EvLlT3lm

    Reply

  7. @nyconyco
    22 Aug 2012 @ 01:55:20

    @clochix Chaque commit (±240 par jour) dans le dépôt de Mozilla déclenche 137h machine de compil et tests http://t.co/mOmOz26K

    Reply

  8. And
    22 Aug 2012 @ 02:02:25

    Couldn’t both builds and tests be run in a batch or bisect mode, so a new build would only be kicked of every hour (or such) and if a test changed status the check-in range would be bisected until the offending check-in was localized (with the triggering test run first, so the bisect can continue, while the other tests are run for completeness). Probably wouldn’t be applicable for try, where build-requests are explicit. For inbound a script could try to land the rest of the queue without the offending check-in.

    Reply

  9. And
    25 Aug 2012 @ 02:31:57

    I think there is a problem with the times you have listed. They seem way to similar and it would be odd if every test suite and every build took the same amount of time. If I look in the build logs (and am interpreting them correctly) the job “WINNT 5.2 mozilla-central build” (that you list as 1420 sec) took about 3872 sec (with make build:47%, make check:20%, hg-mirror:11%, make buildsymbols:6%, make update-packaging: 4%, …, for those curious), whereas the job “jetpack-mozilla-central-w764-debug” (you list as 2471) seems to have taken just about 70sec.

    The same job “jetpack-mozilla-central-w764-debug” seems a little odd since it purports to run a test on a debug build for win64. But tinderbox doesn’t seem to create debug builds for win64. And if you look in the build log this job seems to fail with a file-not-found error.

    It also looks odd that the jetpack tests seem to be run twice (both as “jetpack-mozilla-central-…” and “Rev3 … mozilla-central … test jetpack”). Both seem to run the “run_jetpack.py” script, although with different options.

    I don’t think windows have the best performance when it comes to many small files. Have you tried enabling folder compression on the \talos-slave\ folder. It have helped read performance for me (on something unrelated).

    I assume that at least mochitests doesn’t alter the files in the test folder, so perhaps it would be an idea to not unzip the test files, but map the zip as drive Presumably “Pismo Mount File” can do this, or you could distribute the tests to the slaves as a zipped iso-file, which should be fast to unzip and multiple programs can mount.

    Reply

Leave a Reply