“Oh, the Places You’ll Go!” by Dr Seuss

After last week, I’ve proudly added this to my bookshelf.

The book is, of course, fun. But the biggest reason I’m was because I received it at LEAD. This 4-session LEAD series ended Friday, and has been more then just an experience; its “an inflection point”. The excellently designed “coursework” and setup/moderation by Athena, Kate, Debbie and Mihca provided the framework. But the intensity, passion and honesty that everyone engaged within LEAD left me stunned, and immensely proud to know each and every one of them. The book is a small talisman.

ps: (possible NSFW?) below is a really fun reading of the book done at Burning Man 2011.

41,598 test jobs in a 24 hour day

On 26sep2012, we processed 41,598 test jobs. Thats yet another new record.

To put that in perspective, almost exactly 5 months ago, on 03may2012, was the first time we ever did over 40,000 test jobs in a 24 hour day. See my previous blog for details and trend graph. Since then, we’ve crossed that threshold:

  • 40,207 (03may2012)
  • 40,648 (24may2012)
  • 40,972 (30may2012)
  • 41,492 (30jul2012)
  • 41,598 (26sep2012)

We’re continuing to build out systems to get ahead of this increasing load, but its worth pausing to note the sheer scale and rate of growth here.

ps: on the same 26sep2012, we also did 3,433 builds in one day. Busy day near the end of quarter?

AOSA(vol2) and Mozilla’s Release Engineering: now on kindle and nook!

“The Architecture of Open Source Applications (vol2)” was published in paperback format in May2012. More details here. A few days ago, we were also published in ebook format, so now you can download the same book from Amazon (for your kindle) or from Barnes&Noble (for your nook).

Very cool to see this, and big thanks to Amy Brown for making this happen.

(As always, proceeds from book sales go to Amnesty International.)

linux on a Boeing 777

Last week, at SFO, while waiting for clearance to pushback from the gate, I noticed that the headrest entertainment system had crashed and was rebooting…and it was linux!

I couldnt read all the messages – the bootup messages scrolled by so quickly – but there was at least one error message about not finding a config file?!? They had to reboot the system a few times before it finally booted up successfully. And yes, once booted, it stayed up and running for the entire flight to LHR.

Here’s the best I could do with some rushed photos, in poor lighting, while they were telling me to turn off all electronic devices.

(click images for enlarged photos)

Oh, and while writing this post, I found someone else’s photo of linux booting on another Boeing 777.
linux on a Boeing 777

Calling all remoties

tl;dr: If you are a remotie…or if you work with someone who is a remotie… I’d love to hear from you.


Whenever “remoties” come up in discussion, I continue to be surprised by the level of interest people have about this.

Its not just a polite “oh, that’s interesting”. Its a suddenly intense outpouring of personal war stories – “oh really? Let me tell you about the time when…”. Some of those stories were told as validation (“yes, we did what you do, and we’re happy it worked for us also” or “we didn’t do what you do, and it ended badly“). Some of these stories were told in denial (“we tried that once, it didn’t work out, which proves it is not ever possible“). Some of these stories were told in despair (“…so now my company wont hire any remoties“). But all of these stories were told with intense personal fervor, sometimes years after the fact!

This shouldn’t have surprised me. As Homa Bahrami pointed out when I met her in Mozilla Summit 2011, and again in meetings this summer, working with remoties is a hard people-organizational problem, not a software-organization problem. Homa also pointed out the intense, long term impact this can have on someone’s personal life and entire career, which explained some of the passionate responses I’ve received so far.

Stepping back, I realized that while most of the people I’ve talked with so far are in the computer business, I’ve also heard similar stories from university lecturers, book publishers, public relations people, medical doctors and traveling sales reps.

This got me thinking about how to contact even more people who work remotely… hence this blog post.

If you are a remotie…or if you work with someone who is a remotie… I’d be really interested to hear from you.

  • Do you have examples of things that did (or did not!) work for you?
  • Do you have ideas of things you haven’t tried, but which you think might help?

As usual, you can post comments below. I do also understand this is a personal thing, especially if you are still working in the situation. Therefore, if you want to email me privately instead, please email me, and put “remoties” somewhere in the subject. I will, of course, honor any requests to keep feedback anonymous, all I ask that you give me any working email address in case something is unclear, and I want to contact you with any followup questions.

Thanks!
John.
=====

Meanwhile, here’s a collection of useful links I’ve found about working remotely. If you know of others, please let me know.

(UPDATED: added another link, joduinn 16mar2014, 09nov2014)

Increasing capacity

tl;dr: We’ve had a lot of infrastructure changes go live in the last 2-3 weeks, so now build and test wait times are MUCH better. The changes made were:

1) Turn off obsolete or broken jobs.
2) Re-image some linux32/linux64/win32 machines as extra Win2008 (64bit) build machines.
3) Enable pymake in production
4) Turn on more tegras
5) Re-imaging 40 OSX10.5 test machines as 20 WinXP and 20 Win7 test machines.


After all the recent meetings and newsgroup posts, I thought it useful to followup with a quick summary of the last few weeks of behind the scenes work on how RelEng infrastructure is handling the increased checkin load, as well as the increased number of builds/tests per checkin. Oh, and its worth noting for the record that all these changes have been done without needing a downtime. 🙂

1) Turn off obsolete or broken jobs.
This is not glamorous work, but there were lots of these and it had a massive impact in terms of reducing load in some areas, which allowed us to reshuffle machines to other areas. Specifically, we’ve disabled:

  • perma-orange/red test suites
  • android-xul tests, no longer needed
  • standalone talos v8 test suite, now included in the dromaeojs suite
  • B2G-on-gingerbread builds. (B2G-on-gingerbread builds were intentionally broken as part of setting up B2G-on-icecreamsandwich builds but left running hidden on tbpl.m.o. This was bad because a) B2G-on-gingerbread builds continued to waste CPU cycles and b) having broken B2G-on-gingerbread builds caused B2G IceCreamSandwich nightly builds to not be run because automation thought that there were no good/green changesets for B2G. Disabling the broken B2G-on-gingerbread builds fixed both these issues. (bug#780915).
  • Disabled osx10.5 tests on mozilla-central, related project branches and try, because FF17 no longer supports osx10.5. However, we still need some osx10.5 machines in order to run osx10.5 tests for mozilla-aurora, mozilla-beta, mozilla-release and ESR.
  • There is still an open question about linux32/64 builds/tests. Can we reduce our linux test capacity on one architecture to use these as test machines for other OS? From the thread it seems like the preference would be to turn down/off 32-bit builds/tests, but if you are interested, please respond in the dev.planning thread.
  • Of course, if you know of other builds or tests that aren’t used, or are perma-red/orange, let RelEng or Sheriff know and we can disable them until they can be fixed! Tests still to be disabled are tracked in bug#784681

2) Re-image some linux32/linux64/win32 machines as extra Win2008 (64bit) build machines.

  • linux32/linux64 builds continue to migrate to AWS, which frees up linux32/linux64 build machines for reimaging as win2008 (64bit) build machines. This requires changing desktop toolchain, so we have to be careful about not breaking binary compat, and needed to roll this change out on the trains. Details in bug#772446. As of today, we’ve increased from ~40 to 102 win2008 (64bit) build machines in production, with even more coming online soon. Details in bug#780022 and bug#784891.
  • This is important because the Win2008 (64bit) machines are used to do both win32 *and* win64 builds.
  • win32 l10n nightlies have code dependencies that require win32. Fixing this to run on win2008 (64bit) will free up even more win32 machines to convert to win2008 (64bit) builders. More details in bug#784848.

3) Enable pymake in production (bug# bug#593585).
This reduced build time on windows significantly. Combined with the extra windows machines, this is all good. Coop blogged more details here, and if you like eyecandy, he even has even a cool graph showing reduced duration of builds!

4) Turn on more tegras
86 new tegras have now been added to our test pool, bringing our current total up to 284 tegras. bug#767456

5) Re-imaging 40 OSX10.5 test machines as 20 WinXP and 20 Win7 test machines.
This increases us to 70 machines for WinXP and 70 machines for Win7x32, which helped improve WinXP, Win7 test wait times.

We’ve still got lots to do, of course. After all, the faster our infrastructure can process builds and tests, the more checkins developers will do, which means the more builds and tests we will need to handle… but things are definitely better, and our numbers in dev.tree-management for the last two weeks shows that!

Firefox: now testing on OSX 10.8

Late last week, tbpl.mozilla.org started showing two extra rows of green unittest+talos results. Those are because our existing OSX 32bit/64bit builds, which we already test on OSX10.6 and OSX10.7, are now also being tested on OSX10.8!

We test these opt and debug builds on OSX10.8 for incremental builds per checkin during the day, full clobber builds every night and all builds available on ftp.m.o. All the usual goodness, just like we do for OSX10.6, OSX10.7. For anyone interested in more details, check out kmoir’s blogpost, or bug#731278. Congrats to kmoir, on all the behind-the-scenes work needed to cat-herd this large, multiheaded project across the production line, only a few months after joining Mozilla.

This was our second big OSX10.8 project. In early July, bhearsum and edransch got OSX signing automation into production in preparation for the release of OSX10.8. For details of that work, the curious can read bug#730924 and bhearsum’s blog.

Infrastructure load for August 2012

  • #checkins-per-month: We had 5,803 checkins in August2012, another new record, which breaks last month’s record (5,635 checkins), which in turn broke the previous month’s record (5,194)…
  • #checkins-per-day: We had consistently high load across the month, 17-of-31 days had over 200 checkins-per-day.
  • #checkins-per-hour: The peak this month was 12.3 checkins per hour, and throughout the month, we sustained over 11-checkins-per-hour for 5 out of 24 hours in a day.

mozilla-inbound, fx-team:
Ratios this month identical within one percentage of last month. Again, mozilla-inbound continues to be heavily used as an integration branch, with 25.7% of all checkins, far more then the other integration branches fx-team (1.4% of checkins) or mozilla-central (2.9% of checkins). For comparison, I note that more people landed on mozilla-aurora then on mozilla-central.

mozilla-aurora, mozilla-beta:

  • 3.1% of our total monthly checkins landed into mozilla-aurora.
  • 1.9% of our total monthly checkins landed into mozilla-beta.

(Standard disclaimer: I’m always glad whenever we catch a problem *before* we ship a release; it avoids us having to do a chemspill release and also we ship better code to our Firefox users in the first place.)

misc other details:

  • Pushes per day
    • You can clearly see weekends through the month.

  • Pushes by hour of day
    • It is worth noting that for 5 hours in every 24 hour day, we did over 11 checkins-per-hour. For two of those hours, we did over 12 checkins-per-hour. Phrased another way, thats at least one checkin every 5mins for 5 hours.

137 hours compute hours every ~6 minutes

Followup from today’s Dev platform meeting.

We talk a lot about how the number of checkins continue to rise – its always great to see developers being able to do more work – and the “Infrastructure load” series of posts is one way to measure that. However, its worth noting that the amount of work we trigger per checkin also continues to rise.

Some of this is good (new builds, new test suites), but some of this is bad (consistently failing tests; obsolete tests).

How many hours of builds and tests do we run per commit?

  • In 2009, each commit triggered ~40 compute hours
  • In Jan 2012, each commit triggered 110 compute hours
  • In Aug 2012, each commit triggered 137 compute hours

137 hours of builds and tests per checkin is a lot – especially when we do a checkin every ~6 minutes. Put another way, that’s ~3.5 years of builds and tests being run per day.

If you look at the “wait times” posts in dev.tree-management, you’ll see that we’re having a tough time keeping up. We’re busily venting excess production load over to AWS for nativefennec / B2G desktop builds / linux desktop …. and are then recycling existing hardware to help grow inhouse capacity for win32 and win64.

But you can help.

Please file a bug if you know of:

  • any build or test that is being run but no longer needed OR
  • any build or test being hidden on tbpl.m.o because it constantly fails

We’ll disable the job to stop wasting CPU time, which will help everyone’s wait times. As needed, we’ll also file bugs to track fixing the problem and then reenable once the build/test is running green.

While any one given test suite might not seem like a lot of time to save, we run all these suites literally thousands of times per day. Every little saving helps makes things faster for everyone!

Thank you!
===========================================
Full list (with duration of job in seconds)
Android Armv6 mozilla-central build 1523.29213483
Android Debug mozilla-central build 1461.34090909
Android Tegra 250 mozilla-central opt test crashtest-2 1509.39215686
Android Tegra 250 mozilla-central opt test crashtest-3 1627.30769231
Android Tegra 250 mozilla-central opt test jsreftest-1 1635.7047619
Android Tegra 250 mozilla-central opt test jsreftest-2 2023.08181818
Android Tegra 250 mozilla-central opt test jsreftest-3 1498.89215686
Android Tegra 250 mozilla-central opt test mochitest-1 1723.55445545
Android Tegra 250 mozilla-central opt test mochitest-2 1433.60909091
Android Tegra 250 mozilla-central opt test mochitest-3 1533.75247525
Android Tegra 250 mozilla-central opt test mochitest-4 1311.97222222
Android Tegra 250 mozilla-central opt test mochitest-5 1332.30392157
Android Tegra 250 mozilla-central opt test mochitest-6 1373.30188679
Android Tegra 250 mozilla-central opt test mochitest-7 1373.56190476
Android Tegra 250 mozilla-central opt test mochitest-8 1691.24528302
Android Tegra 250 mozilla-central opt test reftest-1 1481.94166667
Android Tegra 250 mozilla-central opt test reftest-2 1585.47572816
Android Tegra 250 mozilla-central opt test reftest-3 1606.37864078
Android Tegra 250 mozilla-central opt test robocop 1813.01941748
Android Tegra 250 mozilla-central talos remote-tdhtml 1506.13861386
Android Tegra 250 mozilla-central talos remote-tp4m_nochrome 1592.23529412
Android Tegra 250 mozilla-central talos remote-trobocheck 1398.76146789
Android Tegra 250 mozilla-central talos remote-trobocheck2 1584.97142857
Android Tegra 250 mozilla-central talos remote-trobocheck3 1460.77669903
Android Tegra 250 mozilla-central talos remote-trobopan 1567.03883495
Android Tegra 250 mozilla-central talos remote-troboprovider 1592.09345794
Android Tegra 250 mozilla-central talos remote-ts 1761.95145631
Android Tegra 250 mozilla-central talos remote-tsvg 1490.09803922
Android XUL Tegra 250 mozilla-central opt test crashtest-2 1647.91428571
Android XUL Tegra 250 mozilla-central opt test crashtest-3 1430.54368932
Android XUL Tegra 250 mozilla-central opt test jsreftest-1 1550.04761905
Android XUL Tegra 250 mozilla-central opt test jsreftest-2 1593.39805825
Android XUL Tegra 250 mozilla-central opt test jsreftest-3 1754.47058824
Android XUL Tegra 250 mozilla-central opt test mochitest-1 1409.15686275
Android XUL Tegra 250 mozilla-central opt test mochitest-2 1521.00980392
Android XUL Tegra 250 mozilla-central opt test mochitest-3 1230.08737864
Android XUL Tegra 250 mozilla-central opt test mochitest-4 1525.06796117
Android XUL Tegra 250 mozilla-central opt test mochitest-5 1431.87254902
Android XUL Tegra 250 mozilla-central opt test mochitest-6 1476.26923077
Android XUL Tegra 250 mozilla-central opt test mochitest-7 1272.8490566
Android XUL Tegra 250 mozilla-central opt test mochitest-8 1799.10576923
Android XUL Tegra 250 mozilla-central opt test reftest-1 1569.52427184
Android XUL Tegra 250 mozilla-central opt test reftest-2 1658.66666667
Android XUL Tegra 250 mozilla-central opt test reftest-3 1551.54205607
Android XUL mozilla-central build 1277.25
Android mozilla-central build 1486.5
B2G gb_armv7a_gecko mozilla-central build 1264.95652174
B2G gb_armv7a_gecko-debug mozilla-central build 1539.45977011
B2G ics_armv7a_gecko mozilla-central build 1498.29850746
B2G ics_armv7a_gecko-debug mozilla-central build 1439.40298507
Linux mozilla-central build 1429.60227273
Linux mozilla-central leak test build 1413.39772727
Linux x86-64 mozilla-central build 1362.26136364
Linux x86-64 mozilla-central leak test build 1859.20454545
OS X 10.7 64-bit mozilla-central leak test build 1418.80681818
OS X 10.7 mozilla-central build 1324.79545455
Rev3 Fedora 12 mozilla-central debug test crashtest 1793.67857143
Rev3 Fedora 12 mozilla-central debug test jetpack 1527.98809524
Rev3 Fedora 12 mozilla-central debug test jsreftest 1477.39285714
Rev3 Fedora 12 mozilla-central debug test mochitest-other 1443.39285714
Rev3 Fedora 12 mozilla-central debug test mochitests-1/5 1704.18823529
Rev3 Fedora 12 mozilla-central debug test mochitests-2/5 1576.8452381
Rev3 Fedora 12 mozilla-central debug test mochitests-3/5 1615.10714286
Rev3 Fedora 12 mozilla-central debug test mochitests-4/5 1448.51190476
Rev3 Fedora 12 mozilla-central debug test mochitests-5/5 1751.01190476
Rev3 Fedora 12 mozilla-central debug test reftest 1603.91666667
Rev3 Fedora 12 mozilla-central debug test xpcshell 1416.51190476
Rev3 Fedora 12 mozilla-central opt test crashtest 1610.98809524
Rev3 Fedora 12 mozilla-central opt test crashtest-ipc 1559.86904762
Rev3 Fedora 12 mozilla-central opt test jetpack 1546.75
Rev3 Fedora 12 mozilla-central opt test jsreftest 1443.92857143
Rev3 Fedora 12 mozilla-central opt test mochitest-other 1358.32142857
Rev3 Fedora 12 mozilla-central opt test mochitests-1/5 1444.32941176
Rev3 Fedora 12 mozilla-central opt test mochitests-2/5 1247.67857143
Rev3 Fedora 12 mozilla-central opt test mochitests-3/5 1574.0952381
Rev3 Fedora 12 mozilla-central opt test mochitests-4/5 1452.3452381
Rev3 Fedora 12 mozilla-central opt test mochitests-5/5 1345.52380952
Rev3 Fedora 12 mozilla-central opt test peptest 1388.17857143
Rev3 Fedora 12 mozilla-central opt test reftest 1365.30952381
Rev3 Fedora 12 mozilla-central opt test reftest-ipc 1832.04761905
Rev3 Fedora 12 mozilla-central opt test reftest-no-accel 1419.16666667
Rev3 Fedora 12 mozilla-central opt test xpcshell 1645.51764706
Rev3 Fedora 12 mozilla-central talos chromer 1427.22619048
Rev3 Fedora 12 mozilla-central talos dirty 1282.8
Rev3 Fedora 12 mozilla-central talos dirtypaint 1629.80952381
Rev3 Fedora 12 mozilla-central talos dromaeo 1365.82222222
Rev3 Fedora 12 mozilla-central talos dromaeojs 1465.46153846
Rev3 Fedora 12 mozilla-central talos jstests 1725.84444444
Rev3 Fedora 12 mozilla-central talos nochromer 1409.21428571
Rev3 Fedora 12 mozilla-central talos other 1230.77380952
Rev3 Fedora 12 mozilla-central talos svgr 1486.88095238
Rev3 Fedora 12 mozilla-central talos tpn 1460.95238095
Rev3 Fedora 12×64 mozilla-central debug test crashtest 1460.81176471
Rev3 Fedora 12×64 mozilla-central debug test jetpack 1299.35294118
Rev3 Fedora 12×64 mozilla-central debug test jsreftest 1620.12790698
Rev3 Fedora 12×64 mozilla-central debug test mochitest-other 1441.22727273
Rev3 Fedora 12×64 mozilla-central debug test mochitests-1/5 1636.98888889
Rev3 Fedora 12×64 mozilla-central debug test mochitests-2/5 1292.28235294
Rev3 Fedora 12×64 mozilla-central debug test mochitests-3/5 1520.34117647
Rev3 Fedora 12×64 mozilla-central debug test mochitests-4/5 1539.4
Rev3 Fedora 12×64 mozilla-central debug test mochitests-5/5 1563.67058824
Rev3 Fedora 12×64 mozilla-central debug test reftest 1460.58823529
Rev3 Fedora 12×64 mozilla-central debug test xpcshell 1292.23529412
Rev3 Fedora 12×64 mozilla-central opt test crashtest 1539.57647059
Rev3 Fedora 12×64 mozilla-central opt test jetpack 1716.28235294
Rev3 Fedora 12×64 mozilla-central opt test jsreftest 1619.31764706
Rev3 Fedora 12×64 mozilla-central opt test mochitest-other 1523.48837209
Rev3 Fedora 12×64 mozilla-central opt test mochitests-1/5 1461.10588235
Rev3 Fedora 12×64 mozilla-central opt test mochitests-2/5 1548.6744186
Rev3 Fedora 12×64 mozilla-central opt test mochitests-3/5 1630.16470588
Rev3 Fedora 12×64 mozilla-central opt test mochitests-4/5 1540.01176471
Rev3 Fedora 12×64 mozilla-central opt test mochitests-5/5 1639.5047619
Rev3 Fedora 12×64 mozilla-central opt test peptest 1845.95294118
Rev3 Fedora 12×64 mozilla-central opt test reftest 1786.08235294
Rev3 Fedora 12×64 mozilla-central opt test xpcshell 1321.78823529
Rev3 Fedora 12×64 mozilla-central talos chromer 1399.71764706
Rev3 Fedora 12×64 mozilla-central talos dirty 1582.13333333
Rev3 Fedora 12×64 mozilla-central talos dirtypaint 1609.03529412
Rev3 Fedora 12×64 mozilla-central talos dromaeo 1793.82222222
Rev3 Fedora 12×64 mozilla-central talos dromaeojs 1266.35
Rev3 Fedora 12×64 mozilla-central talos jstests 1830.17777778
Rev3 Fedora 12×64 mozilla-central talos nochromer 1310.87058824
Rev3 Fedora 12×64 mozilla-central talos other 1684.15294118
Rev3 Fedora 12×64 mozilla-central talos svgr 1864.89411765
Rev3 Fedora 12×64 mozilla-central talos tpn 1607.82352941
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test crashtest 1567.03960396
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test jetpack 1587.17821782
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test jsreftest 1711.85148515
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitest-other 1638.87128713
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-1/5 1424.64356436
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-2/5 1589.06930693
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-3/5 1592.65346535
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-4/5 1477.56435644
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-5/5 1662.66336634
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test peptest 1594.99009901
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test reftest 1704.13861386
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test xpcshell 1524.73267327
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos chromer 1540.58415842
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty 1663.2037037
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirtypaint 1493.07920792
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dromaeo 1467.05555556
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dromaeojs 1631.04255319
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos jstests 1470.88888889
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos nochromer 1586.46534653
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos other 1580.38613861
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos svgr 1610.73267327
Rev3 MacOSX Leopard 10.5.8 mozilla-central talos tpn 1768.98019802
Rev3 WINNT 5.1 mozilla-central debug test crashtest 1346.4
Rev3 WINNT 5.1 mozilla-central debug test jetpack 1568.725
Rev3 WINNT 5.1 mozilla-central debug test jsreftest 1465.05
Rev3 WINNT 5.1 mozilla-central debug test mochitest-other 1893.2625
Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5 1669.0875
Rev3 WINNT 5.1 mozilla-central debug test mochitests-2/5 1697.8875
Rev3 WINNT 5.1 mozilla-central debug test mochitests-3/5 1615.8375
Rev3 WINNT 5.1 mozilla-central debug test mochitests-4/5 1627.75
Rev3 WINNT 5.1 mozilla-central debug test mochitests-5/5 1597.425
Rev3 WINNT 5.1 mozilla-central debug test reftest 1757.05
Rev3 WINNT 5.1 mozilla-central debug test xpcshell 1720.375
Rev3 WINNT 5.1 mozilla-central opt test crashtest 1719.84146341
Rev3 WINNT 5.1 mozilla-central opt test jetpack 1640.7804878
Rev3 WINNT 5.1 mozilla-central opt test jsreftest 1725.46341463
Rev3 WINNT 5.1 mozilla-central opt test mochitest-other 1554.07317073
Rev3 WINNT 5.1 mozilla-central opt test mochitests-1/5 1405.34146341
Rev3 WINNT 5.1 mozilla-central opt test mochitests-2/5 1378.73170732
Rev3 WINNT 5.1 mozilla-central opt test mochitests-3/5 1467.54878049
Rev3 WINNT 5.1 mozilla-central opt test mochitests-4/5 1545.7195122
Rev3 WINNT 5.1 mozilla-central opt test mochitests-5/5 1382.70731707
Rev3 WINNT 5.1 mozilla-central opt test peptest 1382.92682927
Rev3 WINNT 5.1 mozilla-central opt test reftest 1650.84146341
Rev3 WINNT 5.1 mozilla-central opt test xpcshell 1608.17073171
Rev3 WINNT 5.1 mozilla-central talos chromer 1694.86585366
Rev3 WINNT 5.1 mozilla-central talos dirty 1556.30952381
Rev3 WINNT 5.1 mozilla-central talos dirtypaint 1651.80487805
Rev3 WINNT 5.1 mozilla-central talos dromaeo 1415.38095238
Rev3 WINNT 5.1 mozilla-central talos dromaeojs 1341.225
Rev3 WINNT 5.1 mozilla-central talos jstests 1612.71428571
Rev3 WINNT 5.1 mozilla-central talos nochromer 1556.0
Rev3 WINNT 5.1 mozilla-central talos other 1326.64634146
Rev3 WINNT 5.1 mozilla-central talos svgr 1524.90243902
Rev3 WINNT 5.1 mozilla-central talos tpn 1711.64634146
Rev3 WINNT 6.1 mozilla-central debug test crashtest 1308.125
Rev3 WINNT 6.1 mozilla-central debug test jetpack 1633.7875
Rev3 WINNT 6.1 mozilla-central debug test jsreftest 1833.8625
Rev3 WINNT 6.1 mozilla-central debug test mochitest-other 1391.5375
Rev3 WINNT 6.1 mozilla-central debug test mochitests-1/5 1398.2625
Rev3 WINNT 6.1 mozilla-central debug test mochitests-2/5 1269.4875
Rev3 WINNT 6.1 mozilla-central debug test mochitests-3/5 1545.325
Rev3 WINNT 6.1 mozilla-central debug test mochitests-4/5 1451.45
Rev3 WINNT 6.1 mozilla-central debug test mochitests-5/5 1265.275
Rev3 WINNT 6.1 mozilla-central debug test reftest 1292.25
Rev3 WINNT 6.1 mozilla-central debug test xpcshell 1563.7875
Rev3 WINNT 6.1 mozilla-central opt test crashtest 1645.51219512
Rev3 WINNT 6.1 mozilla-central opt test jetpack 1549.59756098
Rev3 WINNT 6.1 mozilla-central opt test jsreftest 1926.76829268
Rev3 WINNT 6.1 mozilla-central opt test mochitest-other 1845.15853659
Rev3 WINNT 6.1 mozilla-central opt test mochitests-1/5 1392.75609756
Rev3 WINNT 6.1 mozilla-central opt test mochitests-2/5 1516.2804878
Rev3 WINNT 6.1 mozilla-central opt test mochitests-3/5 1643.31707317
Rev3 WINNT 6.1 mozilla-central opt test mochitests-4/5 1658.19512195
Rev3 WINNT 6.1 mozilla-central opt test mochitests-5/5 1658.73170732
Rev3 WINNT 6.1 mozilla-central opt test peptest 1412.69512195
Rev3 WINNT 6.1 mozilla-central opt test reftest 1557.63529412
Rev3 WINNT 6.1 mozilla-central opt test reftest-no-accel 1773.42682927
Rev3 WINNT 6.1 mozilla-central opt test xpcshell 1849.8902439
Rev3 WINNT 6.1 mozilla-central talos chromer 1764.8902439
Rev3 WINNT 6.1 mozilla-central talos dirty 1440.97619048
Rev3 WINNT 6.1 mozilla-central talos dirtypaint 1697.02439024
Rev3 WINNT 6.1 mozilla-central talos dromaeo 1511.4047619
Rev3 WINNT 6.1 mozilla-central talos dromaeojs 1442.2
Rev3 WINNT 6.1 mozilla-central talos jstests 1709.88095238
Rev3 WINNT 6.1 mozilla-central talos nochromer 1299.13414634
Rev3 WINNT 6.1 mozilla-central talos other 1595.7195122
Rev3 WINNT 6.1 mozilla-central talos svgr 1865.98780488
Rev3 WINNT 6.1 mozilla-central talos tpn 1861.63414634
Rev3 WINNT 6.1 mozilla-central talos xperf 1551.48809524
Rev3 WINNT 6.1 x64 mozilla-central opt test crashtest 1598.8375
Rev3 WINNT 6.1 x64 mozilla-central opt test jetpack 1258.225
Rev3 WINNT 6.1 x64 mozilla-central opt test jsreftest 1472.35
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitest-other 1490.725
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-1/5 1404.075
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-2/5 1376.5875
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-3/5 1592.975
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-4/5 1463.95
Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-5/5 1449.9
Rev3 WINNT 6.1 x64 mozilla-central opt test peptest 1529.0
Rev3 WINNT 6.1 x64 mozilla-central opt test reftest 1672.975
Rev3 WINNT 6.1 x64 mozilla-central opt test xpcshell 1316.7625
Rev3 WINNT 6.1 x64 mozilla-central talos chromer 1588.0875
Rev3 WINNT 6.1 x64 mozilla-central talos dirty 1083.07142857
Rev3 WINNT 6.1 x64 mozilla-central talos dirtypaint 1317.36842105
Rev3 WINNT 6.1 x64 mozilla-central talos dromaeo 1551.97619048
Rev3 WINNT 6.1 x64 mozilla-central talos dromaeojs 1061.57894737
Rev3 WINNT 6.1 x64 mozilla-central talos nochromer 1117.725
Rev3 WINNT 6.1 x64 mozilla-central talos other 1464.9625
Rev3 WINNT 6.1 x64 mozilla-central talos svgr 1294.5875
Rev3 WINNT 6.1 x64 mozilla-central talos tpn 1587.9875
Rev4 MacOSX Lion 10.7 mozilla-central debug test crashtest 1413.51190476
Rev4 MacOSX Lion 10.7 mozilla-central debug test jetpack 1886.20238095
Rev4 MacOSX Lion 10.7 mozilla-central debug test jsreftest 1689.70238095
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-other 1419.03529412
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-1/5 1367.14285714
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-2/5 1447.96428571
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-3/5 1306.36904762
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-4/5 1560.03571429
Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-5/5 1346.22619048
Rev4 MacOSX Lion 10.7 mozilla-central debug test reftest 1160.25
Rev4 MacOSX Lion 10.7 mozilla-central debug test xpcshell 1641.98809524
Rev4 MacOSX Lion 10.7 mozilla-central opt test crashtest 1567.85148515
Rev4 MacOSX Lion 10.7 mozilla-central opt test jetpack 1488.48514851
Rev4 MacOSX Lion 10.7 mozilla-central opt test jsreftest 1610.36633663
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-other 1559.95049505
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-1/5 1551.06930693
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-2/5 1439.67326733
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-3/5 1611.44554455
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-4/5 1471.16831683
Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-5/5 1648.7029703
Rev4 MacOSX Lion 10.7 mozilla-central opt test peptest 1444.3960396
Rev4 MacOSX Lion 10.7 mozilla-central opt test reftest 1564.81553398
Rev4 MacOSX Lion 10.7 mozilla-central opt test xpcshell 1543.82178218
Rev4 MacOSX Lion 10.7 mozilla-central talos chromer 1400.03960396
Rev4 MacOSX Lion 10.7 mozilla-central talos dirty 1479.14814815
Rev4 MacOSX Lion 10.7 mozilla-central talos dirtypaint 1492.04950495
Rev4 MacOSX Lion 10.7 mozilla-central talos dromaeo 1390.87037037
Rev4 MacOSX Lion 10.7 mozilla-central talos dromaeojs 1378.08510638
Rev4 MacOSX Lion 10.7 mozilla-central talos jstests 1653.78181818
Rev4 MacOSX Lion 10.7 mozilla-central talos nochromer 1320.79207921
Rev4 MacOSX Lion 10.7 mozilla-central talos other 1492.05940594
Rev4 MacOSX Lion 10.7 mozilla-central talos svgr 1500.45544554
Rev4 MacOSX Lion 10.7 mozilla-central talos tpn 1516.94059406
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test crashtest 1336.52380952
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jetpack 1568.20238095
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jsreftest 1532.78823529
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-other 1355.49425287
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-1/5 1452.45238095
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-2/5 1497.10714286
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-3/5 1482.0952381
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-4/5 1443.01190476
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-5/5 1237.79761905
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test reftest 1343.89411765
Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test xpcshell 1545.88372093
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test crashtest 1669.87128713
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jetpack 1679.46534653
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jsreftest 1621.41584158
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-other 1568.45098039
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-1/5 1644.94059406
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-2/5 1851.26732673
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-3/5 1618.50485437
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-4/5 1634.09803922
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-5/5 1557.76153846
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test peptest 1636.86138614
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test reftest 1602.41176471
Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test xpcshell 1590.94059406
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos chromer 1665.55882353
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dirty 1248.79245283
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dirtypaint 1670.63366337
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dromaeo 1626.81481481
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dromaeojs 1625.02083333
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos jstests 1739.57407407
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos nochromer 1436.52941176
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos other 1602.79207921
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos svgr 1676.52475248
Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos tpn 1481.43564356
WINNT 5.2 mozilla-central build 1420.93258427
WINNT 5.2 mozilla-central leak test build 1364.53932584
WINNT 6.1 x86-64 mozilla-central build 1437.09090909
jetpack-mozilla-central-fedora-debug 2145.93333333
jetpack-mozilla-central-fedora-opt 1462.06896552
jetpack-mozilla-central-fedora64-debug 1806.75862069
jetpack-mozilla-central-fedora64-opt 1603.75862069
jetpack-mozilla-central-leopard-debug 1972.31034483
jetpack-mozilla-central-leopard-opt 2482.82758621
jetpack-mozilla-central-lion-debug 1446.65517241
jetpack-mozilla-central-lion-opt 1509.27586207
jetpack-mozilla-central-snowleopard-debug 1982.5862069
jetpack-mozilla-central-snowleopard-opt 1654.34482759
jetpack-mozilla-central-w764-debug 2471.62068966
jetpack-mozilla-central-w764-opt 1800.65517241
jetpack-mozilla-central-win7-debug 1794.37931034
jetpack-mozilla-central-win7-opt 1567.93103448
jetpack-mozilla-central-xp-debug 2147.37931034
jetpack-mozilla-central-xp-opt 1786.24137931
============================================

HOWTO: Use crutches

I spent the best part of 2 years on crutches after a venom bite and an unrelated fall. Thankfully, I’ve made a full recovery. With all that time on crutches, I built up some tips/tricks. Since then, whenever I see someone on the street who is obviously new-to-crutches, I strike up a casual conversation about how long I spent on crutches, empathize about the various awkward realities of crutches and generally try to be encouraging. If things feel ok, then I ask if its ok to share some tips I learned from my time on crutches. This happened recently with someone in the office, and afterwards, they encouraged me to blog this in case it is helpful to anyone reading who is also newly on crutches.

1) buy fingerless cycling gloves
Sore hands from pressure of the crutch handle is a problem. Some people prefer to pad/wrap the crutch handle, but I found this still left lots of friction / movement-under-pressure while using crutches. Over time this friction led to blisters on the palm of your hands, between your thumb and first finger – exactly the spot that you needed to press your weight on while using crutches. Painful.

Padded cycling gloves solved this problem perfectly. These are padded in exactly the right area… between the base-of-thumb and forefinger…and they’re cheap! Being fingerless, I could still type and answer my cellphone with them on. And the mesh backing kept my hands from getting too hot while wearing them indoors all the time, so I could just easily leave them on all day.

2) be aware of possible nerve damage to armpits
If you get sore hands (see above), its easy to then start resting your full weight on your armpits on the top of the crutches. Every now and then is typically ok. However, its easy to get into the habit of resting your full weight on your armpits on the crutches. Dont. Leaning your weight on your armpits on your crutches can lead to nerve damage in your armpits, which will seriously limit future use of your arms and hands.

From wikipedia, “A condition known as crutch paralysis, or crutch palsy can arise from pressure on nerves in the armpit, or axilla. Specifically, the brachial plexus in the axilla is often damaged from the pressure of a crutch…In these cases the radial is the nerve most frequently implicated; the ulnar nerve suffers next in frequency”. More details are also at  http://en.wikipedia.org/wiki/Axillary_nerve.

Instead of resting your bodyweight on your armpits on the top of your crutches, use your arms to hold up your body weight. As a fulcrum point, squeeze in your arms to pin the top of the crutch between your upperarm and ribs – without contact to your armpit. Then try to move around the room. I found my arms were not use to this much work and quickly got tired. Start with short distances. Over time, build up to longer sustained use of crutches as your arms build up strength.

On the bright side: when I finally came off crutches, my arms were in the best shape ever. And I had no nerve damage in my arms and hands!

3) buy a jogger fanny pack, specifically one that has holders for full-size water bottles.
One of my most frustrating episodes soon after being on crutches was my first attempt to “have a normal day at home”, by myself, making my own coffee and hoping to sit at the computer with my leg elevated. It was tricky to wrangle the machine/filter/coffee/water while on crutches, but the brew smelled great – and the first sip was wonderful.

Success! If I can make my own coffee, how bad can life be?! Then I realized I had no way of getting from the kitchen to the chair, on crutches, while holding my prized mug of coffee. Foiled!

Weeks later, I had a brainwave. The water bottle holders in fanny packs is the same diameter as most car coffee travel mugs. Coffee in a travel mug in a fanny pack doesn’t spill while you are on crutches.

Make coffee. Pour into travel mug. Put travel mug into fanny-pack-bottle-holder. Use crutches to get to chair. Sit. Drink coffee from travel mug. Rejoice.

ps: While you have this fanny pack, you might as well also pack it with:

  • house keys
  • powerbars
  • cellphone
  • tissues

Thats it. Hope those hints help someone out there… and of course, if you have other tips or hints, please email me or add them to comments below.

take care
John.