Please be gentle with the shiny new TryServer

Since the awesome redesign of TryServer that Lukas did recently, we’ve seen the load on TryServer jump significantly as people use it more. Since the start of the year, TryServer had ~15-35 pushes per day. Last week, TryServer had 91 pushes on Thursday, and 137 pushes on Friday.

Thats great proof of how much people like the new TryServer – cool – we want people to love it, and use it. After all, we made it better, because we hoped developers will find it even more valuable. We even added extra machines to handle possible growth throughout the rest of the year. However, the recent jump in usage over the last few weeks is already approaching the extra growth we’d predicted for the *year*.

We still want people to continue using TryServer, but from a scan through TryServer usage logs we’ve found two things that everyone can do to help avoid accidentally wasting TryServer capacity :

1) If you want to rerun a specific test suite on a specific build on TryServer, please DO NOT re-push the same source code to TryServer again because:

  • resubmitting source code tells TryServer to build new binaries and then rerun all tests on the new binaries, which may defeat the entire purpose of retesting!
  • you have to wait significantly longer for results, because it has to recompile everything first.
  • this will run a bunch of build/unittest/talos jobs you dont care about – well over 65 hours of computer time *per* push doing 32bit+64bit builds, opt+debug builds, unittests, talos, etc – which ties up those TryServer slaves from doing work for other people.

Instead, it is much faster, and more efficient, to file a bug in and have the RelEng buildduty person manually trigger just the test suites you want on your *existing* build. This is an interim step until we have a self-service UI for this. This same UI will also allow you to choose what you want run for your job on TryServer, an improvement over the current situation where we default to running everything.

2) Consider refreshing your local repo to latest changeset *before* pushing to TryServer. If your patch works as you hoped, you can then ask to land it on mozilla-central. By contrast, if you were based on 6-week-old repo when you pushed to TryServer, and everything worked, it doesnt really tell you much. You’re 6 weeks out of date – who knows what other code has changed in 6 weeks – so you’ll still have to refresh your local repo forward, and resubmit to TryServer *again* anyway to make sure your code is valid before asking to landing on mozilla-central.

Both of these changes could really help – every CPU cycle saved helps!!

3 thoughts on “Please be gentle with the shiny new TryServer

  1. Perhaps this is a sign that actually building and testing locally (i.e. running the full test suite) is becoming intractable to do locally, and therefore everybody just do it on the try server? I know people have written tests that had no chance of passing unless you did packaged tests, for example – which seems rather unlikely to happen in a development environment. Also: tests that require you to not use your machine (e.g. things that want focus) exacerbate the problem, as well as the sheer size of the full test suite. I know that I’ve never been able to actually finish one run locally, ever, even without any changes…

    It might be useful to figure out ways to make it easier to test things locally, so that it becomes more attractive to not need to use the try server.