Feb 05
JohnUncategorized
To help with capacity planning, I pulled together some numbers for January. I’m still sorting through all this, but thought these early results were worth sharing.
In January, people pushed 1,128 code changes into the mercurial-based repos here in Mozilla.
As each of these pushes triggers multiple different types of builds/unittest jobs, the *theoretical* total amount of work done by the pool-of-slaves in January was 11,511. For each push, we do:
- mozilla-central: 11 jobs per push (L/M/W opt, L/M/W leaktest, L/M/W unittest, linux64 opt, linux-arm)
- mozilla-1.9.1: 10 jobs per push (L/M/W opt, L/M/W leaktest, L/M/W unittest, linux64 opt)
- tracemonkey: 7 jobs per push (L/M/W opt, L/M/W unittest, linux64 opt)
- theoretical total: (681 x 11) + (297 x 10) + (150 x 7) = 11,511 jobs. Or ~371 jobs per day. Or ~15 jobs per hour. (Considering how many of our jobs take over an hour to complete, this is quite scary!)
I say “theoretical total” because there are two complications here, which would slightly reduce numbers, so I dont yet have *actual* numbers:
- if two pushes arrive into hg.m.o on same repo in hg.m.o within 2 minutes of each other, we count them as one push, not two.
- if the entire pool-of-slaves is busy, then any pending build/unittest jobs get queued up for the next available slave. To stop the slaves from falling behind in peak times like that, we “collapse the queue”, and have the next available slave take *all* pending jobs. This is good from the point of view of keeping turnaround times as quick as possible, and keeping up with incoming jobs. However, it complicates regression hunting. Part of the reason for getting these numbers is to measure and see what we should do here.
…but this theoretical total is very close. I’m still working on this.
Some other details:
- a developer making ‘n’ changesets locally from a local repo and pushing them all up to hg.m.o at one time is counted as only one push. Put another way, this only counts changes landed into the mozilla-191/mozilla-central/tracemonkey repos based at hg.mozilla.org; this explicitly excludes “hg commits” to a developer’s local repo – which is what you see if you use “hg log”.
- its interesting to see the focus of activity, and the number of pushes, to a given repo change over time. This matches with the gut sense you get from irc/bugzilla, seeing people focus on one area and then move to another, but thats just my guess? Having the pool-of-slaves dynamically shift from one repo to another as-needed is really working well here.
- I’ve excluded all talos jobs, because those machines are organized differently, and I’ll need different math for that. Also excluded are all try-server jobs. Also excluded are all changes to FF2.0.0.x, TB2.0.0.x, FF3.0.x. Once I get the hg-based numbers going routinely, I’ll start to look at the cvs-based numbers.
Hopefully people find this is interesting, I’ll keep digging here.
Feb 03
JohnUncategorized
Aki got automated WinCE builds going yesterday! Isnt this awesome??!
Some quick details:
- These builds are available for download from ftp.mozilla.org since yesterday.
- The one broken build since yesterday afternoon / evening was caused by a patch merge conflict, which the developer since fixed, hence builds are back to green again.
- We are building using the mozilla-central and mobile-browser repos that will be used for the upcoming fennec-on-wince alpha.
- Disclaimer: the builds produce a zip file. Ted, Wolfe and Aki are working in bug#474530 to figure out Windows CAB installers. We also don’t yet have nightly updates available, thats still be worked out.
- belzner showed me a link to a demo of a recent WinCE build.
- Aki is swiftly developing a taste for fine Irish whiskey.
Nice work, Aki!
Feb 02
JohnUncategorized
Not sure what magic fix the fennec folks landed recently, but charts like this are really pretty, both because of the performance improvement, and also because of the improved stability of the results!
(Aki just pointed me to mfinkle’s post with details on the same nice improvement.)
Feb 02
JohnUncategorized
Two weeks ago, I added http://www.whitehouse.gov/blog/ to my daily RSS feeds.
Since then, each morning, I get to glimpse the volume, and sheer dynamic range, of work going on there. The speed of context switching makes my head spin. Lobbying restrictions. Child health care. Plans for Guantanamo. Weather emergencies in Arkansas and Kentucky. Interviews on Al Arabiya. Freedom of Information Act. A gender-equal payment law… Nice going for just two weeks, and that’s just the highlights of the unclassified stuff. For extra fun, you can now also see the full text of the new law, and even comment on it.
“…The law is now up on our website, where you can review its full text and and submit your thoughts, comments, and ideas.”
Lots of this information was probably available in different forms in the past, if you knew where to look and who to ask. But not easily to mere mortals. Getting this out of the hands of gatekeepers, or media spin-doctors and instead making it available directly to the public feels to me like how government is supposed to be. The phrase “…by the people, for the people…” seems refreshingly appropriate once again.
I really never thought I’d see the day, and I have to say this openness really makes me proud.
As an aside: If you take top-of-the-world financial institutions, and drive them literally to bankruptcy (or in some cases to rescue by government handouts), how exactly you can justify getting a bonus in that situation is beyond me. That $18billion (yes, billion with a B) could instead pay for a lot of salaries. I wonder how many of those laid-off financial workers, including some computer engineer friends of mine, would still have a job if that “bonus” money was used instead for employee salaries. These are the same financial institutions that make me sit silently every time I look at the carnage in my 401k. Shameful is one word for it.
Jan 26
JohnUncategorized
After buying an Apple iPhone 3G recently, I quickly discovered the reports about short battery life were all true – the Achilles Heel of an otherwise cool device. Making it through a day on just one charge was a challenge. Plugging in overnight was not optional. Once, I’ve even run out of power *despite* having started on a full charge – in frustration, I bought spare charger cables for the office, the backpack, adapters for the car. [Sigh]
Then I found Solio.
Basically, its a battery that you can charge using built-in solar panels or from a USB port on a laptop. Once charged, you can carry it in your bag, and then use it to charge an assortment of small electronics as needed.
 
Pro:
- Seems to charge quickly, and in various light levels.
- Holds that charge for at least up to a couple of weeks of being carried around in a bag, and still works as hoped.
- Brain dead simple to use.
- When fully charged, it can recharge a dead iPhone, and a GPS receiver, and a bluetooth headset, with still some extra juice left over for at least some part of another charge (still experimenting with power limits).
Con:
- The device has two connections on it; one for charging the battery (power in), and one for using the battery to charge other devices (power out). Personally, I’d prefer to have the power out connection (and maybe even the power in connection?) be standard mini-USB. All the electronics I use have cables that connect directly to USB/mini-USB, so this would have made sense to me. Instead, the Solio uses some proprietary form of pseudo-mini-headphone jack connection. They give you a cable to convert from that to USB, which you can then plug your USB-to-device cable, so it does work fine… but….well… it just seems to me that it could have been neater if there were fewer cables to carry around. Not sure how much money they make on selling the extra cables, and not sure of the design/manufacturing consequences of using a slightly larger mini-USB, but this feels to me like a v1.0 nit.
Summary: I’ve used this for about a month, and really like it so far. There’s something refreshing about a device that works as advertised, right out of the box! The nit about connection sockets is just that, a personal nit. Of course the real test will be later this year, when I go camping in the desert. Lets see how that goes!
Jan 21
JohnUncategorized
Last week we shipped our 5th funnelcake release.
Ken has been leading us doing one funnelcake release per month for 3 months now. Ken can best discuss the significance of all the stats/metrics which these funnelcake releases make possible.
However, I’m more interested in the process. I stumbled across this diagram in bugzilla. To me, this diagram accurately captures how different groups within Mozilla are coordinated do a specific funnelcake release. Having the bug dependencies match the reality is really important. In a time-crunch, with lots of people switching hands, it does make it easy to quickly see what needs to be done next. Each time we do a release, the process gets just a little more refined, and more efficient.
(click through to live bugzilla graph – with options at bottom to change how graph is drawn.)
Hope the diagram helps explain some of the mechanics?
John.
(Personally, I think its a little weird how arrows go “towards” the dependent bugs, not “away from” the dependent bugs. This feels like the “wrong” way to me, but hey, thats just my perspective/style. Certainly the dependencies and interconnected-ness of the bugs is accurate. We havent done a formal post-mortem yet, but based on informal discussions last week, each funnelcake releases feels like an improvement on prior releases; we should be able to improve on this release also.)
Jan 12
JohnUncategorized
To help give a rough outline of how 2008 looked like in RelEng, I thought these numbers were interesting :
At 2008-01-01
- 4 people (bhearsum, joduinn, nthomas, rhelmer)
- 3 code lines (FF2.0.x, FF3.0.x, and a few straggler machines on FF1.5.x)
- 89 machines
At 2008-12-31
- 9 people (alice, aki, bhearsum, catlee, coop, joduinn, nthomas, as well as part time: armen, lsblakk)
- 5 code lines (FF2.0.x, FF3.0.x, mozilla-191, mozilla-central, tracemonkey) all with full sets of build/unittest/talos machines
- 269 machines
I tried to figure out how many releases we did during the year. According to https://wiki.mozilla.org/Releases , we did 40 releases during 2008. However, there’s some other releases I remember happening which never made it to that list, so I actually don’t know what the fully accurate count is. Regardless, “at least 40 releases”, that’s a lot!!
Dec 26
JohnUncategorized
A friend told me she was unable to login to hotmail using Firefox anymore, getting some warning about not being a supported browser. She also had similar problems with Netflix. We’d recently released new updates on both Firefox 2.x and Firefox 3.0.x, so I stopped by her house to figure out what was going on, just in case…
Sure enough, we could easily reproduce the problems, so I started investigating. “hmmm… wonder what version she is using?
Ahhhhhhh!! A few minutes later I had installed FF3.0.5 over her existing FF1.0.7 installation.
Lessons learnt:
- Firefox1.0.7 works just fine on PPC based Mac OSX 10.5.5. It had never crashed for her, not once.
- A pave-over install of FF3.0.5 over FF1.0.7 works just fine; history and home page were correctly handled in the pave-over install and worked perfectly. I didn’t check bookmarks, because she never uses them, preferring to use URLbar history instead.
- The amount of websites that still work with really old browsers is quite impressive. Obviously, new functionality might not be visible/usable, but having a website gracefully fallback in functionality so it still does *something* reasonable on older browsers is tricky to do, and I was impressed by the number of sites that made that effort.
“Oh, thanks for fixing that – but whats changed in this new version?”. Oh boy – whats changed between FF1.0.7 and FF3.0.5? Honestly, I didn’t know where to start. Tabs? Memory improvements? JS performance improvements? Awesome bar? Phishing protection? …? After a brief hesitation, I decided to only describe one improvement: how the browser can now check for new updates automatically, and showed her where this was set in preferences, so she should never be out-of-date like this again.
Heading home, thinking about it further, I still think that was a good choice, but it got me wondering what other people would choose. So, I’m curious – if you had to list just one “best new feature” since FF1.0.7, what would you choose?
Dec 15
JohnUncategorized
Aki’s been busy; he’s rounded up 12 more n800/n810 devices, an old guitar pedal board, some velcro and sticky tape. Some firewall hacking, and we’re now seeing our first automated numbers up on staging graph server. My personal favorite details are the custom made stylus holders.
So far the heat from all the power supplies inside the case seems fine, but we’re keeping an eye on it.
This is really exciting progress, but we’ve still got a lot of work to do here.
Normal Talos runs do a lot of work to reduce noise and machine variance in the test results; we do a total of 15 iterations of the same test; (5 iterations on each of three separate machines). On these *slow* mobile devices, *one* iteration takes approximately 3 hours; good enough for us to check infrastructure is in place, but too noisy to be usable by developer – thats the results now visible on staging graphs above. Doing 5 iterations, like we do for PCs, would take 15 hours. Ideas we are investigating are:
- having multiple sets of devices running at staggered different times (one different set running every 8 hours, for example). We weren’t able to do this for PC builds because there was too much machine variance noise in the results, but maybe the ongoing rebootabilty work might finally solve that.
- only have one set of slaves running, but always test the latest pending build (skipping over other builds in the process). This avoids the machine variance noise, but would still peak out at one talos run every 15 hours, and if you miss that, you’d have to wait for the next 15 hour cycle to come around.
We’d like to be able to do more frequent runs if at all possible…any/all suggestions welcome!
Dec 11
JohnUncategorized
When: 9th-11th December 1968.
Where: The Fall Joint Computer Conference 1968, here in San Francisco
What: Doug Engelbart demonstrates a wooden box with wheels which moves a “tracking spot” on the screen. Click here for the presentation. He then went on to demo hyperlinking in clip7, clip8 here. I found the entire presentation oddly reminded me of the movie “Dr Strangelove“, maybe it was the dress code, B&W camera angles and voice echos, but still…surreal!
Talking points:
- he showed two different mice – one with the wire out the front, one with the wire out the back.
- quote of the day: “I dont know why we called it a mouse, sometimes I apologize; it started that way, and we never did change it”.
- seems that integrating todo lists with maps is still a killer app!
- in related news, Logitech built its billionth mouse last week – that is billion with a B!!
Older Entries Newer Entries
RSS