Year: 2012

Infrastructure load for November 2012

John2 Comments
  • #checkins-per-month: We had 5,646 checkins in November 2012. This is slightly down from last month’s record of 5,893, but still the 3rd highest number of checkins in 2012, after October (5,893 checkins) and August (5,803 checkins). Again, we handled this record load with >95% of all builds consistently being started within 15mins. Sadly, our test pools continue to have a hard time, both with the increased rate of checkins, and the increased number of test suites being run per checkin. We’re continuing to work on buying and powering up more test machines, please continue to bear with us. Meanwhile, if you know of any test suites that no longer need to be run per-checkin, please let us know.
  • #checkins-per-day: During November, 19-of-30 days had over 200 checkins-per-day. Worth noting was spikes of 295 checkins on 13nov and 301 checkins on 27nov.
  • #checkins-per-hour: The peak across this month was a new record 13.36 checkins per hour between 10-11am. Of interest, I note that for 25% of every day (6 of every 24 hours), we sustained over 10 checkins per hour.

mozilla-inbound, mozilla-central, fx-team:
Ratios of checkins across these branches remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch, with 26.9% of all checkins, consistently far more then the other integration branches combined (fx-team has 0.8% of checkins, mozilla-central has 2.2% of checkins). As usual, very few people land directly on mozilla-central – in fact more people go through approval process to land on mozilla-aurora.

mozilla-aurora, mozilla-beta:

  • 4.9% of our total monthly checkins landed into mozilla-aurora. This is an decrease from last month’s high, but still higher then is typical. I believe this is caused by the number of b2g changes being landed into aurora, which then stopped ~20th November, with the migration of b2g from aurora to beta.
  • 2.2% of our total monthly checkins landed into mozilla-beta. This is typical for beta. I suspect the recent transition of b2g to beta just before holiday season here in the US, means we didnt see many b2g-related checkins on beta for November and I do expect beta to show increased checkins for December.

(Standard disclaimer: I’m always glad whenever we catch a problem *before* we ship a release; it avoids us having to do a chemspill release and also we ship better code to our Firefox users in the first place.)

misc other details:

Android x86 builds now on tbpl.mozilla.org

John3 Comments

This week, kmoir quietly enabled Android x86 builds on tbpl.m.o. This was already announced in this week’s Mozilla developer platform meeting but is important enough to repeat here too.

These builds are being run per-checkin, as well as nightly. There are updates for nightly builds. Obviously, there’s a lot of details to still work through (crash symbols, release builds, signing, etc…), but this is a major milestone worth noting. You can follow progress, and offer help to kmoir, on the remaining work in bug#750366.

As always with standing up new platforms in production, if you see problems on tbpl.m.o, please do not just hide the breakages; instead, please file bugs in mozilla.org:ReleaseEngineering and we’ll be happy to investigate.

Big thanks to kmoir, callek, jmaher, blassey and others for making this happen.

Yoda does powerpoint

John2 Comments

Concise. Accurate. Perfect. Just perfect.

(credits: Looks like graphjam.com is now part of cheezburger.com empire, which confused tracking down the original author. Digging around, I found a few versions of this on different sites going back through 2011. I *think* its originally from Nathan Yau on flowingdata.com, or Garr Reynolds, posted here but if you know anything about the original author, please let me know.)

multi-locale b2g builds now on tbpl.mozilla.org

John1 Comment

(This is so important to FirefoxOS localization that its worth cross-posting.)

In case you missed it, bhearsum just announced that b2g builds on tbpl.m.o are now multi-locale.

There’s two sets of locales here, so to be precise:

  • existing desktop and unagi, otoro builds: ar, en-US, fr, es, pt-BR, zh-TW. These locales are specifically chosen because they help developers debug various font-specific issues (right-to-left, etc).
  • new desktop builds: ar, as, bn-BD, ca, cs, cy, de, el, en-US, eo, es, et, eu, ff, fr, fy-NL, ga-IE, gd, gl, he, hi-IN, ht, hu, id, it, ja, ko, lij, ml, nl, or, pa, pl, pt-BR, ro, ru, sl, sq, sv-SE, te, tr, ur, zh-CN, zh-TW. These are the list of locales being actively worked on. We expect this list to continue growing. For progress, track this file: https://github.com/mozilla-b2g/gaia/blob/master/shared/resources/languages-all.json

For more info, check out bhearsum’s blogpost1 or blogpost2 or bug#766962.

Big thanks to bhearsum, stas, aki and others for making this happen. And hey, if you want to help with localizing boot2gecko / FirefoxOS into your locale, please contact the friendly folks in dev.l10n newsgroups – they’d be delighted with your help.

Otoro and Unagi builds now on tbpl.mozilla.org

John1 Comment

In case you missed it, Catlee has quietly enabled Otoro builds on mozilla-beta. This means that now:

  • per checkin, we generate a full oroto build. This otoro build is on the same changeset as the unagi build, the gecko-compiled-for-arm-with-b2g-enabled builds, and all the usual desktop Firefox and mobile Fennec builds that we already generate per checkin. Having these builds on tbpl helps narrow down build regressions. However, as these full otoro builds are ~164MB each, and we do a *lot* of builds per day, we do not plan on uploading these until we have tests to run against them. Of course, if you have a need for these, please let us know, and we’ll enable them, its an easy change.
  • every night, we generate and publish an oroto build, an unagi build, the gecko-compiled-for-arm-with-b2g-enabled builds, b2g desktop builds, on the same identical changeset. As usual, all the desktop Firefox and mobile Fennec builds are built on the one changeset.

These new otoro builds have already been quickly evaluated by RelEng and by QA, and as far as we can tell, they look fine to us. However, if you have an Otoro phone, and have access to the private share, you can help! Please grab one of these new Otoro builds, which you’ll find alongside the previous otoro builds, install it on your Otoro phone and give it a try. We’re expecting to transition everyone over to these builds in the coming days, so if you see anything which makes you think we should stop rollout, please file a bug and we’ll get right on it.

RelEng production systems now in 3 AWS regions

John4 Comments

tl;dr: We’re now running our build infrastructure across 3 different Amazon regions. This makes us more robust and *cheaper*?! Whats not to love?! 🙂

This is important for 3 reasons:

1) It means RelEng can keep up with load, and hence keep all trees open, even if an Amazon region goes offline, or if a VPN link fails. Amazon doesn’t lose a region often, but it can happen. Mozilla doesn’t lose a VPN link often, but it can happen. Using 3 different regions, with 3 different VPN links, makes it unlikely we’d lose all at one time. In fact, multi-region outages on AWS are so rare, that the most recent multi-region outage I could find was in June2008.

2) As our first “go hybrid on AWS VPC” is a clear success, we’re experimenting with VPCs that are further away (ie slightly slower connection) from our inhouse colos. This allows us to start using regions that are cheaper (good!), and not inside the same earthquake zone (also good!).

3) We have a month or so of realistic usage data on AWS, as all the builds which we can run on AWS are now running on AWS. (The only exceptions are recent requests for new B2G builds, which we’re still setting up). This means that we can now make decisions about bulk-purchase-in-advance instances (called “reserved instances” in AWS lingo), which buys us the same compute power for ~1/4 the price. As these reserved instances are region-specific, and cannot be refunded or swapped around to other regions later, it was worth bringing new cheaper regions online first before we start bulk purchases in those cheaper regions.

All in all, this is a big deal.

Oh, and it is worth noting for the record that these additional regions were brought online, and into production, without needing any downtime, and without any hiccups! Big thanks to catlee and rail and ravi for their work.

Khmer team at MozCamp Singapore

John2 Comments

At MozCamp Singapore, I was delighted to meet up with some of the Khmer localization team again. From left to right, here’s Vannak Eng, Sokhem Khoem, John O’Duinn, Sophea Sok, Mark West and Piseth Kheng. (Javier Sola was unable to make the trip at the last minute, so is sadly missing from the photo.)

They are very modest, great fun, and super sharp. As a sneak surprise, it was great to see them hack together a Khmer version of FirefoxOS during a quick hackfest arranged by Stas at MozCamp.

Since we last met at my workshop in Phnom Penh in Jan 2012, we officially released Khmer as part of the Firefox13.0 release and government policy now has Khmer Firefox being used in government departments in Cambodia! Great stuff.

If you want to find out more about the Khmer team, or want to join them in localizing, please look here: https://wiki.mozilla.org/L10n:Teams:km or http://www.mozillakm.org/

HOWTO use an unlocked Android phone in Malaysia

John2 Comments

Here what I used in my trip to Malaysia in Nov2012, in case others find this helpful:

Disclaimer:

  • In the US, buying a cellphone “out-of-contract” is not the same as buying a cellphone “unlocked”. All of the following only works for an unlocked phone. Make sure your phone is unlocked before you get on the plane.
  • Different cellphone companies have different policies on this. AT&T declared that, despite my being a multi-year customer, with no contract, they would not unlock my phone per policy. T-Mobile said upfront that they would need ~40days from date-of-purchase of “out-of-contact” phone before I could ask to have it unlocked. On the 40th day, when I asked T-Mobile to unlock my phone, they sent me the phone unlock codes within 48hours.
  • Make sure your phone supports GSM. Sounds obvious, but still needs to be said, as most countries use GSM.
  • NOTE: You need to show your passport, or national ID card, when buying a pay-as-you-go SIM, or a phone, in Malaysia.
  • Buy a “HotLink” pay-as-you-go SIM card. I bought mine at the airport in Kuala Lumpur, but they are also for sale on most small street corner stores. While there are several mobile companies selling pay-as-you-go, I went with HotLink because they had the best price for data at 4G-LTE speeds, great high speed coverage everywhere I went, and no hassle about using your cellphone as a hotspot. Oh, and comparable prices for voice calls and text messaging.
  • Disassemble your phone to swap out sim card, insert new HotLink sim card and power up the phone.
  • On the phone, enter “*122#″ and press dial (typically, the green handset button). This will connect you to an automated voice service which will tell you your balance.
  • Assuming that works, you should now attempt to call any local number. By habit, I now call the mobile phone of the person at the store selling me the SIM card.
  • Cultural tip: I never setup voicemail – as discovered in my other recent trips to SEasia, most people dont both leaving voice messages – if they cant reach you when they phone, they hangup and send you a text message instead.
  • At this point you should be able to make/receive calls.
  • To make my Android 2.2 phone transmit/receive data, I had to add the following APN settings:
    * on home screen, go into “settings”
    * go into “wireless & network settings”
    * go into “mobile networks”
    * go into “access point names”
    * if there is not already a “maxis” APN, then create one as follows:
    ** Name = maxis
    ** APN == bbnet
    ** Proxy == 202.75.133.49
    ** Port == 80
    ** Username == Not set
    ** Password == Not set
    ** Server == Not set
    ** MMSC == Not set
    ** MMS proxy == Not set
    ** MMS port == Not set
    ** MCC == 502
    ** MNC == 12
    ** Authentication Type == Not set
    ** APN Type == Not set
    …hit save, and go back to “Access Point Names”.

  • verify that this new “maxis” APK is present, and is selected.
  • verify that “Use only 2G networks” is not selected.
  • Reboot the phone to see if that helps.
  • Now that the phone is configured correctly, I selected the 500mb-per-day data plan, as follows:
    enter “*100*9*1#” and press dialer
    read menu, enter “2” and click “ok”
    read menu, enter “1” and click “ok”
    …thats it.

  • At this point you should be able to make/receive calls, send/receive text messages, surf the web, and use your cellphone as a wifi hotspot.
  • To check your account balance call *122#.
  • When you need additional credits, buy a one-time use scratch-refill “top up” card at almost any corner store, and follow the instructions on the back. Alternatively, their website says you can topup using PayPal as well as credit cards, but I never personally tried that. Either way, you’ll receive a text message with the new balance when the credits are added to your account.

Infrastructure load for October 2012

JohnLeave a comment
  • #checkins-per-month: We had 5,893 checkins in October2012. This is a new record, breaking the previous record in August of 5,803. Even more impressively, we handled this record load with >95% of all builds consistently being started within 15mins.
  • #checkins-per-day: During October, 19-of-30 days had over 200 checkins-per-day.
  • #checkins-per-hour: The peak across this month was 12.87 checkins per hour between 10-11am, a new record, closely followed by 12.35 checkins per hour between 1pm-2pm.

mozilla-inbound, fx-team:
Ratios of checkins remain fairly consistent. mozilla-inbound continues to be heavily used as an integration branch, with 28.6% of all checkins, far more then the other integration branches fx-team (0.7% of checkins) or mozilla-central (2.5% of checkins). As usual, very few people land directly on mozilla-central – in fact more people go through approval process to land on mozilla-aurora.

mozilla-aurora, mozilla-beta:

  • 5.7% of our total monthly checkins landed into mozilla-aurora. This is an increase over previous months, and I suspect is caused by the number of b2g changes being landed into aurora.
  • 2.6% of our total monthly checkins landed into mozilla-beta.

(Standard disclaimer: I’m always glad whenever we catch a problem *before* we ship a release; it avoids us having to do a chemspill release and also we ship better code to our Firefox users in the first place.)

misc other details:

Infrastructure load for September 2012

JohnLeave a comment
  • #checkins-per-month: We had 5,315 checkins in September2012, a drop from August.
  • #checkins-per-day: We had a new record of 315 checkins in one day (25sep). Thats a checkin every 4.5mins for 24 hours. This is a significant jump above our previous record of 284 checkins per day. During September, 16-of-30 days had over 200 checkins-per-day, but it looks like beginning of September was quieter than end of September. Not sure if this is because of vacations at the start of the month, or because of B2G deadlines at the end of September… or a combination of both?!
  • #checkins-per-hour: The per-hour peak across this month was slightly down, at “only” 11.3 checkins per hour.

mozilla-inbound, fx-team:
Ratios of checkins remain consistent. mozilla-inbound continues to be heavily used as an integration branch, with 24.1% of all checkins, far more then the other integration branches fx-team (1.5% of checkins) or mozilla-central (2.8% of checkins). As usual, very few people land directly on mozilla-central – in fact more people go through approval process to land on mozilla-aurora.

mozilla-aurora, mozilla-beta:

  • 3.3% of our total monthly checkins landed into mozilla-aurora.
  • 2.2% of our total monthly checkins landed into mozilla-beta.

(Standard disclaimer: I’m always glad whenever we catch a problem *before* we ship a release; it avoids us having to do a chemspill release and also we ship better code to our Firefox users in the first place.)

misc other details: