Dec 11
JohnUncategorized
Ted’s recent landing of bug#421534 was blogged about by Ben Hearsum here. While it might initially seem like a real snooze, its actually really important for our ability to support multiple active project branches.
Each build slave keeps copies of the srcdir/objdir directory tree. Fair enough. However, we keep multiple copies. By the time you keep a directory tree for incremental builds, debug builds, leak builds, nightly builds, and release builds… and do that for each of mozilla-central, tracemonkey and mozilla-1.9.1… suddenly that all adds up.
By far the worst offender here was the Mac, with the “-save-temp” workaround for a compiler problem forcing each tree we kept to be huge (10GB); with Ted’s fix in place, that same tree is 3GB. Ted’s work on DWARF buys us a bunch of breathing room so we can still create new project branches as needed right now, while we continue to look for a more scalable solution.(any ideas/suggestions very very very welcome!)
Given the number of directory trees, and the number of active branches, buying more disks is only a short term stopgap solution. With the number of machines we have, its also an expensive short term stopgap. In terms of building scalable infrastructure, it would be great if each slave be able to support 5-6 different types of builds, across 8 or more simultaneous different project branches. We might never quite reach a perfectly clean “leave-no-trace” build, but every time we reduce the amount of space a build tree consumes on the slaves, it helps us scale up the number of build types, and project branches, we can provide for developers.
Thanks Ted.
Dec 08
JohnUncategorized
“…1941 — a date which will live in infamy — the United States of America was suddenly and deliberately attacked by naval and air forces of the Empire of Japan.”
A lot has changed in the last 67 years, politically, economically and personally, it was the year my father was born. But listening to President Roosevelt’s live speech the morning after Pearl Harbor still gives me goose-bumps. Every time. Its only 6 mins long, so worth a listen (click here for wikipedia’s ogg/vorbis version). The written speech misses out the emotion, timing, and reaction of the people present.
The interesting little legal detail here is that, back in 1941, the only way people got to hear these broadcasts were over the commercially owned airwaves, or commercial newsreels in cinemas. This has the unusual side effect of many recordings of historically important national events being owned by private commercial broadcasting companies, not by the public.
Here in the US, the incoming Obama administration is making an important move to use new technologies to get government owned content directly to the public. This makes lots of sense in a generation where people get their news on The Daily Show, and google.com/news while NPR and traditional printed newspapers try to survive with plummeting audiences. Changing the broadcast medium is just one part. It’s also important that the new content be available to the public, not just owned by just a different private commercial company. The work that Mitchell and John Lilly have been doing here with Creative Commons licensing is literally history in the making.
Dec 08
JohnUncategorized
Since Feb 2008, we’ve fixed:
- a talos redness that hit us 2-3 times *every* day since Talos first started (caused by a synchronization problem between the build and talos systems)
- made it possible for IT to support these Talos machines as part of their oncall work (making all the Talos machines boot cleanly into working state).
- simplify (at least a little) how to setup new Talos machines on a new project branch.
Building on those fixes, we’re now focusing again on reducing the variances between the different Talos machines. We’ve always tried making the machines be as identical as possible to each other (sequential serial numbers, carefully controlling what is installed, etc), but even so, still there was a lot of variance in the test results. Last week however, after weeks of testing, Alice and Chris AtLee made a breakthrough:
Having Talos machines cleanly reboot after completing every 5th job, and before accepting another job means that developers never see any burning. There’s still details to be worked out before we can roll this out across all Talos machines across all branches… but so far, this looks really encouraging.
For background info, have a look at bug#463020 and the live graph on graphserver. If you have any suggestions, or ideas which might help, we’d love to hear them.
Dec 08
JohnUncategorized
Canadian journalist Melissa Fung was recently released after being kidnapped and held hostage in Afghanistan for a month. Just after her release, she did an interview with CBC (her employer) which struck me as extraordinarily honest, direct, personal and respectful retelling of everything that happened.
http://www.cbc.ca/mrl3/8752/news/features/invu-fung-cbc.wmv
I found her calm presence of mind throughout the experience quite impressive, and at the same time, humbling. Some of that probably came from the specialized K&R training before traveling on assignment, and the rest had to be from the strength of her core inner personality.
The interview is almost an hour long and well worth watching.
Dec 05
JohnUncategorized
We had a lot going on during Monday’s downtime. One small, yet important, event was the powering down of the 7 Talos machines running on FF2.0.0.x / mozilla-1.8.0.
These machines came online in December2007, and were originally needed for performance comparisons with the “new development work” going into Firefox3.0 at the time. Which was great. But, we’ve long since shipped Firefox 3.0, and most developers have moved on to FF3.1 and FF3.next. The world has moved on, so powering off these machines seemed useful because:
- these machines can now be recycled and put to use where we need them more – doing talos or build work on other, more active, code lines.
- this simplifies our life when doing Talos improvements for FF3.0, FF3.1 and mozilla-central. There’s interesting differences in how Talos collects info across the various release branches. By turning off this set of Talos machines, it removes one set of cross-testing we have to do. This improves our “drag coefficient” when making changes to Talos for other branches.
- minor incremental reduction of load on graphserver (recall that on FF2, we build continuously, meaning we also test, and post results continuously!)
- no-one was using them anymore. We spend much of our time supporting machines that people actually use. This made powering off unused machines, especially before the EOL of FF2.0.0.x, a psychologically important milestone.
Many thanks to everyone for the honest, and flexible, discussions in bug#463325 and the “Power off unused FF2.0, FF3.0 machines?” thread in dev.planning leading up to this.
Nov 28
JohnUncategorized
“The answers you seek are in Norway”
huh???
Suddenly, I realize that *making* fortune cookies could be a lot of
fun! 
Nov 26
JohnUncategorized
Heading home Friday evening, I stopped by Aki’s desk and took these quick photos. At first glance, these might not look all that exciting:
…but look closer… closer… see the little bits of paper taped on the machines??
This is actually really really big! This is the beginning of a pool of mobile devices slaves to automatically run unittest and talos on mobile builds.
Aki’s been using these two machines to figure out linux mobile hardware setup problems around the hardware setup, toolchain setup, memory limitations as well as how to get a buildbot slave running on the device, communicating to a buildbot master.
Since end of Sept, we’ve been producing linux-arm builds of fennec automatically on checkin and every night. However, humans were still needed to manually download specific builds onto the physical devices when they had time, and manually run some tests. There’s still no automated unittests, talos, graphserver for mobile.
The build automation was done by updating all the linux slaves in our existing pool of slaves, and then doing cross-compile builds. That allowed us to re-use a bunch of existing infrastruture. However, we need to use actual devices for testing and performance… which opens a whole set of problems we have to figure out for the first time. For example:
- if the device hangs during a test, how do you powercycle it? (Most people first suggest removing battery and running on remote switchable a/c; however, these devices refuse to boot unless a battery is present, and it looks like they need a physical button pushed on after a power reset, so we’re still looking for a solution – all suggestions welcome!)
- How do these devices communicate to other machines; how do we get builds onto, and test data off from, these devices? The buildbot master needs to tell slaves when and what to do. The slaves need to post results to buildbot master as well as ftp.m.o and graphserver. Something (nagios?) needs to monitor to see if devices are ok. If we use wifi, does that cause noisy deviations in our test results? If we use ethernet, what s/w drivers and cable converters do we need?
- If we setup devices in office, for easier manual reboots, what ssh/firewall changes do we need to get access to other build systems in secured area in colo?
- What toolchain do we need to install on these devices? Both of these devices were “polluted” when we got them, meaning no-one was exactly sure whats on them, and we’re fairly sure they’re different from each other. We’ve got some more machines on order, so will have to figure out clean-setup-instructions as part of this setup.
- Is there enough memory in the device to run o.s. + buildbot + talos + talos-tools + fennec? Can far can we increase memory? Do we have to rewrite buildbot/talos to fit into low memory situations?
- We can’t build on these devices, but unittests currently requires building first before testing.
- What can we do about turnaround time? These devices are *SLOW*… Break testsuites into chunks that can be run in parallel?
And all that is before you get to the the fun stuff… the usually expected problems, like how many tests fail because of different display properties, memory availability, slower CPUs causing test timeouts, environment differences, etc. etc. etc.
While there’s still lots to solve along the way, its exciting to watch Aki making methodical, persistent and rapid progress. Very very cool.
Nov 24
JohnUncategorized
After upgrading from OSX10.4 -> OSX10.5, I was surprised to discover that Spotlight was no longer indexing Thunderbird emails.
I went back and rechecked all the steps in my earlier blog post, thinking maybe some files were lost in the upgrade, but all appeared ok. Using “/usr/bin/mdimport -L" I verified that the importer was present and still running. I even tried "/usr/bin/mdimportmdutil -E" to do a complete new re-index, in case it was somehow corrupted. Still no success.
Re-reading my earlier instructions, this step caught my eye:
- Move Thunderbird.mdimporter to either “~/Library/Spotlight/†(which I did) or “/Library/Spotlight/†(as suggested in some other posts)
…so as an experiment, I moved the Thunderbird.mdimporter directory from “~/Library/Spotlight/” to “/Library/Spotlight/”, used "/usr/bin/mdimport -L" to verify that it was running in the new location. Immediately, Spotlight was finding matches within my Thunderbird emails.
The only change I did was to move the Thunderbird.mdimporter – could the location of this really be o.s. version specific? Maybe that explains why some people used one location, and some the other?
(If anyone reading this has done these same steps, I’d be very curious what location are you using for Thunderbird.mdimporter, and what version of OSX you are using!)
UPDATE: At first, Spotlight worked fine, but problems arose soon after the original blog post. Spotlight started giving garbage results pointing to unrelated files, showing confusing icons and filenames for matches, and eventually a system crash. I suspect that Spotlight from 10.5 got confused by the existing indexed data of Spotlight from 10.4, but have no data to back that theory.
To get Spotlight working with Thunderbird properly, I did the following:
- move the Thunderbird.mdimporter directory from “~/Library/Spotlight/” to “/Library/Spotlight/”
- use
"/usr/bin/mdutil -E" to clean out the existing indices.
- reboot the computer
- use
"/usr/bin/mdutil -E" to clean out the existing indices again. (maybe unneeded but by now I was being paranoid!)
- used
"/usr/bin/mdimport -L" to verify that it was running in the new “/Library/Spotlight” location.
- wait a few mins, then try searching your email.
- smile and celebrate with a cup of coffee.
Hope that helps – John. 04dec2008
Nov 24
JohnUncategorized
Last Wednesday, we setup the new Firefox3.1 project branch. (Details on the machine-setup work coming in a separate blog post.) Meanwhile, I thought people might find this diagram useful to help understand the repository cloning/branching work that happened.
Note:
- To keep terminology consistent, last week we created a “project branch”. We’ll be creating “release branches” on this project branch for each milestone.
Before we allow checkins to the new mozilla-1.9.1 project branch, we will need to carefully cross-check all the changes landed on mozilla-central since the mozilla-1.9.1 branch was created (The area in the dotted circle) and verify that those changes were also landed on mozilla1.9.1.- We want the mozilla-1.9.1 branch to start from FF3.1beta2. To speedup opening the mozilla-1.9.1 branch, we branched early. This means that before we allow checkins to the new mozilla-1.9.1 project branch, we will need to carefully cross-check all the changes that landed on mozilla-central since the mozilla-1.9.1 branch was created (The area in the dotted circle), up to the branch for FF3.1beta2, and verify that those changes were also landed on mozilla1.9.1. Any later FF3.1beta2 respin would require patches to also be landed on mozilla-1.9.1.
- There’s been discussions recently on whether we’d have a FF3.1b3 or not. I included FF3.1b3 in the diagram so that, if we do decide to do Beta3, people could see where it would be branched from. This is not meant to state or imply that we are doing Beta3!
- The diagram above shows logical active-branches-of-code. At the SCM repository level, some of these active-branches-of-code are implemented as in-repo-branches, and some are implemented as separate-cloned-repos. For the sake of clarity, I’ve omitted those details in this post, but would be happy to go into further detail, if anyone is interested.
Hope that helps?
tc
John.
UPDATE#1: added extra text in 2nd bullet point, describing the cross-branch synchronization, to clarify how we branched and why.
UPDATE#2: Sam Sidler and and Gavin Sharp both pointed out that the branching diagram had an error when describing how FF1.5 was branched from FF2.0. They are correct, and I thank them. The branch diagram above has been updated, and is now correct. I since found the same error in a presentation we gave here at the 2008 Mozilla Summit (diagram had same error), and a brownbag in Mountain View office (diagram had same error).
Nov 23
JohnMozilla, Tech tips
The basic technique here is to export all your Contact info from Palm’s format to Microsoft Address format, and then import from there into the Apple iPhone Contacts app. The best instructions I’ve found so far were here, but I’ve added extra gotchas below in case it helps others. Note: this only transfers Contact info, and does not transfer Calendaring, ToDo or anything else.
Before starting, you need to do the following:
- On a MS Windows PC, install palm desktop and iTunes. (If you primarily use iTunes on another computer, its ok to just install iTunes on the windows PC, do this one-off data transfer, and then throw away that iTunes installation).
- Start Programs->Accessories->AddressBook and make sure that the Microsoft Address book is empty. (Note: this is not to be confused with MS Outlook, which is very different!)
- On the Palm Pilot/Treo, look through all the contacts for the following gotchas:
- If any contact has two entries of the same type (for example, a person with two mobile numbers, or two work phone numbers), you will need to manually remove/rename one of these numbers before you start. I noticed that any contacts with multiple entries of the same type ended up losing all but one.
- If you have the same person listed multiple times in your Palm Pilot/Treo, this will cause an error later on. I discovered that I accidentally had the same person entered twice, in two different categories. These duplicate names caused errors later on in the export process, so its best to check and fix this before you start. Worst case, if you miss a duplicate, its quick and easy to just throw away all the conversion work and restart from the beginning again….but it sure it annoying!
- Check for any occurrences of ‘= (single-quote followed by equals) in your contacts, or attached notes. If you find this anywhere, you must change them to something else before you can continue. It seems that ‘= (single-quote followed by equals) is used as a delimiter somewhere in the conversion process, so anything after that gets cut from that specific person’s info, and never transferred.
- “Custom fields” are not transferred, so you should either migrate that data to one of the “standard” fields, or make a note of them, and come back later to cleanup by manually copying from palm desktop into MS Address book.
OK, thats it. Now we’re ready to begin:
- Hotsync your Palm pilot/treo with the Palm Desktop one final time. Remember, backups are your friend!
- Start Palm Desktop and view the Address page. Select the contacts you want to bring over (ctrl-A on keyboard if you want them all), then choose File->ExportvCard… to save all your contacts into one single file on your desktop.
- Start Programs->Accessories->AddressBook
- Drag and drop the newly created file from your desktop into Microsoft AddressBook. You’ll be asked to press “ok” for each entry being imported into AddressBook. Annoying but its very quick.
- Unplug your Palm Pilot/Treo, and plug in your iPhone.
- Start iTunes. On the iPhone page, Info tab, choose to sync contacts with Microsoft Address Book and click Apply.
- Sync your iPhone with iTunes – this will in turn pull the contacts from the Microsoft Address Book.
- On iPhone, look in Contacts app, and verify that all your contacts transferred over correctly. Specifically, look for long attached notes to make sure nothing was truncated.
- Undo step (6) in iTunes.
Thats it!
Again, this only transfers Contact info; I’m still investigating exporting Calendaring, ToDo and other types of “legacy data” – any hints?
Big tip-o-the-hat to: http://www.fixya.com/support/t161691-downloading_contacts_from_palm_iphone and also http://www.dba2csv.com. Both were wonderful help!
All this left me wondering why Apple didnt provide an import utility to handle migrating from Palm Pilot to iPhone…or from BlackBerry to iPhone. That would sure make it easier for business users to migrate over to iPhone. Oh well.
UPDATE: These same instructions worked great tonight when migrating a friend from a Palm Pilot to a new iPhone 4Gs. Added warning about custom fields, which I missed in this first blog. joduinn 04-dec-2011
Older Entries Newer Entries
RSS