Mozilla

Is it git or is it github?

joduinn9 Comments

Background:
At the Mozilla AllHands in Sept2011, I was surprised to find that my proposed session on git was accepted, and even more surprised at how well this session was attended! The room was full of people from all different groups across Mozilla. And boy, were people passionate. The slides don’t capture the lively back-forth discussions, but this is obviously a topic that people care deeply about, hence my blog post.

Here’s a quick summary of the main speaking points:

  • Some projects in Mozilla are now basing their code in github.com, not on Mozilla’s existing release infrastructure. They do this despite the fact that github in one sense has reduced functionality (doesnt do all builds/test/automated-regression alerts/etc), and despite that using github like this causes extra manual headaches of periodically importing/exporting code, and complicates branch mechanics of releases. So why do this?
  • Theory#1: git vs hg
    • Some people prefer to use the git commandline tool. At same time, some people prefer to use the hg commandline tool. Each Distributed Version Control System has various technical merits, and drawbacks, so I put this git-vs-hg debate into the same category as an emacs-vs-vi debate. (I do *not* mean that as a put down – I mean this in the nicest possible way – the reality of life is that people have their own unique hard-earned preferences)
  • Theory#2: github vs hg+bugzilla+graphserver+tbpl
    • Instead of this debate being about the mechanical differences of the command line tools, I instead believe the debate is actually about Developer Workflow. This is not just theoretical; it makes a day-to-day difference to how developers get their job done.
    • Every mozilla developer ends up using hg+bugzilla+graphserver+tbpl. These are organically grown, over the years, and each went from hacky-experiments-to-mission-critical once people started to rely on them. However, they are not smoothly cross-integrated.
    • Mozilla saw a jump in checkins-per-day when tbpl.mozilla.org went live; tree closures were shorter, in part because people could more easily figure out who broke the build, and who/what to back out. (Of course, there’s more to it then *just* tbpl, but the usability issue here is the key point).
    • While different people have talked about this, in different companies over the years, I think github.com is a really compelling proof point that good developer workflow makes a difference. If you can make it easier for a developer to find a bug, get the bugfix reviewed and landed, and be able to see if it made a difference, developers will WANT to use your production systems.
    • Apart from workflow, there’s also security and autonomy concerns. How valuable are your bug discussions, review history, regression tracking? Who else would you entrust this valuable data to? If you find someone you trust, how do you know they’ll be around as long/longer then Mozilla?

    Hal Wine has now got up-to-speed in this area because of his work on git-staging. Next, Hal is starting to see how feasible would it be to support both git command line tools and hg command line tools in our production RelEng + WebDev + IT systems. And meanwhile, Hal, LauraT, myself and some others are gathering to see what we can do to improve developer workflow. As Mozilla grows, its important to make it easier to do coding work here – after all, this is one way to encourage more people to contribute, and to help Mozilla scale.

    If you have suggestions, or want to help, we’d love to hear from you – or if you prefer, you can just follow the tip of the iceberg in bug#713782.

9 thoughts on “Is it git or is it github?”

  1. I see a high degree of autonomy and decentralization within Mozilla – in some ways this is a blessing, and it others this is a curse.

    Github in particular has one major feature that the Mozilla infrastructure completely lacks, which is the push request. Seamless submission of patch requests, without jumping through any major hurdles, without need to be approved, vouched, vetted, etc to have your own public repository as a fork off of an official one. and to have patches sent directly to developers, with tools right there to review them and merge them in.

    Mozilla’s developer tools on the other hand are downright ugly. They are highly functional, but the rough edges are everywhere, and prospective users are getting cut on them before they can learn to appreciate them. Consequently, newer teams are avoiding these tools, even when doing so is an (obvious, at least to experienced developers) step backwards.

    Now, you could impose a top down edict as far as what tools to use, but that would just hide the problem – that the tools aren’t working for some people.

  3. Github in particular has one major feature that the Mozilla infrastructure completely lacks, which is the push request.

    I think there’s another – one which I don’t think we can, or even should, go for.

    There is a network effect to DVCS. This perhaps shouldn’t come as a surprise since DVSCs are about sharing. A traditional VCS is fine if you don’t want to distribute/share/network.

    So Github has pull requests, which we can get, but it also has the winning network, which we probably should not try to get.

    Since we want to distribute/share, perhaps we should try to integrate as closely as we can with the winning network rather than copy it.

  5. I partly agree with your assessment of the git versus hg command-line interfaces as an issue of personal taste. I also agree with you that workflow matters much more. However, git-versus-hg does have one major workflow difference not related to github and other surrounding communities: the acceptance and optimization of history-editing as a standard workflow technique. “git rebase” and “git rebase -i”, “git commit –amend”, and numerous other commands help reinforce the notion that you can freely edit any history you haven’t yet published to get it into the state you want to push or to create the set of patches you want to mail. hg, by contrast, strongly discourages this kind of workflow, and does not optimize for it at all.

  6. Interesting post. I’m an Eclipse release engineer and I migrated many of our project’s code repositories from CVS to Git this summer. For legal reasons all of our code must reside at eclipse.org. We don’t ship code from external repos. (The exception being third-party libraries which have been cleared by our IP team). Thus eclipse.org provides Git infrastructure and all of our repos are mirrored at GitHub. I gave a presentation at Eclipsecon Europe 2011 on our Git migration experience if you are interested :-). http://www.slideshare.net/mobile/k2moir/migrating-to-git-rethinking-the-commit

  7. I’m quite nervous about github because it’s 1) a centralized (instead of decentralized) platform, 2) not open source by itself, and 3) not controllable by Mozilla and therefore potentially losing info if it goes down (apart from a disturbance factor we can’t control ourselves).

    git itself is great (even though I like the hg commands better) and the DVCS workflow is also great, of course – but making ourselves dependent on a proprietary platform is not really in line with what Mozilla should advocate.

  8. re: history editing

    I agree that there’s a difference between git and hg, but… well, the other way around. The majority of mercurial-using Moz developers seem to use mq, which is a much more flexible way to reorganize unpublished history. Base mercurial also has hg rebase, hg pull –rebase, hg crecord, hg qcrecord, etc. which seem to cover the same ground as the commands you listed. (Note that git has stgit and guilt, which are equivalent to mq, but seemingly less commonly used.)

    I actually think there’s a more substantial difference in the common workflows with published changes — specifically, whether people want to maintain linear history (“fast-forward commits” in git terminology) or are ok with lots of branchiness. Both git and hg have similar support for both, but there seem to be social differences between the two communities. The one major functionality difference between the two — namely, whether commits are permanently associated with a line of development or not — is somewhat relevant, though there are good (and vehement) arguments on both sides of that question.

  9. re: sfink

    I switched from hg to git for day-to-day development because b2g uses git; I was kind of forced to.

    But now that I’ve learned enough git to get by (no small task; git is way harder to learn than hg+mq), the killer feature for me is: I don’t have to rebase my patches until I want to.

    With hg, I’d commonly do something like the following:

    1) Work on patches for bug A.
    2) Forget about those patches for three weeks. In the meantime, I’ve hg pull -u’ed many times.
    3) Try to push patches for bug A. Now none of my patches apply, and I have to manually apply the .rej files.
    4) Go back to step (2).

    With git, my patches for bug A are attached to a base revision of m-c which only changes when I want it to. This is a huge improvement!