Bluish Coder: git

Performing Compatible Updates of Mozilla Central Git Forks

2011-10-28T16:00:00+13:00

Earlier this year I posted about how I created my git mirror of the mozilla-central mercurial repository. I’ve been keeping a fork on github updated regularly since then that a number of people have started using.

Users of my mirror have wanted to be able to keep up to date with the mercurial mozilla-central repository themselves, or add updates from other branches of the Mozilla source. It takes a large amount of time to start a conversion from scratch but it’s possible to start from the existing git mirror, perform the incremental updates from mercurial yourself, and still stay compatible with the incremental updates I push to my github mirror.

hg-git uses a git-mapfile in the .hg directory of the mercurial clone to keep track of the mapping between mercurial and git commits. By using the same git-mapfile that I use for my mirror you can start your own incremental update from the latest data in the git-mapfile instead of from the beginning. I keep an up-to-date git-mapfile in git-mapfile.bz2. It’s compressed with bzip2 as the uncompressed version is quite large.

The steps to start your own update process are:

Clone the mercurial mozilla-central repository
Clone my git mirror as a bare repository in .hg/git
Place the git-mapfile in .hg
Do an hg bookmark -f -r default master to mark the commit to convert to
Perform hg gexport to update .hg/git with recent mercurial commits

The .hg/git directory should now be up-to-date with respect to the mercurial repository. And the additional git commits will have the same SHA id as any commits I push into my mirror when I perform my own update.

The steps to do an incremental update are the normal:

Pull from the mercurial mozilla-central repository
Run hg bookmark -f -r default master
Run hg gexport
Push or pull to/from .hg/git as needed

As a working example, the following shell commands on a Linux system should set up your own repository ready for incremental updating:

$ hg clone https://hg.mozilla.org/mozilla-central
$ cd mozilla-central/.hg
$ git clone --bare git://github.com/doublec/mozilla-central.git git
$ wget http://www.bluishcoder.co.nz/git-mapfile.bz2
$ bunzip2 git-mapfile.bz2
$ cd ..
$ hg bookmark -f -r default master
$ hg gexport
$ cd .hg/git
$ git push --all ~/some/other/repo.git

Incremental updates just repeat the last few commands above:

$ hg pull -u
$ hg bookmark -f -r default master
$ hg gexport
$ cd .hg/git
$ git push --all ~/some/other/repo.git

Note that I use hg gexport and push from the .hg/git repository using the git command instead of doing an hg push and relying on hg-git to do the conversion and push. In my original article I did the latter. The hg-git push method uses slower python based routines to do the push which can take a long time and large amounts of memory on big repositories like mozilla-central. By splitting this up into gexport and using the native git command I save a lot of time and memory.

My Git Workflow for Mozilla Development

2011-04-16T17:00:00+12:00

Since my post on converting mozilla-central to git I’ve had a few requests about what my git workflow is for Mozilla Development.

When working on bugs I’ll create a git branch for development. The workflow in this branch looks like:

git fetch origin
git checkout -b bug/123456 origin/master
...make changes...
git commit
...make more changes...
git commit

Occasionally I’ll want to merge with the latest code from mozilla-central. I do this by rebasing my changes on top of the latest trunk code:

git fetch origin
git rebase origin/master

I’ll often have multiple temporary branches as I work on different ideas during the fix. I find being able to diff between the branches, cherry pick patches, etc useful.

When I’ve finished with the fix and want to generate a patch for review I rebase on top of trunk and squash all my commits down into one patch. I do this use git’s interactive rebase:

git fetch origin
git rebase -i origin/master
...squash commits into one commit and set commit message to my checkin commit message...
git hgp >~/mypatch.patch

The last command, git hgp, uses an alias I got from Rafael Espindola’s Blog. You can install this alias by adding the following to your ~/.gitconfig file:

[alias]
hgp = show --format=\"From: %an <%ae>%n%s%n%b\" -U8

The adds an ‘hgp’ git command that does the same as git show but includes a header for the committers name. This allows an hg import of the patch to include the correct patch authors details which is useful if you use the checkin-needed keyword for others to commit the patch.

I attach the patch file to the bug for review. If I get review comments I then go back to the git branch and make the necessary changes. Regenerating the patch involves repeating the steps above.

When the patch needs to be committed to mozilla-central it can be imported and pushed directly using mercurial:

hg clone https://hg.mozilla.org/mozilla-central
cd mozilla-central
hg import ~/mypatch.patch
hg push

Git Conversion of Mozilla Central

2011-02-10T14:00:00+13:00

The Mozilla repository for Firefox development, mozilla-central, is a mercurial repository. I prefer using git for version control so I usually do my Firefox development using a git mirror of the mozilla-central repository.

To convert the repository into git I used hg-fast-export. This has worked well over the last couple of years. Unfortunately I’ve noticed that my git mirror and the official mercurial repository differ slightly. The git mirror contains files that have been deleted or moved in the official repository and lately the mirror has stopped building due to some commits not being correctly converted. As a result I stopped updating my mirror and used mercurial directly.

I still prefer developing using git though so I tried the hg-git mercurial plugin. This took about six days on my laptop to do the conversion and peaked at 14GB of memory. The result was a git repository with a working tree that, according to diff, is an exact copy of the original mercurial repositories working tree. Another nice feature of the hg-git is that commit id’s in the resulting git repository are the same across multiple runs of hg-git. This would allow other users to push and pull from a git repository that they converted independently.

To perform the conversion I installed hg-git and ran the following series of commands:

$ mkdir git
$ cd git
$ git init --bare
$ cd ..
$ hg clone https://hg.mozilla.org/mozilla-central hg
$ cd hg
$ hg bookmark -r default master
$ hg push ../git

Six days and 14GB later, the resulting git repository ended up being about 7GB in size. I did a git gc --aggressive on it to get this down to 200MB. Incremental updates of new mercurial commits are done with:

$ cd hg
$ hg pull -u
$ hg bookmark -f -r default master
$ hg push ../git

I’ve pushed a mirror of this to github at http://github.com/doublec/mozilla-central. I’m using this for development and will see if it is able to continue importing without diverging from the original mozilla-central source.

More information about the use of git for Mozilla development can be found in the followup posts I did on the subject:

Changes to git mirror of Firefox repository

2008-12-04T17:32:00+13:00

Original Post Changes to git mirror of Firefox repository

Git, Binary Files and Cherry Picking Patches

2007-09-06T17:03:00+12:00

Original Post Git, Binary Files and Cherry Picking Patches