Divmod Combinator : Tutorial

Combinator is a tool to help developers use and manage multiple branches of software hosted in a Subversion (SVN from now on) repository. If you are not familiar with branches and merging in SVN, I recommend you read Chapter 4 of Version Control with Subversion at some point - it's not a pre-requisite for using Combinator or reading this tutorial, but it definitely helps to have an understanding of what's going on behind the scenes.

Let's Play

Firstly, create a directory in which we can work our way through Combinator usage:

$ mkdir CombinatorTutorial
$ cd CombinatorTutorial
$

Combinator expects to find the trunk/branches/tags SVN repository directory structure recommended by Version Control with Subversion. I recommend you create a new repository just for this tutorial and add the standard directories to it:

$ svnadmin create repository
$ svn mkdir -m "trunk/branches/tags structure" "file://`pwd`/repository/trunk" "file://`pwd`/repository/branches" "file://`pwd`/repository/tags"
$

Note: the three file:// arguments above are enclosed in double quotes in case the path to your current working directory contains whitespace.

Note: if you're using Bash, you can use "file://`pwd`/repository/"{trunk,branches,tags}

Note: If you want to use an existing repository in working through this tutorial, you can. Just remember to use your repository when the examples below use file://`pwd`/repository and note that you'll not see a directory named repository as shown in the examples below. We will be making new branches, so if you take this route, you should be comfortable doing that to your existing repository.

Install Combinator (sort of)

Make sure you're in the right place:

$ pwd
/home/matt/CombinatorTutorial

Combinator is currently only available from Divmod's Subversion repository. Seeing as you have Subversion installed, it's easy to fetch. Either install the full Divmod suite (which includes Combinator):

$ svn co -q http://divmod.org/svn/Divmod/trunk Divmod/trunk

Or, if you only want to check out Combinator, just:

$ svn co -q http://divmod.org/svn/Divmod/trunk/Combinator Divmod/trunk/Combinator

Note: In either case, you must check out Combinator into a directory named /Divmod/trunk/Combinator - omitting the required trunk path segment will result in a broken installation, with neither Combinator itself nor the Divmod package functioning properly. Also, if you are installing Combinator without the other Divmod packages, you must still put it in a directory called Divmod/trunk/Combinator - those three path segments are the way Combinator discovers where to import itself from, and also where to place other files, such as the ones listed below. In this tutorial I've done the check out in /home/matt/CombinatorTutorial. You are of course free to do the check out wherever you like.

Now we'll set some shell environment variables:

$ pwd
/home/matt/CombinatorTutorial
$ eval `python Divmod/trunk/Combinator/environment.py`
[...]
$ ls
combinator_paths  Divmod repository
$

The eval command sets some Combinator variables into your current shell's environment. If you like Combinator you'll probably want to add that line to your shell's startup file (e.g., ~/.bashrc), but it's a bit early for that right now ;-).

combinator_paths is a directory that Combinator uses to record what branches you're working on and any binaries/scripts the branch provides.

Note: If you want to see just what this command will do to your environment, simply run python Divmod/trunk/Combinator/environment.py. You'll eventually want to do this if you're trying to get a fuller picture of how Combinator fiddles things on your behalf.

Note: Combinator tries to figure out what shell you're using, so the above eval command should Just Work. Let us know if not.

Starting work on the Project

Now we'll actually start using Combinator. Firstly, fetch your project's trunk from the SVN repository and register it with Combinator.

$ chbranch YourProject trunk "file://`pwd`/repository/trunk"
C: Checked out revision 1.
$ ls
combinator_paths  Divmod  YourProject  repository
$ ls YourProject
trunk
$

Here YourProject is the name we'll give to your project. Note that this could have been anything - it's just the name by which Combinator will know your project. file://`pwd`/repository/trunk is the repository you created up above - supposing you're not using your own pre-existing repository.

As you can see, Combinator created the main directory for YourProject development, and checked out the trunk.

Note: You may be wondering: "Where did that chbranch command come from?" The answer is that it's a python program that lives in combinator_paths/bincache and that that directory was sneakily added to

your shell's PATH variable in the eval command above.

Obviously, this is going to be a Twisted project ;-) so we'll get Twisted from SVN too. (Don't worry, we're just doing that so we have another respository checked out for the purposes of illustration.)

$ pwd
/home/matt/CombinatorTutorial
$ chbranch Twisted trunk svn://svn.twistedmatrix.com/svn/Twisted/trunk
[...]
$ ls
combinator_paths  Divmod  YourProject  repository  Twisted
$ ls Twisted
trunk

I mentioned above that Combinator records what branches you're currently working on. Let's find out what Combinator thinks I'm working on.

$ whbranch
YourProject: trunk
Twisted: trunk
$

As you can probably guess, whbranch magically popped into existence in the same way that chbranch did.

You can see that chbranch has not only checked a branch out for you, but it has also switched you to it (your trunk is just another branch after all, even if you're not used to thinking of it that way).

Create a Development Branch

At some point you decide to add a substantial new feature to YourProject. Following the principles laid down in Divmod's Ultimate Quality Development System we'd obviously never work directly on trunk, so we need a new branch to work on.

$ mkbranch YourProject say-hello
[...]
$ ls YourProject
branches  trunk
$ ls YourProject/branches
say-hello
$ whbranch
YourProject: say-hello
Twisted: trunk
$

The mkbranch command creates us a new branch, checks it out, and switches us to it (you'll see the corresponding svn commands in the output), as confirmed by whbranch.

Let's add an exciting new feature.

$ pwd
/home/matt/CombinatorTutorial
$ cd YourProject/branches/say-hello/
$ echo "print 'Hello, Combinator.'" > test.py
$ svn add test.py
$ svn commit -m ""
$

Hurray, done already! Note that the new change is only commited in the say-hello branch. Merge back to trunk so we can roll this out to our eager customers.

$ unbranch YourProject
[...]
$ 

If you understand svn, you will know that the svn merge done by unbranch actually merges only to the local copy of trunk. Check that the changes look sane, and if so, commit them. Note that unbranch is acting on your currently active branch (as determined and maintained by Combinator), so it's a good idea to know what this is before unbranching (though as unbranching is just a local merge into your trunk, it's easily undone via svn revert (you do have a clean trunk, right? if not, re-read the UQDS page to see why you should).

$ cd ../../trunk
$ svn status
A  +   test.py
$ svn commit -m "Merged say-hello to trunk."
$ whbranch
YourProject: trunk
Twisted: trunk
$

Note that after unbranching, Combinator has put you back on the trunk.

Merging Forward

A new feature means a new branch.

$ pwd
/home/matt/CombinatorTutorial/YourProject
$ mkbranch YourProject another-feature
[...]
$ whbranch
YourProject: another-feature
Twisted: trunk
$ cd branches/another-feature
$ echo "print 'Hello, Bloatware.'" > test.py
$ svn commit -m "Yes, another feature."
[...]
$

Suppose that half-way through your development of this new feature, someone commits new code to the trunk that would be really useful to this branch too. It's easy to pick it up and keep moving.

First, let's add the new feature to the trunk (note that you wouldn't do this normally - the new feature should be added on a branch, merged into the trunk, and then committed there.)

$ cd ../../trunk
$ touch another.py
$ svn add another.py
$ svn commit -m "A must-have new feature."
$

Now the trunk has the new feature. We can pick up the change and continue development on a branch by "merging forward". First, check in your changes on your branch. Then:

$ unbranch YourProject
$ mkbranch YourProject another-feature-2
$ whbranch
YourProject: another-feature-2
Twisted: trunk
$ cd ../another-feature-2/
$ svn status
M      test.py
$ ls
another.py  test.py
$

There are a couple of things to note here. Firstly, when we unbranch, we are merging in the another-feature branch (our original branch, the one we were working on before we noticed the irresistible new feature added to the trunk). We then make a brand new branch, another-feature-2.

Note: This may seem slightly wasteful, but branches are cheap in svn, so it's no big deal. If you understand what's going on here, you might consider cutting Combinator out of the loop at this point and entering an svn merge command to get the trunk changes into your current branch by hand. Don't. Combinator is handling remembering what revision numbers need to be applied - that's one of the main reasons to use Combinator. If you start messing around manually behind its back, things will break.

Finally, note that test.py was committed in our original branch, but not in the trunk. When we unbranch, Combinator is picking up that change and bringing into our local trunk. It's still not checked in to the trunk. Then, when we make a new branch (another-feature-2), Combinator again picks up the uncommitted test.py and puts it into the new branch. That's exactly what you want, and you can move on with your development just as though you were still on the original branch. This is another good reason why you should keep a clean trunk. If you happened to have some other changes sitting around in your trunk, mkbranch would sweep them up and put them into the new (another-feature-2) branch - which is not at all what you want. Note too that your trunk is now unclean as a local copy of test.py from your branch has been placed into it and svn knows nothing about this file (you should probably remove it, otherwise when you merge the new branch back into the trunk you might be in a slight mess --- someone needs to verify what actually happens here, I guess a merge, perhaps resulting in conflicts, or does svn notice that you have a local file whose name is the same as an incoming newly checked in file?)

Now complete the development of the new feature, merge to the trunk and commit, as before:

$ svn commit -m "Finally completed (punctuated) development of new feature."
$ unbranch YourProject
$ cd ../../trunk
$ svn status
M      test.py
$ svn commit -m "Merged another-feature-2"
$

What Does All This Buy You?

If you've read about branches and merging in the standard text on SVN, you'll know that 1) you're going to be typing some long and possibly hard-to-remember commands, and 2) that you're going to have to keep track of revision numbers and/or dig them out of svn stat --verbose output. This requires real care. And you're going to need to discipline yourself to put revision numbers into merge log messages to save future problems. All that is a hassle, and if you're a programmer your first thought is probably to write a couple of scripts to take care of these tasks. But there's no need: Combinator was written for this purpose.

But there's more. If you're working on a complex project, you'll have an environment that likely includes a test suite, some shell scripts, etc. It's a pain to set everything up every time you switch between branches in a project. You want to be sure that you're really running the test code on the branched source, not picking up part of one tree and part of another. You don't want to be constantly rolling tarballs or installing your code on your machine (which could also affect others). You want all that to happen locally. Combinator does this for you too.

If you understand how and why Combinator does its thing, you'll likely wind up with a setup in which you have Divmod/trunk/Combinator, combinator_paths, and YourProject all sitting as siblings in a directory (perhaps alongside Divmod/trunk/other-stuff, Nevow, Twisted, etc) and none of these will be installed in your python site-packages directory. Instead, you'll tell Combinator about them (usually just checking out the trunk "branch" of the projects you're using but not a developer on), and it will take care of fiddling python to find everything properly when your code does an import. That's a big win.

Final Notes

If it's not clear, Combinator does not preclude you from checking out and working on many branches of a project at the same time. For example, if you wanted to review (and run tests against) a couple different branches, you could do so with the following:

$ chbranch Divmod sprocket-123
$ unbranch Divmod
$ cd Divmod/trunk
$ svn diff | less
$ trial Divmod/Nevow/nevow/test
$ svn revert
$ cd ../../

$ chbranch Divmod petmonkey-456
$ unbranch Divmod
$ cd Divmod/trunk
$ svn diff | less
$ trial Divmod/Nevow/nevow/test
$ svn revert
$ cd ../../

Of course, if you will be committing the changes to trunk, there is no need to revert.

Combinator is a tool for managing branches and your python environment. Combinator remembers a current or default branch of a project for you, and makes sure that when you unbranch or import your project in python, that's the version of your code that's the subject of the action. That doesn't mean that you can't work on various branches simultaneously and use Combinator as a tool to manipulate branches. But given that Combinator alters your python startup to make sure you pick up its notion of the current branch, you're probably better off getting into the habit of switching to a branch with Combinator each time you go to work on it.

When you use the Combinator unbranch command to merge a branch into your local copy of the trunk, Combinator does not remove the branch. If you want to get rid of old branches you will need to remove them yourself using normal SVN commands.

You will need to follow the Divmod directory layout a little in order to be able to import your project successfully into python when using Combinator. Your project will not be installed into your python site-packages directory, so Combinator needs to know where to find it to set python up properly. For this reason (in order to import yourproject) you'll need to create a yourproject directory under your YourProject directory, and set up a __init__.py file. Try copying the setup of a Divmod project to make this work.

It might help to think of the chbranch command as "check-out (if necessary) and change to".

Combinator will insist on checking out your project as a sibling of the Divmod directory you created at the top of this page. You can't change that at present. If you've followed this tutorial and have decided you'd like to use Combinator for real, you'll likely want to install it elsewhere. To do this, make sure you use a shell that does not have the Combinator environment variables in it (unset COMBINATOR_PATHS and unset COMBINATOR_PROJECTS should help if you're using the Bash shell. Check the output of the environment.py command (without the eval) above for a hint on what to clean up though, as these names may change).

Combinator is a useful tool for BranchBasedDevelopment although I think it's still in its infancy. There are other aspects of merging that Combinator could manage that may make it more useful. For instance, some assistance with finalizing (commitbranch?) and aborting (rmbranch?) a branch might be nice. If I ever work out for sure what additional features are required I will create a ticket.

jethro@divmod.org