Continuous Integration

Whilst Nix's strict control of software versions and dependencies is excellent when handling other people's programs, it can be a little frustrating when developing one's own software.

In particular, version control systems like Git have their own notion of immutable state (commits) and dependencies (each commit's hash includes that of its parent, just like Nix derivations include their dependencies' hashes). This similarity can often lead to overlaps, where I need to synchronise Nix packages with git updates, or vice versa.

I've tried several methods of resolving this, but have finally found one that I'm happy with. Here I'll explain these approaches.

Manual Git Revisions

This is the most tedious, but it ensures you're using a known-good configuration. For example, this is the way you'd manage an official package in nixpkgs.

Let's say we're maintaining the following packages foo and bar. For the sake of argument, let's say we're maintaining these definitions in our ~/.nixpkgs/config.nix file:

foo = stdenv.mkDerivation {
        name = "foo";
        src  = fetchgit {
                 url    = "https://example.com/foo.git";
                 rev    = "f000001";
                 sha256 = "f000000000000000000000000000000000000000000000000001";
               };
      };

bar = stdenv.mkDerivation {
        name = "bar";
        src  = fetchgit {
                 url    = "https://example.com/bar.git";
                 rev    = "b000001";
                 sha256 = "b000000000000000000000000000000000000000000000000001";
               };
        buildInputs = [ foo ];
      };

Now let's say we make a change to foo, which lives in a new git revision f000002. We need to go and edit our package definition:

foo = stdenv.mkDerivation {
        name = "foo";
        src  = fetchgit {
                 url    = "https://example.com/foo.git";
                 rev    = "f000002";
                 sha256 = "f000000000000000000000000000000000000000000000000001";
               };
      };

In fact, this package will not build, since the SHA256 hash will not match. Keep in mind that the SHA sum of the git commit doesn't have anything to do with the SHA sum of the nix derivation, so we can't just copy it over. Instead, we must ask Nix what the new SHA sum should be, and the easiest way to do that is to attempt to build the package:

$ nix-shell -p foo
...
output path ‘/nix/store/...foo’ should have r:sha256 hash ‘f000000000000000000000000000000000000000000000000001’, instead has ‘f000000000000000000000000000000000000000000000000002’
...

This is what we were expecting, so we can now safely copy that new hash in place of the old one:

foo = stdenv.mkDerivation {
        name = "foo";
        src  = fetchgit {
                 url    = "https://example.com/foo.git";
                 rev    = "f000002";
                 sha256 = "f000000000000000000000000000000000000000000000000002";
               };
      };

Clearly, this is a tedious process. The worst part is that we need to do the same thing even for tiny changes. Let's say that bar requires some small change to the API of foo, e.g. changing a private function into a public one (before you scoff, read on for a treatment of the software engineering aspects!).

Even if we have a nice development workflow for bar, with this setup we will still need to bump the version of foo globally, just to have the new version available to bar.

This is problematic in a couple of ways: firstly, all package definitions will now point to the new version of foo. Of course, thanks to the way Nix works this won't affect any existing derivations (i.e. all of our installed packages will carry on working as-is), however it will affect new derivations, like those instantiated by nix-shell. Symptoms might include a load of expensive rebuilds, or subtle breakages that we don't want to contend with right now (software engineers, read on!).

The other problem is that our change might not work! With this approach, we must go to all this effort and bump the global version of foo just to try something out!

As for the software engineering perspective, it's certainly true that any change in public API should lead to a version increase, and is not considered "tiny". It's also true that we should try not to release new API versions if they're known to break existing clients, at least without understanding whether the breakages are acceptable.

However, our problem shouldn't have anything to do with releases and versions! Even at the stage of experimenting and prototyping, to see if the proposed change will even work in the first place, we are suddenly forced to deal with a release management scenario!

This is obviously a bad state of affairs, in particular because it penalises modularity: these headaches in tying packages together will subconsciously bias us against re-using existing components, or splitting up our task into smaller sub-tasks.

Ignoring Git

We can also go to the opposite extreme and ignore version control altogether! Consider the following:

foo = stdenv.mkDerivation {
        name = "foo";
        src  = ~/Programming/foo;
      };

bar = stdenv.mkDerivation {
        name = "bar";
        src  = ~/Programming/bar;
        buildInputs = [ foo ];
      };

Using hard-coded paths is clearly a bit of a code smell, as these package definitions are no longer portable to other machines. However, look at what we've gained! We no longer need to specify a git revision or an SHA256 checksum: Nix will scan the contents of the directories, and if they've changed from the last build it will copy them into the store and make a new derivation.

Unfortunately, there are obvious problems with this approach. In particular, we've still got the issue that any changes we make will have a global effect. In fact, we no longer have to commit something in order to alter our system! If we have unstaged changes in the foo or bar directories, any new derivations depending on these packages will get those changes, regardless of whether we intended them to or not.

This undermines a lot of the purpose of Nix, since we want to have confidence in our system integrity and the freedom to develop software without fear of breaking anything with half-finished changes.

Automating Updates

Another solution I tried is to automate the process of updating revisions and hashes. I wrote two scripts, called bumpCommit and bumpEverything, to aid this process. First, the packages would be defined to load their revision and hash attributes from external files:

foo = stdenv.mkDerivation {
        name = "foo";
        src  = fetchgit {
                 url    = "https://example.com/foo.git";
                 rev    = import "./foo.rev.nix";
                 sha256 = import "./foo.sha256.nix";
               };
      };

bar = stdenv.mkDerivation {
        name = "bar";
        src  = fetchgit {
                 url    = "https://example.com/bar.git";
                 rev    = import "./bar.rev.nix";
                 sha256 = import "./bar.sha256.nix";
               };
        buildInputs = [ foo ];
      };

bumpCommit would look at the current working directory, and use a look up table to find the corresponding rev.nix and sha256.nix files, which it would overwrite with the new versions.

Since these package definitions themselves are stored in git repositories, running bumpCommit on a project may cause changes in other projects, which requires more commits, and so on.

That's where bumpEverything came in: it would loop through all projects in dependency order, commit any outstanding changes to rev.nix or sha256.nix files, then invoke bumpCommit.

This was clearly a hack, and was very fragile in the face of changing projects, e.g. splitting a project into two parts. It also generated a large number of git commits, which would do nothing other than bump revision numbers and hashes.

latestGit

My current solution abandons the bumpCommit and bumpEverything commands. Instead, we try to combine the best features of the git approach and the no-git approach.

The aim is to treat a git URL as if it were a local directory: no need to specify a particular revision (just fetch a sensible default like HEAD, master, etc.) and no need to specify a particular hash (just like with local directories, if it's changed then build a new derivation; of course, in practice a particular git revision will not change, and is hence safe to cache).

To do this, we abuse Nix's idea of derivations in order to run arbitrary scripts and cache the results in the Nix store for subsequent access.

First, we write a "package" which is just a single file, containing the latest git revision of a particular repository:

with import <nixpkgs> {};
with builtins;

getHeadRev = { url, ref ? "HEAD" }:
  stdenv.mkDerivation {
    inherit url ref;
    name    = "repo-head-${hashString "sha256" url}";
    version = toString currentTime;

    # Required for SSL
    GIT_SSL_CAINFO = "${cacert}/etc/ssl/certs/ca-bundle.crt";

    buildInputs  = [ git gnused ];
    buildCommand = ''
      source $stdenv/setup
      # printf is an ugly way to avoid trailing newlines
      printf "%s" $(git ls-remote "$url" "$ref" | head -n1 | sed -e 's/\s.*//g') > "$out"
    '';
  };

The getHeadRev function takes a url parameter and, optionally, a ref (defaulting to HEAD). It then generates a package with a name derived from the given URL, and a version based on the current time; this ensures that we avoid the Nix cache.

The contents of the package, stored in the file at location $out, are generated by the buildCommand, which queries the given URL for the latest revision ID.

Next, we need a way to access this revision information from within Nix. We do this by coercing the packages generated by getHeadRev into strings, which will correspond the the $out path at which they're stored. We use readFile to read the contents of the generated files:

rev = args: unsafeDiscardStringContext (readFile "${getHeadRev args}");

We use unsafeDiscardStringContext to ignore the "dependencies" of this string, i.e. the particular invocation of the git ls-remote command; all we care about is the revision, not the time at which it was looked up.

Now that we have a git revision, we just need to avoid the hash requirement of fetchgit, to prevent it from being a so-called "fixed-output derivation". We do this in two steps, utilising Nix's lazy evaluation. First, we do a regular fetchgit invocation:

fg = args: fetchgit {
       url = args.url;
       rev = rev args;

       # Dummy hash
       sha256 = hashString "sha256" args.url;
     };

Finally, we override this derivation to strip out all of the hashing information:

latestGit = args: stdenv.lib.overrideDerivation (fg args) (old: {
              outputHash     = null;
              outputHashAlgo = null;
              outputHashMode = null;
              sha256         = null;
            });

Without these details, Nix will use the hashes it calculates from the source. Now we can use this latestGit function to specify our package sources:

foo = stdenv.mkDerivation {
        name = "foo";
        src  = latestGit {
                 url    = "https://example.com/foo.git";
               };
      };

bar = stdenv.mkDerivation {
        name = "bar";
        src  = latestGit {
                 url    = "https://example.com/bar.git";
               };
        buildInputs = [ foo ];
      };

This ensures that Nix and git are always synchronised, since git revisions are now the canonical form of package versions, and Nix will ask Git if these have changed when a new derivation is being instantiated.

From a development point of view, this gives us the freedom to tinker and experiment without fear of breaking our system. When our local changes are suitable for wider use, we can do the usual git push to make them available to the world; which now includes our installation of Nix!

Our packages are also portable, as long as latestGit is made available somewhere.

This doesn't quite solve the problem of using experimental versions of foo from within bar, however we're free to define extra packages, like foo-unstable, which use the local directory as their source. If we make foo a parameter of the bar package, we can override it with nix-shell to get a one-off development shell using foo-unstable, without breaking anything, without exposing our dodgy prototypes to the world, and without building a mountain of trivial git commits.