asv-nix

Readme


# Nix Plugin for Airspeed Velocity #

## About Airspeed Velocity ##

Airspeed Velocity (asv) is a benchmarking tool for Python. Its features include:

 - Measuring CPU time, wallclock time, peak memory usage, or any numeric value.
 - Integration with version control, e.g. to benchmark a range of git commits.
 - Measuring with different dependencies (e.g. alternative libraries).
 - Measuring on different machines.
 - Generating HTML reports, with graphs, step-detection, etc.
 - Running benchmarks in isolated environments.

All of this is very nice, but this last feature is rather tricky. There is a
plugin mechanism for defining environments, and plugins are provided for
virtualenv, conda and a "dummy" plugin which uses the system's installed
packages.

This is a bit unfortunate for two reasons:

 - These plugins can only manage dependencies which are Python packages. If a
   project requires non-Python dependencies, they'll have to be hacked in using
   setup/teardown functions or similar.
 - This limitation on dependencies prevents asv from being used (to its full
   extent, at least) on non-Python projects. Even if we write our benchmarks in
   Python, e.g. using `subprocess` to call out to our "real" project, that just
   turns our "real" project into a dependency of the benchmarks (as the previous
   point states) such dependencies can't be managed unless they're Python
   packages.

If we solve the first problem, we've automatically solved the second. That's
what this project tries to do.

## About Nix ##

Nix is a build/packaging system which is very powerful and modular. Packages can
be defined using a Turing-complete, pure functional, lazily evaluated, domain
specific programming language. Every package is built in isolation, the results
are cached along with the hashes of all inputs, and only rebuilt if some input
changes.

Rather than defining packages directly, it's common to write a bunch of helper
functions and define a package in terms of those. The 'nixpkgs' repository
contains a large collection of packages and helper functions, for example
`buildPythonPackage` which takes care of Python-specific tasks like setting
module directories, or `haskellPackages.mkDerivation` which handles tasks
specific to Haskell projects. This flexibility makes Nix usable for almost any
project in any programming language.

## About This Plugin ##

This project provides an asv plugin, which uses Nix to manage dependencies
instead of conda or virtualenv. The architecture of asv is understandably a
little Python-centric, so we take a few liberties in our configuration to make
using Nix more seamless.

## Using The Plugin ##

To run a benchmark, you'll need a Python package with both `asv` and this plugin
installed as modules. You could do this using `pip`, `apt-get`, etc. but we
might as well use Nix: the `default.nix` file defines a Nix package for this
plugin. It depends on an `asv` package, but at the time of writing `nixpkgs`
doesn't include one. The `shell.nix` file has instructions to fetch one, so you
should be able to just run `nix-shell` and be dropped into an environment with
both `asv` and this plugin available.

Most of the asv instructions apply as normal, so we won't cover them here.
Instead, we'll highlight the aspects which are unique to the Nix plugin.

### Quick Start ###

`asv` allows benchmarks to be managed separately to your project. That
complicates things a little, so let's assume that your benchmarks live alongside
your code. In your project repo's top-level directory, run `asv quickstart` to
generate some example benchmarks and a config file.

In your `asv.conf.json`, make sure `plugins` is set to `[ "asv_nix" ]` and
`environment_type` is `"nix"`. Also set `builders` to `{}` and `matrix` to `{}`.

The easiest way to specify your benchmarking environment is to define it using a
Nix file. Let's make a file called `benchmarks.nix` in your project's root
directory. For the sake of argument, we'll assume your project has a
`default.nix` which defines a package containing some binaries in `bin/`. The
following `benchmarks.nix` file will provide a `python` executable, which has
access to that `bin/` directory in its `PATH`:

    args:
    with import  {};
    with { pkg = callPackage args.root {}; };
    runCommand "env" { buildInputs = [ makeWrapper ]; } ''
      mkdir -p "$out/bin"
      makeWrapper "${python}/bin/python" "$out/bin/python" \
        --prefix PATH : "${pkg}/bin"
    ''

Now we'll tell `asv` to use this as our environment. In our `asv.conf.json` we
set `installer` to the following:

    "args: import (args.root + ''/benchmarks.nix'') args"

This should be everything we need to run the benchmarks, using `asv run`.
However, there is an annoying problem: we're looking for `benchmarks.nix` in
`root`, which is the version of the project that's checked out of version
control. Hence we need to commit `benchmarks.nix` to the repo before it will be
available; this isn't a good idea, since it forces us to commit changes before
we've tested them, and different commits may end up being tested with different
`benchmarks.nix` files, making comparisons unjustified.

To improve this, we need to load `benchmarks.nix` from our working tree instead.
We can do this by defining a "builder", since they're given the working tree as
their `dir` argument.

First, alter your `builders` attribute to the following:

    { "myDep": "{ dir, version }: dir" }

Now we need to specify a "version" to use; this can be an arbitrary Nix
expression, and since we're ignoring it we might as well use `null`. Alter your
`matrix` to be:

    { "myDep": [ "null" ] }

Now we need to make use of `myDep` in `installer`. Change your `installer` to:

    "args: import (args.myDep + ''/benchmarks.nix'') args"

Now this will read `benchmarks.nix` from the working tree (the `dir` value we
returned from the `myDeps` builder), whilst `pkg` will still use the checked out
copy (which is what we want to benchmark).

If your needs are simple, this is enough to work with: just adjust the contents
of `benchmarks.nix` (and rename/move the file) as appropriate to define an
environment for your project. Alter the benchmarks as you like.

The recommended way to access features of the environment from within a
benchmark is to use environment variables (for data), or augment `PATH` (for
programs). We use `makeWrapper` to do the latter in the above example; the same
can be done for environment variables using the `--set` option.

### Detailed Explanation ###

To benchmark a project, you'll need an `asv.conf.json` file. You can use
`asv quickstart` to generate one, and follow the `asv` documentation for setting
the fields. The settings which are specific to the Nix plugin are as follows:

 - `plugins`: This tells asv what Python modules to import for plugins. You
   should make sure `"asv_nix"` is included. For example:

       "plugins": [ "asv_nix" ]

 - `environment_type`: This tells asv how it should build the environment that
   the benchmarks will be run in. Use the value `"nix"`. For example:

       "environment_type": "nix"

 - `env_dir`: This tells asv where to store environments, so they don't have to
   be rebuilt from scratch. The Nix plugin extends this by including a "garbage
   collector root" at this path *inside each environment*. This prevents Nix
   from deleting the environment when its garbage collector runs. Deleting these
   roots (symlinks) will cause the environments to be garbage collected. You can
   also just delete the entire contents of `env_dir` if you need to.

   For example:

       "env_dir": ".asv/env"

   This will tell `asv` to make a directory `.asv/env` in the benchmarking
   project's directory. Each commit of the project being benchmarked will be put
   in its own directory in here; and *inside each of those* will be *another*
   `.asv/env` directory, containing a garbage collector root for each set of
   dependencies that commit has been benchmarked with.

 - `matrix`: This tells asv which versions of which dependencies to include in
   the environment. It is an object where each property names a dependency, and
   each value is a list of "versions" for that dependency to benchmark. For
   example if the dependencies are `{"a": [ 1, 2, 3 ], "b": [ 7 ]}` then the
   benchmarks will be run with in an environment with `a` version 1 and `b`
   version 7, then again in an environment with `a` version 2 and `b` version 7,
   and again in an environment with `a` version 3 and `b` version 7.

   In normal `asv` projects, these dependencies are Python packages. Since Nix
   is more general, we add a layer of indirection: each dependency name is a key
   for the `builders` object described below. Each "version" is a string
   containing a Nix expression, which will be passed as an argument to that
   "builder".

   For example, here are two ways we might specify a package to use: `python`
   explicitly imports `nixpkgs` and projects out the desired package attribute;
   `shell` just gives a string of the attribute's name (Nix strings can use `"`
   or `''`); `myLib` seems to be specifying git revisions:

       "matrix": {
         "python": ["(import  {}).python2",
                    "(import  {}).python3",
                    "(import  {}).pypy" ],
        "shell":  ["''bash''", "''dash''", "''zsh''" ],
         "myLib":  ["''42db882''", "''2ddf9fe''"]
       }

 - `builders`: This property isn't part of a standard `asv` project. It defines,
   for each dependency, a Nix expression for building it. The keys correspond to
   those used in `matrix`. The values should be strings containing a Nix
   expression, which should be a function taking a set of two arguments:
   `version`, which comes from the `matrix` entry; and `dir`, which is the path
   to the benchmarks' root directory. The latter is useful when we want to write
   large Nix expressions in a separate file and have the configuration import
   them. Note that `dir` is the root of the *benchmarking project*; this isn't
   necessarily the *project being benchmarked*, since asv allows benchmarks to
   be managed separately.

   Continuing the example from `matrix`, we might have these implementations:
   Since `python` passes in a package for each "version", the implementation can
   just return them as-is; `shell` only provides names, so its implementation
   must look them up. `myLib` uses `dir` to import a Nix expression from disk,
   and passes on the given `version`.

       "builders": {
         "python": "args.version,
         "shell":  "args: builtins.getAttr args.version (import  {})",
         "myLib":  "args: import args.dir { inherit (args) version; }"
       }

 - `installer`: This is another Nix-specific property. It is a string containing
   a Nix expression, which will build the whole benchmark environment. It should
   be a function accepting a set containing all of the dependencies (so `shell`,
   `python` and `myLib` in the above example), as well as `root` which is the
   path to the root directory of the project being benchmarked (i.e. this is the
   *opposite* of the `dir` argument for builders). Importantly, the contents of
   `root` will change as different versions (e.g. git commits) of the project
   are benchmarked.

   The result of this function should include a `bin/python` executable, which
   will be used to run the benchmarks. Presumably, you will want other programs,
   environment variables, etc. to be set during the benchmark's execution; the
   standard way to do this is to "wrap" the python executable, e.g. using
   `makeWrapper` from `nixpkgs`, or some similar mechanism.

   For example:

       "installer": "args: with import  {};
                           with {
                             pkg = callPackage args.root { shell = args.shell; };
                           };
                           runCommand ''bench-env''
                             {
                               inherit pkg;
                               inherit (args) python;
                               buildInputs = [ makeWrapper ];
                             }
                             ''
                               mkdir -p $out/bin
                               makeWrapper $python/bin/python $out/bin/python \
                                 --prefix PATH : $pkg/bin
                                 --set    MYLIB $myLib
                             ''"

  More likely, we'll write expressions like these in a file and either have the
  `installer` import them from `root`, or have a `builder` import them from
  `dir`.

To see a working example, take a look at `example.nix`, which is used as a test
during installation.

Repository

Clone this repository using:
  git clone http://chriswarbo.net/git/asv-nix.git 

Branches


Generated by git2html.