asv-nix

Last updated: 2017-06-23 04:27:09 +0100

Upstream URL: git clone http://chriswarbo.net/git/asv-nix.git

Repo

View repository

View issue tracker

Contents of follows


Nix Plugin for Airspeed Velocity

About Airspeed Velocity

Airspeed Velocity (asv) is a benchmarking tool for Python. Its features include:

All of this is very nice, but this last feature is rather tricky. There is a plugin mechanism for defining environments, and plugins are provided for virtualenv, conda and a "dummy" plugin which uses the system's installed packages.

This is a bit unfortunate for two reasons:

If we solve the first problem, we've automatically solved the second. That's what this project tries to do.

About Nix

Nix is a build/packaging system which is very powerful and modular. Packages can be defined using a Turing-complete, pure functional, lazily evaluated, domain specific programming language. Every package is built in isolation, the results are cached along with the hashes of all inputs, and only rebuilt if some input changes.

Rather than defining packages directly, it's common to write a bunch of helper functions and define a package in terms of those. The 'nixpkgs' repository contains a large collection of packages and helper functions, for example buildPythonPackage which takes care of Python-specific tasks like setting module directories, or haskellPackages.mkDerivation which handles tasks specific to Haskell projects. This flexibility makes Nix usable for almost any project in any programming language.

About This Plugin

This project provides an asv plugin, which uses Nix to manage dependencies instead of conda or virtualenv. The architecture of asv is understandably a little Python-centric, so we take a few liberties in our configuration to make using Nix more seamless.

Using The Plugin

To run a benchmark, you'll need a Python package with both asv and this plugin installed as modules. You could do this using pip, apt-get, etc. but we might as well use Nix: the default.nix file defines a Nix package for this plugin. It depends on an asv package, but at the time of writing nixpkgs doesn't include one. The shell.nix file has instructions to fetch one, so you should be able to just run nix-shell and be dropped into an environment with both asv and this plugin available.

Most of the asv instructions apply as normal, so we won't cover them here. Instead, we'll highlight the aspects which are unique to the Nix plugin.

Quick Start

asv allows benchmarks to be managed separately to your project. That complicates things a little, so let's assume that your benchmarks live alongside your code. In your project repo's top-level directory, run asv quickstart to generate some example benchmarks and a config file.

In your asv.conf.json, make sure plugins is set to [ "asv_nix" ] and environment_type is "nix". Also set builders to {} and matrix to {}.

The easiest way to specify your benchmarking environment is to define it using a Nix file. Let's make a file called benchmarks.nix in your project's root directory. For the sake of argument, we'll assume your project has a default.nix which defines a package containing some binaries in bin/. The following benchmarks.nix file will provide a python executable, which has access to that bin/ directory in its PATH:

args:
with import <lt;nixpkgs>gt; {};
with { pkg = callPackage args.root {}; };
runCommand "env" { buildInputs = [ makeWrapper ]; } ''
  mkdir -p "$out/bin"
  makeWrapper "${python}/bin/python" "$out/bin/python" \
    --prefix PATH : "${pkg}/bin"
''

Now we'll tell asv to use this as our environment. In our asv.conf.json we set installer to the following:

"args: import (args.root + ''/benchmarks.nix'') args"

This should be everything we need to run the benchmarks, using asv run. However, there is an annoying problem: we're looking for benchmarks.nix in root, which is the version of the project that's checked out of version control. Hence we need to commit benchmarks.nix to the repo before it will be available; this isn't a good idea, since it forces us to commit changes before we've tested them, and different commits may end up being tested with different benchmarks.nix files, making comparisons unjustified.

To improve this, we need to load benchmarks.nix from our working tree instead. We can do this by defining a "builder", since they're given the working tree as their dir argument.

First, alter your builders attribute to the following:

{ "myDep": "{ dir, version }: dir" }

Now we need to specify a "version" to use; this can be an arbitrary Nix expression, and since we're ignoring it we might as well use null. Alter your matrix to be:

{ "myDep": [ "null" ] }

Now we need to make use of myDep in installer. Change your installer to:

"args: import (args.myDep + ''/benchmarks.nix'') args"

Now this will read benchmarks.nix from the working tree (the dir value we returned from the myDeps builder), whilst pkg will still use the checked out copy (which is what we want to benchmark).

If your needs are simple, this is enough to work with: just adjust the contents of benchmarks.nix (and rename/move the file) as appropriate to define an environment for your project. Alter the benchmarks as you like.

The recommended way to access features of the environment from within a benchmark is to use environment variables (for data), or augment PATH (for programs). We use makeWrapper to do the latter in the above example; the same can be done for environment variables using the --set option.

Detailed Explanation

To benchmark a project, you'll need an asv.conf.json file. You can use asv quickstart to generate one, and follow the asv documentation for setting the fields. The settings which are specific to the Nix plugin are as follows:

For example:

   "env_dir": ".asv/env"

This will tell asv to make a directory .asv/env in the benchmarking project's directory. Each commit of the project being benchmarked will be put in its own directory in here; and inside each of those will be another .asv/env directory, containing a garbage collector root for each set of dependencies that commit has been benchmarked with.

In normal asv projects, these dependencies are Python packages. Since Nix is more general, we add a layer of indirection: each dependency name is a key for the builders object described below. Each "version" is a string containing a Nix expression, which will be passed as an argument to that "builder".

For example, here are two ways we might specify a package to use: python explicitly imports nixpkgs and projects out the desired package attribute; shell just gives a string of the attribute's name (Nix strings can use " or ''); myLib seems to be specifying git revisions:

   "matrix": {
     "python": ["(import <lt;nixpkgs>gt; {}).python2",
                "(import <lt;nixpkgs>gt; {}).python3",
                "(import <lt;nixpkgs>gt; {}).pypy" ],
    "shell":  ["''bash''", "''dash''", "''zsh''" ],
     "myLib":  ["''42db882''", "''2ddf9fe''"]
   }

Continuing the example from matrix, we might have these implementations: Since python passes in a package for each "version", the implementation can just return them as-is; shell only provides names, so its implementation must look them up. myLib uses dir to import a Nix expression from disk, and passes on the given version.

   "builders": {
     "python": "args.version,
     "shell":  "args: builtins.getAttr args.version (import <lt;nixpkgs>gt; {})",
     "myLib":  "args: import args.dir { inherit (args) version; }"
   }

The result of this function should include a bin/python executable, which will be used to run the benchmarks. Presumably, you will want other programs, environment variables, etc. to be set during the benchmark's execution; the standard way to do this is to "wrap" the python executable, e.g. using makeWrapper from nixpkgs, or some similar mechanism.

For example:

   "installer": "args: with import <lt;nixpkgs>gt; {};
                       with {
                         pkg = callPackage args.root { shell = args.shell; };
                       };
                       runCommand ''bench-env''
                         {
                           inherit pkg;
                           inherit (args) python;
                           buildInputs = [ makeWrapper ];
                         }
                         ''
                           mkdir -p $out/bin
                           makeWrapper $python/bin/python $out/bin/python \
                             --prefix PATH : $pkg/bin
                             --set    MYLIB $myLib
                         ''"

More likely, we'll write expressions like these in a file and either have the installer import them from root, or have a builder import them from dir.

To see a working example, take a look at example.nix, which is used as a test during installation.