asv-nix

Last updated: 2022-01-16 19:47:49 +0000

Upstream URL: git clone http://chriswarbo.net/git/asv-nix.git

Repo

View repository

View issue tracker

Contents of README follows

Nix Plugin for Airspeed Velocity

About Airspeed Velocity

Airspeed Velocity (asv) is a benchmarking tool for Python. Its features include:

Measuring CPU time, wallclock time, peak memory usage, or any numeric value.
Integration with version control, e.g. to benchmark a range of git commits.
Measuring with different dependencies (e.g. alternative libraries).
Measuring on different machines.
Generating HTML reports, with graphs, step-detection, etc.
Running benchmarks in isolated environments.

All of this is very nice, but this last feature is rather tricky. There is a plugin mechanism for defining environments, and plugins are provided for virtualenv, conda and a “dummy” plugin which uses the system’s installed packages.

This is a bit unfortunate for two reasons:

These plugins can only manage dependencies which are Python packages. If a project requires non-Python dependencies, they’ll have to be hacked in using setup/teardown functions or similar.
This limitation on dependencies prevents asv from being used (to its full extent, at least) on non-Python projects. Even if we write our benchmarks in Python, e.g. using subprocess to call out to our “real” project, that just turns our “real” project into a dependency of the benchmarks (as the previous point states) such dependencies can’t be managed unless they’re Python packages.

If we solve the first problem, we’ve automatically solved the second. That’s what this project tries to do.

About Nix

Nix is a build/packaging system which is very powerful and modular. Packages can be defined using a Turing-complete, pure functional, lazily evaluated, domain specific programming language. Every package is built in isolation, the results are cached along with the hashes of all inputs, and only rebuilt if some input changes.

Rather than defining packages directly, it’s common to write a bunch of helper functions and define a package in terms of those. The ‘nixpkgs’ repository contains a large collection of packages and helper functions, for example buildPythonPackage which takes care of Python-specific tasks like setting module directories, or haskellPackages.mkDerivation which handles tasks specific to Haskell projects. This flexibility makes Nix usable for almost any project in any programming language.

About This Plugin

This project provides an asv plugin, which uses Nix to manage dependencies instead of conda or virtualenv. The architecture of asv is understandably a little Python-centric, so we take a few liberties in our configuration to make using Nix more seamless.

Using The Plugin

To run a benchmark, you’ll need a Python package with both asv and this plugin installed as modules. You could do this using pip, apt-get, etc. but we might as well use Nix: the default.nix file defines a Nix package for this plugin. It depends on an asv package, but at the time of writing nixpkgs doesn’t include one. The shell.nix file has instructions to fetch one, so you should be able to just run nix-shell and be dropped into an environment with both asv and this plugin available.

Most of the asv instructions apply as normal, so we won’t cover them here. Instead, we’ll highlight the aspects which are unique to the Nix plugin.

Quick Start

asv allows benchmarks to be managed separately to your project. That complicates things a little, so let’s assume that your benchmarks live alongside your code. In your project repo’s top-level directory, run asv quickstart to generate some example benchmarks and a config file.

In your asv.conf.json, make sure plugins is set to [ "asv_nix" ] and environment_type is "nix". Also set builders to {} and matrix to {}.

The easiest way to specify your benchmarking environment is to define it using a Nix file. Let’s make a file called benchmarks.nix in your project’s root directory. For the sake of argument, we’ll assume your project has a default.nix which defines a package containing some binaries in bin/. The following benchmarks.nix file will provide a python executable, which has access to that bin/ directory in its PATH:

args:
with import <nixpkgs> {};
with { pkg = callPackage args.root {}; };
runCommand "env" { buildInputs = [ makeWrapper ]; } ''
  mkdir -p "$out/bin"
  makeWrapper "${python}/bin/python" "$out/bin/python" \
    --prefix PATH : "${pkg}/bin"
''

Now we’ll tell asv to use this as our environment. In our asv.conf.json we set installer to the following:

"args: import (args.root + ''/benchmarks.nix'') args"

This should be everything we need to run the benchmarks, using asv run. However, there is an annoying problem: we’re looking for benchmarks.nix in root, which is the version of the project that’s checked out of version control. Hence we need to commit benchmarks.nix to the repo before it will be available; this isn’t a good idea, since it forces us to commit changes before we’ve tested them, and different commits may end up being tested with different benchmarks.nix files, making comparisons unjustified.

To improve this, we need to load benchmarks.nix from our working tree instead. We can do this by defining a “builder”, since they’re given the working tree as their dir argument.

First, alter your builders attribute to the following:

{ "myDep": "{ dir, version }: dir" }

Now we need to specify a “version” to use; this can be an arbitrary Nix expression, and since we’re ignoring it we might as well use null. Alter your matrix to be:

{ "myDep": [ "null" ] }

Now we need to make use of myDep in installer. Change your installer to:

"args: import (args.myDep + ''/benchmarks.nix'') args"

Now this will read benchmarks.nix from the working tree (the dir value we returned from the myDeps builder), whilst pkg will still use the checked out copy (which is what we want to benchmark).

If your needs are simple, this is enough to work with: just adjust the contents of benchmarks.nix (and rename/move the file) as appropriate to define an environment for your project. Alter the benchmarks as you like.

The recommended way to access features of the environment from within a benchmark is to use environment variables (for data), or augment PATH (for programs). We use makeWrapper to do the latter in the above example; the same can be done for environment variables using the --set option.

Detailed Explanation

To benchmark a project, you’ll need an asv.conf.json file. You can use asv quickstart to generate one, and follow the asv documentation for setting the fields. The settings which are specific to the Nix plugin are as follows:

plugins: This tells asv what Python modules to import for plugins. You should make sure "asv_nix" is included. For example:
```
"plugins": [ "asv_nix" ]
```
environment_type: This tells asv how it should build the environment that the benchmarks will be run in. Use the value "nix". For example:
```
"environment_type": "nix"
```
env_dir: This tells asv where to store environments, so they don’t have to be rebuilt from scratch. The Nix plugin extends this by including a “garbage collector root” at this path inside each environment. This prevents Nix from deleting the environment when its garbage collector runs. Deleting these roots (symlinks) will cause the environments to be garbage collected. You can also just delete the entire contents of env_dir if you need to.

For example:
```
"env_dir": ".asv/env"
```
This will tell asv to make a directory .asv/env in the benchmarking project’s directory. Each commit of the project being benchmarked will be put in its own directory in here; and inside each of those will be another .asv/env directory, containing a garbage collector root for each set of dependencies that commit has been benchmarked with.
matrix: This tells asv which versions of which dependencies to include in the environment. It is an object where each property names a dependency, and each value is a list of “versions” for that dependency to benchmark. For example if the dependencies are {"a": [ 1, 2, 3 ], "b": [ 7 ]} then the benchmarks will be run with in an environment with a version 1 and b version 7, then again in an environment with a version 2 and b version 7, and again in an environment with a version 3 and b version 7.

In normal asv projects, these dependencies are Python packages. Since Nix is more general, we add a layer of indirection: each dependency name is a key for the builders object described below. Each “version” is a string containing a Nix expression, which will be passed as an argument to that “builder”.

For example, here are two ways we might specify a package to use: python explicitly imports nixpkgs and projects out the desired package attribute; shell just gives a string of the attribute’s name (Nix strings can use " or ''); myLib seems to be specifying git revisions:
```
"matrix": {
  "python": ["(import <nixpkgs> {}).python2",
             "(import <nixpkgs> {}).python3",
             "(import <nixpkgs> {}).pypy" ],
 "shell":  ["''bash''", "''dash''", "''zsh''" ],
  "myLib":  ["''42db882''", "''2ddf9fe''"]
}
```
builders: This property isn’t part of a standard asv project. It defines, for each dependency, a Nix expression for building it. The keys correspond to those used in matrix. The values should be strings containing a Nix expression, which should be a function taking a set of two arguments: version, which comes from the matrix entry; and dir, which is the path to the benchmarks’ root directory. The latter is useful when we want to write large Nix expressions in a separate file and have the configuration import them. Note that dir is the root of the benchmarking project; this isn’t necessarily the project being benchmarked, since asv allows benchmarks to be managed separately.

Continuing the example from matrix, we might have these implementations: Since python passes in a package for each “version”, the implementation can just return them as-is; shell only provides names, so its implementation must look them up. myLib uses dir to import a Nix expression from disk, and passes on the given version.
```
"builders": {
  "python": "args.version,
  "shell":  "args: builtins.getAttr args.version (import <nixpkgs> {})",
  "myLib":  "args: import args.dir { inherit (args) version; }"
}
```
installer: This is another Nix-specific property. It is a string containing a Nix expression, which will build the whole benchmark environment. It should be a function accepting a set containing all of the dependencies (so shell, python and myLib in the above example), as well as root which is the path to the root directory of the project being benchmarked (i.e. this is the opposite of the dir argument for builders). Importantly, the contents of root will change as different versions (e.g. git commits) of the project are benchmarked.

The result of this function should include a bin/python executable, which will be used to run the benchmarks. Presumably, you will want other programs, environment variables, etc. to be set during the benchmark’s execution; the standard way to do this is to “wrap” the python executable, e.g. using makeWrapper from nixpkgs, or some similar mechanism.

For example:
```
"installer": "args: with import <nixpkgs> {};
                    with {
                      pkg = callPackage args.root { shell = args.shell; };
                    };
                    runCommand ''bench-env''
                      {
                        inherit pkg;
                        inherit (args) python;
                        buildInputs = [ makeWrapper ];
                      }
                      ''
                        mkdir -p $out/bin
                        makeWrapper $python/bin/python $out/bin/python \
                          --prefix PATH : $pkg/bin
                          --set    MYLIB $myLib
                      ''"
```

More likely, we’ll write expressions like these in a file and either have the installer import them from root, or have a builder import them from dir.

To see a working example, take a look at example.nix, which is used as a test during installation.