asv-nix

Last updated: 2022-01-16 19:47:49 +0000

Upstream URL: git clone http://chriswarbo.net/git/asv-nix.git

Repo

View repository

View issue tracker

Contents of README follows

Nix Plugin for Airspeed Velocity

About Airspeed Velocity

Airspeed Velocity (asv) is a benchmarking tool for Python. Its features include:

All of this is very nice, but this last feature is rather tricky. There is a plugin mechanism for defining environments, and plugins are provided for virtualenv, conda and a “dummy” plugin which uses the system’s installed packages.

This is a bit unfortunate for two reasons:

If we solve the first problem, we’ve automatically solved the second. That’s what this project tries to do.

About Nix

Nix is a build/packaging system which is very powerful and modular. Packages can be defined using a Turing-complete, pure functional, lazily evaluated, domain specific programming language. Every package is built in isolation, the results are cached along with the hashes of all inputs, and only rebuilt if some input changes.

Rather than defining packages directly, it’s common to write a bunch of helper functions and define a package in terms of those. The ‘nixpkgs’ repository contains a large collection of packages and helper functions, for example <code>buildPythonPackage</code> which takes care of Python-specific tasks like setting module directories, or <code>haskellPackages.mkDerivation</code> which handles tasks specific to Haskell projects. This flexibility makes Nix usable for almost any project in any programming language.

About This Plugin

This project provides an asv plugin, which uses Nix to manage dependencies instead of conda or virtualenv. The architecture of asv is understandably a little Python-centric, so we take a few liberties in our configuration to make using Nix more seamless.

Using The Plugin

To run a benchmark, you’ll need a Python package with both <code>asv</code> and this plugin installed as modules. You could do this using <code>pip</code>, <code>apt-get</code>, etc. but we might as well use Nix: the <code>default.nix</code> file defines a Nix package for this plugin. It depends on an <code>asv</code> package, but at the time of writing <code>nixpkgs</code> doesn’t include one. The <code>shell.nix</code> file has instructions to fetch one, so you should be able to just run <code>nix-shell</code> and be dropped into an environment with both <code>asv</code> and this plugin available.

Most of the asv instructions apply as normal, so we won’t cover them here. Instead, we’ll highlight the aspects which are unique to the Nix plugin.

Quick Start

<code>asv</code> allows benchmarks to be managed separately to your project. That complicates things a little, so let’s assume that your benchmarks live alongside your code. In your project repo’s top-level directory, run <code>asv quickstart</code> to generate some example benchmarks and a config file.

In your <code>asv.conf.json</code>, make sure <code>plugins</code> is set to <code>[ "asv_nix" ]</code> and <code>environment_type</code> is <code>"nix"</code>. Also set <code>builders</code> to <code>{}</code> and <code>matrix</code> to <code>{}</code>.

The easiest way to specify your benchmarking environment is to define it using a Nix file. Let’s make a file called <code>benchmarks.nix</code> in your project’s root directory. For the sake of argument, we’ll assume your project has a <code>default.nix</code> which defines a package containing some binaries in <code>bin/</code>. The following <code>benchmarks.nix</code> file will provide a <code>python</code> executable, which has access to that <code>bin/</code> directory in its <code>PATH</code>:

Now we’ll tell <code>asv</code> to use this as our environment. In our <code>asv.conf.json</code> we set <code>installer</code> to the following:

This should be everything we need to run the benchmarks, using <code>asv run</code>. However, there is an annoying problem: we’re looking for <code>benchmarks.nix</code> in <code>root</code>, which is the version of the project that’s checked out of version control. Hence we need to commit <code>benchmarks.nix</code> to the repo before it will be available; this isn’t a good idea, since it forces us to commit changes before we’ve tested them, and different commits may end up being tested with different <code>benchmarks.nix</code> files, making comparisons unjustified.

To improve this, we need to load <code>benchmarks.nix</code> from our working tree instead. We can do this by defining a “builder”, since they’re given the working tree as their <code>dir</code> argument.

First, alter your <code>builders</code> attribute to the following:

Now we need to specify a “version” to use; this can be an arbitrary Nix expression, and since we’re ignoring it we might as well use <code>null</code>. Alter your <code>matrix</code> to be:

Now we need to make use of <code>myDep</code> in <code>installer</code>. Change your <code>installer</code> to:

Now this will read <code>benchmarks.nix</code> from the working tree (the <code>dir</code> value we returned from the <code>myDeps</code> builder), whilst <code>pkg</code> will still use the checked out copy (which is what we want to benchmark).

If your needs are simple, this is enough to work with: just adjust the contents of <code>benchmarks.nix</code> (and rename/move the file) as appropriate to define an environment for your project. Alter the benchmarks as you like.

The recommended way to access features of the environment from within a benchmark is to use environment variables (for data), or augment <code>PATH</code> (for programs). We use <code>makeWrapper</code> to do the latter in the above example; the same can be done for environment variables using the <code>--set</code> option.

Detailed Explanation

To benchmark a project, you’ll need an <code>asv.conf.json</code> file. You can use <code>asv quickstart</code> to generate one, and follow the <code>asv</code> documentation for setting the fields. The settings which are specific to the Nix plugin are as follows:

<code>plugins</code>: This tells asv what Python modules to import for plugins. You should make sure <code>"asv_nix"</code> is included. For example:

<code>environment_type</code>: This tells asv how it should build the environment that the benchmarks will be run in. Use the value <code>"nix"</code>. For example:

<code>env_dir</code>: This tells asv where to store environments, so they don’t have to be rebuilt from scratch. The Nix plugin extends this by including a “garbage collector root” at this path <em>inside each environment</em>. This prevents Nix from deleting the environment when its garbage collector runs. Deleting these roots (symlinks) will cause the environments to be garbage collected. You can also just delete the entire contents of <code>env_dir</code> if you need to.

For example:

This will tell <code>asv</code> to make a directory <code>.asv/env</code> in the benchmarking project’s directory. Each commit of the project being benchmarked will be put in its own directory in here; and <em>inside each of those</em> will be <em>another</em> <code>.asv/env</code> directory, containing a garbage collector root for each set of dependencies that commit has been benchmarked with.

<code>matrix</code>: This tells asv which versions of which dependencies to include in the environment. It is an object where each property names a dependency, and each value is a list of “versions” for that dependency to benchmark. For example if the dependencies are <code>{"a": [ 1, 2, 3 ], "b": [ 7 ]}</code> then the benchmarks will be run with in an environment with <code>a</code> version 1 and <code>b</code> version 7, then again in an environment with <code>a</code> version 2 and <code>b</code> version 7, and again in an environment with <code>a</code> version 3 and <code>b</code> version 7.

In normal <code>asv</code> projects, these dependencies are Python packages. Since Nix is more general, we add a layer of indirection: each dependency name is a key for the <code>builders</code> object described below. Each “version” is a string containing a Nix expression, which will be passed as an argument to that “builder”.

For example, here are two ways we might specify a package to use: <code>python</code> explicitly imports <code>nixpkgs</code> and projects out the desired package attribute; <code>shell</code> just gives a string of the attribute’s name (Nix strings can use <code>"</code> or <code>''</code>); <code>myLib</code> seems to be specifying git revisions:

<code>builders</code>: This property isn’t part of a standard <code>asv</code> project. It defines, for each dependency, a Nix expression for building it. The keys correspond to those used in <code>matrix</code>. The values should be strings containing a Nix expression, which should be a function taking a set of two arguments: <code>version</code>, which comes from the <code>matrix</code> entry; and <code>dir</code>, which is the path to the benchmarks’ root directory. The latter is useful when we want to write large Nix expressions in a separate file and have the configuration import them. Note that <code>dir</code> is the root of the <em>benchmarking project</em>; this isn’t necessarily the <em>project being benchmarked</em>, since asv allows benchmarks to be managed separately.

Continuing the example from <code>matrix</code>, we might have these implementations: Since <code>python</code> passes in a package for each “version”, the implementation can just return them as-is; <code>shell</code> only provides names, so its implementation must look them up. <code>myLib</code> uses <code>dir</code> to import a Nix expression from disk, and passes on the given <code>version</code>.

<code>installer</code>: This is another Nix-specific property. It is a string containing a Nix expression, which will build the whole benchmark environment. It should be a function accepting a set containing all of the dependencies (so <code>shell</code>, <code>python</code> and <code>myLib</code> in the above example), as well as <code>root</code> which is the path to the root directory of the project being benchmarked (i.e. this is the <em>opposite</em> of the <code>dir</code> argument for builders). Importantly, the contents of <code>root</code> will change as different versions (e.g. git commits) of the project are benchmarked.

The result of this function should include a <code>bin/python</code> executable, which will be used to run the benchmarks. Presumably, you will want other programs, environment variables, etc. to be set during the benchmark’s execution; the standard way to do this is to “wrap” the python executable, e.g. using <code>makeWrapper</code> from <code>nixpkgs</code>, or some similar mechanism.

For example:

More likely, we’ll write expressions like these in a file and either have the <code>installer</code> import them from <code>root</code>, or have a <code>builder</code> import them from <code>dir</code>.

To see a working example, take a look at <code>example.nix</code>, which is used as a test during installation.