Last updated: 2019-06-10 16:29:49 +0100
Upstream URL: git clone http://chriswarbo.net/git/benchmark-runner.git
Contents of README follows
The Nix derivation in default.nix allows a Git
repository to be benchmarked using Airspeed Velocity (ASV). This is
useful for continuous integration, since normally we will only get a
single Git revision (to aid reproducibility, Nix deletes the
.git directory by default), whilst ASV wants access to all
of the commits in order to benchmark them.
The default.nix file defines a function. This should be
called with an attrset of arguments, including a repo
attribute containing the URL (remote or local) of the desired Git
repository. This can be done “manually”, or if you’re on Hydra by
setting it as a build input.
Other arguments are optional, and include:
commitCount is the number of commits (ancestors of
HEAD, inclusive) to run benchmarks for. If
null is given, the range NEW will be used,
which causes ASV to look for the latest benchmark result (if any) and
benchmark all commits since then. Note that null may
require a crazy amount of work, if there are thousands of commits. For
this reason we default to 10.cacheDir defaults to null. If a string is
given, we will treat it as a path and copy existing results from there
into our ASV results directory, so that they don’t get re-run. Any new
benchmark results get copied into the cacheDir after we’re
finished.Using a cache can be a huge performance gain, especially when benchmarking the same repos over and over (e.g. a build server watching for new commits). However it also makes our derivations impure. It’s up to you whether that’s worth it.
Many repo’s results can co-exist in the same cache, since we identify them by hashing the relevant ASV config file. This means that changing the config file will cause old commits to be benchmarked from scratch. Whilst this may be inconvenient in some circumstances (e.g. if the benchmarks haven’t changed, and there are thousands of existing results), we consider this behaviour to be reasonable since it is simple and predictable. If you want to re-use results in a “smarter” way (AKA more complicated/fragile/confusing/etc.), feel free to write your own scripts to copy the cache contents around.
Note that you should probably avoid giving a Nix path value
(e.g. /tmp/cache) as a cacheDir argument, and
should almost always use a Nix string value instead
(e.g. "/tmp/cache"). This is because Nix copies path values
to its store, and passes those immutable copies to the build scripts.
This would prevent new results getting copied to the cache.