Last updated: 2019-06-10 16:29:49 +0100
Upstream URL: git clone http://chriswarbo.net/git/benchmark-runner.git
Contents of README follows
The Nix derivation in <code>default.nix</code> allows a Git repository to be benchmarked using Airspeed Velocity (ASV). This is useful for continuous integration, since normally we will only get a single Git revision (to aid reproducibility, Nix deletes the <code>.git</code> directory by default), whilst ASV wants access to all of the commits in order to benchmark them.
The <code>default.nix</code> file defines a function. This should be called with an attrset of arguments, including a <code>repo</code> attribute containing the URL (remote or local) of the desired Git repository. This can be done “manually”, or if you’re on Hydra by setting it as a build input.
Other arguments are optional, and include:
<ul> <li><code>commitCount</code> is the number of commits (ancestors of <code>HEAD</code>, inclusive) to run benchmarks for. If <code>null</code> is given, the range <code>NEW</code> will be used, which causes ASV to look for the latest benchmark result (if any) and benchmark all commits since then. Note that <code>null</code> may require a crazy amount of work, if there are thousands of commits. For this reason we default to <code>10</code>.</li> <li><code>cacheDir</code> defaults to <code>null</code>. If a string is given, we will treat it as a path and copy existing results from there into our ASV results directory, so that they don’t get re-run. Any new benchmark results get copied into the <code>cacheDir</code> after we’re finished.</li> </ul>Using a cache can be a huge performance gain, especially when benchmarking the same repos over and over (e.g. a build server watching for new commits). However it also makes our derivations impure. It’s up to you whether that’s worth it.
Many repo’s results can co-exist in the same cache, since we identify them by hashing the relevant ASV config file. This means that changing the config file will cause old commits to be benchmarked from scratch. Whilst this may be inconvenient in some circumstances (e.g. if the benchmarks haven’t changed, and there are thousands of existing results), we consider this behaviour to be reasonable since it is <em>simple</em> and <em>predictable</em>. If you want to re-use results in a “smarter” way (AKA more complicated/fragile/confusing/etc.), feel free to write your own scripts to copy the cache contents around.
Note that you should probably avoid giving a Nix <em>path</em> value (e.g. <code>/tmp/cache</code>) as a <code>cacheDir</code> argument, and should almost always use a Nix <em>string</em> value instead (e.g. <code>"/tmp/cache"</code>). This is because Nix copies path values to its store, and passes those immutable copies to the build scripts. This would prevent new results getting copied to the cache.