general-tests

Last updated: 2018-06-10 11:29:44 +0100

Upstream URL: git clone http://chriswarbo.net/git/general-tests.git

Repo

View repository

View issue tracker

Contents of README follows


Simple test suite

I use this as a high-level test harness, testing as many of my projects as possible; like a poor man's continuous integration server.

Architecture

Tests are defined in the <code>*.nix</code> files in <code>tests/</code> (except for <code>tests/default.nix</code>). Each test is a Nix derivation which produces a script: running that script runs the test. Each file in <code>tests/</code> can either define a single test or an attrset of tests (nested to arbitrary depth).

For example, <code>tests/hlint.nix</code> defines an attrset, where each value is a script for running hlint on a different Haskell project.

Running

The top-level <code>default.nix</code> file defines a script which runs all of these tests, counts passes/fails and writes output to <code>/tmp</code> suitable for embedding in <code>xmobar</code>. To build this wrapper script, just run <code>nix-build</code> in the top-level directory: the path to the resulting script will be printed on stdout, and a symlink called <code>result</code> will also point to it. Running that script will invoke the whole test suite.

The advantage to this two step build-then-run approach is that we can have the build step perform some time-consuming tasks, like using <code>find</code> to look for shell scripts, git repos, etc. and generate test scripts based on the results. Once they're built, we can run those scripts again and again without having to perform the slow tasks again. When we want to "flush the cache" we just run <code>nix-build</code> to generate new scripts.

For convenience, the included <code>run</code> script will build and run the test suite.

Debugging

The top-level script prints out the path to each test script as it goes. To investigate why a particular test is failing, just run it individually to see the stdout, stderr and exit code.

There are two ways to find an individual test without running the whole suite. One way is to look at the env vars defined in the top-level script, which include the directory storing the generated test scripts.

Another way is to look in the Nix derivations, for example using the <code>nix-repl</code> command. The top-level <code>default.nix</code> file takes a <code>packageOnly</code> parameter, defaulting to <code>true</code>: when this is <code>true</code> only the top-level script is produced; when <code>false</code> it returns an attrset containing the top-level script, all of the individual test derivations and the utilities from <code>helpers/</code>.

Extras

The <code>release.nix</code> file allows tests to be run on the Hydra continuous integration server. This is nice to have, but some tests only make sense on my development machine; for example <code>tests/all-committed.nix</code> checks that my git repositories don't contain any uncommitted things. This wouldn't make sense on a continuous integration server, since it will <em>only</em> see committed things.