Last updated: 2019-01-22 19:29:23 +0000
Upstream URL: git clone http://chriswarbo.net/git/general-tests.git
Contents of README follows
I use this as a high-level test harness, testing as many of my projects as possible; like a poor man’s continuous integration server.
Tests are defined in the *.nix files in
tests/ (except for tests/default.nix). Each
test is a Nix derivation which produces a script: running that script
runs the test. Each file in tests/ can either define a
single test or an attrset of tests (nested to arbitrary depth).
For example, tests/hlint.nix defines an attrset, where
each value is a script for running hlint on a different Haskell
project.
The top-level default.nix file defines a script which
runs all of these tests and counts passes/fails. Each test script is
wrapped in a script which writes either PASS or
FAIL to an appropriately named file in /tmp,
so we can find out which tests have failed, etc. The main runner will
delete any of these files which don’t correspond to a known test.
To build the main runner, and hence all of the tests scripts and
their wrappers, just run nix-build in the top-level
directory: the path to the runner script will be printed on stdout, and
a symlink called result will also point to it. Running that
script will invoke the whole test suite. If you want to run an
individual test (via its wrapper), look for the tests
variable in the main runner script: that’s where all of the individual
tests can be found.
The advantage to this two step build-then-run approach is that we can
have the build step perform some time-consuming tasks, like using
find to look for shell scripts, git repos, etc. and
generate test scripts based on what’s found. Once they’re built, we can
run those scripts again and again without having to perform the slow
tasks again. When we want to “flush the cache” we just run
nix-build to generate new scripts.
For convenience, the included run script will build and
run the test suite.
The top-level script prints out the path to each test script as it goes. To investigate why a particular test is failing, just run it individually to see the stdout, stderr and exit code.
There are two ways to find an individual test without running the whole suite. One way is to look at the env vars defined in the top-level script, which include the directory storing the generated test scripts.
Another way is to look in the Nix derivations, for example using the
nix-repl command. The top-level default.nix
file takes a packageOnly parameter, defaulting to
true: when this is true only the top-level
script is produced; when false it returns an attrset
containing the top-level script, all of the individual test derivations
and the utilities from helpers/.
The release.nix file allows tests to be run on the Hydra
continuous integration server. This is nice to have, but some tests only
make sense on my development machine; for example
tests/all-committed.nix checks that my git repositories
don’t contain any uncommitted things. This wouldn’t make sense on a
continuous integration server, since it will only see committed
things.