# Simple test suite #

I use this as a high-level test harness, testing as many of my projects as
possible; like a poor man's continuous integration server.

## Architecture ##

Tests are defined in the `*.nix` files in `tests/` (except for
`tests/default.nix`). Each test is a Nix derivation which produces a script:
running that script runs the test. Each file in `tests/` can either define a
single test or an attrset of tests (nested to arbitrary depth).

For example, `tests/hlint.nix` defines an attrset, where each value is a script
for running hlint on a different Haskell project.

## Running ##

The top-level `default.nix` file defines a script which runs all of these tests
and counts passes/fails. Each test script is wrapped in a script which writes
either `PASS` or `FAIL` to an appropriately named file in `/tmp`, so we can find
out which tests have failed, etc. The main runner will delete any of these files
which don't correspond to a known test.

To build the main runner, and hence all of the tests scripts and their wrappers,
just run `nix-build` in the top-level directory: the path to the runner script
will be printed on stdout, and a symlink called `result` will also point to it.
Running that script will invoke the whole test suite. If you want to run an
individual test (via its wrapper), look for the `tests` variable in the main
runner script: that's where all of the individual tests can be found.

The advantage to this two step build-then-run approach is that we can have the
build step perform some time-consuming tasks, like using `find` to look for
shell scripts, git repos, etc. and generate test scripts based on what's found.
Once they're built, we can run those scripts again and again without having to
perform the slow tasks again. When we want to "flush the cache" we just run
`nix-build` to generate new scripts.

For convenience, the included `run` script will build and run the test suite.

## Debugging ##

The top-level script prints out the path to each test script as it goes. To
investigate why a particular test is failing, just run it individually to see
the stdout, stderr and exit code.

There are two ways to find an individual test without running the whole suite.
One way is to look at the env vars defined in the top-level script, which
include the directory storing the generated test scripts.

Another way is to look in the Nix derivations, for example using the `nix-repl`
command. The top-level `default.nix` file takes a `packageOnly` parameter,
defaulting to `true`: when this is `true` only the top-level script is produced;
when `false` it returns an attrset containing the top-level script, all of the
individual test derivations and the utilities from `helpers/`.

## Extras ##

The `release.nix` file allows tests to be run on the Hydra continuous
integration server. This is nice to have, but some tests only make sense on my
development machine; for example `tests/all-committed.nix` checks that my git
repositories don't contain any uncommitted things. This wouldn't make sense on a
continuous integration server, since it will *only* see committed things.