Work log 2016-07-20

Desktop was frozen; pressing power button caused shutdown and turning back on brought Hydra back up (with some scripts, etc.)

haskell-te job is failing on Hydra; maybe we should treat the equation-generation as a build output, giving us 3 ‘phases’:

• Sanity checks
• Exploration
• Validity checks

Actually, it’s even easier! We just make the outputs a derivation, rather than a ‘pure’ Nix expression.

Each test is a trivial derivation, with exit code 1 from its build script on error, and 0 otherwise.

Our ‘sanity checks’ can actually be implemented as assertions in the expressions which build the derivations; i.e. fail immediately if we’re not even calling stuff in the right way (poor man’s type system).

Use buildEnv to combine results, instead of && or all; put all tests in the paths attribute.

This change effects quite a lot, but shouldn’t need too much refactoring, as it’s only the return value of testMsg that’s different. Callers shouldn’t care, but users should (e.g. those which combine results).

To prevent running benchmarks unnecessarily, they should be turned into derivations, and use buildInputs to depend on each other. How about:

...benchmark/result.json
            /bin/getBenchmarkResult

Where getBenchmarkResult spits out the contents of result.json’s stdout? Hmm, get is redundant, and we might as well just print the directory name rather than assuming what is wanted.

Real issue is Hydra not fetching network resources, loading arbitrary paths or running Nix recursively; partially due to ‘restricted mode’ in Nix. Lots of ongoing discussions around this, but for now seems easiest to try each feature out and see which part is failing; building up to a full test suite, rather than refactoring tests ‘until they work’, when we don’t know it we’re headed in the right direction!