[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Use asv for running benchmarks
- Subject: Use asv for running benchmarks
- From: Chris Warburton
- Date: Wed, 12 Jul 2017 13:30:44 +0100
- Resolution: fixed
- State: resolved
We should benchmark:
- hashspec
- quickspec
- mlspec
These should probably be different dependencies.
We should add benchmarks for:
- Runtime
- Precision
- Recall
It would be nice to share the same execution for the results of these,
e.g. running `choose_sample 1 1` once, running it through quickspec
once, then spitting out a runtime, precision and recall from those
results.
What we could do is run it in a setup_cache function, which would be
re-used across runs for the same version. Unfortunately that's not
parameterised, so it would all have to be done up front. Similarly,
doing the run inside a dependency would require all results to be
generated up front. Also, these wouldn't let us use asv's built-in
support for repetitions and parameters, which match what we want quite
closely.
How about, for now, we just implement these as standalone things? We can
worry about optimising them through re-use later.