diff --git a/docs/BENCHMARKING.md b/docs/BENCHMARKING.md new file mode 100644 index 0000000..12e9fde --- /dev/null +++ b/docs/BENCHMARKING.md @@ -0,0 +1,44 @@ +# Benchmarking + +How the performance numbers attached to this project are measured, so that a +number in a changelog entry or a release note can be reproduced and trusted. + +## The suite + +The benchmarks live in `bench_test.go`, next to the code they measure: + +| Benchmark | What it measures | +|---|---| +| `BenchmarkParse` | `Parse` over a representative configuration document | +| `BenchmarkMarshal` | `Marshal` of the tree `Parse` produced from the same document | +| `BenchmarkStrictDecode` | `Decode` into a struct under `DisallowUnknownFields` | +| `BenchmarkParseLong` | `Parse` over a generated document with about 2000 array-of-tables entries | + +## Running + +```sh +just bench +``` + +The recipe runs the suite with `-benchmem -count=5`. Every benchmark uses +`b.Loop`, so setup runs outside the timed region, and `ReportAllocs` records +allocations per operation. The parse and marshal benchmarks set `SetBytes`, so +their results read as input bytes per second. + +## Method + +- An idle machine only: a loaded box times whatever else is running, and the + fastest sample can land on the wrong function. +- An A/B comparison runs both variants inside one process, in one binary; + separate processes of identical binaries differ by more than the effect + being measured. +- The five counts are compared through their medians, allocations and bytes + per operation alongside the times. Differences within 1 to 2 percent are + noise; only a difference beyond that is a result. +- When timing is hopeless, the allocation and byte counts are the result. + +## Reports + +The repository stores no benchmark reports. A performance claim in +`CHANGELOG.md` is measured with the method above on the change that makes it, +and the number travels with the claim. diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 097030e..dbdd1b1 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -77,7 +77,9 @@ just bench ``` Benchmark on an idle machine, and compare only runs made in one process against -each other. The recipe sweeps `./...` five times with `-benchmem`. +each other. The recipe sweeps `./...` five times with `-benchmem`. The binding +measurement method, and what counts as a result, is in +[docs/BENCHMARKING.md](BENCHMARKING.md). ## Debugging the build