45 lines
1.7 KiB
Markdown
45 lines
1.7 KiB
Markdown
|
# Benchmarking
|
||
|
|
|
||
|
|
How the performance numbers attached to this project are measured, so that a
|
||
|
|
number in a changelog entry or a release note can be reproduced and trusted.
|
||
|
|
|
||
|
|
## The suite
|
||
|
|
|
||
|
|
The benchmarks live in `bench_test.go`, next to the code they measure:
|
||
|
|
|
||
|
|
| Benchmark | What it measures |
|
||
|
|
|---|---|
|
||
|
|
| `BenchmarkParse` | `Parse` over a representative configuration document |
|
||
|
|
| `BenchmarkMarshal` | `Marshal` of the tree `Parse` produced from the same document |
|
||
|
|
| `BenchmarkStrictDecode` | `Decode` into a struct under `DisallowUnknownFields` |
|
||
|
|
| `BenchmarkParseLong` | `Parse` over a generated document with about 2000 array-of-tables entries |
|
||
|
|
|
||
|
|
## Running
|
||
|
|
|
||
|
|
```sh
|
||
|
|
just bench
|
||
|
|
```
|
||
|
|
|
||
|
|
The recipe runs the suite with `-benchmem -count=5`. Every benchmark uses
|
||
|
|
`b.Loop`, so setup runs outside the timed region, and `ReportAllocs` records
|
||
|
|
allocations per operation. The parse and marshal benchmarks set `SetBytes`, so
|
||
|
|
their results read as input bytes per second.
|
||
|
|
|
||
|
|
## Method
|
||
|
|
|
||
|
|
- An idle machine only: a loaded box times whatever else is running, and the
|
||
|
|
fastest sample can land on the wrong function.
|
||
|
|
- An A/B comparison runs both variants inside one process, in one binary;
|
||
|
|
separate processes of identical binaries differ by more than the effect
|
||
|
|
being measured.
|
||
|
|
- The five counts are compared through their medians, allocations and bytes
|
||
|
|
per operation alongside the times. Differences within 1 to 2 percent are
|
||
|
|
noise; only a difference beyond that is a result.
|
||
|
|
- When timing is hopeless, the allocation and byte counts are the result.
|
||
|
|
|
||
|
|
## Reports
|
||
|
|
|
||
|
|
The repository stores no benchmark reports. A performance claim in
|
||
|
|
`CHANGELOG.md` is measured with the method above on the change that makes it,
|
||
|
|
and the number travels with the claim.
|