49 lines
2.2 KiB
Markdown
49 lines
2.2 KiB
Markdown
# Benchmarking
|
|
|
|
How the performance numbers attached to this project are measured, so that a
|
|
number in a changelog entry or a release note can be reproduced and trusted.
|
|
|
|
## The suite
|
|
|
|
The benchmarks live in `bench_test.go`, next to the code they measure:
|
|
|
|
| Benchmark | What it measures |
|
|
|---|---|
|
|
| `BenchmarkParse` | `ParseMap` over a representative configuration document |
|
|
| `BenchmarkMarshal` | `Marshal` of the tree `ParseMap` produced from the same document |
|
|
| `BenchmarkStrictDecode` | `Unmarshal` into a struct under `RejectUnknownFields` (the targeted parse) |
|
|
| `BenchmarkParseLong` | `ParseMap` over a generated document with about 2000 array-of-tables entries |
|
|
| `BenchmarkStrictDecodeLong` | `Unmarshal` into a typed document under `RejectUnknownFields`, over the same long document |
|
|
| `BenchmarkMarshalLong` | `Marshal` of the tree `ParseMap` produced from the long document |
|
|
| `BenchmarkStrictDecodeTree` | the tree-path reference decode of the representative document: parse, then the reflection walk |
|
|
| `BenchmarkStrictDecodeTreeLong` | the tree-path reference decode of the long document, the A/B baseline of the targeted parse |
|
|
|
|
## Running
|
|
|
|
```sh
|
|
just bench
|
|
```
|
|
|
|
The recipe runs the suite with `-benchmem -count=5`. Every benchmark uses
|
|
`b.Loop`, so setup runs outside the timed region, and `ReportAllocs` records
|
|
allocations per operation. The parse and marshal benchmarks set `SetBytes`, so
|
|
their results read as input bytes per second.
|
|
|
|
## Method
|
|
|
|
- An idle machine only: a loaded box times whatever else is running, and the
|
|
fastest sample can land on the wrong function.
|
|
- An A/B comparison runs both variants inside one process, in one binary;
|
|
separate processes of identical binaries differ by more than the effect
|
|
being measured.
|
|
- The five counts are compared through their medians, allocations and bytes
|
|
per operation alongside the times. Differences within 1 to 2 percent are
|
|
noise; only a difference beyond that is a result.
|
|
- When timing is hopeless, the allocation and byte counts are the result.
|
|
|
|
## Reports
|
|
|
|
The repository stores no benchmark reports. A performance claim in
|
|
`CHANGELOG.md` is measured with the method above on the change that makes it,
|
|
and the number travels with the claim.
|