# Benchmarking

How the performance numbers attached to this project are measured, so that a
number in a changelog entry or a release note can be reproduced and trusted.

## The suite

The benchmarks live in `bench_test.go`, next to the code they measure:

| Benchmark | What it measures |
|---|---|
| `BenchmarkParse` | `ParseMap` over a representative configuration document |
| `BenchmarkMarshal` | `Marshal` of the tree `ParseMap` produced from the same document |
| `BenchmarkStrictDecode` | `Decode` into a struct under `DisallowUnknownFields` |
| `BenchmarkParseLong` | `ParseMap` over a generated document with about 2000 array-of-tables entries |
| `BenchmarkStrictDecodeLong` | `Decode` into a typed document under `DisallowUnknownFields`, over the same long document |
| `BenchmarkMarshalLong` | `Marshal` of the tree `ParseMap` produced from the long document |

## Running

```sh
just bench
```

The recipe runs the suite with `-benchmem -count=5`. Every benchmark uses
`b.Loop`, so setup runs outside the timed region, and `ReportAllocs` records
allocations per operation. The parse and marshal benchmarks set `SetBytes`, so
their results read as input bytes per second.

## Method

- An idle machine only: a loaded box times whatever else is running, and the
  fastest sample can land on the wrong function.
- An A/B comparison runs both variants inside one process, in one binary;
  separate processes of identical binaries differ by more than the effect
  being measured.
- The five counts are compared through their medians, allocations and bytes
  per operation alongside the times. Differences within 1 to 2 percent are
  noise; only a difference beyond that is a result.
- When timing is hopeless, the allocation and byte counts are the result.

## Reports

The repository stores no benchmark reports. A performance claim in
`CHANGELOG.md` is measured with the method above on the change that makes it,
and the number travels with the claim.