interpres/docs/BENCHMARKING.md

Benchmarking

How the performance numbers attached to this project are measured, so that a number in a changelog entry or a release note can be reproduced and trusted.

The suite

The benchmarks live in bench_test.go, next to the code they measure:

Benchmark	What it measures
BenchmarkParse	Parse over a representative configuration document
BenchmarkMarshal	Marshal of the tree Parse produced from the same document
BenchmarkStrictDecode	Decode into a struct under DisallowUnknownFields
BenchmarkParseLong	Parse over a generated document with about 2000 array-of-tables entries

Running

just bench

The recipe runs the suite with -benchmem -count=5. Every benchmark uses b.Loop, so setup runs outside the timed region, and ReportAllocs records allocations per operation. The parse and marshal benchmarks set SetBytes, so their results read as input bytes per second.

Method

• An idle machine only: a loaded box times whatever else is running, and the fastest sample can land on the wrong function.
• An A/B comparison runs both variants inside one process, in one binary; separate processes of identical binaries differ by more than the effect being measured.
• The five counts are compared through their medians, allocations and bytes per operation alongside the times. Differences within 1 to 2 percent are noise; only a difference beyond that is a result.
• When timing is hopeless, the allocation and byte counts are the result.

Reports

The repository stores no benchmark reports. A performance claim in CHANGELOG.md is measured with the method above on the change that makes it, and the number travels with the claim.