diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..774408b --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,69 @@ +# Changelog + +All notable changes to **interpres** are documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [development] + +### Added + +**Parsing** + +- `Parse(data []byte) (map[string]any, error)`: decode a TOML document into an + untyped tree. +- Full TOML 1.0 syntax: comments; bare, quoted, and dotted keys; tables + (`[a.b]`) and arrays of tables (`[[a]]`); basic and literal strings, including + multiline (`"""` / `'''`) with escape sequences and line-ending backslash + trimming; integers in decimal, hex (`0x`), octal (`0o`), and binary (`0b`) + with `_` separators; floats with exponents and `inf` / `nan`; booleans; + offset/local date-times, dates, and times; arrays; and inline tables. +- Distinct date-time types: offset date-times decode to `time.Time`, while + `LocalDateTime`, `LocalDate`, and `LocalTime` represent the local variants, + each with a `String()` method returning the TOML-canonical rendering. +- Strict, spec-conformant validation that rejects invalid documents: leading + zeros, misplaced underscores, malformed floats and radix literals, control + characters in strings and comments, bare carriage returns, non-UTF-8 input, + out-of-range Unicode escapes, multiline strings used as keys, single-digit + date-time components, duplicate/overwriting inline-table keys, and the full + family of table redefinitions (header vs. header, header vs. dotted key, + array of tables vs. table, and dotted-key appends to defined tables). +- `SyntaxError` carrying the 1-based line of a malformed document. + +**Decoding** + +- `Unmarshal(data []byte, v any)`: parse and map onto a struct or + `map[string]any` via reflection, with overflow-checked numeric conversion, + nested structs, slices, and `map[string]T`. +- `toml:"name"` field tags, case-insensitive name fallback, and `toml:"-"` to + skip a field. +- `Decoder` with `DisallowUnknownFields` for strict decoding that rejects keys + without a destination field, at every struct depth. +- `Unmarshaler` interface (`UnmarshalTOML(data any) error`) for types that take + full control of their decode. + +**Encoding** + +- `Marshal(v any) ([]byte, error)`: encode a struct or `map[string]V` value to + a TOML 1.0 document that re-parses to an equivalent value tree. +- `Encoder` with chainable policy options: `GroupByKind` (group-by-kind layout + versus declaration order), `OmitEmptyArrays`, and `UseLiteralMultiline`. +- `Marshaler` interface (`MarshalTOML() (any, error)`) for types that need a + custom TOML shape; the returned value is encoded normally. + +**Cancellation** + +- `ParseContext`, `UnmarshalContext`, `MarshalContext`, + `(*Decoder).DecodeContext` and `(*Encoder).MarshalContext` honour a + `context.Context`, checked up front and every 64 statements or fields. + +**Project** + +- Standard library only: zero third-party modules. +- `cmd/interpres-decode`, a toml-test harness adapter (TOML on stdin, tagged + JSON on stdout). +- A runnable example, a table-driven Go test suite, and a canonical `just` + recipe set whose `gates` recipe is the definition of done. +- Hand-written CI pipelines for the test, race and release gates. +- `SECURITY.md` for private vulnerability reports. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..f59c33c --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,111 @@ +# Contributing + +Thanks for contributing to **interpres**. + +## Development setup + +Requirements: Go 1.27.0, the version `go.mod` declares, and +[just](https://github.com/casey/just) for the recipes. + +```sh +git clone https://sourcedock.dev/petrbalvin/interpres.git +cd interpres +just build +just test +``` + +## Workflow + +1. Branch from `development`. Never commit directly to `main`, which is + release-only. +2. Commit in [Conventional Commits](https://www.conventionalcommits.org/) form: + `type(scope): description`, subject line only, imperative mood, lowercase + after the colon, no trailing full stop. Allowed types: `feat`, `fix`, `docs`, + `style`, `refactor`, `perf`, `test`, `chore`, `ci`, `build`, `revert`. +3. One logical change per commit. A refactor, a behaviour change and a + formatting pass are three commits, never one. +4. Record every user-visible change in `CHANGELOG.md` under `## [development]`. +5. Add or update tests. Coverage stays at 80 percent or more; it is a hard + gate. Parser and decoder changes must also keep the toml-test suite at zero + failures, checked with `just toml-test`. +6. Update the documentation when the public API, the configuration or the + behaviour changes; the documents move in the same commit as the behaviour + they describe. +7. Open a pull request against `development`. + +Releases are cut by merging `development` into `main` and tagging `vX.Y.Z`. The +release workflow runs the full gate set including the race detector and +publishes the Gitea release with the matching `CHANGELOG.md` section as its +notes. + +## Code style + +The project is standard library only: no third-party Go module enters `go.mod`, +because that constraint is the point of the project. + +The formatter is `gofmt` and the linters are `go vet` and `go fix -diff`, run +through the recipes: `just fmt` formats in place, `just fmt-check` demands a +zero diff, `just vet` runs both static gates, and `just gates` is the +definition of done in one command. Errors are checked explicitly and wrapped as +`fmt.Errorf("context: %w", err)`; nothing panics outside `main`. Tests are +table-driven and live next to the code they cover. + +New source files open with the project's two-line MIT licence header, whose +SPDX identifier matches `LICENSE`. Configuration files, workflows and dotfiles +do not carry it. + +## AI contribution policy + +AI tools are welcome as productivity aids and are a normal part of modern +software development. What matters is that the contribution stays +understandable, reviewable and genuinely useful. + +- **Disclose the assistance.** If AI helped draft any part of a commit, issue, + pull request or review, say so. +- **Commit messages carry exactly one trailer**, on the line after the subject: + + ``` + Assisted-by: MODEL + ``` + + Name the model that did the work, spelled the way its maker spells it, for + example `GLM 5.3`, `DeepSeek V4.1 Flash` or `Qwen 3.8 Flash`. No + `Co-Authored-By`, no `Signed-off-by`, no other trailers, and no prose: the + trailer is the disclosure. +- **Issues and pull requests** attribute the assistance in a comment, for + example `_Assisted-by: GLM 5.3_`. It does not belong in the pull request + description. +- **Take responsibility.** You are accountable for the accuracy, completeness + and intent of everything you submit, whether or not AI produced it. +- **Review before marking ready.** Read the diff carefully, run it locally, and + add the tests it needs. Do not mark a pull request ready until you can defend + every change in it. +- **Quality over quantity.** Contributions that look like un-reviewed output, + or whose author cannot engage substantively during review, may be closed. +- **Preferred models.** Prefer open-weight models with transparent training + data and minimal output filtering. + +AI assists. It does not replace judgement. + +## Continuous integration + +Workflows live in `.gitea/workflows/` and run on the project's own runners: + +| Workflow | Trigger | What it does | +|---|---|---| +| Test | push or pull request to `development` | format check, vet, modernisation, build, the test suite with the coverage floor, the toml-test compliance suite | +| Race | `workflow_dispatch`, by hand | the suite under the race detector, the same race gate the local `just gates` runs | +| Release | a `v*` tag | the same gates plus the race detector, then the Gitea release created from the `CHANGELOG.md` section | + +The local equivalent is `just gates`, which is the same set plus the race +detector. + +## Reporting bugs + +Open an issue at `https://sourcedock.dev/petrbalvin/interpres/issues` with the +version, the operating system and architecture, the exact command, the full +output, and the expected against the actual behaviour. For a parser bug, the +smallest TOML document that triggers it decides how fast it is fixed. + +**Security issues do not go in the issue tracker.** Report them as +[SECURITY.md](SECURITY.md) describes. diff --git a/README.md b/README.md new file mode 100644 index 0000000..84763ae --- /dev/null +++ b/README.md @@ -0,0 +1,169 @@ +# interpres + +A TOML 1.0 parser and encoder for Go, written with the standard library alone. +`interpres` (Latin for *interpreter*) gives zero-dependency programs an +`encoding/json`-style API for reading and writing TOML, and passes the entire +official [toml-test](https://github.com/toml-lang/toml-test) suite: 185 valid +and 371 invalid cases, zero failures. + +## Features + +- **Full TOML 1.0**: bare, quoted and dotted keys; tables and arrays of tables; + basic and literal strings including multiline; integers in the four radixes + with `_` separators; floats with exponents, `inf` and `nan`; booleans; the + four date-time kinds; arrays and inline tables. +- **Decoding and encoding**: `Parse` for an untyped tree, `Unmarshal` and + `Marshal` for structs and maps, mirroring `encoding/json`. +- **Strict decoding**: `NewDecoder().DisallowUnknownFields()` rejects keys that + match no destination field, at every struct depth. +- **Custom types**: `Marshaler` and `Unmarshaler` let a type control its own + TOML representation in both directions. +- **Cancellation**: every entry point has a `*Context` sibling that honours a + `context.Context`. +- **Configurable emission**: `Encoder` options for declaration-order output, + omitting empty arrays, and literal multiline strings. + +## Install + +As a library: + +```sh +go get sourcedock.dev/petrbalvin/interpres +``` + +Requires Go 1.27.0 or newer. The module imports only the standard library. + +## Quick start + +```sh +git clone https://sourcedock.dev/petrbalvin/interpres.git +cd interpres +just example +``` + +`just example` runs the tour in `examples/basic`: it decodes an embedded +document into a struct, prints it, and re-encodes it under both `Encoder` +layouts. + +## Usage + +### Decode into a struct + +```go +type Config struct { + Title string `toml:"title"` + Server struct { + Host string `toml:"host"` + Port int `toml:"port"` + } `toml:"server"` +} + +var cfg Config +err := interpres.Unmarshal(data, &cfg) +``` + +Fields match by the `toml:"name"` tag, or by the lower-cased field name when no +tag is present; `toml:"-"` skips a field. `Parse` returns the untyped +`map[string]any` tree instead, and `UnmarshalContext` accepts a context. + +### Encode from a struct + +```go +out, err := interpres.Marshal(cfg) +``` + +writes: + +```toml +title = "example" + +[server] +host = "127.0.0.1" +port = 9090 +``` + +Tables are laid out scalars first, then sub-tables, then arrays of tables, +which is the layout that re-parses to the same tree. + +### Strict decoding + +```go +err := interpres.NewDecoder(). + DisallowUnknownFields(). + Decode(data, &cfg) +``` + +A key with no matching field becomes an error instead of a silent drop. + +### Custom types + +```go +type Port int + +func (p Port) MarshalTOML() (any, error) { + return int64(p), nil +} + +type IP struct{ net.IP } + +func (ip *IP) UnmarshalTOML(data any) error { + s, ok := data.(string) + if !ok { + return fmt.Errorf("ip: not a string") + } + ip.IP = net.ParseIP(s) + return nil +} +``` + +The value `MarshalTOML` returns is encoded in place of the receiver; +`UnmarshalTOML` receives the parsed value verbatim. + +### Encoder options + +```go +out, err := interpres.NewEncoder(). + GroupByKind(false). // preserve declaration order + OmitEmptyArrays(). // skip empty scalar arrays + UseLiteralMultiline(80). // long multi-line strings as literal blocks + Marshal(cfg) +``` + +### Cancellation + +```go +ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) +defer cancel() + +out, err := interpres.MarshalContext(ctx, cfg) +``` + +`ParseContext`, `UnmarshalContext`, `(*Decoder).DecodeContext` and +`(*Encoder).MarshalContext` follow the same pattern. + +The full rules for field matching, numeric conversion and emission live in +[docs/API.md](docs/API.md). + +## Development + +```sh +just build # compile bin/interpres-decode +just test # the suite, with the 80 percent coverage floor +just toml-test # the official compliance suite, needs toml-test on PATH +just fmt # gofmt in place +``` + +See [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) for the full workflow, and +[CONTRIBUTING.md](CONTRIBUTING.md) for how to contribute. + +## Documentation + +- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md): components and data flow +- [docs/API.md](docs/API.md): the API reference, decoding and encoding rules +- [docs/CLI.md](docs/CLI.md): the interpres-decode toml-test adapter + +## Licence + +MIT. See [LICENSE](LICENSE). + +Copyright © 2026 [Petr Balvín](https://petrbalvin.org) diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..a719649 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,42 @@ +# Security policy + +## Supported versions + +Security fixes go to the newest release and to the `development` branch. Older +releases do not receive them. + +| Version | Supported | +|---|---| +| 1.0.0 | yes | +| older releases | no | + +## Reporting a vulnerability + +**Do not open a public issue for a security problem.** A public report tells +everyone about the flaw before there is a fix. Report it privately to +**opensource@petrbalvin.org**. + +Include: + +- the version or commit you tested, and the platform +- what the problem is, and what an attacker gains from it +- the smallest reproducer you have, ideally a test or a single command +- a suggested fix, if you have one + +## What to expect + +- A human reads the report, and you get an acknowledgement. +- You are kept informed while the fix is being made, and told when it ships. +- The fix is released before the details are published, and the timing is + agreed with you. +- You are credited in the `Security` section of `CHANGELOG.md` if you want to + be. + +## Out of scope + +- Findings that require the attacker to already run code as the user, or to + have local access. +- Missing hardening with no demonstrated impact. +- Flaws in a third-party dependency. This module has none; the standard + library is the only import, and standard library issues belong with the Go + project. diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 0000000..dfab6c0 --- /dev/null +++ b/docs/API.md @@ -0,0 +1,431 @@ +# API + +The library exports the surface below from the `sourcedock.dev/petrbalvin/interpres` +package. The snippets assume: + +```go +import "sourcedock.dev/petrbalvin/interpres" +``` + +## Functions + +### `func Parse(data []byte) (map[string]any, error)` + +Decodes a TOML document into an untyped tree, using the value mapping in the +[Decoding](#decoding) section below. Returns `*SyntaxError` on a malformed +document. Input that is not valid UTF-8 is rejected before the parser runs. +Equivalent to `ParseContext(context.Background(), data)`. + +```go +tree, err := interpres.Parse([]byte("title = \"x\"\nport = 8080\n")) +``` + +### `func ParseContext(ctx context.Context, data []byte) (map[string]any, error)` + +The cancellable variant of `Parse`. An already-cancelled context returns +`ctx.Err()` before any work. During parsing the context is checked every 64 +top-level statements, so a long document aborts without running to completion. + +### `func Unmarshal(data []byte, v any) error` + +Parses `data` and stores the result in the value pointed to by `v`, typically a +pointer to a struct or to `map[string]any`. Equivalent to +`UnmarshalContext(context.Background(), data, v)`. + +```go +var cfg Config +if err := interpres.Unmarshal(data, &cfg); err != nil { + return err +} +``` + +### `func UnmarshalContext(ctx context.Context, data []byte, v any) error` + +The cancellable variant of `Unmarshal`. + +### `func Marshal(v any) ([]byte, error)` + +Encodes a `struct` or `map[string]V` value, or a non-nil pointer to one, into a +TOML 1.0 document. The emission rules are in the [Encoding](#encoding) section +below. Equivalent to `MarshalContext(context.Background(), v)`. + +```go +out, err := interpres.Marshal(cfg) +``` + +### `func MarshalContext(ctx context.Context, v any) ([]byte, error)` + +The cancellable variant of `Marshal`. The context is checked before any work +and every 64 fields during the reflection walk. + +## Decoding + +### Value mapping + +`Parse` and `Unmarshal` map TOML values to Go types as follows: + +| TOML value | Go type in the parsed tree | +|---|---| +| string | `string` | +| integer | `int64` | +| float | `float64` | +| boolean | `bool` | +| offset date-time | `time.Time` | +| local date-time | `LocalDateTime` | +| local date | `LocalDate` | +| local time | `LocalTime` | +| array | `[]any` | +| table, inline table | `map[string]any` | +| array of tables | `[]map[string]any` | + +When decoding into a struct, these values convert onto the destination's +concrete types: any integer or unsigned width, floats, slices, nested structs +and `map[string]T`. + +### Target constraints + +`Unmarshal` and `(*Decoder).Decode` write into a non-nil pointer: + +- `*struct`, matched per the field rules below +- `*map[string]any` or `*map[string]T`, keys become map keys and values decode + into `T` recursively +- `*any`, receives the whole parsed tree unchanged + +Anything else returns `interpres: decode target must be a non-nil pointer`. + +### Field matching + +For a struct destination, a TOML key matches a field as follows: + +1. The `toml:"name"` tag, using the part before any comma. The literal `-` + excludes the field. +2. Without a tag, the lower-cased field name. +3. The key itself is lower-cased before lookup, so the match is + case-insensitive on both sides: `DATABASEURL` matches a field named + `DatabaseUrl`. + +The match is exact after lower-casing. No separator is inserted, so a TOML key +`database_url` does not match a field named `DatabaseUrl`; tag such a field +(`toml:"database_url"`) or use the lower-cased name as the key. When two fields +resolve to the same name, the one declared later wins. + +Unknown keys are ignored by default; see [Strict decoding](#strict-decoding). + +### Numeric conversion + +The parser produces `int64` for every integer and `float64` for every float. +The decoder converts to the destination type with explicit overflow checks: + +| Destination kind | Rule | +|---|---| +| `int`, `int8`, `int16`, `int32`, `int64` | the `int64` value must not overflow the destination | +| `uint`, `uint8`, `uint16`, `uint32`, `uint64` | the value must be non-negative; `uint8`, `uint16` and `uint32` enforce their own maxima; `uint64` accepts any non-negative `int64` | +| `float32`, `float64` | copied verbatim; an integer also coerces, so TOML `5` decodes into `5.0` | +| `bool`, `string` | exact kind match only, no coercion across kinds | +| `time.Time` | offset date-times only; no implicit conversion to or from the local variants | + +A conversion that the rules do not allow produces an error wrapped with the +offending key or index, for example `p: interpres: integer 300 overflows uint8`. + +### Date-time values + +Offset date-times decode into `time.Time` and keep their offset. The local +variants decode into `LocalDateTime`, `LocalDate` and `LocalTime`, whose +embedded `time.Time` is normalised to UTC (midnight UTC for a local date, the +zero date for a local time). There is no implicit conversion between the offset +and local kinds; assigning one to the other is an error. + +### Arrays of tables + +A `[[a]]` block parses into a `[]map[string]any` element of the tree. When the +destination is a slice, each element decodes into the slice's element type +(`[]struct` or `[]map[string]V`); a mismatch on one element surfaces as an +error wrapped with `[i]:` and the element index. + +### Custom decoding: `Unmarshaler` + +A type that wants full control of its decode implements: + +```go +type Unmarshaler interface { + UnmarshalTOML(data any) error +} +``` + +`data` is whatever the parser produced for that key: `string`, `bool`, `int64`, +`float64`, `time.Time`, `LocalDateTime`, `LocalDate`, `LocalTime`, `[]any`, or +`map[string]any`. The method inspects the value and mutates its own receiver; +the decoder keeps whatever state the receiver stored. + +The method is usually on a pointer receiver (`*T`). The decoder invokes it when +the destination type or its pointer implements the interface, so a +pointer-receiver implementation on an addressable struct field is found +automatically, and a nil pointer destination is allocated first. An error +returned from `UnmarshalTOML` halts the decode and propagates wrapped with the +key path, for example `addr: unmarshal: not a string`. + +### Strict decoding + +By default unknown keys are dropped silently. A `Decoder` built with +`DisallowUnknownFields` rejects them instead: + +```go +err := interpres.NewDecoder(). + DisallowUnknownFields(). + Decode(data, &cfg) +``` + +A typo such as `database_urls` then fails with +`interpres: unknown field "database_urls" for main.Config` instead of a silent +default-zero run. Strictness applies to every struct the decode reaches, at any +depth, including struct elements inside slices; map destinations accept every +key by nature. + +### Cancellation + +`ParseContext`, `UnmarshalContext` and `(*Decoder).DecodeContext` accept a +`context.Context`. An already-cancelled context short-circuits with +`context.Canceled` before any work begins; afterwards the context is checked +every 64 top-level statements. + +### Flow + +```mermaid +sequenceDiagram + participant Caller + participant Unmarshal as Unmarshal + participant Parser as parser + participant Decoder as decoder + Caller->>Unmarshal: data, v + Unmarshal->>Parser: ParseContext(ctx, data) + Parser-->>Unmarshal: tree or *SyntaxError + Unmarshal->>Decoder: decode(tree, reflect value) + Decoder-->>Unmarshal: nil or wrapped field error + Unmarshal-->>Caller: error +``` + +## Encoding + +### Input constraints + +`Marshal` and `(*Encoder).Marshal` accept a `struct`, a `map[string]V`, or a +non-nil pointer to one, where `V` is any value `Marshal` itself understands. A +different top-level value fails: + +| Input | Error | +|---|---| +| a bare scalar or array | `interpres: top-level value must be a struct or map[string]V, got ` | +| a nil `any` | `interpres: cannot marshal nil value` | +| a nil pointer | `interpres: cannot marshal nil pointer` | + +### Field matching + +Struct fields become TOML keys as follows: + +1. The `toml:"name"` tag, using the part before any comma. The literal `-` + skips the field. +2. Without a tag, the lower-cased field name. The key emitted for a field named + `DatabaseUrl` is `databaseurl`; tag the field to emit `database_url`. +3. An anonymous (embedded) field without a tag is inlined into the parent + table; with a tag it is a regular field under that name. + +Keys that match `[A-Za-z0-9_-]+` are emitted bare, all others quoted. A +`map[string]V` emits its keys in sorted order for deterministic output, and a +nil map emits nothing. + +Note the asymmetry: the encoder inlines untagged embedded structs, while the +decoder expects them under their lower-cased type name. A struct with an +untagged embedded struct therefore does not round-trip through `Unmarshal` into +the same type. + +### Group-by-kind layout + +By default every table is emitted with its entries grouped by kind: + +1. scalars (`string`, `int64`, `float64`, `bool`, `time.Time`, + `LocalDateTime`, `LocalDate`, `LocalTime`) +2. sub-tables (structs and `map[string]V` values) +3. arrays of tables (`[]struct` and `[]map[string]V`) + +Within each group the order follows struct field declaration order, or sorted +key order for maps. This is the only layout that reliably re-parses to the same +tree: once a `[header]` is written, later scalars at the parent level would be +parsed as keys of the sub-table. + +### Preserving declaration order + +`GroupByKind(false)` on an `Encoder` walks the entries in declaration order +instead, emitting each header immediately before its content: + +```go +out, err := interpres.NewEncoder().GroupByKind(false).Marshal(cfg) +``` + +The output remains parseable, but a scalar declared after a sub-table lands +under that sub-table's header when the document is read back. Use this layout +for presentation only, not when the output must round-trip. + +### Custom encoding: `Marshaler` + +A type that wants a non-default TOML shape implements: + +```go +type Marshaler interface { + MarshalTOML() (any, error) +} +``` + +The returned value is encoded as if it had been passed in place of the +receiver, so it may be a scalar, a slice, an array of tables, or another +struct or map, including the `Marshaler` result of another type; the encoder +recurses. An error returned from `MarshalTOML` fails the marshal wrapped with +the key path, for example `interpres: server.port: bad timestamp`. + +```go +type Port int + +func (p Port) MarshalTOML() (any, error) { + return int64(p), nil +} +``` + +### Empty arrays + +A nil slice is always omitted. An empty (length 0) array of tables is always +omitted, because TOML forbids an empty `[[a]]`. Other empty arrays emit as +`key = []` by default; `OmitEmptyArrays()` skips them as well, so +`[]string{}` is treated like a nil slice. + +### Long strings + +By default every string is emitted as a basic `"..."` string with the escapes +TOML requires, and a string containing a newline is emitted as an escaped +multi-line basic string. `UseLiteralMultiline(threshold)` switches strings that +contain a newline and are at least `threshold` bytes long to the literal +`'''...'''` form, which carries the newlines verbatim: + +```go +out, err := interpres.NewEncoder().UseLiteralMultiline(80).Marshal(cfg) +``` + +Single-line strings keep the basic form regardless of the threshold, and a +threshold of `0` or less disables the option. + +### Cancellation + +`MarshalContext` and `(*Encoder).MarshalContext` accept a `context.Context`. The +context is checked before any work and every 64 fields during the reflection +walk. + +### What is not preserved + +The output is not byte-identical to any document that produced the value: + +- comments are dropped, and whitespace inside expressions is normalised +- map keys are emitted in sorted order +- the choice between `[table]` headers and inline tables is not preserved +- strings use the basic quoted form unless the literal option above applies +- floats always carry a `.` or an exponent, so a float `1` is emitted as `1.0` + and stays distinguishable from the integer `1` across a round-trip; negative + zero is normalised to `0.0` + +The output is guaranteed to re-parse through `Parse` into an equivalent value +tree. `Marshal` cannot encode cyclic data structures. + +### Flow + +```mermaid +sequenceDiagram + participant Caller + participant Marshal as Marshal + participant Walk as reflection walk + participant Emit as emitter + Caller->>Marshal: v any + Marshal->>Walk: build tomlDoc from struct or map + Walk-->>Marshal: tomlDoc or wrapped error + Marshal->>Emit: emitDoc(doc) + Emit-->>Marshal: bytes or error + Marshal-->>Caller: bytes, error +``` + +## Types + +### `type SyntaxError struct{ Line int; Msg string }` + +Describes a malformed TOML document; `Line` is 1-based and `Error()` renders as +`interpres: line N: msg`. Read the structured fields with a type assertion or +`errors.AsType`: + +```go +if se, ok := errors.AsType[*interpres.SyntaxError](err); ok { + fmt.Println(se.Line, se.Msg) +} +``` + +### `type Decoder` + +Configurable strictness for decoding, constructed with `NewDecoder`. Set up +with `DisallowUnknownFields`, then call `Decode` or `DecodeContext` any number +of times. A configured `Decoder` holds no per-call state and is safe for +concurrent use. + +### `type Encoder` + +Configurable emission policy, constructed with `NewEncoder`. The option state +is private; set it with the chainable methods, each of which returns the +encoder: + +| Method | Default | Effect | +|---|---|---| +| `GroupByKind(v bool)` | `true` | group entries as scalars, then sub-tables, then arrays of tables; `false` preserves declaration order | +| `OmitEmptyArrays()` | off | skip `key = []` for empty scalar arrays | +| `UseLiteralMultiline(threshold int)` | `0` | emit multi-line strings of at least `threshold` bytes as literal `'''...'''` | + +```go +out, err := interpres.NewEncoder(). + GroupByKind(false). + OmitEmptyArrays(). + UseLiteralMultiline(80). + MarshalContext(ctx, cfg) +``` + +A configured `Encoder` holds no per-call state; each `Marshal` or +`MarshalContext` call copies the options and is safe for concurrent use, as +long as no setter races with a call. + +### `type Marshaler interface{ MarshalTOML() (any, error) }` + +See [Custom encoding](#custom-encoding-marshaler). + +### `type Unmarshaler interface{ UnmarshalTOML(data any) error }` + +See [Custom decoding](#custom-decoding-unmarshaler). + +### Date-time wrappers + +```go +type LocalDateTime struct{ time.Time } // 1979-05-27T07:32:00 +type LocalDate struct{ time.Time } // 1979-05-27 +type LocalTime struct{ time.Time } // 07:32:00.999999 +``` + +Each carries a `String()` method returning the TOML-canonical rendering, with +the fractional second zero-padded to nanosecond precision when present. The +types are produced by `Parse` and accepted by `Marshal`. + +## Errors + +The entry points return: + +- `*SyntaxError` for a malformed document, with the 1-based line +- a plain error for everything else: a non-pointer decode target, a type + mismatch, an overflow, a marshal policy violation, a cancelled context + +Decode and encode failures are wrapped with the key path or element index using +`fmt.Errorf`, so `errors.Is` and `errors.AsType` see through them. + +## Notes + +The exported surface is documented in godoc form in the source, and `go doc .` +run from the module root is the authority on signatures and types. This file +explains what the surface is for and how the parts fit together. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..1bcff50 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,113 @@ +# Architecture + +How interpres is put together. Every file, package and arrow below exists in the +source tree; nothing is aspirational. + +## Overview + +interpres is one public library package, one command, and one example. The +library implements the whole of TOML 1.0, decoding and encoding, in the +standard library alone; the command wraps the parser for the toml-test +compliance harness, against which it stands at 185 valid and 371 invalid cases +with zero failures; the example demonstrates the API. + +```mermaid +flowchart TD + CLI[cmd/interpres-decode
toml-test adapter] --> API + EX[examples/basic
usage demo] --> API + subgraph Lib [package interpres] + API[interpres.go
public API and types] + API --> P[parser.go
recursive-descent parser] + API --> DEC[decode.go
tree onto Go values] + API --> ENC[encode.go
Go values to TOML] + P --> N[number.go
numeric tokens] + P --> DT[datetime.go
date-time atoms and types] + end +``` + +The public API in `interpres.go` is a thin facade: every entry point funnels +into `ParseContext` for parsing and into the unexported `decoder` and `encoder` +for the reflection work. `parser.go` owns the grammar; it leans on +`number.go` and `datetime.go` for the two token families that need their own +strict validation. + +## Packages + +| Path | Responsibility | +|---|---| +| `.` (package `interpres`) | The whole library. `interpres.go` declares the exported surface (`Parse`, `Unmarshal`, `Marshal`, the `*Context` variants, `Decoder`, `Encoder`, `Marshaler`, `Unmarshaler`, `SyntaxError`, the local date-time types); everything below it is unexported. | +| `cmd/interpres-decode` | The toml-test adapter. Reads TOML on stdin, writes tagged JSON on stdout. Owns no parsing logic. | +| `examples/basic` | A runnable tour of the API. Documentation in executable form, not part of the library. | + +Inside the library package, one file owns one concern: + +| File | Responsibility | +|---|---| +| `parser.go` | The recursive-descent parser. Produces the `map[string]any` tree and enforces the structural rules of TOML 1.0 (table redefinitions, dotted keys, arrays of tables). Reports a 1-based line on failure. | +| `number.go` | Strict numeric tokens: integers in the four radixes with `_` separators, and floats including `inf` and `nan`. Rejects leading zeros, misplaced underscores and malformed fractions. | +| `datetime.go` | The three local date-time wrapper types and `parseDateTime`, which classifies a token into the four date-time kinds under the strict TOML grammar. | +| `decode.go` | Maps the parsed tree onto Go values by reflection: struct fields, maps, slices, scalar conversion with overflow checks, `Unmarshaler` dispatch. | +| `encode.go` | The reverse walk: builds an intermediate `tomlDoc` per table (which is what preserves declaration order and enables the group-by-kind partition) and then emits it as TOML. | + +The boundary that matters: `parser.go` produces only untyped trees +(`map[string]any`, `[]any`, `[]map[string]any`, scalars); `decode.go` and +`encode.go` are the only files that touch `reflect`; the command never touches +either, it consumes `Parse` alone. + +## Data flow + +Decoding is parse, then one reflection walk. `SyntaxError` values are produced +inside `parser.go` and returned as-is; conversion errors are produced inside +`decode.go` and wrapped with the key path as they unwind. + +```mermaid +sequenceDiagram + participant Caller + participant API as interpres.go + participant P as parser.go + participant D as decode.go + Caller->>API: Unmarshal(data, v) + API->>P: ParseContext(ctx, data) + P->>P: number and datetime atoms + P-->>API: map tree or *SyntaxError + API->>D: decode(tree, reflect value) + D-->>API: nil or wrapped field error + API-->>Caller: error +``` + +Encoding walks the other way. `encode.go` first builds a `tomlDoc` from the +value, then emits it; the two phases are why `GroupByKind` can reorder entries +without a second reflection pass, and why cancellation is checked during both. + +```mermaid +sequenceDiagram + participant Caller + participant API as interpres.go + participant B as encode.go build + participant E as encode.go emit + Caller->>API: Marshal(v) + API->>B: build tomlDoc from struct or map + B-->>API: tomlDoc or error + API->>E: emitDoc(doc) + E-->>API: bytes or error + API-->>Caller: bytes, error +``` + +## State and lifetime + +- The exported `Decoder` and `Encoder` hold configuration only. Every + `Decode`, `DecodeContext`, `Marshal` and `MarshalContext` call allocates its + own unexported worker, so a configured type is safe for concurrent use; the + setter methods are not, and must finish before the value is shared. +- The parser is allocated per `ParseContext` call; nothing is cached between + documents. +- The date-time wrappers are values, not pointers, and are immutable in use. +- Nothing in the library starts goroutines or holds locks; concurrency safety + comes from having no shared mutable state. + +## Dependencies + +None. `go.mod` declares the module and the Go version and carries no requires; +the library imports the standard library only, which is the point of the +project. The `toml-test` binary is a development and CI tool, never a module +dependency. diff --git a/docs/CLI.md b/docs/CLI.md new file mode 100644 index 0000000..f0ee4a2 --- /dev/null +++ b/docs/CLI.md @@ -0,0 +1,71 @@ +# Command line + +The reference below is taken from the program itself. `interpres-decode` is the +toml-test harness adapter, not a general-purpose tool: it takes no flags and no +arguments, reads one TOML document from stdin, and writes the toml-test +tagged-JSON form to stdout. + +## Synopsis + +```sh +interpres-decode < document.toml +``` + +Build it with `just build`, which compiles it into `bin/interpres-decode`, or +run it straight from the module directory with `just run`. + +## Exit codes + +| Code | Meaning | +|---|---| +| `0` | the document parsed, tagged JSON written to stdout | +| `1` | parse error, the document is malformed; the message goes to stderr | +| `2` | reading stdin failed, or a value has no tagged representation | + +## Wire format + +Tables become JSON objects, arrays become JSON arrays, and every scalar is +wrapped in an object with a `type` and a `value`: + +```json +{ + "title": {"type": "string", "value": "hello"}, + "port": {"type": "integer", "value": "9090"}, + "enabled": {"type": "bool", "value": "true"}, + "ratio": {"type": "float", "value": "3.14"} +} +``` + +| TOML value | Tag | Rendering | +|---|---|---| +| string | `string` | the string verbatim | +| integer | `integer` | decimal | +| float | `float` | decimal, or `inf`, `-inf`, `nan` | +| boolean | `bool` | `true` or `false` | +| offset date-time | `datetime` | RFC 3339 with nanoseconds | +| local date-time | `datetime-local` | `1979-05-27T07:32:00` | +| local date | `date-local` | `1979-05-27` | +| local time | `time-local` | `07:32:00.999999` | + +## Examples + +Echo a small document through the adapter: + +```sh +echo 'title = "hello" +[server] +host = "127.0.0.1" +port = 9090 +' | ./bin/interpres-decode +``` + +The output is the equivalent value tree as one JSON object. Run the official +compliance suite against the binary: + +```sh +just toml-test +``` + +That recipe needs the `toml-test` binary on `PATH`, installed with +`go install github.com/toml-lang/toml-test/cmd/toml-test@v1.6.0`. The full +reference for the library itself is [API.md](API.md). diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md new file mode 100644 index 0000000..b4e5d90 --- /dev/null +++ b/docs/DEVELOPMENT.md @@ -0,0 +1,108 @@ +# Development + +How to work on interpres. + +## Prerequisites + +- Go 1.27.0, the version the `go` directive in `go.mod` declares. +- [just](https://github.com/casey/just) for the recipes. +- The `toml-test` binary on `PATH` for the compliance recipe, installed with + `go install github.com/toml-lang/toml-test/cmd/toml-test@v1.6.0`. + +The module has no third-party dependencies, so there is nothing else to fetch. + +## Setup + +```sh +git clone https://sourcedock.dev/petrbalvin/interpres.git +cd interpres +just build +just test +``` + +## Recipes + +Every recipe in the project's justfile, and what it does. Taken from the file +itself, so the names and the list match it exactly; `just` with no arguments +prints the same list. + +| Recipe | What it does | +|---|---| +| `just gates` | the definition of done: build, format check, vet, the test suite with the coverage floor, and the race detector | +| `just build` | compiles `./cmd/interpres-decode` into `bin/interpres-decode` | +| `just test` | the suite with no cache, then the coverage floor of 80 percent from `coverage.out` | +| `just race` | the same suite under the race detector | +| `just unit ./... TestName` | a fast scoped run for iterating; the second argument is a `-run` pattern, `.*` by default | +| `just fuzz FuzzParse . 30s` | time-boxed fuzzing of one target in exactly one package; `go test -fuzz` rejects `./...`; never a gate | +| `just bench` | benchmarks, `-benchmem -count=5`, on an idle machine only | +| `just fmt` | `gofmt -w .`, format in place | +| `just fmt-check` | `gofmt -l .`, zero diff | +| `just vet` | `go vet ./...` and `go fix -diff ./...` | +| `just run` | `go run ./cmd/interpres-decode`, reads TOML from stdin | +| `just dev` | the same run, for iterating | +| `just example` | `go run ./examples/basic`, the usage tour | +| `just toml-test` | builds the adapter and runs the official toml-test compliance suite against it | +| `just coverage-html` | `just test`, then `go tool cover -html` into `coverage.html` | +| `just install` | builds, then copies the binary into `~/.local/bin` (`BINDIR` overrides) | +| `just uninstall` | removes the installed binary | +| `just clean` | removes `bin/` and `coverage.out` | + +## Running a single test + +```sh +just unit ./... TestParseMultilineString +go test -run 'TestRejectsSpecInvalid/table_over_array' ./... +go test ./cmd/interpres-decode/ +``` + +Add `-v` for the sub-test names, and `-race` when the change touches +concurrency. `-count=1` defeats the test cache when a result looks stale. + +## Coverage + +```sh +just test +go tool cover -func=coverage.out +just coverage-html +``` + +`just test` prints the total itself and fails below 80 percent, which is the +same floor CI enforces. The profile is `coverage.out`; the HTML map is +`coverage.html`. Both are ignored by git. + +## Benchmarks + +```sh +just bench +``` + +Benchmark on an idle machine, and compare only runs made in one process against +each other. The recipe sweeps `./...` five times with `-benchmem`. + +## Debugging the build + +```sh +go build -gcflags='-m' ./... # inlining decisions +go build -gcflags='-S' ./... # what the compiler generated +``` + +## Continuous integration + +Workflows live in `.gitea/workflows/` and run on the project's own runners. +They are written by hand rather than through `just`, but they enforce the same +set of gates, so a green `just gates` locally is the fastest way to a green +pipeline. + +| Workflow | Trigger | What it does | +|---|---|---| +| `test.yml` | push or pull request to `development` | format check, vet, modernisation, build, the test suite with the 80 percent coverage floor, then the toml-test compliance suite | +| `race.yml` | `workflow_dispatch`, by hand | the suite under the race detector; the same race gate `just gates` runs locally | +| `release.yml` | a `v*` tag | the same gates plus the race detector, then the Gitea release from the CHANGELOG section | + +## Releases + +Releases are cut by merging `development` into `main` and tagging `vX.Y.Z`. The +tag drives the release workflow: it validates the tag, runs the full gate set +including the race detector, extracts the matching `## [X.Y.Z]` section from +`CHANGELOG.md`, and publishes the release with that section as its body. A +library ships no binaries, so the release carries the notes and nothing else.