5 Commits
Author SHA1 Message Date
petrbalvin 2ae487e117 perf(encode): emit in place, pool the buffer and cache interface flags
Test / test (push) Successful in 1m33s
Grouped emission walks the entries slice in three passes instead of copying them into per-kind slices, error paths render their key path only when an error names it, inline tables are measured once because a measuring encoder no longer nests another measurement, clean string runs are written in one write, and date-times render through the single-buffer path. The output buffer comes from a sync.Pool and returns to it within a 1 MiB retention cap, the output handed to the caller as a copy; repeated marshals keep the live heap flat, verified over 500 runs. The encoder resolves Marshaler and the text interfaces through the same per-type flag cache and hint the decoder uses, and docs/ARCHITECTURE.md now lists the caches and the pool as the library's shared state.

Representative document: 177 to 141 allocations, 12 230 to 6170 bytes per operation; long document: 97 175 to 63 660 allocations, 6.97 to 2.79 MB per operation, about 7.8 to about 3.4 ms.
2026-09-20 22:15:38 +02:00
petrbalvin 9874464213 perf(decode): resolve interfaces through cached type flags
The decoder asked every value whether it implements Unmarshaler or encoding.TextUnmarshaler by boxing it into an interface and asserting, which allocated on every scalar field. A per-type flag cache answers first and an interface value is built only where the assertion can succeed; interface destinations are still asked dynamically. A monomorphic hint in front of the cache keeps the hot walk off the sync.Map probe, and it re-points at published cache entries so a miss allocates nothing.

Representative document: 233 to 167 allocations; long document typed decode: 107 674 to 63 772 allocations, about 6.1 to about 3.8 ms.
2026-09-20 22:15:23 +02:00
petrbalvin d00cbb983a perf(parse): cut allocations and validate UTF-8 in the scan
Per-statement work no longer allocates where it does not need to: parseKeyPath returns the first key component directly, absolute paths and key buffers sit on the parser, the definition maps are created on first write, arrays presize, and a string without escapes is copied out in one piece instead of built byte by byte. Repeated keys share one string across array-of-tables elements through a parser-local intern table whose lookup works on the bytes, so a repeated key costs no allocation.

UTF-8 validity is no longer a whole-input pass before the parser runs: the scan validates where it meets a multi-byte sequence, and a SyntaxError now names the line the invalid byte sits on. TestParseRejectsInvalidUTF8 covers nine positions; docs/API.md describes the new reporting.

Long document: 67 664 to 31 765 allocations per parse, about 51 to about 106 MB/s; representative document: 157 to 104 allocations.
2026-09-20 22:15:10 +02:00
petrbalvin f37c05f9a2 perf(datetime): scan the token shape and render in one pass
Date-time tokens were validated by two regular expressions and then tried against up to sixteen time.Parse layouts; the profile named the regexp backtracker among the hottest nodes, and the failed attempts allocated ParseErrors by the million. scanDateTimeShape walks the strict TOML grammar as bytes and dispatches one layout per shape, which time.Parse accepts because parsing takes a fractional second whether the layout signs it or not. The String methods build their output in a single buffer instead of concatenating Format results.
2026-09-20 22:15:10 +02:00
petrbalvin 9a3d8dd87a test(bench): measure the long document on the typed decode and marshal paths
The suite gains BenchmarkStrictDecodeLong and BenchmarkMarshalLong over the same 2000-entry document ParseLong uses, so long-input work is visible on every path, not the parse alone. docs/BENCHMARKING.md lists them.
2026-09-20 22:15:10 +02:00
12 changed files with 1130 additions and 323 deletions
+29
View File
@@ -74,6 +74,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- The module path carries the /v2 suffix the Go toolchain requires of - The module path carries the /v2 suffix the Go toolchain requires of
every major version 2 module: imports change to every major version 2 module: imports change to
`sourcedock.dev/petrbalvin/interpres/v2`. `sourcedock.dev/petrbalvin/interpres/v2`.
- Input that is not valid UTF-8 is now rejected where the parser's scan
meets the invalid byte, with a `SyntaxError` naming that line, instead of
a whole-input check that always reported line 1. Invalid input is still
rejected; the reported location is now the byte's own.
**Performance**
- Parsing is faster than in 1.1.0 while carrying the new document layer:
the suite's representative document decodes at about 79 MB/s with 104
allocations per call, and the long array-of-tables document at about
106 MB/s against 56 MB/s in 1.1.0, with allocations on that document
halved from 67 664 to 31 765. Date-time tokens are validated by a byte
scan instead of regular expressions, repeated keys share one string
across array-of-tables elements, and per-statement buffers are reused.
- Typed decoding is 12 percent faster than in 1.1.0 on the representative
document (9792 ns against 11 147 ns) with 24 percent fewer allocations
(167 against 220); interface lookups resolve through a cached per-type
flag set instead of boxing every value into an interface to ask.
- `Marshal` runs at the 1.1.0 speed while emitting the new TOML 1.1 output
form, at half the bytes per operation (6170 against 11 348 on the
representative document), and writes through a pooled output buffer with
a 1 MiB retention cap; repeated marshals keep the live heap flat.
- Two benchmarks measure the shapes that drove the work:
`BenchmarkStrictDecodeLong` and `BenchmarkMarshalLong` run the 2000-entry
document at about 3.8 ms and 3.4 ms per call, at 63 772 and 63 660
allocations.
### Fixed ### Fixed
@@ -81,6 +107,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
as `type Name string`, panicked instead of storing the value, because a as `type Name string`, panicked instead of storing the value, because a
value of the predeclared type is not assignable to a defined type and the value of the predeclared type is not assignable to a defined type and the
decoder assigned it without a conversion. decoder assigned it without a conversion.
- A top-level value the encoder could not normalise reported its path with
a leading dot, `interpres: .port: ...`; the message now reads
`interpres: port: ...`, the shape `EncodeError.Path` already used.
## [1.1.0] - 2026-09-18 ## [1.1.0] - 2026-09-18
+41
View File
@@ -127,3 +127,44 @@ func BenchmarkParseLong(b *testing.B) {
} }
} }
} }
// benchLongEntry mirrors one [[entry]] element of longDoc for the typed
// decode of the long document.
type benchLongEntry struct {
Name string `toml:"name"`
Weight int `toml:"weight"`
When time.Time `toml:"when"`
Ratio float64 `toml:"ratio"`
Tags []string `toml:"tags"`
}
type benchLongDoc struct {
Title string `toml:"title"`
Entry []benchLongEntry `toml:"entry"`
}
func BenchmarkStrictDecodeLong(b *testing.B) {
dec := NewDecoder().DisallowUnknownFields()
b.ReportAllocs()
b.SetBytes(int64(len(longDoc)))
for b.Loop() {
var doc benchLongDoc
if err := dec.Decode(longDoc, &doc); err != nil {
b.Fatal(err)
}
}
}
func BenchmarkMarshalLong(b *testing.B) {
tree, err := ParseMap(longDoc)
if err != nil {
b.Fatal(err)
}
b.ReportAllocs()
b.SetBytes(int64(len(longDoc)))
for b.Loop() {
if _, err := Marshal(tree); err != nil {
b.Fatal(err)
}
}
}
+198 -69
View File
@@ -5,8 +5,6 @@ package interpres
import ( import (
"fmt" "fmt"
"regexp"
"strconv"
"strings" "strings"
"time" "time"
) )
@@ -44,7 +42,8 @@ func (odt OffsetDateTime) String() string { return offsetString(odt.Time) }
// fractional second. TOML 1.1 makes the seconds optional, so they appear only // fractional second. TOML 1.1 makes the seconds optional, so they appear only
// when they are non-zero, and a fraction drops its trailing zeros. // when they are non-zero, and a fraction drops its trailing zeros.
func (ldt LocalDateTime) String() string { func (ldt LocalDateTime) String() string {
return ldt.Format("2006-01-02T") + clockString(ldt.Time) buf := ldt.Time.AppendFormat(make([]byte, 0, 32), "2006-01-02T")
return string(appendClock(buf, ldt.Time))
} }
// String returns the TOML-canonical rendering of the local date, e.g. // String returns the TOML-canonical rendering of the local date, e.g.
@@ -55,70 +54,187 @@ func (ld LocalDate) String() string { return ld.Format("2006-01-02") }
// or "07:32:00.5" when the time carries a fractional second. // or "07:32:00.5" when the time carries a fractional second.
func (lt LocalTime) String() string { return clockString(lt.Time) } func (lt LocalTime) String() string { return clockString(lt.Time) }
// clockString renders a time of day the way TOML writes it: the seconds appear // appendClock appends the clock part of a TOML time to buf: HH:MM, seconds
// only when the value carries them, and a fractional second drops its trailing // only when the value carries them, and a fraction with its trailing zeros
// zeros, so half a second is "00.5" and not "00.500000000". Both are the same // dropped, so half a second is ".5" and not ".500000000". Both are the same
// value either way; the shorter form is the one TOML 1.1 allows. // value either way; the shorter form is the one TOML 1.1 allows. The whole
// rendering is built in one buffer, because the encoder writes a date-time
// per entry of a large document.
func appendClock(buf []byte, t time.Time) []byte {
buf = t.AppendFormat(buf, "15:04")
if t.Second() != 0 || t.Nanosecond() != 0 {
buf = t.AppendFormat(buf, ":05")
}
if ns := t.Nanosecond(); ns > 0 {
buf = append(buf, '.')
buf = append(buf, strings.TrimRight(fmt.Sprintf("%09d", ns), "0")...)
}
return buf
}
// clockString renders a time of day the way TOML writes it.
func clockString(t time.Time) string { func clockString(t time.Time) string {
out := t.Format("15:04") return string(appendClock(make([]byte, 0, 16), t))
ns := t.Nanosecond()
if t.Second() != 0 || ns != 0 {
out += t.Format(":05")
}
if ns > 0 {
out += "." + strings.TrimRight(fmt.Sprintf("%09d", ns), "0")
}
return out
} }
// offsetString renders an offset date-time, the fourth TOML kind, in the same // offsetString renders an offset date-time, the fourth TOML kind, in the same
// shape: no zero seconds, no trailing zeros in the fraction, and the offset // shape: no zero seconds, no trailing zeros in the fraction, and the offset
// written as "Z" when it is zero. // written as "Z" when it is zero.
func offsetString(t time.Time) string { func offsetString(t time.Time) string {
return t.Format("2006-01-02T") + clockString(t) + t.Format("Z07:00") buf := t.AppendFormat(make([]byte, 0, 32), "2006-01-02T")
buf = appendClock(buf, t)
buf = t.AppendFormat(buf, "Z07:00")
return string(buf)
} }
var ( // dateTimeKind names the date-time shape a bare token has, as the scanner
offsetDateTimeLayouts = []string{ // below classifies it.
"2006-01-02T15:04:05.999999999Z07:00", type dateTimeKind int
"2006-01-02T15:04:05Z07:00",
"2006-01-02 15:04:05.999999999Z07:00", const (
"2006-01-02 15:04:05Z07:00", dateTimeNone dateTimeKind = iota
// TOML 1.1 makes the seconds optional. dateTimeOffset
"2006-01-02T15:04Z07:00", dateTimeLocal
"2006-01-02 15:04Z07:00", dateTimeDate
} dateTimeClock
localDateTimeLayouts = []string{
"2006-01-02T15:04:05.999999999",
"2006-01-02T15:04:05",
"2006-01-02 15:04:05.999999999",
"2006-01-02 15:04:05",
"2006-01-02T15:04",
"2006-01-02 15:04",
}
localTimeLayouts = []string{
"15:04:05.999999999",
"15:04:05",
"15:04",
}
) )
// dateTimeShape enforces the strict TOML grammar (two-digit components, // The layouts the time package parses each shape with. Parsing accepts a
// seconds optional since 1.1, a fraction only after seconds) that time.Parse // fractional second even when the layout does not carry one, so each shape
// would otherwise accept loosely (e.g. a single-digit hour). // needs a single layout, chosen by whether the token has seconds.
var dateTimeShape = regexp.MustCompile( const (
`^\d{4}-\d{2}-\d{2}([Tt ]\d{2}:\d{2}(:\d{2}(\.\d+)?)?([Zz]|[+-]\d{2}:\d{2})?)?$` + offsetDateTimeLayout = "2006-01-02T15:04:05Z07:00"
`|^\d{2}:\d{2}(:\d{2}(\.\d+)?)?$`, offsetClockLayout = "2006-01-02T15:04Z07:00"
localDateTimeLayout = "2006-01-02T15:04:05"
localClockLayout = "2006-01-02T15:04"
localTimeLayout = "15:04:05"
localTimeClockLayout = "15:04"
localDateOnlyLayout = "2006-01-02"
) )
// offsetBounds extracts the numeric offset of a date-time. The ABNF bounds it // scanDateTimeShape validates a bare token against the strict TOML date-time
// to 00:00 through 23:59, but time.Parse accepts values outside that range // grammar and reports which kind it is: two-digit components, seconds
// and rolls them over (for example "+00:60" becomes "+01:00"), so the bounds // optional since TOML 1.1, a fraction only after seconds, an offset only
// are enforced here. // after a time, and an offset bounded to 00:00 through 23:59. The grammar is
var offsetBounds = regexp.MustCompile(`([+-])(\d{2}):(\d{2})$`) // a fixed byte shape, so the scan is a byte walk; the regular expressions
// this replaced cost the parser measurably per token, and a shape that fails
// the scan is simply not a date-time.
func scanDateTimeShape(tok string) (kind dateTimeKind, seconds bool) {
// A local clock on its own: HH:MM[:SS[.fraction]].
if len(tok) >= 5 && tok[2] == ':' {
n, secs, ok := scanClock(tok, 0)
if !ok || n != len(tok) {
return dateTimeNone, false
}
return dateTimeClock, secs
}
// A date, optionally followed by a time and an offset.
if len(tok) < 10 || tok[4] != '-' || tok[7] != '-' {
return dateTimeNone, false
}
for _, i := range [8]int{0, 1, 2, 3, 5, 6, 8, 9} {
if !isDecDigit(tok[i]) {
return dateTimeNone, false
}
}
if len(tok) == 10 {
return dateTimeDate, false
}
if sep := tok[10]; sep != 'T' && sep != 't' && sep != ' ' {
return dateTimeNone, false
}
n, secs, ok := scanClock(tok, 11)
if !ok {
return dateTimeNone, false
}
if n == len(tok) {
return dateTimeLocal, secs
}
// The offset: Z/z, or a signed HH:MM bounded as the ABNF requires.
switch c := tok[n]; {
case c == 'Z' || c == 'z':
if n+1 != len(tok) {
return dateTimeNone, false
}
case c == '+' || c == '-':
if n+6 != len(tok) || tok[n+3] != ':' ||
!isDecDigit(tok[n+1]) || !isDecDigit(tok[n+2]) ||
!isDecDigit(tok[n+4]) || !isDecDigit(tok[n+5]) ||
tok[n+1] > '2' || (tok[n+1] == '2' && tok[n+2] > '3') ||
tok[n+4] > '5' {
return dateTimeNone, false
}
default:
return dateTimeNone, false
}
return dateTimeOffset, secs
}
// scanClock validates HH:MM[:SS[.fraction]] starting at i and returns the
// position after the clock, whether seconds were present, and whether the
// shape is valid.
func scanClock(tok string, i int) (pos int, seconds bool, ok bool) {
if i+5 > len(tok) || tok[i+2] != ':' ||
!isDecDigit(tok[i]) || !isDecDigit(tok[i+1]) ||
!isDecDigit(tok[i+3]) || !isDecDigit(tok[i+4]) {
return 0, false, false
}
i += 5
if i == len(tok) || tok[i] != ':' {
return i, false, true
}
if i+3 > len(tok) || !isDecDigit(tok[i+1]) || !isDecDigit(tok[i+2]) {
return 0, false, false
}
i += 3
if i == len(tok) || tok[i] != '.' {
return i, true, true
}
i++
digits := i
for i < len(tok) && isDecDigit(tok[i]) {
i++
}
if i == digits {
return 0, false, false
}
return i, true, true
}
// normaliseDateTimeToken rewrites the date/time separator to 'T' and the
// offset marker to 'Z', the characters the layouts above carry. A token that
// already has them is returned as it is, without a copy.
func normaliseDateTimeToken(tok string, kind dateTimeKind) string {
if kind == dateTimeDate || kind == dateTimeClock {
return tok
}
needs := false
for i := range len(tok) {
c := tok[i]
if c == 't' || c == 'z' || (c == ' ' && i == 10) {
needs = true
break
}
}
if !needs {
return tok
}
b := []byte(tok)
for i, c := range b {
switch {
case c == 't':
b[i] = 'T'
case c == 'z':
b[i] = 'Z'
case c == ' ' && i == 10:
b[i] = 'T'
}
}
return string(b)
}
// parseDateTime classifies and parses a bare token as a TOML date-time value. // parseDateTime classifies and parses a bare token as a TOML date-time value.
// It returns the decoded value (time.Time, LocalDateTime, LocalDate, or // It returns the decoded value (OffsetDateTime, LocalDateTime, LocalDate or
// LocalTime) and whether the token was a date-time at all. // LocalTime) and whether the token was a date-time at all.
func parseDateTime(tok string) (any, bool) { func parseDateTime(tok string) (any, bool) {
if tok == "" || tok[0] < '0' || tok[0] > '9' { if tok == "" || tok[0] < '0' || tok[0] > '9' {
@@ -127,36 +243,49 @@ func parseDateTime(tok string) (any, bool) {
if !strings.ContainsAny(tok, "-:") { if !strings.ContainsAny(tok, "-:") {
return nil, false return nil, false
} }
if !dateTimeShape.MatchString(tok) { kind, seconds := scanDateTimeShape(tok)
if kind == dateTimeNone {
return nil, false return nil, false
} }
if m := offsetBounds.FindStringSubmatch(tok); m != nil { norm := normaliseDateTimeToken(tok, kind)
hour, _ := strconv.Atoi(m[2]) switch kind {
minute, _ := strconv.Atoi(m[3]) case dateTimeOffset:
if hour > 23 || minute > 59 { layout := offsetClockLayout
if seconds {
layout = offsetDateTimeLayout
}
t, err := time.Parse(layout, norm)
if err != nil {
return nil, false return nil, false
} }
}
// The ABNF accepts lowercase "t"/"z"; time.Parse only matches uppercase.
norm := strings.ToUpper(tok)
for _, layout := range offsetDateTimeLayouts {
if t, err := time.Parse(layout, norm); err == nil {
return OffsetDateTime{t}, true return OffsetDateTime{t}, true
case dateTimeLocal:
layout := localClockLayout
if seconds {
layout = localDateTimeLayout
} }
t, err := time.Parse(layout, norm)
if err != nil {
return nil, false
} }
for _, layout := range localDateTimeLayouts {
if t, err := time.Parse(layout, norm); err == nil {
return LocalDateTime{t}, true return LocalDateTime{t}, true
case dateTimeDate:
t, err := time.Parse(localDateOnlyLayout, norm)
if err != nil {
return nil, false
} }
}
if t, err := time.Parse("2006-01-02", norm); err == nil {
return LocalDate{t}, true return LocalDate{t}, true
case dateTimeClock:
layout := localTimeClockLayout
if seconds {
layout = localTimeLayout
}
t, err := time.Parse(layout, norm)
if err != nil {
return nil, false
} }
for _, layout := range localTimeLayouts {
if t, err := time.Parse(layout, norm); err == nil {
return LocalTime{t}, true return LocalTime{t}, true
} }
}
return nil, false return nil, false
} }
+113 -11
View File
@@ -10,6 +10,7 @@ import (
"slices" "slices"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"time" "time"
) )
@@ -22,6 +23,97 @@ func newDecoder() *decoder { return &decoder{} }
var timeType = reflect.TypeFor[time.Time]() var timeType = reflect.TypeFor[time.Time]()
var (
unmarshalerType = reflect.TypeFor[Unmarshaler]()
textUnmarshalerType = reflect.TypeFor[encoding.TextUnmarshaler]()
)
// The per-type flags record which interface lookups a decode into that type
// can succeed at, so the hot path consults the cache instead of boxing every
// value into an interface to ask. The bits name the receiver the method is
// found on: the value itself, or its address.
const (
flagUnmarshaler uint8 = 1 << iota
flagAddrUnmarshaler
flagTextUnmarshaler
flagAddrTextUnmarshaler
)
// typeFlagCache holds one flag entry per destination type. A set is immutable
// once published, the same trade-off structSchemaCache makes; the cache grows
// with the number of distinct types decoded, never per document. The hint
// below re-points at these published entries, so a hot lookup allocates
// nothing.
var typeFlagCache sync.Map // reflect.Type -> *flagHintEntry
// flagHintEntry pairs a type with its cached flags for the monomorphic hint
// below. Both caches share the entry shape.
type flagHintEntry struct {
typ reflect.Type
flags uint8
}
// typeFlagHint remembers the entry resolved last, because a decode walks one
// type across consecutive fields and elements. A lost race loses only the
// hint: every value it can hold came from the cache.
var typeFlagHint atomic.Pointer[flagHintEntry]
func typeFlags(t reflect.Type) uint8 {
if e := typeFlagHint.Load(); e != nil && e.typ == t {
return e.flags
}
if v, ok := typeFlagCache.Load(t); ok {
entry := v.(*flagHintEntry)
typeFlagHint.Store(entry)
return entry.flags
}
var f uint8
if t.Implements(unmarshalerType) {
f |= flagUnmarshaler
}
pt := reflect.PointerTo(t)
if pt.Implements(unmarshalerType) {
f |= flagAddrUnmarshaler
}
// The date-time types are excluded from the text path: they carry
// time.Time's UnmarshalText through an embedded field while their only
// accepted form is a bare timestamp.
if !isDateTimeType(t) {
if t.Implements(textUnmarshalerType) {
f |= flagTextUnmarshaler
}
if pt.Implements(textUnmarshalerType) {
f |= flagAddrTextUnmarshaler
}
}
actual, _ := typeFlagCache.LoadOrStore(t, &flagHintEntry{t, f})
published := actual.(*flagHintEntry)
typeFlagHint.Store(published)
return published.flags
}
// unmarshalerOf resolves the Unmarshaler for dst through the flag cache, so
// an interface value is built only where the cache says the assertion can
// succeed. An interface destination is asked dynamically, because the value
// it will hold may implement the interface even when the interface type
// itself does not.
func unmarshalerOf(dst reflect.Value) (Unmarshaler, bool) {
if dst.Kind() == reflect.Interface {
u, ok := dst.Interface().(Unmarshaler)
return u, ok
}
f := typeFlags(dst.Type())
if f&flagUnmarshaler != 0 {
u, ok := dst.Interface().(Unmarshaler)
return u, ok
}
if f&flagAddrUnmarshaler != 0 && dst.CanAddr() {
u, ok := dst.Addr().Interface().(Unmarshaler)
return u, ok
}
return nil, false
}
func (d *decoder) decode(tree map[string]any, v any) error { func (d *decoder) decode(tree map[string]any, v any) error {
rv := reflect.ValueOf(v) rv := reflect.ValueOf(v)
if rv.Kind() != reflect.Pointer || rv.IsNil() { if rv.Kind() != reflect.Pointer || rv.IsNil() {
@@ -109,22 +201,24 @@ func (d *decoder) assign(data any, dst reflect.Value) error {
} }
} }
// textUnmarshalerOf finds the encoding.TextUnmarshaler for dst: on the value // textUnmarshalerOf is the same resolution for encoding.TextUnmarshaler,
// itself, or on its address, so a pointer-receiver UnmarshalText is invoked on // with the date-time types excluded for the reason typeFlags records.
// an addressable struct field. The TOML date-time types are excluded, because
// they carry time.Time's UnmarshalText through an embedded field while their
// only accepted form is a bare timestamp.
func textUnmarshalerOf(dst reflect.Value) (encoding.TextUnmarshaler, bool) { func textUnmarshalerOf(dst reflect.Value) (encoding.TextUnmarshaler, bool) {
if !dst.CanInterface() || isDateTimeType(dst.Type()) { if !dst.CanInterface() || isDateTimeType(dst.Type()) {
return nil, false return nil, false
} }
if u, ok := dst.Interface().(encoding.TextUnmarshaler); ok { if dst.Kind() == reflect.Interface {
return u, true tu, ok := dst.Interface().(encoding.TextUnmarshaler)
return tu, ok
} }
if dst.CanAddr() { f := typeFlags(dst.Type())
if u, ok := dst.Addr().Interface().(encoding.TextUnmarshaler); ok { if f&flagTextUnmarshaler != 0 {
return u, true tu, ok := dst.Interface().(encoding.TextUnmarshaler)
return tu, ok
} }
if f&flagAddrTextUnmarshaler != 0 && dst.CanAddr() {
tu, ok := dst.Addr().Interface().(encoding.TextUnmarshaler)
return tu, ok
} }
return nil, false return nil, false
} }
@@ -147,6 +241,9 @@ func (d *decoder) assignStruct(tbl map[string]any, dst reflect.Value) error {
// deterministically: the smallest one. // deterministically: the smallest one.
unknown := "" unknown := ""
for key := range tbl { for key := range tbl {
if _, ok := schema.byName[key]; ok {
continue
}
if _, ok := schema.byName[strings.ToLower(key)]; ok { if _, ok := schema.byName[strings.ToLower(key)]; ok {
continue continue
} }
@@ -159,7 +256,12 @@ func (d *decoder) assignStruct(tbl map[string]any, dst reflect.Value) error {
} }
} }
for key, val := range tbl { for key, val := range tbl {
field, ok := schema.byName[strings.ToLower(key)] // A key that is already lowercase, which document keys usually are,
// hits the map directly; only a miss pays for the case fold.
field, ok := schema.byName[key]
if !ok {
field, ok = schema.byName[strings.ToLower(key)]
}
if !ok { if !ok {
if schema.embedMaps != nil { if schema.embedMaps != nil {
// Leftover keys land in an untagged embedded map, the inverse // Leftover keys land in an untagged embedded map, the inverse
+25 -6
View File
@@ -24,19 +24,38 @@ func TestSyntaxErrorMessage(t *testing.T) {
} }
func TestParseRejectsInvalidUTF8(t *testing.T) { func TestParseRejectsInvalidUTF8(t *testing.T) {
_, err := ParseMap([]byte("v = \"\xff\"\n")) // The scan validates UTF-8 where it meets the byte, so the reported line
// is the invalid byte's own, wherever in the document it sits.
cases := []struct {
name string
doc string
line int
}{
{"in a basic string", "v = \"\xff\"\n", 1},
{"in a literal string", "v = '\xff'\n", 1},
{"in a multiline string", "v = \"\"\"\n\xff\"\"\"\n", 2},
{"in a comment", "v = 1\n# caf\xe9\xff\n", 2},
{"in a bare key", "va\xfflue = 1\n", 1},
{"as a statement", "\xff = 1\n", 1},
{"in a bare value", "v = \xff1\n", 1},
{"after a value", "v = 1 \xff\n", 1},
{"after the first line", "a = 1\nb = \"\xff\"\n", 2},
}
for _, c := range cases {
_, err := ParseMap([]byte(c.doc))
if err == nil { if err == nil {
t.Fatal("expected a UTF-8 validation error") t.Fatalf("%s: expected a UTF-8 validation error", c.name)
} }
se, ok := err.(*SyntaxError) se, ok := err.(*SyntaxError)
if !ok { if !ok {
t.Fatalf("err is %T, want *SyntaxError", err) t.Fatalf("%s: err is %T, want *SyntaxError", c.name, err)
} }
if !strings.Contains(se.Msg, "UTF-8") { if !strings.Contains(se.Msg, "UTF-8") {
t.Errorf("Msg = %q, want it to mention UTF-8", se.Msg) t.Errorf("%s: Msg = %q, want it to mention UTF-8", c.name, se.Msg)
}
if se.Line != c.line {
t.Errorf("%s: Line = %d, want %d", c.name, se.Line, c.line)
} }
if se.Line != 1 {
t.Errorf("Line = %d, want 1", se.Line)
} }
} }
+2 -1
View File
@@ -19,7 +19,8 @@ Decodes a TOML document into a [Document](#documents): the values, the order the
keys were written in, whether a table was written inline, and the comments. keys were written in, whether a table was written inline, and the comments.
The values follow the mapping in the [Decoding](#decoding) section below. The values follow the mapping in the [Decoding](#decoding) section below.
Returns `*SyntaxError` on a malformed document. Input that is not valid UTF-8 Returns `*SyntaxError` on a malformed document. Input that is not valid UTF-8
is rejected before the parser runs. Equivalent to is rejected with a `SyntaxError` naming the line where the invalid byte
appears, because validity is checked during the scan. Equivalent to
`ParseContext(context.Background(), data)`. `ParseContext(context.Background(), data)`.
```go ```go
+16 -8
View File
@@ -102,15 +102,23 @@ sequenceDiagram
setter methods are not, and must finish before the value is shared. setter methods are not, and must finish before the value is shared.
- The parser is allocated per `ParseContext` call; the parser itself caches - The parser is allocated per `ParseContext` call; the parser itself caches
nothing between documents. nothing between documents.
- The one piece of shared state is the struct-schema cache in `decode.go`: a - The shared state is a set of caches and pools whose entries are immutable
`sync.Map` keyed by `reflect.Type`, holding the flattened field layout the once published, each growing with the number of distinct types rather than
decoder and the encoder both consult. A schema is immutable once published, with document size: the struct-schema cache in `decode.go` (a `sync.Map`
so concurrent callers only race to build an identical value, the same keyed on `reflect.Type`, holding the flattened field layout the decoder and
trade-off `encoding/json`'s field cache makes. The cache grows with the the encoder both consult), the per-type interface flag caches in `decode.go`
number of distinct struct types, never with document size. and `encode.go` (recording where `Marshaler`, `Unmarshaler` and the text
interfaces can be found, so a walk builds an interface value only where the
assertion can succeed), each fronted by a monomorphic hint holding the type
resolved last, and the encoder's output-buffer pool in `encode.go`
(`sync.Pool`, buffers returned to it only within a 1 MiB retention cap). A
published schema or flag set never mutates, so concurrent callers only race
to build an identical value, the same trade-off `encoding/json`'s field
cache makes.
- The date-time wrappers are values, not pointers, and are immutable in use. - The date-time wrappers are values, not pointers, and are immutable in use.
- Nothing in the library starts goroutines; apart from the schema cache above, - Nothing in the library starts goroutines; apart from the caches and the pool
which never mutates a published entry, there is no shared mutable state. above, which never mutate a published entry, there is no shared mutable
state.
## Dependencies ## Dependencies
+5 -3
View File
@@ -9,10 +9,12 @@ The benchmarks live in `bench_test.go`, next to the code they measure:
| Benchmark | What it measures | | Benchmark | What it measures |
|---|---| |---|---|
| `BenchmarkParse` | `Parse` over a representative configuration document | | `BenchmarkParse` | `ParseMap` over a representative configuration document |
| `BenchmarkMarshal` | `Marshal` of the tree `Parse` produced from the same document | | `BenchmarkMarshal` | `Marshal` of the tree `ParseMap` produced from the same document |
| `BenchmarkStrictDecode` | `Decode` into a struct under `DisallowUnknownFields` | | `BenchmarkStrictDecode` | `Decode` into a struct under `DisallowUnknownFields` |
| `BenchmarkParseLong` | `Parse` over a generated document with about 2000 array-of-tables entries | | `BenchmarkParseLong` | `ParseMap` over a generated document with about 2000 array-of-tables entries |
| `BenchmarkStrictDecodeLong` | `Decode` into a typed document under `DisallowUnknownFields`, over the same long document |
| `BenchmarkMarshalLong` | `Marshal` of the tree `ParseMap` produced from the long document |
## Running ## Running
+300 -106
View File
@@ -15,6 +15,8 @@ import (
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
"sync"
"sync/atomic"
"time" "time"
"unicode/utf8" "unicode/utf8"
) )
@@ -27,8 +29,63 @@ var (
timeGoType = reflect.TypeFor[time.Time]() timeGoType = reflect.TypeFor[time.Time]()
durationType = reflect.TypeFor[time.Duration]() durationType = reflect.TypeFor[time.Duration]()
textMarshalerType = reflect.TypeFor[encoding.TextMarshaler]() textMarshalerType = reflect.TypeFor[encoding.TextMarshaler]()
marshalerIfaceType = reflect.TypeFor[Marshaler]()
) )
// The encoder's per-type flags mirror the decoder's cache: an interface value
// is built only where the cache says the assertion can succeed. The bits name
// the receiver the method is found on.
const (
encFlagMarshaler uint8 = 1 << iota
encFlagAddrMarshaler
encFlagTextMarshaler
encFlagAddrTextMarshaler
)
// encTypeFlagCache holds one flag entry per value type the encoder walks,
// immutable once published, the same trade-off structSchemaCache makes. The
// hint below re-points at these published entries, so a hot lookup allocates
// nothing.
var encTypeFlagCache sync.Map // reflect.Type -> *flagHintEntry
// encTypeFlagHint is the encoder's monomorphic hint, for the same reason the
// decoder's one exists: an encode walk repeats one type across consecutive
// fields and elements, and the hint answers without a cache probe.
var encTypeFlagHint atomic.Pointer[flagHintEntry]
func encTypeFlags(t reflect.Type) uint8 {
if e := encTypeFlagHint.Load(); e != nil && e.typ == t {
return e.flags
}
if v, ok := encTypeFlagCache.Load(t); ok {
entry := v.(*flagHintEntry)
encTypeFlagHint.Store(entry)
return entry.flags
}
var f uint8
if t.Implements(marshalerIfaceType) {
f |= encFlagMarshaler
}
pt := reflect.PointerTo(t)
if pt.Implements(marshalerIfaceType) {
f |= encFlagAddrMarshaler
}
// The date-time types are excluded from the text path for the reason
// textValue records; no caller reaches textMarshalerOf for them.
if !isDateTimeType(t) {
if t.Implements(textMarshalerType) {
f |= encFlagTextMarshaler
}
if pt.Implements(textMarshalerType) {
f |= encFlagAddrTextMarshaler
}
}
actual, _ := encTypeFlagCache.LoadOrStore(t, &flagHintEntry{t, f})
published := actual.(*flagHintEntry)
encTypeFlagHint.Store(published)
return published.flags
}
// inlineLimit is the column past which an inline table is written across // inlineLimit is the column past which an inline table is written across
// lines. TOML 1.1 lets an inline table carry newlines and a trailing comma, so // lines. TOML 1.1 lets an inline table carry newlines and a trailing comma, so
// a long one stays readable instead of running off the line. // a long one stays readable instead of running off the line.
@@ -38,10 +95,26 @@ const inlineLimit = 100
// form it renders is always the single-line one. // form it renders is always the single-line one.
const noInlineBreak = 1 << 30 const noInlineBreak = 1 << 30
// encoderBufRetention is the largest output buffer put back into the pool.
// The cap is the pool's memory rule: repeated marshals of a document within it
// reuse one buffer and keep the heap flat, while a document larger than the
// cap pins nothing per processor once written.
const encoderBufRetention = 1 << 20
// encoderBufPool holds output buffers between Marshal calls.
var encoderBufPool = sync.Pool{New: func() any { return new(bytes.Buffer) }}
// getEncoderBuf takes a cleared output buffer from the pool.
func getEncoderBuf() *bytes.Buffer {
b := encoderBufPool.Get().(*bytes.Buffer)
b.Reset()
return b
}
// encoder produces a TOML document from a Go value via a small intermediate // encoder produces a TOML document from a Go value via a small intermediate
// representation that preserves the order in which fields were declared. // representation that preserves the order in which fields were declared.
type encoder struct { type encoder struct {
buf bytes.Buffer buf *bytes.Buffer
ctx context.Context ctx context.Context
opts Encoder opts Encoder
@@ -54,15 +127,32 @@ type encoder struct {
limit int limit int
} }
func newEncoder() *encoder { return &encoder{limit: inlineLimit} } func newEncoder() *encoder {
e := &encoder{limit: inlineLimit}
e.buf = getEncoderBuf()
return e
}
// flat returns an encoder that measures a value by rendering it on one line, // flat returns an encoder that measures a value by rendering it on one line,
// so a caller can decide which form to write before writing it. // so a caller can decide which form to write before writing it.
func (e *encoder) flat() *encoder { func (e *encoder) flat() *encoder {
return &encoder{ctx: e.ctx, opts: e.opts, limit: noInlineBreak} f := &encoder{ctx: e.ctx, opts: e.opts, limit: noInlineBreak}
f.buf = getEncoderBuf()
return f
} }
func (e *encoder) bytes() []byte { return e.buf.Bytes() } // release returns the encoder's output buffer to the pool, within the
// retention cap. It is safe to call twice; the buffer travels only once.
func (e *encoder) release() {
if e.buf == nil {
return
}
b := e.buf
e.buf = nil
if b.Cap() <= encoderBufRetention {
encoderBufPool.Put(b)
}
}
// column reports how many bytes the current line already holds, so a form can // column reports how many bytes the current line already holds, so a form can
// be measured against the limit before it is written. // be measured against the limit before it is written.
@@ -106,13 +196,14 @@ func (e *encoder) encode(v any) error {
rv = rv.Elem() rv = rv.Elem()
} }
doc := &tomlDoc{ctx: e.ctx, opts: e.opts} doc := &tomlDoc{ctx: e.ctx, opts: e.opts}
root := encPath{}
switch rv.Kind() { switch rv.Kind() {
case reflect.Struct: case reflect.Struct:
if err := buildStructDoc(rv, doc, ""); err != nil { if err := buildStructDoc(rv, doc, root); err != nil {
return err return err
} }
case reflect.Map: case reflect.Map:
if err := buildMapDoc(rv, doc, ""); err != nil { if err := buildMapDoc(rv, doc, root); err != nil {
return err return err
} }
default: default:
@@ -133,15 +224,20 @@ const (
) )
// entry is one binding in a tomlDoc. entries live in a single slice in the // entry is one binding in a tomlDoc. entries live in a single slice in the
// order they were added; emission either walks that order directly // order they were added; emission walks that order directly, either as it is
// (Encoder with GroupByKind(false)) or partitions by kind first // (Encoder with GroupByKind(false)) or in kind-grouped passes over the same
// (Encoder with GroupByKind(true), the default). // slice (the default).
type entry struct { type entry struct {
kind entryKind kind entryKind
key string key string
val any // entryScalar val any // entryScalar
doc *tomlDoc // entryTable doc *tomlDoc // entryTable
docs []*tomlDoc docs []*tomlDoc
// emitted records that the grouped emission wrote this table inline, so
// the header pass that follows skips it. The representation is built
// fresh per Marshal call.
emitted bool
} }
// tomlDoc holds the entries of one TOML table in declaration order. // tomlDoc holds the entries of one TOML table in declaration order.
@@ -170,26 +266,62 @@ func (d *tomlDoc) addArray(key string, subs []*tomlDoc) {
d.entries = append(d.entries, entry{kind: entryArray, key: key, docs: subs}) d.entries = append(d.entries, entry{kind: entryArray, key: key, docs: subs})
} }
// partitionedEntries returns the entries grouped by kind, preserving each // --- error paths -----------------------------------------------------------
// group's relative order. The only allocation is the three slice headers.
func (d *tomlDoc) partitionedEntries() (scalars []entry, tables []entry, arrays []entry) { // encPath names a value the way an error message needs it, "server.ports[2]",
for _, e := range d.entries { // without building the string unless an error actually asks for one. A zero
switch e.kind { // encPath is the document root. The chain is stack-allocated: a segment holds
case entryScalar: // a pointer to its parent's frame-local value, and the rendered string exists
scalars = append(scalars, e) // only while an error is being built.
case entryTable: type encPath struct {
tables = append(tables, e) parent *encPath
case entryArray: name string
arrays = append(arrays, e) index int
}
// key returns the child path of a named key.
func (p *encPath) key(name string) encPath {
return encPath{parent: p, name: name, index: -1}
}
// elem returns the child path of an array element by index.
func (p *encPath) elem(i int) encPath {
return encPath{parent: p, index: i}
}
// String renders the path root first: keys join with dots and an array
// element carries its bracketed index, so the third port under server reads
// "server.ports[2]".
func (p encPath) String() string {
var parts []string
for s := &p; s != nil; s = s.parent {
switch {
case s.name == "" && s.parent == nil:
// The root: nothing to write.
case s.name == "":
parts = append(parts, "["+strconv.Itoa(s.index)+"]")
default:
parts = append(parts, s.name)
} }
} }
return var b strings.Builder
for _, part := range slices.Backward(parts) {
if strings.HasPrefix(part, "[") {
b.WriteString(part)
continue
}
if b.Len() > 0 {
b.WriteByte('.')
}
b.WriteString(part)
}
return b.String()
} }
// --- reflection walk: struct --------------------------------------------- // --- reflection walk: struct ---------------------------------------------
func buildStructDoc(v reflect.Value, doc *tomlDoc, ctx string) error { func buildStructDoc(v reflect.Value, doc *tomlDoc, path encPath) error {
return walkStructDoc(v, doc, ctx, nil, cachedStructSchema(v.Type())) return walkStructDoc(v, doc, path, nil, cachedStructSchema(v.Type()))
} }
// walkStructDoc emits the fields of v into doc. prefix is v's index path from // walkStructDoc emits the fields of v into doc. prefix is v's index path from
@@ -198,8 +330,11 @@ func buildStructDoc(v reflect.Value, doc *tomlDoc, ctx string) error {
// decoder's rule: the shallower field wins, the later declaration at equal // decoder's rule: the shallower field wins, the later declaration at equal
// depth. A field another field shadows is skipped, because emitting both // depth. A field another field shadows is skipped, because emitting both
// would duplicate the key and the output would not re-parse. // would duplicate the key and the output would not re-parse.
func walkStructDoc(v reflect.Value, doc *tomlDoc, ctx string, prefix []int, schema structSchema) error { func walkStructDoc(v reflect.Value, doc *tomlDoc, path encPath, prefix []int, schema structSchema) error {
t := v.Type() t := v.Type()
if cap(doc.entries) == 0 {
doc.entries = make([]entry, 0, t.NumField())
}
for i := range t.NumField() { for i := range t.NumField() {
if i%ctxCheckInterval == 0 { if i%ctxCheckInterval == 0 {
if err := doc.checkCtx(); err != nil { if err := doc.checkCtx(); err != nil {
@@ -210,7 +345,7 @@ func walkStructDoc(v reflect.Value, doc *tomlDoc, ctx string, prefix []int, sche
if f.PkgPath != "" { if f.PkgPath != "" {
continue continue
} }
path := append(append([]int{}, prefix...), i) fpath := append(append([]int{}, prefix...), i)
if f.Anonymous { if f.Anonymous {
tag, _ := f.Tag.Lookup("toml") tag, _ := f.Tag.Lookup("toml")
if tag == "-" { if tag == "-" {
@@ -225,20 +360,18 @@ func walkStructDoc(v reflect.Value, doc *tomlDoc, ctx string, prefix []int, sche
case reflect.Struct: case reflect.Struct:
if isScalarStruct(fv.Type()) { if isScalarStruct(fv.Type()) {
name := strings.ToLower(f.Name) name := strings.ToLower(f.Name)
if !schema.ownsKey(name, path) { if !schema.ownsKey(name, fpath) {
continue continue
} }
if err := doc.appendScalar(name, fv.Interface(), ctx); err != nil { doc.addScalar(name, fv.Interface())
return err
}
continue continue
} }
if err := walkStructDoc(fv, doc, ctx, path, schema); err != nil { if err := walkStructDoc(fv, doc, path, fpath, schema); err != nil {
return err return err
} }
continue continue
case reflect.Map: case reflect.Map:
if err := buildMapDoc(fv, doc, ctx); err != nil { if err := buildMapDoc(fv, doc, path); err != nil {
return err return err
} }
continue continue
@@ -249,13 +382,13 @@ func walkStructDoc(v reflect.Value, doc *tomlDoc, ctx string, prefix []int, sche
if name == "-" { if name == "-" {
continue continue
} }
if !schema.ownsKey(strings.ToLower(name), path) { if !schema.ownsKey(strings.ToLower(name), fpath) {
continue continue
} }
if fieldOmitted(f, v.Field(i)) { if fieldOmitted(f, v.Field(i)) {
continue continue
} }
if err := addField(doc, name, v.Field(i), ctx); err != nil { if err := addField(doc, name, v.Field(i), path); err != nil {
return err return err
} }
} }
@@ -322,10 +455,13 @@ func fieldName(f reflect.StructField) string {
// --- reflection walk: map ------------------------------------------------ // --- reflection walk: map ------------------------------------------------
func buildMapDoc(v reflect.Value, doc *tomlDoc, ctx string) error { func buildMapDoc(v reflect.Value, doc *tomlDoc, path encPath) error {
if v.Type().Key().Kind() != reflect.String { if v.Type().Key().Kind() != reflect.String {
return fmt.Errorf("interpres: map key must be string, got %s", v.Type().Key()) return fmt.Errorf("interpres: map key must be string, got %s", v.Type().Key())
} }
if cap(doc.entries) == 0 {
doc.entries = make([]entry, 0, v.Len())
}
keys := v.MapKeys() keys := v.MapKeys()
slices.SortFunc(keys, func(a, b reflect.Value) int { slices.SortFunc(keys, func(a, b reflect.Value) int {
return strings.Compare(a.String(), b.String()) return strings.Compare(a.String(), b.String())
@@ -336,7 +472,7 @@ func buildMapDoc(v reflect.Value, doc *tomlDoc, ctx string) error {
return err return err
} }
} }
if err := addField(doc, k.String(), v.MapIndex(k), ctx); err != nil { if err := addField(doc, k.String(), v.MapIndex(k), path); err != nil {
return err return err
} }
} }
@@ -351,14 +487,14 @@ func buildMapDoc(v reflect.Value, doc *tomlDoc, ctx string) error {
// contract violation. // contract violation.
var errNilMarshalTOML = errors.New("MarshalTOML returned a nil value") var errNilMarshalTOML = errors.New("MarshalTOML returned a nil value")
func addField(doc *tomlDoc, name string, v reflect.Value, ctx string) error { func addField(doc *tomlDoc, name string, v reflect.Value, path encPath) error {
if m, ok := marshalerOf(v); ok { if m, ok := marshalerOf(v); ok {
mv, err := m.MarshalTOML() mv, err := m.MarshalTOML()
if err != nil { if err != nil {
return &EncodeError{Path: joinKey(ctx, name), Err: err} return &EncodeError{Path: path.key(name).String(), Err: err}
} }
if mv == nil { if mv == nil {
return &EncodeError{Path: joinKey(ctx, name), Err: errNilMarshalTOML} return &EncodeError{Path: path.key(name).String(), Err: errNilMarshalTOML}
} }
v = reflect.ValueOf(mv) v = reflect.ValueOf(mv)
} }
@@ -366,10 +502,11 @@ func addField(doc *tomlDoc, name string, v reflect.Value, ctx string) error {
// a scalar kind or a struct. // a scalar kind or a struct.
s, isText, err := textValue(v) s, isText, err := textValue(v)
if err != nil { if err != nil {
return &EncodeError{Path: joinKey(ctx, name), Err: err} return &EncodeError{Path: path.key(name).String(), Err: err}
} }
if isText { if isText {
return doc.appendScalar(name, s, ctx) doc.addScalar(name, s)
return nil
} }
v = followPtr(v) v = followPtr(v)
if !v.IsValid() { if !v.IsValid() {
@@ -384,37 +521,33 @@ func addField(doc *tomlDoc, name string, v reflect.Value, ctx string) error {
switch v.Kind() { switch v.Kind() {
case reflect.Struct: case reflect.Struct:
if isScalarStruct(v.Type()) { if isScalarStruct(v.Type()) {
return doc.appendScalar(name, v.Interface(), ctx) doc.addScalar(name, v.Interface())
return nil
} }
return addSubTable(doc, name, v, ctx) return addSubTable(doc, name, v, path)
case reflect.Map: case reflect.Map:
return addSubTable(doc, name, v, ctx) return addSubTable(doc, name, v, path)
case reflect.Slice, reflect.Array: case reflect.Slice, reflect.Array:
return addArrayValue(doc, name, v, ctx) return addArrayValue(doc, name, v, path)
default: default:
val, err := normaliseValue(v) val, err := normaliseValue(v)
if err != nil { if err != nil {
return fmt.Errorf("interpres: %s.%s: %w", ctx, name, err) return fmt.Errorf("interpres: %s: %w", path.key(name), err)
} }
return doc.appendScalar(name, val, ctx) doc.addScalar(name, val)
}
}
// appendScalar wraps addScalar with a uniform error path.
func (d *tomlDoc) appendScalar(name string, val any, ctx string) error {
d.addScalar(name, val)
return nil return nil
}
} }
func addSubTable(doc *tomlDoc, name string, v reflect.Value, ctx string) error { func addSubTable(doc *tomlDoc, name string, v reflect.Value, path encPath) error {
sub := &tomlDoc{ctx: doc.ctx, opts: doc.opts} sub := &tomlDoc{ctx: doc.ctx, opts: doc.opts}
switch v.Kind() { switch v.Kind() {
case reflect.Struct: case reflect.Struct:
if err := buildStructDoc(v, sub, joinKey(ctx, name)); err != nil { if err := buildStructDoc(v, sub, path.key(name)); err != nil {
return err return err
} }
case reflect.Map: case reflect.Map:
if err := buildMapDoc(v, sub, joinKey(ctx, name)); err != nil { if err := buildMapDoc(v, sub, path.key(name)); err != nil {
return err return err
} }
} }
@@ -422,7 +555,7 @@ func addSubTable(doc *tomlDoc, name string, v reflect.Value, ctx string) error {
return nil return nil
} }
func addArrayValue(doc *tomlDoc, name string, v reflect.Value, ctx string) error { func addArrayValue(doc *tomlDoc, name string, v reflect.Value, path encPath) error {
if v.Kind() == reflect.Slice && v.IsNil() { if v.Kind() == reflect.Slice && v.IsNil() {
// A nil slice has no explicit representation in TOML, so it is skipped. // A nil slice has no explicit representation in TOML, so it is skipped.
return nil return nil
@@ -436,9 +569,14 @@ func addArrayValue(doc *tomlDoc, name string, v reflect.Value, ctx string) error
if doc.opts.omitEmptyArrays { if doc.opts.omitEmptyArrays {
return nil return nil
} }
return doc.appendScalar(name, []any{}, ctx) doc.addScalar(name, []any{})
return nil
} }
// The element paths hang off this one; they render only when an error
// names them.
apath := path.key(name)
// Every element is resolved through MarshalTOML first, so an element that // Every element is resolved through MarshalTOML first, so an element that
// renders itself as a scalar, a table or a value array is classified by // renders itself as a scalar, a table or a value array is classified by
// what it produces rather than by its Go kind, and its method runs once. // what it produces rather than by its Go kind, and its method runs once.
@@ -449,7 +587,7 @@ func addArrayValue(doc *tomlDoc, name string, v reflect.Value, ctx string) error
return err return err
} }
} }
ev, err := resolveElement(v.Index(i), fmt.Sprintf("%s[%d]", joinKey(ctx, name), i)) ev, err := resolveElement(v.Index(i), apath.elem(i))
if err != nil { if err != nil {
return err return err
} }
@@ -485,17 +623,17 @@ func addArrayValue(doc *tomlDoc, name string, v reflect.Value, ctx string) error
switch ev.Kind() { switch ev.Kind() {
case reflect.Struct: case reflect.Struct:
if isScalarStruct(ev.Type()) { if isScalarStruct(ev.Type()) {
return &EncodeError{Path: fmt.Sprintf("%s[%d]", joinKey(ctx, name), i), Err: errors.New("heterogeneous array contains scalar")} return &EncodeError{Path: apath.elem(i).String(), Err: errors.New("heterogeneous array contains scalar")}
} }
if err := buildStructDoc(ev, sub, joinKey(ctx, fmt.Sprintf("%s[%d]", name, i))); err != nil { if err := buildStructDoc(ev, sub, apath.elem(i)); err != nil {
return err return err
} }
case reflect.Map: case reflect.Map:
if err := buildMapDoc(ev, sub, joinKey(ctx, fmt.Sprintf("%s[%d]", name, i))); err != nil { if err := buildMapDoc(ev, sub, apath.elem(i)); err != nil {
return err return err
} }
default: default:
return &EncodeError{Path: joinKey(ctx, name), Err: errors.New("heterogeneous array, expected table")} return &EncodeError{Path: apath.elem(i).String(), Err: errors.New("heterogeneous array, expected table")}
} }
subs[i] = sub subs[i] = sub
} }
@@ -514,27 +652,37 @@ func addArrayValue(doc *tomlDoc, name string, v reflect.Value, ctx string) error
} }
val, err := normaliseValue(ev) val, err := normaliseValue(ev)
if err != nil { if err != nil {
return &EncodeError{Path: fmt.Sprintf("%s[%d]", joinKey(ctx, name), i), Err: err} return &EncodeError{Path: apath.elem(i).String(), Err: err}
} }
items[i] = val items[i] = val
} }
return doc.appendScalar(name, items, ctx) doc.addScalar(name, items)
return nil
} }
// marshalerOf finds the Marshaler a value carries: on the value itself, or on // marshalerOf finds the Marshaler a value carries: on the value itself, or on
// its address, so a pointer-receiver MarshalTOML is found on an addressable // its address, so a pointer-receiver MarshalTOML is found on an addressable
// struct field or slice element, exactly as textMarshalerOf finds MarshalText. // struct field or slice element, exactly as textMarshalerOf finds MarshalText.
// The lookup consults the per-type flag cache, so an interface value is built
// only where the assertion can succeed; an interface-typed value is asked
// dynamically, because what it holds may implement the interface when the
// interface type itself does not.
func marshalerOf(v reflect.Value) (Marshaler, bool) { func marshalerOf(v reflect.Value) (Marshaler, bool) {
if !v.CanInterface() { if !v.CanInterface() {
return nil, false return nil, false
} }
if m, ok := v.Interface().(Marshaler); ok { if v.Kind() == reflect.Interface {
return m, true m, ok := v.Interface().(Marshaler)
return m, ok
} }
if v.CanAddr() { f := encTypeFlags(v.Type())
if m, ok := v.Addr().Interface().(Marshaler); ok { if f&encFlagMarshaler != 0 {
return m, true m, ok := v.Interface().(Marshaler)
return m, ok
} }
if f&encFlagAddrMarshaler != 0 && v.CanAddr() {
m, ok := v.Addr().Interface().(Marshaler)
return m, ok
} }
return nil, false return nil, false
} }
@@ -542,10 +690,10 @@ func marshalerOf(v reflect.Value) (Marshaler, bool) {
// resolveElement looks through pointers and runs MarshalTOML, so an array // resolveElement looks through pointers and runs MarshalTOML, so an array
// element is classified by what its method produces. path names the element, // element is classified by what its method produces. path names the element,
// for the errors the method can raise. // for the errors the method can raise.
func resolveElement(v reflect.Value, path string) (reflect.Value, error) { func resolveElement(v reflect.Value, path encPath) (reflect.Value, error) {
ev := followPtr(v) ev := followPtr(v)
if !ev.IsValid() { if !ev.IsValid() {
return ev, &EncodeError{Path: path, Err: errors.New("nil element")} return ev, &EncodeError{Path: path.String(), Err: errors.New("nil element")}
} }
m, ok := marshalerOf(ev) m, ok := marshalerOf(ev)
if !ok { if !ok {
@@ -553,14 +701,14 @@ func resolveElement(v reflect.Value, path string) (reflect.Value, error) {
} }
mv, err := m.MarshalTOML() mv, err := m.MarshalTOML()
if err != nil { if err != nil {
return reflect.Value{}, &EncodeError{Path: path, Err: err} return reflect.Value{}, &EncodeError{Path: path.String(), Err: err}
} }
if mv == nil { if mv == nil {
return reflect.Value{}, &EncodeError{Path: path, Err: errNilMarshalTOML} return reflect.Value{}, &EncodeError{Path: path.String(), Err: errNilMarshalTOML}
} }
ev = followPtr(reflect.ValueOf(mv)) ev = followPtr(reflect.ValueOf(mv))
if !ev.IsValid() { if !ev.IsValid() {
return ev, &EncodeError{Path: path, Err: errors.New("nil element")} return ev, &EncodeError{Path: path.String(), Err: errors.New("nil element")}
} }
return ev, nil return ev, nil
} }
@@ -731,20 +879,25 @@ func textValue(v reflect.Value) (string, bool, error) {
return string(b), true, nil return string(b), true, nil
} }
// textMarshalerOf finds the encoding.TextMarshaler for v: on the value itself, // textMarshalerOf finds the encoding.TextMarshaler for v through the same
// or on its address, so a pointer-receiver MarshalText is found on an // flag cache: on the value itself, or on its address, so a pointer-receiver
// addressable struct field. // MarshalText is found on an addressable struct field.
func textMarshalerOf(v reflect.Value) (encoding.TextMarshaler, bool) { func textMarshalerOf(v reflect.Value) (encoding.TextMarshaler, bool) {
if !v.CanInterface() { if !v.CanInterface() {
return nil, false return nil, false
} }
if m, ok := v.Interface().(encoding.TextMarshaler); ok { if v.Kind() == reflect.Interface {
return m, true m, ok := v.Interface().(encoding.TextMarshaler)
return m, ok
} }
if v.CanAddr() { f := encTypeFlags(v.Type())
if m, ok := v.Addr().Interface().(encoding.TextMarshaler); ok { if f&encFlagTextMarshaler != 0 {
return m, true m, ok := v.Interface().(encoding.TextMarshaler)
return m, ok
} }
if f&encFlagAddrTextMarshaler != 0 && v.CanAddr() {
m, ok := v.Addr().Interface().(encoding.TextMarshaler)
return m, ok
} }
return nil, false return nil, false
} }
@@ -767,13 +920,6 @@ func isTableElementValue(v reflect.Value) bool {
return isTableElementType(v.Type()) return isTableElementType(v.Type())
} }
func joinKey(ctx, name string) string {
if ctx == "" {
return name
}
return ctx + "." + name
}
// --- emission ------------------------------------------------------------ // --- emission ------------------------------------------------------------
// writeBlankLine writes a single newline before a table or array-of-tables // writeBlankLine writes a single newline before a table or array-of-tables
@@ -788,8 +934,15 @@ func (e *encoder) writeBlankLine() {
func (e *encoder) emitDoc(doc *tomlDoc, prefix []string) error { func (e *encoder) emitDoc(doc *tomlDoc, prefix []string) error {
if e.opts.groupByKind { if e.opts.groupByKind {
scalars, tables, arrays := doc.partitionedEntries() // Scalars first, then inline sub-tables as value lines, then the
for _, kv := range scalars { // remaining tables as headers, then arrays of tables. Each pass walks
// the entries in place; grouping copies of them cost the encoder a
// third of its allocations for nothing.
for i := range doc.entries {
kv := &doc.entries[i]
if kv.kind != entryScalar {
continue
}
if err := e.writeKV(kv.key, kv.val); err != nil { if err := e.writeKV(kv.key, kv.val); err != nil {
return err return err
} }
@@ -797,17 +950,22 @@ func (e *encoder) emitDoc(doc *tomlDoc, prefix []string) error {
// An inlined sub-table is a value line, so it has to precede every // An inlined sub-table is a value line, so it has to precede every
// header of this document: a line written after a [header] would be // header of this document: a line written after a [header] would be
// read back as part of that table. // read back as part of that table.
headers := make([]entry, 0, len(tables)) for i := range doc.entries {
for _, t := range tables { t := &doc.entries[i]
if t.kind != entryTable {
continue
}
inlined, err := e.writeInlineSubTableIfSmall(t.key, t.doc) inlined, err := e.writeInlineSubTableIfSmall(t.key, t.doc)
if err != nil { if err != nil {
return err return err
} }
if !inlined { t.emitted = inlined
headers = append(headers, t)
} }
for i := range doc.entries {
t := &doc.entries[i]
if t.kind != entryTable || t.emitted {
continue
} }
for _, t := range headers {
path := append(append([]string{}, prefix...), t.key) path := append(append([]string{}, prefix...), t.key)
e.writeBlankLine() e.writeBlankLine()
e.buf.WriteByte('[') e.buf.WriteByte('[')
@@ -819,7 +977,11 @@ func (e *encoder) emitDoc(doc *tomlDoc, prefix []string) error {
return err return err
} }
} }
for _, a := range arrays { for i := range doc.entries {
a := &doc.entries[i]
if a.kind != entryArray {
continue
}
path := append(append([]string{}, prefix...), a.key) path := append(append([]string{}, prefix...), a.key)
for _, sub := range a.docs { for _, sub := range a.docs {
e.writeBlankLine() e.writeBlankLine()
@@ -918,21 +1080,37 @@ func (e *encoder) writeKey(key string) error {
if !utf8.ValidString(key) { if !utf8.ValidString(key) {
return fmt.Errorf("interpres: key %q is not valid UTF-8", key) return fmt.Errorf("interpres: key %q is not valid UTF-8", key)
} }
return writeQuotedString(&e.buf, key) return writeQuotedString(e.buf, key)
} }
// writeQuotedString writes s as a TOML basic string (double-quoted) to buf. // writeQuotedString writes s as a TOML basic string (double-quoted) to buf.
// Returns an error only if s is not valid UTF-8; invalid byte sequences // Returns an error only if s is not valid UTF-8.
// within a valid UTF-8 string are encoded as \ufffd replacement characters.
func writeQuotedString(buf *bytes.Buffer, s string) error { func writeQuotedString(buf *bytes.Buffer, s string) error {
if !utf8.ValidString(s) { if !utf8.ValidString(s) {
return fmt.Errorf("interpres: string is not valid UTF-8") return fmt.Errorf("interpres: string is not valid UTF-8")
} }
buf.WriteByte('"') buf.WriteByte('"')
for i := 0; i < len(s); { for i := 0; i < len(s); {
// Write the run of plain characters in one go; the scan stops at a
// byte that needs an escape, a control character, or a multi-byte rune.
j := i
for j < len(s) {
c := s[j]
if c < 0x20 || c == 0x7f || c == '"' || c == '\\' || c >= 0x80 {
break
}
j++
}
if j > i {
buf.WriteString(s[i:j])
i = j
}
if i >= len(s) {
break
}
r, size := utf8.DecodeRuneInString(s[i:]) r, size := utf8.DecodeRuneInString(s[i:])
if r == utf8.RuneError && size == 1 { if r == utf8.RuneError && size == 1 {
buf.WriteString(`\ufffd`) buf.WriteString("\\ufffd")
i++ i++
continue continue
} }
@@ -963,7 +1141,7 @@ func writeEscapedRune(buf *bytes.Buffer, r rune) {
buf.WriteString(`\r`) buf.WriteString(`\r`)
default: default:
if r < 0x20 || r == 0x7f { if r < 0x20 || r == 0x7f {
fmt.Fprintf(buf, `\u%04X`, r) fmt.Fprintf(buf, "\\u%04X", r)
} else { } else {
buf.WriteRune(r) buf.WriteRune(r)
} }
@@ -1034,12 +1212,23 @@ func (e *encoder) writeValue(val any) error {
// writeInlineMap renders m as a TOML inline table, on one line when it fits // writeInlineMap renders m as a TOML inline table, on one line when it fits
// there and across lines when it does not. // there and across lines when it does not.
func (e *encoder) writeInlineMap(m map[string]any) error { func (e *encoder) writeInlineMap(m map[string]any) error {
// An encoder that is itself measuring renders single-line by contract, so
// a nested inline table needs no measuring pass of its own.
if e.limit >= noInlineBreak {
return e.writeInlineMapFlat(m)
}
flat := e.flat() flat := e.flat()
if err := flat.writeInlineMapFlat(m); err != nil { err := flat.writeInlineMapFlat(m)
if err != nil {
flat.release()
return err return err
} }
if e.column()+flat.buf.Len() <= e.limit { fits := e.column()+flat.buf.Len() <= e.limit
if fits {
e.buf.Write(flat.buf.Bytes()) e.buf.Write(flat.buf.Bytes())
}
flat.release()
if fits {
return nil return nil
} }
return e.writeInlineMapMultiline(m) return e.writeInlineMapMultiline(m)
@@ -1180,20 +1369,25 @@ func (e *encoder) writeInlineSubTableIfSmall(name string, doc *tomlDoc) (bool, e
} }
flat := e.flat() flat := e.flat()
if err := flat.writeInlineDoc(doc); err != nil { if err := flat.writeInlineDoc(doc); err != nil {
flat.release()
return false, err return false, err
} }
if flat.buf.Len() > e.opts.inlineTablesAt { if flat.buf.Len() > e.opts.inlineTablesAt {
flat.release()
return false, nil return false, nil
} }
if err := e.writeKey(name); err != nil { if err := e.writeKey(name); err != nil {
flat.release()
return false, err return false, err
} }
e.buf.WriteString(" = ") e.buf.WriteString(" = ")
if e.column()+flat.buf.Len() <= e.limit { if e.column()+flat.buf.Len() <= e.limit {
e.buf.Write(flat.buf.Bytes()) e.buf.Write(flat.buf.Bytes())
} else if err := e.writeInlineDocMultiline(doc); err != nil { } else if err := e.writeInlineDocMultiline(doc); err != nil {
flat.release()
return false, err return false, err
} }
flat.release()
e.buf.WriteByte('\n') e.buf.WriteByte('\n')
return true, nil return true, nil
} }
@@ -1201,9 +1395,9 @@ func (e *encoder) writeInlineSubTableIfSmall(name string, doc *tomlDoc) (bool, e
func (e *encoder) writeStringVal(s string) error { func (e *encoder) writeStringVal(s string) error {
if e.opts.literalMultilineAt > 0 && strings.ContainsRune(s, '\n') && if e.opts.literalMultilineAt > 0 && strings.ContainsRune(s, '\n') &&
len(s) >= e.opts.literalMultilineAt && canBeLiteralMultiline(s) { len(s) >= e.opts.literalMultilineAt && canBeLiteralMultiline(s) {
return writeLiteralMultilineString(&e.buf, s) return writeLiteralMultilineString(e.buf, s)
} }
return writeQuotedString(&e.buf, s) return writeQuotedString(e.buf, s)
} }
// canBeLiteralMultiline reports whether s can be carried verbatim by the // canBeLiteralMultiline reports whether s can be carried verbatim by the
+10 -5
View File
@@ -24,7 +24,7 @@ import (
"context" "context"
"errors" "errors"
"fmt" "fmt"
"unicode/utf8" "slices"
) )
// A SyntaxError describes a malformed TOML document, including the 1-based // A SyntaxError describes a malformed TOML document, including the 1-based
@@ -143,9 +143,9 @@ func parseWithOptions(ctx context.Context, data []byte, opts parseOptions, wantD
if opts.maxInputSize > 0 && len(data) > opts.maxInputSize { if opts.maxInputSize > 0 && len(data) > opts.maxInputSize {
return nil, nil, fmt.Errorf("interpres: input is %d bytes, over the limit of %d", len(data), opts.maxInputSize) return nil, nil, fmt.Errorf("interpres: input is %d bytes, over the limit of %d", len(data), opts.maxInputSize)
} }
if !utf8.Valid(data) { // UTF-8 validity is not checked in a pass of its own: the scanner
return nil, nil, &SyntaxError{Line: 1, Msg: "input is not valid UTF-8"} // validates the multi-byte sequences where it meets them, so an invalid
} // byte is reported on its own line instead of always on line 1.
maxDepth := opts.maxDepth maxDepth := opts.maxDepth
if maxDepth <= 0 { if maxDepth <= 0 {
maxDepth = maxNestingDepth maxDepth = maxNestingDepth
@@ -424,7 +424,12 @@ func (e *Encoder) MarshalContext(ctx context.Context, v any) ([]byte, error) {
enc.ctx = ctx enc.ctx = ctx
enc.opts = *e enc.opts = *e
if err := enc.encode(v); err != nil { if err := enc.encode(v); err != nil {
enc.release()
return nil, err return nil, err
} }
return enc.bytes(), nil // The output leaves the pooled buffer as a copy, so the next Marshal
// reuses the buffer without touching what the caller holds.
out := slices.Clone(enc.buf.Bytes())
enc.release()
return out, nil
} }
+9
View File
@@ -41,6 +41,15 @@ func decodeDecimalInt(tok string) (any, error) {
if err := checkNoLeadingZero(digits); err != nil { if err := checkNoLeadingZero(digits); err != nil {
return nil, err return nil, err
} }
// An unsigned token parses in place; only a sign needs the concatenated
// copy, and concatenating an empty sign still allocated.
if sign == "" {
i, err := strconv.ParseInt(digits, 10, 64)
if err != nil {
return nil, fmt.Errorf("integer %q out of range", tok)
}
return i, nil
}
i, err := strconv.ParseInt(sign+digits, 10, 64) i, err := strconv.ParseInt(sign+digits, 10, 64)
if err != nil { if err != nil {
return nil, fmt.Errorf("integer %q out of range", tok) return nil, fmt.Errorf("integer %q out of range", tok)
+358 -90
View File
@@ -18,10 +18,10 @@ const ctxCheckInterval = 64
// parser is a recursive-descent TOML parser producing a map[string]any tree. // parser is a recursive-descent TOML parser producing a map[string]any tree.
// //
// The scanner works on bytes, not runes: the input is validated UTF-8 before // The scanner works on bytes, not runes: every character that drives the
// the parser runs, every character that drives the grammar (quotes, // grammar (quotes, separators, newlines, bare-key characters) is ASCII, the
// separators, newlines, bare-key characters) is ASCII, and multi-byte runes // scan validates a multi-byte sequence where it meets one, and multi-byte
// matter only as string content, where they are decoded on the spot. Holding // runes matter only as content, where they are decoded on the spot. Holding
// the source as []rune instead would cost a conversion pass plus four bytes // the source as []rune instead would cost a conversion pass plus four bytes
// per rune of extra memory before parsing even starts. // per rune of extra memory before parsing even starts.
type parser struct { type parser struct {
@@ -45,6 +45,21 @@ type parser struct {
currentPath []string currentPath []string
// keys interns key strings: a document that repeats a key across
// array-of-tables elements stores one string per distinct key instead of
// one per occurrence. The table is parser-local and dies with the parse;
// the tree keeps sharing the strings it was handed.
keys map[string]string
// keyBuf backs the transient single-segment result of parseKeyPath. A
// caller that keeps the path copies it out first, which is what
// retainPath does for the current section.
keyBuf [1]string
// absScratch backs the absolute path of a top-level key, which lives only
// for the statement being parsed.
absScratch [1]string
// wantDoc asks for the node tree the Document is built from; doc is that // wantDoc asks for the node tree the Document is built from; doc is that
// tree, and it stays nil when only the value tree is wanted. currentNode // tree, and it stays nil when only the value tree is wanted. currentNode
// is the node of p.current; pending collects the comment lines since the // is the node of p.current; pending collects the comment lines since the
@@ -87,10 +102,9 @@ func (p *parser) leaveNesting() { p.depth-- }
func (p *parser) parse() (map[string]any, error) { func (p *parser) parse() (map[string]any, error) {
p.root = map[string]any{} p.root = map[string]any{}
p.current = p.root p.current = p.root
p.headers = map[string]bool{} // The definition maps start unallocated: a document with no headers, no
p.frozen = map[string]bool{} // dotted keys and no inline tables never pays for them, and a nil map
p.dotted = map[string]bool{} // reads as empty. Each is created on its first write.
p.arrays = map[string]bool{}
p.currentPath = nil p.currentPath = nil
if p.wantDoc { if p.wantDoc {
p.doc = newTable(p.root) p.doc = newTable(p.root)
@@ -166,6 +180,56 @@ func (p *parser) checkCtx() error {
return p.ctx.Err() return p.ctx.Err()
} }
// --- definition maps --------------------------------------------------------
// The definition maps record what a document has already defined, so a later
// statement cannot redefine it. Each is created on first write: reads on a
// nil map answer false, which is exactly the state of a map never written.
func (p *parser) markHeader(pk string) {
if p.headers == nil {
p.headers = make(map[string]bool, 4)
}
p.headers[pk] = true
}
func (p *parser) markFrozen(pk string) {
if p.frozen == nil {
p.frozen = make(map[string]bool, 4)
}
p.frozen[pk] = true
}
func (p *parser) markDotted(pk string) {
if p.dotted == nil {
p.dotted = make(map[string]bool, 4)
}
p.dotted[pk] = true
}
func (p *parser) markArray(pk string) {
if p.arrays == nil {
p.arrays = make(map[string]bool, 2)
}
p.arrays[pk] = true
}
// internKey returns the shared string for key bytes. The lookup works on the
// bytes directly, which the compiler lets run without allocating, so a
// repeated key costs no allocation at all and the tree stores one string per
// distinct key.
func (p *parser) internKey(b []byte) string {
if p.keys == nil {
p.keys = make(map[string]string, 16)
}
if s, ok := p.keys[string(b)]; ok {
return s
}
s := string(b)
p.keys[s] = s
return s
}
// --- table headers --------------------------------------------------------- // --- table headers ---------------------------------------------------------
func (p *parser) parseTableHeader() error { func (p *parser) parseTableHeader() error {
@@ -176,7 +240,7 @@ func (p *parser) parseTableHeader() error {
p.pos++ p.pos++
} }
key, err := p.parseKeyPath() first, rest, err := p.parseKeyPath()
if err != nil { if err != nil {
return err return err
} }
@@ -193,6 +257,15 @@ func (p *parser) parseTableHeader() error {
p.pos++ p.pos++
} }
// The key the rest of the header handling reads. parseKeyPath hands back
// a transient buffer for the single-segment case, the shape every
// repeated array-of-tables header has; anything longer is copied once.
key := p.keyBuf[:1]
key[0] = first
if len(rest) > 0 {
key = append([]string{first}, rest...)
}
if array { if array {
tbl, elem, err := p.appendArrayTable(key) tbl, elem, err := p.appendArrayTable(key)
if err != nil { if err != nil {
@@ -201,9 +274,9 @@ func (p *parser) parseTableHeader() error {
// A new array-of-tables element starts a fresh scope: sub-table headers // A new array-of-tables element starts a fresh scope: sub-table headers
// and inline-table freezes from the previous element no longer apply. // and inline-table freezes from the previous element no longer apply.
p.resetScopeUnder(key) p.resetScopeUnder(key)
p.arrays[pathKey(key)] = true p.markArray(pathKey(key))
p.current = tbl p.current = tbl
p.currentPath = key p.currentPath = p.retainPath(key)
p.currentNode = elem p.currentNode = elem
p.lastTable = elem p.lastTable = elem
return nil return nil
@@ -213,30 +286,53 @@ func (p *parser) parseTableHeader() error {
if p.headers[pk] || p.dotted[pk] || p.arrays[pk] { if p.headers[pk] || p.dotted[pk] || p.arrays[pk] {
return p.errf("table %q is defined more than once", strings.Join(key, ".")) return p.errf("table %q is defined more than once", strings.Join(key, "."))
} }
p.headers[pk] = true p.markHeader(pk)
tbl, node, err := p.tableAt(key) tbl, node, err := p.tableAt(key)
if err != nil { if err != nil {
return err return err
} }
p.current = tbl p.current = tbl
p.currentPath = key p.currentPath = p.retainPath(key)
p.currentNode = node p.currentNode = node
p.lastTable = node p.lastTable = node
return nil return nil
} }
// retainPath copies key into the parser-owned storage currentPath holds, so
// the transient key buffer is free to serve the next statement.
func (p *parser) retainPath(key []string) []string {
if cap(p.currentPath) < len(key) {
p.currentPath = make([]string, len(key))
} else {
p.currentPath = p.currentPath[:len(key)]
}
copy(p.currentPath, key)
return p.currentPath
}
// tableAt walks (creating intermediate tables) to the table named by key, // tableAt walks (creating intermediate tables) to the table named by key,
// relative to the document root, rejecting any step into a frozen inline table. // relative to the document root, rejecting any step into a frozen inline table.
func (p *parser) tableAt(key []string) (map[string]any, *Table, error) { func (p *parser) tableAt(key []string) (map[string]any, *Table, error) {
cur := p.root cur := p.root
node := p.doc node := p.doc
path := make([]string, 0, len(key)) // The intermediate-path bookkeeping allocates only when the key actually
// has intermediate segments; a single-segment key checks its own name.
var path []string
if len(key) > 1 {
path = make([]string, 0, len(key))
}
for _, k := range key { for _, k := range key {
if len(key) == 1 {
if p.frozen[k] {
return nil, nil, p.errf("cannot extend inline table %q", k)
}
} else {
path = append(path, k) path = append(path, k)
if p.frozen[pathKey(path)] { if p.frozen[pathKey(path)] {
return nil, nil, p.errf("cannot extend inline table %q", strings.Join(path, ".")) return nil, nil, p.errf("cannot extend inline table %q", strings.Join(path, "."))
} }
}
existing, ok := cur[k] existing, ok := cur[k]
if !ok { if !ok {
next := map[string]any{} next := map[string]any{}
@@ -271,7 +367,12 @@ func (p *parser) tableAt(key []string) (map[string]any, *Table, error) {
func (p *parser) appendArrayTable(key []string) (map[string]any, *Table, error) { func (p *parser) appendArrayTable(key []string) (map[string]any, *Table, error) {
parent := p.root parent := p.root
node := p.doc node := p.doc
path := make([]string, 0, len(key)) // As in tableAt, the path slice exists only for a multi-segment key; the
// loop below runs for those alone.
var path []string
if len(key) > 1 {
path = make([]string, 0, len(key))
}
for _, k := range key[:len(key)-1] { for _, k := range key[:len(key)-1] {
path = append(path, k) path = append(path, k)
if p.frozen[pathKey(path)] { if p.frozen[pathKey(path)] {
@@ -323,7 +424,7 @@ func (p *parser) appendArrayTable(key []string) (map[string]any, *Table, error)
// --- key/value ------------------------------------------------------------- // --- key/value -------------------------------------------------------------
func (p *parser) parseKeyValue() error { func (p *parser) parseKeyValue() error {
key, err := p.parseKeyPath() first, rest, err := p.parseKeyPath()
if err != nil { if err != nil {
return err return err
} }
@@ -340,47 +441,46 @@ func (p *parser) parseKeyValue() error {
} }
dest := p.current dest := p.current
// One allocation covers the current section plus the dotted key; a // The absolute path of the key drives the dotted-key bookkeeping and the
// top-level statement reuses it for the leaf. // inline-table freeze. A single top-level key needs it only for the
abs := make([]string, 0, len(p.currentPath)+len(key)) // freeze, where a one-element path sits in the parser's scratch.
var abs []string
if len(rest) > 0 || len(p.currentPath) > 0 {
abs = make([]string, 0, len(p.currentPath)+len(rest)+1)
abs = append(abs, p.currentPath...) abs = append(abs, p.currentPath...)
abs = append(abs, first)
} else {
abs = append(p.absScratch[:0], first)
}
// dests collects the map each dotted key descended into, which the node // dests collects the map each dotted key descended into, which the node
// tree needs to build the matching tables around the value. // tree needs to build the matching tables around the value.
var dests []map[string]any var dests []map[string]any
for _, k := range key[:len(key)-1] { leaf := first
if len(rest) > 0 {
if err := p.descendKey(&dest, first, abs, &dests); err != nil {
return err
}
for _, k := range rest[:len(rest)-1] {
abs = append(abs, k) abs = append(abs, k)
if p.frozen[pathKey(abs)] { if err := p.descendKey(&dest, k, abs, &dests); err != nil {
return p.errf("cannot extend inline table %q", strings.Join(abs, ".")) return err
} }
if p.headers[pathKey(abs)] {
return p.errf("cannot extend table %q with a dotted key", strings.Join(abs, "."))
} }
p.dotted[pathKey(abs)] = true leaf = rest[len(rest)-1]
existing, ok := dest[k]
if !ok {
next := map[string]any{}
dest[k] = next
dest = next
dests = append(dests, next)
continue
}
m, ok := existing.(map[string]any)
if !ok {
return p.errf("key %q is not a table", k)
}
dest = m
dests = append(dests, m)
}
leaf := key[len(key)-1]
abs = append(abs, leaf) abs = append(abs, leaf)
}
if _, exists := dest[leaf]; exists { if _, exists := dest[leaf]; exists {
return p.errf("duplicate key %q", leaf) return p.errf("duplicate key %q", leaf)
} }
dest[leaf] = val dest[leaf] = val
if p.doc != nil { if p.doc != nil {
node := p.currentNode node := p.currentNode
for i, k := range key[:len(key)-1] { if len(rest) > 0 {
node = node.addTable(k, dests[i]) node = node.addTable(first, dests[0])
for i, k := range rest[:len(rest)-1] {
node = node.addTable(k, dests[i+1])
}
} }
_, inline := val.(map[string]any) _, inline := val.(map[string]any)
entry := node.addValue(leaf, val, inline) entry := node.addValue(leaf, val, inline)
@@ -396,6 +496,35 @@ func (p *parser) parseKeyValue() error {
return nil return nil
} }
// descendKey walks dest into the sub-table named key on the dotted path abs,
// recording the path in the definition maps; dests collects the maps
// descended into.
func (p *parser) descendKey(dest *map[string]any, key string, abs []string, dests *[]map[string]any) error {
ak := pathKey(abs)
if p.frozen[ak] {
return p.errf("cannot extend inline table %q", strings.Join(abs, "."))
}
if p.headers[ak] {
return p.errf("cannot extend table %q with a dotted key", strings.Join(abs, "."))
}
p.markDotted(ak)
existing, ok := (*dest)[key]
if !ok {
next := map[string]any{}
(*dest)[key] = next
*dest = next
*dests = append(*dests, next)
return nil
}
m, ok := existing.(map[string]any)
if !ok {
return p.errf("key %q is not a table", key)
}
*dest = m
*dests = append(*dests, m)
return nil
}
// takeInline returns the node of the inline table just parsed, when v is that // takeInline returns the node of the inline table just parsed, when v is that
// table's value, and clears it so a later value cannot pick it up. // table's value, and clears it so a later value cannot pick it up.
func (p *parser) takeInline(v any) *Table { func (p *parser) takeInline(v any) *Table {
@@ -419,16 +548,17 @@ func (p *parser) takeArrayElems(v any) []*Table {
} }
// freezeInline marks the path of an inline table (and any nested inline tables) // freezeInline marks the path of an inline table (and any nested inline tables)
// as immutable, so a later header or dotted key cannot extend it. // as immutable, so a later header or dotted key cannot extend it. The
// recursion appends into the caller's path slice; the frozen map keeps the
// joined strings, never the slice, so the backing is free to be reused.
func (p *parser) freezeInline(path []string, val any) { func (p *parser) freezeInline(path []string, val any) {
m, ok := val.(map[string]any) m, ok := val.(map[string]any)
if !ok { if !ok {
return return
} }
p.frozen[pathKey(path)] = true p.markFrozen(pathKey(path))
for k, v := range m { for k, v := range m {
child := append(append([]string{}, path...), k) p.freezeInline(append(path, k), v)
p.freezeInline(child, v)
} }
} }
@@ -438,33 +568,54 @@ func (p *parser) freezeInline(path []string, val any) {
// fresh in the new element. // fresh in the new element.
func (p *parser) resetScopeUnder(key []string) { func (p *parser) resetScopeUnder(key []string) {
prefix := pathKey(key) + "\x00" prefix := pathKey(key) + "\x00"
for _, m := range []map[string]bool{p.headers, p.frozen, p.dotted, p.arrays} { p.resetMapUnder(p.headers, prefix)
p.resetMapUnder(p.frozen, prefix)
p.resetMapUnder(p.dotted, prefix)
p.resetMapUnder(p.arrays, prefix)
}
// resetMapUnder deletes the entries m holds under prefix. An empty or
// unallocated map holds none, so the common case walks nothing.
func (p *parser) resetMapUnder(m map[string]bool, prefix string) {
if len(m) == 0 {
return
}
for k := range m { for k := range m {
if strings.HasPrefix(k, prefix) { if strings.HasPrefix(k, prefix) {
delete(m, k) delete(m, k)
} }
} }
}
} }
// parseKeyPath parses a dotted key into its components. // parseKeyPath parses a dotted key. The first component comes back directly
func (p *parser) parseKeyPath() ([]string, error) { // and the rest as a usually nil slice, because a single-component key is the
var parts []string // common shape and a fresh slice per statement is what the allocation profile
// showed. The single-key slice a caller sees is parser-owned and transient.
func (p *parser) parseKeyPath() (string, []string, error) {
p.skipInline()
first, err := p.parseKeyComponent()
if err != nil {
return "", nil, err
}
p.skipInline()
if p.eof() || p.peek() != '.' {
return first, nil, nil
}
p.pos++
var rest []string
for { for {
p.skipInline() p.skipInline()
part, err := p.parseKeyComponent() part, err := p.parseKeyComponent()
if err != nil { if err != nil {
return nil, err return "", nil, err
} }
parts = append(parts, part) rest = append(rest, part)
p.skipInline() p.skipInline()
if !p.eof() && p.peek() == '.' { if p.eof() || p.peek() != '.' {
return first, rest, nil
}
p.pos++ p.pos++
continue
} }
break
}
return parts, nil
} }
func (p *parser) parseKeyComponent() (string, error) { func (p *parser) parseKeyComponent() (string, error) {
@@ -493,11 +644,18 @@ func (p *parser) parseKeyComponent() (string, error) {
} }
break break
} }
// The stopping byte decides the message: a multi-byte sequence that
// does not decode names that, before any grammar message can.
if !p.eof() && p.peek() >= utf8.RuneSelf {
if r, size := utf8.DecodeRune(p.src[p.pos:]); r == utf8.RuneError && size == 1 {
return "", p.errf("invalid UTF-8 in key")
}
}
if p.pos == start { if p.pos == start {
r, _ := utf8.DecodeRune(p.src[p.pos:]) r, _ := utf8.DecodeRune(p.src[p.pos:])
return "", p.errf("invalid key character %q", string(r)) return "", p.errf("invalid key character %q", string(r))
} }
return string(p.src[start:p.pos]), nil return p.internKey(p.src[start:p.pos]), nil
} }
} }
@@ -544,6 +702,9 @@ func (p *parser) parseAtom() (any, error) {
if tok == "" { if tok == "" {
return nil, p.errf("expected a value") return nil, p.errf("expected a value")
} }
if hasHighByte(tok) && !utf8.ValidString(tok) {
return nil, p.errf("invalid UTF-8 in value")
}
// A date may be followed by a space and a time, forming one date-time. // A date may be followed by a space and a time, forming one date-time.
if isDateToken(tok) && !p.eof() && p.peek() == ' ' { if isDateToken(tok) && !p.eof() && p.peek() == ' ' {
if next, ok := p.peekAt(1); ok && next >= '0' && next <= '9' { if next, ok := p.peekAt(1); ok && next >= '0' && next <= '9' {
@@ -576,6 +737,17 @@ func (p *parser) scanBareToken() {
} }
} }
// hasHighByte reports whether s holds any byte outside ASCII, the cheap gate
// in front of a full UTF-8 check.
func hasHighByte(s string) bool {
for i := range len(s) {
if s[i] >= utf8.RuneSelf {
return true
}
}
return false
}
// --- strings --------------------------------------------------------------- // --- strings ---------------------------------------------------------------
func (p *parser) parseBasicString() (string, error) { func (p *parser) parseBasicString() (string, error) {
@@ -583,7 +755,36 @@ func (p *parser) parseBasicString() (string, error) {
return p.parseMultilineString('"', true) return p.parseMultilineString('"', true)
} }
p.pos++ // opening quote p.pos++ // opening quote
start := p.pos
// A run of plain characters up to the closing quote needs no builder, only
// one copy at the end; escapes, controls and multi-byte runes fall through
// to the builder loop, which validates them on the spot.
for p.pos < len(p.src) {
c := p.src[p.pos]
if c == '"' {
s := string(p.src[start:p.pos])
p.pos++
return s, nil
}
if c == '\\' || c == '\n' || c == '\r' || c >= utf8.RuneSelf ||
(c < 0x20 && c != '\t') || c == 0x7f {
break
}
p.pos++
}
if p.eof() {
return "", p.errf("unterminated string")
}
var b strings.Builder var b strings.Builder
b.Grow(p.pos - start)
b.Write(p.src[start:p.pos])
return p.parseBasicStringRest(&b)
}
// parseBasicStringRest continues a basic string whose fast scan has met a byte
// it does not handle: an escape, a control character, a multi-byte rune, or a
// bare newline, which the loop rejects.
func (p *parser) parseBasicStringRest(b *strings.Builder) (string, error) {
for { for {
if p.eof() { if p.eof() {
return "", p.errf("unterminated string") return "", p.errf("unterminated string")
@@ -605,7 +806,7 @@ func (p *parser) parseBasicString() (string, error) {
} }
b.WriteRune(r) b.WriteRune(r)
default: default:
if err := p.writeContentRune(&b); err != nil { if err := p.writeContentRune(b); err != nil {
return "", err return "", err
} }
} }
@@ -617,7 +818,27 @@ func (p *parser) parseLiteralString() (string, error) {
return p.parseMultilineString('\'', false) return p.parseMultilineString('\'', false)
} }
p.pos++ // opening quote p.pos++ // opening quote
start := p.pos
// The same fast scan as the basic string, without the escape case.
for p.pos < len(p.src) {
c := p.src[p.pos]
if c == '\'' {
s := string(p.src[start:p.pos])
p.pos++
return s, nil
}
if c == '\n' || c == '\r' || c >= utf8.RuneSelf ||
(c < 0x20 && c != '\t') || c == 0x7f {
break
}
p.pos++
}
if p.eof() {
return "", p.errf("unterminated literal string")
}
var b strings.Builder var b strings.Builder
b.Grow(p.pos - start)
b.Write(p.src[start:p.pos])
for { for {
if p.eof() { if p.eof() {
return "", p.errf("unterminated literal string") return "", p.errf("unterminated literal string")
@@ -641,8 +862,8 @@ func (p *parser) parseLiteralString() (string, error) {
// writeContentRune appends the rune at the cursor to b and advances past it. // writeContentRune appends the rune at the cursor to b and advances past it.
// An ASCII byte, which includes every control character the grammar forbids, // An ASCII byte, which includes every control character the grammar forbids,
// is checked and written directly; a multi-byte rune is decoded and can never // is checked and written directly; a multi-byte rune is decoded, and a
// be a control character. // sequence that does not decode is the UTF-8 error reported where it sits.
func (p *parser) writeContentRune(b *strings.Builder) error { func (p *parser) writeContentRune(b *strings.Builder) error {
c := p.peek() c := p.peek()
if c < utf8.RuneSelf { if c < utf8.RuneSelf {
@@ -654,6 +875,9 @@ func (p *parser) writeContentRune(b *strings.Builder) error {
return nil return nil
} }
r, size := utf8.DecodeRune(p.src[p.pos:]) r, size := utf8.DecodeRune(p.src[p.pos:])
if r == utf8.RuneError && size == 1 {
return p.errf("invalid UTF-8 in string")
}
p.pos += size p.pos += size
b.WriteRune(r) b.WriteRune(r)
return nil return nil
@@ -831,7 +1055,10 @@ func (p *parser) parseArray() (val any, err error) {
} }
defer p.leaveNesting() defer p.leaveNesting()
p.pos++ // '[' p.pos++ // '['
arr := []any{} // A small presize covers the arrays documents actually hold, and trades a
// little capacity on tiny arrays for the growth chain an append-from-nil
// costs per array.
arr := make([]any, 0, 4)
// elems carries the node of each element that is an inline table, so the // elems carries the node of each element that is an inline table, so the
// caller can keep its key order; the entries are nil for other values. // caller can keep its key order; the entries are nil for other values.
var elems []*Table var elems []*Table
@@ -886,7 +1113,9 @@ func (p *parser) parseInlineTable() (val any, err error) {
defer p.leaveNesting() defer p.leaveNesting()
p.pos++ // '{' p.pos++ // '{'
tbl := map[string]any{} tbl := map[string]any{}
assigned := map[string]bool{} // assigned tracks the dotted paths written into this table. It is created
// on the first key, so an empty inline table allocates nothing for it.
var assigned map[string]bool
// The inline table is a node of its own, so the keys keep their order; the // The inline table is a node of its own, so the keys keep their order; the
// caller picks the node up when the table parses. // caller picks the node up when the table parses.
var node *Table var node *Table
@@ -913,7 +1142,7 @@ func (p *parser) parseInlineTable() (val any, err error) {
if err := p.skipNestedSpace(); err != nil { if err := p.skipNestedSpace(); err != nil {
return nil, err return nil, err
} }
key, err := p.parseKeyPath() first, rest, err := p.parseKeyPath()
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -929,39 +1158,46 @@ func (p *parser) parseInlineTable() (val any, err error) {
} }
dest := tbl dest := tbl
path := make([]string, 0, len(key)) var path []string
var dests []map[string]any var dests []map[string]any
for _, k := range key[:len(key)-1] { leaf := first
if len(rest) > 0 {
path = append(p.absScratch[:0], first)
d, err := p.descendInline(&dest, first, path, assigned)
if err != nil {
return nil, err
}
dests = append(dests, d)
for _, k := range rest[:len(rest)-1] {
path = append(path, k) path = append(path, k)
if assigned[pathKey(path)] { d, err := p.descendInline(&dest, k, path, assigned)
return nil, p.errf("key %q is already defined", strings.Join(path, ".")) if err != nil {
return nil, err
} }
existing, ok := dest[k] dests = append(dests, d)
if !ok {
m := map[string]any{}
dest[k] = m
dest = m
dests = append(dests, m)
continue
} }
m, isMap := existing.(map[string]any) leaf = rest[len(rest)-1]
if !isMap {
return nil, p.errf("key %q is already defined", k)
}
dest = m
dests = append(dests, m)
}
leaf := key[len(key)-1]
path = append(path, leaf) path = append(path, leaf)
}
if _, exists := dest[leaf]; exists { if _, exists := dest[leaf]; exists {
return nil, p.errf("duplicate key %q in inline table", leaf) return nil, p.errf("duplicate key %q in inline table", leaf)
} }
dest[leaf] = val dest[leaf] = val
if assigned == nil {
assigned = make(map[string]bool, 4)
}
if len(rest) == 0 {
assigned[first] = true
} else {
assigned[pathKey(path)] = true assigned[pathKey(path)] = true
}
if node != nil { if node != nil {
child := node child := node
for i, k := range key[:len(key)-1] { if len(rest) > 0 {
child = child.addTable(k, dests[i]) child = child.addTable(first, dests[0])
for i, k := range rest[:len(rest)-1] {
child = child.addTable(k, dests[i+1])
}
} }
_, inline := val.(map[string]any) _, inline := val.(map[string]any)
entry := child.addValue(leaf, val, inline) entry := child.addValue(leaf, val, inline)
@@ -998,6 +1234,28 @@ func (p *parser) parseInlineTable() (val any, err error) {
} }
} }
// descendInline walks dest into the sub-table named key inside an inline
// table, rejecting a dotted segment the table has already defined.
func (p *parser) descendInline(dest *map[string]any, key string, path []string, assigned map[string]bool) (map[string]any, error) {
pk := pathKey(path)
if assigned[pk] {
return nil, p.errf("key %q is already defined", strings.Join(path, "."))
}
existing, ok := (*dest)[key]
if !ok {
m := map[string]any{}
(*dest)[key] = m
*dest = m
return m, nil
}
m, isMap := existing.(map[string]any)
if !isMap {
return nil, p.errf("key %q is already defined", key)
}
*dest = m
return m, nil
}
// --- scanning helpers ------------------------------------------------------ // --- scanning helpers ------------------------------------------------------
func (p *parser) eof() bool { return p.pos >= len(p.src) } func (p *parser) eof() bool { return p.pos >= len(p.src) }
@@ -1121,8 +1379,14 @@ func (p *parser) skipComment() (string, error) {
p.pos++ p.pos++
case c < 0x20 || c == 0x7f: case c < 0x20 || c == 0x7f:
return "", p.errf("control character U+%04X is not allowed in a comment", c) return "", p.errf("control character U+%04X is not allowed in a comment", c)
default: case c < utf8.RuneSelf:
p.pos++ p.pos++
default:
r, size := utf8.DecodeRune(p.src[p.pos:])
if r == utf8.RuneError && size == 1 {
return "", p.errf("invalid UTF-8 in comment")
}
p.pos += size
} }
} }
return commentText(string(p.src[start:p.pos])), nil return commentText(string(p.src[start:p.pos])), nil
@@ -1176,7 +1440,10 @@ func (p *parser) expectLineEnd() error {
p.pos++ p.pos++
return nil return nil
} }
r, _ := utf8.DecodeRune(p.src[p.pos:]) r, size := utf8.DecodeRune(p.src[p.pos:])
if r == utf8.RuneError && size == 1 {
return p.errf("invalid UTF-8 after value")
}
return p.errf("unexpected %q after value", string(r)) return p.errf("unexpected %q after value", string(r))
} }
@@ -1185,7 +1452,8 @@ func (p *parser) errf(format string, args ...any) error {
} }
// pathKey joins key components with a NUL separator so a dotted path can be // pathKey joins key components with a NUL separator so a dotted path can be
// used as a map key for tracking defined tables. // used as a map key for tracking defined tables. A single component comes
// back as it is, with no join and no copy.
func pathKey(parts []string) string { func pathKey(parts []string) string {
return strings.Join(parts, "\x00") return strings.Join(parts, "\x00")
} }