petrbalvin/scriptorium
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

feat: render Markdown, mathematics and Mermaid diagrams server-side

Browse Source
This commit is contained in:
petrbalvin
2026-09-27 19:15:50 +02:00
commit 35bc42623b
72 changed files with 8914 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/API.md
+124
View File
@@ -0,0 +1,124 @@
# API
Scriptorium is a library with one package and five exported functions. The
signatures below are the whole public surface; everything else lives under
`internal/` and reaches no caller. `go doc sourcedock.dev/petrbalvin/scriptorium`
is the authority on the signatures, and this file explains what each function
does, when it fails, and where the edges of the rendered surfaces are.
## Library
### `Render(source []byte) []byte`
Renders Markdown to HTML. The grammar is CommonMark 0.31.2 in full, with the
GitHub Flavored Markdown extensions on top: tables, strikethrough, task lists
and extended autolinks, plus footnotes and definition lists. Rendering never
fails and never returns an error: any input parses, and a construct the
grammar does not know stays in the output as literal text.
The extended autolinks supersede three CommonMark examples: a bare
`www.`/`http(s)://`/`ftp://` address or a bare email address becomes a link
where CommonMark without the extension would leave it as text. That is the
specification's own trade, and GitHub's reference implementation makes it
too.
```go
html := scriptorium.Render([]byte("# Title\n\nText with [a link](/uri).\n"))
```
The Markdown engine embeds no mathematics and no diagrams: a `$…$` run stays
what the CommonMark grammar says it is. A consumer that wants mathematics
inside its documents calls `RenderMath` on the math spans it recognises and
splices the results into the HTML itself.
### `RenderMath(source []byte) []byte`
Renders TeX mathematics to a complete MathML Core `<math>` element in the
inline form. Returns one line of markup with the namespace declared, ready to
embed in HTML. Never fails: see the degradation contract below.
### `RenderMathDisplay(source []byte) []byte`
The display form: the same engine, with `display="block"` on the element and
the movable limits of big operators set above and below instead of beside.
### `RenderDiagram(source []byte) ([]byte, error)`
Renders Mermaid source to a self-contained SVG document. The first line names
the diagram type; only two families are carried:
- `flowchart` and its historical `graph` alias, with any direction
(`TB`/`TD`, `BT`, `LR`, `RL`), every node shape, every edge kind, subgraphs
with their own direction, and the `classDef`, `class`, `style` and
`linkStyle` statements;
- `sequenceDiagram`, with participants and actors, all arrow kinds, notes,
activations in both spellings, the `alt`, `opt`, `loop`, `par`, `critical`
and `break` blocks, coloured `rect` regions, `autonumber` and dividers.
Returns an error when the type is outside the two families, when a direction
is unknown, or when a statement does not parse; every error names the diagram
type or the source line, so the author can find the sentence at fault. The
`click` statement is refused on purpose: the SVG is static and carries no
interactivity.
### `SupportedDiagram(kind string) bool`
Reports whether `RenderDiagram` carries the named Mermaid type. It answers
the question the error path would answer, before the call.
## What every function guarantees
- **Determinism.** The same input produces byte-identical output on every
call, in every process, on every machine. No layout decision reads a map
iteration order; the tests assert byte equality across repeated renders.
- **No sanitisation.** The input is trusted, and the output is written to
HTML, MathML or SVG verbatim. Whether the output may reach a given audience
is the consumer's policy: a consumer of untrusted input keeps its own
sanitiser and applies it to the rendered result.
- **Concurrency safety.** The engines hold no state between calls and share
nothing, so every function is safe for concurrent use.
- **No dependencies.** The standard library alone; there is no `go.sum`
because there is nothing to sum.
## The mathematics surface
The mathematics engine renders the standard TeX command surface that maps
into MathML Core, measured against the coverage list of KaTeX as the external
standard: the Greek letters including the variants, the relation and operator
tables including the negations and the colon relations, the arrows including
the extensible `\x` family, the big operators, the delimiters, the amsmath
environments, accents, the font and spacing commands, colours, and bounded
macros (`\newcommand`, `\renewcommand`, `\providecommand`, `\def`,
`\DeclareMathOperator`) with expansion limits.
Multi-character relations have no single Unicode character, so they render as
the linear two-character operators KaTeX's own MathML branch produces
(`\coloneqq` as `≔`, `\coloneq` as `:−`, `\Coloneqq` as `∷=`).
### Degradation, not failure
A construct outside the mappable surface degrades in place: the offending
command stays in the output as its verbatim source inside an `<merror>`
element, and the rest of the expression still renders. The author sees what
was not understood, in the document, at the place it happened.
The commands that degrade are the ones no MathML Core construct can carry:
positioning (`\raisebox`, `\vcenter`, `\smash`, the `\llap`/`\rlap`/
`\mathclap` family, `\kern`, `\mkern`, `\hskip`), boxes and overlays
(`\boxed`, `\colorbox`, `\fcolorbox`, `\sideset`), text annotations of the
page rather than the formula (`\tag`, `\label`, `\notag`, `\verb`,
`\char`, `\mathstrut`), and the environments outside the table family
(`\begin{CD}`, `\begin{tikzpicture}` and every other unknown name).
A macro whose expansion would not terminate is cut by the expansion limits
and degrades at the call site.
A degraded render is still a valid render: the output remains well-formed
MathML Core, and the tests assert well-formedness for every degraded corpus
case.
## Notes
The exported surface is documented in godoc form in the source, and `go doc ./...` is
the authority on signatures and types. This file explains what the surface is for and
how the parts fit together. It does not repeat the signatures, because a copy of a
signature is a future lie.
docs/ARCHITECTURE.md
+91
View File
@@ -0,0 +1,91 @@
# Architecture
How scriptorium is put together. Every node, package and arrow below exists in the
source tree; nothing is aspirational.
## Overview
```mermaid
flowchart TD
Caller[consumer] --> Root[scriptorium, public surface]
Root --> Markdown[internal/markdown]
Root --> Math[internal/mathml]
Root --> Diagram[internal/diagram]
Markdown --> Block[parse.go, block parser]
Markdown --> Inline[inline.go, inline parser]
Markdown --> Writer[render.go, HTML writer]
Math --> Tok[token.go, tokeniser]
Math --> MParse[parse.go and command.go, parser]
Math --> Tables[symbols.go, symbol tables]
Math --> Env[environments.go, tables]
Math --> Macros[macros.go, expansion]
Diagram --> DParse[flowparse.go and sequence.go, parsers]
Diagram --> DLayout[flowlayout.go and sequence.go, layout]
Diagram --> DWrite[svg.go, writer]
```
The root package is the whole public surface: `Render` hands Markdown source to
the Markdown engine, `RenderMath` and `RenderMathDisplay` hand TeX source to the
mathematics engine, `RenderDiagram` hands Mermaid source to the diagram engine,
and `SupportedDiagram` states the Mermaid scope. Everything else lives in
internal packages and reaches no caller.
The Markdown engine parses line by line into the block tree CommonMark defines;
`RenderHTML` walks the tree, parses the inline content of every leaf against the
document's link reference definitions and footnotes, and writes HTML.
The mathematics engine tokenises the TeX source, expands the bounded macros,
parses atoms with their scripts into a MathML tree and writes one line of XML. A
construct the tables do not carry degrades in place to its verbatim source inside
an merror element.
The diagram engine parses statement lines into flowchart or sequence models,
lays the flowchart out by rank and barycentre and the sequence along its
lifelines, and writes SVG. A diagram type outside the two grammars is refused
with an error naming it.
## Packages
| Package | Responsibility |
|---|---|
| `scriptorium` (root) | the public surface: `Render`, `RenderMath`, `RenderMathDisplay`, `RenderDiagram`, `SupportedDiagram`. Owns nothing else. |
| `internal/markdown` | the Markdown engine: block parsing, inline parsing, HTML writing, and the scanners both phases share. Carries no state between calls. |
| `internal/mathml` | the mathematics engine: tokenising, macro expansion, symbol tables, environments, accents, styles and the XML writer. Carries no state between calls. |
| `internal/diagram` | the diagram engine: the flowchart and sequence parsers, both layouts and the SVG writer. Carries no state between calls. |
## Data flow
```mermaid
sequenceDiagram
participant Caller
participant S as scriptorium.RenderMath
participant T as mathml tokeniser
participant P as mathml parser
participant W as mathml writer
Caller->>S: TeX source
S->>T: source
T-->>P: tokens
P->>P: expand macros, parse atoms and scripts
P-->>W: MathML tree
W-->>Caller: one math element
```
Rendering cannot fail. A construct outside the grammar stays in the output as its
literal source rather than becoming an error, and nothing is sanitised on the way
out: that is the consumer's policy.
Parsing and rendering are pure: no globals, no caches, no goroutines. The same source
produces byte-identical output on every call, which the tests assert.
## State and lifetime
- Everything is per-call: the block tree, the delimiter stack, the reference map,
the token list, the macro table, the diagram models and the layouts die with
the render.
- All functions are safe for concurrent use, because nothing is shared.
## Dependencies
None beyond the standard library. That is a contract of the project, not an
accident: every engine writes its own scanners and tables where the standard
library has nothing to offer.
docs/DEVELOPMENT.md
+76
View File
@@ -0,0 +1,76 @@
# Development
How to work on scriptorium.
## Prerequisites
- Go 1.27.1, the newest stable release.
- [just](https://github.com/casey/just) for the recipes.
- gcc, because the race detector in `just gates` needs cgo.
## Setup
```sh
git clone https://sourcedock.dev/petrbalvin/scriptorium.git
cd scriptorium
just build
```
## Recipes
Every recipe in the project's file, and what it does. Taken from the file itself, so
the names and the list match it exactly. A library produces no binary, so the
recipes that carry one do not exist here.
| Recipe | What it does |
|---|---|
| `just gates` | the definition of done: compile, format check, vet, modernisation, the test suite with the coverage floor, and the race detector |
| `just build` | compiles every package, zero warnings |
| `just test` | the full suite as CI runs it, with the coverage floor |
| `just unit . 'TestName'` | a fast scoped run for iterating |
| `just fuzz <target> <package>` | a time-boxed fuzz of one target, never a gate |
| `just bench` | benchmarks, on an idle machine only |
| `just fmt` | gofmt, in place |
| `just fmt-check` | gofmt, zero diff |
| `just vet` | `go vet` and `go fix -diff` |
| `just clean` | removes the coverage profile |
## Running a single test
```sh
go test -run TestName ./...
```
Add `-v` for the sub-test names, and `-race` when the change touches concurrency.
`-count=1` defeats the test cache when a result looks stale.
## Coverage
```sh
just test
go tool cover -func=coverage.out
```
The `total:` line is the number that matters, and it stays at 80 percent or more.
## Benchmarks
```sh
go test -run='^$' -bench=. -benchmem -count=5 ./...
```
Benchmark on an idle machine, and compare only runs made in one process against each
other: runs in separate processes, or on a loaded machine, differ by more than the
effects being measured.
## Continuous integration
Workflows live in `.gitea/workflows/` and run on the project's own runners. They are
written by hand rather than through `just`, but they enforce the same set of gates, so
a green `just gates` locally is the fastest way to a green pipeline.
## Releases
Releases are cut by merging `development` into `main` and tagging `vX.Y.Z`. The tag
drives the release workflow, which publishes the notes it extracted from
`CHANGELOG.md`.