volumen/docs/ARCHITECTURE.md

# Architecture

How Volumen is put together. Every node, package and arrow below exists in the
source tree; nothing is aspirational.

## Overview

```mermaid
flowchart TD
    cli["cmd/volumen, the flag dispatcher"] --> app["internal/app, server assembly and middleware chain"]
    cli --> backup["internal/backup, the tar.gz archive"]
    cli --> scheduler["internal/scheduler, the publish_at sweep"]
    app --> api["internal/httpapi, the public JSON API"]
    app --> admin["internal/admin, the server-rendered admin"]
    admin --> backup
    api --> preview["internal/preview, the signed preview link"]
    admin --> preview
    api --> store["internal/store, snapshot cache, atomic writes, revisions"]
    admin --> store
    scheduler --> store
    store --> post["internal/post, the domain object"]
    store --> disk[("content directory: posts, media, revisions")]
    post --> markdown["internal/markdown, scriptorium then bluemonday"]
    admin --> i18n["internal/i18n, the admin interface catalogue"]
    admin --> diff["internal/diff, the revision comparison"]
    app --> config["internal/config, TOML over built-in defaults"]
    admin --> fediverse["internal/fediverse, the @user@host rule"]
    config --> fediverse
    admin --> users["internal/users, internal/tokens, internal/templates"]
```

`cmd/volumen` is the only entry point. `serve` assembles `internal/app`, which
builds the file-backed stores from the configuration, wires the middleware
chain in front of the public API and the admin, and starts the scheduler loop
when the configuration enables it; a deployment is founded through the admin
itself, by the first-run wizard. `export`, `import` and
`publish-due` reach the filesystem and the archive without starting a server.
Volumen does not render the public site: the front end consumes the JSON API,
and the admin is the only HTML it serves.

## Packages

| Package | Responsibility |
|---|---|
| `config` | Reads `config.toml` into a typed value over the built-in defaults, applies the command-line overrides and validates it; with no file the defaults move the state under the user's home. It owns the commented template (`template.go`) shipped as `config.toml.example` and nothing about the content. |
| `store` | Owns the content directory through one `os.Root`: reading and writing posts, the snapshot cache, per-target write locks, the revision archive and its tombstones, and the media library. It decides where a file goes and never what a post means. |
| `imagefile` | The upload rule: which extensions are accepted, the WebP, AVIF and SVG signatures, the extension a byte string is, and the pixel size the header carries (the WebP chunks, the AVIF `ispe` box, the SVG root's size attributes or `viewBox`), so the media library can show it. The store asks it what an upload is; the media route asks it what a name may be. |
| `post` | The `Post` domain object: frontmatter metadata, the raw Markdown body, and the lazily rendered HTML and table of contents. It derives the slug, language, date, excerpt, reading time and publication status, and it clones deeply, because the store hands one instance to several readers. |
| `frontmatter` | Parses and serialises the `+++` block on an interpres `Document`, so a save keeps what the author wrote: the key order at every level, the comments, and whether a table is a header section or an inline table. |
| `markdown` | Owns rendering and sanitisation: scriptorium renders the body (CommonMark with the GFM extensions, footnotes and definition lists), a pre-render scan lifts `$$…$$` and `$…$` runs out of the source and splices the MathML back after rendering, fenced mermaid blocks become their SVG, headings gain ids and a table of contents, a figure pass wraps titled images, and the bluemonday allowlist sanitises the result. This is the only place that turns a body into HTML. |
| `feeds` | Renders RSS 2.0, Atom 1.0, JSON Feed 1.1 and the sitemap from the same posts the API serves. |
| `payloads` | Builds the API's wire shapes: site metadata, post summaries and details, tag and series listings, pagination in both modes, and the validation used by the write endpoints and the admin forms. |
| `httpapi` | Serves `/api/volumen/*`: reads, feeds, the sitemap, and the token-authenticated writes. It never writes a file; it calls `store`. |
| `admin` | Serves `/admin`: the first-run wizard that founds the installation, login, the dashboard, post CRUD, the editor and its preview, import and download, history, media, settings, users, tokens, webhooks, backups and self-update, with the session, role and CSRF guards. |
| `users`, `tokens`, `templates` | The file-backed stores beside `users.toml`: accounts with roles (the first one created by the wizard's serialised `AddFirst`), API token digests with scopes, and named post templates. They own their files and their atomic writes. |
| `session` | The signed session cookie: load, verify, sign, expire. The cookie also carries a fingerprint of the account's password hash, so changing a password retires every session issued before the change. |
| `preview` | The shared preview-link rule: an HMAC over the slug and an expiry stamp. The admin issues links, the API honours them, and neither implements the rule itself. |
| `fediverse` | The `@user@host` rule, a leaf so that the configuration, the admin account form and the post payload validation share one validator. |
| `identifiers` | The DOI and ORCID rules, a leaf like `fediverse`: syntax and normalisation for a DOI, shape and ISO 7064 check digit for an ORCID, and the resolver URLs both are published under. |
| `biblio` | The bibliography leaf: the `refs` frontmatter parsed into numbered entries, inline `[n]` citations linked to them, DOIs, arXiv ids and ORCIDs turned into resolver links, and the `[[refs]]` marker replaced by the rendered list. It imports no other domain package; the post annotates the same-instance links. |
| `app` | Assembles the server: stores, route tree, middleware chain, and the health, robots, favicon, media and 404 handlers. |
| `web` | The shared middleware (gzip, cross-origin refusal, security headers, the request logger and its id, the client address) and access to the embedded templates and assets. |
| `i18n` | The admin interface catalogue: English source strings, Czech translations, and the plural rules both languages need. The public API's messages stay English by contract. |
| `diff` | The line-based comparison behind the revision history's diff view, with a context collapse and a bounded table, so an oversized input degrades to a whole-text replacement instead of burning memory. |
| `ratelimit` | The sliding-window counter behind both the public API limit and the login limit, with a key bound enforced on every request. |
| `webhooks` | Delivers signed JSON events to the configured endpoints, with a bounded number in flight, retries, and an in-memory delivery history. Hooks come from `config.toml` and from the admin-managed `webhooks.toml`, whose changes apply through `SetHooks` without a restart. |
| `backup` | The one writer and reader of the tar.gz archive, shared by the CLI and the admin. It owns the layout and the containment of a restore. |
| `audit` | The append-only JSON-lines audit log. |
| `scheduler` | Publishes posts whose `publish_at` has arrived, from the in-app loop or the CLI, and reports each one to the webhook sink. |
| `updater` | The Gitea release check, the checksum-verified download, and the in-place replacement of the running binary. |
| `password` | scrypt hashing and verification with fixed parameters. |
| `tomlfile` | The shared atomic TOML writer (temp file, `fsync`, rename, mode `0600`). |
| `version` | Reports the version the toolchain recorded in the build information. Nothing writes a version number. |

## Data flow

```mermaid
sequenceDiagram
    participant C as Client
    participant G as web.Gzip
    participant X as web.CrossOrigin
    participant H as web.SecurityHeaders
    participant R as apiRateLimit
    participant M as session.Middleware
    participant L as web.RequestLogger
    participant A as httpapi handleSingle
    participant S as store
    participant P as the rendering pipeline

    C->>G: GET /api/volumen/posts/hello-world
    G->>X: buffers the body when the client accepts gzip
    X->>H: refuses a state-changing request from another origin
    H->>R: baseline security headers, CSP nonce for /admin
    alt over the limit
        R-->>C: 429 with Retry-After
    else within the limit
        R->>M: X-RateLimit-Limit and X-RateLimit-Remaining set
        M->>L: the signed session cookie is loaded and verified
        L->>A: the request id is assigned and carried in the context
        A->>S: Find(slug, lang)
        S->>S: compare the path, mtime and size snapshot with the cache
        S-->>A: the post, an alias to redirect, or nothing
        A->>P: render, splice mathematics and diagrams, sanitise
        P-->>A: the HTML, cached on the post
        A-->>C: 200 with CORS, an ETag and Cache-Control
    end
```

The chain is built outermost first, so a request passes `web.Gzip`,
`web.CrossOrigin`, `web.SecurityHeaders`, `apiRateLimit`, `session.Middleware`
and `web.RequestLogger` before the route mux. The cross-origin gate is the
outer one of the two write guards: it refuses a request a browser sent from
another site before a handler runs, and the admin's per-session CSRF token is
the inner one, which also covers a same-site request from another port. The
request logger assigns a 16-character id, answers with it in `X-Request-Id`,
puts a logger carrying it into the context so a handler's own lines share it,
and writes one access line when the request finishes. The media route is split
off before all of it and wrapped only in the security headers, because the
session and gzip layers buffer a whole response and would hold entire images in
memory.

Errors are produced close to their cause and mapped once, at the edge: the
store returns an error for a failed write, the API turns it into an error
envelope, and the admin renders the same message in the form it came from. A
post file that cannot be parsed is never an error at the edge; the store skips
it, records it, and `volumen validate`, `volumen doctor` and `/healthz` report
it.

Admin handlers add `requireLogin` or `requireAdmin` in front of the handler
and validate the CSRF token before doing any work. Write endpoints
authenticate a `Bearer` token and check its scope. Both paths converge on the
same store calls, so a post saved from the admin and one saved from the API
land on disk the same way, including a rename, which moves the file and
tombstones the old one.

## State and lifetime

- **The content directory is the only durable state.** Posts, media,
  revisions, `users.toml`, `templates.toml`, `tokens.toml`, `webhooks.toml`
  and the audit log all live on disk; a restart loses only the in-memory
  caches, the sitemap memo and the webhook delivery history.
- **One process, one content directory.** Every cache and limiter is
  in-process, so two servers must never share a content directory. The lockers
  are for concurrent requests inside one process, not for two writers on one
  tree.
- **Cached posts are shared.** `store.All` and `store.Find` return the same
  `*post.Post` to several readers, so a writer clones first (`Post.Clone`
  deep-copies the metadata). The rendered HTML is cached on the instance with
  a `sync.Once`, which is safe for concurrent readers by construction.
- **Locks.** The store holds one mutex over its cache and snapshot, a
  reference-counted mutex per write target, and its own guard for that map.
  The users, tokens, templates and audit stores each hold one mutex; the rate
  limiters hold one each and bound their key sets.
- **Sessions live in the cookie**, signed with the secret from
  `[admin].session_key`, or from the `secret.key` the server generated beside
  the users file when the config leaves the key empty, and
  bound to the account's password hash by a fingerprint, so a password change
  ends every session issued before it while the device that made the change
  re-signs itself. A request that carries a valid cookie is authenticated
  without server state, so signing out expires the browser's copy rather than
  revoking the value, and rotating the key is what ends every session at once.
- **Long-lived goroutines** are the scheduler loop and the HTTP server; both
  are joined on shutdown, which drains in-flight requests for up to 15 seconds
  after `SIGINT` or `SIGTERM`. Webhook deliveries and the release check are
  bounded, detached, and die with the process.
- **Post files are written atomically**: a temp file in the target directory,
  `fsync`, rename, directory `fsync`. The previous version is archived before
  the rename, and a delete is a move into that archive, so both are reversible.
- **The content directory is reached through an `os.Root`.** Every read, write
  and delete inside it goes through the handle, which refuses a path that
  would escape the tree through `..` or a symlink, so confinement is a
  property of the store rather than a check each caller has to remember.

## Dependencies

The direct requires in `go.mod` are the whole list, and each is there because
the standard library does not do the job:

- `sourcedock.dev/petrbalvin/scriptorium` renders the Markdown body, the TeX
  mathematics and the Mermaid diagrams, deterministically and on the standard
  library alone.
- `bluemonday` sanitises the
  rendered HTML against an allowlist. It is the reason a post body can be
  treated as untrusted even though its author is authenticated.
- `sourcedock.dev/petrbalvin/interpres/v2` parses TOML. It keeps a real date a
  date, rather than turning `date = 2026-01-15` into a locale-dependent guess,
  and its `Document` keeps key order and comments so a save round-trips the
  author's own frontmatter.
- `golang.org/x/crypto` provides scrypt, which the standard library does not
  ship, for password hashing.
- `golang.org/x/text` provides the NFKC normalisation applied to passwords
  before hashing, so a password typed with a different Unicode form still
  verifies.

Everything else is the standard library: `net/http` for the server and its
routing, `html/template` and `embed` for the admin, `archive/tar` and
`compress/gzip` for the archive, `encoding/json/v2` for the API, and
`log/slog` for diagnostics.