scripts/docs/ARCHITECTURE.md

# Architecture

How the scripts collection is put together. Every script, helper and arrow below
exists in the repository; nothing is aspirational.

## Overview

There is no shared library. Each script is one file that can be fetched and run on its
own, and the six of them share a skeleton rather than a module: the same command
runner, the same progress reporting, the same state checks before every action, and the
same shape of a run.

```mermaid
flowchart TD
    Start[invocation] --> Args[parse_args reads ARGV]
    Args --> Version{version or help?}
    Version -->|yes| Print[print and exit]
    Version -->|no| Root[check_root, detect_os]
    Root --> Sections[sections run in order]
    Sections --> Each[each section checks state first]
    Each --> Done{changed?}
    Done -->|no| Report[report what is already in place]
    Done -->|yes| Act[act, then verify what was done]
    Act --> Report
    Report --> Summary[print_summary on stderr]
```

A run is idempotent because of that loop: a section reads the state of the system,
compares it with the desired one, and acts only on the difference, so a second run
reports everything as already in place and changes nothing.

## The seven scripts

| Script | Owns | Deliberately does not |
|---|---|---|
| `network-diag.pl` | Latency, loss, DNS timing, MTU discovery, dual-stack reachability, listening ports, and the grade of the connection | Touch any configuration; it only measures |
| `system-diag.pl` | Reading CPU, memory, disk, network, GPU, services, security and performance, and grading the result | Change anything on the host; it is read-only |
| `server-setup.pl` | Base packages, firewall (with the SSH rule verified before the service starts), SELinux, Podman, automatic updates | Install the services themselves; that is the operator's, and the deploy scripts' |
| `workstation-setup.pl` | A Fedora desktop: Brave, the official Go toolchain, Rust, GoLand, Flatpak applications, firewall, SELinux | Anything on a server distribution; it targets Fedora Workstation |
| `sglang-deploy.pl` | The SGLang deployment: the container engine, the systemd unit, nginx with TLS, the API key, the firewall rule and the SELinux boolean | The host's own setup, which `server-setup.pl` does first |
| `system-optimise.pl` | Old kernels (the running one and one fallback always stay), journals, temporary files, core dumps, and the package audit | Touch an rpm-ostree system, which it refuses |

## Data flow: a forked section collection

`system-diag.pl` is the one script whose shape is not linear. Its sections are
independent and dominated by waiting, so each runs in its own child process, writes its
result into a private scratch directory as a Perl literal, and the parent reads the
results back after reaping. The runner keeps stdout and stderr apart in that same
directory, which is why the file names carry the process id.

```mermaid
sequenceDiagram
    participant Parent
    participant Child as Section child
    participant Disk as Scratch directory
    Parent->>Child: fork, one per requested section
    Child->>Child: collect one section
    Child->>Disk: write the result as a Perl literal
    Parent->>Parent: waitpid, report the section as it lands
    Parent->>Disk: read_literal for each child
    Parent->>Parent: compute the grade and render the report
    Parent->>Disk: remove the scratch directory
```

Only the process that created the scratch directory removes it: a forked child inherits
the `END` block, so the cleanup is guarded by a parent pid comparison. A child that
cannot write its result is reported as a missing section, and the grade drops that
component rather than inventing a value for it.

## Conventions every script follows

- **Builtins only.** No module beyond the interpreter is loaded, because the systems
  this targets package modules separately. What Perl has no builtin for is written out:
  the command runner, RPM's version comparison, the JSON decoders and the encoder, the
  IPv4 and IPv6 arithmetic, `sockaddr` packing, a `which`, and argument parsing.
- **External binaries as transport.** `dnf`, `rpm`, `curl`, `openssl`, `podman`,
  `systemctl`, `journalctl`, `find`, `stat`, `sha256sum`, `tar`, `gpg`, `lspci` and the
  rest are driven as argument lists through `run`, which keeps stdout and stderr apart,
  forces `LANG=C` and `LC_ALL=C` for parseable output, and turns a timeout, a missing
  binary and a failed exec into exit codes 124, 127 and 126 rather than exceptions.
- **One optional module.** `Time::HiRes` is loaded inside an `eval` in the scripts that
  time something. Where it is absent the affected figures are reported as not measured
  and the grade drops that component, rather than the script failing.
- **Atomic writes for system configuration.** A file that a boot depends on is written
  to a temporary name, flushed, given its mode and owner, and renamed over the target.
  Perl has no `fsync`, so that barrier is delegated to `sync` where the binary exists.
- **Errors carry what the tool said.** A failure reproduces the errno, the reason and
  the path, so a message can be searched for as it stands.

## The SGLang deployment

`sglang-deploy.pl` is the only script that deploys a service, and its engine runs as a
container. That is a consequence of the material rather than a preference: SGLang
publishes no ROCm wheel, and its AMD install paths are the project's container images
or a source build against a full ROCm toolchain that Fedora carries only partly and
that CentOS Stream and openEuler cannot carry at all.

```mermaid
flowchart LR
    Client[client] -->|443| Nginx[nginx on the host, TLS]
    Nginx -->|loopback, plain HTTP| Engine[SGLang in a Podman container]
    Engine -->|device nodes| GPU[/dev/kfd, /dev/dri]
    Engine -->|bind mount| Cache[state directory, model cache]
    Unit[systemd unit] -->|podman run| Engine
    EnvFile[EnvironmentFile, mode 0600] -->|API key| Unit
```

The host therefore needs the amdgpu kernel driver and its device nodes, never a ROCm
userland. The image tag is resolved from the card's family (mi30x for an MI300 or
MI325, mi35x for an MI350 or MI355) and from the host's ROCm, because the container's
ROCm userland must not be newer than the host's kernel driver. A Radeon 8060S or 8050S
(gfx1151) has no stable tag anywhere; AMD's dated development builds are resolved
instead, and `--image` pins one. Radeon cards need `SGLANG_USE_AITER=false` and
`SGLANG_ROCM_FUSED_DECODE_MLA=false` in the unit, which the script writes for them and
never for an Instinct host.

## Dependencies

Nothing outside the interpreter, and nothing that has to be installed beyond the tools
each script's own dependency section installs. The non-obvious ones and their reasons:

- `podman` for `sglang-deploy.pl`, because the engine is a container.
- `lspci` for the GPU family, and `rocm-smi` in `system-diag.pl` for AMD memory and
  utilisation figures.
- `sha256sum` in `workstation-setup.pl`, because Perl's builtins have no hash.
- `sync` for the write barrier after an atomic write.
- `getent` or `host` in `network-diag.pl` as the resolver transport, since `getent` on
  FreeBSD implements neither of the `ahosts` databases.