petrbalvin/scripts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: describe the sglang endpoint behind caddy

Browse Source 
Assisted-by: GLM 5.3 Flash
This commit is contained in:
petrbalvin
2026-09-29 00:18:15 +02:00
parent 9455c65f10
commit 37e06d090c
6 changed files with 35 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+15 -3
View File
@@ -37,7 +37,7 @@ reports everything as already in place and changes nothing.
| `system-diag.pl` | Reading CPU, memory, disk, network, GPU, services, security and performance, and grading the result | Change anything on the host; it is read-only |
| `server-setup.pl` | Base packages, firewall (with the SSH rule verified before the service starts), SELinux, Podman, automatic updates | Install the services themselves; that is the operator's, and the deploy scripts' |
| `workstation-setup.pl` | A Fedora desktop: Brave, the official Go toolchain, Rust, GoLand, Flatpak applications, firewall, SELinux | Anything on a server distribution; it targets Fedora Workstation |
| `sglang-deploy.pl` | The SGLang deployment: the container engine, the systemd unit, nginx with TLS, the API key, the firewall rule and the SELinux boolean | The host's own setup, which `server-setup.pl` does first |
| `sglang-deploy.pl` | The SGLang deployment: the container engine, the systemd unit, Caddy with TLS, the API key, the firewall rule and the SELinux checks | The host's own setup, which `server-setup.pl` does first |
| `system-optimise.pl` | Old kernels (the running one and one fallback always stay), journals, temporary files, core dumps, and the package audit | Touch an rpm-ostree system, which it refuses |
## Data flow: a forked section collection
@@ -97,8 +97,8 @@ that CentOS Stream and openEuler cannot carry at all.
```mermaid
flowchart LR
Client[client] -->|443| Nginx[nginx on the host, TLS]
Nginx -->|loopback, plain HTTP| Engine[SGLang in a Podman container]
Client[client] -->|443| Caddy[Caddy on the host, TLS]
Caddy -->|loopback, plain HTTP| Engine[SGLang in a Podman container]
Engine -->|device nodes| GPU[/dev/kfd, /dev/dri]
Engine -->|bind mount| Cache[state directory, model cache]
Unit[systemd unit] -->|podman run| Engine
@@ -114,12 +114,24 @@ instead, and `--image` pins one. Radeon cards need `SGLANG_USE_AITER=false` and
`SGLANG_ROCM_FUSED_DECODE_MLA=false` in the unit, which the script writes for them and
never for an Instinct host.
Caddy serves the endpoint from a drop-in under `/etc/caddy/Caddyfile.d`, which the
distribution's default Caddyfile imports. Fedora carries the caddy package and CentOS
Stream gets it from EPEL, whose repository file the script installs first; openEuler
packages no caddy at all, so there the official release binary is installed instead,
with the unit file the package would have carried. The service runs as the caddy user,
so the private key is made group-readable for the caddy group, and on an
SELinux-enforcing host the packaged caddy runs unconfined: no boolean is needed. A
deployment made by an earlier release of this script carries an nginx configuration,
which both a deploy and an uninstall remove.
## Dependencies
Nothing outside the interpreter, and nothing that has to be installed beyond the tools
each script's own dependency section installs. The non-obvious ones and their reasons:
- `podman` for `sglang-deploy.pl`, because the engine is a container.
- `caddy` for `sglang-deploy.pl`, because the endpoint serves TLS on 443; the
release binary and `tar` stand in where no repository carries the package.
- `lspci` for the GPU family, and `rocm-smi` in `system-diag.pl` for AMD memory and
utilisation figures.
- `sha256sum` in `workstation-setup.pl`, because Perl's builtins have no hash.
docs/CLI.md
+5 -5
View File
@@ -111,7 +111,7 @@ verified by checksum; Brave's repository key is verified by fingerprint before i
```
Usage: sglang-deploy.pl [options]
--model ID Hugging Face model ID (menu when omitted)
--model ID ModelScope model ID (menu when omitted)
--port N internal engine port (default: 8000, not 443)
--tensor-parallel N GPUs for tensor parallelism (default: 1, written as
the engine's --tp-size)
@@ -130,19 +130,19 @@ Usage: sglang-deploy.pl [options]
--api-key KEY API key for the endpoint (default: generate and
store in /etc/sysconfig)
--dry-run preview without making changes
--uninstall tear down the service, container, nginx config
--uninstall tear down the service, container, caddy drop-in
and certificates
--help show this help
--version show the version
```
Needs root. Without `--model` an interactive menu offers GLM 5.3, GLM 5.3 Flash,
DeepSeek V4 Pro, DeepSeek V4 Flash, MiMo V2.5 Pro and MiMo V2.5, plus a free-form
entry. `--tensor-parallel`, `--max-model-len` and `--gpu-memory-utilization` are
Qwen 3.8 Max, Qwen 3.8 Flash and DeepSeek V4.1 Flash, plus a free-form entry.
`--tensor-parallel`, `--max-model-len` and `--gpu-memory-utilization` are
written to the unit as the engine's own `--tp-size`, `--context-length` and
`--mem-fraction-static`. The API key is shown once when it is generated; a key given
with `--api-key` is never echoed. `--uninstall` stops and disables the service, removes
the unit, the container, the nginx configuration and the certificates, and keeps the
the unit, the container, the caddy drop-in and the certificates, and keeps the
image and the model cache.
## system-optimise.pl
docs/DEPLOYMENT.md
+3 -3
View File
@@ -84,7 +84,7 @@ deployment an operator cares about, so the pipeline that publishes them never ru
| Script | What it leaves behind | Where it is documented |
|---|---|---|
| `server-setup.pl` | Packages, firewalld with the SSH rule already allowed, SELinux enforcing, Podman, unattended updates | [docs/CLI.md](CLI.md) |
| `sglang-deploy.pl` | A systemd unit running the engine in a Podman container, nginx with TLS in front, an API key file, the firewall rule and the SELinux boolean | [docs/ARCHITECTURE.md](ARCHITECTURE.md) |
| `sglang-deploy.pl` | A systemd unit running the engine in a Podman container, Caddy with TLS in front, an API key file, the firewall rule and the SELinux checks | [docs/ARCHITECTURE.md](ARCHITECTURE.md) |
| `workstation-setup.pl` | A Fedora desktop with the toolchains and applications installed | [docs/CLI.md](CLI.md) |
The SGLang deployment is the only service in the collection, and the host needs no ROCm
@@ -100,6 +100,6 @@ where the script reads them:
- The published copies are unversioned: whatever is on `main` is what is served.
- `sglang-deploy.pl` keeps the engine's API key in `/etc/sysconfig/sglang`, mode 0600,
written by the script and never committed. A Hugging Face token for a gated model goes
into that same file as `HF_TOKEN`, which the unit forwards to the container.
written by the script and never committed. A ModelScope token for a gated model goes
into that same file as `MODELSCOPE_TOKEN`, which the unit forwards to the container.
- The deploy secrets live in the Gitea repository settings, under Actions, Secrets.
docs/DEVELOPMENT.md
+9 -7
View File
@@ -61,13 +61,15 @@ only be exercised as root.
`tests/container/rig.pl` runs the real `sglang-deploy.pl` as root inside a container
against a stub `PATH`: every command the script drives (`dnf`, `rpm`, `podman`,
`systemctl`, `curl`, `openssl`, `nginx`, `firewall-cmd`, `getsebool`, `lspci`) is a
stub that answers from a fixture, so a run is deterministic and needs no network and no
GPU. It covers the deploy end to end: the resolved image tag, the unit file, the nginx
configuration, the TLS certificate and its modes, the API key file, the SELinux
boolean, the firewall rule, the idempotent second run, the dry run, the uninstall, the
Radeon and MI300 paths, the offline and unpublished-tag failures, the argument
validation and the non-root refusal.
`systemctl`, `curl`, `openssl`, `caddy`, `tar`, `useradd`, `semodule`, `firewall-cmd`,
`getsebool`, `lspci`) is a stub that answers from a fixture, so a run is deterministic
and needs no network and no GPU. It covers the deploy end to end: the resolved image
tag, the unit file, the caddy drop-in and the main Caddyfile's import, the release
binary install for the hosts without a caddy package, the EPEL bootstrap on CentOS
Stream, the legacy nginx clean-up, the TLS certificate and its modes, the API key file,
the SELinux decisions, the firewall rule, the idempotent second run, the dry run, the
uninstall, the Radeon and MI300 paths, the offline and unpublished-tag failures, the
argument validation and the non-root refusal.
Prepare the platform image once, since the base images carry no Perl: