253 lines
14 KiB
Markdown
253 lines
14 KiB
Markdown
# Architecture
|
||
|
||
How gasm-devkit is put together and why.
|
||
|
||
## Design goals
|
||
|
||
1. **A real AST, not a grammar hack.** The linter, analyser, assembler and
|
||
language server all need to *reason* about assembly — not just colour it.
|
||
So the centre of the toolkit is a hand-written lexer and a parser that
|
||
produce a typed AST with source positions on every node.
|
||
2. **Architecture as data, not code.** Per-architecture differences (amd64,
|
||
arm64, riscv64, loong64) live in register and instruction *tables* (`arch`),
|
||
never in `if arch == …` branches scattered through the logic. The
|
||
instruction tables are generated from the Go toolchain's own assembler
|
||
source (`just gen`), so adding or refreshing an architecture is a data
|
||
operation, not a coding one.
|
||
3. **Open integration surface.** Everything the toolkit can do is reachable
|
||
through two vendor-neutral interfaces: a CLI and an LSP server. No editor
|
||
owns the toolkit; the toolkit is offered to editors on standard terms.
|
||
|
||
## Pipeline
|
||
|
||
```mermaid
|
||
graph TD
|
||
SRC["source .s"] --> LEX["lexer<br/>token stream"]
|
||
LEX --> PAR["parser<br/>AST + diagnostics"]
|
||
LEX --> FMT["format<br/>re-space tokens"]
|
||
PAR --> LINT["lint<br/>static checks"]
|
||
PAR --> LSP["lsp server"]
|
||
LEX --> LSP
|
||
ARCH["arch tables<br/>amd64 / arm64 / riscv64 / loong64"] --> LINT
|
||
ARCH --> LSP
|
||
LINT --> LSP
|
||
FMT --> CLI["gasm CLI"]
|
||
LINT --> CLI
|
||
PAR --> CLI
|
||
LEX --> CLI
|
||
LSP --> EDITOR["any LSP editor"]
|
||
```
|
||
|
||
The lexer is the shared foundation: the parser builds the AST from it, the
|
||
formatter re-spaces its tokens directly, and the language server uses it for
|
||
semantic highlighting.
|
||
|
||
## Components
|
||
|
||
### `token` and `lexer`
|
||
|
||
The scanner is hand-written and permissive: it never panics and maps anything
|
||
it cannot classify to an `Illegal` token, so every downstream tool still works
|
||
on malformed input. Newlines are significant tokens, because Plan 9 assembly
|
||
is line-oriented and the parser relies on line structure.
|
||
|
||
The middle dot (`·`, U+00B7) is treated as an identifier character so that
|
||
`·funcName(SB)` lexes as one symbol. Multi-character operators (`<<`, `>>`,
|
||
`->`) are recognised so arm64 shift operands scan correctly. A backslash
|
||
immediately before a newline is a C-preprocessor line continuation (used by
|
||
`#define` macros in the runtime `.s` files); the lexer splices the lines
|
||
together so a multi-line macro becomes one logical line the parser treats as an
|
||
opaque preprocessor directive.
|
||
|
||
### `ast` and `parser`
|
||
|
||
The parser is **line-oriented**, matching how the Plan 9 assembler reads a
|
||
file: it groups tokens into lines, classifies each line (directive, label,
|
||
instruction, comment, preprocessor) and dispatches. A malformed line is
|
||
reported and skipped; it never aborts the file.
|
||
|
||
Operands are parsed into a faithful, flat representation. The amd64
|
||
addressing modes — `reg`, `$imm`, `(base)`, `off(base)`, `(base)(index*scale)`,
|
||
`name+off(FP)`, `name<>(SB)` — are all captured structurally, and the original
|
||
token text is retained for fidelity.
|
||
|
||
A deliberate boundary: the AST records **syntax only**. Whether a bare
|
||
identifier is a register or a label is an *architecture* question, so it is
|
||
left to `arch` and resolved in the lint/lsp layers. This keeps the parser
|
||
arch-agnostic and its output deterministic.
|
||
|
||
### `arch`
|
||
|
||
Register files are generated programmatically (the regular `R8`–`R15`,
|
||
`X0`–`X15`, `Y0`–`Y15`, `Z0`–`Z31`, `K0`–`K7` ranges) plus the irregularly
|
||
named registers listed explicitly. Instruction names are **generated from the
|
||
Go toolchain's own assembler source** (`cmd/internal/obj/<arch>/anames.go`,
|
||
plus the common opcodes and the per-architecture front-end aliases such as the
|
||
arm64 `B`/`BL` branches and the `.P`/`.W` load-store addressing suffixes) by
|
||
`just gen`, so the tables always match what the real assembler accepts. Each
|
||
mnemonic maps to a summary and an optional operand-count range; counts are
|
||
recorded only where unambiguous (`-1` disables the operand-count lint for that
|
||
instruction) so the linter stays silent rather than guess. For architectures
|
||
with highly variable operand forms (arm64, riscv64, loong64) only a few
|
||
fixed-arity instructions (`RET`, `NOP`, `JMP`, `CALL`) carry counts at all.
|
||
|
||
### `lint`
|
||
|
||
Rules are conservative by design — silence beats a false positive. The rules
|
||
are `unknown-instruction`, `operand-count`, `undefined-label`,
|
||
`duplicate-label`, `missing-ret`, `missing-textflag-include`, `abi-argsize` and
|
||
`unreachable-code`. Every diagnostic carries a stable code so callers can
|
||
disable rules individually, and arch-specific rules switch off entirely when
|
||
the target architecture cannot be inferred from the file name.
|
||
|
||
Two things keep the rules honest on real-world code:
|
||
|
||
- **Pseudo-ops and macros are not instructions.** `unknown-instruction` knows
|
||
the assembler pseudo-ops (`BYTE`, `WORD`, `FUNCDATA`, `PCDATA`, …) and
|
||
recognises macro invocations — an in-file `#define` name, or any identifier
|
||
containing an underscore (no Plan 9 mnemonic ever does).
|
||
- **Macro-heavy files get the label/RET heuristics turned off.** Without a
|
||
preprocessor, labels a macro defines are invisible, so `undefined-label` and
|
||
`missing-ret` are suppressed for files that use macros (an in-file `#define`
|
||
or a `#include` of anything other than `textflag.h`). `missing-ret` also
|
||
treats a trailing unconditional jump and `UNDEF` as valid terminators.
|
||
|
||
The result is validated by `TestGoRuntimeCorpus`, which parses and lints every
|
||
`src/runtime/*.s` file the toolchain ships for all four architectures and
|
||
asserts zero parse errors and zero error-severity diagnostics.
|
||
|
||
Two deeper analyses sit on top of the AST:
|
||
|
||
- **`abi-argsize`.** Hand-written kernels document their signature in a
|
||
`// func …` comment above the `TEXT`. The linter parses that signature with
|
||
the standard library's Go parser, lays out the parameters and results under
|
||
Go's ABI0 stack rules (results begin on a word boundary after the
|
||
parameters), and checks the total against the argument size declared in the
|
||
`TEXT` directive. It only runs for stack-argument functions (a non-zero
|
||
declared arg area that is actually addressed through `FP`), and aborts
|
||
silently on a type whose size it cannot determine — so it never guesses.
|
||
- **`unreachable-code`.** Code after a `RET` and before the next label is
|
||
dead. The check is suppressed for any function whose reachability cannot be
|
||
decided statically: those using PC-relative jumps (`JMP 2(PC)`),
|
||
register-indirect branches (`JALR`/`JR`/`JIRL`/`BR`/`BLR`), or living in a
|
||
file with `#ifdef` conditionals. `UNDEF` is deliberately not a terminator —
|
||
code after it is occasionally intentional metadata.
|
||
- **`register-clobber` (register liveness).** The linter builds the function's
|
||
control-flow graph (basic blocks split at labels and after branches, with
|
||
fall-through and jump-target edges), computes a conservative per-instruction
|
||
register def/use, and runs the standard backward liveness iteration to a fixed
|
||
point. On top of that it flags writes to the registers the **Go ABI** fixes
|
||
across calls that are never saved and restored — calibrated from
|
||
`cmd/compile/abi-internal.md`, *not* the platform ABI: Go's stack-based ABI0
|
||
has no System V style callee-saved registers (amd64 `BX`, `R12`–`R15` and
|
||
the like are caller-saved or permanent scratch, and hand-written kernels may
|
||
clobber them freely). The audited set is the frame pointer and the
|
||
the frame pointer, the goroutine pointer per architecture (amd64 `BP`/`R14`, arm64 `R18`/`R28`/
|
||
`R29`, riscv64 `X27`, loong64 `R22`); the goroutine pointer is reported only
|
||
when the function can reach the runtime — it is not `NOSPLIT` or makes a
|
||
call — since the ABI0 transition machinery restores it on those paths, and
|
||
NOSPLIT call-free leaves may use it (the runtime's own assembly does). It
|
||
runs only on macro-free files, where
|
||
no opaque macro can perform the save/restore.
|
||
- **`funcdata-pcdata`.** `FUNCDATA $idx, sym(SB)` and `PCDATA $idx, $val` are
|
||
checked for well-formed operands (arity, immediate index and value, symbol
|
||
reference) and a literal index is range-checked; a named index constant such
|
||
as `$PCDATA_StackMapIndex` is accepted without a range check.
|
||
|
||
### `format`
|
||
|
||
The formatter works on the **token stream, not the AST**, so it preserves
|
||
every line — comments and blanks included. It normalises indentation, operand
|
||
spacing, per-function mnemonic alignment and blank-line layout: a new block
|
||
(a label, `TEXT` or `GLOBL`) is preceded by exactly one blank line (comments
|
||
leading a block stay with it), runs of blanks collapse to one, and a `RET`
|
||
terminates the body so the next function's doc comment stays at column 0. It
|
||
is idempotent and its output always round-trips through the parser. With a
|
||
directory argument — or none — it reformats every `.s` file below it in
|
||
place and lists the files changed, the way `go fmt` does (`.` and `_`
|
||
directories are skipped).
|
||
|
||
### `lsp`
|
||
|
||
The server speaks JSON-RPC 2.0 with `Content-Length` framing over any
|
||
`io.Reader`/`io.Writer` (normally stdin/stdout). It maintains an in-memory
|
||
document store, republishes diagnostics on every change, and provides:
|
||
|
||
- **completion** — instructions, registers, pseudo-registers, textflag macros
|
||
and local labels;
|
||
- **hover** — instruction summaries and register descriptions from `arch`;
|
||
- **document symbols** — `TEXT` functions with their labels, plus `GLOBL`/`DATA`;
|
||
- **semantic tokens** — syntax highlighting delivered as LSP semantic tokens,
|
||
classified with the lexer plus `arch` (instructions, registers by class,
|
||
pseudo-registers, labels, immediates, comments, directives, textflag macros).
|
||
|
||
Semantic tokens are the key to editor-agnostic highlighting: the editor renders
|
||
them from the standard LSP legend, so no editor-specific grammar is needed.
|
||
|
||
### `asm`
|
||
|
||
The standalone assembler (Phase 2). Its core is an amd64 instruction encoder:
|
||
a REX/ModR-M/SIB/displacement/immediate engine plus the scalar instruction set,
|
||
with the Plan 9 operand order (source first) mapped onto the x86 encoding.
|
||
Every encoding is validated by decoding it again with `golang.org/x/arch` — the
|
||
one module dependency, used in tests only and never linked into the binary.
|
||
|
||
On top of the encoder, `Assemble` walks a parsed `TEXT` body, converts each
|
||
operand to an encoder operand, and lays the instructions out so local labels
|
||
resolve to relative jump offsets: jumps start in the short (rel8) form and
|
||
expand to rel32 when the settled displacement does not fit, iterating to a
|
||
fixed point, and jump-to-jump chains are folded (a conditional jump to a label
|
||
whose only instruction is an unconditional jump is redirected to the ultimate
|
||
target) exactly as the Go toolchain's linker does before it encodes branches.
|
||
The `FP`/`SP` pseudo-
|
||
registers are translated onto the hardware stack pointer — `x+N(FP)` becomes
|
||
`(N+8)(SP)` for a zero-frame function and `(N+frame+16)(SP)` once a frame
|
||
pointer is set up, with the matching Go prologue/epilogue generated — so the
|
||
output is byte-identical to the Go assembler for these cases. SIMD is handled
|
||
by a VEX (AVX/AVX2) encoder — the two- and three-byte VEX prefixes with XMM/YMM
|
||
registers — across eight operand forms: the three-operand NDS form, the
|
||
two-operand reg/rm form, the immediate-shift form (plus the variable-count
|
||
shifts, which share the NDS shape with the count in an XMM register or
|
||
memory), the immediate shuffle form (`VPSHUFD`, `VPERMQ`), the
|
||
three-operand-plus-immediate form (`VSHUFPD`,
|
||
`VPERM2I128`, `VINSERTI128`), the lane-extract form (`VEXTRACTI128`,
|
||
`VEXTRACTF128`, where the YMM source occupies the reg field and the XMM or
|
||
memory destination r/m), the direction-sensitive moves (`VMOVDQU`, `VMOVUPD`,
|
||
`VMOVD`, `VMOVQ`, `VMOVSD`), the floating-point and FMA arithmetic (`VADDPD`,
|
||
`VMULPD`, `VXORPD`, `VUNPCKHPD`, the scalar `VADDSD`/`VMULSD`, `VCVTDQ2PD`,
|
||
`VFMADD231PD`) and the no-operand `VZEROUPPER` — together with `VPERMD` and
|
||
the scalar families (`CMOVcc`, `SETcc`, `LZCNT`/`TZCNT`, the extending moves,
|
||
`CVTSx2SD`, `IMUL3`) and the EVEX (AVX-512) prefix — the four-byte prefix with
|
||
5-bit register fields (Z0–Z31, X/Y 16–31, with the mod=11 quirk that carries
|
||
rm[4] in X̄), opmask registers (K0–K7 as operands, mask destinations and
|
||
explicit merging/zeroing masks — written the way Go writes them, as a K
|
||
operand among the operands plus a `.Z` mnemonic suffix), and the compressed
|
||
disp8×N displacement, whose multiplier follows the memory operand's size —
|
||
covering every instruction the go-flac and go-lz4 AVX2/AVX-512 kernels use,
|
||
plus the common AVX-512 F/BW integer set. Every encoding is validated two ways: by
|
||
round-trip decoding through `golang.org/x/arch`, and byte-for-byte against
|
||
the machine code the real Go assembler emits — a comparison that holds for
|
||
whole functions: all 27 functions of both kernels assemble to exactly the Go
|
||
toolchain's bytes, the lone exception being the displacements of the
|
||
static-constant loads, which the Go linker fills at link time.
|
||
|
||
File-level assembly (`AssembleFile`) goes beyond single functions: it
|
||
materialises the file's static symbols (`GLOBL`/`DATA`) in a data section
|
||
behind the code and resolves references to them (`mask<>(SB)`) to
|
||
RIP-relative loads whose displacements point inside the resulting image, so
|
||
the bytes are self-consistent at any base address. External (non-file-local)
|
||
symbols are rejected: they need object-file emission, which — together with
|
||
EVEX masking/zeroing and the other architectures — is the rest of Phase 2.
|
||
|
||
## Extension points
|
||
|
||
- **New architecture:** add an entry to the generator in `_gen`, run
|
||
`just gen`, and add a `buildXXX()` register file plus a case in `ForArch`.
|
||
- **New lint rule:** add a function in `lint` and a rule-code constant.
|
||
- **New LSP feature:** add a method case in `dispatch` and a handler.
|
||
|
||
The phases follow a dependency chain. Phase 1 (static analysis) builds only on
|
||
the AST; Phase 2 (the standalone assembler) emits object code; Phases 3
|
||
(dynamic analysis) and 4 (the debugger) both consume the execution substrate
|
||
that the assembler provides.
|