goget/docs/architecture.md

# Architecture

**goget** is a command-line download utility built on the Go standard library. It follows a layered architecture with a protocol abstraction layer, a transport layer, and utility modules for authentication, compression, output, and more.

## High-Level Overview

```mermaid
graph TD
    User[User / Terminal]
    Script[Script / CI]
    CLI[cmd/goget/ - Main entry]
    API[pkg/api/ - Library interface]
    Core[internal/core/ - Types, Protocol Interface, Errors]
    Registry[internal/protocol/ - Protocol Registry]
    HTTP[internal/protocol/http/]
    FTP[internal/protocol/ftp/]
    SFTP[internal/protocol/sftp/]
    FILE[internal/protocol/file/]
    DataURL[internal/protocol/dataurl/]
    Gopher[internal/protocol/gopher/]
    Gemini[internal/protocol/gemini/]
    WebDAV[internal/protocol/webdav/]
    Transport[internal/transport/ - Dialer, Proxy, TLS, Rate Limit]
    Config[internal/config/]
    Output[internal/output/ - Atomic write, Resume, WARC]
    Auth[internal/auth/ - Basic, Digest, OAuth, netrc]
    Crypto[internal/crypto/ - Checksum, PGP, Certs]
    Cookie[internal/cookie/ - Netscape jar]
    HSTS[internal/hsts/ - RFC 6797 cache]
    Archive[internal/archive/ - Extraction]
    Recursive[internal/recursive/ - Crawler, Parser, Filter]
    Mirror[internal/mirror/]
    Metalink[internal/metalink/]
    Queue[internal/queue/]
    Compression[internal/compression/ - gzip, bzip2, zlib, flate, lzw]
    Log[internal/log/ - Structured logging]
    WARC[internal/warc/ - Archiving]
    Register[internal/protocol/register/ - Single registration]
    Format[internal/format/ - Bytes, Speed formatting]
    LinkRewrite[internal/linkrewrite/ - HTML link conversion]
    CLIUtil[internal/cli/ - Help, Progress bar, Colors]
    Pool[internal/pool/ - Unified worker pool]

    User --> CLI
    Script --> API
    CLI --> Log
    CLI --> CLIUtil
    CLI --> Config
    CLI --> Core
    CLI -.-> API
    API --> Core
    API --> Registry

    subgraph "Download Pipeline"
        Core --> Registry
        Registry --> HTTP
        Registry --> FTP
        Registry --> SFTP
        Registry --> FILE
        Registry --> DataURL
        Registry --> Gopher
        Registry --> Gemini
        Registry --> WebDAV
        HTTP --> Transport
        HTTP --> Cookie
        HTTP --> HSTS
        HTTP --> Output
        FTP --> Transport
        FTP --> Pool
        SFTP --> Transport
        SFTP --> Pool
        WebDAV --> Pool
    end

    subgraph "Post-Download"
        CLI --> Crypto
        CLI --> Archive
        CLI --> Compression
    end

    subgraph "Download Management"
        CLI --> Register
        CLI --> WARC
        CLI --> Recursive
        CLI --> Mirror
        CLI --> Metalink
        CLI --> Queue
        Recursive --> WARC
        Recursive --> LinkRewrite
        Mirror --> LinkRewrite
    end

    Transport --> Auth
    Transport --> Config
```

## Layers

### 1. Entry Points

| Layer | Package | Role |
|---|---|---|
| **CLI** | `cmd/goget/` | Flag parsing, command routing, signal handling |
| **Library API** | `pkg/api/` | Public Go API for programmatic downloads, uploads, and queue management |

The CLI and API share the same internal packages. The CLI is built on top of the same `internal/core` types that the API exposes.

### 2. Protocol Abstraction

Defined in `internal/core` and `internal/protocol`:

```go
// core.Protocol is the interface every protocol handler implements.
type Protocol interface {
    Scheme() string
    CanHandle(url *url.URL) bool
    Download(ctx context.Context, req *DownloadRequest) (*DownloadResult, error)
    Capabilities() []string
    SupportsResume() bool
    SupportsCompression() bool
    SupportsParallel() bool
    SupportsRecursive() bool
}

// core.Uploader is an optional interface for protocols supporting uploads.
type Uploader interface {
    Upload(ctx context.Context, req *UploadRequest) (*UploadResult, error)
    SupportsUpload() bool
}
```

Protocols register themselves by scheme in a global `Registry`. Scheme normalization maps HTTPS → HTTP, FTPS → FTP, etc. Supported protocols include HTTP, FTP, SFTP, File, DataURL, Gopher, Gemini, and WebDAV (new in `internal/protocol/webdav/`).

### 3. Transport Layer

`internal/transport` provides a custom HTTP transport with:

- **IPv6-first dialer** — Resolves DNS, prefers IPv6 addresses, falls back to IPv4 only when no IPv6 record exists
- **Interface binding** — Linux-specific socket binding via `SO_BINDTODEVICE`
- **Rate limiting** — Token bucket algorithm for bandwidth capping
- **Retry logic** — Exponential backoff with configurable max retries
- **TLS configuration** — Certificate pinning, client certificates, key log file
- **Proxy support** — HTTP CONNECT and SOCKS5 (`socks5://` with local DNS, `socks5h://` with remote DNS, RFC 1929 username/password auth)

### 4. Output Layer

`internal/output` handles atomic file writes:

1. Write to a `.goget.tmp` file during download
2. Store resume metadata in `.goget.meta` alongside the temp file
3. Atomically rename from `.goget.tmp` to the final filename on completion

### 5. Utility Modules

| Module | Responsibility |
|---|---|
| `auth` | HTTP Basic, Digest, OAuth 2.0 (client credentials, authorization code, refresh token), `.netrc` auto-reading |
| `cookie` | Netscape-format cookie jar with import/export |
| `crypto` | Checksum verification (SHA-256, SHA-512, SHA3-256/512, BLAKE2b, MD5), PGP signature verification and decryption, certificate handling |
| `compression` | Decompression wrappers for gzip, bzip2, zlib, flate, LZW |
| `hsts` | HSTS cache (RFC 6797) for automatic HTTPS upgrade |
| `archive` | Archive extraction for tar, tar.gz, tar.bz2, zip with auto-detection |
| `format` | Human-readable byte and speed formatting |
| `linkrewrite` | HTML link rewriting for offline mirror viewing |
| `log` | Structured logger with text and JSON output modes, level filtering, ANSI colors |
| `warc` | WARC (Web ARChive) format writing for legal and archival compliance |
| `cli` | ANSI color output, progress bar, help text, completion |
| `pool` | Protocol-agnostic recursive worker pool — bounded goroutine execution with recursive task submission. Replaces per-protocol semaphore+WaitGroup code in FTP, SFTP, and WebDAV recursive-parallel downloads |

## Download Flow

```mermaid
sequenceDiagram
    participant User
    participant CLI as cmd/goget/
    participant Config as internal/config/
    participant Core as internal/core/
    participant Registry as internal/protocol/
    participant Proto as Protocol Handler
    participant Transport as internal/transport/
    participant Output as internal/output/

    User->>CLI: goget --url https://example.com/file.zip
    CLI->>Config: Load config file
    Config-->>CLI: Config (with defaults + overrides)
    CLI->>Core: Create DownloadRequest
    CLI->>Registry: Get protocol for URL scheme
    Registry-->>CLI: HTTP protocol handler
    CLI->>Proto: Download(ctx, request)
    Proto->>Transport: Dial + send HTTP request
    Transport-->>Proto: HTTP response stream
    Proto->>Output: Write chunks atomically
    Output-->>Proto: Progress + completion
    Proto-->>CLI: DownloadResult
    CLI-->>User: Progress bar + summary
```

## Config Merging

Configuration is resolved in this priority order (highest first):

1. CLI flags (`--timeout 5m`)
2. Config file (`~/.config/goget/config.toml`)
3. Default values

The `config.Merge()` method applies CLI override top of the loaded config.

## Error Handling

All operations return structured `core.GogetError` with:

- **Type** — Category (`NETWORK`, `PROTOCOL`, `FILE`, `TIMEOUT`, `AUTH`, `CHECKSUM`, `UNSUPPORTED`, `CANCELLED`)
- **Message** — Human-readable description
- **Cause** — Wrapped underlying error (available via `errors.Unwrap`)
- **URL** — The URL that caused the error (credentials stripped)
- **Details** — Optional key-value metadata (e.g., expected vs actual checksum)

## Build Tags

All source files use the build constraint `//go:build linux || freebsd`. The project targets Linux and FreeBSD only. Platform-specific code (e.g., network interface binding) lives in `internal/transport/bind_linux.go` vs `bind_other.go`.