223 lines
8.4 KiB
Markdown
223 lines
8.4 KiB
Markdown
|
# Architecture
|
||
|
|
|
||
|
|
**goget** is a command-line download utility built on the Go standard library. It follows a layered architecture with a protocol abstraction layer, a transport layer, and utility modules for authentication, compression, output, and more.
|
||
|
|
|
||
|
|
## High-Level Overview
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
graph TD
|
||
|
|
User[User / Terminal]:::accent6
|
||
|
|
Script[Script / CI]:::accent6
|
||
|
|
CLI[cmd/goget/ - Main entry]:::accent0
|
||
|
|
API[pkg/api/ - Library interface]:::accent3
|
||
|
|
Core[internal/core/ - Types, Protocol Interface, Errors]:::accent1
|
||
|
|
Registry[internal/protocol/ - Protocol Registry]:::accent1
|
||
|
|
HTTP[internal/protocol/http/]:::accent1
|
||
|
|
FTP[internal/protocol/ftp/]:::accent1
|
||
|
|
SFTP[internal/protocol/sftp/]:::accent1
|
||
|
|
FILE[internal/protocol/file/]:::accent1
|
||
|
|
DataURL[internal/protocol/dataurl/]:::accent1
|
||
|
|
Gopher[internal/protocol/gopher/]:::accent1
|
||
|
|
Gemini[internal/protocol/gemini/]:::accent1
|
||
|
|
WebDAV[internal/protocol/webdav/]:::accent1
|
||
|
|
Transport[internal/transport/ - Dialer, Proxy, TLS, Rate Limit]:::accent7
|
||
|
|
Config[internal/config/]:::accent7
|
||
|
|
Output[internal/output/ - Atomic write, Resume, WARC]:::accent2
|
||
|
|
Auth[internal/auth/ - Basic, Digest, OAuth, netrc]:::accent7
|
||
|
|
Crypto[internal/crypto/ - Checksum, PGP, Certs]:::accent7
|
||
|
|
Cookie[internal/cookie/ - Netscape jar]:::accent7
|
||
|
|
HSTS[internal/hsts/ - RFC 6797 cache]:::accent7
|
||
|
|
Archive[internal/archive/ - Extraction]:::accent7
|
||
|
|
Recursive[internal/recursive/ - Crawler, Parser, Filter]:::accent2
|
||
|
|
Mirror[internal/mirror/]:::accent2
|
||
|
|
Metalink[internal/metalink/]:::accent2
|
||
|
|
Queue[internal/queue/]:::accent2
|
||
|
|
Compression[internal/compression/ - gzip, bzip2, zlib, flate, lzw]:::accent7
|
||
|
|
Log[internal/log/ - Structured logging]:::accent7
|
||
|
|
WARC[internal/warc/ - Archiving]:::accent7
|
||
|
|
Register[internal/protocol/register/ - Single registration]:::accent7
|
||
|
|
Format[internal/format/ - Bytes, Speed formatting]:::accent7
|
||
|
|
LinkRewrite[internal/linkrewrite/ - HTML link conversion]:::accent7
|
||
|
|
CLIUtil[internal/cli/ - Help, Progress bar, Colors]:::accent7
|
||
|
|
|
||
|
|
User --> CLI
|
||
|
|
Script --> API
|
||
|
|
CLI --> Log
|
||
|
|
CLI --> CLIUtil
|
||
|
|
CLI --> Config
|
||
|
|
CLI --> Core
|
||
|
|
CLI -.-> API
|
||
|
|
API --> Core
|
||
|
|
API --> Registry
|
||
|
|
|
||
|
|
subgraph "Download Pipeline"
|
||
|
|
Core --> Registry
|
||
|
|
Registry --> HTTP
|
||
|
|
Registry --> FTP
|
||
|
|
Registry --> SFTP
|
||
|
|
Registry --> FILE
|
||
|
|
Registry --> DataURL
|
||
|
|
Registry --> Gopher
|
||
|
|
Registry --> Gemini
|
||
|
|
Registry --> WebDAV
|
||
|
|
HTTP --> Transport
|
||
|
|
HTTP --> Cookie
|
||
|
|
HTTP --> HSTS
|
||
|
|
HTTP --> Output
|
||
|
|
FTP --> Transport
|
||
|
|
SFTP --> Transport
|
||
|
|
end
|
||
|
|
|
||
|
|
subgraph "Post-Download"
|
||
|
|
CLI --> Crypto
|
||
|
|
CLI --> Archive
|
||
|
|
CLI --> Compression
|
||
|
|
end
|
||
|
|
|
||
|
|
subgraph "Download Management"
|
||
|
|
CLI --> Register
|
||
|
|
CLI --> WARC
|
||
|
|
CLI --> Recursive
|
||
|
|
CLI --> Mirror
|
||
|
|
CLI --> Metalink
|
||
|
|
CLI --> Queue
|
||
|
|
Recursive --> WARC
|
||
|
|
Recursive --> LinkRewrite
|
||
|
|
Mirror --> LinkRewrite
|
||
|
|
end
|
||
|
|
|
||
|
|
Transport --> Auth
|
||
|
|
Transport --> Config
|
||
|
|
|
||
|
|
classDef accent0 fill:#3B82F6,stroke:#2563EB,color:#fff
|
||
|
|
classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
|
||
|
|
classDef accent2 fill:#F59E0B,stroke:#D97706,color:#000
|
||
|
|
classDef accent3 fill:#A855F7,stroke:#7C3AED,color:#fff
|
||
|
|
classDef accent6 fill:#6366F1,stroke:#4F46E5,color:#fff
|
||
|
|
classDef accent7 fill:#64748B,stroke:#475569,color:#fff
|
||
|
|
```
|
||
|
|
|
||
|
|
## Layers
|
||
|
|
|
||
|
|
### 1. Entry Points
|
||
|
|
|
||
|
|
| Layer | Package | Role |
|
||
|
|
|---|---|---|
|
||
|
|
| **CLI** | `cmd/goget/` | Flag parsing, command routing, signal handling |
|
||
|
|
| **Library API** | `pkg/api/` | Public Go API for programmatic downloads, uploads, and queue management |
|
||
|
|
|
||
|
|
The CLI and API share the same internal packages. The CLI is built on top of the same `internal/core` types that the API exposes.
|
||
|
|
|
||
|
|
### 2. Protocol Abstraction
|
||
|
|
|
||
|
|
Defined in `internal/core` and `internal/protocol`:
|
||
|
|
|
||
|
|
```go
|
||
|
|
// core.Protocol is the interface every protocol handler implements.
|
||
|
|
type Protocol interface {
|
||
|
|
Scheme() string
|
||
|
|
CanHandle(url *url.URL) bool
|
||
|
|
Download(ctx context.Context, req *DownloadRequest) (*DownloadResult, error)
|
||
|
|
Capabilities() []string
|
||
|
|
SupportsResume() bool
|
||
|
|
SupportsCompression() bool
|
||
|
|
SupportsParallel() bool
|
||
|
|
SupportsRecursive() bool
|
||
|
|
}
|
||
|
|
|
||
|
|
// core.Uploader is an optional interface for protocols supporting uploads.
|
||
|
|
type Uploader interface {
|
||
|
|
Upload(ctx context.Context, req *UploadRequest) (*UploadResult, error)
|
||
|
|
SupportsUpload() bool
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
Protocols register themselves by scheme in a global `Registry`. Scheme normalization maps HTTPS → HTTP, FTPS → FTP, etc. Supported protocols include HTTP, FTP, SFTP, File, DataURL, Gopher, Gemini, and WebDAV (new in `internal/protocol/webdav/`).
|
||
|
|
|
||
|
|
### 3. Transport Layer
|
||
|
|
|
||
|
|
`internal/transport` provides a custom HTTP transport with:
|
||
|
|
|
||
|
|
- **IPv6-first dialer** — Resolves DNS, prefers IPv6 addresses, falls back to IPv4 only when no IPv6 record exists
|
||
|
|
- **Interface binding** — Linux-specific socket binding via `SO_BINDTODEVICE`
|
||
|
|
- **Rate limiting** — Token bucket algorithm for bandwidth capping
|
||
|
|
- **Retry logic** — Exponential backoff with configurable max retries
|
||
|
|
- **TLS configuration** — Certificate pinning, client certificates, key log file
|
||
|
|
- **Proxy support** — HTTP CONNECT and SOCKS5 (`socks5://` with local DNS, `socks5h://` with remote DNS, RFC 1929 username/password auth)
|
||
|
|
|
||
|
|
### 4. Output Layer
|
||
|
|
|
||
|
|
`internal/output` handles atomic file writes:
|
||
|
|
|
||
|
|
1. Write to a `.goget.tmp` file during download
|
||
|
|
2. Store resume metadata in `.goget.meta` alongside the temp file
|
||
|
|
3. Atomically rename from `.goget.tmp` to the final filename on completion
|
||
|
|
|
||
|
|
### 5. Utility Modules
|
||
|
|
|
||
|
|
| Module | Responsibility |
|
||
|
|
|---|---|
|
||
|
|
| `auth` | HTTP Basic, Digest, OAuth 2.0 (client credentials, authorization code, refresh token), `.netrc` auto-reading |
|
||
|
|
| `cookie` | Netscape-format cookie jar with import/export |
|
||
|
|
| `crypto` | Checksum verification (SHA-256, SHA-512, SHA3-256/512, BLAKE2b, MD5), PGP signature verification and decryption, certificate handling |
|
||
|
|
| `compression` | Decompression wrappers for gzip, bzip2, zlib, flate, LZW |
|
||
|
|
| `hsts` | HSTS cache (RFC 6797) for automatic HTTPS upgrade |
|
||
|
|
| `archive` | Archive extraction for tar, tar.gz, tar.bz2, zip with auto-detection |
|
||
|
|
| `format` | Human-readable byte and speed formatting |
|
||
|
|
| `linkrewrite` | HTML link rewriting for offline mirror viewing |
|
||
|
|
| `log` | Structured logger with text and JSON output modes, level filtering, ANSI colors |
|
||
|
|
| `warc` | WARC (Web ARChive) format writing for legal and archival compliance |
|
||
|
|
| `cli` | ANSI color output, progress bar, help text, completion |
|
||
|
|
|
||
|
|
## Download Flow
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
sequenceDiagram
|
||
|
|
participant User
|
||
|
|
participant CLI as cmd/goget/
|
||
|
|
participant Config as internal/config/
|
||
|
|
participant Core as internal/core/
|
||
|
|
participant Registry as internal/protocol/
|
||
|
|
participant Proto as Protocol Handler
|
||
|
|
participant Transport as internal/transport/
|
||
|
|
participant Output as internal/output/
|
||
|
|
|
||
|
|
User->>CLI: goget --url https://example.com/file.zip
|
||
|
|
CLI->>Config: Load config file
|
||
|
|
Config-->>CLI: Config (with defaults + overrides)
|
||
|
|
CLI->>Core: Create DownloadRequest
|
||
|
|
CLI->>Registry: Get protocol for URL scheme
|
||
|
|
Registry-->>CLI: HTTP protocol handler
|
||
|
|
CLI->>Proto: Download(ctx, request)
|
||
|
|
Proto->>Transport: Dial + send HTTP request
|
||
|
|
Transport-->>Proto: HTTP response stream
|
||
|
|
Proto->>Output: Write chunks atomically
|
||
|
|
Output-->>Proto: Progress + completion
|
||
|
|
Proto-->>CLI: DownloadResult
|
||
|
|
CLI-->>User: Progress bar + summary
|
||
|
|
```
|
||
|
|
|
||
|
|
## Config Merging
|
||
|
|
|
||
|
|
Configuration is resolved in this priority order (highest first):
|
||
|
|
|
||
|
|
1. CLI flags (`--timeout 5m`)
|
||
|
|
2. Config file (`~/.config/goget/config.toml`)
|
||
|
|
3. Default values
|
||
|
|
|
||
|
|
The `config.Merge()` method applies CLI override top of the loaded config.
|
||
|
|
|
||
|
|
## Error Handling
|
||
|
|
|
||
|
|
All operations return structured `core.GogetError` with:
|
||
|
|
|
||
|
|
- **Type** — Category (`NETWORK`, `PROTOCOL`, `FILE`, `TIMEOUT`, `AUTH`, `CHECKSUM`, `UNSUPPORTED`, `CANCELLED`)
|
||
|
|
- **Message** — Human-readable description
|
||
|
|
- **Cause** — Wrapped underlying error (available via `errors.Unwrap`)
|
||
|
|
- **URL** — The URL that caused the error (credentials stripped)
|
||
|
|
- **Details** — Optional key-value metadata (e.g., expected vs actual checksum)
|
||
|
|
|
||
|
|
## Build Tags
|
||
|
|
|
||
|
|
All source files use the build constraint `//go:build linux || freebsd`. The project targets Linux and FreeBSD only. Platform-specific code (e.g., network interface binding) lives in `internal/transport/bind_linux.go` vs `bind_other.go`.
|