feat: initial goget release — modern IPv6-first download utility
This commit is contained in:
@@ -0,0 +1,222 @@
|
||||
# Architecture
|
||||
|
||||
**goget** is a command-line download utility built on the Go standard library. It follows a layered architecture with a protocol abstraction layer, a transport layer, and utility modules for authentication, compression, output, and more.
|
||||
|
||||
## High-Level Overview
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
User[User / Terminal]:::accent6
|
||||
Script[Script / CI]:::accent6
|
||||
CLI[cmd/goget/ - Main entry]:::accent0
|
||||
API[pkg/api/ - Library interface]:::accent3
|
||||
Core[internal/core/ - Types, Protocol Interface, Errors]:::accent1
|
||||
Registry[internal/protocol/ - Protocol Registry]:::accent1
|
||||
HTTP[internal/protocol/http/]:::accent1
|
||||
FTP[internal/protocol/ftp/]:::accent1
|
||||
SFTP[internal/protocol/sftp/]:::accent1
|
||||
FILE[internal/protocol/file/]:::accent1
|
||||
DataURL[internal/protocol/dataurl/]:::accent1
|
||||
Gopher[internal/protocol/gopher/]:::accent1
|
||||
Gemini[internal/protocol/gemini/]:::accent1
|
||||
WebDAV[internal/protocol/webdav/]:::accent1
|
||||
Transport[internal/transport/ - Dialer, Proxy, TLS, Rate Limit]:::accent7
|
||||
Config[internal/config/]:::accent7
|
||||
Output[internal/output/ - Atomic write, Resume, WARC]:::accent2
|
||||
Auth[internal/auth/ - Basic, Digest, OAuth, netrc]:::accent7
|
||||
Crypto[internal/crypto/ - Checksum, PGP, Certs]:::accent7
|
||||
Cookie[internal/cookie/ - Netscape jar]:::accent7
|
||||
HSTS[internal/hsts/ - RFC 6797 cache]:::accent7
|
||||
Archive[internal/archive/ - Extraction]:::accent7
|
||||
Recursive[internal/recursive/ - Crawler, Parser, Filter]:::accent2
|
||||
Mirror[internal/mirror/]:::accent2
|
||||
Metalink[internal/metalink/]:::accent2
|
||||
Queue[internal/queue/]:::accent2
|
||||
Compression[internal/compression/ - gzip, bzip2, zlib, flate, lzw]:::accent7
|
||||
Log[internal/log/ - Structured logging]:::accent7
|
||||
WARC[internal/warc/ - Archiving]:::accent7
|
||||
Register[internal/protocol/register/ - Single registration]:::accent7
|
||||
Format[internal/format/ - Bytes, Speed formatting]:::accent7
|
||||
LinkRewrite[internal/linkrewrite/ - HTML link conversion]:::accent7
|
||||
CLIUtil[internal/cli/ - Help, Progress bar, Colors]:::accent7
|
||||
|
||||
User --> CLI
|
||||
Script --> API
|
||||
CLI --> Log
|
||||
CLI --> CLIUtil
|
||||
CLI --> Config
|
||||
CLI --> Core
|
||||
CLI -.-> API
|
||||
API --> Core
|
||||
API --> Registry
|
||||
|
||||
subgraph "Download Pipeline"
|
||||
Core --> Registry
|
||||
Registry --> HTTP
|
||||
Registry --> FTP
|
||||
Registry --> SFTP
|
||||
Registry --> FILE
|
||||
Registry --> DataURL
|
||||
Registry --> Gopher
|
||||
Registry --> Gemini
|
||||
Registry --> WebDAV
|
||||
HTTP --> Transport
|
||||
HTTP --> Cookie
|
||||
HTTP --> HSTS
|
||||
HTTP --> Output
|
||||
FTP --> Transport
|
||||
SFTP --> Transport
|
||||
end
|
||||
|
||||
subgraph "Post-Download"
|
||||
CLI --> Crypto
|
||||
CLI --> Archive
|
||||
CLI --> Compression
|
||||
end
|
||||
|
||||
subgraph "Download Management"
|
||||
CLI --> Register
|
||||
CLI --> WARC
|
||||
CLI --> Recursive
|
||||
CLI --> Mirror
|
||||
CLI --> Metalink
|
||||
CLI --> Queue
|
||||
Recursive --> WARC
|
||||
Recursive --> LinkRewrite
|
||||
Mirror --> LinkRewrite
|
||||
end
|
||||
|
||||
Transport --> Auth
|
||||
Transport --> Config
|
||||
|
||||
classDef accent0 fill:#3B82F6,stroke:#2563EB,color:#fff
|
||||
classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
|
||||
classDef accent2 fill:#F59E0B,stroke:#D97706,color:#000
|
||||
classDef accent3 fill:#A855F7,stroke:#7C3AED,color:#fff
|
||||
classDef accent6 fill:#6366F1,stroke:#4F46E5,color:#fff
|
||||
classDef accent7 fill:#64748B,stroke:#475569,color:#fff
|
||||
```
|
||||
|
||||
## Layers
|
||||
|
||||
### 1. Entry Points
|
||||
|
||||
| Layer | Package | Role |
|
||||
|---|---|---|
|
||||
| **CLI** | `cmd/goget/` | Flag parsing, command routing, signal handling |
|
||||
| **Library API** | `pkg/api/` | Public Go API for programmatic downloads, uploads, and queue management |
|
||||
|
||||
The CLI and API share the same internal packages. The CLI is built on top of the same `internal/core` types that the API exposes.
|
||||
|
||||
### 2. Protocol Abstraction
|
||||
|
||||
Defined in `internal/core` and `internal/protocol`:
|
||||
|
||||
```go
|
||||
// core.Protocol is the interface every protocol handler implements.
|
||||
type Protocol interface {
|
||||
Scheme() string
|
||||
CanHandle(url *url.URL) bool
|
||||
Download(ctx context.Context, req *DownloadRequest) (*DownloadResult, error)
|
||||
Capabilities() []string
|
||||
SupportsResume() bool
|
||||
SupportsCompression() bool
|
||||
SupportsParallel() bool
|
||||
SupportsRecursive() bool
|
||||
}
|
||||
|
||||
// core.Uploader is an optional interface for protocols supporting uploads.
|
||||
type Uploader interface {
|
||||
Upload(ctx context.Context, req *UploadRequest) (*UploadResult, error)
|
||||
SupportsUpload() bool
|
||||
}
|
||||
```
|
||||
|
||||
Protocols register themselves by scheme in a global `Registry`. Scheme normalization maps HTTPS → HTTP, FTPS → FTP, etc. Supported protocols include HTTP, FTP, SFTP, File, DataURL, Gopher, Gemini, and WebDAV (new in `internal/protocol/webdav/`).
|
||||
|
||||
### 3. Transport Layer
|
||||
|
||||
`internal/transport` provides a custom HTTP transport with:
|
||||
|
||||
- **IPv6-first dialer** — Resolves DNS, prefers IPv6 addresses, falls back to IPv4 only when no IPv6 record exists
|
||||
- **Interface binding** — Linux-specific socket binding via `SO_BINDTODEVICE`
|
||||
- **Rate limiting** — Token bucket algorithm for bandwidth capping
|
||||
- **Retry logic** — Exponential backoff with configurable max retries
|
||||
- **TLS configuration** — Certificate pinning, client certificates, key log file
|
||||
- **Proxy support** — HTTP CONNECT and SOCKS5 (`socks5://` with local DNS, `socks5h://` with remote DNS, RFC 1929 username/password auth)
|
||||
|
||||
### 4. Output Layer
|
||||
|
||||
`internal/output` handles atomic file writes:
|
||||
|
||||
1. Write to a `.goget.tmp` file during download
|
||||
2. Store resume metadata in `.goget.meta` alongside the temp file
|
||||
3. Atomically rename from `.goget.tmp` to the final filename on completion
|
||||
|
||||
### 5. Utility Modules
|
||||
|
||||
| Module | Responsibility |
|
||||
|---|---|
|
||||
| `auth` | HTTP Basic, Digest, OAuth 2.0 (client credentials, authorization code, refresh token), `.netrc` auto-reading |
|
||||
| `cookie` | Netscape-format cookie jar with import/export |
|
||||
| `crypto` | Checksum verification (SHA-256, SHA-512, SHA3-256/512, BLAKE2b, MD5), PGP signature verification and decryption, certificate handling |
|
||||
| `compression` | Decompression wrappers for gzip, bzip2, zlib, flate, LZW |
|
||||
| `hsts` | HSTS cache (RFC 6797) for automatic HTTPS upgrade |
|
||||
| `archive` | Archive extraction for tar, tar.gz, tar.bz2, zip with auto-detection |
|
||||
| `format` | Human-readable byte and speed formatting |
|
||||
| `linkrewrite` | HTML link rewriting for offline mirror viewing |
|
||||
| `log` | Structured logger with text and JSON output modes, level filtering, ANSI colors |
|
||||
| `warc` | WARC (Web ARChive) format writing for legal and archival compliance |
|
||||
| `cli` | ANSI color output, progress bar, help text, completion |
|
||||
|
||||
## Download Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant CLI as cmd/goget/
|
||||
participant Config as internal/config/
|
||||
participant Core as internal/core/
|
||||
participant Registry as internal/protocol/
|
||||
participant Proto as Protocol Handler
|
||||
participant Transport as internal/transport/
|
||||
participant Output as internal/output/
|
||||
|
||||
User->>CLI: goget --url https://example.com/file.zip
|
||||
CLI->>Config: Load config file
|
||||
Config-->>CLI: Config (with defaults + overrides)
|
||||
CLI->>Core: Create DownloadRequest
|
||||
CLI->>Registry: Get protocol for URL scheme
|
||||
Registry-->>CLI: HTTP protocol handler
|
||||
CLI->>Proto: Download(ctx, request)
|
||||
Proto->>Transport: Dial + send HTTP request
|
||||
Transport-->>Proto: HTTP response stream
|
||||
Proto->>Output: Write chunks atomically
|
||||
Output-->>Proto: Progress + completion
|
||||
Proto-->>CLI: DownloadResult
|
||||
CLI-->>User: Progress bar + summary
|
||||
```
|
||||
|
||||
## Config Merging
|
||||
|
||||
Configuration is resolved in this priority order (highest first):
|
||||
|
||||
1. CLI flags (`--timeout 5m`)
|
||||
2. Config file (`~/.config/goget/config.toml`)
|
||||
3. Default values
|
||||
|
||||
The `config.Merge()` method applies CLI override top of the loaded config.
|
||||
|
||||
## Error Handling
|
||||
|
||||
All operations return structured `core.GogetError` with:
|
||||
|
||||
- **Type** — Category (`NETWORK`, `PROTOCOL`, `FILE`, `TIMEOUT`, `AUTH`, `CHECKSUM`, `UNSUPPORTED`, `CANCELLED`)
|
||||
- **Message** — Human-readable description
|
||||
- **Cause** — Wrapped underlying error (available via `errors.Unwrap`)
|
||||
- **URL** — The URL that caused the error (credentials stripped)
|
||||
- **Details** — Optional key-value metadata (e.g., expected vs actual checksum)
|
||||
|
||||
## Build Tags
|
||||
|
||||
All source files use the build constraint `//go:build linux || freebsd`. The project targets Linux and FreeBSD only. Platform-specific code (e.g., network interface binding) lives in `internal/transport/bind_linux.go` vs `bind_other.go`.
|
||||
Reference in New Issue
Block a user