feat: initial goget release — modern IPv6-first download utility

2026-06-26 21:21:58 +02:00
commit 53db81e1f7
147 changed files with 45931 additions and 0 deletions
@@ -0,0 +1,312 @@
+# Download Pipeline
+
+This document explains how goget processes a download request from start to finish — the phase structure, parallel chunking, resume logic, and output writing.
+
+## High-Level Flow
+
+```mermaid
+flowchart TD
+    Start([User invokes goget]):::accent0
+    ParseFlags["Parse CLI flags"]:::accent7
+    LoadConfig["Load config + merge overrides"]:::accent7
+    ResolveProtocol["Resolve protocol handler\nfrom URL scheme"]:::accent1
+    CreateRequest["Build core.DownloadRequest"]:::accent1
+    CheckResume{"Resume metadata\nexists?"}:::accent2
+    LoadResume["Load .goget.meta\n(resume info + chunk map)"]:::accent2
+    SetupOutput["Setup output writer\n(atomic temp file + resume)"]:::accent2
+    CreateDirs{"--create-dirs\n(parent missing)?"}:::accent2
+    MkdirAll["os.MkdirAll → create\nparent directories"]:::accent2
+    CheckParallel{"File > 100 MB\nor --parallel set?"}:::accent1
+    SequentialDownload["Sequential download\n(single connection)"]:::accent1
+    ParallelDownload["Parallel chunked download\n(multiple Range requests)"]:::accent1
+    Decompress{"Auto-decompress\nenabled?"}:::accent7
+    Decompression["Decompress response\n(gzip, deflate, bzip2, zlib)"]:::accent7
+    VerifyChecksum{"Checksum\nspecified?"}:::accent7
+    ChecksumVerify["Verify checksum\n(SHA-256, SHA-512, etc.)"]:::accent7
+    PGPVerify{"PGP verify/\ndecrypt?"}:::accent7
+    PGPProcess["Verify signature /\ndecrypt file"]:::accent7
+    SaveHSTS["Save HSTS cache\n(RFC 6797)"]:::accent7
+    AtomicRename["Atomic rename\ntemp → final"]:::accent2
+    SigInt{"SIGINT received?"}:::accent4
+    SaveSidecar["Save .goget.meta\nfor resume"]:::accent2
+    Exit130["Exit 130"]:::accent4
+    RunHook{"--on-complete\nset?"}:::accent7
+    PostHook["Run post-download command\n(GOGET_OUTPUT, GOGET_SIZE, GOGET_URL)"]:::accent7
+    Done([Done]):::accent0
+
+    Start --> ParseFlags --> LoadConfig --> ResolveProtocol
+    ResolveProtocol --> CreateRequest --> CheckResume
+    CheckResume -->|Yes| LoadResume --> SetupOutput
+    CheckResume -->|No| SetupOutput
+    SetupOutput --> CreateDirs
+    CreateDirs -->|Yes| MkdirAll --> CheckParallel
+    CreateDirs -->|No| CheckParallel
+    CheckParallel -->|Yes| ParallelDownload
+    CheckParallel -->|No| SequentialDownload
+    SequentialDownload --> Decompress
+    ParallelDownload --> Decompress
+    Decompress -->|Yes| Decompression --> VerifyChecksum
+    Decompress -->|No| VerifyChecksum
+    VerifyChecksum -->|Yes| ChecksumVerify -->|pass| PGPVerify
+    ChecksumVerify -->|fail| Done
+    VerifyChecksum -->|No| PGPVerify
+    PGPVerify -->|Yes| PGPProcess --> SaveHSTS
+    PGPVerify -->|No| SaveHSTS
+    SaveHSTS --> AtomicRename
+    AtomicRename --> SigInt
+    SigInt -->|Yes| SaveSidecar --> Exit130
+    SigInt -->|No| RunHook
+    RunHook -->|Yes| PostHook --> Done
+    RunHook -->|No| Done
+
+    classDef accent0 fill:#3B82F6,stroke:#2563EB,color:#fff
+    classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
+    classDef accent2 fill:#F59E0B,stroke:#D97706,color:#000
+    classDef accent4 fill:#EF4444,stroke:#DC2626,color:#fff
+    classDef accent7 fill:#64748B,stroke:#475569,color:#fff
+```
+
+## Phase 1: Request Construction
+
+A `core.DownloadRequest` is built from CLI flags and config:
+
+```go
+req := &core.DownloadRequest{
+    URL:              parsedURL,
+    Output:           outputPath,
+    Resume:           resumeEnabled,
+    Timeout:          effectiveTimeout,
+    Verbose:          verbose,
+    Headers:          customHeaders,
+    Proxy:            proxyURL,
+    Checksum:         expectedChecksum,
+    Parallel:         parallelConfig,
+    Recursive:        recursiveEnabled,
+    MaxDepth:         maxDepth,
+    ProgressCallback: progressFn,
+    Ctx:              ctx,
+}
+```
+
+## Phase 2: Protocol Resolution
+
+The protocol registry resolves the handler by URL scheme:
+
+```mermaid
+flowchart LR
+    URL["https://example.com"]:::accent6
+    Parse["Parse scheme\n→ https"]:::accent7
+    Normalized["Normalize\nhttps → http"]:::accent7
+    Registry["Registry lookup\nprotocols[http]"]:::accent1
+    Handler["HTTP protocol\nhandler"]:::accent1
+
+    URL --> Parse --> Normalized --> Registry --> Handler
+
+    classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
+    classDef accent6 fill:#6366F1,stroke:#4F46E5,color:#fff
+    classDef accent7 fill:#64748B,stroke:#475569,color:#fff
+```
+
+Scheme normalization maps:
+- `https://` → `http` (same handler)
+- `ftps://` → `ftp` (same handler)
+- `webdavs://` → `webdav` (same handler)
+
+## Phase 3: Output Setup
+
+The `internal/output` package creates an atomic writer:
+
+1. **Temp file** — Writes to `<filename>.goget.tmp` during download
+2. **Parent directories** — If `--create-dirs` is enabled (default: `true`), missing parent directories are created via `os.MkdirAll` before opening the output file. Pass `--create-dirs=false` to restore strict behaviour
+3. **Resume metadata** — Reads `<filename>.goget.meta` if resuming
+4. **Progress callback** — Hooks into the writer for real-time speed/ETA
+
+```go
+writer, err := output.NewWriter(&output.WriterConfig{
+    Output:           outputFile,
+    Atomic:           true,
+    Resume:           resumeEnabled,
+    CreateDirs:       true,  // auto-create parent dirs (curl --create-dirs)
+    ProgressCallback: progressFunc,
+})
+```
+
+### Resume Metadata Format
+
+```json
+{
+    "downloaded_bytes": 524288000,
+    "etag": "\"abc123\"",
+    "last_modified": "Mon, 01 Jun 2026 12:00:00 GMT",
+    "url": "https://example.com/file.zip",
+    "last_write": "2026-06-01T12:05:00Z",
+    "chunks": {
+        "0": 131072000,
+        "1": 131072000,
+        "2": 131072000,
+        "3": 131072000
+    }
+}
+```
+
+For parallel downloads, each chunk's progress is individually tracked.
+
+## Phase 4: Download Strategy
+
+### Sequential Download
+
+Used when the file is under 100 MB or `--parallel 1` is set:
+
+```go
+resp, _ := client.Do(request)
+io.Copy(writer, resp.Body)
+```
+
+### Parallel Chunked Download
+
+Triggered automatically for files over 100 MB, or explicitly with `--parallel N`:
+
+```mermaid
+sequenceDiagram
+    participant Main
+    participant Chunk1
+    participant Chunk2
+    participant Chunk3
+    participant Chunk4
+    participant Writer as Atomic Writer
+
+    Main->>Main: HEAD request → get file size
+    Main->>Main: Split into N equal chunks
+    Main->>Chunk1: Start: bytes 0-13107199
+    Main->>Chunk2: Start: bytes 13107200-26214399
+    Main->>Chunk3: Start: bytes 26214400-39321599
+    Main->>Chunk4: Start: bytes 39321600-52428799
+    Chunk1->>Writer: Write bytes to temp/chunk_0
+    Chunk2->>Writer: Write bytes to temp/chunk_1
+    Chunk3->>Writer: Write bytes to temp/chunk_2
+    Chunk4->>Writer: Write bytes to temp/chunk_3
+    Main->>Writer: Merge chunks → final file
+```
+
+Each chunk downloads via a separate HTTP `Range` request:
+
+```http
+GET /large.iso HTTP/1.1
+Host: example.com
+Range: bytes=13107200-26214399
+```
+
+### Concurrency Control
+
+- Max 4 parallel connections by default
+- Customizable via `--parallel N` or config `parallel` key
+- Min chunk size: 1 MB
+- Max chunk size: 50 MB
+
+## Phase 5: Post-Processing
+
+### Decompression
+
+If `auto_decompress` is enabled (default) and the server sends compressed content, the response body is transparently decompressed:
+
+| Content-Encoding | Handler |
+|---|---|
+| `gzip` | `compress/gzip` |
+| `deflate` | `compress/flate` |
+| `zlib` | Internal |
+| `bzip2` | Internal |
+| `lzw` | Internal |
+
+Use `--no-decompress` to preserve the compressed response.
+
+### Checksum Verification
+
+If `--checksum` or `--checksum-file` is provided, the downloaded file is hashed and compared:
+
+```
+Expected: a1b2c3d4...
+Actual:   a1b2c3d4...
+Checksum OK
+```
+
+Supported algorithms: SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b, MD5.
+
+### Atomic Rename
+
+On successful completion, the temp file is atomically renamed:
+
+```go
+os.Rename("file.zip.goget.tmp", "file.zip")
+```
+
+If the download fails or is interrupted, the temp file remains for resume.
+
+### PGP Verification and Decryption
+
+If `--pgp-verify` or `--pgp-decrypt` is set, the downloaded file is processed via `golang.org/x/crypto/openpgp`:
+
+- **Detached signature verification** — `--pgp-sig file.sig --pgp-key public.key`
+- **Decryption** — `--pgp-decrypt --pgp-key private.key --pgp-passphrase "secret"`
+- Auto-detect signature files: `.asc`, `.sig`
+
+The decrypted file replaces the encrypted one (`.gpg` → stripped extension, or `.decrypted` suffix).
+
+### HSTS Cache
+
+After every successful HTTPS connection, the HSTS cache (`~/.config/goget/hsts`) is updated per RFC 6797. Expired entries are pruned on load. This ensures that future `http://` requests to known hosts are automatically upgraded to `https://`.
+
+### Graceful Shutdown (SIGINT)
+
+When the user sends Ctrl+C (SIGINT) during a download, goget persists partial progress to the `.goget.meta` sidecar for **all protocols** (HTTP, FTP, SFTP, WebDAV) and exits with code 130. A subsequent `goget --resume` picks up where it left off. The signal handler uses `signal.NotifyContext` and propagates cancellation through the download pipeline via `ctx.Err()` checks after every read.
+
+## Smart Timeout
+
+When no explicit `--timeout` is set, goget calculates a timeout based on file size:
+
+```
+timeout = (fileSize / minSpeed) × safetyFactor
+```
+
+Where:
+- `minSpeed` = 10 KB/s (conservative minimum)
+- `safetyFactor` = 3.0
+- `minTimeout` = 30 seconds
+- `maxTimeout` = 24 hours
+
+A 1 GB file at 10 KB/s → 34 hours (capped to 24 hours max).
+A 10 MB file at 10 KB/s → 51 minutes.
+
+## Rate Limiting
+
+`--rate-limit` uses a token bucket algorithm:
+
+```go
+bucket := transport.NewTokenBucket(rate)
+// For each read:
+bucket.Wait(n)
+```
+
+The token bucket allows short bursts above the limit while maintaining the average rate.
+
+### Speed Format
+
+```
+--rate-limit 1MB/s    # 1,000,000 bytes/sec
+--rate-limit 500KB/s  # 500,000 bytes/sec
+--rate-limit 1GB/s    # 1,000,000,000 bytes/sec
+--rate-limit 100      # 100 bytes/sec (plain number)
+```
+
+## Retry Logic
+
+When a download fails, goget retries with exponential backoff:
+
+```
+Attempt 1: wait 1s, retry
+Attempt 2: wait 2s, retry
+Attempt 3: wait 4s, retry
+...
+Attempt N: wait min(2^(N-1) × 1s, 30s), retry
+```
+
+Max retries default to 3. Use `--max-retries N` to increase. Use `--retry-all-errors` to retry on HTTP 4xx/5xx (by default, only 5xx server errors trigger retries).