# Download Pipeline This document explains how goget processes a download request from start to finish — the phase structure, parallel chunking, resume logic, and output writing. ## High-Level Flow ```mermaid flowchart TD Start([User invokes goget]):::accent0 ParseFlags["Parse CLI flags"]:::accent7 LoadConfig["Load config + merge overrides"]:::accent7 ResolveProtocol["Resolve protocol handler\nfrom URL scheme"]:::accent1 CreateRequest["Build core.DownloadRequest"]:::accent1 CheckResume{"Resume metadata\nexists?"}:::accent2 LoadResume["Load .goget.meta\n(resume info + chunk map)"]:::accent2 SetupOutput["Setup output writer\n(atomic temp file + resume)"]:::accent2 CreateDirs{"--create-dirs\n(parent missing)?"}:::accent2 MkdirAll["os.MkdirAll → create\nparent directories"]:::accent2 CheckParallel{"File > 100 MB\nor --parallel set?"}:::accent1 SequentialDownload["Sequential download\n(single connection)"]:::accent1 ParallelDownload["Parallel chunked download\n(multiple Range requests)"]:::accent1 Decompress{"Auto-decompress\nenabled?"}:::accent7 Decompression["Decompress response\n(gzip, deflate, bzip2, zlib)"]:::accent7 VerifyChecksum{"Checksum\nspecified?"}:::accent7 ChecksumVerify["Verify checksum\n(SHA-256, SHA-512, etc.)"]:::accent7 PGPVerify{"PGP verify/\ndecrypt?"}:::accent7 PGPProcess["Verify signature /\ndecrypt file"]:::accent7 SaveHSTS["Save HSTS cache\n(RFC 6797)"]:::accent7 AtomicRename["Atomic rename\ntemp → final"]:::accent2 SigInt{"SIGINT received?"}:::accent4 SaveSidecar["Save .goget.meta\nfor resume"]:::accent2 Exit130["Exit 130"]:::accent4 RunHook{"--on-complete\nset?"}:::accent7 PostHook["Run post-download command\n(GOGET_OUTPUT, GOGET_SIZE, GOGET_URL)"]:::accent7 Done([Done]):::accent0 Start --> ParseFlags --> LoadConfig --> ResolveProtocol ResolveProtocol --> CreateRequest --> CheckResume CheckResume -->|Yes| LoadResume --> SetupOutput CheckResume -->|No| SetupOutput SetupOutput --> CreateDirs CreateDirs -->|Yes| MkdirAll --> CheckParallel CreateDirs -->|No| CheckParallel CheckParallel -->|Yes| ParallelDownload CheckParallel -->|No| SequentialDownload SequentialDownload --> Decompress ParallelDownload --> Decompress Decompress -->|Yes| Decompression --> VerifyChecksum Decompress -->|No| VerifyChecksum VerifyChecksum -->|Yes| ChecksumVerify -->|pass| PGPVerify ChecksumVerify -->|fail| Done VerifyChecksum -->|No| PGPVerify PGPVerify -->|Yes| PGPProcess --> SaveHSTS PGPVerify -->|No| SaveHSTS SaveHSTS --> AtomicRename AtomicRename --> SigInt SigInt -->|Yes| SaveSidecar --> Exit130 SigInt -->|No| RunHook RunHook -->|Yes| PostHook --> Done RunHook -->|No| Done classDef accent0 fill:#3B82F6,stroke:#2563EB,color:#fff classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff classDef accent2 fill:#F59E0B,stroke:#D97706,color:#000 classDef accent4 fill:#EF4444,stroke:#DC2626,color:#fff classDef accent7 fill:#64748B,stroke:#475569,color:#fff ``` ## Phase 1: Request Construction A `core.DownloadRequest` is built from CLI flags and config: ```go req := &core.DownloadRequest{ URL: parsedURL, Output: outputPath, Resume: resumeEnabled, Timeout: effectiveTimeout, Verbose: verbose, Headers: customHeaders, Proxy: proxyURL, Checksum: expectedChecksum, Parallel: parallelConfig, Recursive: recursiveEnabled, MaxDepth: maxDepth, ProgressCallback: progressFn, Ctx: ctx, } ``` ## Phase 2: Protocol Resolution The protocol registry resolves the handler by URL scheme: ```mermaid flowchart LR URL["https://example.com"]:::accent6 Parse["Parse scheme\n→ https"]:::accent7 Normalized["Normalize\nhttps → http"]:::accent7 Registry["Registry lookup\nprotocols[http]"]:::accent1 Handler["HTTP protocol\nhandler"]:::accent1 URL --> Parse --> Normalized --> Registry --> Handler classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff classDef accent6 fill:#6366F1,stroke:#4F46E5,color:#fff classDef accent7 fill:#64748B,stroke:#475569,color:#fff ``` Scheme normalization maps: - `https://` → `http` (same handler) - `ftps://` → `ftp` (same handler) - `webdavs://` → `webdav` (same handler) ## Phase 3: Output Setup The `internal/output` package creates an atomic writer: 1. **Temp file** — Writes to `.goget.tmp` during download 2. **Parent directories** — If `--create-dirs` is enabled (default: `true`), missing parent directories are created via `os.MkdirAll` before opening the output file. Pass `--create-dirs=false` to restore strict behaviour 3. **Resume metadata** — Reads `.goget.meta` if resuming 4. **Progress callback** — Hooks into the writer for real-time speed/ETA ```go writer, err := output.NewWriter(&output.WriterConfig{ Output: outputFile, Atomic: true, Resume: resumeEnabled, CreateDirs: true, // auto-create parent dirs (curl --create-dirs) ProgressCallback: progressFunc, }) ``` ### Resume Metadata Format ```json { "downloaded_bytes": 524288000, "etag": "\"abc123\"", "last_modified": "Mon, 01 Jun 2026 12:00:00 GMT", "url": "https://example.com/file.zip", "last_write": "2026-06-01T12:05:00Z", "chunks": { "0": 131072000, "1": 131072000, "2": 131072000, "3": 131072000 } } ``` For parallel downloads, each chunk's progress is individually tracked. ## Phase 4: Download Strategy ### Sequential Download Used when the file is under 100 MB or `--parallel 1` is set: ```go resp, _ := client.Do(request) io.Copy(writer, resp.Body) ``` ### Parallel Chunked Download Triggered automatically for files over 100 MB, or explicitly with `--parallel N`: ```mermaid sequenceDiagram participant Main participant Chunk1 participant Chunk2 participant Chunk3 participant Chunk4 participant Writer as Atomic Writer Main->>Main: HEAD request → get file size Main->>Main: Split into N equal chunks Main->>Chunk1: Start: bytes 0-13107199 Main->>Chunk2: Start: bytes 13107200-26214399 Main->>Chunk3: Start: bytes 26214400-39321599 Main->>Chunk4: Start: bytes 39321600-52428799 Chunk1->>Writer: Write bytes to temp/chunk_0 Chunk2->>Writer: Write bytes to temp/chunk_1 Chunk3->>Writer: Write bytes to temp/chunk_2 Chunk4->>Writer: Write bytes to temp/chunk_3 Main->>Writer: Merge chunks → final file ``` Each chunk downloads via a separate HTTP `Range` request: ```http GET /large.iso HTTP/1.1 Host: example.com Range: bytes=13107200-26214399 ``` ### Concurrency Control - Max 4 parallel connections by default - Customizable via `--parallel N` or config `parallel` key - Min chunk size: 1 MB - Max chunk size: 50 MB ## Phase 5: Post-Processing ### Decompression If `auto_decompress` is enabled (default) and the server sends compressed content, the response body is transparently decompressed: | Content-Encoding | Handler | |---|---| | `gzip` | `compress/gzip` | | `deflate` | `compress/flate` | | `zlib` | Internal | | `bzip2` | Internal | | `lzw` | Internal | Use `--no-decompress` to preserve the compressed response. ### Checksum Verification If `--checksum` or `--checksum-file` is provided, the downloaded file is hashed and compared: ``` Expected: a1b2c3d4... Actual: a1b2c3d4... Checksum OK ``` Supported algorithms: SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b, MD5. ### Atomic Rename On successful completion, the temp file is atomically renamed: ```go os.Rename("file.zip.goget.tmp", "file.zip") ``` If the download fails or is interrupted, the temp file remains for resume. ### PGP Verification and Decryption If `--pgp-verify` or `--pgp-decrypt` is set, the downloaded file is processed via `golang.org/x/crypto/openpgp`: - **Detached signature verification** — `--pgp-sig file.sig --pgp-key public.key` - **Decryption** — `--pgp-decrypt --pgp-key private.key --pgp-passphrase "secret"` - Auto-detect signature files: `.asc`, `.sig` The decrypted file replaces the encrypted one (`.gpg` → stripped extension, or `.decrypted` suffix). ### HSTS Cache After every successful HTTPS connection, the HSTS cache (`~/.config/goget/hsts`) is updated per RFC 6797. Expired entries are pruned on load. This ensures that future `http://` requests to known hosts are automatically upgraded to `https://`. ### Graceful Shutdown (SIGINT) When the user sends Ctrl+C (SIGINT) during a download, goget persists partial progress to the `.goget.meta` sidecar for **all protocols** (HTTP, FTP, SFTP, WebDAV) and exits with code 130. A subsequent `goget --resume` picks up where it left off. The signal handler uses `signal.NotifyContext` and propagates cancellation through the download pipeline via `ctx.Err()` checks after every read. ## Smart Timeout When no explicit `--timeout` is set, goget calculates a timeout based on file size: ``` timeout = (fileSize / minSpeed) × safetyFactor ``` Where: - `minSpeed` = 10 KB/s (conservative minimum) - `safetyFactor` = 3.0 - `minTimeout` = 30 seconds - `maxTimeout` = 24 hours A 1 GB file at 10 KB/s → 34 hours (capped to 24 hours max). A 10 MB file at 10 KB/s → 51 minutes. ## Rate Limiting `--rate-limit` uses a token bucket algorithm: ```go bucket := transport.NewTokenBucket(rate) // For each read: bucket.Wait(n) ``` The token bucket allows short bursts above the limit while maintaining the average rate. ### Speed Format ``` --rate-limit 1MB/s # 1,000,000 bytes/sec --rate-limit 500KB/s # 500,000 bytes/sec --rate-limit 1GB/s # 1,000,000,000 bytes/sec --rate-limit 100 # 100 bytes/sec (plain number) ``` ## Retry Logic When a download fails, goget retries with exponential backoff: ``` Attempt 1: wait 1s, retry Attempt 2: wait 2s, retry Attempt 3: wait 4s, retry ... Attempt N: wait min(2^(N-1) × 1s, 30s), retry ``` Max retries default to 3. Use `--max-retries N` to increase. Use `--retry-all-errors` to retry on HTTP 4xx/5xx (by default, only 5xx server errors trigger retries).