goget/docs/download-pipeline.md

Download Pipeline

This document explains how goget processes a download request from start to finish — the phase structure, parallel chunking, resume logic, and output writing.

High-Level Flow

flowchart TD
    Start([User invokes goget]):::accent0
    ParseFlags["Parse CLI flags"]:::accent7
    LoadConfig["Load config + merge overrides"]:::accent7
    ResolveProtocol["Resolve protocol handler\nfrom URL scheme"]:::accent1
    CreateRequest["Build core.DownloadRequest"]:::accent1
    CheckResume{"Resume metadata\nexists?"}:::accent2
    LoadResume["Load .goget.meta\n(resume info + chunk map)"]:::accent2
    SetupOutput["Setup output writer\n(atomic temp file + resume)"]:::accent2
    CreateDirs{"--create-dirs\n(parent missing)?"}:::accent2
    MkdirAll["os.MkdirAll → create\nparent directories"]:::accent2
    CheckParallel{"File > 100 MB\nor --parallel set?"}:::accent1
    SequentialDownload["Sequential download\n(single connection)"]:::accent1
    ParallelDownload["Parallel chunked download\n(multiple Range requests)"]:::accent1
    Decompress{"Auto-decompress\nenabled?"}:::accent7
    Decompression["Decompress response\n(gzip, deflate, bzip2, zlib)"]:::accent7
    VerifyChecksum{"Checksum\nspecified?"}:::accent7
    ChecksumVerify["Verify checksum\n(SHA-256, SHA-512, etc.)"]:::accent7
    PGPVerify{"PGP verify/\ndecrypt?"}:::accent7
    PGPProcess["Verify signature /\ndecrypt file"]:::accent7
    SaveHSTS["Save HSTS cache\n(RFC 6797)"]:::accent7
    AtomicRename["Atomic rename\ntemp → final"]:::accent2
    SigInt{"SIGINT received?"}:::accent4
    SaveSidecar["Save .goget.meta\nfor resume"]:::accent2
    Exit130["Exit 130"]:::accent4
    RunHook{"--on-complete\nset?"}:::accent7
    PostHook["Run post-download command\n(GOGET_OUTPUT, GOGET_SIZE, GOGET_URL)"]:::accent7
    Done([Done]):::accent0

    Start --> ParseFlags --> LoadConfig --> ResolveProtocol
    ResolveProtocol --> CreateRequest --> CheckResume
    CheckResume -->|Yes| LoadResume --> SetupOutput
    CheckResume -->|No| SetupOutput
    SetupOutput --> CreateDirs
    CreateDirs -->|Yes| MkdirAll --> CheckParallel
    CreateDirs -->|No| CheckParallel
    CheckParallel -->|Yes| ParallelDownload
    CheckParallel -->|No| SequentialDownload
    SequentialDownload --> Decompress
    ParallelDownload --> Decompress
    Decompress -->|Yes| Decompression --> VerifyChecksum
    Decompress -->|No| VerifyChecksum
    VerifyChecksum -->|Yes| ChecksumVerify -->|pass| PGPVerify
    ChecksumVerify -->|fail| Done
    VerifyChecksum -->|No| PGPVerify
    PGPVerify -->|Yes| PGPProcess --> SaveHSTS
    PGPVerify -->|No| SaveHSTS
    SaveHSTS --> AtomicRename
    AtomicRename --> SigInt
    SigInt -->|Yes| SaveSidecar --> Exit130
    SigInt -->|No| RunHook
    RunHook -->|Yes| PostHook --> Done
    RunHook -->|No| Done

    classDef accent0 fill:#3B82F6,stroke:#2563EB,color:#fff
    classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
    classDef accent2 fill:#F59E0B,stroke:#D97706,color:#000
    classDef accent4 fill:#EF4444,stroke:#DC2626,color:#fff
    classDef accent7 fill:#64748B,stroke:#475569,color:#fff

Phase 1: Request Construction

A core.DownloadRequest is built from CLI flags and config:

req := &core.DownloadRequest{
    URL:              parsedURL,
    Output:           outputPath,
    Resume:           resumeEnabled,
    Timeout:          effectiveTimeout,
    Verbose:          verbose,
    Headers:          customHeaders,
    Proxy:            proxyURL,
    Checksum:         expectedChecksum,
    Parallel:         parallelConfig,
    Recursive:        recursiveEnabled,
    MaxDepth:         maxDepth,
    ProgressCallback: progressFn,
    Ctx:              ctx,
}

Phase 2: Protocol Resolution

The protocol registry resolves the handler by URL scheme:

flowchart LR
    URL["https://example.com"]:::accent6
    Parse["Parse scheme\n→ https"]:::accent7
    Normalized["Normalize\nhttps → http"]:::accent7
    Registry["Registry lookup\nprotocols[http]"]:::accent1
    Handler["HTTP protocol\nhandler"]:::accent1

    URL --> Parse --> Normalized --> Registry --> Handler

    classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
    classDef accent6 fill:#6366F1,stroke:#4F46E5,color:#fff
    classDef accent7 fill:#64748B,stroke:#475569,color:#fff

Scheme normalization maps:

• https:// → http (same handler)
• ftps:// → ftp (same handler)
• webdavs:// → webdav (same handler)

Phase 3: Output Setup

The internal/output package creates an atomic writer:

1. Temp file — Writes to <filename>.goget.tmp during download
2. Parent directories — If --create-dirs is enabled (default: true), missing parent directories are created via os.MkdirAll before opening the output file. Pass --create-dirs=false to restore strict behaviour
3. Resume metadata — Reads <filename>.goget.meta if resuming
4. Progress callback — Hooks into the writer for real-time speed/ETA

writer, err := output.NewWriter(&output.WriterConfig{
    Output:           outputFile,
    Atomic:           true,
    Resume:           resumeEnabled,
    CreateDirs:       true,  // auto-create parent dirs (curl --create-dirs)
    ProgressCallback: progressFunc,
})

Resume Metadata Format

{
    "downloaded_bytes": 524288000,
    "etag": "\"abc123\"",
    "last_modified": "Mon, 01 Jun 2026 12:00:00 GMT",
    "url": "https://example.com/file.zip",
    "last_write": "2026-06-01T12:05:00Z",
    "chunks": {
        "0": 131072000,
        "1": 131072000,
        "2": 131072000,
        "3": 131072000
    }
}

For parallel downloads, each chunk's progress is individually tracked.

Phase 4: Download Strategy

Sequential Download

Used when the file is under 100 MB or --parallel 1 is set:

resp, _ := client.Do(request)
io.Copy(writer, resp.Body)

Parallel Chunked Download

Triggered automatically for files over 100 MB, or explicitly with --parallel N:

sequenceDiagram
    participant Main
    participant Chunk1
    participant Chunk2
    participant Chunk3
    participant Chunk4
    participant Writer as Atomic Writer

    Main->>Main: HEAD request → get file size
    Main->>Main: Split into N equal chunks
    Main->>Chunk1: Start: bytes 0-13107199
    Main->>Chunk2: Start: bytes 13107200-26214399
    Main->>Chunk3: Start: bytes 26214400-39321599
    Main->>Chunk4: Start: bytes 39321600-52428799
    Chunk1->>Writer: Write bytes to temp/chunk_0
    Chunk2->>Writer: Write bytes to temp/chunk_1
    Chunk3->>Writer: Write bytes to temp/chunk_2
    Chunk4->>Writer: Write bytes to temp/chunk_3
    Main->>Writer: Merge chunks → final file

Each chunk downloads via a separate HTTP Range request:

GET /large.iso HTTP/1.1
Host: example.com
Range: bytes=13107200-26214399

Concurrency Control

• Max 4 parallel connections by default
• Customizable via --parallel N or config parallel key
• Min chunk size: 1 MB
• Max chunk size: 50 MB

Phase 5: Post-Processing

Decompression

If auto_decompress is enabled (default) and the server sends compressed content, the response body is transparently decompressed:

Content-Encoding	Handler
gzip	compress/gzip
deflate	compress/flate
zlib	Internal
bzip2	Internal
lzw	Internal

Use --no-decompress to preserve the compressed response.

Checksum Verification

If --checksum or --checksum-file is provided, the downloaded file is hashed and compared:

Expected: a1b2c3d4...
Actual:   a1b2c3d4...
Checksum OK

Supported algorithms: SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b, MD5.

Atomic Rename

On successful completion, the temp file is atomically renamed:

os.Rename("file.zip.goget.tmp", "file.zip")

If the download fails or is interrupted, the temp file remains for resume.

PGP Verification and Decryption

If --pgp-verify or --pgp-decrypt is set, the downloaded file is processed via golang.org/x/crypto/openpgp:

• Detached signature verification — --pgp-sig file.sig --pgp-key public.key
• Decryption — --pgp-decrypt --pgp-key private.key --pgp-passphrase "secret"
• Auto-detect signature files: .asc, .sig

The decrypted file replaces the encrypted one (.gpg → stripped extension, or .decrypted suffix).

HSTS Cache

After every successful HTTPS connection, the HSTS cache (~/.config/goget/hsts) is updated per RFC 6797. Expired entries are pruned on load. This ensures that future http:// requests to known hosts are automatically upgraded to https://.

Graceful Shutdown (SIGINT)

When the user sends Ctrl+C (SIGINT) during a download, goget persists partial progress to the .goget.meta sidecar for all protocols (HTTP, FTP, SFTP, WebDAV) and exits with code 130. A subsequent goget --resume picks up where it left off. The signal handler uses signal.NotifyContext and propagates cancellation through the download pipeline via ctx.Err() checks after every read.

Smart Timeout

When no explicit --timeout is set, goget calculates a timeout based on file size:

timeout = (fileSize / minSpeed) × safetyFactor

Where:

• minSpeed = 10 KB/s (conservative minimum)
• safetyFactor = 3.0
• minTimeout = 30 seconds
• maxTimeout = 24 hours

A 1 GB file at 10 KB/s → 34 hours (capped to 24 hours max). A 10 MB file at 10 KB/s → 51 minutes.

Rate Limiting

--rate-limit uses a token bucket algorithm:

bucket := transport.NewTokenBucket(rate)
// For each read:
bucket.Wait(n)

The token bucket allows short bursts above the limit while maintaining the average rate.

Speed Format

--rate-limit 1MB/s    # 1,000,000 bytes/sec
--rate-limit 500KB/s  # 500,000 bytes/sec
--rate-limit 1GB/s    # 1,000,000,000 bytes/sec
--rate-limit 100      # 100 bytes/sec (plain number)

Retry Logic

When a download fails, goget retries with exponential backoff:

Attempt 1: wait 1s, retry
Attempt 2: wait 2s, retry
Attempt 3: wait 4s, retry
...
Attempt N: wait min(2^(N-1) × 1s, 30s), retry

Max retries default to 3. Use --max-retries N to increase. Use --retry-all-errors to retry on HTTP 4xx/5xx (by default, only 5xx server errors trigger retries).