petrbalvin/goget
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity
Files
11aed3771dbf2a006fb61a80568c84f18eb06409
goget/docs/architecture.md
T

8.4 KiB
Raw Blame History

Architecture

goget is a command-line download utility built on the Go standard library. It follows a layered architecture with a protocol abstraction layer, a transport layer, and utility modules for authentication, compression, output, and more.

High-Level Overview

graph TD
    User[User / Terminal]:::accent6
    Script[Script / CI]:::accent6
    CLI[cmd/goget/ - Main entry]:::accent0
    API[pkg/api/ - Library interface]:::accent3
    Core[internal/core/ - Types, Protocol Interface, Errors]:::accent1
    Registry[internal/protocol/ - Protocol Registry]:::accent1
    HTTP[internal/protocol/http/]:::accent1
    FTP[internal/protocol/ftp/]:::accent1
    SFTP[internal/protocol/sftp/]:::accent1
    FILE[internal/protocol/file/]:::accent1
    DataURL[internal/protocol/dataurl/]:::accent1
    Gopher[internal/protocol/gopher/]:::accent1
    Gemini[internal/protocol/gemini/]:::accent1
    WebDAV[internal/protocol/webdav/]:::accent1
    Transport[internal/transport/ - Dialer, Proxy, TLS, Rate Limit]:::accent7
    Config[internal/config/]:::accent7
    Output[internal/output/ - Atomic write, Resume, WARC]:::accent2
    Auth[internal/auth/ - Basic, Digest, OAuth, netrc]:::accent7
    Crypto[internal/crypto/ - Checksum, PGP, Certs]:::accent7
    Cookie[internal/cookie/ - Netscape jar]:::accent7
    HSTS[internal/hsts/ - RFC 6797 cache]:::accent7
    Archive[internal/archive/ - Extraction]:::accent7
    Recursive[internal/recursive/ - Crawler, Parser, Filter]:::accent2
    Mirror[internal/mirror/]:::accent2
    Metalink[internal/metalink/]:::accent2
    Queue[internal/queue/]:::accent2
    Compression[internal/compression/ - gzip, bzip2, zlib, flate, lzw]:::accent7
    Log[internal/log/ - Structured logging]:::accent7
    WARC[internal/warc/ - Archiving]:::accent7
    Register[internal/protocol/register/ - Single registration]:::accent7
    Format[internal/format/ - Bytes, Speed formatting]:::accent7
    LinkRewrite[internal/linkrewrite/ - HTML link conversion]:::accent7
    CLIUtil[internal/cli/ - Help, Progress bar, Colors]:::accent7

    User --> CLI
    Script --> API
    CLI --> Log
    CLI --> CLIUtil
    CLI --> Config
    CLI --> Core
    CLI -.-> API
    API --> Core
    API --> Registry

    subgraph "Download Pipeline"
        Core --> Registry
        Registry --> HTTP
        Registry --> FTP
        Registry --> SFTP
        Registry --> FILE
        Registry --> DataURL
        Registry --> Gopher
        Registry --> Gemini
        Registry --> WebDAV
        HTTP --> Transport
        HTTP --> Cookie
        HTTP --> HSTS
        HTTP --> Output
        FTP --> Transport
        SFTP --> Transport
    end

    subgraph "Post-Download"
        CLI --> Crypto
        CLI --> Archive
        CLI --> Compression
    end

    subgraph "Download Management"
        CLI --> Register
        CLI --> WARC
        CLI --> Recursive
        CLI --> Mirror
        CLI --> Metalink
        CLI --> Queue
        Recursive --> WARC
        Recursive --> LinkRewrite
        Mirror --> LinkRewrite
    end

    Transport --> Auth
    Transport --> Config

    classDef accent0 fill:#3B82F6,stroke:#2563EB,color:#fff
    classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
    classDef accent2 fill:#F59E0B,stroke:#D97706,color:#000
    classDef accent3 fill:#A855F7,stroke:#7C3AED,color:#fff
    classDef accent6 fill:#6366F1,stroke:#4F46E5,color:#fff
    classDef accent7 fill:#64748B,stroke:#475569,color:#fff

Layers

1. Entry Points

Layer Package Role
CLI cmd/goget/ Flag parsing, command routing, signal handling
Library API pkg/api/ Public Go API for programmatic downloads, uploads, and queue management

The CLI and API share the same internal packages. The CLI is built on top of the same internal/core types that the API exposes.

2. Protocol Abstraction

Defined in internal/core and internal/protocol:

// core.Protocol is the interface every protocol handler implements.
type Protocol interface {
    Scheme() string
    CanHandle(url *url.URL) bool
    Download(ctx context.Context, req *DownloadRequest) (*DownloadResult, error)
    Capabilities() []string
    SupportsResume() bool
    SupportsCompression() bool
    SupportsParallel() bool
    SupportsRecursive() bool
}

// core.Uploader is an optional interface for protocols supporting uploads.
type Uploader interface {
    Upload(ctx context.Context, req *UploadRequest) (*UploadResult, error)
    SupportsUpload() bool
}

Protocols register themselves by scheme in a global Registry. Scheme normalization maps HTTPS → HTTP, FTPS → FTP, etc. Supported protocols include HTTP, FTP, SFTP, File, DataURL, Gopher, Gemini, and WebDAV (new in internal/protocol/webdav/).

3. Transport Layer

internal/transport provides a custom HTTP transport with:

  • IPv6-first dialer — Resolves DNS, prefers IPv6 addresses, falls back to IPv4 only when no IPv6 record exists
  • Interface binding — Linux-specific socket binding via SO_BINDTODEVICE
  • Rate limiting — Token bucket algorithm for bandwidth capping
  • Retry logic — Exponential backoff with configurable max retries
  • TLS configuration — Certificate pinning, client certificates, key log file
  • Proxy support — HTTP CONNECT and SOCKS5 (socks5:// with local DNS, socks5h:// with remote DNS, RFC 1929 username/password auth)

4. Output Layer

internal/output handles atomic file writes:

  1. Write to a .goget.tmp file during download
  2. Store resume metadata in .goget.meta alongside the temp file
  3. Atomically rename from .goget.tmp to the final filename on completion

5. Utility Modules

Module Responsibility
auth HTTP Basic, Digest, OAuth 2.0 (client credentials, authorization code, refresh token), .netrc auto-reading
cookie Netscape-format cookie jar with import/export
crypto Checksum verification (SHA-256, SHA-512, SHA3-256/512, BLAKE2b, MD5), PGP signature verification and decryption, certificate handling
compression Decompression wrappers for gzip, bzip2, zlib, flate, LZW
hsts HSTS cache (RFC 6797) for automatic HTTPS upgrade
archive Archive extraction for tar, tar.gz, tar.bz2, zip with auto-detection
format Human-readable byte and speed formatting
linkrewrite HTML link rewriting for offline mirror viewing
log Structured logger with text and JSON output modes, level filtering, ANSI colors
warc WARC (Web ARChive) format writing for legal and archival compliance
cli ANSI color output, progress bar, help text, completion

Download Flow

sequenceDiagram
    participant User
    participant CLI as cmd/goget/
    participant Config as internal/config/
    participant Core as internal/core/
    participant Registry as internal/protocol/
    participant Proto as Protocol Handler
    participant Transport as internal/transport/
    participant Output as internal/output/

    User->>CLI: goget --url https://example.com/file.zip
    CLI->>Config: Load config file
    Config-->>CLI: Config (with defaults + overrides)
    CLI->>Core: Create DownloadRequest
    CLI->>Registry: Get protocol for URL scheme
    Registry-->>CLI: HTTP protocol handler
    CLI->>Proto: Download(ctx, request)
    Proto->>Transport: Dial + send HTTP request
    Transport-->>Proto: HTTP response stream
    Proto->>Output: Write chunks atomically
    Output-->>Proto: Progress + completion
    Proto-->>CLI: DownloadResult
    CLI-->>User: Progress bar + summary

Config Merging

Configuration is resolved in this priority order (highest first):

  1. CLI flags (--timeout 5m)
  2. Config file (~/.config/goget/config.toml)
  3. Default values

The config.Merge() method applies CLI override top of the loaded config.

Error Handling

All operations return structured core.GogetError with:

  • Type — Category (NETWORK, PROTOCOL, FILE, TIMEOUT, AUTH, CHECKSUM, UNSUPPORTED, CANCELLED)
  • Message — Human-readable description
  • Cause — Wrapped underlying error (available via errors.Unwrap)
  • URL — The URL that caused the error (credentials stripped)
  • Details — Optional key-value metadata (e.g., expected vs actual checksum)

Build Tags

All source files use the build constraint //go:build linux || freebsd. The project targets Linux and FreeBSD only. Platform-specific code (e.g., network interface binding) lives in internal/transport/bind_linux.go vs bind_other.go.

Reference in New Issue View Git Blame Copy Permalink