petrbalvin/goget
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity
Files
main
goget/docs/architecture.md
T

8.0 KiB
Raw Permalink Blame History

Architecture

goget is a command-line download utility built on the Go standard library. It follows a layered architecture with a protocol abstraction layer, a transport layer, and utility modules for authentication, compression, output, and more.

High-Level Overview

graph TD
    User[User / Terminal]
    Script[Script / CI]
    CLI[cmd/goget/ - Main entry]
    API[pkg/api/ - Library interface]
    Core[internal/core/ - Types, Protocol Interface, Errors]
    Registry[internal/protocol/ - Protocol Registry]
    HTTP[internal/protocol/http/]
    FTP[internal/protocol/ftp/]
    SFTP[internal/protocol/sftp/]
    FILE[internal/protocol/file/]
    DataURL[internal/protocol/dataurl/]
    Gopher[internal/protocol/gopher/]
    Gemini[internal/protocol/gemini/]
    WebDAV[internal/protocol/webdav/]
    Transport[internal/transport/ - Dialer, Proxy, TLS, Rate Limit]
    Config[internal/config/]
    Output[internal/output/ - Atomic write, Resume, WARC]
    Auth[internal/auth/ - Basic, Digest, OAuth, netrc]
    Crypto[internal/crypto/ - Checksum, PGP, Certs]
    Cookie[internal/cookie/ - Netscape jar]
    HSTS[internal/hsts/ - RFC 6797 cache]
    Archive[internal/archive/ - Extraction]
    Recursive[internal/recursive/ - Crawler, Parser, Filter]
    Mirror[internal/mirror/]
    Metalink[internal/metalink/]
    Queue[internal/queue/]
    Compression[internal/compression/ - gzip, bzip2, zlib, flate, lzw]
    Log[internal/log/ - Structured logging]
    WARC[internal/warc/ - Archiving]
    Register[internal/protocol/register/ - Single registration]
    Format[internal/format/ - Bytes, Speed formatting]
    LinkRewrite[internal/linkrewrite/ - HTML link conversion]
    CLIUtil[internal/cli/ - Help, Progress bar, Colors]
    Pool[internal/pool/ - Unified worker pool]

    User --> CLI
    Script --> API
    CLI --> Log
    CLI --> CLIUtil
    CLI --> Config
    CLI --> Core
    CLI -.-> API
    API --> Core
    API --> Registry

    subgraph "Download Pipeline"
        Core --> Registry
        Registry --> HTTP
        Registry --> FTP
        Registry --> SFTP
        Registry --> FILE
        Registry --> DataURL
        Registry --> Gopher
        Registry --> Gemini
        Registry --> WebDAV
        HTTP --> Transport
        HTTP --> Cookie
        HTTP --> HSTS
        HTTP --> Output
        FTP --> Transport
        FTP --> Pool
        SFTP --> Transport
        SFTP --> Pool
        WebDAV --> Pool
    end

    subgraph "Post-Download"
        CLI --> Crypto
        CLI --> Archive
        CLI --> Compression
    end

    subgraph "Download Management"
        CLI --> Register
        CLI --> WARC
        CLI --> Recursive
        CLI --> Mirror
        CLI --> Metalink
        CLI --> Queue
        Recursive --> WARC
        Recursive --> LinkRewrite
        Mirror --> LinkRewrite
    end

    Transport --> Auth
    Transport --> Config

Layers

1. Entry Points

Layer Package Role
CLI cmd/goget/ Flag parsing, command routing, signal handling
Library API pkg/api/ Public Go API for programmatic downloads, uploads, and queue management

The CLI and API share the same internal packages. The CLI is built on top of the same internal/core types that the API exposes.

2. Protocol Abstraction

Defined in internal/core and internal/protocol:

// core.Protocol is the interface every protocol handler implements.
type Protocol interface {
    Scheme() string
    CanHandle(url *url.URL) bool
    Download(ctx context.Context, req *DownloadRequest) (*DownloadResult, error)
    Capabilities() []string
    SupportsResume() bool
    SupportsCompression() bool
    SupportsParallel() bool
    SupportsRecursive() bool
}

// core.Uploader is an optional interface for protocols supporting uploads.
type Uploader interface {
    Upload(ctx context.Context, req *UploadRequest) (*UploadResult, error)
    SupportsUpload() bool
}

Protocols register themselves by scheme in a global Registry. Scheme normalization maps HTTPS → HTTP, FTPS → FTP, etc. Supported protocols include HTTP, FTP, SFTP, File, DataURL, Gopher, Gemini, and WebDAV (new in internal/protocol/webdav/).

3. Transport Layer

internal/transport provides a custom HTTP transport with:

  • IPv6-first dialer — Resolves DNS, prefers IPv6 addresses, falls back to IPv4 only when no IPv6 record exists
  • Interface binding — Linux-specific socket binding via SO_BINDTODEVICE
  • Rate limiting — Token bucket algorithm for bandwidth capping
  • Retry logic — Exponential backoff with configurable max retries
  • TLS configuration — Certificate pinning, client certificates, key log file
  • Proxy support — HTTP CONNECT and SOCKS5 (socks5:// with local DNS, socks5h:// with remote DNS, RFC 1929 username/password auth)

4. Output Layer

internal/output handles atomic file writes:

  1. Write to a .goget.tmp file during download
  2. Store resume metadata in .goget.meta alongside the temp file
  3. Atomically rename from .goget.tmp to the final filename on completion

5. Utility Modules

Module Responsibility
auth HTTP Basic, Digest, OAuth 2.0 (client credentials, authorization code, refresh token), .netrc auto-reading
cookie Netscape-format cookie jar with import/export
crypto Checksum verification (SHA-256, SHA-512, SHA3-256/512, BLAKE2b, MD5), PGP signature verification and decryption, certificate handling
compression Decompression wrappers for gzip, bzip2, zlib, flate, LZW
hsts HSTS cache (RFC 6797) for automatic HTTPS upgrade
archive Archive extraction for tar, tar.gz, tar.bz2, zip with auto-detection
format Human-readable byte and speed formatting
linkrewrite HTML link rewriting for offline mirror viewing
log Structured logger with text and JSON output modes, level filtering, ANSI colors
warc WARC (Web ARChive) format writing for legal and archival compliance
cli ANSI color output, progress bar, help text, completion
pool Protocol-agnostic recursive worker pool — bounded goroutine execution with recursive task submission. Replaces per-protocol semaphore+WaitGroup code in FTP, SFTP, and WebDAV recursive-parallel downloads

Download Flow

sequenceDiagram
    participant User
    participant CLI as cmd/goget/
    participant Config as internal/config/
    participant Core as internal/core/
    participant Registry as internal/protocol/
    participant Proto as Protocol Handler
    participant Transport as internal/transport/
    participant Output as internal/output/

    User->>CLI: goget --url https://example.com/file.zip
    CLI->>Config: Load config file
    Config-->>CLI: Config (with defaults + overrides)
    CLI->>Core: Create DownloadRequest
    CLI->>Registry: Get protocol for URL scheme
    Registry-->>CLI: HTTP protocol handler
    CLI->>Proto: Download(ctx, request)
    Proto->>Transport: Dial + send HTTP request
    Transport-->>Proto: HTTP response stream
    Proto->>Output: Write chunks atomically
    Output-->>Proto: Progress + completion
    Proto-->>CLI: DownloadResult
    CLI-->>User: Progress bar + summary

Config Merging

Configuration is resolved in this priority order (highest first):

  1. CLI flags (--timeout 5m)
  2. Config file (~/.config/goget/config.toml)
  3. Default values

The config.Merge() method applies CLI override top of the loaded config.

Error Handling

All operations return structured core.GogetError with:

  • Type — Category (NETWORK, PROTOCOL, FILE, TIMEOUT, AUTH, CHECKSUM, UNSUPPORTED, CANCELLED)
  • Message — Human-readable description
  • Cause — Wrapped underlying error (available via errors.Unwrap)
  • URL — The URL that caused the error (credentials stripped)
  • Details — Optional key-value metadata (e.g., expected vs actual checksum)

Build Tags

All source files use the build constraint //go:build linux || freebsd. The project targets Linux and FreeBSD only. Platform-specific code (e.g., network interface binding) lives in internal/transport/bind_linux.go vs bind_other.go.

Reference in New Issue View Git Blame Copy Permalink