petrbalvin/goget
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

2 Releases 2 Tags

RSS Feed
  • v1.0.0 53db81e1f7
    Compare

    v1.0.0 Stable

    petrbalvin released this 2026-06-26 19:22:01 +00:00 | 28 commits to main since this release

    First public release: a modern IPv6-first command-line download utility built on
    the Go standard library. goget provides parallel chunked downloads, resume
    support, checksum verification, recursive website mirroring, and support for 9
    network protocols — all in a single static binary with zero runtime dependencies.

    Added

    Core engine

    • Unified download and upload architecture via core.Protocol interface.
    • Protocol registry with scheme normalization (https→http, ftps→ftp).
    • Structured error types (NETWORK, PROTOCOL, FILE, TIMEOUT, AUTH,
      CHECKSUM, UNSUPPORTED, CANCELLED).
    • Request context with protocol chain tracking, attempt counting, and
      cancellation.
    • Smart timeout calculation based on file size, minimum speed, and safety factor.
    • IPv6-first transport dialer with automatic IPv4 fallback.

    Protocols

    • HTTP/HTTPS — HTTP/1.1 and HTTP/2, TLS 1.2/1.3, content negotiation,
      auto-decompression.
    • FTP/FTPS — active and passive modes, explicit TLS (AUTH TLS), resume via
      REST command.
    • SFTP — custom SFTP v3 implementation over SSH, known_hosts verification,
      resume, private key and password authentication.
    • WebDAV — file download, recursive directory download, upload via HTTP PUT,
      MKCOL for directory creation.
    • file:// — local file copy with recursive directory support.
    • data: — RFC 2397 data URIs (base64 and plain text).
    • Gopher — RFC 1436 with item type parsing.
    • Gemini — TLS 1.2+ connections, redirect following, status code handling
      (1x–6x).

    Network & transport

    • Concurrent chunked downloads for files over 100 MB with Range requests.
    • Per-chunk resume metadata for interrupted parallel downloads.
    • HTTP CONNECT and SOCKS5 (socks5://, socks5h://) proxy support with
      username/password authentication.
    • Custom DNS servers via --dns-servers.
    • Network interface binding via SO_BINDTODEVICE (Linux).
    • SSRF protection — blocks connections to private/internal IPs by default,
      override with --allow-private.
    • IPv4-mapped IPv6 normalization to prevent SSRF bypass.
    • Token bucket rate limiting.
    • Exponential backoff retry logic with configurable max retries.
    • --retry-all-errors to retry on HTTP 4xx/5xx responses.
    • --pinned-cert SHA-256 certificate pinning.

    Authentication

    • HTTP Basic authentication (RFC 7617).
    • HTTP Digest authentication (RFC 7616).
    • OAuth 2.0 — client credentials, authorization code, password, and refresh
      token grant types with 30-second expiry grace period.
    • OAuth error sanitization (RFC 6749 §5.2).
    • .netrc auto-reading with machine and default entry resolution.
    • Netscape-format cookie jar with import/export.
    • Per-request cookies via --cookie CLI flag.
    • SFTP SSH key auto-detection.
    • --password-file for secure password loading.
    • Config sanitization — strips credentials when saving to disk.
    • core.SafeURL() — strips credentials from URLs in error messages and logs.

    Recursive downloading & mirroring

    • Recursive HTML link following with configurable depth limit.
    • Full website mirroring with infinite depth and automatic link conversion.
    • HTML parsing via golang.org/x/net/html.
    • Link filtering by domain, accept pattern (glob and MIME), reject pattern, and
      exclusion list.
    • Page requisites — automatic CSS, JavaScript, and image downloading.
    • --no-parent, --span-hosts, --domains, --random-wait.
    • --adjust-extension, --timestamping, --cut-dirs.
    • Link rewriting for offline viewing — absolute, protocol-relative, and
      root-relative URLs.
    • --backup-converted keeps .orig backups before link conversion.
    • robots.txt compliance with --no-robots override.

    Verification & security

    • Checksum verification — SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b, MD5.
    • SHA256SUMS file parsing (plain and binary-mode).
    • PGP detached signature verification (.asc, .sig, .gpg).
    • PGP decryption with private key and passphrase.
    • HSTS cache (RFC 6797) — automatic HTTP→HTTPS upgrade.
    • TLS certificate pinning, mTLS support, custom CA bundle.
    • --ssl-key-log TLS session key logging (Wireshark-compatible).
    • Atomic file writes with mode 0600 for config, resume metadata, and temp
      files.

    Download management

    • Persistent download queue with priority (1–10) and JSON storage.
    • Multi-source Metalink downloads (.meta4, .metalink) with automatic
      failover.
    • Batch URL loading from files.
    • Cron-based scheduling (MIN HOUR DOM MON DOW and YYYY-MM-DD HH:MM).
    • Post-download command hooks with environment variables.
    • WARC (Web ARChive) output for archival compliance.
    • Upload support — HTTP PUT and POST with multipart form data.

    Output & progress

    • Unicode progress bar with filled/empty blocks and real-time speed.
    • Speed sparkline — mini ASCII chart of recent speed history.
    • Estimated time remaining (ETA) calculation.
    • Dot progress bar style, JSON progress output for scripting.
    • --write-out metadata output, --trace-time HTTP timing breakdown.
    • --show-headers, --keep-timestamps, --content-disposition.
    • Color output with always, never, and auto modes (honors NO_COLOR).

    Archive & compression

    • Transparent decompression — gzip, deflate, zlib, bzip2, LZW.
    • Archive extraction — TAR, TAR.GZ, TAR.BZ2, ZIP with auto-detection.

    Internationalization

    • IDN (International Domain Names) support with Punycode conversion via
      golang.org/x/net/idna.

    CLI

    • Long flags only (--url, --output, --verbose).
    • Shell completion generation — bash, zsh, and fish.
    • Man page generation in troff format.
    • Category-organized help text.
    • --info, --benchmark, --spider, --dry-run, --glob.

    Configuration

    • TOML configuration file at ~/.config/goget/config.toml.
    • GOGET_CONFIG environment variable for alternative config path.
    • CLI flag overrides merged on top of config and defaults.
    • Config commands — init, get, set, list, unset.

    Library API (pkg/api/)

    • Client type with functional options (WithTimeout, WithUserAgent,
      WithVerbose).
    • Client.Download() and Client.Upload() with structured request/result types.
    • QueueClient for programmatic queue management.
    • ParseMetalink(), FetchMetalink(), LoadConfig(), DefaultConfig(),
      ParseSpeed(), Version().
    • One-shot convenience functions: Download(), Upload().

    Build & platforms

    • Linux — amd64, arm64, riscv64, loong64.
    • FreeBSD — amd64, arm64, riscv64.
    • Static binary with zero runtime dependencies (stdlib only +
      golang.org/x/ packages).
    • Race detector enabled in tests, fuzz tests for security-critical parsers.
    • Coverage gate at 40%.

    Project & tooling

    • Documentation set: architecture, API, authentication, CLI reference,
      configuration, download pipeline, protocols, recursive & mirror, security.
    • just task set (install, run, build, test, uninstall).
    • Forgejo Actions CI (go vet, test + race, fuzz, release).
    • CONTRIBUTING.md with Conventional Commits and release process.
    Downloads