petrbalvin/goget
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

2 Releases 2 Tags

RSS Feed
  • v1.1.0 b5fa009ec5
    Compare

    v1.1.0 Stable

    petrbalvin released this 2026-06-30 19:46:23 +00:00 | 0 commits to main since this release

    Added

    Documentation

    • Standalone troff manpage — man/goget.1 is now the single source
      of truth for the goget(1) manual. The previous embedded manpage listed
      only ~30 of the 138 CLI flags and was missing every standard section;
      the new file documents every flag with a dedicated entry, the
      full set of conventional .SH sections (NAME, SYNOPSIS, DESCRIPTION,
      OPTIONS, EXAMPLES, EXIT STATUS, FILES, ENVIRONMENT, SEE ALSO, HISTORY,
      AUTHORS, REPORTING BUGS, COPYRIGHT), curl/wget compatibility notes, and
      a release-history entry. The manpage is embedded into the binary via
      //go:embed, so --generate-man-page and man goget now agree.
    • just man — new task that runs groff -man -ww -Kutf-8 -z over
      man/goget.1 to validate the troff source. Skips cleanly when groff
      is not installed (CI on minimal images still passes).
    • just install-man and just uninstall-man — install the
      manpage to $HOME/.local/share/man/man1/goget.1.gz (XDG rootless,
      gzipped, reinstall-safe). After just install-man, man goget
      resolves the page automatically via the user's MANPATH.

    Core engine

    • Unified worker pool — single protocol-agnostic worker pool (internal/pool)
      used by FTP, SFTP, and WebDAV recursive-parallel downloads. Replaces three
      near-identical semaphore+WaitGroup implementations with a shared, tested
      concurrency primitive.
    • Graceful shutdown on SIGINT — persist .goget.meta sidecar on interrupt
      for SFTP single-file downloads, ensuring progress is saved for resume.
    • Gemini resume & metadata — Gemini downloads now persist .goget.meta
      sidecar on interrupt and delete it on successful completion. --resume
      continues interrupted Gemini transfers from the saved byte offset.
    • Gopher resume & metadata — Gopher typeHTML and typeTextFile
      responses now persist .goget.meta sidecar on interrupt and delete it on
      successful completion. --resume continues interrupted Gopher text
      transfers, skipping already-written output bytes (post-transform length
      because Gopher line metadata is stripped before writing).

    Security

    • Certificate pinning across all TLS protocols — --pinned-cert now
      applies to FTPS, Gemini, and WebDAVS in addition to HTTP/HTTPS. The leaf
      certificate SHA-256 hash is verified via tls.Config.VerifyPeerCertificate
      on every TLS-bearing protocol.
    • SSH host-key pinning for SFTP — new --pinned-host-key flag pins the
      SSH host key by its OpenSSH fingerprint (SHA256:<base64>, as produced by
      ssh-keygen -lf). Raw lowercase-hex SHA-256 of the key wire format is also
      accepted. Pinning takes precedence over --ssh-insecure and known_hosts.
    • WebDAV TLS config fix — webdav.Protocol.getHTTPTransport now builds
      its tls.Config from the caller-provided transport.TLSConfig instead of
      a fresh config that silently dropped CA certs, mTLS, keylog, and pinned-cert
      settings.

    Recursive downloading & mirroring

    • --adjust-extension extended to non-HTTP protocols — applies .html
      extension to text/html files downloaded via Gopher and WebDAV.
    • --backup-converted — keep .orig backups of files before link conversion
      during recursive mirroring. Previously only available for HTTP/HTTPS;
      now works for WebDAV mirrors too.

    Download management

    • --show-metadata — displays .goget.meta sidecar contents for debugging
      interrupted / resumed downloads. Accepts a path to the downloaded output
      (the sidecar is located next to it) or a direct path to the sidecar file.
      Renders a human-readable summary by default and switches to raw JSON
      output when combined with --json. Credentials embedded in the stored
      URL are masked before display (the password placeholder is xxxxx).

    Fixed

    • Gopher: ctx.Done() cancellation goroutine leak — the goroutine
      spawned in Download to close the socket on ctx.Done() previously
      waited indefinitely on the parent context, leaking every time the
      context outlived the function. Now bounded by a defer-closed
      done channel so the goroutine exits on every return path.

    • Gemini: ctx.Done() cancellation goroutine leak — same bug,
      same fix in downloadWithRedirects.

    • Gopher: partial state lost on cancel-then-close — when the server
      closed the connection after ctx was cancelled, downloadText
      treated the resulting EOF as a successful end-of-stream and never
      persisted .goget.meta. Mirrors the equivalent Gemini behavior
      (already present from the Gemini resume feature): if ctx.Err() != nil,
      save partial state and return the context error.

    • Gopher: --resume ignored when caller supplies Writer — when
      --resume was set together with a caller-supplied Writer, the
      partial-state sidecar would track bytes that landed in the Writer
      rather than Output. --resume is now silently disabled in that
      case (with a --verbose warning), keeping metadata consistent with
      the actual on-disk state.

    • Gemini: --resume ignored when caller supplies Writer — same
      fix applied to downloadWithRedirects.

    Testing

    • cmd/goget/man_page_test.go — new test file. Asserts the embedded
      manpage is non-empty, carries a .TH GOGET 1 header, contains every
      mandatory .SH section, and documents every flag registered by
      defineFlags (so adding a new flag without documenting it now breaks
      the build). When groff is installed, also runs groff -man -ww -Kutf-8 -z over the embedded source and fails on any warning.
    • Resume tests deterministic across the protocol stack
      (TestGeminiResume{SaveMetadata,Download} and
      TestGopherResume{SaveMetadata,Download}) — the four tests previously
      relied on a fixed time.Sleep before calling cancel(), making them
      flaky on slow CI schedulers. They now wait for a ProgressCallback
      signal, which fires synchronously after the first writer.Write
      lands on disk.
    • Goroutine leak regression suite
      (TestGeminiCancelGoroutineExits, TestGopherCancelGoroutineExits)
      — runs 50 downloads back-to-back, settles the runtime via
      runtime.GC(), and asserts runtime.NumGoroutine() returns to
      baseline + 1 within a 2-second window.
    • TestGopherResumeInformation — new regression test covering the
      typeInformation text-mode path through resume, which differs from
      typeHTML/typeTextFile because lines are passed through verbatim
      (no metadata strip).
    • TestGopherResumeStaleMetadata now asserts that an orphan
      .goget.meta carried over from a different URL is deleted once a
      successful fresh download completes, preventing it from being
      picked up by a future --resume run.
    Downloads
  • v1.0.0 53db81e1f7
    Compare

    v1.0.0 Stable

    petrbalvin released this 2026-06-26 19:22:01 +00:00 | 28 commits to main since this release

    First public release: a modern IPv6-first command-line download utility built on
    the Go standard library. goget provides parallel chunked downloads, resume
    support, checksum verification, recursive website mirroring, and support for 9
    network protocols — all in a single static binary with zero runtime dependencies.

    Added

    Core engine

    • Unified download and upload architecture via core.Protocol interface.
    • Protocol registry with scheme normalization (https→http, ftps→ftp).
    • Structured error types (NETWORK, PROTOCOL, FILE, TIMEOUT, AUTH,
      CHECKSUM, UNSUPPORTED, CANCELLED).
    • Request context with protocol chain tracking, attempt counting, and
      cancellation.
    • Smart timeout calculation based on file size, minimum speed, and safety factor.
    • IPv6-first transport dialer with automatic IPv4 fallback.

    Protocols

    • HTTP/HTTPS — HTTP/1.1 and HTTP/2, TLS 1.2/1.3, content negotiation,
      auto-decompression.
    • FTP/FTPS — active and passive modes, explicit TLS (AUTH TLS), resume via
      REST command.
    • SFTP — custom SFTP v3 implementation over SSH, known_hosts verification,
      resume, private key and password authentication.
    • WebDAV — file download, recursive directory download, upload via HTTP PUT,
      MKCOL for directory creation.
    • file:// — local file copy with recursive directory support.
    • data: — RFC 2397 data URIs (base64 and plain text).
    • Gopher — RFC 1436 with item type parsing.
    • Gemini — TLS 1.2+ connections, redirect following, status code handling
      (1x–6x).

    Network & transport

    • Concurrent chunked downloads for files over 100 MB with Range requests.
    • Per-chunk resume metadata for interrupted parallel downloads.
    • HTTP CONNECT and SOCKS5 (socks5://, socks5h://) proxy support with
      username/password authentication.
    • Custom DNS servers via --dns-servers.
    • Network interface binding via SO_BINDTODEVICE (Linux).
    • SSRF protection — blocks connections to private/internal IPs by default,
      override with --allow-private.
    • IPv4-mapped IPv6 normalization to prevent SSRF bypass.
    • Token bucket rate limiting.
    • Exponential backoff retry logic with configurable max retries.
    • --retry-all-errors to retry on HTTP 4xx/5xx responses.
    • --pinned-cert SHA-256 certificate pinning.

    Authentication

    • HTTP Basic authentication (RFC 7617).
    • HTTP Digest authentication (RFC 7616).
    • OAuth 2.0 — client credentials, authorization code, password, and refresh
      token grant types with 30-second expiry grace period.
    • OAuth error sanitization (RFC 6749 §5.2).
    • .netrc auto-reading with machine and default entry resolution.
    • Netscape-format cookie jar with import/export.
    • Per-request cookies via --cookie CLI flag.
    • SFTP SSH key auto-detection.
    • --password-file for secure password loading.
    • Config sanitization — strips credentials when saving to disk.
    • core.SafeURL() — strips credentials from URLs in error messages and logs.

    Recursive downloading & mirroring

    • Recursive HTML link following with configurable depth limit.
    • Full website mirroring with infinite depth and automatic link conversion.
    • HTML parsing via golang.org/x/net/html.
    • Link filtering by domain, accept pattern (glob and MIME), reject pattern, and
      exclusion list.
    • Page requisites — automatic CSS, JavaScript, and image downloading.
    • --no-parent, --span-hosts, --domains, --random-wait.
    • --adjust-extension, --timestamping, --cut-dirs.
    • Link rewriting for offline viewing — absolute, protocol-relative, and
      root-relative URLs.
    • --backup-converted keeps .orig backups before link conversion.
    • robots.txt compliance with --no-robots override.

    Verification & security

    • Checksum verification — SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b, MD5.
    • SHA256SUMS file parsing (plain and binary-mode).
    • PGP detached signature verification (.asc, .sig, .gpg).
    • PGP decryption with private key and passphrase.
    • HSTS cache (RFC 6797) — automatic HTTP→HTTPS upgrade.
    • TLS certificate pinning, mTLS support, custom CA bundle.
    • --ssl-key-log TLS session key logging (Wireshark-compatible).
    • Atomic file writes with mode 0600 for config, resume metadata, and temp
      files.

    Download management

    • Persistent download queue with priority (1–10) and JSON storage.
    • Multi-source Metalink downloads (.meta4, .metalink) with automatic
      failover.
    • Batch URL loading from files.
    • Cron-based scheduling (MIN HOUR DOM MON DOW and YYYY-MM-DD HH:MM).
    • Post-download command hooks with environment variables.
    • WARC (Web ARChive) output for archival compliance.
    • Upload support — HTTP PUT and POST with multipart form data.

    Output & progress

    • Unicode progress bar with filled/empty blocks and real-time speed.
    • Speed sparkline — mini ASCII chart of recent speed history.
    • Estimated time remaining (ETA) calculation.
    • Dot progress bar style, JSON progress output for scripting.
    • --write-out metadata output, --trace-time HTTP timing breakdown.
    • --show-headers, --keep-timestamps, --content-disposition.
    • Color output with always, never, and auto modes (honors NO_COLOR).

    Archive & compression

    • Transparent decompression — gzip, deflate, zlib, bzip2, LZW.
    • Archive extraction — TAR, TAR.GZ, TAR.BZ2, ZIP with auto-detection.

    Internationalization

    • IDN (International Domain Names) support with Punycode conversion via
      golang.org/x/net/idna.

    CLI

    • Long flags only (--url, --output, --verbose).
    • Shell completion generation — bash, zsh, and fish.
    • Man page generation in troff format.
    • Category-organized help text.
    • --info, --benchmark, --spider, --dry-run, --glob.

    Configuration

    • TOML configuration file at ~/.config/goget/config.toml.
    • GOGET_CONFIG environment variable for alternative config path.
    • CLI flag overrides merged on top of config and defaults.
    • Config commands — init, get, set, list, unset.

    Library API (pkg/api/)

    • Client type with functional options (WithTimeout, WithUserAgent,
      WithVerbose).
    • Client.Download() and Client.Upload() with structured request/result types.
    • QueueClient for programmatic queue management.
    • ParseMetalink(), FetchMetalink(), LoadConfig(), DefaultConfig(),
      ParseSpeed(), Version().
    • One-shot convenience functions: Download(), Upload().

    Build & platforms

    • Linux — amd64, arm64, riscv64, loong64.
    • FreeBSD — amd64, arm64, riscv64.
    • Static binary with zero runtime dependencies (stdlib only +
      golang.org/x/ packages).
    • Race detector enabled in tests, fuzz tests for security-critical parsers.
    • Coverage gate at 40%.

    Project & tooling

    • Documentation set: architecture, API, authentication, CLI reference,
      configuration, download pipeline, protocols, recursive & mirror, security.
    • just task set (install, run, build, test, uninstall).
    • Forgejo Actions CI (go vet, test + race, fuzz, release).
    • CONTRIBUTING.md with Conventional Commits and release process.
    Downloads