goget/CHANGELOG.md

Changelog

All notable changes to goget are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

1.1.0 — 2026-06-30

Added

Documentation

• Standalone troff manpage — man/goget.1 is now the single source of truth for the goget(1) manual. The previous embedded manpage listed only ~30 of the 138 CLI flags and was missing every standard section; the new file documents every flag with a dedicated entry, the full set of conventional .SH sections (NAME, SYNOPSIS, DESCRIPTION, OPTIONS, EXAMPLES, EXIT STATUS, FILES, ENVIRONMENT, SEE ALSO, HISTORY, AUTHORS, REPORTING BUGS, COPYRIGHT), curl/wget compatibility notes, and a release-history entry. The manpage is embedded into the binary via //go:embed, so --generate-man-page and man goget now agree.
• just man — new task that runs groff -man -ww -Kutf-8 -z over man/goget.1 to validate the troff source. Skips cleanly when groff is not installed (CI on minimal images still passes).
• just install-man and just uninstall-man — install the manpage to $HOME/.local/share/man/man1/goget.1.gz (XDG rootless, gzipped, reinstall-safe). After just install-man, man goget resolves the page automatically via the user's MANPATH.

Core engine

• Unified worker pool — single protocol-agnostic worker pool (internal/pool) used by FTP, SFTP, and WebDAV recursive-parallel downloads. Replaces three near-identical semaphore+WaitGroup implementations with a shared, tested concurrency primitive.
• Graceful shutdown on SIGINT — persist .goget.meta sidecar on interrupt for SFTP single-file downloads, ensuring progress is saved for resume.
• Gemini resume & metadata — Gemini downloads now persist .goget.meta sidecar on interrupt and delete it on successful completion. --resume continues interrupted Gemini transfers from the saved byte offset.
• Gopher resume & metadata — Gopher typeHTML and typeTextFile responses now persist .goget.meta sidecar on interrupt and delete it on successful completion. --resume continues interrupted Gopher text transfers, skipping already-written output bytes (post-transform length because Gopher line metadata is stripped before writing).

Security

• Certificate pinning across all TLS protocols — --pinned-cert now applies to FTPS, Gemini, and WebDAVS in addition to HTTP/HTTPS. The leaf certificate SHA-256 hash is verified via tls.Config.VerifyPeerCertificate on every TLS-bearing protocol.
• SSH host-key pinning for SFTP — new --pinned-host-key flag pins the SSH host key by its OpenSSH fingerprint (SHA256:<base64>, as produced by ssh-keygen -lf). Raw lowercase-hex SHA-256 of the key wire format is also accepted. Pinning takes precedence over --ssh-insecure and known_hosts.
• WebDAV TLS config fix — webdav.Protocol.getHTTPTransport now builds its tls.Config from the caller-provided transport.TLSConfig instead of a fresh config that silently dropped CA certs, mTLS, keylog, and pinned-cert settings.

Recursive downloading & mirroring

• --adjust-extension extended to non-HTTP protocols — applies .html extension to text/html files downloaded via Gopher and WebDAV.
• --backup-converted — keep .orig backups of files before link conversion during recursive mirroring. Previously only available for HTTP/HTTPS; now works for WebDAV mirrors too.

Download management

• --show-metadata — displays .goget.meta sidecar contents for debugging interrupted / resumed downloads. Accepts a path to the downloaded output (the sidecar is located next to it) or a direct path to the sidecar file. Renders a human-readable summary by default and switches to raw JSON output when combined with --json. Credentials embedded in the stored URL are masked before display (the password placeholder is xxxxx).

Fixed

• Gopher: ctx.Done() cancellation goroutine leak — the goroutine spawned in Download to close the socket on ctx.Done() previously waited indefinitely on the parent context, leaking every time the context outlived the function. Now bounded by a defer-closed done channel so the goroutine exits on every return path.

• Gemini: ctx.Done() cancellation goroutine leak — same bug, same fix in downloadWithRedirects.

• Gopher: partial state lost on cancel-then-close — when the server closed the connection after ctx was cancelled, downloadText treated the resulting EOF as a successful end-of-stream and never persisted .goget.meta. Mirrors the equivalent Gemini behavior (already present from the Gemini resume feature): if ctx.Err() != nil, save partial state and return the context error.

• Gopher: --resume ignored when caller supplies Writer — when --resume was set together with a caller-supplied Writer, the partial-state sidecar would track bytes that landed in the Writer rather than Output. --resume is now silently disabled in that case (with a --verbose warning), keeping metadata consistent with the actual on-disk state.

• Gemini: --resume ignored when caller supplies Writer — same fix applied to downloadWithRedirects.

Testing

• cmd/goget/man_page_test.go — new test file. Asserts the embedded manpage is non-empty, carries a .TH GOGET 1 header, contains every mandatory .SH section, and documents every flag registered by defineFlags (so adding a new flag without documenting it now breaks the build). When groff is installed, also runs groff -man -ww -Kutf-8 -z over the embedded source and fails on any warning.
• Resume tests deterministic across the protocol stack (TestGeminiResume{SaveMetadata,Download} and TestGopherResume{SaveMetadata,Download}) — the four tests previously relied on a fixed time.Sleep before calling cancel(), making them flaky on slow CI schedulers. They now wait for a ProgressCallback signal, which fires synchronously after the first writer.Write lands on disk.
• Goroutine leak regression suite (TestGeminiCancelGoroutineExits, TestGopherCancelGoroutineExits) — runs 50 downloads back-to-back, settles the runtime via runtime.GC(), and asserts runtime.NumGoroutine() returns to baseline + 1 within a 2-second window.
• TestGopherResumeInformation — new regression test covering the typeInformation text-mode path through resume, which differs from typeHTML/typeTextFile because lines are passed through verbatim (no metadata strip).
• TestGopherResumeStaleMetadata now asserts that an orphan .goget.meta carried over from a different URL is deleted once a successful fresh download completes, preventing it from being picked up by a future --resume run.

1.0.0 — 2026-06-26

First public release: a modern IPv6-first command-line download utility built on the Go standard library. goget provides parallel chunked downloads, resume support, checksum verification, recursive website mirroring, and support for 9 network protocols — all in a single static binary with zero runtime dependencies.

Added

Core engine

• Unified download and upload architecture via core.Protocol interface.
• Protocol registry with scheme normalization (https→http, ftps→ftp).
• Structured error types (NETWORK, PROTOCOL, FILE, TIMEOUT, AUTH, CHECKSUM, UNSUPPORTED, CANCELLED).
• Request context with protocol chain tracking, attempt counting, and cancellation.
• Smart timeout calculation based on file size, minimum speed, and safety factor.
• IPv6-first transport dialer with automatic IPv4 fallback.

Protocols

• HTTP/HTTPS — HTTP/1.1 and HTTP/2, TLS 1.2/1.3, content negotiation, auto-decompression.
• FTP/FTPS — active and passive modes, explicit TLS (AUTH TLS), resume via REST command.
• SFTP — custom SFTP v3 implementation over SSH, known_hosts verification, resume, private key and password authentication.
• WebDAV — file download, recursive directory download, upload via HTTP PUT, MKCOL for directory creation.
• file:// — local file copy with recursive directory support.
• data: — RFC 2397 data URIs (base64 and plain text).
• Gopher — RFC 1436 with item type parsing.
• Gemini — TLS 1.2+ connections, redirect following, status code handling (1x–6x).

Network & transport

• Concurrent chunked downloads for files over 100 MB with Range requests.
• Per-chunk resume metadata for interrupted parallel downloads.
• HTTP CONNECT and SOCKS5 (socks5://, socks5h://) proxy support with username/password authentication.
• Custom DNS servers via --dns-servers.
• Network interface binding via SO_BINDTODEVICE (Linux).
• SSRF protection — blocks connections to private/internal IPs by default, override with --allow-private.
• IPv4-mapped IPv6 normalization to prevent SSRF bypass.
• Token bucket rate limiting.
• Exponential backoff retry logic with configurable max retries.
• --retry-all-errors to retry on HTTP 4xx/5xx responses.
• --pinned-cert SHA-256 certificate pinning.

Authentication

• HTTP Basic authentication (RFC 7617).
• HTTP Digest authentication (RFC 7616).
• OAuth 2.0 — client credentials, authorization code, password, and refresh token grant types with 30-second expiry grace period.
• OAuth error sanitization (RFC 6749 §5.2).
• .netrc auto-reading with machine and default entry resolution.
• Netscape-format cookie jar with import/export.
• Per-request cookies via --cookie CLI flag.
• SFTP SSH key auto-detection.
• --password-file for secure password loading.
• Config sanitization — strips credentials when saving to disk.
• core.SafeURL() — strips credentials from URLs in error messages and logs.

Recursive downloading & mirroring

• Recursive HTML link following with configurable depth limit.
• Full website mirroring with infinite depth and automatic link conversion.
• HTML parsing via golang.org/x/net/html.
• Link filtering by domain, accept pattern (glob and MIME), reject pattern, and exclusion list.
• Page requisites — automatic CSS, JavaScript, and image downloading.
• --no-parent, --span-hosts, --domains, --random-wait.
• --adjust-extension, --timestamping, --cut-dirs.
• Link rewriting for offline viewing — absolute, protocol-relative, and root-relative URLs.
• --backup-converted keeps .orig backups before link conversion.
• robots.txt compliance with --no-robots override.

Verification & security

• Checksum verification — SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b, MD5.
• SHA256SUMS file parsing (plain and binary-mode).
• PGP detached signature verification (.asc, .sig, .gpg).
• PGP decryption with private key and passphrase.
• HSTS cache (RFC 6797) — automatic HTTP→HTTPS upgrade.
• TLS certificate pinning, mTLS support, custom CA bundle.
• --ssl-key-log TLS session key logging (Wireshark-compatible).
• Atomic file writes with mode 0600 for config, resume metadata, and temp files.

Download management

• Persistent download queue with priority (1–10) and JSON storage.
• Multi-source Metalink downloads (.meta4, .metalink) with automatic failover.
• Batch URL loading from files.
• Cron-based scheduling (MIN HOUR DOM MON DOW and YYYY-MM-DD HH:MM).
• Post-download command hooks with environment variables.
• WARC (Web ARChive) output for archival compliance.
• Upload support — HTTP PUT and POST with multipart form data.

Output & progress

• Unicode progress bar with filled/empty blocks and real-time speed.
• Speed sparkline — mini ASCII chart of recent speed history.
• Estimated time remaining (ETA) calculation.
• Dot progress bar style, JSON progress output for scripting.
• --write-out metadata output, --trace-time HTTP timing breakdown.
• --show-headers, --keep-timestamps, --content-disposition.
• Color output with always, never, and auto modes (honors NO_COLOR).

Archive & compression

• Transparent decompression — gzip, deflate, zlib, bzip2, LZW.
• Archive extraction — TAR, TAR.GZ, TAR.BZ2, ZIP with auto-detection.

Internationalization

• IDN (International Domain Names) support with Punycode conversion via golang.org/x/net/idna.

CLI

• Long flags only (--url, --output, --verbose).
• Shell completion generation — bash, zsh, and fish.
• Man page generation in troff format.
• Category-organized help text.
• --info, --benchmark, --spider, --dry-run, --glob.

Configuration

• TOML configuration file at ~/.config/goget/config.toml.
• GOGET_CONFIG environment variable for alternative config path.
• CLI flag overrides merged on top of config and defaults.
• Config commands — init, get, set, list, unset.

Library API (pkg/api/)

• Client type with functional options (WithTimeout, WithUserAgent, WithVerbose).
• Client.Download() and Client.Upload() with structured request/result types.
• QueueClient for programmatic queue management.
• ParseMetalink(), FetchMetalink(), LoadConfig(), DefaultConfig(), ParseSpeed(), Version().
• One-shot convenience functions: Download(), Upload().

Build & platforms

• Linux — amd64, arm64, riscv64, loong64.
• FreeBSD — amd64, arm64, riscv64.
• Static binary with zero runtime dependencies (stdlib only + golang.org/x/ packages).
• Race detector enabled in tests, fuzz tests for security-critical parsers.
• Coverage gate at 40%.

Project & tooling

• Documentation set: architecture, API, authentication, CLI reference, configuration, download pipeline, protocols, recursive & mirror, security.
• just task set (install, run, build, test, uninstall).
• Forgejo Actions CI (go vet, test + race, fuzz, release).
• CONTRIBUTING.md with Conventional Commits and release process.