goget/CHANGELOG.md

# Changelog

All notable changes to **goget** are documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.1.0] — 2026-06-30

### Added

**Documentation**

- **Standalone troff manpage** — `man/goget.1` is now the single source
  of truth for the goget(1) manual. The previous embedded manpage listed
  only ~30 of the 138 CLI flags and was missing every standard section;
  the new file documents every flag with a dedicated entry, the
  full set of conventional `.SH` sections (NAME, SYNOPSIS, DESCRIPTION,
  OPTIONS, EXAMPLES, EXIT STATUS, FILES, ENVIRONMENT, SEE ALSO, HISTORY,
  AUTHORS, REPORTING BUGS, COPYRIGHT), curl/wget compatibility notes, and
  a release-history entry. The manpage is embedded into the binary via
  `//go:embed`, so `--generate-man-page` and `man goget` now agree.
- **`just man`** — new task that runs `groff -man -ww -Kutf-8 -z` over
  `man/goget.1` to validate the troff source. Skips cleanly when `groff`
  is not installed (CI on minimal images still passes).
- **`just install-man`** and **`just uninstall-man`** — install the
  manpage to `$HOME/.local/share/man/man1/goget.1.gz` (XDG rootless,
  gzipped, reinstall-safe). After `just install-man`, `man goget`
  resolves the page automatically via the user's `MANPATH`.

**Core engine**

- **Unified worker pool** — single protocol-agnostic worker pool (`internal/pool`)
  used by FTP, SFTP, and WebDAV recursive-parallel downloads. Replaces three
  near-identical semaphore+WaitGroup implementations with a shared, tested
  concurrency primitive.
- **Graceful shutdown on SIGINT** — persist `.goget.meta` sidecar on interrupt
  for SFTP single-file downloads, ensuring progress is saved for resume.
- **Gemini resume & metadata** — Gemini downloads now persist `.goget.meta`
  sidecar on interrupt and delete it on successful completion. `--resume`
  continues interrupted Gemini transfers from the saved byte offset.
- **Gopher resume & metadata** — Gopher `typeHTML` and `typeTextFile`
  responses now persist `.goget.meta` sidecar on interrupt and delete it on
  successful completion. `--resume` continues interrupted Gopher text
  transfers, skipping already-written output bytes (post-transform length
  because Gopher line metadata is stripped before writing).

**Security**

- **Certificate pinning across all TLS protocols** — `--pinned-cert` now
  applies to FTPS, Gemini, and WebDAVS in addition to HTTP/HTTPS. The leaf
  certificate SHA-256 hash is verified via `tls.Config.VerifyPeerCertificate`
  on every TLS-bearing protocol.
- **SSH host-key pinning for SFTP** — new `--pinned-host-key` flag pins the
  SSH host key by its OpenSSH fingerprint (`SHA256:<base64>`, as produced by
  `ssh-keygen -lf`). Raw lowercase-hex SHA-256 of the key wire format is also
  accepted. Pinning takes precedence over `--ssh-insecure` and `known_hosts`.
- **WebDAV TLS config fix** — `webdav.Protocol.getHTTPTransport` now builds
  its `tls.Config` from the caller-provided `transport.TLSConfig` instead of
  a fresh config that silently dropped CA certs, mTLS, keylog, and pinned-cert
  settings.

**Recursive downloading & mirroring**

- `--adjust-extension` extended to non-HTTP protocols — applies `.html`
  extension to `text/html` files downloaded via Gopher and WebDAV.
- `--backup-converted` — keep `.orig` backups of files before link conversion
  during recursive mirroring. Previously only available for HTTP/HTTPS;
  now works for WebDAV mirrors too.

**Download management**

- `--show-metadata` — displays `.goget.meta` sidecar contents for debugging
  interrupted / resumed downloads. Accepts a path to the downloaded output
  (the sidecar is located next to it) or a direct path to the sidecar file.
  Renders a human-readable summary by default and switches to raw JSON
  output when combined with `--json`. Credentials embedded in the stored
  URL are masked before display (the password placeholder is `xxxxx`).

### Fixed

- **Gopher: `ctx.Done()` cancellation goroutine leak** — the goroutine
  spawned in `Download` to close the socket on `ctx.Done()` previously
  waited indefinitely on the parent context, leaking every time the
  context outlived the function. Now bounded by a `defer`-closed
  `done` channel so the goroutine exits on every return path.

- **Gemini: `ctx.Done()` cancellation goroutine leak** — same bug,
  same fix in `downloadWithRedirects`.

- **Gopher: partial state lost on cancel-then-close** — when the server
  closed the connection after `ctx` was cancelled, `downloadText`
  treated the resulting EOF as a successful end-of-stream and never
  persisted `.goget.meta`. Mirrors the equivalent Gemini behavior
  (already present from the Gemini resume feature): if `ctx.Err() != nil`,
  save partial state and return the context error.

- **Gopher: `--resume` ignored when caller supplies `Writer`** — when
  `--resume` was set together with a caller-supplied `Writer`, the
  partial-state sidecar would track bytes that landed in the `Writer`
  rather than `Output`. `--resume` is now silently disabled in that
  case (with a `--verbose` warning), keeping metadata consistent with
  the actual on-disk state.

- **Gemini: `--resume` ignored when caller supplies `Writer`** — same
  fix applied to `downloadWithRedirects`.

### Testing

- `cmd/goget/man_page_test.go` — new test file. Asserts the embedded
  manpage is non-empty, carries a `.TH GOGET 1` header, contains every
  mandatory `.SH` section, and documents every flag registered by
  `defineFlags` (so adding a new flag without documenting it now breaks
  the build). When `groff` is installed, also runs `groff -man -ww
  -Kutf-8 -z` over the embedded source and fails on any warning.
- Resume tests deterministic across the protocol stack
  (`TestGeminiResume{SaveMetadata,Download}` and
  `TestGopherResume{SaveMetadata,Download}`) — the four tests previously
  relied on a fixed `time.Sleep` before calling `cancel()`, making them
  flaky on slow CI schedulers. They now wait for a `ProgressCallback`
  signal, which fires synchronously after the first `writer.Write`
  lands on disk.
- Goroutine leak regression suite
  (`TestGeminiCancelGoroutineExits`, `TestGopherCancelGoroutineExits`)
  — runs 50 downloads back-to-back, settles the runtime via
  `runtime.GC()`, and asserts `runtime.NumGoroutine()` returns to
  baseline + 1 within a 2-second window.
- `TestGopherResumeInformation` — new regression test covering the
  `typeInformation` text-mode path through resume, which differs from
  `typeHTML`/`typeTextFile` because lines are passed through verbatim
  (no metadata strip).
- `TestGopherResumeStaleMetadata` now asserts that an orphan
  `.goget.meta` carried over from a different URL is deleted once a
  successful fresh download completes, preventing it from being
  picked up by a future `--resume` run.

## [1.0.0] — 2026-06-26

First public release: a modern IPv6-first command-line download utility built on
the Go standard library. goget provides parallel chunked downloads, resume
support, checksum verification, recursive website mirroring, and support for 9
network protocols — all in a single static binary with zero runtime dependencies.

### Added

**Core engine**

- Unified download and upload architecture via `core.Protocol` interface.
- Protocol registry with scheme normalization (https→http, ftps→ftp).
- Structured error types (`NETWORK`, `PROTOCOL`, `FILE`, `TIMEOUT`, `AUTH`,
  `CHECKSUM`, `UNSUPPORTED`, `CANCELLED`).
- Request context with protocol chain tracking, attempt counting, and
  cancellation.
- Smart timeout calculation based on file size, minimum speed, and safety factor.
- IPv6-first transport dialer with automatic IPv4 fallback.

**Protocols**

- **HTTP/HTTPS** — HTTP/1.1 and HTTP/2, TLS 1.2/1.3, content negotiation,
  auto-decompression.
- **FTP/FTPS** — active and passive modes, explicit TLS (AUTH TLS), resume via
  REST command.
- **SFTP** — custom SFTP v3 implementation over SSH, `known_hosts` verification,
  resume, private key and password authentication.
- **WebDAV** — file download, recursive directory download, upload via HTTP PUT,
  MKCOL for directory creation.
- **file://** — local file copy with recursive directory support.
- **data:** — RFC 2397 data URIs (base64 and plain text).
- **Gopher** — RFC 1436 with item type parsing.
- **Gemini** — TLS 1.2+ connections, redirect following, status code handling
  (1x–6x).

**Network & transport**

- Concurrent chunked downloads for files over 100 MB with `Range` requests.
- Per-chunk resume metadata for interrupted parallel downloads.
- HTTP CONNECT and SOCKS5 (`socks5://`, `socks5h://`) proxy support with
  username/password authentication.
- Custom DNS servers via `--dns-servers`.
- Network interface binding via `SO_BINDTODEVICE` (Linux).
- SSRF protection — blocks connections to private/internal IPs by default,
  override with `--allow-private`.
- IPv4-mapped IPv6 normalization to prevent SSRF bypass.
- Token bucket rate limiting.
- Exponential backoff retry logic with configurable max retries.
- `--retry-all-errors` to retry on HTTP 4xx/5xx responses.
- `--pinned-cert` SHA-256 certificate pinning.

**Authentication**

- HTTP Basic authentication (RFC 7617).
- HTTP Digest authentication (RFC 7616).
- OAuth 2.0 — client credentials, authorization code, password, and refresh
  token grant types with 30-second expiry grace period.
- OAuth error sanitization (RFC 6749 §5.2).
- `.netrc` auto-reading with machine and default entry resolution.
- Netscape-format cookie jar with import/export.
- Per-request cookies via `--cookie` CLI flag.
- SFTP SSH key auto-detection.
- `--password-file` for secure password loading.
- Config sanitization — strips credentials when saving to disk.
- `core.SafeURL()` — strips credentials from URLs in error messages and logs.

**Recursive downloading & mirroring**

- Recursive HTML link following with configurable depth limit.
- Full website mirroring with infinite depth and automatic link conversion.
- HTML parsing via `golang.org/x/net/html`.
- Link filtering by domain, accept pattern (glob and MIME), reject pattern, and
  exclusion list.
- Page requisites — automatic CSS, JavaScript, and image downloading.
- `--no-parent`, `--span-hosts`, `--domains`, `--random-wait`.
- `--adjust-extension`, `--timestamping`, `--cut-dirs`.
- Link rewriting for offline viewing — absolute, protocol-relative, and
  root-relative URLs.
- `--backup-converted` keeps `.orig` backups before link conversion.
- `robots.txt` compliance with `--no-robots` override.

**Verification & security**

- Checksum verification — SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b, MD5.
- SHA256SUMS file parsing (plain and binary-mode).
- PGP detached signature verification (`.asc`, `.sig`, `.gpg`).
- PGP decryption with private key and passphrase.
- HSTS cache (RFC 6797) — automatic HTTP→HTTPS upgrade.
- TLS certificate pinning, mTLS support, custom CA bundle.
- `--ssl-key-log` TLS session key logging (Wireshark-compatible).
- Atomic file writes with mode `0600` for config, resume metadata, and temp
  files.

**Download management**

- Persistent download queue with priority (1–10) and JSON storage.
- Multi-source Metalink downloads (`.meta4`, `.metalink`) with automatic
  failover.
- Batch URL loading from files.
- Cron-based scheduling (`MIN HOUR DOM MON DOW` and `YYYY-MM-DD HH:MM`).
- Post-download command hooks with environment variables.
- WARC (Web ARChive) output for archival compliance.
- Upload support — HTTP PUT and POST with multipart form data.

**Output & progress**

- Unicode progress bar with filled/empty blocks and real-time speed.
- Speed sparkline — mini ASCII chart of recent speed history.
- Estimated time remaining (ETA) calculation.
- Dot progress bar style, JSON progress output for scripting.
- `--write-out` metadata output, `--trace-time` HTTP timing breakdown.
- `--show-headers`, `--keep-timestamps`, `--content-disposition`.
- Color output with `always`, `never`, and `auto` modes (honors `NO_COLOR`).

**Archive & compression**

- Transparent decompression — gzip, deflate, zlib, bzip2, LZW.
- Archive extraction — TAR, TAR.GZ, TAR.BZ2, ZIP with auto-detection.

**Internationalization**

- IDN (International Domain Names) support with Punycode conversion via
  `golang.org/x/net/idna`.

**CLI**

- Long flags only (`--url`, `--output`, `--verbose`).
- Shell completion generation — bash, zsh, and fish.
- Man page generation in troff format.
- Category-organized help text.
- `--info`, `--benchmark`, `--spider`, `--dry-run`, `--glob`.

**Configuration**

- TOML configuration file at `~/.config/goget/config.toml`.
- `GOGET_CONFIG` environment variable for alternative config path.
- CLI flag overrides merged on top of config and defaults.
- Config commands — init, get, set, list, unset.

**Library API (`pkg/api/`)**

- `Client` type with functional options (`WithTimeout`, `WithUserAgent`,
  `WithVerbose`).
- `Client.Download()` and `Client.Upload()` with structured request/result types.
- `QueueClient` for programmatic queue management.
- `ParseMetalink()`, `FetchMetalink()`, `LoadConfig()`, `DefaultConfig()`,
  `ParseSpeed()`, `Version()`.
- One-shot convenience functions: `Download()`, `Upload()`.

**Build & platforms**

- Linux — amd64, arm64, riscv64, loong64.
- FreeBSD — amd64, arm64, riscv64.
- Static binary with zero runtime dependencies (stdlib only +
  `golang.org/x/` packages).
- Race detector enabled in tests, fuzz tests for security-critical parsers.
- Coverage gate at 40%.

**Project & tooling**

- Documentation set: architecture, API, authentication, CLI reference,
  configuration, download pipeline, protocols, recursive & mirror, security.
- `just` task set (`install`, `run`, `build`, `test`, `uninstall`).
- Forgejo Actions CI (go vet, test + race, fuzz, release).
- `CONTRIBUTING.md` with Conventional Commits and release process.

[1.0.0]: https://codeberg.org/petrbalvin/goget/releases/tag/v1.0.0
[1.1.0]: https://codeberg.org/petrbalvin/goget/releases/tag/v1.1.0