# Goget — Modern IPv6-First Command-Line Download Utility **goget** is a modern command-line download utility built on the **Go standard library**. It provides IPv6-first connections, parallel chunked downloads, resume support, checksum verification, recursive website mirroring, and support for 9 network protocols — all in a single static binary with zero runtime dependencies. It is designed to be: - **IPv6-first** with automatic IPv4 fallback — always connects via the fastest available path. - **Safe by default** — TLS 1.2 minimum, certificate verification, SSRF protection, and HSTS cache (RFC 6797). - **Self-contained** — one static binary, zero runtime dependencies beyond the Go standard library and `golang.org/x/` packages. - **Scriptable** — JSON progress output, `--write-out` metadata, post-download hooks, and a public Go library API. --- ## Stack | Concern | Choice | |-----------------|------------------------------------------| | Language | Go (≥ 1.26) | | Protocols | HTTP/1.1, HTTP/2, FTP, SFTP, WebDAV, Gemini, Gopher | | HTTP transport | Custom dialer with IPv6-first resolution | | TLS | Go `crypto/tls` (1.2 minimum) | | Markup | `golang.org/x/net/html` for recursive downloads | | Crypto | stdlib + `golang.org/x/crypto` for PGP | --- ## Features - **9 protocols** — HTTP/1.1, HTTP/2, FTP (active/passive, explicit TLS), SFTP with `known_hosts` verification, WebDAV (`webdav://`, `webdavs://`) with recursive directory download and upload, Gemini, Gopher, `data:`, `file://`. - **Parallel chunked downloads** — automatically splits files over 100 MB into concurrent chunks using byte-range requests. - **Resume interrupted transfers** — metadata-driven resume with `.goget.meta` tracking across HTTP, FTP, SFTP, WebDAV, and Gemini. - **Recursive downloading & mirroring** — follow links, filter by pattern or MIME type, limit depth, stay within domain. Full site mirrors with link conversion for offline viewing and `robots.txt` compliance. - **Checksum verification** — SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b, MD5; auto-parse SHA256SUMS files. - **PGP support** — detached signature verification and OpenPGP decryption. - **Metalink** — multi-source downloads with automatic failover (`.meta4`, `.metalink`). - **Download queue** — persistent queue with priority (1–10). - **Scheduling** — absolute datetime or cron expressions. - **OAuth 2.0** — client credentials, authorization code, and refresh token flows. - **Proxy support** — HTTP CONNECT, SOCKS5 (`socks5://`, `socks5h://`). - **HSTS cache** — RFC 6797 compliance, automatic HTTP→HTTPS upgrade. - **WARC output** — Web ARChive format for legal and archival compliance. - **Library API** — `pkg/api/` package for programmatic downloads, uploads, and queue management. - **Archive extraction** — TAR, TAR.GZ, TAR.BZ2, ZIP with auto-detection. - **Shell completion** — bash, zsh, and fish. - **IDN support** — international domain names with automatic Punycode conversion. --- ## Quick Start ```bash git clone https://codeberg.org/petrbalvin/goget && cd goget && just install && just run --version ``` > `just install` downloads Go module dependencies. `just build` builds a static > binary to `bin/goget`. ```bash # Download a file goget --url https://example.com/file.zip # Parallel chunked download (auto-enabled for files >100 MB) goget --url https://example.com/large.iso --parallel 4 # Resume interrupted download goget --url https://example.com/file.zip --resume # Multiple URLs goget --url https://a.com/1.zip --url https://b.com/2.zip # Batch download from file goget --batch-file urls.txt # Custom output directory goget --url https://example.com/file.zip --output ./downloads/file.zip ``` --- ## Usage ### Common Flags ```bash # Custom HTTP headers goget --header "Authorization: Bearer token" --url https://api.example.com # Cookies goget --cookie "session=abc123" --url https://example.com # POST data goget --data "key=value" --url https://httpbin.org/post # Skip TLS verification (⚠ testing only) goget --insecure --url https://self-signed.example.com # Client certificate (mTLS) goget --cert client.pem --key client.key --url https://mtls.example.com # HTTP trace timing goget --trace-time --url https://example.com # → DNS: 12ms | TCP: 34ms | TLS: 56ms | TTFB: 78ms | Total: 180ms # Timeouts goget --max-time 30s --url https://slow.example.com goget --connect-timeout 5s --url https://example.com # Rate limiting goget --rate-limit 1MB/s --url https://example.com/file.zip # Proxy goget --proxy http://proxy.example.com:8080 --url https://example.com/file.zip goget --proxy socks5h://127.0.0.1:1080 --url https://example.com/file.zip # Pin server certificate (applies to HTTPS, FTPS, Gemini, WebDAVS) goget --pinned-cert a1b2c3d4e5f6... --url https://example.com/file goget --pinned-cert a1b2c3d4e5f6... --url gemini://gemini.example.com/ # Pin SSH host key for SFTP (OpenSSH fingerprint from `ssh-keygen -lf`) goget --pinned-host-key "SHA256:AAAAC3NzaC1lZDI1NTE5AAAAIE..." \ --url sftp://example.com/file ``` ### Recursive & Mirroring ```bash # Recursive download with depth limit goget --url https://example.com/docs/ --recursive --max-depth 3 # Pattern filtering goget --recursive --accept "*.pdf,*.html" --url https://example.com # Dry-run — list without downloading goget --url https://example.com/docs/ --recursive --dry-run # Mirror entire website with link conversion goget --url https://example.com --mirror --convert-links --output ./mirror # WARC archiving goget --warc-file archive.warc --url https://example.com/page.html ``` ### Verification ```bash # Checksum goget --checksum abc123... --url https://example.com/file.zip goget --checksum-file SHA256SUMS --url https://example.com/file.zip # PGP goget --pgp-verify --pgp-sig file.sig --url https://example.com/file # Metalink multi-source goget --metalink --url https://example.com/release.meta4 ``` ### Advanced ```bash # Schedule a download goget --schedule "2026-06-01 02:00" --url https://example.com/daily.zip goget --schedule "0 2 * * *" --url https://example.com/daily.zip # Post-download hook (env: GOGET_OUTPUT, GOGET_SIZE, GOGET_URL) goget --on-complete "process.sh" --url https://example.com/data.csv # JSON progress for scripting goget --json --no-progress --url https://example.com/file.zip # Inspect a partially-downloaded transfer's resume metadata goget --show-metadata ./file.zip goget --show-metadata ./file.zip.goget.meta --json # Shell completion goget --completion bash > /etc/bash_completion.d/goget goget --completion zsh > /usr/share/zsh/site-functions/_goget goget --completion fish > ~/.config/fish/completions/goget.fish # Configuration management goget --init-config goget --config-get timeout goget --config-set timeout 5m goget --config-list ``` --- ## Configuration Default location: `~/.config/goget/config.toml`. See [`docs/configuration.md`](docs/configuration.md) for the full schema. ```toml timeout = "30m0s" parallel = 0 user_agent = "Goget/1.0.0" max_speed = 0 auto_resume = true auto_decompress = true insecure_skip_verify = false checksum_algo = "sha256" color = "auto" progress_style = "unicode" max_retries = 0 proxy = "" [recursive] enabled = false max_depth = 3 follow_external = false delay = "100ms" ``` --- ## Library Usage ```go package main import ( "fmt" "log" "time" "codeberg.org/petrbalvin/goget/pkg/api" ) func main() { client := api.NewClient( api.WithTimeout(5 * time.Minute), api.WithUserAgent("MyApp/1.0"), ) defer client.Close() result, err := client.Download(&api.DownloadRequest{ URL: "https://example.com/file.zip", Output: "output.zip", Resume: true, }) if err != nil { log.Fatalf("download failed: %v", err) } fmt.Printf("Downloaded %d bytes in %v (%.1f MB/s, IPv%d)\n", result.BytesDownloaded, result.Duration, result.Speed/1024/1024, result.IPVersion) } ``` Full API reference: [`docs/api.md`](docs/api.md). --- ## Development ### Requirements - Go 1.26+ - Linux (amd64, arm64, riscv64, loong64) or FreeBSD (amd64, arm64, riscv64) ```bash just # list recipes just install # go mod download just run # run from source (development) just build # go vet + gofmt check + build → bin/goget just test # go test -race -count=1 ./... just man # validate man/goget.1 with groff just install-man # install manpage to ~/.local/share/man/man1/ just uninstall # remove build artifacts just uninstall-man# remove installed manpage ``` --- ## Documentation - [`man/goget.1`](man/goget.1) — troff manual page (`man goget` after `just install-man`) - [`docs/architecture.md`](docs/architecture.md) — components and request flow - [`docs/configuration.md`](docs/configuration.md) — config schema (TOML) - [`docs/api.md`](docs/api.md) — public Go library API - [`docs/cli-reference.md`](docs/cli-reference.md) — full CLI flag reference - [`docs/protocols.md`](docs/protocols.md) — protocol handler details - [`docs/recursive-mirror.md`](docs/recursive-mirror.md) — recursive downloading and mirroring - [`docs/download-pipeline.md`](docs/download-pipeline.md) — download flow - [`docs/authentication.md`](docs/authentication.md) — auth mechanisms - [`docs/security.md`](docs/security.md) — threat model and controls - [`docs/migration.md`](docs/migration.md) — curl/wget → goget flag mapping - [`docs/development.md`](docs/development.md) — dev workflow, releases, CI ## Contributing See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, code style, commit conventions, the pull request flow, and how releases are cut. ## License MIT — see [LICENSE](LICENSE). Copyright © 2026 [Petr Balvín](https://petrbalvin.org)