Goget — Modern IPv6-First Command-Line Download Utility

goget is a modern command-line download utility built on the Go standard library. It provides IPv6-first connections, parallel chunked downloads, resume support, checksum verification, recursive website mirroring, and support for 9 network protocols — all in a single static binary with zero runtime dependencies.

It is designed to be:

IPv6-first with automatic IPv4 fallback — always connects via the fastest available path.
Safe by default — TLS 1.2 minimum, certificate verification, SSRF protection, and HSTS cache (RFC 6797).
Self-contained — one static binary, zero runtime dependencies beyond the Go standard library and golang.org/x/ packages.
Scriptable — JSON progress output, --write-out metadata, post-download hooks, and a public Go library API.

Stack

Concern	Choice
Language	Go (≥ 1.26)
Protocols	HTTP/1.1, HTTP/2, FTP, SFTP, WebDAV, Gemini, Gopher
HTTP transport	Custom dialer with IPv6-first resolution
TLS	Go `crypto/tls` (1.2 minimum)
Markup	`golang.org/x/net/html` for recursive downloads
Crypto	stdlib + `golang.org/x/crypto` for PGP

Features

9 protocols — HTTP/1.1, HTTP/2, FTP (active/passive, explicit TLS), SFTP with known_hosts verification, WebDAV (webdav://, webdavs://) with recursive directory download and upload, Gemini, Gopher, data:, file://.
Parallel chunked downloads — automatically splits files over 100 MB into concurrent chunks using byte-range requests.
Resume interrupted transfers — metadata-driven resume with .goget.meta tracking across HTTP, FTP, SFTP, WebDAV, and Gemini.
Recursive downloading & mirroring — follow links, filter by pattern or MIME type, limit depth, stay within domain. Full site mirrors with link conversion for offline viewing and robots.txt compliance.
Checksum verification — SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b, MD5; auto-parse SHA256SUMS files.
PGP support — detached signature verification and OpenPGP decryption.
Metalink — multi-source downloads with automatic failover (.meta4, .metalink).
Download queue — persistent queue with priority (1–10).
Scheduling — absolute datetime or cron expressions.
OAuth 2.0 — client credentials, authorization code, and refresh token flows.
Proxy support — HTTP CONNECT, SOCKS5 (socks5://, socks5h://).
HSTS cache — RFC 6797 compliance, automatic HTTP→HTTPS upgrade.
WARC output — Web ARChive format for legal and archival compliance.
Library API — pkg/api/ package for programmatic downloads, uploads, and queue management.
Archive extraction — TAR, TAR.GZ, TAR.BZ2, ZIP with auto-detection.
Shell completion — bash, zsh, and fish.
IDN support — international domain names with automatic Punycode conversion.

Quick Start

git clone https://codeberg.org/petrbalvin/goget && cd goget && just install && just run --version

just install downloads Go module dependencies. just build builds a static binary to bin/goget.

# Download a file
goget --url https://example.com/file.zip

# Parallel chunked download (auto-enabled for files >100 MB)
goget --url https://example.com/large.iso --parallel 4

# Resume interrupted download
goget --url https://example.com/file.zip --resume

# Multiple URLs
goget --url https://a.com/1.zip --url https://b.com/2.zip

# Batch download from file
goget --batch-file urls.txt

# Custom output directory
goget --url https://example.com/file.zip --output ./downloads/file.zip

Usage

Common Flags

# Custom HTTP headers
goget --header "Authorization: Bearer token" --url https://api.example.com

# Cookies
goget --cookie "session=abc123" --url https://example.com

# POST data
goget --data "key=value" --url https://httpbin.org/post

# Skip TLS verification (⚠ testing only)
goget --insecure --url https://self-signed.example.com

# Client certificate (mTLS)
goget --cert client.pem --key client.key --url https://mtls.example.com

# HTTP trace timing
goget --trace-time --url https://example.com
# → DNS: 12ms | TCP: 34ms | TLS: 56ms | TTFB: 78ms | Total: 180ms

# Timeouts
goget --max-time 30s --url https://slow.example.com
goget --connect-timeout 5s --url https://example.com

# Rate limiting
goget --rate-limit 1MB/s --url https://example.com/file.zip

# Proxy
goget --proxy http://proxy.example.com:8080 --url https://example.com/file.zip
goget --proxy socks5h://127.0.0.1:1080 --url https://example.com/file.zip

# Pin server certificate (applies to HTTPS, FTPS, Gemini, WebDAVS)
goget --pinned-cert a1b2c3d4e5f6... --url https://example.com/file
goget --pinned-cert a1b2c3d4e5f6... --url gemini://gemini.example.com/

# Pin SSH host key for SFTP (OpenSSH fingerprint from `ssh-keygen -lf`)
goget --pinned-host-key "SHA256:AAAAC3NzaC1lZDI1NTE5AAAAIE..." \
      --url sftp://example.com/file

Recursive & Mirroring

# Recursive download with depth limit
goget --url https://example.com/docs/ --recursive --max-depth 3

# Pattern filtering
goget --recursive --accept "*.pdf,*.html" --url https://example.com

# Dry-run — list without downloading
goget --url https://example.com/docs/ --recursive --dry-run

# Mirror entire website with link conversion
goget --url https://example.com --mirror --convert-links --output ./mirror

# WARC archiving
goget --warc-file archive.warc --url https://example.com/page.html

Verification

# Checksum
goget --checksum abc123... --url https://example.com/file.zip
goget --checksum-file SHA256SUMS --url https://example.com/file.zip

# PGP
goget --pgp-verify --pgp-sig file.sig --url https://example.com/file

# Metalink multi-source
goget --metalink --url https://example.com/release.meta4

Advanced

# Schedule a download
goget --schedule "2026-06-01 02:00" --url https://example.com/daily.zip
goget --schedule "0 2 * * *" --url https://example.com/daily.zip

# Post-download hook (env: GOGET_OUTPUT, GOGET_SIZE, GOGET_URL)
goget --on-complete "process.sh" --url https://example.com/data.csv

# JSON progress for scripting
goget --json --no-progress --url https://example.com/file.zip

# Inspect a partially-downloaded transfer's resume metadata
goget --show-metadata ./file.zip
goget --show-metadata ./file.zip.goget.meta --json

# Shell completion
goget --completion bash > /etc/bash_completion.d/goget
goget --completion zsh > /usr/share/zsh/site-functions/_goget
goget --completion fish > ~/.config/fish/completions/goget.fish

# Configuration management
goget --init-config
goget --config-get timeout
goget --config-set timeout 5m
goget --config-list

Configuration

Default location: ~/.config/goget/config.toml. See docs/configuration.md for the full schema.

timeout = "30m0s"
parallel = 0
user_agent = "Goget/1.0.0"
max_speed = 0
auto_resume = true
auto_decompress = true
insecure_skip_verify = false
checksum_algo = "sha256"
color = "auto"
progress_style = "unicode"
max_retries = 0
proxy = ""

[recursive]
enabled = false
max_depth = 3
follow_external = false
delay = "100ms"

Library Usage

package main

import (
    "fmt"
    "log"
    "time"

    "codeberg.org/petrbalvin/goget/pkg/api"
)

func main() {
    client := api.NewClient(
        api.WithTimeout(5 * time.Minute),
        api.WithUserAgent("MyApp/1.0"),
    )
    defer client.Close()

    result, err := client.Download(&api.DownloadRequest{
        URL:    "https://example.com/file.zip",
        Output: "output.zip",
        Resume: true,
    })
    if err != nil {
        log.Fatalf("download failed: %v", err)
    }
    fmt.Printf("Downloaded %d bytes in %v (%.1f MB/s, IPv%d)\n",
        result.BytesDownloaded, result.Duration,
        result.Speed/1024/1024, result.IPVersion)
}

Full API reference: docs/api.md.

Development

Requirements

Go 1.26+
Linux (amd64, arm64, riscv64, loong64) or FreeBSD (amd64, arm64, riscv64)

just              # list recipes
just install      # go mod download
just run          # run from source (development)
just build        # go vet + gofmt check + build → bin/goget
just test         # go test -race -count=1 ./...
just man          # validate man/goget.1 with groff
just install-man  # install manpage to ~/.local/share/man/man1/
just uninstall    # remove build artifacts
just uninstall-man# remove installed manpage

Documentation

man/goget.1 — troff manual page (man goget after just install-man)
docs/architecture.md — components and request flow
docs/configuration.md — config schema (TOML)
docs/api.md — public Go library API
docs/cli-reference.md — full CLI flag reference
docs/protocols.md — protocol handler details
docs/recursive-mirror.md — recursive downloading and mirroring
docs/download-pipeline.md — download flow
docs/authentication.md — auth mechanisms
docs/security.md — threat model and controls
docs/migration.md — curl/wget → goget flag mapping
docs/development.md — dev workflow, releases, CI

Contributing

See CONTRIBUTING.md for development setup, code style, commit conventions, the pull request flow, and how releases are cut.

9.8 KiB Raw Permalink Blame History Unescape Escape