301 lines
8.9 KiB
Markdown
301 lines
8.9 KiB
Markdown
# Goget — Modern IPv6-First Command-Line Download Utility
|
||
|
||
**goget** is a modern command-line download utility built on the **Go standard
|
||
library**. It provides IPv6-first connections, parallel chunked downloads,
|
||
resume support, checksum verification, recursive website mirroring, and support
|
||
for 9 network protocols — all in a single static binary with zero runtime
|
||
dependencies.
|
||
|
||
It is designed to be:
|
||
|
||
- **IPv6-first** with automatic IPv4 fallback — always connects via the fastest
|
||
available path.
|
||
- **Safe by default** — TLS 1.2 minimum, certificate verification, SSRF
|
||
protection, and HSTS cache (RFC 6797).
|
||
- **Self-contained** — one static binary, zero runtime dependencies beyond the
|
||
Go standard library and `golang.org/x/` packages.
|
||
- **Scriptable** — JSON progress output, `--write-out` metadata, post-download
|
||
hooks, and a public Go library API.
|
||
|
||
---
|
||
|
||
## Stack
|
||
|
||
| Concern | Choice |
|
||
|-----------------|------------------------------------------|
|
||
| Language | Go (≥ 1.26) |
|
||
| Protocols | HTTP/1.1, HTTP/2, FTP, SFTP, WebDAV, Gemini, Gopher |
|
||
| HTTP transport | Custom dialer with IPv6-first resolution |
|
||
| TLS | Go `crypto/tls` (1.2 minimum) |
|
||
| Markup | `golang.org/x/net/html` for recursive downloads |
|
||
| Crypto | stdlib + `golang.org/x/crypto` for PGP |
|
||
|
||
---
|
||
|
||
## Features
|
||
|
||
- **9 protocols** — HTTP/1.1, HTTP/2, FTP (active/passive, explicit TLS), SFTP
|
||
with `known_hosts` verification, WebDAV (`webdav://`, `webdavs://`) with
|
||
recursive directory download and upload, Gemini, Gopher, `data:`, `file://`.
|
||
- **Parallel chunked downloads** — automatically splits files over 100 MB into
|
||
concurrent chunks using byte-range requests.
|
||
- **Resume interrupted transfers** — metadata-driven resume with `.goget.meta`
|
||
tracking across HTTP, FTP, SFTP, and WebDAV.
|
||
- **Recursive downloading & mirroring** — follow links, filter by pattern or
|
||
MIME type, limit depth, stay within domain. Full site mirrors with link
|
||
conversion for offline viewing and `robots.txt` compliance.
|
||
- **Checksum verification** — SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b,
|
||
MD5; auto-parse SHA256SUMS files.
|
||
- **PGP support** — detached signature verification and OpenPGP decryption.
|
||
- **Metalink** — multi-source downloads with automatic failover (`.meta4`,
|
||
`.metalink`).
|
||
- **Download queue** — persistent queue with priority (1–10).
|
||
- **Scheduling** — absolute datetime or cron expressions.
|
||
- **OAuth 2.0** — client credentials, authorization code, and refresh token
|
||
flows.
|
||
- **Proxy support** — HTTP CONNECT, SOCKS5 (`socks5://`, `socks5h://`).
|
||
- **HSTS cache** — RFC 6797 compliance, automatic HTTP→HTTPS upgrade.
|
||
- **WARC output** — Web ARChive format for legal and archival compliance.
|
||
- **Library API** — `pkg/api/` package for programmatic downloads, uploads, and
|
||
queue management.
|
||
- **Archive extraction** — TAR, TAR.GZ, TAR.BZ2, ZIP with auto-detection.
|
||
- **Shell completion** — bash, zsh, and fish.
|
||
- **IDN support** — international domain names with automatic Punycode
|
||
conversion.
|
||
|
||
---
|
||
|
||
## Quick Start
|
||
|
||
```bash
|
||
git clone https://codeberg.org/petrbalvin/goget && cd goget && just install && just run --version
|
||
```
|
||
|
||
> `just install` downloads Go module dependencies. `just build` builds a static
|
||
> binary to `bin/goget`.
|
||
|
||
```bash
|
||
# Download a file
|
||
goget --url https://example.com/file.zip
|
||
|
||
# Parallel chunked download (auto-enabled for files >100 MB)
|
||
goget --url https://example.com/large.iso --parallel 4
|
||
|
||
# Resume interrupted download
|
||
goget --url https://example.com/file.zip --resume
|
||
|
||
# Multiple URLs
|
||
goget --url https://a.com/1.zip --url https://b.com/2.zip
|
||
|
||
# Batch download from file
|
||
goget --batch-file urls.txt
|
||
|
||
# Custom output directory
|
||
goget --url https://example.com/file.zip --output ./downloads/file.zip
|
||
```
|
||
|
||
---
|
||
|
||
## Usage
|
||
|
||
### Common Flags
|
||
|
||
```bash
|
||
# Custom HTTP headers
|
||
goget --header "Authorization: Bearer token" --url https://api.example.com
|
||
|
||
# Cookies
|
||
goget --cookie "session=abc123" --url https://example.com
|
||
|
||
# POST data
|
||
goget --data "key=value" --url https://httpbin.org/post
|
||
|
||
# Skip TLS verification (⚠ testing only)
|
||
goget --insecure --url https://self-signed.example.com
|
||
|
||
# Client certificate (mTLS)
|
||
goget --cert client.pem --key client.key --url https://mtls.example.com
|
||
|
||
# HTTP trace timing
|
||
goget --trace-time --url https://example.com
|
||
# → DNS: 12ms | TCP: 34ms | TLS: 56ms | TTFB: 78ms | Total: 180ms
|
||
|
||
# Timeouts
|
||
goget --max-time 30s --url https://slow.example.com
|
||
goget --connect-timeout 5s --url https://example.com
|
||
|
||
# Rate limiting
|
||
goget --rate-limit 1MB/s --url https://example.com/file.zip
|
||
|
||
# Proxy
|
||
goget --proxy http://proxy.example.com:8080 --url https://example.com/file.zip
|
||
goget --proxy socks5h://127.0.0.1:1080 --url https://example.com/file.zip
|
||
```
|
||
|
||
### Recursive & Mirroring
|
||
|
||
```bash
|
||
# Recursive download with depth limit
|
||
goget --url https://example.com/docs/ --recursive --max-depth 3
|
||
|
||
# Pattern filtering
|
||
goget --recursive --accept "*.pdf,*.html" --url https://example.com
|
||
|
||
# Dry-run — list without downloading
|
||
goget --url https://example.com/docs/ --recursive --dry-run
|
||
|
||
# Mirror entire website with link conversion
|
||
goget --url https://example.com --mirror --convert-links --output ./mirror
|
||
|
||
# WARC archiving
|
||
goget --warc-file archive.warc --url https://example.com/page.html
|
||
```
|
||
|
||
### Verification
|
||
|
||
```bash
|
||
# Checksum
|
||
goget --checksum abc123... --url https://example.com/file.zip
|
||
goget --checksum-file SHA256SUMS --url https://example.com/file.zip
|
||
|
||
# PGP
|
||
goget --pgp-verify --pgp-sig file.sig --url https://example.com/file
|
||
|
||
# Metalink multi-source
|
||
goget --metalink --url https://example.com/release.meta4
|
||
```
|
||
|
||
### Advanced
|
||
|
||
```bash
|
||
# Schedule a download
|
||
goget --schedule "2026-06-01 02:00" --url https://example.com/daily.zip
|
||
goget --schedule "0 2 * * *" --url https://example.com/daily.zip
|
||
|
||
# Post-download hook (env: GOGET_OUTPUT, GOGET_SIZE, GOGET_URL)
|
||
goget --on-complete "process.sh" --url https://example.com/data.csv
|
||
|
||
# JSON progress for scripting
|
||
goget --json --no-progress --url https://example.com/file.zip
|
||
|
||
# Shell completion
|
||
goget --completion bash > /etc/bash_completion.d/goget
|
||
goget --completion zsh > /usr/share/zsh/site-functions/_goget
|
||
goget --completion fish > ~/.config/fish/completions/goget.fish
|
||
|
||
# Configuration management
|
||
goget --init-config
|
||
goget --config-get timeout
|
||
goget --config-set timeout 5m
|
||
goget --config-list
|
||
```
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
Default location: `~/.config/goget/config.toml`. See
|
||
[`docs/configuration.md`](docs/configuration.md) for the full schema.
|
||
|
||
```toml
|
||
timeout = "30m0s"
|
||
parallel = 0
|
||
user_agent = "Goget/1.0.0"
|
||
max_speed = 0
|
||
auto_resume = true
|
||
auto_decompress = true
|
||
insecure_skip_verify = false
|
||
checksum_algo = "sha256"
|
||
color = "auto"
|
||
progress_style = "unicode"
|
||
max_retries = 0
|
||
proxy = ""
|
||
|
||
[recursive]
|
||
enabled = false
|
||
max_depth = 3
|
||
follow_external = false
|
||
delay = "100ms"
|
||
```
|
||
|
||
---
|
||
|
||
## Library Usage
|
||
|
||
```go
|
||
package main
|
||
|
||
import (
|
||
"fmt"
|
||
"log"
|
||
"time"
|
||
|
||
"codeberg.org/petrbalvin/goget/pkg/api"
|
||
)
|
||
|
||
func main() {
|
||
client := api.NewClient(
|
||
api.WithTimeout(5 * time.Minute),
|
||
api.WithUserAgent("MyApp/1.0"),
|
||
)
|
||
defer client.Close()
|
||
|
||
result, err := client.Download(&api.DownloadRequest{
|
||
URL: "https://example.com/file.zip",
|
||
Output: "output.zip",
|
||
Resume: true,
|
||
})
|
||
if err != nil {
|
||
log.Fatalf("download failed: %v", err)
|
||
}
|
||
fmt.Printf("Downloaded %d bytes in %v (%.1f MB/s, IPv%d)\n",
|
||
result.BytesDownloaded, result.Duration,
|
||
result.Speed/1024/1024, result.IPVersion)
|
||
}
|
||
```
|
||
|
||
Full API reference: [`docs/api.md`](docs/api.md).
|
||
|
||
---
|
||
|
||
## Development
|
||
|
||
### Requirements
|
||
|
||
- Go 1.26+
|
||
- Linux (amd64, arm64, riscv64, loong64) or FreeBSD (amd64, arm64, riscv64)
|
||
|
||
```bash
|
||
just # list recipes
|
||
just install # go mod download
|
||
just run # run from source (development)
|
||
just build # go vet + gofmt check + build → bin/goget
|
||
just test # go test -race -count=1 ./...
|
||
just uninstall # remove build artifacts
|
||
```
|
||
|
||
---
|
||
|
||
## Documentation
|
||
|
||
- [`docs/architecture.md`](docs/architecture.md) — components and request flow
|
||
- [`docs/configuration.md`](docs/configuration.md) — config schema (TOML)
|
||
- [`docs/api.md`](docs/api.md) — public Go library API
|
||
- [`docs/cli-reference.md`](docs/cli-reference.md) — full CLI flag reference
|
||
- [`docs/protocols.md`](docs/protocols.md) — protocol handler details
|
||
- [`docs/recursive-mirror.md`](docs/recursive-mirror.md) — recursive downloading
|
||
and mirroring
|
||
- [`docs/download-pipeline.md`](docs/download-pipeline.md) — download flow
|
||
- [`docs/authentication.md`](docs/authentication.md) — auth mechanisms
|
||
- [`docs/security.md`](docs/security.md) — threat model and controls
|
||
|
||
## Contributing
|
||
|
||
See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, code style, commit
|
||
conventions, the pull request flow, and how releases are cut.
|
||
|
||
## License
|
||
|
||
MIT — see [LICENSE](LICENSE).
|
||
Copyright © 2026 [Petr Balvín](https://petrbalvin.org)
|