feat: initial goget release — modern IPv6-first download utility

2026-06-26 21:21:58 +02:00
commit 53db81e1f7
147 changed files with 45931 additions and 0 deletions
@@ -0,0 +1,300 @@
+# Goget — Modern IPv6-First Command-Line Download Utility
+
+**goget** is a modern command-line download utility built on the **Go standard
+library**. It provides IPv6-first connections, parallel chunked downloads,
+resume support, checksum verification, recursive website mirroring, and support
+for 9 network protocols — all in a single static binary with zero runtime
+dependencies.
+
+It is designed to be:
+
+- **IPv6-first** with automatic IPv4 fallback — always connects via the fastest
+  available path.
+- **Safe by default** — TLS 1.2 minimum, certificate verification, SSRF
+  protection, and HSTS cache (RFC 6797).
+- **Self-contained** — one static binary, zero runtime dependencies beyond the
+  Go standard library and `golang.org/x/` packages.
+- **Scriptable** — JSON progress output, `--write-out` metadata, post-download
+  hooks, and a public Go library API.
+
+---
+
+## Stack
+
+| Concern         | Choice                                   |
+|-----------------|------------------------------------------|
+| Language        | Go (≥ 1.26)                              |
+| Protocols       | HTTP/1.1, HTTP/2, FTP, SFTP, WebDAV, Gemini, Gopher |
+| HTTP transport  | Custom dialer with IPv6-first resolution |
+| TLS             | Go `crypto/tls` (1.2 minimum)            |
+| Markup          | `golang.org/x/net/html` for recursive downloads |
+| Crypto          | stdlib + `golang.org/x/crypto` for PGP   |
+
+---
+
+## Features
+
+- **9 protocols** — HTTP/1.1, HTTP/2, FTP (active/passive, explicit TLS), SFTP
+  with `known_hosts` verification, WebDAV (`webdav://`, `webdavs://`) with
+  recursive directory download and upload, Gemini, Gopher, `data:`, `file://`.
+- **Parallel chunked downloads** — automatically splits files over 100 MB into
+  concurrent chunks using byte-range requests.
+- **Resume interrupted transfers** — metadata-driven resume with `.goget.meta`
+  tracking across HTTP, FTP, SFTP, and WebDAV.
+- **Recursive downloading & mirroring** — follow links, filter by pattern or
+  MIME type, limit depth, stay within domain. Full site mirrors with link
+  conversion for offline viewing and `robots.txt` compliance.
+- **Checksum verification** — SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b,
+  MD5; auto-parse SHA256SUMS files.
+- **PGP support** — detached signature verification and OpenPGP decryption.
+- **Metalink** — multi-source downloads with automatic failover (`.meta4`,
+  `.metalink`).
+- **Download queue** — persistent queue with priority (1–10).
+- **Scheduling** — absolute datetime or cron expressions.
+- **OAuth 2.0** — client credentials, authorization code, and refresh token
+  flows.
+- **Proxy support** — HTTP CONNECT, SOCKS5 (`socks5://`, `socks5h://`).
+- **HSTS cache** — RFC 6797 compliance, automatic HTTP→HTTPS upgrade.
+- **WARC output** — Web ARChive format for legal and archival compliance.
+- **Library API** — `pkg/api/` package for programmatic downloads, uploads, and
+  queue management.
+- **Archive extraction** — TAR, TAR.GZ, TAR.BZ2, ZIP with auto-detection.
+- **Shell completion** — bash, zsh, and fish.
+- **IDN support** — international domain names with automatic Punycode
+  conversion.
+
+---
+
+## Quick Start
+
+```bash
+git clone https://codeberg.org/petrbalvin/goget && cd goget && just install && just run --version
+```
+
+> `just install` downloads Go module dependencies. `just build` builds a static
+> binary to `bin/goget`.
+
+```bash
+# Download a file
+goget --url https://example.com/file.zip
+
+# Parallel chunked download (auto-enabled for files >100 MB)
+goget --url https://example.com/large.iso --parallel 4
+
+# Resume interrupted download
+goget --url https://example.com/file.zip --resume
+
+# Multiple URLs
+goget --url https://a.com/1.zip --url https://b.com/2.zip
+
+# Batch download from file
+goget --batch-file urls.txt
+
+# Custom output directory
+goget --url https://example.com/file.zip --output ./downloads/file.zip
+```
+
+---
+
+## Usage
+
+### Common Flags
+
+```bash
+# Custom HTTP headers
+goget --header "Authorization: Bearer token" --url https://api.example.com
+
+# Cookies
+goget --cookie "session=abc123" --url https://example.com
+
+# POST data
+goget --data "key=value" --url https://httpbin.org/post
+
+# Skip TLS verification (⚠ testing only)
+goget --insecure --url https://self-signed.example.com
+
+# Client certificate (mTLS)
+goget --cert client.pem --key client.key --url https://mtls.example.com
+
+# HTTP trace timing
+goget --trace-time --url https://example.com
+# → DNS: 12ms | TCP: 34ms | TLS: 56ms | TTFB: 78ms | Total: 180ms
+
+# Timeouts
+goget --max-time 30s --url https://slow.example.com
+goget --connect-timeout 5s --url https://example.com
+
+# Rate limiting
+goget --rate-limit 1MB/s --url https://example.com/file.zip
+
+# Proxy
+goget --proxy http://proxy.example.com:8080 --url https://example.com/file.zip
+goget --proxy socks5h://127.0.0.1:1080 --url https://example.com/file.zip
+```
+
+### Recursive & Mirroring
+
+```bash
+# Recursive download with depth limit
+goget --url https://example.com/docs/ --recursive --max-depth 3
+
+# Pattern filtering
+goget --recursive --accept "*.pdf,*.html" --url https://example.com
+
+# Dry-run — list without downloading
+goget --url https://example.com/docs/ --recursive --dry-run
+
+# Mirror entire website with link conversion
+goget --url https://example.com --mirror --convert-links --output ./mirror
+
+# WARC archiving
+goget --warc-file archive.warc --url https://example.com/page.html
+```
+
+### Verification
+
+```bash
+# Checksum
+goget --checksum abc123... --url https://example.com/file.zip
+goget --checksum-file SHA256SUMS --url https://example.com/file.zip
+
+# PGP
+goget --pgp-verify --pgp-sig file.sig --url https://example.com/file
+
+# Metalink multi-source
+goget --metalink --url https://example.com/release.meta4
+```
+
+### Advanced
+
+```bash
+# Schedule a download
+goget --schedule "2026-06-01 02:00" --url https://example.com/daily.zip
+goget --schedule "0 2 * * *" --url https://example.com/daily.zip
+
+# Post-download hook (env: GOGET_OUTPUT, GOGET_SIZE, GOGET_URL)
+goget --on-complete "process.sh" --url https://example.com/data.csv
+
+# JSON progress for scripting
+goget --json --no-progress --url https://example.com/file.zip
+
+# Shell completion
+goget --completion bash > /etc/bash_completion.d/goget
+goget --completion zsh > /usr/share/zsh/site-functions/_goget
+goget --completion fish > ~/.config/fish/completions/goget.fish
+
+# Configuration management
+goget --init-config
+goget --config-get timeout
+goget --config-set timeout 5m
+goget --config-list
+```
+
+---
+
+## Configuration
+
+Default location: `~/.config/goget/config.toml`. See
+[`docs/configuration.md`](docs/configuration.md) for the full schema.
+
+```toml
+timeout = "30m0s"
+parallel = 0
+user_agent = "Goget/1.0.0"
+max_speed = 0
+auto_resume = true
+auto_decompress = true
+insecure_skip_verify = false
+checksum_algo = "sha256"
+color = "auto"
+progress_style = "unicode"
+max_retries = 0
+proxy = ""
+
+[recursive]
+enabled = false
+max_depth = 3
+follow_external = false
+delay = "100ms"
+```
+
+---
+
+## Library Usage
+
+```go
+package main
+
+import (
+    "fmt"
+    "log"
+    "time"
+
+    "codeberg.org/petrbalvin/goget/pkg/api"
+)
+
+func main() {
+    client := api.NewClient(
+        api.WithTimeout(5 * time.Minute),
+        api.WithUserAgent("MyApp/1.0"),
+    )
+    defer client.Close()
+
+    result, err := client.Download(&api.DownloadRequest{
+        URL:    "https://example.com/file.zip",
+        Output: "output.zip",
+        Resume: true,
+    })
+    if err != nil {
+        log.Fatalf("download failed: %v", err)
+    }
+    fmt.Printf("Downloaded %d bytes in %v (%.1f MB/s, IPv%d)\n",
+        result.BytesDownloaded, result.Duration,
+        result.Speed/1024/1024, result.IPVersion)
+}
+```
+
+Full API reference: [`docs/api.md`](docs/api.md).
+
+---
+
+## Development
+
+### Requirements
+
+- Go 1.26+
+- Linux (amd64, arm64, riscv64, loong64) or FreeBSD (amd64, arm64, riscv64)
+
+```bash
+just              # list recipes
+just install      # go mod download
+just run          # run from source (development)
+just build        # go vet + gofmt check + build → bin/goget
+just test         # go test -race -count=1 ./...
+just uninstall    # remove build artifacts
+```
+
+---
+
+## Documentation
+
+- [`docs/architecture.md`](docs/architecture.md) — components and request flow
+- [`docs/configuration.md`](docs/configuration.md) — config schema (TOML)
+- [`docs/api.md`](docs/api.md) — public Go library API
+- [`docs/cli-reference.md`](docs/cli-reference.md) — full CLI flag reference
+- [`docs/protocols.md`](docs/protocols.md) — protocol handler details
+- [`docs/recursive-mirror.md`](docs/recursive-mirror.md) — recursive downloading
+  and mirroring
+- [`docs/download-pipeline.md`](docs/download-pipeline.md) — download flow
+- [`docs/authentication.md`](docs/authentication.md) — auth mechanisms
+- [`docs/security.md`](docs/security.md) — threat model and controls
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, code style, commit
+conventions, the pull request flow, and how releases are cut.
+
+## License
+
+MIT — see [LICENSE](LICENSE).  
+Copyright © 2026 [Petr Balvín](https://petrbalvin.org)