feat: initial goget release — modern IPv6-first download utility
This commit is contained in:
+454
@@ -0,0 +1,454 @@
|
||||
# API Reference
|
||||
|
||||
**goget** exposes a public Go library in `pkg/api/` for programmatic downloads, uploads, queue management, and configuration.
|
||||
|
||||
## Installation
|
||||
|
||||
```go
|
||||
import "codeberg.org/petrbalvin/goget/pkg/api"
|
||||
```
|
||||
|
||||
## Types
|
||||
|
||||
### `Client`
|
||||
|
||||
The main client for downloads and uploads. Created via `NewClient()`.
|
||||
|
||||
```go
|
||||
type Client struct { /* unexported */ }
|
||||
|
||||
func NewClient(opts ...ClientOption) *Client
|
||||
func (c *Client) Download(req *DownloadRequest) (*DownloadResult, error)
|
||||
func (c *Client) Upload(req *UploadRequest) (*UploadResult, error)
|
||||
func (c *Client) Close()
|
||||
```
|
||||
|
||||
### `ClientOption`
|
||||
|
||||
Functional options for client configuration.
|
||||
|
||||
```go
|
||||
func WithTimeout(d time.Duration) ClientOption
|
||||
func WithUserAgent(ua string) ClientOption
|
||||
func WithVerbose(v bool) ClientOption
|
||||
```
|
||||
|
||||
### `DownloadRequest`
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `URL` | `string` | URL to download (required) |
|
||||
| `Output` | `string` | Output file path (empty = auto-detect) |
|
||||
| `Resume` | `bool` | Resume interrupted download |
|
||||
| `Verbose` | `bool` | Enable verbose logging |
|
||||
|
||||
### `DownloadResult`
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `BytesDownloaded` | `int64` | Total bytes transferred |
|
||||
| `TotalSize` | `int64` | Total file size (-1 if unknown) |
|
||||
| `Speed` | `float64` | Average speed in bytes/second |
|
||||
| `Duration` | `time.Duration` | Total transfer time |
|
||||
| `OutputPath` | `string` | Path to saved file |
|
||||
| `Protocol` | `string` | Protocol used (e.g., "http") |
|
||||
| `IPVersion` | `int` | 4 or 6 |
|
||||
| `Resumed` | `bool` | True if download was resumed |
|
||||
| `Parallel` | `bool` | True if parallel chunks were used |
|
||||
| `ChunksCount` | `int` | Number of parallel chunks |
|
||||
|
||||
### `UploadRequest`
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `URL` | `string` | Destination URL (required) |
|
||||
| `FilePath` | `string` | Path to file to upload (required) |
|
||||
| `Method` | `string` | HTTP method: `"PUT"` or `"POST"` |
|
||||
| `Verbose` | `bool` | Enable verbose logging |
|
||||
|
||||
### `UploadResult`
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `BytesUploaded` | `int64` | Total bytes uploaded |
|
||||
| `Duration` | `time.Duration` | Total upload time |
|
||||
| `Protocol` | `string` | Protocol used |
|
||||
| `ResultURL` | `string` | Result URL (may differ from input after redirects) |
|
||||
|
||||
---
|
||||
|
||||
## Client
|
||||
|
||||
### `NewClient`
|
||||
|
||||
Creates a new goget client. Registers HTTP and FTP protocol handlers automatically.
|
||||
|
||||
```go
|
||||
func NewClient(opts ...ClientOption) *Client
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
| Option | Effect |
|
||||
|---|---|
|
||||
| `WithTimeout(5 * time.Minute)` | Set request timeout (default: 30 minutes) |
|
||||
| `WithUserAgent("MyApp/1.0")` | Custom User-Agent header |
|
||||
| `WithVerbose(true)` | Enable verbose logging |
|
||||
|
||||
**Example:**
|
||||
```go
|
||||
client := api.NewClient(
|
||||
api.WithTimeout(10 * time.Minute),
|
||||
api.WithUserAgent("MyDownloader/1.0"),
|
||||
)
|
||||
defer client.Close()
|
||||
```
|
||||
|
||||
### `Client.Download`
|
||||
|
||||
Downloads a file from a URL.
|
||||
|
||||
```go
|
||||
func (c *Client) Download(req *DownloadRequest) (*DownloadResult, error)
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|---|---|---|
|
||||
| `req.URL` | `string` | URL to download — supports http, https, ftp, sftp, file, data, gopher, gemini |
|
||||
| `req.Output` | `string` | Output path. Empty string uses the filename derived from the URL. |
|
||||
| `req.Resume` | `bool` | Resume an interrupted download if resume metadata exists |
|
||||
|
||||
**Returns:** `*DownloadResult` — bytes transferred, speed, duration, protocol info.
|
||||
|
||||
**Throws:**
|
||||
- `error` — if URL is empty or invalid
|
||||
- `error` — if protocol is unsupported
|
||||
- `*core.GogetError` — typed errors for network, protocol, timeout, auth, checksum failures
|
||||
|
||||
**Example:**
|
||||
```go
|
||||
result, err := client.Download(&api.DownloadRequest{
|
||||
URL: "https://example.com/file.zip",
|
||||
Output: "downloads/file.zip",
|
||||
Resume: true,
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("Download failed: %v", err)
|
||||
return
|
||||
}
|
||||
log.Printf("Downloaded %d bytes at %.1f KB/s", result.BytesDownloaded, result.Speed/1024)
|
||||
```
|
||||
|
||||
### `Client.Upload`
|
||||
|
||||
Uploads a file to a URL. The protocol handler must implement `core.Uploader`.
|
||||
|
||||
```go
|
||||
func (c *Client) Upload(req *UploadRequest) (*UploadResult, error)
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|---|---|---|
|
||||
| `req.URL` | `string` | Destination URL (required) |
|
||||
| `req.FilePath` | `string` | Path to local file (required) |
|
||||
| `req.Method` | `string` | `"PUT"` or `"POST"` (default: `"PUT"`) |
|
||||
| `req.Verbose` | `bool` | Enable verbose logging |
|
||||
|
||||
**Returns:** `*UploadResult` — bytes uploaded, duration, protocol, result URL.
|
||||
|
||||
**Throws:**
|
||||
- `error` — if URL or file path is empty
|
||||
- `error` — if protocol does not support upload
|
||||
|
||||
**Example:**
|
||||
```go
|
||||
result, err := client.Upload(&api.UploadRequest{
|
||||
URL: "https://httpbin.org/put",
|
||||
FilePath: "./data.csv",
|
||||
Method: "PUT",
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("Upload failed: %v", err)
|
||||
return
|
||||
}
|
||||
log.Printf("Uploaded %d bytes", result.BytesUploaded)
|
||||
```
|
||||
|
||||
### `Client.Close`
|
||||
|
||||
Releases resources and cancels any pending operations.
|
||||
|
||||
```go
|
||||
func (c *Client) Close()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Convenience Functions
|
||||
|
||||
### `Download`
|
||||
|
||||
One-shot download. Creates a client, downloads, and closes.
|
||||
|
||||
```go
|
||||
func Download(urlStr, output string) (*DownloadResult, error)
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```go
|
||||
result, err := api.Download("https://example.com/file.zip", "output.zip")
|
||||
```
|
||||
|
||||
### `Upload`
|
||||
|
||||
One-shot upload. Creates a client, uploads, and closes.
|
||||
|
||||
```go
|
||||
func Upload(urlStr, filePath, method string) (*UploadResult, error)
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```go
|
||||
result, err := api.Upload("https://httpbin.org/put", "./data.csv", "PUT")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Queue API
|
||||
|
||||
### `QueueClient`
|
||||
|
||||
Manages a download queue persisted to a JSON file.
|
||||
|
||||
```go
|
||||
type QueueClient struct { /* unexported */ }
|
||||
|
||||
func NewQueueClient(queueFile string) (*QueueClient, error)
|
||||
func (qc *QueueClient) Add(urlStr, output string, priority int) string
|
||||
func (qc *QueueClient) List() []*queue.QueueItem
|
||||
func (qc *QueueClient) Stats() queue.QueueStats
|
||||
func (qc *QueueClient) Remove(id string) bool
|
||||
func (qc *QueueClient) ClearCompleted() int
|
||||
```
|
||||
|
||||
### `NewQueueClient`
|
||||
|
||||
```go
|
||||
func NewQueueClient(queueFile string) (*QueueClient, error)
|
||||
```
|
||||
|
||||
Creates a queue client backed by a JSON file.
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|---|---|---|
|
||||
| `queueFile` | `string` | Path to the queue JSON file |
|
||||
|
||||
**Returns:** `*QueueClient`, `error` — error if the file cannot be read or created.
|
||||
|
||||
### `QueueClient.Add`
|
||||
|
||||
```go
|
||||
func (qc *QueueClient) Add(urlStr, output string, priority int) string
|
||||
```
|
||||
|
||||
Adds a URL to the queue.
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|---|---|---|
|
||||
| `urlStr` | `string` | URL to download |
|
||||
| `output` | `string` | Output file path |
|
||||
| `priority` | `int` | Priority (1–10, higher = processed first) |
|
||||
|
||||
**Returns:** `string` — unique ID for the queued item.
|
||||
|
||||
### `QueueClient.List`
|
||||
|
||||
```go
|
||||
func (qc *QueueClient) List() []*queue.QueueItem
|
||||
```
|
||||
|
||||
Returns all items in the queue.
|
||||
|
||||
**Returns:** `[]*queue.QueueItem`
|
||||
|
||||
### `QueueClient.Stats`
|
||||
|
||||
```go
|
||||
func (qc *QueueClient) Stats() queue.QueueStats
|
||||
```
|
||||
|
||||
Returns queue statistics (total, pending, completed, failed counts).
|
||||
|
||||
### `QueueClient.Remove`
|
||||
|
||||
```go
|
||||
func (qc *QueueClient) Remove(id string) bool
|
||||
```
|
||||
|
||||
Removes an item from the queue by ID.
|
||||
|
||||
**Returns:** `bool` — true if the item was found and removed.
|
||||
|
||||
### `QueueClient.ClearCompleted`
|
||||
|
||||
```go
|
||||
func (qc *QueueClient) ClearCompleted() int
|
||||
```
|
||||
|
||||
Removes all completed items from the queue.
|
||||
|
||||
**Returns:** `int` — number of removed items.
|
||||
|
||||
---
|
||||
|
||||
## Metalink API
|
||||
|
||||
### `ParseMetalink`
|
||||
|
||||
Parses a local Metalink file.
|
||||
|
||||
```go
|
||||
func ParseMetalink(path string) (*metalink.Metalink, error)
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|---|---|---|
|
||||
| `path` | `string` | Path to a local `.meta4` or `.metalink` file |
|
||||
|
||||
**Returns:** `*metalink.Metalink` — parsed metalink with sources, checksums, and metadata.
|
||||
|
||||
### `FetchMetalink`
|
||||
|
||||
Downloads and parses a Metalink from a URL.
|
||||
|
||||
```go
|
||||
func FetchMetalink(urlStr string) (*metalink.Metalink, error)
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|---|---|---|
|
||||
| `urlStr` | `string` | URL pointing to a `.meta4` or `.metalink` file |
|
||||
|
||||
**Returns:** `*metalink.Metalink` — parsed metalink with sources, checksums, and metadata.
|
||||
|
||||
---
|
||||
|
||||
## Config API
|
||||
|
||||
### `LoadConfig`
|
||||
|
||||
```go
|
||||
func LoadConfig(path string) (*config.Config, error)
|
||||
```
|
||||
|
||||
Loads the TOML configuration file. Returns default config if the file does not exist.
|
||||
|
||||
**Parameters:**
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|---|---|---|
|
||||
| `path` | `string` | Path to config file (empty = default location) |
|
||||
|
||||
**Returns:** `*config.Config`, `error`
|
||||
|
||||
### `DefaultConfig`
|
||||
|
||||
```go
|
||||
func DefaultConfig() *config.Config
|
||||
```
|
||||
|
||||
Returns the default configuration values.
|
||||
|
||||
### `ParseSpeed`
|
||||
|
||||
```go
|
||||
func ParseSpeed(s string) int64
|
||||
```
|
||||
|
||||
Parses a speed string like `"10MB/s"` into bytes per second.
|
||||
|
||||
| Input | Output |
|
||||
|---|---|
|
||||
| `"1MB/s"` | `1000000` |
|
||||
| `"500KB/s"` | `500000` |
|
||||
| `"1GB/s"` | `1000000000` |
|
||||
|
||||
---
|
||||
|
||||
## Version
|
||||
|
||||
### `Version`
|
||||
|
||||
```go
|
||||
func Version() string
|
||||
```
|
||||
|
||||
Returns the current goget version.
|
||||
|
||||
---
|
||||
|
||||
## Full Example
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"codeberg.org/petrbalvin/goget/pkg/api"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// Create client with custom timeout
|
||||
client := api.NewClient(
|
||||
api.WithTimeout(5 * time.Minute),
|
||||
api.WithVerbose(true),
|
||||
)
|
||||
defer client.Close()
|
||||
|
||||
// Download
|
||||
result, err := client.Download(&api.DownloadRequest{
|
||||
URL: "https://example.com/large-file.iso",
|
||||
Output: "./downloads/large-file.iso",
|
||||
Resume: true,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalf("download failed: %v", err)
|
||||
}
|
||||
|
||||
fmt.Printf("Downloaded %d bytes in %v\n", result.BytesDownloaded, result.Duration)
|
||||
fmt.Printf("Average speed: %.1f MB/s\n", result.Speed/1024/1024)
|
||||
fmt.Printf("Protocol: %s (IPv%d)\n", result.Protocol, result.IPVersion)
|
||||
|
||||
if result.Resumed {
|
||||
fmt.Println("Download was resumed from previous state")
|
||||
}
|
||||
if result.Parallel {
|
||||
fmt.Printf("Used %d parallel chunks\n", result.ChunksCount)
|
||||
}
|
||||
|
||||
// Queue a download
|
||||
qc, err := api.NewQueueClient("./queue.json")
|
||||
if err != nil {
|
||||
log.Fatalf("queue init failed: %v", err)
|
||||
}
|
||||
|
||||
id := qc.Add("https://example.com/another.zip", "./downloads/", 5)
|
||||
fmt.Printf("Queued item ID: %s\n", id)
|
||||
fmt.Printf("Queue stats: %+v\n", qc.Stats())
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,222 @@
|
||||
# Architecture
|
||||
|
||||
**goget** is a command-line download utility built on the Go standard library. It follows a layered architecture with a protocol abstraction layer, a transport layer, and utility modules for authentication, compression, output, and more.
|
||||
|
||||
## High-Level Overview
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
User[User / Terminal]:::accent6
|
||||
Script[Script / CI]:::accent6
|
||||
CLI[cmd/goget/ - Main entry]:::accent0
|
||||
API[pkg/api/ - Library interface]:::accent3
|
||||
Core[internal/core/ - Types, Protocol Interface, Errors]:::accent1
|
||||
Registry[internal/protocol/ - Protocol Registry]:::accent1
|
||||
HTTP[internal/protocol/http/]:::accent1
|
||||
FTP[internal/protocol/ftp/]:::accent1
|
||||
SFTP[internal/protocol/sftp/]:::accent1
|
||||
FILE[internal/protocol/file/]:::accent1
|
||||
DataURL[internal/protocol/dataurl/]:::accent1
|
||||
Gopher[internal/protocol/gopher/]:::accent1
|
||||
Gemini[internal/protocol/gemini/]:::accent1
|
||||
WebDAV[internal/protocol/webdav/]:::accent1
|
||||
Transport[internal/transport/ - Dialer, Proxy, TLS, Rate Limit]:::accent7
|
||||
Config[internal/config/]:::accent7
|
||||
Output[internal/output/ - Atomic write, Resume, WARC]:::accent2
|
||||
Auth[internal/auth/ - Basic, Digest, OAuth, netrc]:::accent7
|
||||
Crypto[internal/crypto/ - Checksum, PGP, Certs]:::accent7
|
||||
Cookie[internal/cookie/ - Netscape jar]:::accent7
|
||||
HSTS[internal/hsts/ - RFC 6797 cache]:::accent7
|
||||
Archive[internal/archive/ - Extraction]:::accent7
|
||||
Recursive[internal/recursive/ - Crawler, Parser, Filter]:::accent2
|
||||
Mirror[internal/mirror/]:::accent2
|
||||
Metalink[internal/metalink/]:::accent2
|
||||
Queue[internal/queue/]:::accent2
|
||||
Compression[internal/compression/ - gzip, bzip2, zlib, flate, lzw]:::accent7
|
||||
Log[internal/log/ - Structured logging]:::accent7
|
||||
WARC[internal/warc/ - Archiving]:::accent7
|
||||
Register[internal/protocol/register/ - Single registration]:::accent7
|
||||
Format[internal/format/ - Bytes, Speed formatting]:::accent7
|
||||
LinkRewrite[internal/linkrewrite/ - HTML link conversion]:::accent7
|
||||
CLIUtil[internal/cli/ - Help, Progress bar, Colors]:::accent7
|
||||
|
||||
User --> CLI
|
||||
Script --> API
|
||||
CLI --> Log
|
||||
CLI --> CLIUtil
|
||||
CLI --> Config
|
||||
CLI --> Core
|
||||
CLI -.-> API
|
||||
API --> Core
|
||||
API --> Registry
|
||||
|
||||
subgraph "Download Pipeline"
|
||||
Core --> Registry
|
||||
Registry --> HTTP
|
||||
Registry --> FTP
|
||||
Registry --> SFTP
|
||||
Registry --> FILE
|
||||
Registry --> DataURL
|
||||
Registry --> Gopher
|
||||
Registry --> Gemini
|
||||
Registry --> WebDAV
|
||||
HTTP --> Transport
|
||||
HTTP --> Cookie
|
||||
HTTP --> HSTS
|
||||
HTTP --> Output
|
||||
FTP --> Transport
|
||||
SFTP --> Transport
|
||||
end
|
||||
|
||||
subgraph "Post-Download"
|
||||
CLI --> Crypto
|
||||
CLI --> Archive
|
||||
CLI --> Compression
|
||||
end
|
||||
|
||||
subgraph "Download Management"
|
||||
CLI --> Register
|
||||
CLI --> WARC
|
||||
CLI --> Recursive
|
||||
CLI --> Mirror
|
||||
CLI --> Metalink
|
||||
CLI --> Queue
|
||||
Recursive --> WARC
|
||||
Recursive --> LinkRewrite
|
||||
Mirror --> LinkRewrite
|
||||
end
|
||||
|
||||
Transport --> Auth
|
||||
Transport --> Config
|
||||
|
||||
classDef accent0 fill:#3B82F6,stroke:#2563EB,color:#fff
|
||||
classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
|
||||
classDef accent2 fill:#F59E0B,stroke:#D97706,color:#000
|
||||
classDef accent3 fill:#A855F7,stroke:#7C3AED,color:#fff
|
||||
classDef accent6 fill:#6366F1,stroke:#4F46E5,color:#fff
|
||||
classDef accent7 fill:#64748B,stroke:#475569,color:#fff
|
||||
```
|
||||
|
||||
## Layers
|
||||
|
||||
### 1. Entry Points
|
||||
|
||||
| Layer | Package | Role |
|
||||
|---|---|---|
|
||||
| **CLI** | `cmd/goget/` | Flag parsing, command routing, signal handling |
|
||||
| **Library API** | `pkg/api/` | Public Go API for programmatic downloads, uploads, and queue management |
|
||||
|
||||
The CLI and API share the same internal packages. The CLI is built on top of the same `internal/core` types that the API exposes.
|
||||
|
||||
### 2. Protocol Abstraction
|
||||
|
||||
Defined in `internal/core` and `internal/protocol`:
|
||||
|
||||
```go
|
||||
// core.Protocol is the interface every protocol handler implements.
|
||||
type Protocol interface {
|
||||
Scheme() string
|
||||
CanHandle(url *url.URL) bool
|
||||
Download(ctx context.Context, req *DownloadRequest) (*DownloadResult, error)
|
||||
Capabilities() []string
|
||||
SupportsResume() bool
|
||||
SupportsCompression() bool
|
||||
SupportsParallel() bool
|
||||
SupportsRecursive() bool
|
||||
}
|
||||
|
||||
// core.Uploader is an optional interface for protocols supporting uploads.
|
||||
type Uploader interface {
|
||||
Upload(ctx context.Context, req *UploadRequest) (*UploadResult, error)
|
||||
SupportsUpload() bool
|
||||
}
|
||||
```
|
||||
|
||||
Protocols register themselves by scheme in a global `Registry`. Scheme normalization maps HTTPS → HTTP, FTPS → FTP, etc. Supported protocols include HTTP, FTP, SFTP, File, DataURL, Gopher, Gemini, and WebDAV (new in `internal/protocol/webdav/`).
|
||||
|
||||
### 3. Transport Layer
|
||||
|
||||
`internal/transport` provides a custom HTTP transport with:
|
||||
|
||||
- **IPv6-first dialer** — Resolves DNS, prefers IPv6 addresses, falls back to IPv4 only when no IPv6 record exists
|
||||
- **Interface binding** — Linux-specific socket binding via `SO_BINDTODEVICE`
|
||||
- **Rate limiting** — Token bucket algorithm for bandwidth capping
|
||||
- **Retry logic** — Exponential backoff with configurable max retries
|
||||
- **TLS configuration** — Certificate pinning, client certificates, key log file
|
||||
- **Proxy support** — HTTP CONNECT and SOCKS5 (`socks5://` with local DNS, `socks5h://` with remote DNS, RFC 1929 username/password auth)
|
||||
|
||||
### 4. Output Layer
|
||||
|
||||
`internal/output` handles atomic file writes:
|
||||
|
||||
1. Write to a `.goget.tmp` file during download
|
||||
2. Store resume metadata in `.goget.meta` alongside the temp file
|
||||
3. Atomically rename from `.goget.tmp` to the final filename on completion
|
||||
|
||||
### 5. Utility Modules
|
||||
|
||||
| Module | Responsibility |
|
||||
|---|---|
|
||||
| `auth` | HTTP Basic, Digest, OAuth 2.0 (client credentials, authorization code, refresh token), `.netrc` auto-reading |
|
||||
| `cookie` | Netscape-format cookie jar with import/export |
|
||||
| `crypto` | Checksum verification (SHA-256, SHA-512, SHA3-256/512, BLAKE2b, MD5), PGP signature verification and decryption, certificate handling |
|
||||
| `compression` | Decompression wrappers for gzip, bzip2, zlib, flate, LZW |
|
||||
| `hsts` | HSTS cache (RFC 6797) for automatic HTTPS upgrade |
|
||||
| `archive` | Archive extraction for tar, tar.gz, tar.bz2, zip with auto-detection |
|
||||
| `format` | Human-readable byte and speed formatting |
|
||||
| `linkrewrite` | HTML link rewriting for offline mirror viewing |
|
||||
| `log` | Structured logger with text and JSON output modes, level filtering, ANSI colors |
|
||||
| `warc` | WARC (Web ARChive) format writing for legal and archival compliance |
|
||||
| `cli` | ANSI color output, progress bar, help text, completion |
|
||||
|
||||
## Download Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant CLI as cmd/goget/
|
||||
participant Config as internal/config/
|
||||
participant Core as internal/core/
|
||||
participant Registry as internal/protocol/
|
||||
participant Proto as Protocol Handler
|
||||
participant Transport as internal/transport/
|
||||
participant Output as internal/output/
|
||||
|
||||
User->>CLI: goget --url https://example.com/file.zip
|
||||
CLI->>Config: Load config file
|
||||
Config-->>CLI: Config (with defaults + overrides)
|
||||
CLI->>Core: Create DownloadRequest
|
||||
CLI->>Registry: Get protocol for URL scheme
|
||||
Registry-->>CLI: HTTP protocol handler
|
||||
CLI->>Proto: Download(ctx, request)
|
||||
Proto->>Transport: Dial + send HTTP request
|
||||
Transport-->>Proto: HTTP response stream
|
||||
Proto->>Output: Write chunks atomically
|
||||
Output-->>Proto: Progress + completion
|
||||
Proto-->>CLI: DownloadResult
|
||||
CLI-->>User: Progress bar + summary
|
||||
```
|
||||
|
||||
## Config Merging
|
||||
|
||||
Configuration is resolved in this priority order (highest first):
|
||||
|
||||
1. CLI flags (`--timeout 5m`)
|
||||
2. Config file (`~/.config/goget/config.toml`)
|
||||
3. Default values
|
||||
|
||||
The `config.Merge()` method applies CLI override top of the loaded config.
|
||||
|
||||
## Error Handling
|
||||
|
||||
All operations return structured `core.GogetError` with:
|
||||
|
||||
- **Type** — Category (`NETWORK`, `PROTOCOL`, `FILE`, `TIMEOUT`, `AUTH`, `CHECKSUM`, `UNSUPPORTED`, `CANCELLED`)
|
||||
- **Message** — Human-readable description
|
||||
- **Cause** — Wrapped underlying error (available via `errors.Unwrap`)
|
||||
- **URL** — The URL that caused the error (credentials stripped)
|
||||
- **Details** — Optional key-value metadata (e.g., expected vs actual checksum)
|
||||
|
||||
## Build Tags
|
||||
|
||||
All source files use the build constraint `//go:build linux || freebsd`. The project targets Linux and FreeBSD only. Platform-specific code (e.g., network interface binding) lives in `internal/transport/bind_linux.go` vs `bind_other.go`.
|
||||
@@ -0,0 +1,251 @@
|
||||
# Authentication
|
||||
|
||||
goget supports multiple authentication mechanisms for HTTP, FTP, and SFTP connections. Credentials are resolved in priority order, with `.netrc` auto-reading as a fallback.
|
||||
|
||||
## Priority Order
|
||||
|
||||
1. CLI flags (`--username`, `--password`, `--password-file`)
|
||||
2. OAuth 2.0 tokens (if configured via CLI or config)
|
||||
3. `.netrc` auto-reading (`~/.netrc`)
|
||||
4. Anonymous / no auth
|
||||
|
||||
## HTTP Basic Authentication (RFC 7617)
|
||||
|
||||
The simplest mechanism. Credentials are Base64-encoded in the `Authorization` header.
|
||||
|
||||
```bash
|
||||
goget --username admin --password secret --url https://api.example.com/data
|
||||
```
|
||||
|
||||
```
|
||||
Authorization: Basic YWRtaW46c2VjcmV0
|
||||
```
|
||||
|
||||
### Security Notes
|
||||
|
||||
- Basic auth sends credentials in plain text (Base64 is not encryption)
|
||||
- Always use HTTPS with Basic auth
|
||||
- Prefer `--password-file` for scripting to avoid credentials in shell history:
|
||||
|
||||
```bash
|
||||
goget --username admin --password-file ./secret.txt --url https://api.example.com
|
||||
```
|
||||
|
||||
## HTTP Digest Authentication (RFC 7616)
|
||||
|
||||
Digest auth never sends the password in plain text. Instead, it uses a challenge-response mechanism with MD5 or SHA-256 hashing:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant Server
|
||||
|
||||
Client->>Server: GET /protected
|
||||
Server-->>Client: 401 + WWW-Authenticate: Digest nonce="abc", realm="app"
|
||||
Client->>Server: GET /protected + Authorization: Digest response="hash"
|
||||
Server-->>Client: 200 OK
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
goget --username admin --password secret --auth-type digest --url https://api.example.com
|
||||
goget --username admin --password secret --auth-type auto --url https://api.example.com
|
||||
```
|
||||
|
||||
With `--auth-type auto`, goget first tries to make the request. If the server returns 401 with a `WWW-Authenticate: Digest` header, it automatically switches to Digest auth.
|
||||
|
||||
### Implementation
|
||||
|
||||
The `internal/auth` package implements Digest auth per RFC 7616:
|
||||
|
||||
- **HA1** = `MD5(username:realm:password)`
|
||||
- **HA2** = `MD5(method:digestURI)`
|
||||
- **response** = `MD5(HA1:nonce:nc:cnonce:qop:HA2)`
|
||||
|
||||
The `cnonce` (client nonce) is generated using `crypto/rand` for each request.
|
||||
|
||||
## OAuth 2.0
|
||||
|
||||
goget supports four OAuth 2.0 grant types:
|
||||
|
||||
### Client Credentials Grant
|
||||
|
||||
For server-to-server communication without user interaction:
|
||||
|
||||
```bash
|
||||
goget \
|
||||
--oauth-client-id "my-client" \
|
||||
--oauth-client-secret "my-secret" \
|
||||
--oauth-token-url "https://auth.example.com/token" \
|
||||
--oauth-grant-type client_credentials \
|
||||
--oauth-scopes "read,write" \
|
||||
--url https://api.example.com/data
|
||||
```
|
||||
|
||||
### Authorization Code Grant
|
||||
|
||||
For web applications with user interaction:
|
||||
|
||||
```bash
|
||||
# Step 1: Generate authorization URL
|
||||
goget \
|
||||
--oauth-client-id "my-client" \
|
||||
--oauth-auth-url "https://auth.example.com/authorize" \
|
||||
--oauth-redirect-uri "http://localhost:8080/callback" \
|
||||
--url https://api.example.com/data
|
||||
|
||||
# Step 2: Exchange code for token (after user authorizes)
|
||||
goget \
|
||||
--oauth-client-id "my-client" \
|
||||
--oauth-client-secret "my-secret" \
|
||||
--oauth-token-url "https://auth.example.com/token" \
|
||||
--oauth-grant-type authorization_code \
|
||||
--url https://api.example.com/data
|
||||
```
|
||||
|
||||
### Refresh Token Grant
|
||||
|
||||
For maintaining long-lived access:
|
||||
|
||||
```bash
|
||||
goget \
|
||||
--oauth-client-id "my-client" \
|
||||
--oauth-client-secret "my-secret" \
|
||||
--oauth-token-url "https://auth.example.com/token" \
|
||||
--oauth-refresh-token "ref_xxx" \
|
||||
--oauth-grant-type refresh_token \
|
||||
--url https://api.example.com/data
|
||||
```
|
||||
|
||||
### Token Management
|
||||
|
||||
```go
|
||||
client := auth.NewOAuthClient(cfg)
|
||||
|
||||
// Check if token is valid (not expired)
|
||||
if !client.IsTokenValid() {
|
||||
// Refresh automatically
|
||||
client.RefreshToken(ctx)
|
||||
}
|
||||
|
||||
// Apply token to HTTP request
|
||||
client.ApplyToken(httpReq)
|
||||
// → Authorization: Bearer eyJhbG...
|
||||
```
|
||||
|
||||
The token expiry is tracked with a 30-second grace period.
|
||||
|
||||
### Security
|
||||
|
||||
- **Token URL enforced to HTTPS** — plain HTTP token endpoints are rejected unless they're `localhost` or `127.0.0.1`
|
||||
- **Tokens never logged** — The config `Save()` method strips `access_token`, `refresh_token`, and `client_secret` before writing to disk
|
||||
|
||||
## .netrc Auto-Reading
|
||||
|
||||
When no explicit credentials are provided via CLI flags or OAuth config, goget automatically checks `~/.netrc`:
|
||||
|
||||
```
|
||||
machine api.example.com
|
||||
login admin
|
||||
password secret
|
||||
|
||||
machine default
|
||||
login guest
|
||||
password guest123
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# No --username or --password needed if ~/.netrc has a matching entry
|
||||
goget --url https://api.example.com/data
|
||||
```
|
||||
|
||||
### Resolution Order
|
||||
|
||||
1. **Exact hostname match** — `api.example.com` matches `machine api.example.com`
|
||||
2. **Default entry** — `machine default` used as fallback
|
||||
3. **No match** — proceeds without credentials
|
||||
|
||||
### Skip .netrc
|
||||
|
||||
Use `--no-private` to disable .netrc and auth file reading:
|
||||
|
||||
```bash
|
||||
goget --no-private --url https://api.example.com
|
||||
```
|
||||
|
||||
## Cookies
|
||||
|
||||
goget supports Netscape-format cookie jars:
|
||||
|
||||
### CLI Cookies
|
||||
|
||||
```bash
|
||||
goget --cookie "session=abc123" --cookie "theme=dark" --url https://example.com
|
||||
```
|
||||
|
||||
### Cookie Jar
|
||||
|
||||
Persist cookies across sessions:
|
||||
|
||||
```bash
|
||||
# Save cookies to file
|
||||
goget --cookie-jar cookies.txt --url https://example.com
|
||||
|
||||
# Load cookies from file
|
||||
goget --cookie-jar cookies.txt --url https://example.com/protected
|
||||
```
|
||||
|
||||
The cookie jar uses the standard Netscape format:
|
||||
|
||||
## SFTP Authentication
|
||||
|
||||
SFTP connections authenticate via SSH:
|
||||
|
||||
### SSH Key Authentication
|
||||
|
||||
goget auto-detects keys from standard locations:
|
||||
|
||||
- `~/.ssh/id_rsa`
|
||||
- `~/.ssh/id_ecdsa`
|
||||
- `~/.ssh/id_ed25519`
|
||||
|
||||
```bash
|
||||
goget --url sftp://user@host:/path/to/file.zip
|
||||
```
|
||||
|
||||
### Password Authentication
|
||||
|
||||
```bash
|
||||
goget --url sftp://user@host:/path/to/file.zip --password "secret"
|
||||
```
|
||||
|
||||
### Known Hosts
|
||||
|
||||
Host key verification uses `~/.ssh/known_hosts` by default:
|
||||
|
||||
```bash
|
||||
# Custom known_hosts file
|
||||
goget --url sftp://user@host:/path/to/file.zip --known-hosts ~/.ssh/known_hosts
|
||||
|
||||
# Skip verification (first connection)
|
||||
goget --url sftp://user@host:/path/to/file.zip --ssh-insecure
|
||||
```
|
||||
|
||||
## FTP Authentication
|
||||
|
||||
FTP connections default to anonymous login. For authenticated FTP:
|
||||
|
||||
```bash
|
||||
# Username/password
|
||||
goget --url ftp://ftp.example.com/file.zip --username user --password pass
|
||||
|
||||
# Anonymous (default)
|
||||
goget --url ftp://anonymous@ftp.example.com/file.zip
|
||||
```
|
||||
|
||||
## Access Tokens in URLs
|
||||
|
||||
While `https://user:pass@example.com` URLs are supported, prefer `--username` and `--password` CLI flags. When a URL contains credentials, they're automatically stripped from error messages and logs via `core.SafeURL()`.
|
||||
@@ -0,0 +1,225 @@
|
||||
# CLI Reference
|
||||
|
||||
Complete reference for all goget command-line flags, organized by category.
|
||||
|
||||
```bash
|
||||
goget --url <URL> [OPTIONS]
|
||||
```
|
||||
|
||||
## Target
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--url` | `<URL>` | URL to download (repeatable for multiple URLs) |
|
||||
|
||||
## Output
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--output` | `<FILE>` | Output filename or directory (default: auto-derived from URL) |
|
||||
| `--restart` | — | Delete existing file and restart download from scratch |
|
||||
| `--overwrite` | — | Overwrite existing file without atomic rename |
|
||||
| `--content-disposition` | — | Use filename from server's `Content-Disposition` header |
|
||||
| `--create-dirs` | — | Create missing parent directories of `--output` (default: true). Use `--create-dirs=false` to disable (curl `--create-dirs` / wget behavior). Ignored for stdout output |
|
||||
| `--keep-timestamps` | — | Set file modification time to server's `Last-Modified` value |
|
||||
|
||||
## Progress
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--verbose` | — | Verbose output with Unicode progress bar (default enabled) |
|
||||
| `--no-progress` | — | Hide progress bar, keep other output |
|
||||
| `--progress` | `dot` | Dot progress bar style |
|
||||
| `--quiet` | — | Suppress all output except errors |
|
||||
| `--json` | — | Output progress as JSON lines for scripting (format: `{"type":"progress","current":X,"total":X,"speed":X}`) |
|
||||
| `--debug` | — | Enable debug output with detailed connection info |
|
||||
|
||||
## Network
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--timeout` | `<DURATION>` | Connection timeout (`0` = smart auto-calculation based on file size) |
|
||||
| `--connect-timeout` | `<DURATION>` | TCP connection establishment timeout |
|
||||
| `--max-time` | `<DURATION>` | Maximum total transfer time (abort if exceeded) |
|
||||
| `--max-retries` | `<N>` | Maximum retry attempts on failure |
|
||||
| `--max-filesize` | `<SIZE>` | Abort if file exceeds size limit (e.g. `100MB`, `1GB`) |
|
||||
| `--proxy` | `<URL>` | Proxy server URL — `http://` for HTTP CONNECT, `socks5://` (local DNS) or `socks5h://` (remote DNS) for SOCKS5, with optional `user:pass@` authentication |
|
||||
| `--no-ipv4` | — | Disable IPv4 fallback (IPv6 only) |
|
||||
| `--no-redirect` | — | Disable following HTTP redirects (3xx) |
|
||||
| `--dns-servers` | `<IPs>` | Custom DNS servers, comma-separated (e.g. `1.1.1.1,8.8.8.8`) |
|
||||
| `--bind-interface` | `<IFACE>` | Bind outgoing connections to a specific network interface (Linux only) |
|
||||
| `--allow-private` | — | Allow connections to private/internal IPs (disabled by default for SSRF protection) |
|
||||
| `--pinned-cert` | `<SHA256>` | Certificate pinning — abort if server cert SHA-256 doesn't match |
|
||||
|
||||
## HTTP / Network Flags
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--header` | `"K: V"` | Custom HTTP header (repeatable) |
|
||||
| `--insecure` | — | Skip TLS certificate verification |
|
||||
| `--cert` | `<FILE>` | Client certificate in PEM format (mTLS) |
|
||||
| `--key` | `<FILE>` | Client private key in PEM format (mTLS) |
|
||||
| `--data` | `"key=value"` | Send POST data (implies POST) |
|
||||
| `--form` | `"field=value"` | Multipart form field (repeatable; `--form "file=@./path"` for file upload) |
|
||||
| `--spider` | — | Check URL existence via HEAD request — no body downloaded |
|
||||
| `--write-out` | `<FORMAT>` | Print metadata after transfer (format: `%{http_code}`, `%{size_download}`, `%{time_total}`, `%{speed_download}`) |
|
||||
| `--trace-time` | — | Add HTTP timing breakdown to output (DNS, TCP, TLS, TTFB, Total) |
|
||||
| `--fail` | — | Exit non-zero on HTTP 4xx/5xx response |
|
||||
| `--compressed` | — | Accept-Encoding header for compressed transfer |
|
||||
| `--range` | `<N>-<M>` | Download a byte range (e.g. `--range 0-999`) |
|
||||
| `--post-file` | `<FILE>` | POST file contents as request body |
|
||||
| `--referer` | `<URL>` | Set Referer header |
|
||||
|
||||
## Recursive / Spider Flags
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--accept` | `"*.pdf"` | Accept patterns — filename glob or MIME type (comma-separated) |
|
||||
| `--no-parent` | — | Do not ascend to parent directories during recursive download |
|
||||
| `--adjust-extension` | — | Append `.html` extension to text/html files without one |
|
||||
| `--wait` | `<DURATION>` | Delay between requests in recursive mode |
|
||||
| `--random-wait` | — | Randomize wait time (0.5x–1.5x of `--wait` value) |
|
||||
| `--page-requisites` | — | Download CSS, JS, and images required to render HTML pages |
|
||||
| `--span-hosts` | — | Follow links to external domains in recursive mode |
|
||||
| `--convert-links` | — | Rewrite links in downloaded HTML for local offline viewing |
|
||||
| `--warc-file` | `<FILE>` | Write WARC (Web ARChive) output for archival |
|
||||
| `--dry-run` | — | List all URLs that would be downloaded in recursive/mirror mode without saving files to disk |
|
||||
| `--progress` | `dot` | Dot progress bar style |
|
||||
| `--no-clobber` | — | Skip download if file already exists |
|
||||
|
||||
## Recursive / Mirror
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--recursive` | — | Enable recursive downloading — follow links from HTML pages |
|
||||
| `--mirror` | — | Mirror entire website (infinite depth, convert links) — shorthand for `--recursive --convert-links --page-requisites` |
|
||||
| `--max-depth` | `<N>` | Maximum recursion depth |
|
||||
| `--follow-external` | — | Follow links to domains other than the starting domain |
|
||||
| `--exclude-pattern` | `<P>` | Exclude URL patterns (glob, comma-separated) |
|
||||
| `--no-convert-links` | — | Don't convert links in HTML during mirror (faster, but not offline-ready) |
|
||||
| `--no-mirror-assets` | — | Don't download CSS, JS, and images during mirror |
|
||||
| `--no-robots` | — | Don't respect `robots.txt` during mirroring |
|
||||
| `--rate-limit` | `<RATE>` | Maximum download speed (e.g. `1MB/s`, `500KB/s`, `0` = unlimited) |
|
||||
| `--domains` | `<LIST>` | Restrict recursion to listed domains (comma-separated) |
|
||||
| `--reject` | `<PATTERN>` | Reject URL patterns (glob) |
|
||||
| `--cut-dirs` | `<N>` | Ignore N directory components when creating local filenames |
|
||||
| `--proto-dirs` | — | Create protocol-prefixed directories (http/, ftp/ etc.) |
|
||||
| `--adjust-extension` | — | Append `.html` to text/html files missing an extension |
|
||||
| `--timestamping` | — | Only download files newer than the local copy |
|
||||
| `--backup-converted` | — | Back up original `.orig` files before link conversion |
|
||||
| `--recursive-parallel` | `<N>` | Number of concurrent file downloads in recursive mode across all protocols (HTTP, WebDAV, FTP, SFTP). `0` = sequential (default) |
|
||||
| `--continue` | — | Continue getting partially-downloaded file |
|
||||
|
||||
## Download Management
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--queue` | — | Add URL to the download queue (see `--queue-file`) |
|
||||
| `--queue-list` | — | List all items in the queue |
|
||||
| `--queue-process` | — | Process (download) all pending items in the queue |
|
||||
| `--queue-clear` | — | Remove completed items from the queue |
|
||||
| `--queue-file` | `<PATH>` | Custom queue file path (default: `~/.config/goget/queue.json`) |
|
||||
| `--queue-priority` | `<N>` | Priority for queued item (1–10, higher = processed first) |
|
||||
| `--batch-file` | `<FILE>` | Read URLs from a file, one URL per line |
|
||||
| `--input-file` | `<FILE>` | Read URLs from a file (alias for `--batch-file`) |
|
||||
| `--schedule` | `<TIME>` | Schedule download — absolute (`"2026-06-01 02:00"`) or cron (`"0 2 * * *"`) |
|
||||
| `--on-complete` | `<CMD>` | Run shell command after successful download (env: `GOGET_OUTPUT`, `GOGET_SIZE`, `GOGET_URL`) |
|
||||
|
||||
## Authentication
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--username` | `<USER>` | Username for HTTP Basic/Digest authentication |
|
||||
| `--password` | `<PASS>` | Password (prefer `--password-file` for security) |
|
||||
| `--password-file` | `<FILE>` | Read password from file |
|
||||
| `--auth-type` | `<TYPE>` | Authentication type: `basic`, `digest`, or `auto` |
|
||||
| `--cookie-jar` | `<FILE>` | Load and save cookies in Netscape format |
|
||||
| `--cookie` | `"n=v"` | Set cookie from CLI (repeatable) |
|
||||
| `--oauth-client-id` | `<ID>` | OAuth 2.0 Client ID |
|
||||
| `--oauth-client-secret` | `<S>` | OAuth 2.0 Client Secret |
|
||||
| `--oauth-token-url` | `<URL>` | OAuth 2.0 Token endpoint URL |
|
||||
| `--oauth-auth-url` | `<URL>` | OAuth 2.0 Authorization endpoint URL |
|
||||
| `--oauth-redirect-uri` | `<URI>` | OAuth 2.0 Redirect URI |
|
||||
| `--oauth-scopes` | `<S>` | OAuth 2.0 scopes (comma-separated) |
|
||||
| `--oauth-grant-type` | `<TYPE>` | OAuth 2.0 grant type: `client_credentials`, `authorization_code`, `password`, `refresh_token` |
|
||||
| `--oauth-access-token` | `<T>` | OAuth 2.0 access token |
|
||||
| `--oauth-refresh-token` | `<T>` | OAuth 2.0 refresh token |
|
||||
|
||||
## Upload
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--upload` | — | Upload file instead of downloading |
|
||||
| `--upload-method` | `<M>` | HTTP method: `PUT` or `POST` (default: `PUT`) |
|
||||
| `--upload-file` | `<FILE>` | Path to file to upload |
|
||||
| `--form-data` | `"k=v"` | Form data for multipart upload (comma-separated) |
|
||||
| `--file-field` | `<NAME>` | Form field name for file upload (default: `file`) |
|
||||
|
||||
## Verification
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--checksum` | `<HASH>` | Expected checksum (hex string) |
|
||||
| `--checksum-algo` | `<ALG>` | Algorithm: `sha256`, `sha512`, `blake2b`, `sha3-256`, `sha3-512`, `md5` (default: `sha256`) |
|
||||
| `--checksum-file` | `<FILE>` | Read expected checksum from SHA256SUMS-format file |
|
||||
|
||||
## PGP / GPG
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--pgp-decrypt` | — | Decrypt PGP-encrypted file after download |
|
||||
| `--pgp-verify` | — | Verify PGP detached or inline signature |
|
||||
| `--pgp-sig` | `<FILE>` | Path to detached signature file (`.asc`, `.sig`) |
|
||||
| `--pgp-key` | `<FILE>` | Path to PGP key file |
|
||||
| `--pgp-passphrase` | `<PASS>` | Passphrase for private key |
|
||||
| `--pgp-passphrase-file` | `<F>` | Read passphrase from file |
|
||||
|
||||
## Metalink
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--metalink` | — | Treat URL as a Metalink file (multi-source download) |
|
||||
| `--metalink-file` | `<PATH>` | Path to local Metalink file (`.meta4`, `.metalink`) |
|
||||
|
||||
## SSL / TLS
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--ssl-key-log` | `<FILE>` | Log TLS master secrets to file (SSLKEYLOGFILE for Wireshark) |
|
||||
| `--known-hosts` | `<FILE>` | Custom SSH `known_hosts` file path for SFTP |
|
||||
| `--ssh-insecure` | — | Skip SSH host key verification for SFTP (accept any key) |
|
||||
| `--cacert` | `<FILE>` | Custom CA certificate bundle |
|
||||
|
||||
## Configuration
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--config` | `<FILE>` | Path to configuration file |
|
||||
| `--init-config` | — | Generate default config file at `~/.config/goget/config.toml` |
|
||||
| `--config-get` | `<KEY>` | Get value (e.g. `timeout`, `parallel`) |
|
||||
| `--config-set` | `<K>` `<V>` | Set value (e.g. `--config-set timeout 5m`) |
|
||||
| `--config-list` | — | List all config keys and values |
|
||||
| `--config-unset` | `<KEY>` | Remove config key |
|
||||
|
||||
## Information
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--info` | `<URL>` | Show file metadata without downloading (size, type, server info) |
|
||||
| `--benchmark` | `<URL>` | Benchmark download speed (downloads first 10 MB) |
|
||||
| `--generate-man-page` | — | Generate man page in troff format and exit |
|
||||
| `--completion` | `<SHELL>` | Generate shell completion for `bash`, `zsh`, or `fish` |
|
||||
| `--help` | — | Show help text and exit |
|
||||
| `--version` | — | Show version and exit |
|
||||
|
||||
## Additional Flags
|
||||
|
||||
| Flag | Argument | Description |
|
||||
|---|---|---|
|
||||
| `--parallel` | `<N>` | Number of parallel chunk connections (`0` = auto, `1` = sequential) |
|
||||
| `--max-speed` | `<RATE>` | Maximum download speed in bytes/sec or human format |
|
||||
| `--no-private` | — | Disable `netrc` and auth file reading |
|
||||
| `--glob` | `<PATTERN>` | URL glob pattern expansion (e.g. `--glob "https://example.com/file[1-3].zip"`) |
|
||||
| `--retry-all-errors` | — | Retry on any HTTP 4xx/5xx response |
|
||||
| `--retry-delay` | `<DURATION>` | Custom delay before retry |
|
||||
| `--hsts-file` | `<FILE>` | Custom HSTS cache file path |
|
||||
@@ -0,0 +1,122 @@
|
||||
# Configuration Reference
|
||||
|
||||
goget loads configuration from a TOML file at `~/.config/goget/config.toml`. CLI flags override config file values, which override defaults. The `GOGET_CONFIG` environment variable can be used to specify an alternative config path.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Generate default config
|
||||
goget --init-config
|
||||
|
||||
# Read a value
|
||||
goget --config-get timeout
|
||||
|
||||
# Set a value
|
||||
goget --config-set timeout 5m
|
||||
|
||||
# List all values
|
||||
goget --config-list
|
||||
|
||||
# Remove a value
|
||||
goget --config-unset timeout
|
||||
```
|
||||
|
||||
## Configuration Keys
|
||||
|
||||
### Network
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `timeout` | duration string | `30m` | Request timeout. Set to `0` for auto-calculation based on file size |
|
||||
| `parallel` | int | `0` | Parallel connections. `0` = auto (4 for files >100 MB, 1 otherwise) |
|
||||
| `proxy` | string | `""` | Proxy server URL (HTTP CONNECT) |
|
||||
| `max_speed` | int64 | `0` | Bytes per second cap. `0` = unlimited |
|
||||
| `max_retries` | int | `0` | Retry attempts on failure |
|
||||
| `dns_servers` | []string | `[]` | Custom DNS servers (e.g. `["1.1.1.1","8.8.8.8"]`) |
|
||||
| `ipv4_fallback` | bool | `true` | Whether to fall back to IPv4 if IPv6 is unavailable |
|
||||
|
||||
### HTTP
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `user_agent` | string | `Goget/1.0.0` | HTTP User-Agent header value |
|
||||
| `auto_decompress` | bool | `true` | Auto-decompress gzip/deflate responses |
|
||||
| `insecure_skip_verify` | bool | `false` | Skip TLS certificate verification (insecure) |
|
||||
| `auto_resume` | bool | `true` | Resume download when `.goget.meta` metadata exists |
|
||||
| `cookie_jar` | string | `""` | Path to Netscape-format cookie jar file |
|
||||
| `keep_metadata` | bool | `false` | Keep `.goget.meta` files after download completes |
|
||||
|
||||
### Authentication
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `auth.username` | string | `""` | Default username for HTTP Auth |
|
||||
| `auth.password` | string | `""` | Default password (⚠ stored in plain text — prefer `password_file`) |
|
||||
| `auth.password_file` | string | `""` | Path to file containing the password |
|
||||
| `auth.auth_type` | string | `"auto"` | `basic`, `digest`, or `auto` |
|
||||
| `auth.oauth.client_id` | string | `""` | OAuth 2.0 Client ID |
|
||||
| `auth.oauth.client_secret` | string | `""` | OAuth 2.0 Client Secret |
|
||||
| `auth.oauth.token_url` | string | `""` | OAuth 2.0 Token endpoint URL |
|
||||
| `auth.oauth.auth_url` | string | `""` | OAuth 2.0 Authorization endpoint URL |
|
||||
| `auth.oauth.redirect_uri` | string | `""` | OAuth 2.0 Redirect URI |
|
||||
| `auth.oauth.scopes` | []string | `[]` | OAuth 2.0 scopes |
|
||||
| `auth.oauth.access_token` | string | `""` | OAuth 2.0 Access Token |
|
||||
| `auth.oauth.refresh_token` | string | `""` | OAuth 2.0 Refresh Token |
|
||||
| `auth.oauth.grant_type` | string | `""` | `client_credentials`, `authorization_code`, `password`, `refresh_token` |
|
||||
|
||||
### Recursive Downloads
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `recursive.enabled` | bool | `false` | Enable recursive downloading |
|
||||
| `recursive.max_depth` | int | `3` | Maximum recursion depth |
|
||||
| `recursive.follow_external` | bool | `false` | Follow links to external domains |
|
||||
| `recursive.exclude_patterns` | []string | `[]` | URL patterns to exclude (simple substring match) |
|
||||
| `recursive.include_patterns` | []string | `[]` | URL patterns to include (simple substring match) |
|
||||
| `recursive.delay` | duration string | `100ms` | Delay between requests in recursive mode |
|
||||
| `recursive.parallel` | int | `0` | Concurrent file downloads in recursive mode. `0` = sequential, `N` = up to N concurrent workers (applies to HTTP, WebDAV, FTP, SFTP) |
|
||||
|
||||
### Archive Extraction
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `extract.enabled` | bool | `false` | Auto-extract archives after download |
|
||||
| `extract.output_dir` | string | `""` | Extraction target directory (empty = same as download dir) |
|
||||
| `extract.strip_components` | int | `0` | Number of leading path components to strip |
|
||||
| `extract.exclude_patterns` | []string | `[]` | File patterns to exclude from extraction |
|
||||
| `extract.preserve_permissions` | bool | `true` | Preserve file permissions from archive |
|
||||
| `extract.auto_detect` | bool | `true` | Auto-detect archive format (tar, tar.gz, tar.bz2, zip) |
|
||||
|
||||
### Output / Display
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `output_dir` | string | `"."` | Default output directory |
|
||||
| `verbose` | bool | `true` | Verbose output with progress bar |
|
||||
| `debug` | bool | `false` | Debug output with detailed connection info |
|
||||
| `color` | string | `"auto"` | Color output: `always`, `never`, or `auto` (honors `NO_COLOR` env var) |
|
||||
| `progress_style` | string | `"unicode"` | Progress bar style: `"unicode"` or `"dot"` |
|
||||
| `show_eta` | bool | `true` | Display estimated time remaining |
|
||||
| `checksum_algo` | string | `"sha256"` | Default checksum algorithm |
|
||||
|
||||
## Password Security
|
||||
|
||||
Passwords are stored in plain text in the config file by default. For better security, use `auth.password_file` to reference an external file, and set `GOGET_PASSWORD` as an environment variable — the config loader will prefer it.
|
||||
|
||||
When `config.Save()` is called, the `password`, `client_secret`, `access_token`, and `refresh_token` fields are stripped from the output to prevent accidental credential leakage.
|
||||
|
||||
## Config Resolution Order
|
||||
|
||||
1. **CLI flags** (highest priority) — override everything below
|
||||
2. **Config file** — `~/.config/goget/config.toml` (or `GOGET_CONFIG` path)
|
||||
3. **Defaults** (lowest priority) — hardcoded in `DefaultConfig()`
|
||||
|
||||
The `config.Merge()` method applies CLI overrides after loading:
|
||||
|
||||
```go
|
||||
cfg, _ := config.Load("")
|
||||
cfg.Merge(&config.CLIFlags{
|
||||
Timeout: 5 * time.Minute,
|
||||
Parallel: 8,
|
||||
})
|
||||
```
|
||||
@@ -0,0 +1,312 @@
|
||||
# Download Pipeline
|
||||
|
||||
This document explains how goget processes a download request from start to finish — the phase structure, parallel chunking, resume logic, and output writing.
|
||||
|
||||
## High-Level Flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([User invokes goget]):::accent0
|
||||
ParseFlags["Parse CLI flags"]:::accent7
|
||||
LoadConfig["Load config + merge overrides"]:::accent7
|
||||
ResolveProtocol["Resolve protocol handler\nfrom URL scheme"]:::accent1
|
||||
CreateRequest["Build core.DownloadRequest"]:::accent1
|
||||
CheckResume{"Resume metadata\nexists?"}:::accent2
|
||||
LoadResume["Load .goget.meta\n(resume info + chunk map)"]:::accent2
|
||||
SetupOutput["Setup output writer\n(atomic temp file + resume)"]:::accent2
|
||||
CreateDirs{"--create-dirs\n(parent missing)?"}:::accent2
|
||||
MkdirAll["os.MkdirAll → create\nparent directories"]:::accent2
|
||||
CheckParallel{"File > 100 MB\nor --parallel set?"}:::accent1
|
||||
SequentialDownload["Sequential download\n(single connection)"]:::accent1
|
||||
ParallelDownload["Parallel chunked download\n(multiple Range requests)"]:::accent1
|
||||
Decompress{"Auto-decompress\nenabled?"}:::accent7
|
||||
Decompression["Decompress response\n(gzip, deflate, bzip2, zlib)"]:::accent7
|
||||
VerifyChecksum{"Checksum\nspecified?"}:::accent7
|
||||
ChecksumVerify["Verify checksum\n(SHA-256, SHA-512, etc.)"]:::accent7
|
||||
PGPVerify{"PGP verify/\ndecrypt?"}:::accent7
|
||||
PGPProcess["Verify signature /\ndecrypt file"]:::accent7
|
||||
SaveHSTS["Save HSTS cache\n(RFC 6797)"]:::accent7
|
||||
AtomicRename["Atomic rename\ntemp → final"]:::accent2
|
||||
SigInt{"SIGINT received?"}:::accent4
|
||||
SaveSidecar["Save .goget.meta\nfor resume"]:::accent2
|
||||
Exit130["Exit 130"]:::accent4
|
||||
RunHook{"--on-complete\nset?"}:::accent7
|
||||
PostHook["Run post-download command\n(GOGET_OUTPUT, GOGET_SIZE, GOGET_URL)"]:::accent7
|
||||
Done([Done]):::accent0
|
||||
|
||||
Start --> ParseFlags --> LoadConfig --> ResolveProtocol
|
||||
ResolveProtocol --> CreateRequest --> CheckResume
|
||||
CheckResume -->|Yes| LoadResume --> SetupOutput
|
||||
CheckResume -->|No| SetupOutput
|
||||
SetupOutput --> CreateDirs
|
||||
CreateDirs -->|Yes| MkdirAll --> CheckParallel
|
||||
CreateDirs -->|No| CheckParallel
|
||||
CheckParallel -->|Yes| ParallelDownload
|
||||
CheckParallel -->|No| SequentialDownload
|
||||
SequentialDownload --> Decompress
|
||||
ParallelDownload --> Decompress
|
||||
Decompress -->|Yes| Decompression --> VerifyChecksum
|
||||
Decompress -->|No| VerifyChecksum
|
||||
VerifyChecksum -->|Yes| ChecksumVerify -->|pass| PGPVerify
|
||||
ChecksumVerify -->|fail| Done
|
||||
VerifyChecksum -->|No| PGPVerify
|
||||
PGPVerify -->|Yes| PGPProcess --> SaveHSTS
|
||||
PGPVerify -->|No| SaveHSTS
|
||||
SaveHSTS --> AtomicRename
|
||||
AtomicRename --> SigInt
|
||||
SigInt -->|Yes| SaveSidecar --> Exit130
|
||||
SigInt -->|No| RunHook
|
||||
RunHook -->|Yes| PostHook --> Done
|
||||
RunHook -->|No| Done
|
||||
|
||||
classDef accent0 fill:#3B82F6,stroke:#2563EB,color:#fff
|
||||
classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
|
||||
classDef accent2 fill:#F59E0B,stroke:#D97706,color:#000
|
||||
classDef accent4 fill:#EF4444,stroke:#DC2626,color:#fff
|
||||
classDef accent7 fill:#64748B,stroke:#475569,color:#fff
|
||||
```
|
||||
|
||||
## Phase 1: Request Construction
|
||||
|
||||
A `core.DownloadRequest` is built from CLI flags and config:
|
||||
|
||||
```go
|
||||
req := &core.DownloadRequest{
|
||||
URL: parsedURL,
|
||||
Output: outputPath,
|
||||
Resume: resumeEnabled,
|
||||
Timeout: effectiveTimeout,
|
||||
Verbose: verbose,
|
||||
Headers: customHeaders,
|
||||
Proxy: proxyURL,
|
||||
Checksum: expectedChecksum,
|
||||
Parallel: parallelConfig,
|
||||
Recursive: recursiveEnabled,
|
||||
MaxDepth: maxDepth,
|
||||
ProgressCallback: progressFn,
|
||||
Ctx: ctx,
|
||||
}
|
||||
```
|
||||
|
||||
## Phase 2: Protocol Resolution
|
||||
|
||||
The protocol registry resolves the handler by URL scheme:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
URL["https://example.com"]:::accent6
|
||||
Parse["Parse scheme\n→ https"]:::accent7
|
||||
Normalized["Normalize\nhttps → http"]:::accent7
|
||||
Registry["Registry lookup\nprotocols[http]"]:::accent1
|
||||
Handler["HTTP protocol\nhandler"]:::accent1
|
||||
|
||||
URL --> Parse --> Normalized --> Registry --> Handler
|
||||
|
||||
classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
|
||||
classDef accent6 fill:#6366F1,stroke:#4F46E5,color:#fff
|
||||
classDef accent7 fill:#64748B,stroke:#475569,color:#fff
|
||||
```
|
||||
|
||||
Scheme normalization maps:
|
||||
- `https://` → `http` (same handler)
|
||||
- `ftps://` → `ftp` (same handler)
|
||||
- `webdavs://` → `webdav` (same handler)
|
||||
|
||||
## Phase 3: Output Setup
|
||||
|
||||
The `internal/output` package creates an atomic writer:
|
||||
|
||||
1. **Temp file** — Writes to `<filename>.goget.tmp` during download
|
||||
2. **Parent directories** — If `--create-dirs` is enabled (default: `true`), missing parent directories are created via `os.MkdirAll` before opening the output file. Pass `--create-dirs=false` to restore strict behaviour
|
||||
3. **Resume metadata** — Reads `<filename>.goget.meta` if resuming
|
||||
4. **Progress callback** — Hooks into the writer for real-time speed/ETA
|
||||
|
||||
```go
|
||||
writer, err := output.NewWriter(&output.WriterConfig{
|
||||
Output: outputFile,
|
||||
Atomic: true,
|
||||
Resume: resumeEnabled,
|
||||
CreateDirs: true, // auto-create parent dirs (curl --create-dirs)
|
||||
ProgressCallback: progressFunc,
|
||||
})
|
||||
```
|
||||
|
||||
### Resume Metadata Format
|
||||
|
||||
```json
|
||||
{
|
||||
"downloaded_bytes": 524288000,
|
||||
"etag": "\"abc123\"",
|
||||
"last_modified": "Mon, 01 Jun 2026 12:00:00 GMT",
|
||||
"url": "https://example.com/file.zip",
|
||||
"last_write": "2026-06-01T12:05:00Z",
|
||||
"chunks": {
|
||||
"0": 131072000,
|
||||
"1": 131072000,
|
||||
"2": 131072000,
|
||||
"3": 131072000
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For parallel downloads, each chunk's progress is individually tracked.
|
||||
|
||||
## Phase 4: Download Strategy
|
||||
|
||||
### Sequential Download
|
||||
|
||||
Used when the file is under 100 MB or `--parallel 1` is set:
|
||||
|
||||
```go
|
||||
resp, _ := client.Do(request)
|
||||
io.Copy(writer, resp.Body)
|
||||
```
|
||||
|
||||
### Parallel Chunked Download
|
||||
|
||||
Triggered automatically for files over 100 MB, or explicitly with `--parallel N`:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Main
|
||||
participant Chunk1
|
||||
participant Chunk2
|
||||
participant Chunk3
|
||||
participant Chunk4
|
||||
participant Writer as Atomic Writer
|
||||
|
||||
Main->>Main: HEAD request → get file size
|
||||
Main->>Main: Split into N equal chunks
|
||||
Main->>Chunk1: Start: bytes 0-13107199
|
||||
Main->>Chunk2: Start: bytes 13107200-26214399
|
||||
Main->>Chunk3: Start: bytes 26214400-39321599
|
||||
Main->>Chunk4: Start: bytes 39321600-52428799
|
||||
Chunk1->>Writer: Write bytes to temp/chunk_0
|
||||
Chunk2->>Writer: Write bytes to temp/chunk_1
|
||||
Chunk3->>Writer: Write bytes to temp/chunk_2
|
||||
Chunk4->>Writer: Write bytes to temp/chunk_3
|
||||
Main->>Writer: Merge chunks → final file
|
||||
```
|
||||
|
||||
Each chunk downloads via a separate HTTP `Range` request:
|
||||
|
||||
```http
|
||||
GET /large.iso HTTP/1.1
|
||||
Host: example.com
|
||||
Range: bytes=13107200-26214399
|
||||
```
|
||||
|
||||
### Concurrency Control
|
||||
|
||||
- Max 4 parallel connections by default
|
||||
- Customizable via `--parallel N` or config `parallel` key
|
||||
- Min chunk size: 1 MB
|
||||
- Max chunk size: 50 MB
|
||||
|
||||
## Phase 5: Post-Processing
|
||||
|
||||
### Decompression
|
||||
|
||||
If `auto_decompress` is enabled (default) and the server sends compressed content, the response body is transparently decompressed:
|
||||
|
||||
| Content-Encoding | Handler |
|
||||
|---|---|
|
||||
| `gzip` | `compress/gzip` |
|
||||
| `deflate` | `compress/flate` |
|
||||
| `zlib` | Internal |
|
||||
| `bzip2` | Internal |
|
||||
| `lzw` | Internal |
|
||||
|
||||
Use `--no-decompress` to preserve the compressed response.
|
||||
|
||||
### Checksum Verification
|
||||
|
||||
If `--checksum` or `--checksum-file` is provided, the downloaded file is hashed and compared:
|
||||
|
||||
```
|
||||
Expected: a1b2c3d4...
|
||||
Actual: a1b2c3d4...
|
||||
Checksum OK
|
||||
```
|
||||
|
||||
Supported algorithms: SHA-256, SHA-512, SHA3-256, SHA3-512, BLAKE2b, MD5.
|
||||
|
||||
### Atomic Rename
|
||||
|
||||
On successful completion, the temp file is atomically renamed:
|
||||
|
||||
```go
|
||||
os.Rename("file.zip.goget.tmp", "file.zip")
|
||||
```
|
||||
|
||||
If the download fails or is interrupted, the temp file remains for resume.
|
||||
|
||||
### PGP Verification and Decryption
|
||||
|
||||
If `--pgp-verify` or `--pgp-decrypt` is set, the downloaded file is processed via `golang.org/x/crypto/openpgp`:
|
||||
|
||||
- **Detached signature verification** — `--pgp-sig file.sig --pgp-key public.key`
|
||||
- **Decryption** — `--pgp-decrypt --pgp-key private.key --pgp-passphrase "secret"`
|
||||
- Auto-detect signature files: `.asc`, `.sig`
|
||||
|
||||
The decrypted file replaces the encrypted one (`.gpg` → stripped extension, or `.decrypted` suffix).
|
||||
|
||||
### HSTS Cache
|
||||
|
||||
After every successful HTTPS connection, the HSTS cache (`~/.config/goget/hsts`) is updated per RFC 6797. Expired entries are pruned on load. This ensures that future `http://` requests to known hosts are automatically upgraded to `https://`.
|
||||
|
||||
### Graceful Shutdown (SIGINT)
|
||||
|
||||
When the user sends Ctrl+C (SIGINT) during a download, goget persists partial progress to the `.goget.meta` sidecar for **all protocols** (HTTP, FTP, SFTP, WebDAV) and exits with code 130. A subsequent `goget --resume` picks up where it left off. The signal handler uses `signal.NotifyContext` and propagates cancellation through the download pipeline via `ctx.Err()` checks after every read.
|
||||
|
||||
## Smart Timeout
|
||||
|
||||
When no explicit `--timeout` is set, goget calculates a timeout based on file size:
|
||||
|
||||
```
|
||||
timeout = (fileSize / minSpeed) × safetyFactor
|
||||
```
|
||||
|
||||
Where:
|
||||
- `minSpeed` = 10 KB/s (conservative minimum)
|
||||
- `safetyFactor` = 3.0
|
||||
- `minTimeout` = 30 seconds
|
||||
- `maxTimeout` = 24 hours
|
||||
|
||||
A 1 GB file at 10 KB/s → 34 hours (capped to 24 hours max).
|
||||
A 10 MB file at 10 KB/s → 51 minutes.
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
`--rate-limit` uses a token bucket algorithm:
|
||||
|
||||
```go
|
||||
bucket := transport.NewTokenBucket(rate)
|
||||
// For each read:
|
||||
bucket.Wait(n)
|
||||
```
|
||||
|
||||
The token bucket allows short bursts above the limit while maintaining the average rate.
|
||||
|
||||
### Speed Format
|
||||
|
||||
```
|
||||
--rate-limit 1MB/s # 1,000,000 bytes/sec
|
||||
--rate-limit 500KB/s # 500,000 bytes/sec
|
||||
--rate-limit 1GB/s # 1,000,000,000 bytes/sec
|
||||
--rate-limit 100 # 100 bytes/sec (plain number)
|
||||
```
|
||||
|
||||
## Retry Logic
|
||||
|
||||
When a download fails, goget retries with exponential backoff:
|
||||
|
||||
```
|
||||
Attempt 1: wait 1s, retry
|
||||
Attempt 2: wait 2s, retry
|
||||
Attempt 3: wait 4s, retry
|
||||
...
|
||||
Attempt N: wait min(2^(N-1) × 1s, 30s), retry
|
||||
```
|
||||
|
||||
Max retries default to 3. Use `--max-retries N` to increase. Use `--retry-all-errors` to retry on HTTP 4xx/5xx (by default, only 5xx server errors trigger retries).
|
||||
@@ -0,0 +1,173 @@
|
||||
# Migration from curl & wget
|
||||
|
||||
goget is designed as a drop-in replacement for `curl` and `wget` in most workflows. This guide helps you migrate existing scripts and commands.
|
||||
|
||||
## Quick Translation
|
||||
|
||||
### curl → goget
|
||||
|
||||
| curl | goget | Notes |
|
||||
|---|---|---|
|
||||
| `curl -O <URL>` | `goget --url <URL>` | Download with auto filename |
|
||||
| `curl -o file.zip <URL>` | `goget --url <URL> --output file.zip` | Named output |
|
||||
| `curl -C - <URL>` | `goget --url <URL> --resume` | Resume download |
|
||||
| `curl -L <URL>` | `goget --url <URL>` | Redirects followed by default |
|
||||
| `curl -H "K: V" <URL>` | `goget --header "K: V" --url <URL>` | Custom headers |
|
||||
| `curl -d "data" <URL>` | `goget --data "data" --url <URL>` | POST data |
|
||||
| `curl -F "f=@file" <URL>` | `goget --form "f=@file" --url <URL>` | Multipart upload |
|
||||
| `curl -k <URL>` | `goget --insecure --url <URL>` | Skip TLS verify |
|
||||
| `curl -x <proxy> <URL>` | `goget --proxy <proxy> --url <URL>` | HTTP proxy |
|
||||
| `curl -I <URL>` | `goget --spider --url <URL>` | HEAD request only |
|
||||
| `curl --create-dirs <URL>` | `goget --url <URL>` | Create dirs (default in goget) |
|
||||
| `curl --max-time 30 <URL>` | `goget --max-time 30s --url <URL>` | Max transfer time |
|
||||
| `curl --cert cert.pem --key key.pem` | `goget --cert cert.pem --key key.pem --url <URL>` | mTLS |
|
||||
| `curl -w "%{http_code}" <URL>` | `goget --write-out "%{http_code}" --url <URL>` | Metadata output |
|
||||
| `curl -s <URL>` | `goget --quiet --url <URL>` | Silent mode |
|
||||
| `curl -v <URL>` | `goget --verbose --url <URL>` | Verbose mode |
|
||||
| `curl -b cookies.txt <URL>` | `goget --cookie-jar cookies.txt --url <URL>` | Cookies |
|
||||
| `curl --ssl-key-log-file keys.log` | `goget --ssl-key-log keys.log --url <URL>` | SSL key log |
|
||||
| `curl --retry-all-errors <URL>` | `goget --retry-all-errors --url <URL>` | Retry on all errors |
|
||||
|
||||
### wget → goget
|
||||
|
||||
| wget | goget | Notes |
|
||||
|---|---|---|
|
||||
| `wget <URL>` | `goget --url <URL>` | Basic download |
|
||||
| `wget -O file.zip <URL>` | `goget --url <URL> --output file.zip` | Named output |
|
||||
| `wget -c <URL>` | `goget --url <URL> --resume` | Resume download |
|
||||
| `wget -r <URL>` | `goget --url <URL> --recursive` | Recursive download |
|
||||
| `wget -r -l 3 <URL>` | `goget --url <URL> --recursive --max-depth 3` | Depth limit |
|
||||
| `wget -m <URL>` | `goget --url <URL> --mirror` | Full mirror |
|
||||
| `wget --mirror --convert-links` | `goget --url <URL> --mirror` | Mirror + convert links |
|
||||
| `wget -p <URL>` | `goget --url <URL> --page-requisites` | Page requisites |
|
||||
| `wget -A "*.pdf" <URL>` | `goget --url <URL> --accept "*.pdf"` | Accept pattern |
|
||||
| `wget -R "*.zip" <URL>` | `goget --url <URL> --reject "*.zip"` | Reject pattern |
|
||||
| `wget -np <URL>` | `goget --url <URL> --no-parent` | No parent dir |
|
||||
| `wget -H <URL>` | `goget --url <URL> --span-hosts` | Span hosts |
|
||||
| `wget -D a.com,b.com <URL>` | `goget --url <URL> --domains a.com,b.com` | Limit domains |
|
||||
| `wget -w 2 <URL>` | `goget --url <URL> --wait 2s` | Delay between requests |
|
||||
| `wget --random-wait <URL>` | `goget --url <URL> --random-wait` | Random delay |
|
||||
| `wget --limit-rate 1M <URL>` | `goget --url <URL> --rate-limit 1MB/s` | Rate limit |
|
||||
| `wget --warc-file=archive` | `goget --url <URL> --warc-file archive.warc` | WARC output |
|
||||
| `wget --progress=dot <URL>` | `goget --url <URL> --progress=dot` | Dot progress |
|
||||
| `wget --adjust-extension <URL>` | `goget --url <URL> --adjust-extension` | Add .html extension |
|
||||
| `wget --no-clobber <URL>` | `goget --url <URL> --no-clobber` | Skip existing files |
|
||||
| `wget --timestamping <URL>` | `goget --url <URL> --timestamping` | Time-stamp check |
|
||||
| `wget --continue <URL>` | `goget --url <URL> --continue` | Continue partial |
|
||||
| `wget -nc <URL>` | `goget --url <URL> --no-clobber` | No clobber |
|
||||
| `wget -nd <URL>` | `goget --url <URL> --cut-dirs 999` | No directories |
|
||||
| `wget --no-directories <URL>` | `goget --url <URL>` | Create dirs (default in goget) |
|
||||
| `wget -E <URL>` | `goget --url <URL> --adjust-extension` | Adjust extension |
|
||||
|
||||
## Behavior Differences
|
||||
|
||||
### Redirects
|
||||
|
||||
| Tool | Default Behavior |
|
||||
|---|---|
|
||||
| **curl** | Does **not** follow redirects by default |
|
||||
| **wget** | Follows redirects by default |
|
||||
| **goget** | Follows redirects by default (use `--no-redirect` to disable) |
|
||||
|
||||
### Output Filenames
|
||||
|
||||
| Tool | Default Output |
|
||||
|---|---|
|
||||
| **curl** | stdout |
|
||||
| **wget** | Auto-detected from URL |
|
||||
| **goget** | Auto-detected from URL (use `--output -` for stdout) |
|
||||
|
||||
### Verbosity
|
||||
|
||||
| Tool | Default |
|
||||
|---|---|
|
||||
| **curl** | Silent (progress bar) |
|
||||
| **wget** | Verbose (progress + status) |
|
||||
| **goget** | Verbose (progress bar + status, use `--quiet` for silent) |
|
||||
|
||||
## Features Unique to goget
|
||||
|
||||
These features have no direct curl/wget equivalent:
|
||||
|
||||
| Feature | Flag |
|
||||
|---|---|
|
||||
| **IPv6-first connections** | Always on (disable with `--no-ipv4`) |
|
||||
| **Smart timeout** | `--timeout 0` (auto-calculates from file size) |
|
||||
| **9 protocols** | HTTP, FTP, SFTP, WebDAV, Gemini, Gopher, data:, file:// |
|
||||
| **Quantum-safe checksums** | `--checksum-algo sha3-256`, `--checksum-algo sha3-512`, `--checksum-algo blake2b` |
|
||||
| **Download queue** | `--queue`, `--queue-list`, `--queue-process` |
|
||||
| **Scheduling** | `--schedule "0 2 * * *"` |
|
||||
| **Post-download hooks** | `--on-complete "script.sh"` |
|
||||
| **JSON progress** | `--json --no-progress` |
|
||||
| **Metalink** | `--metalink`, `--metalink-file` |
|
||||
| **HSTS cache** | `--hsts-file` (RFC 6797) |
|
||||
| **Go library API** | `import "codeberg.org/petrbalvin/goget/pkg/api"` |
|
||||
| **`--create-dirs` (default true)** | Auto-creates missing parent directories of target file |
|
||||
| **SOCKS5 proxy** | `socks5://` (local DNS) and `socks5h://` (remote DNS) |
|
||||
| **Shell completion** | `--completion bash`, `--completion zsh`, `--completion fish` |
|
||||
|
||||
## Script Migration Patterns
|
||||
|
||||
### CI/CD Download Script
|
||||
|
||||
```bash
|
||||
# Before (wget)
|
||||
wget -q -O data.zip https://example.com/data.zip && unzip data.zip
|
||||
|
||||
# After (goget)
|
||||
goget --quiet --url https://example.com/data.zip --output data.zip && unzip data.zip
|
||||
```
|
||||
|
||||
### API Request
|
||||
|
||||
```bash
|
||||
# Before (curl)
|
||||
curl -s -H "Authorization: Bearer $TOKEN" -o output.json https://api.example.com/data
|
||||
|
||||
# After (goget)
|
||||
goget --quiet --header "Authorization: Bearer $TOKEN" --url https://api.example.com/data --output output.json
|
||||
```
|
||||
|
||||
### Mirror Website
|
||||
|
||||
```bash
|
||||
# Before (wget)
|
||||
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent https://example.com
|
||||
|
||||
# After (goget)
|
||||
goget --url https://example.com --mirror
|
||||
```
|
||||
|
||||
### Scheduled Download
|
||||
|
||||
```bash
|
||||
# Before (cron + wget)
|
||||
# crontab: 0 2 * * * wget -q -O /backup/daily.zip https://example.com/daily.zip
|
||||
|
||||
# After (goget with built-in scheduling)
|
||||
goget --url https://example.com/daily.zip --output /backup/daily.zip --schedule "0 2 * * *"
|
||||
```
|
||||
|
||||
### Multi-File Download with Checksums
|
||||
|
||||
```bash
|
||||
# Before (curl + sha256sum)
|
||||
curl -O https://example.com/file.zip
|
||||
curl -O https://example.com/SHA256SUMS
|
||||
sha256sum -c SHA256SUMS
|
||||
|
||||
# After (goget)
|
||||
goget --url https://example.com/file.zip --checksum-file SHA256SUMS
|
||||
```
|
||||
|
||||
## What's Missing
|
||||
|
||||
Features from curl/wget not yet available in goget:
|
||||
|
||||
| Feature | Status |
|
||||
|---|---|
|
||||
| **Bandwidth throttling per-domain** | Global rate limit only |
|
||||
| **DNS-over-HTTPS** | Not yet implemented |
|
||||
| **Short flags** (`-O`, `-o`, `-s`, `-v`) | Long flags only (`--output`, `--quiet`, `--verbose`) |
|
||||
| **MIME type detection for local files** | Not implemented |
|
||||
| **Brotli decompression** | gzip/deflate only |
|
||||
@@ -0,0 +1,243 @@
|
||||
# Protocol Support
|
||||
|
||||
goget supports 9 network protocols through a unified abstraction layer. Each protocol implements `core.Protocol` and registers with a global registry.
|
||||
|
||||
## Supported Protocols
|
||||
|
||||
| Scheme | Protocol | Resume | Parallel | Upload | Recursive | Library |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `http://` `https://` | HTTP/HTTPS | ✓ | ✓ | ✓ | ✓ | Go stdlib `net/http` |
|
||||
| `ftp://` `ftps://` | FTP/FTPS | ✓ | — | — | ✓ | Custom implementation |
|
||||
| `sftp://` | SFTP | ✓ | ✓ | — | ✓ | `golang.org/x/crypto/ssh` |
|
||||
| `file://` | Local file | — | — | — | ✓ | Go stdlib `os` |
|
||||
| `data:` | Data URI | — | — | — | — | Go stdlib |
|
||||
| `gopher://` | Gopher | — | — | — | — | Custom implementation |
|
||||
| `webdav://` `webdavs://` | WebDAV | ✓ | — | ✓ | ✓ | HTTP (via delegation) |
|
||||
|
||||
| `gemini://` | Gemini | — | — | — | — | Custom implementation |
|
||||
|
||||
## HTTP/HTTPS (`http://`, `https://`)
|
||||
|
||||
The primary protocol handler with the most features.
|
||||
|
||||
### Features
|
||||
|
||||
- **HTTP/1.1 and HTTP/2** — Automatic protocol negotiation via ALPN
|
||||
- **TLS 1.2 and 1.3** — With configurable cipher suites
|
||||
- **IPv6-first** — Prefers IPv6 addresses; falls back to IPv4 only when no IPv6 record exists
|
||||
- **Parallel chunked downloads** — Splits files over 100 MB into chunks and downloads them concurrently using `Range` requests
|
||||
- **Resume** — Supports `Range`/`Accept-Ranges` for resuming interrupted downloads
|
||||
- **Upload** — PUT and POST with `Content-Type` detection
|
||||
- **Compression** — Automatic gzip, deflate, brotli (if available) decompression
|
||||
- **Cookies** — Netscape-format cookie jar for session persistence
|
||||
- **HSTS** — RFC 6797 cache for automatic HTTPS upgrades
|
||||
- **OAuth 2.0** — Client credentials, authorization code, and refresh token flows
|
||||
- **Digest authentication** — RFC 7616
|
||||
- **Basic authentication** — RFC 7617
|
||||
- **Rate limiting** — Token bucket bandwidth capping
|
||||
- **Proxy** — HTTP CONNECT proxy support
|
||||
- **Custom DNS** — Configurable DNS servers via `--dns-servers`
|
||||
- **Interface binding** — Linux-specific `SO_BINDTODEVICE`
|
||||
- **Timing breakdown** — DNS, TCP, TLS, TTFB timing via `--trace-time`
|
||||
- **WARC output** — Web ARChive format for archival
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# TLS settings
|
||||
--insecure Skip TLS certificate verification
|
||||
--cert client.pem Client certificate (mTLS)
|
||||
--key client.key Client private key (mTLS)
|
||||
--pinned-cert SHA256 Certificate pinning
|
||||
--ssl-key-log file TLS key log for Wireshark
|
||||
|
||||
# Proxy
|
||||
--proxy http://proxy:8080
|
||||
|
||||
# Rate limiting
|
||||
--rate-limit 1MB/s
|
||||
|
||||
# Headers
|
||||
--header "X-Custom: value"
|
||||
|
||||
# Authentication
|
||||
--username user --password pass
|
||||
--auth-type basic|digest|auto
|
||||
|
||||
# Timing
|
||||
--trace-time
|
||||
```
|
||||
|
||||
## FTP/FTPS (`ftp://`, `ftps://`)
|
||||
|
||||
Custom FTP implementation supporting active and passive modes.
|
||||
|
||||
### Features
|
||||
|
||||
- **Active and passive modes** — Auto-detection; passive preferred
|
||||
- **Explicit TLS (FTPS)** — AUTH TLS command on port 21
|
||||
- **Resume** — REST command for partial transfers
|
||||
- **Directory listing** — For recursive operations
|
||||
|
||||
### Limitations
|
||||
|
||||
- No parallel downloads
|
||||
- No upload support
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# Explicit TLS (FTPS)
|
||||
--url ftps://ftp.example.com/file.zip
|
||||
|
||||
# Username and password
|
||||
--url ftp://ftp.example.com/file.zip --username user --password pass
|
||||
|
||||
# Anonymous (default)
|
||||
--url ftp://anonymous@ftp.example.com/file.zip
|
||||
```
|
||||
|
||||
## SFTP (`sftp://`)
|
||||
|
||||
SFTP over SSH protocol.
|
||||
|
||||
### Features
|
||||
|
||||
- **SSH key authentication** — Auto-detection from `~/.ssh/id_rsa`, `~/.ssh/id_ed25519`
|
||||
- **Password authentication** — Via `--password` or `.netrc`
|
||||
- **Known hosts verification** — Via `~/.ssh/known_hosts`
|
||||
- **Resume** — Partial transfer support; parallel recursive downloads available via `--recursive-parallel N` with connection pooling
|
||||
- **`known_hosts` verification** — Reads `~/.ssh/known_hosts`; does not write to it. First-connection accept via `--ssh-insecure`
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# With known hosts verification
|
||||
--url sftp://user@host:/path/to/file.zip
|
||||
|
||||
# Skip host key verification (first connection)
|
||||
--url sftp://user@host:/path/to/file.zip --ssh-insecure
|
||||
|
||||
# Custom known hosts file
|
||||
--url sftp://user@host:/path/to/file.zip --known-hosts ~/.ssh/known_hosts
|
||||
|
||||
# With password
|
||||
--url sftp://user@host:/path/to/file.zip --password "secret"
|
||||
```
|
||||
|
||||
## Local File (`file://`)
|
||||
|
||||
Reads or copies local files.
|
||||
|
||||
### Features
|
||||
|
||||
- **File copy** — Copies local files to the output path
|
||||
- **Directory listing** — For recursive operations
|
||||
- **No network required** — Works fully offline
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Copy a local file
|
||||
goget --url file:///home/user/document.pdf
|
||||
|
||||
# Recursive directory copy
|
||||
goget --url file:///home/user/docs/ --recursive --output ./backup/
|
||||
```
|
||||
|
||||
## Data URI (`data:`)
|
||||
|
||||
Handles inline data URIs as defined in RFC 2397.
|
||||
|
||||
### Features
|
||||
|
||||
- **Base64 decoding** — `data:application/zip;base64,UEsDB...`
|
||||
- **Plain text** — `data:text/plain,Hello%20World`
|
||||
- **No network required** — Data is embedded in the URI
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Base64 encoded data
|
||||
goget --url "data:application/zip;base64,UEsDB..." --output decoded.zip
|
||||
|
||||
# Plain text
|
||||
goget --url "data:text/plain,Hello%20World" --output hello.txt
|
||||
```
|
||||
|
||||
## Gopher (`gopher://`)
|
||||
|
||||
Implements the Gopher protocol as specified in RFC 1436.
|
||||
|
||||
### Features
|
||||
|
||||
- **Type parsing** — Respects Gopher type codes (0 = text, 1 = directory, 9 = binary)
|
||||
- **Directory listing** — For recursive operations
|
||||
- **Text and binary downloads**
|
||||
|
||||
### Limitations
|
||||
|
||||
- No resume
|
||||
- No parallel downloads
|
||||
- No encryption (Gopher has no TLS)
|
||||
- No upload
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Download a file from a Gopher server
|
||||
goget --url gopher://gopher.example.com/9/file.zip
|
||||
|
||||
# Browse a Gopher directory
|
||||
goget --url gopher://gopher.example.com/1/
|
||||
```
|
||||
|
||||
## Gemini (`gemini://`)
|
||||
|
||||
Implements the Gemini protocol (a lightweight alternative to HTTP/HTTPS).
|
||||
|
||||
### Features
|
||||
|
||||
- **TLS** — Mandatory TLS connections with TLS 1.2 minimum
|
||||
- **Redirect following** — Respects Gemini redirect status codes (3x), up to 10 redirects
|
||||
- **Error handling** — Input prompts (1x) return an error with the prompt text; temporary (4x) and permanent (5x) failures are handled
|
||||
- **Text and binary downloads** — 2x success responses stream content directly
|
||||
|
||||
### Limitations
|
||||
|
||||
- No resume
|
||||
- No parallel downloads
|
||||
- No upload (Gemini is download-only)
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Download a file
|
||||
goget --url gemini://gemini.example.com/file.zip
|
||||
|
||||
# Browse a capsule
|
||||
goget --url gemini://gemini.example.com/
|
||||
```
|
||||
|
||||
## Protocol Resolution Flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
URL[URL input]:::accent6
|
||||
Parse["Parse URL"]:::accent7
|
||||
Normalize["Normalize scheme\n(https→http, ftps→ftp)"]:::accent7
|
||||
Registry["Look up in protocol registry"]:::accent1
|
||||
Found{"Protocol found?"}:::accent4
|
||||
Error["Unsupported protocol error"]:::accent4
|
||||
Handle["Delegate to protocol handler"]:::accent1
|
||||
|
||||
URL --> Parse --> Normalize --> Registry
|
||||
Registry --> Found
|
||||
Found -->|Yes| Handle
|
||||
Found -->|No| Error
|
||||
|
||||
classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
|
||||
classDef accent4 fill:#EF4444,stroke:#DC2626,color:#fff
|
||||
classDef accent6 fill:#6366F1,stroke:#4F46E5,color:#fff
|
||||
classDef accent7 fill:#64748B,stroke:#475569,color:#fff
|
||||
```
|
||||
@@ -0,0 +1,256 @@
|
||||
# Recursive Downloading & Website Mirroring
|
||||
|
||||
goget can recursively download linked pages and mirror entire websites for offline viewing. This document explains the two modes, their configuration, and the underlying architecture.
|
||||
|
||||
## Quick Comparison
|
||||
|
||||
| Feature | Recursive (`--recursive`) | Mirror (`--mirror`) |
|
||||
|---|---|---|
|
||||
| Depth limit | By default (`--max-depth N`) | Infinite |
|
||||
| Link conversion | Optional (`--convert-links`) | Automatic |
|
||||
| Page requisites | Manual (`--page-requisites`) | Automatic |
|
||||
| robots.txt | Respected by default | Respected by default |
|
||||
| Use case | Download a subtree of a site | Full offline archive |
|
||||
|
||||
## Recursive Mode
|
||||
|
||||
Recursive mode follows links from HTML pages and downloads linked resources.
|
||||
|
||||
```bash
|
||||
# Basic recursive download, depth 3
|
||||
goget --url https://example.com/docs/ --recursive --max-depth 3
|
||||
|
||||
# Only PDF files
|
||||
goget --recursive --accept "*.pdf" --url https://example.com/docs/
|
||||
|
||||
# Only images and HTML
|
||||
goget --recursive --accept "image/*,text/html" --url https://example.com/
|
||||
```
|
||||
|
||||
### Link Filtering
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
HTML[Parse HTML page]:::accent1
|
||||
Extract["Extract all links\n(a, img, link, script)"]:::accent1
|
||||
FilterByDomain{"Domain match?"}:::accent7
|
||||
FilterByPattern{"Accept pattern\nmatch?"}:::accent7
|
||||
FilterByDepth{"Depth ≤ max?"}:::accent7
|
||||
FilterByParent{"No-parent\ncheck?"}:::accent7
|
||||
ExcludeFilter{"Not in exclude\nlist?"}:::accent7
|
||||
Enqueue["Add to download queue"]:::accent1
|
||||
Skip["Skip"]:::accent4
|
||||
|
||||
HTML --> Extract --> FilterByDomain
|
||||
FilterByDomain -->|Yes| FilterByPattern
|
||||
FilterByDomain -->|No, --span-hosts| FilterByPattern
|
||||
FilterByDomain -->|No| Skip
|
||||
FilterByPattern -->|Yes| FilterByDepth
|
||||
FilterByPattern -->|No| Skip
|
||||
FilterByDepth -->|Yes| FilterByParent
|
||||
FilterByDepth -->|No| Skip
|
||||
FilterByParent -->|Pass| ExcludeFilter
|
||||
FilterByParent -->|Fail| Skip
|
||||
ExcludeFilter -->|Pass| Enqueue
|
||||
ExcludeFilter -->|Fail| Skip
|
||||
|
||||
classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
|
||||
classDef accent4 fill:#EF4444,stroke:#DC2626,color:#fff
|
||||
classDef accent7 fill:#64748B,stroke:#475569,color:#fff
|
||||
```
|
||||
|
||||
### Accept Patterns
|
||||
|
||||
The `--accept` flag supports two types:
|
||||
|
||||
| Type | Example | Matches |
|
||||
|---|---|---|
|
||||
| **Glob** | `*.pdf` | Filenames ending in `.pdf` |
|
||||
| **Glob** | `*.html,*.css` | Multiple patterns (comma-separated) |
|
||||
| **MIME** | `text/html` | Exact MIME type |
|
||||
| **MIME** | `image/*` | Any MIME in the `image/` category |
|
||||
|
||||
### Domain Control
|
||||
|
||||
| Flag | Effect |
|
||||
|---|---|
|
||||
| (default) | Only follow links within the starting domain |
|
||||
| `--span-hosts` | Follow links to any domain |
|
||||
| `--domains a.com,b.com` | Restrict to specific domains |
|
||||
| `--follow-external` | Follow external links (aliases for `--span-hosts`) |
|
||||
| `--no-parent` | Don't ascendant above the starting URL path |
|
||||
| `--recursive-parallel` | `<N>` | Number of concurrent file downloads in recursive mode across all protocols (`0` = sequential). HTTP, WebDAV, FTP, and SFTP each get a worker pool bounded by a semaphore; subdirectory recursion propagates the limit. Connection pooling (FTP, SFTP) opens N independent control connections |
|
||||
|
||||
### Request Throttling
|
||||
|
||||
```bash
|
||||
# Fixed 2-second delay between requests
|
||||
goget --recursive --wait 2s --url https://example.com/docs/
|
||||
|
||||
# Randomize delay (0.5x – 1.5x of --wait)
|
||||
goget --recursive --wait 2s --random-wait --url https://example.com/docs/
|
||||
```
|
||||
|
||||
### Page Requisites
|
||||
|
||||
`--page-requisites` downloads CSS, JavaScript, and images needed to render each HTML page:
|
||||
|
||||
```bash
|
||||
goget --recursive --page-requisites --url https://example.com/page.html
|
||||
```
|
||||
|
||||
This parses `<link>`, `<script>`, `<img>`, `<source>`, and `<video>` tags and downloads their `src`/`href` targets.
|
||||
|
||||
## Mirror Mode
|
||||
|
||||
| `--dry-run` | — | List all files that would be downloaded in recursive/mirror mode without saving to disk |
|
||||
|
||||
`--mirror` is a shorthand for `--recursive --convert-links --page-requisites --infinite-depth`:
|
||||
|
||||
```bash
|
||||
# Full site mirror
|
||||
goget --url https://example.com --mirror --output ./mirror
|
||||
|
||||
# With link conversion for offline viewing
|
||||
goget --url https://example.com --mirror --convert-links --output ./mirror
|
||||
```
|
||||
|
||||
### Link Conversion
|
||||
|
||||
When `--convert-links` is enabled, HTML links are rewritten for local offline viewing:
|
||||
|
||||
```
|
||||
Before: <a href="https://example.com/about/">About</a>
|
||||
After: <a href="./about/index.html">About</a>
|
||||
|
||||
Before: <img src="https://example.com/img/logo.png">
|
||||
After: <img src="./img/logo.png">
|
||||
```
|
||||
|
||||
The link rewriter handles:
|
||||
- Absolute URLs → relative paths
|
||||
- Protocol-relative URLs (`//example.com/...`)
|
||||
- Root-relative URLs (`/about/`)
|
||||
- CSS `url()` references
|
||||
|
||||
### Robots.txt
|
||||
|
||||
goget respects `robots.txt` by default. Use `--no-robots` to ignore it:
|
||||
|
||||
```bash
|
||||
goget --mirror --no-robots --url https://example.com
|
||||
```
|
||||
|
||||
### Asset Control
|
||||
|
||||
```bash
|
||||
# Mirror but skip CSS/JS/images
|
||||
goget --mirror --no-mirror-assets --url https://example.com
|
||||
|
||||
# Only mirror specific file types
|
||||
goget --mirror --accept "*.html,*.jpg" --url https://example.com
|
||||
```
|
||||
|
||||
## Output Structure
|
||||
|
||||
Mirrored sites preserve the URL path structure:
|
||||
|
||||
```
|
||||
mirror/
|
||||
├── index.html
|
||||
├── about/
|
||||
│ └── index.html
|
||||
├── blog/
|
||||
│ ├── index.html
|
||||
│ └── post-1.html
|
||||
├── css/
|
||||
│ └── style.css
|
||||
└── img/
|
||||
└── logo.png
|
||||
```
|
||||
|
||||
Use `--cut-dirs N` to strip directory components:
|
||||
|
||||
```bash
|
||||
# Original: https://example.com/pub/docs/file.html
|
||||
# Without cut: ./mirror/pub/docs/file.html
|
||||
# With --cut-dirs 2: ./mirror/doc/file.html
|
||||
goget --mirror --cut-dirs 2 --url https://example.com/pub/docs/
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
### Crawler (`internal/recursive`)
|
||||
|
||||
```go
|
||||
type CrawlerConfig struct {
|
||||
MaxDepth int
|
||||
FollowExternal bool
|
||||
ExcludePatterns []string
|
||||
IncludePatterns []string
|
||||
AcceptPatterns []string
|
||||
NoParent bool
|
||||
PageRequisites bool
|
||||
ConvertLinks bool
|
||||
OutputDir string
|
||||
Delay time.Duration
|
||||
RandomWait bool
|
||||
MaxWorkers int // concurrent download workers (0 = sequential)
|
||||
}
|
||||
```
|
||||
|
||||
The crawler:
|
||||
1. Downloads an HTML page
|
||||
2. Parses it with `golang.org/x/net/html`
|
||||
3. Extracts all links (`a[href]`, `img[src]`, `link[href]`, `script[src]`)
|
||||
4. Filters against accept/reject patterns, domain rules, and depth limits
|
||||
5. Enqueues matching URLs for download
|
||||
6. Applies configurable delay between requests
|
||||
|
||||
### Mirror (`internal/mirror`)
|
||||
|
||||
The mirror wraps the crawler with additional configuration:
|
||||
- Infinite depth (or configurable via `MirrorConfig.MaxDepth`)
|
||||
- Automatic page requisites
|
||||
- Automatic link conversion
|
||||
- robots.txt compliance
|
||||
|
||||
### Link Rewriting (`internal/linkrewrite`)
|
||||
|
||||
After the download completes, links are rewritten in-place:
|
||||
1. Read each HTML file
|
||||
2. Parse with `golang.org/x/net/html`
|
||||
3. Identify all `href` and `src` attributes
|
||||
4. Convert absolute URLs to relative paths
|
||||
5. Write back the modified HTML (backup `.orig` if `--backup-converted`)
|
||||
|
||||
## Timestamping
|
||||
|
||||
`--timestamping` downloads a file only if the server copy is newer than the local copy:
|
||||
|
||||
```bash
|
||||
goget --timestamping --url https://example.com/file.zip
|
||||
```
|
||||
|
||||
goget compares the `Last-Modified` header against the local file's modification time.
|
||||
|
||||
## WARC Archiving
|
||||
|
||||
For legal and archival compliance, output can be written in WARC format:
|
||||
|
||||
```bash
|
||||
goget --warc-file archive.warc --url https://example.com/page.html
|
||||
```
|
||||
|
||||
Each downloaded resource gets a separate WARC record with:
|
||||
- Request metadata (URL, timestamp, headers)
|
||||
- Response metadata (status code, content type, headers)
|
||||
- Raw response body
|
||||
|
||||
## Performance Tips
|
||||
|
||||
- **Limit depth** — `--max-depth 3` prevents runaway recursion on large sites
|
||||
- **Filter aggressively** — Use `--accept "*.pdf,*.html"` and `--reject "*.zip"` to narrow scope
|
||||
- **Use delay** — `--wait 500ms` prevents server overload and IP blocks
|
||||
- **Limit domains** — `--domains example.com` prevents crawling external CDNs
|
||||
- **Skip assets** — `--no-mirror-assets` for text-only archives
|
||||
@@ -0,0 +1,258 @@
|
||||
# Security
|
||||
|
||||
goget implements multiple security features to protect both the client and server during file transfers. This document covers TLS configuration, certificate handling, checksum verification, PGP support, HSTS, and SSRF protection.
|
||||
|
||||
## TLS / HTTPS
|
||||
|
||||
### Default Security
|
||||
|
||||
- **TLS 1.2 minimum** — TLS 1.0 and 1.1 are disabled
|
||||
- **Certificate verification** — Enabled by default against system CA bundle
|
||||
- **HTTP/2 support** — Automatic ALPN negotiation
|
||||
|
||||
### Certificate Pinning
|
||||
|
||||
Pin a specific certificate by its SHA-256 hash:
|
||||
|
||||
```bash
|
||||
goget --pinned-cert "a1b2c3d4e5f6..." --url https://example.com
|
||||
```
|
||||
|
||||
If the server certificate's SHA-256 doesn't match, the connection is aborted.
|
||||
|
||||
### Client Certificates (mTLS)
|
||||
|
||||
Authenticate the client to the server with mutual TLS:
|
||||
|
||||
```bash
|
||||
goget --cert client.pem --key client.key --url https://mtls.example.com
|
||||
```
|
||||
|
||||
### Custom CA Bundle
|
||||
|
||||
```bash
|
||||
goget --cacert custom-ca.pem --url https://internal.example.com
|
||||
```
|
||||
|
||||
### Insecure Mode
|
||||
|
||||
Disable all TLS verification (⚠ **for testing only**):
|
||||
|
||||
```bash
|
||||
goget --insecure --url https://self-signed.local
|
||||
```
|
||||
|
||||
### SSLKEYLOGFILE
|
||||
|
||||
Log TLS session keys for Wireshark debugging:
|
||||
|
||||
```bash
|
||||
goget --ssl-key-log tls.keys --url https://example.com
|
||||
# Then: Wireshark → Preferences → TLS → (Pre-)Master-Secret log filename → tls.keys
|
||||
```
|
||||
|
||||
## HSTS (HTTP Strict Transport Security)
|
||||
|
||||
Per RFC 6797, goget maintains an HSTS cache that automatically upgrades HTTP connections to HTTPS for known hosts.
|
||||
|
||||
### How It Works
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client as goget
|
||||
participant Server as Server
|
||||
|
||||
Client->>Server: GET http://example.com
|
||||
Server-->>Client: 200 OK + Strict-Transport-Security: max-age=31536000
|
||||
Client->>Client: Store: example.com → 365 days
|
||||
Client->>Server: GET http://example.com/page (later)
|
||||
Client->>Client: HSTS cache hit → upgrade to HTTPS
|
||||
Client->>Server: GET https://example.com/page
|
||||
```
|
||||
|
||||
### Cache Location
|
||||
|
||||
```
|
||||
~/.config/goget/hsts
|
||||
```
|
||||
|
||||
Can be overridden with `GOGET_HSTS` environment variable or `--hsts-file`.
|
||||
|
||||
### Cache Format
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"host": "example.com",
|
||||
"include_subdomains": true,
|
||||
"expires": "2027-06-01T00:00:00Z"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Expired entries are automatically pruned on load.
|
||||
|
||||
## Checksum Verification
|
||||
|
||||
Verify file integrity after download:
|
||||
|
||||
### Direct Hash
|
||||
|
||||
```bash
|
||||
goget --checksum "a1b2c3d4e5f6..." --checksum-algo sha256 --url https://example.com/file.zip
|
||||
```
|
||||
|
||||
Supported algorithms:
|
||||
|
||||
| Algorithm | Flag Value | Hash Length |
|
||||
|---|---|---|
|
||||
| SHA-256 | `sha256` | 64 hex chars |
|
||||
| SHA-512 | `sha512` | 128 hex chars |
|
||||
| SHA3-256 | `sha3-256` | 64 hex chars |
|
||||
| SHA3-512 | `sha3-512` | 128 hex chars |
|
||||
| BLAKE2b | `blake2b` | 128 hex chars |
|
||||
| MD5 | `md5` | 32 hex chars |
|
||||
|
||||
### Checksum File
|
||||
|
||||
Supports standard `SHA256SUMS` format:
|
||||
|
||||
```bash
|
||||
# SHA256SUMS file:
|
||||
# a1b2c3d4... file.zip
|
||||
# e5f6a7b8... *file.tar.gz
|
||||
|
||||
goget --checksum-file SHA256SUMS --url https://example.com/file.zip
|
||||
```
|
||||
|
||||
Both plain (`file.zip`) and binary-mode (`*file.tar.gz`) formats are supported.
|
||||
|
||||
### Error Handling
|
||||
|
||||
On mismatch:
|
||||
```
|
||||
Checksum mismatch
|
||||
Expected: a1b2c3d4...
|
||||
Actual: e5f6a7b8...
|
||||
```
|
||||
|
||||
The error is a typed `core.GogetError` with `Type: "CHECKSUM"` and `Details["expected"]` / `Details["actual"]`.
|
||||
|
||||
## PGP / GPG
|
||||
|
||||
### Signature Verification
|
||||
|
||||
Verify PGP detached signatures:
|
||||
|
||||
```bash
|
||||
# With explicit signature file
|
||||
goget --pgp-verify --pgp-sig file.sig --pgp-key public.key --url https://example.com/file
|
||||
|
||||
# Auto-detect signature file (tries: file.asc, file.sig, file.gpg)
|
||||
goget --pgp-verify --pgp-key public.key --url https://example.com/file
|
||||
```
|
||||
|
||||
### Decryption
|
||||
|
||||
Decrypt PGP-encrypted files:
|
||||
|
||||
```bash
|
||||
# Decrypt with key and passphrase
|
||||
goget --pgp-decrypt --pgp-key private.key --pgp-passphrase "secret" --url https://example.com/file.gpg
|
||||
|
||||
# Read passphrase from file (safer)
|
||||
goget --pgp-decrypt --pgp-key private.key --pgp-passphrase-file pass.txt --url https://example.com/file.gpg
|
||||
```
|
||||
|
||||
The decrypted file replaces the encrypted one:
|
||||
- `file.gpg` → `file`
|
||||
- `file.pgp` → `file`
|
||||
- `file.asc` → `file`
|
||||
- Other → `filename.decrypted`
|
||||
|
||||
> **Note:** PGP functionality uses `golang.org/x/crypto/openpgp`, which is frozen. A replacement will be provided in a future release.
|
||||
|
||||
## SSRF Protection
|
||||
|
||||
By default, goget blocks connections to private and internal IP ranges:
|
||||
|
||||
| Range | CIDR | Description |
|
||||
|---|---|---|
|
||||
| `10.0.0.0/8` | Private | RFC 1918 |
|
||||
| `172.16.0.0/12` | Private | RFC 1918 |
|
||||
| `127.0.0.0/8` | Loopback | Internal |
|
||||
| `169.254.0.0/16` | Link-local | Auto-configuration |
|
||||
| `::1/128` | IPv6 loopback | Internal |
|
||||
|
||||
### IPv4-Mapped IPv6 Normalization
|
||||
|
||||
DNS-controlled IPv4-mapped IPv6 addresses (e.g. `::ffff:10.0.0.1`) are now normalized to plain IPv4 before the private-IP check. This prevents an attacker from reaching internal hosts through DNS responses that embed IPv4 addresses in IPv6 form. Covered by a 17-case unit test in the transport layer.
|
||||
|
||||
Override with `--allow-private`:
|
||||
|
||||
```bash
|
||||
goget --allow-private --url https://internal.example.com
|
||||
```
|
||||
|
||||
## DNS Security
|
||||
|
||||
### Custom Resolvers
|
||||
|
||||
Bypass system DNS for specific queries:
|
||||
|
||||
```bash
|
||||
goget --dns-servers 1.1.1.1,8.8.8.8 --url https://example.com
|
||||
```
|
||||
|
||||
### IPv6-First
|
||||
|
||||
The transport dialer prefers IPv6 and only falls back to IPv4 when no IPv6 address is available. This reduces NAT-related attack surface:
|
||||
|
||||
```bash
|
||||
# IPv6 only (abort if no IPv6)
|
||||
goget --no-ipv4 --url https://example.com
|
||||
```
|
||||
|
||||
## Credential Safety
|
||||
|
||||
### Safe Error Logging
|
||||
|
||||
When a URL contains user credentials (`https://user:pass@example.com`), `core.SafeURL()` strips them from error messages and logs:
|
||||
|
||||
```go
|
||||
// Original: https://admin:secret@api.example.com/data
|
||||
// Logged: https://api.example.com/data
|
||||
```
|
||||
|
||||
### OAuth Error Sanitization
|
||||
|
||||
Non-2xx OAuth responses are sanitized to extract only the structured `error` / `error_description` fields (RFC 6749 §5.2). Providers that echo `client_secret` or other secrets in their error response bodies can no longer leak them into goget logs or error displays.
|
||||
|
||||
### Config Sanitization
|
||||
|
||||
`config.Save()` strips sensitive fields before writing to disk:
|
||||
- `auth.password`
|
||||
- `auth.oauth.client_secret`
|
||||
- `auth.oauth.access_token`
|
||||
- `auth.oauth.refresh_token`
|
||||
|
||||
### File Permission Hardening
|
||||
|
||||
Configuration files (`config.toml`), resume metadata sidecars (`.goget.meta`), and atomic temp files are written with mode `0600` (owner-only), so OAuth tokens, source URLs, and downloaded payloads are never world-readable on multi-user systems.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
- `GOGET_CONFIG` — alternative config file path
|
||||
- `GOGET_PASSWORD` — password override (no file storage)
|
||||
- `GOGET_HSTS` — alternative HSTS cache path
|
||||
- `NO_COLOR` — disable color output
|
||||
|
||||
## Timeouts & Limits
|
||||
|
||||
| Feature | Default | Description |
|
||||
|---|---|---|
|
||||
| Connection timeout | 30s (auto) | Smart calculation based on file size |
|
||||
| Max total time | — (`--max-time`) | Abort if exceeded |
|
||||
| Max file size | — (`--max-filesize`) | Abort if server response exceeds limit |
|
||||
| Max retries | 3 | Exponential backoff with 30s cap |
|
||||
| Retry delay | 1s initial, 2× multiplier | Configurable via `--retry-delay` |
|
||||
Reference in New Issue
Block a user