feat: initial goget release — modern IPv6-first download utility
This commit is contained in:
@@ -0,0 +1,243 @@
|
||||
# Protocol Support
|
||||
|
||||
goget supports 9 network protocols through a unified abstraction layer. Each protocol implements `core.Protocol` and registers with a global registry.
|
||||
|
||||
## Supported Protocols
|
||||
|
||||
| Scheme | Protocol | Resume | Parallel | Upload | Recursive | Library |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `http://` `https://` | HTTP/HTTPS | ✓ | ✓ | ✓ | ✓ | Go stdlib `net/http` |
|
||||
| `ftp://` `ftps://` | FTP/FTPS | ✓ | — | — | ✓ | Custom implementation |
|
||||
| `sftp://` | SFTP | ✓ | ✓ | — | ✓ | `golang.org/x/crypto/ssh` |
|
||||
| `file://` | Local file | — | — | — | ✓ | Go stdlib `os` |
|
||||
| `data:` | Data URI | — | — | — | — | Go stdlib |
|
||||
| `gopher://` | Gopher | — | — | — | — | Custom implementation |
|
||||
| `webdav://` `webdavs://` | WebDAV | ✓ | — | ✓ | ✓ | HTTP (via delegation) |
|
||||
|
||||
| `gemini://` | Gemini | — | — | — | — | Custom implementation |
|
||||
|
||||
## HTTP/HTTPS (`http://`, `https://`)
|
||||
|
||||
The primary protocol handler with the most features.
|
||||
|
||||
### Features
|
||||
|
||||
- **HTTP/1.1 and HTTP/2** — Automatic protocol negotiation via ALPN
|
||||
- **TLS 1.2 and 1.3** — With configurable cipher suites
|
||||
- **IPv6-first** — Prefers IPv6 addresses; falls back to IPv4 only when no IPv6 record exists
|
||||
- **Parallel chunked downloads** — Splits files over 100 MB into chunks and downloads them concurrently using `Range` requests
|
||||
- **Resume** — Supports `Range`/`Accept-Ranges` for resuming interrupted downloads
|
||||
- **Upload** — PUT and POST with `Content-Type` detection
|
||||
- **Compression** — Automatic gzip, deflate, brotli (if available) decompression
|
||||
- **Cookies** — Netscape-format cookie jar for session persistence
|
||||
- **HSTS** — RFC 6797 cache for automatic HTTPS upgrades
|
||||
- **OAuth 2.0** — Client credentials, authorization code, and refresh token flows
|
||||
- **Digest authentication** — RFC 7616
|
||||
- **Basic authentication** — RFC 7617
|
||||
- **Rate limiting** — Token bucket bandwidth capping
|
||||
- **Proxy** — HTTP CONNECT proxy support
|
||||
- **Custom DNS** — Configurable DNS servers via `--dns-servers`
|
||||
- **Interface binding** — Linux-specific `SO_BINDTODEVICE`
|
||||
- **Timing breakdown** — DNS, TCP, TLS, TTFB timing via `--trace-time`
|
||||
- **WARC output** — Web ARChive format for archival
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# TLS settings
|
||||
--insecure Skip TLS certificate verification
|
||||
--cert client.pem Client certificate (mTLS)
|
||||
--key client.key Client private key (mTLS)
|
||||
--pinned-cert SHA256 Certificate pinning
|
||||
--ssl-key-log file TLS key log for Wireshark
|
||||
|
||||
# Proxy
|
||||
--proxy http://proxy:8080
|
||||
|
||||
# Rate limiting
|
||||
--rate-limit 1MB/s
|
||||
|
||||
# Headers
|
||||
--header "X-Custom: value"
|
||||
|
||||
# Authentication
|
||||
--username user --password pass
|
||||
--auth-type basic|digest|auto
|
||||
|
||||
# Timing
|
||||
--trace-time
|
||||
```
|
||||
|
||||
## FTP/FTPS (`ftp://`, `ftps://`)
|
||||
|
||||
Custom FTP implementation supporting active and passive modes.
|
||||
|
||||
### Features
|
||||
|
||||
- **Active and passive modes** — Auto-detection; passive preferred
|
||||
- **Explicit TLS (FTPS)** — AUTH TLS command on port 21
|
||||
- **Resume** — REST command for partial transfers
|
||||
- **Directory listing** — For recursive operations
|
||||
|
||||
### Limitations
|
||||
|
||||
- No parallel downloads
|
||||
- No upload support
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# Explicit TLS (FTPS)
|
||||
--url ftps://ftp.example.com/file.zip
|
||||
|
||||
# Username and password
|
||||
--url ftp://ftp.example.com/file.zip --username user --password pass
|
||||
|
||||
# Anonymous (default)
|
||||
--url ftp://anonymous@ftp.example.com/file.zip
|
||||
```
|
||||
|
||||
## SFTP (`sftp://`)
|
||||
|
||||
SFTP over SSH protocol.
|
||||
|
||||
### Features
|
||||
|
||||
- **SSH key authentication** — Auto-detection from `~/.ssh/id_rsa`, `~/.ssh/id_ed25519`
|
||||
- **Password authentication** — Via `--password` or `.netrc`
|
||||
- **Known hosts verification** — Via `~/.ssh/known_hosts`
|
||||
- **Resume** — Partial transfer support; parallel recursive downloads available via `--recursive-parallel N` with connection pooling
|
||||
- **`known_hosts` verification** — Reads `~/.ssh/known_hosts`; does not write to it. First-connection accept via `--ssh-insecure`
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# With known hosts verification
|
||||
--url sftp://user@host:/path/to/file.zip
|
||||
|
||||
# Skip host key verification (first connection)
|
||||
--url sftp://user@host:/path/to/file.zip --ssh-insecure
|
||||
|
||||
# Custom known hosts file
|
||||
--url sftp://user@host:/path/to/file.zip --known-hosts ~/.ssh/known_hosts
|
||||
|
||||
# With password
|
||||
--url sftp://user@host:/path/to/file.zip --password "secret"
|
||||
```
|
||||
|
||||
## Local File (`file://`)
|
||||
|
||||
Reads or copies local files.
|
||||
|
||||
### Features
|
||||
|
||||
- **File copy** — Copies local files to the output path
|
||||
- **Directory listing** — For recursive operations
|
||||
- **No network required** — Works fully offline
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Copy a local file
|
||||
goget --url file:///home/user/document.pdf
|
||||
|
||||
# Recursive directory copy
|
||||
goget --url file:///home/user/docs/ --recursive --output ./backup/
|
||||
```
|
||||
|
||||
## Data URI (`data:`)
|
||||
|
||||
Handles inline data URIs as defined in RFC 2397.
|
||||
|
||||
### Features
|
||||
|
||||
- **Base64 decoding** — `data:application/zip;base64,UEsDB...`
|
||||
- **Plain text** — `data:text/plain,Hello%20World`
|
||||
- **No network required** — Data is embedded in the URI
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Base64 encoded data
|
||||
goget --url "data:application/zip;base64,UEsDB..." --output decoded.zip
|
||||
|
||||
# Plain text
|
||||
goget --url "data:text/plain,Hello%20World" --output hello.txt
|
||||
```
|
||||
|
||||
## Gopher (`gopher://`)
|
||||
|
||||
Implements the Gopher protocol as specified in RFC 1436.
|
||||
|
||||
### Features
|
||||
|
||||
- **Type parsing** — Respects Gopher type codes (0 = text, 1 = directory, 9 = binary)
|
||||
- **Directory listing** — For recursive operations
|
||||
- **Text and binary downloads**
|
||||
|
||||
### Limitations
|
||||
|
||||
- No resume
|
||||
- No parallel downloads
|
||||
- No encryption (Gopher has no TLS)
|
||||
- No upload
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Download a file from a Gopher server
|
||||
goget --url gopher://gopher.example.com/9/file.zip
|
||||
|
||||
# Browse a Gopher directory
|
||||
goget --url gopher://gopher.example.com/1/
|
||||
```
|
||||
|
||||
## Gemini (`gemini://`)
|
||||
|
||||
Implements the Gemini protocol (a lightweight alternative to HTTP/HTTPS).
|
||||
|
||||
### Features
|
||||
|
||||
- **TLS** — Mandatory TLS connections with TLS 1.2 minimum
|
||||
- **Redirect following** — Respects Gemini redirect status codes (3x), up to 10 redirects
|
||||
- **Error handling** — Input prompts (1x) return an error with the prompt text; temporary (4x) and permanent (5x) failures are handled
|
||||
- **Text and binary downloads** — 2x success responses stream content directly
|
||||
|
||||
### Limitations
|
||||
|
||||
- No resume
|
||||
- No parallel downloads
|
||||
- No upload (Gemini is download-only)
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Download a file
|
||||
goget --url gemini://gemini.example.com/file.zip
|
||||
|
||||
# Browse a capsule
|
||||
goget --url gemini://gemini.example.com/
|
||||
```
|
||||
|
||||
## Protocol Resolution Flow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
URL[URL input]:::accent6
|
||||
Parse["Parse URL"]:::accent7
|
||||
Normalize["Normalize scheme\n(https→http, ftps→ftp)"]:::accent7
|
||||
Registry["Look up in protocol registry"]:::accent1
|
||||
Found{"Protocol found?"}:::accent4
|
||||
Error["Unsupported protocol error"]:::accent4
|
||||
Handle["Delegate to protocol handler"]:::accent1
|
||||
|
||||
URL --> Parse --> Normalize --> Registry
|
||||
Registry --> Found
|
||||
Found -->|Yes| Handle
|
||||
Found -->|No| Error
|
||||
|
||||
classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
|
||||
classDef accent4 fill:#EF4444,stroke:#DC2626,color:#fff
|
||||
classDef accent6 fill:#6366F1,stroke:#4F46E5,color:#fff
|
||||
classDef accent7 fill:#64748B,stroke:#475569,color:#fff
|
||||
```
|
||||
Reference in New Issue
Block a user