244 lines
7.1 KiB
Markdown
244 lines
7.1 KiB
Markdown
# Protocol Support
|
|
|
|
goget supports 9 network protocols through a unified abstraction layer. Each protocol implements `core.Protocol` and registers with a global registry.
|
|
|
|
## Supported Protocols
|
|
|
|
| Scheme | Protocol | Resume | Parallel | Upload | Recursive | Library |
|
|
|---|---|---|---|---|---|---|
|
|
| `http://` `https://` | HTTP/HTTPS | ✓ | ✓ | ✓ | ✓ | Go stdlib `net/http` |
|
|
| `ftp://` `ftps://` | FTP/FTPS | ✓ | — | — | ✓ | Custom implementation |
|
|
| `sftp://` | SFTP | ✓ | ✓ | — | ✓ | `golang.org/x/crypto/ssh` |
|
|
| `file://` | Local file | — | — | — | ✓ | Go stdlib `os` |
|
|
| `data:` | Data URI | — | — | — | — | Go stdlib |
|
|
| `gopher://` | Gopher | — | — | — | — | Custom implementation |
|
|
| `webdav://` `webdavs://` | WebDAV | ✓ | — | ✓ | ✓ | HTTP (via delegation) |
|
|
|
|
| `gemini://` | Gemini | — | — | — | — | Custom implementation |
|
|
|
|
## HTTP/HTTPS (`http://`, `https://`)
|
|
|
|
The primary protocol handler with the most features.
|
|
|
|
### Features
|
|
|
|
- **HTTP/1.1 and HTTP/2** — Automatic protocol negotiation via ALPN
|
|
- **TLS 1.2 and 1.3** — With configurable cipher suites
|
|
- **IPv6-first** — Prefers IPv6 addresses; falls back to IPv4 only when no IPv6 record exists
|
|
- **Parallel chunked downloads** — Splits files over 100 MB into chunks and downloads them concurrently using `Range` requests
|
|
- **Resume** — Supports `Range`/`Accept-Ranges` for resuming interrupted downloads
|
|
- **Upload** — PUT and POST with `Content-Type` detection
|
|
- **Compression** — Automatic gzip, deflate, brotli (if available) decompression
|
|
- **Cookies** — Netscape-format cookie jar for session persistence
|
|
- **HSTS** — RFC 6797 cache for automatic HTTPS upgrades
|
|
- **OAuth 2.0** — Client credentials, authorization code, and refresh token flows
|
|
- **Digest authentication** — RFC 7616
|
|
- **Basic authentication** — RFC 7617
|
|
- **Rate limiting** — Token bucket bandwidth capping
|
|
- **Proxy** — HTTP CONNECT proxy support
|
|
- **Custom DNS** — Configurable DNS servers via `--dns-servers`
|
|
- **Interface binding** — Linux-specific `SO_BINDTODEVICE`
|
|
- **Timing breakdown** — DNS, TCP, TLS, TTFB timing via `--trace-time`
|
|
- **WARC output** — Web ARChive format for archival
|
|
|
|
### Configuration
|
|
|
|
```bash
|
|
# TLS settings
|
|
--insecure Skip TLS certificate verification
|
|
--cert client.pem Client certificate (mTLS)
|
|
--key client.key Client private key (mTLS)
|
|
--pinned-cert SHA256 Certificate pinning
|
|
--ssl-key-log file TLS key log for Wireshark
|
|
|
|
# Proxy
|
|
--proxy http://proxy:8080
|
|
|
|
# Rate limiting
|
|
--rate-limit 1MB/s
|
|
|
|
# Headers
|
|
--header "X-Custom: value"
|
|
|
|
# Authentication
|
|
--username user --password pass
|
|
--auth-type basic|digest|auto
|
|
|
|
# Timing
|
|
--trace-time
|
|
```
|
|
|
|
## FTP/FTPS (`ftp://`, `ftps://`)
|
|
|
|
Custom FTP implementation supporting active and passive modes.
|
|
|
|
### Features
|
|
|
|
- **Active and passive modes** — Auto-detection; passive preferred
|
|
- **Explicit TLS (FTPS)** — AUTH TLS command on port 21
|
|
- **Resume** — REST command for partial transfers
|
|
- **Directory listing** — For recursive operations
|
|
|
|
### Limitations
|
|
|
|
- No parallel downloads
|
|
- No upload support
|
|
|
|
### Configuration
|
|
|
|
```bash
|
|
# Explicit TLS (FTPS)
|
|
--url ftps://ftp.example.com/file.zip
|
|
|
|
# Username and password
|
|
--url ftp://ftp.example.com/file.zip --username user --password pass
|
|
|
|
# Anonymous (default)
|
|
--url ftp://anonymous@ftp.example.com/file.zip
|
|
```
|
|
|
|
## SFTP (`sftp://`)
|
|
|
|
SFTP over SSH protocol.
|
|
|
|
### Features
|
|
|
|
- **SSH key authentication** — Auto-detection from `~/.ssh/id_rsa`, `~/.ssh/id_ed25519`
|
|
- **Password authentication** — Via `--password` or `.netrc`
|
|
- **Known hosts verification** — Via `~/.ssh/known_hosts`
|
|
- **Resume** — Partial transfer support; parallel recursive downloads available via `--recursive-parallel N` with connection pooling
|
|
- **`known_hosts` verification** — Reads `~/.ssh/known_hosts`; does not write to it. First-connection accept via `--ssh-insecure`
|
|
|
|
### Configuration
|
|
|
|
```bash
|
|
# With known hosts verification
|
|
--url sftp://user@host:/path/to/file.zip
|
|
|
|
# Skip host key verification (first connection)
|
|
--url sftp://user@host:/path/to/file.zip --ssh-insecure
|
|
|
|
# Custom known hosts file
|
|
--url sftp://user@host:/path/to/file.zip --known-hosts ~/.ssh/known_hosts
|
|
|
|
# With password
|
|
--url sftp://user@host:/path/to/file.zip --password "secret"
|
|
```
|
|
|
|
## Local File (`file://`)
|
|
|
|
Reads or copies local files.
|
|
|
|
### Features
|
|
|
|
- **File copy** — Copies local files to the output path
|
|
- **Directory listing** — For recursive operations
|
|
- **No network required** — Works fully offline
|
|
|
|
### Usage
|
|
|
|
```bash
|
|
# Copy a local file
|
|
goget --url file:///home/user/document.pdf
|
|
|
|
# Recursive directory copy
|
|
goget --url file:///home/user/docs/ --recursive --output ./backup/
|
|
```
|
|
|
|
## Data URI (`data:`)
|
|
|
|
Handles inline data URIs as defined in RFC 2397.
|
|
|
|
### Features
|
|
|
|
- **Base64 decoding** — `data:application/zip;base64,UEsDB...`
|
|
- **Plain text** — `data:text/plain,Hello%20World`
|
|
- **No network required** — Data is embedded in the URI
|
|
|
|
### Usage
|
|
|
|
```bash
|
|
# Base64 encoded data
|
|
goget --url "data:application/zip;base64,UEsDB..." --output decoded.zip
|
|
|
|
# Plain text
|
|
goget --url "data:text/plain,Hello%20World" --output hello.txt
|
|
```
|
|
|
|
## Gopher (`gopher://`)
|
|
|
|
Implements the Gopher protocol as specified in RFC 1436.
|
|
|
|
### Features
|
|
|
|
- **Type parsing** — Respects Gopher type codes (0 = text, 1 = directory, 9 = binary)
|
|
- **Directory listing** — For recursive operations
|
|
- **Text and binary downloads**
|
|
|
|
### Limitations
|
|
|
|
- No resume
|
|
- No parallel downloads
|
|
- No encryption (Gopher has no TLS)
|
|
- No upload
|
|
|
|
### Usage
|
|
|
|
```bash
|
|
# Download a file from a Gopher server
|
|
goget --url gopher://gopher.example.com/9/file.zip
|
|
|
|
# Browse a Gopher directory
|
|
goget --url gopher://gopher.example.com/1/
|
|
```
|
|
|
|
## Gemini (`gemini://`)
|
|
|
|
Implements the Gemini protocol (a lightweight alternative to HTTP/HTTPS).
|
|
|
|
### Features
|
|
|
|
- **TLS** — Mandatory TLS connections with TLS 1.2 minimum
|
|
- **Redirect following** — Respects Gemini redirect status codes (3x), up to 10 redirects
|
|
- **Error handling** — Input prompts (1x) return an error with the prompt text; temporary (4x) and permanent (5x) failures are handled
|
|
- **Text and binary downloads** — 2x success responses stream content directly
|
|
|
|
### Limitations
|
|
|
|
- No resume
|
|
- No parallel downloads
|
|
- No upload (Gemini is download-only)
|
|
|
|
### Usage
|
|
|
|
```bash
|
|
# Download a file
|
|
goget --url gemini://gemini.example.com/file.zip
|
|
|
|
# Browse a capsule
|
|
goget --url gemini://gemini.example.com/
|
|
```
|
|
|
|
## Protocol Resolution Flow
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
URL[URL input]:::accent6
|
|
Parse["Parse URL"]:::accent7
|
|
Normalize["Normalize scheme\n(https→http, ftps→ftp)"]:::accent7
|
|
Registry["Look up in protocol registry"]:::accent1
|
|
Found{"Protocol found?"}:::accent4
|
|
Error["Unsupported protocol error"]:::accent4
|
|
Handle["Delegate to protocol handler"]:::accent1
|
|
|
|
URL --> Parse --> Normalize --> Registry
|
|
Registry --> Found
|
|
Found -->|Yes| Handle
|
|
Found -->|No| Error
|
|
|
|
classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
|
|
classDef accent4 fill:#EF4444,stroke:#DC2626,color:#fff
|
|
classDef accent6 fill:#6366F1,stroke:#4F46E5,color:#fff
|
|
classDef accent7 fill:#64748B,stroke:#475569,color:#fff
|
|
```
|