goget/docs/protocols.md

Protocol Support

goget supports 9 network protocols through a unified abstraction layer. Each protocol implements core.Protocol and registers with a global registry.

Supported Protocols

Scheme	Protocol	Resume	Parallel	Upload	Recursive	Library
http:// https://	HTTP/HTTPS	✓	✓	✓	✓	Go stdlib net/http
ftp:// ftps://	FTP/FTPS	✓	—	—	✓	Custom implementation
sftp://	SFTP	✓	✓	—	✓	golang.org/x/crypto/ssh
file://	Local file	—	—	—	✓	Go stdlib os
data:	Data URI	—	—	—	—	Go stdlib
gopher://	Gopher	✓	—	—	—	Custom implementation
webdav:// webdavs://	WebDAV	✓	—	✓	✓	HTTP (via delegation)

| gemini:// | Gemini | ✓ | — | — | — | Custom implementation |

HTTP/HTTPS (http://, https://)

The primary protocol handler with the most features.

Features

• HTTP/1.1 and HTTP/2 — Automatic protocol negotiation via ALPN
• TLS 1.2 and 1.3 — With configurable cipher suites
• IPv6-first — Prefers IPv6 addresses; falls back to IPv4 only when no IPv6 record exists
• Parallel chunked downloads — Splits files over 100 MB into chunks and downloads them concurrently using Range requests
• Resume — Supports Range/Accept-Ranges for resuming interrupted downloads
• Upload — PUT and POST with Content-Type detection
• Compression — Automatic gzip, deflate, brotli (if available) decompression
• Cookies — Netscape-format cookie jar for session persistence
• HSTS — RFC 6797 cache for automatic HTTPS upgrades
• OAuth 2.0 — Client credentials, authorization code, and refresh token flows
• Digest authentication — RFC 7616
• Basic authentication — RFC 7617
• Rate limiting — Token bucket bandwidth capping
• Proxy — HTTP CONNECT proxy support
• Custom DNS — Configurable DNS servers via --dns-servers
• Interface binding — Linux-specific SO_BINDTODEVICE
• Timing breakdown — DNS, TCP, TLS, TTFB timing via --trace-time
• WARC output — Web ARChive format for archival

Configuration

# TLS settings
--insecure              Skip TLS certificate verification
--cert client.pem       Client certificate (mTLS)
--key client.key        Client private key (mTLS)
--pinned-cert SHA256    Certificate pinning
--ssl-key-log file      TLS key log for Wireshark

# Proxy
--proxy http://proxy:8080

# Rate limiting
--rate-limit 1MB/s

# Headers
--header "X-Custom: value"

# Authentication
--username user --password pass
--auth-type basic|digest|auto

# Timing
--trace-time

FTP/FTPS (ftp://, ftps://)

Custom FTP implementation supporting active and passive modes.

Features

• Active and passive modes — Auto-detection; passive preferred
• Explicit TLS (FTPS) — AUTH TLS command on port 21
• Resume — REST command for partial transfers
• Directory listing — For recursive operations
• Certificate pinning — --pinned-cert enforces a SHA-256 leaf-cert match on FTPS connections

Limitations

• No parallel downloads
• No upload support

Configuration

# Explicit TLS (FTPS)
--url ftps://ftp.example.com/file.zip

# Username and password
--url ftp://ftp.example.com/file.zip --username user --password pass

# Anonymous (default)
--url ftp://anonymous@ftp.example.com/file.zip

# Certificate pinning (FTPS only)
--url ftps://ftp.example.com/file.zip --pinned-cert a1b2c3d4e5f6...

SFTP (sftp://)

SFTP over SSH protocol.

Features

• SSH key authentication — Auto-detection from ~/.ssh/id_rsa, ~/.ssh/id_ed25519
• Password authentication — Via --password or .netrc
• Known hosts verification — Via ~/.ssh/known_hosts
• Resume — Partial transfer support; parallel recursive downloads available via --recursive-parallel N with connection pooling
• known_hosts verification — Reads ~/.ssh/known_hosts; does not write to it. First-connection accept via --ssh-insecure
• Host-key pinning — --pinned-host-key enforces an exact SSH host-key match (takes precedence over --ssh-insecure and known_hosts)

Configuration

# With known hosts verification
--url sftp://user@host:/path/to/file.zip

# Skip host key verification (first connection)
--url sftp://user@host:/path/to/file.zip --ssh-insecure

# Custom known hosts file
--url sftp://user@host:/path/to/file.zip --known-hosts ~/.ssh/known_hosts

# With password
--url sftp://user@host:/path/to/file.zip --password "secret"

# Host-key pinning (OpenSSH fingerprint from ssh-keygen -lf)
--url sftp://user@host:/path/to/file.zip --pinned-host-key SHA256:AAAAC3NzaC1lZDI1NTE5AAAAIE...

Local File (file://)

Reads or copies local files.

Features

• File copy — Copies local files to the output path
• Directory listing — For recursive operations
• No network required — Works fully offline

Usage

# Copy a local file
goget --url file:///home/user/document.pdf

# Recursive directory copy
goget --url file:///home/user/docs/ --recursive --output ./backup/

Data URI (data:)

Handles inline data URIs as defined in RFC 2397.

Features

• Base64 decoding — data:application/zip;base64,UEsDB...
• Plain text — data:text/plain,Hello%20World
• No network required — Data is embedded in the URI

Usage

# Base64 encoded data
goget --url "data:application/zip;base64,UEsDB..." --output decoded.zip

# Plain text
goget --url "data:text/plain,Hello%20World" --output hello.txt

Gopher (gopher://)

Implements the Gopher protocol as specified in RFC 1436.

Features

• Type parsing — Respects Gopher type codes (0 = text, 1 = directory, 9 = binary)
• Directory listing — For recursive operations
• Text and binary downloads
• Resume — .goget.meta sidecar on interrupt for typeHTML, typeTextFile, and typeInformation responses; --resume continues interrupted Gopher transfers (skips already-written output bytes — post-transform length because Gopher line metadata is stripped before writing)

Limitations

• No parallel downloads
• No encryption (Gopher has no TLS)
• No upload

Usage

# Download a file from a Gopher server
goget --url gopher://gopher.example.com/9/file.zip

# Resume an interrupted Gopher text download
goget --url gopher://gopher.example.com/0/article --output article.txt --resume

# Browse a Gopher directory
goget --url gopher://gopher.example.com/1/

Gemini (gemini://)

Implements the Gemini protocol (a lightweight alternative to HTTP/HTTPS).

Features

• TLS — Mandatory TLS connections with TLS 1.2 minimum
• Redirect following — Respects Gemini redirect status codes (3x), up to 10 redirects
• Error handling — Input prompts (1x) return an error with the prompt text; temporary (4x) and permanent (5x) failures are handled
• Text and binary downloads — 2x success responses stream content directly
• Certificate pinning — --pinned-cert enforces a SHA-256 leaf-cert match
• Resume — .goget.meta sidecar tracks partial progress; --resume continues interrupted downloads

Limitations

• No parallel downloads
• No upload (Gemini is download-only)

Usage

# Download a file
goget --url gemini://gemini.example.com/file.zip

# Browse a capsule
goget --url gemini://gemini.example.com/

# Pin the server certificate
goget --url gemini://gemini.example.com/ --pinned-cert a1b2c3d4e5f6...

Protocol Resolution Flow

flowchart TD
    URL[URL input]
    Parse["Parse URL"]
    Normalize["Normalize scheme\n(https→http, ftps→ftp)"]
    Registry["Look up in protocol registry"]
    Found{"Protocol found?"}
    Error["Unsupported protocol error"]
    Handle["Delegate to protocol handler"]

    URL --> Parse --> Normalize --> Registry
    Registry --> Found
    Found -->|Yes| Handle
    Found -->|No| Error