goget/BACKLOG.md

Backlog — goget

This file tracks planned features and improvements across releases. Items are grouped by target version and priority. Completed items move to CHANGELOG.md when the version ships.

---

v1.1.0 — Protocol completeness

Finish the protocol layer so every supported protocol handles recursive, parallel, resume, and shutdown correctly. Also bring certificate pinning and metadata debugging to all TLS protocols.

🔴 Supreme Priority

• Graceful shutdown for all protocols — persist .goget.meta on SIGINT for FTP single-file and SFTP parallel recursive paths.
• --pinned-cert for all TLS protocols — extend certificate pinning to FTPS, Gemini, and WebDAVS (currently HTTP/HTTPS only). Also add SSH host-key pinning for SFTP as --pinned-host-key.
• Unified worker pool — a single protocol-agnostic worker pool used by FTP, SFTP, and WebDAV recursive-parallel paths. Removes the three near-identical semaphore+WaitGroup implementations.

🟡 High Priority

• Gemini resume & metadata — persist .goget.meta during Gemini downloads; --resume support. Currently Gemini has Features: [] — it doesn't advertise resume, and the download loop doesn't save progress.
• Gopher resume & metadata — same as Gemini. Gopher typeHTML and typeTextFile responses should save .goget.meta for resume.

🟢 Medium Priority

• --adjust-extension for non-HTML protocols — apply .html extension to text/html files from WebDAV, Gopher.
• --backup-converted for WebDAV mirrors — keep .orig backups before link conversion.
• --show-metadata — display .goget.meta contents for debugging interrupted/resumed downloads.

---

v1.2.0 — curl/wget parity & UX

Close the gap with curl and wget, and improve scripting/automation UX.

🟢 Medium Priority

• --progress=bar:force:noscroll — curl-compatible progress options.
• --metalink --follow-metalink=mem — keep Metalink in memory instead of writing to disk.
• Bandwidth throttling per-domain — global rate limit only at present.
• MIME type detection for local files — determine Content-Type from file extension when uploading (mime.TypeByExtension from stdlib covers this).
• Notification hooks — --on-error with richer environment variables (exit code, error type, checksum).
• Retry with backoff visualization — show retry attempts in progress bar (attempt 2/3, waiting 4s…).

🔵 Lower Priority

• URL glob expansion (--glob) — expand [1-3] and {a,b} patterns in URLs (flag exists, logic not implemented).

---

v1.3.0 — Security hardening

TLS, authentication, and verification improvements.

🔴 Supreme Priority

• PGP replacement — golang.org/x/crypto/openpgp is frozen (archived by Go team). Use github.com/ProtonMail/go-crypto (BSD-3-Clause) as the replacement — the only permitted third-party dependency exception for the Go codebase.

🟡 High Priority

• SOCKS5 proxy for non-HTTP protocols — FTP and SFTP through SOCKS5 proxy (currently HTTP/HTTPS and WebDAV only). golang.org/x/net/proxy provides the SOCKS5 dialer; SFTP is straightforward, FTP needs dual-connection handling.
• DNS-over-HTTPS (DoH) — optional DoH resolver. Uses golang.org/x/net/dns/dnsmessage for DNS wire format and net/http for the HTTPS transport.

🟢 Medium Priority

• Structured error output with --json — machine-readable error objects with type, message, URL, and hint fields.

---

v2.0.0 — Package registries & CAS cache

Turn goget from a file-in-URL-out downloader into a language-aware package fetcher. The content-addressable cache deduplicates downloads across protocols and projects — a file is never downloaded twice.

🔴 Supreme Priority

• Content-addressable cache (CAS) — every downloaded file is stored under its SHA-256 digest. goget before hitting the network consults the cache. Identical content from different URLs, protocols, or projects hits the cache exactly once.
• crate:// protocol — crate://serde@1.0.0 → fetches .crate from crates.io API. crate://serde@latest → resolves latest version.
• gem:// protocol — gem://rails@7.2 → fetches .gem from RubyGems.org.
• npm:// protocol — npm://react@19 → fetches tarball from registry.npmjs.org.
• git:// protocol — git://github.com/user/repo@v1.0 → archive download via provider API. @tag, @branch, @commit resolution.

🟡 High Priority

• oci:// protocol — oci://alpine:3.20 → pulls OCI image layers via registry v2 API, stores in CAS. No daemon required — goget acts as a standalone OCI image fetcher. Works against any OCI-compatible registry (Docker Hub, Quay.io, GHCR, …). Complex protocol, but purely HTTP + JSON — stdlib covers it.
• Lockfile import — goget crate:// --from-lock Cargo.lock downloads and caches every dependency in one shot. Same for Gemfile.lock and package-lock.json.

---

v2.1.0 — Plugin system & extensibility

🔴 Supreme Priority

• Plugin system — protocol handlers as external binaries. Discovered from ~/.config/goget/plugins/. Line-based protocol over stdin/stdout or Unix socket.
• plugin:// protocol — transparent proxy to external handlers.

🟡 High Priority

• Plugin SDK — Go library (golang.org/x/ only) for writing plugins.
• Plugin registry — community-maintained index of known plugins.

---

v2.2.0 — Distributed download mesh

Goget instances on the same LAN discover each other and share cached files. mDNS/DNS-SD written from scratch on top of golang.org/x/net/dns/dnsmessage and stdlib multicast UDP.

🔴 Supreme Priority

• ggtp:// protocol (goget transfer protocol) — simple single-round-trip request for a file by SHA-256 from a peer.
• LAN discovery (mDNS / DNS-SD) — _goget._tcp announcements. Written from scratch using golang.org/x/net/dns/dnsmessage for message encoding and stdlib net for multicast UDP.

🟡 High Priority

• Mesh-aware CAS — query LAN peers for SHA-256 before downloading from the internet.
• Peer authentication — optional shared secret or mTLS between trusted nodes.

---

v3.0.0 — goget-as-a-service

Long-lived daemon with REST API, TUI, and Web GUI.

🔴 Supreme Priority

• Daemon mode — goget daemon + REST API (localhost-only).
• REST API — POST /downloads, GET /downloads/:id, SSE progress.

🟡 High Priority

• Web GUI — Vue 3 + Vite + Tailwind CSS, daemon REST API + SSE.
• Queue pause/resume/reorder — full download queue management.

🟢 Medium Priority

• S3 protocol — as a plugin (see v2.1.0). Signature v4 auth (HMAC-SHA256, all in stdlib).
• Huawei OBS, Alibaba OSS, Tencent COS — cloud storage plugins.
• systemd unit — for daemon / scheduled mode.
• Download speed test / benchmark mode.

---

v3.1.0 — Streaming & real-time protocols

🟢 Medium Priority

• HLS / MPEG-DASH — download streamed video.
• Icecast / SHOUTcast — capture internet radio.
• WebSocket capture — record WebSocket messages (RFC 6455 framing from scratch on top of net/http hijack).

---

💡 Backlog (unscheduled)

Items needing more design, dependent on external factors, or deferred until foundational libraries are written.

Deferred — waiting for custom libraries

• Brotli decompression — RFC 7932. Will be built as a standalone Go library first, then integrated into goget. No third-party dependency — pure Go from scratch.
• TUI (Terminal User Interface) — interactive download manager. Will be built as a standalone Go TUI toolkit on top of golang.org/x/term, then integrated. Layout engine, widgets, and event loop written from scratch.

Performance

• Memory-mapped I/O for large files — mmap for checksum verification.
• Zero-copy for file:// and data: protocols — sendfile(2) / splice(2) where available on Linux.
• HTTP/3 (QUIC) — when Go stdlib gains QUIC support.

Protocols (rare / niche)

• Rsync — rolling checksum, delta transfer, wire protocol from scratch.
• TFTP — trivial file transfer, embedded devices and PXE.
• NFS — NFS v3/v4 client from scratch (ONC RPC/XDR, portmapper).

Code Quality & Testing

• Integration test matrix — CI against real FTP, SFTP, WebDAV servers.
• Coverage threshold increase — raise from 40% to 60%.
• Benchmark suite — go test -bench in CI.
• Fuzzing corpus in CI — store and replay past crashes automatically.
• internal/log migration — migrate remaining ~140 fmt.Fprintf and cli.Print* callsites to log.Infof.
• unsafe audit — verify no unsafe usage outside build-tag-gated platform-specific code.

Project Infrastructure

• RPM spec file — Fedora / RHEL / openEuler packaging.
• FreeBSD port — official FreeBSD ports tree entry.

Documentation

• Man page packaging — already generated, needs installation via packaging.
• Architecture decision records — document design choices.
• Protocol-specific guides — one doc per protocol with examples.
• Troubleshooting guide — common errors and solutions.