goget/BACKLOG.md

# Backlog — goget

This file tracks planned features and improvements across releases.
Items are grouped by target version and priority. Completed items
move to `CHANGELOG.md` when the version ships.

---

## v1.1.0 — Protocol completeness

Finish the protocol layer so every supported protocol handles
recursive, parallel, resume, and shutdown correctly.  Also bring
certificate pinning and metadata debugging to all TLS protocols.

### 🔴 Supreme Priority

- [x] **Graceful shutdown for all protocols** — persist `.goget.meta` on
  SIGINT for FTP single-file and SFTP parallel recursive paths.
- [ ] **`--pinned-cert` for all TLS protocols** — extend certificate
  pinning to FTPS, Gemini, and WebDAVS (currently HTTP/HTTPS only).
  Also add SSH host-key pinning for SFTP as `--pinned-host-key`.
- [x] **Unified worker pool** — a single protocol-agnostic worker pool
  used by FTP, SFTP, and WebDAV recursive-parallel paths.  Removes
  the three near-identical semaphore+WaitGroup implementations.

### 🟡 High Priority

- [ ] **Gemini resume & metadata** — persist `.goget.meta` during Gemini
  downloads; `--resume` support.  Currently Gemini has `Features: []` —
  it doesn't advertise resume, and the download loop doesn't save
  progress.
- [ ] **Gopher resume & metadata** — same as Gemini.  Gopher `typeHTML`
  and `typeTextFile` responses should save `.goget.meta` for resume.

### 🟢 Medium Priority

- [x] **`--adjust-extension` for non-HTML protocols** — apply `.html`
  extension to text/html files from WebDAV, Gopher.
- [x] **`--backup-converted` for WebDAV mirrors** — keep `.orig` backups
  before link conversion.
- [ ] **`--show-metadata`** — display `.goget.meta` contents for debugging
  interrupted/resumed downloads.

---

## v1.2.0 — curl/wget parity & UX

Close the gap with curl and wget, and improve scripting/automation UX.

### 🟢 Medium Priority

- [ ] **`--progress=bar:force:noscroll`** — curl-compatible progress options.
- [ ] **`--metalink --follow-metalink=mem`** — keep Metalink in memory instead
  of writing to disk.
- [ ] **Bandwidth throttling per-domain** — global rate limit only at present.
- [ ] **MIME type detection for local files** — determine Content-Type from
  file extension when uploading (`mime.TypeByExtension` from stdlib covers this).
- [ ] **Notification hooks** — `--on-error` with richer environment variables
  (exit code, error type, checksum).
- [ ] **Retry with backoff visualization** — show retry attempts in progress
  bar (attempt 2/3, waiting 4s…).

### 🔵 Lower Priority

- [ ] **URL glob expansion (`--glob`)** — expand `[1-3]` and `{a,b}` patterns
  in URLs (flag exists, logic not implemented).

---

## v1.3.0 — Security hardening

TLS, authentication, and verification improvements.

### 🔴 Supreme Priority

- [ ] **PGP replacement** — `golang.org/x/crypto/openpgp` is frozen
  (archived by Go team).  Use `github.com/ProtonMail/go-crypto`
  (BSD-3-Clause) as the replacement — the only permitted third-party
  dependency exception for the Go codebase.

### 🟡 High Priority

- [ ] **SOCKS5 proxy for non-HTTP protocols** — FTP and SFTP through SOCKS5
  proxy (currently HTTP/HTTPS and WebDAV only).  `golang.org/x/net/proxy`
  provides the SOCKS5 dialer; SFTP is straightforward, FTP needs
  dual-connection handling.
- [ ] **DNS-over-HTTPS (DoH)** — optional DoH resolver.  Uses
  `golang.org/x/net/dns/dnsmessage` for DNS wire format and `net/http`
  for the HTTPS transport.

### 🟢 Medium Priority

- [ ] **Structured error output with `--json`** — machine-readable error
  objects with type, message, URL, and hint fields.

---

## v2.0.0 — Package registries & CAS cache

Turn goget from a file-in-URL-out downloader into a language-aware
package fetcher.  The content-addressable cache deduplicates downloads
across protocols and projects — a file is never downloaded twice.

### 🔴 Supreme Priority

- [ ] **Content-addressable cache (CAS)** — every downloaded file is stored
  under its SHA-256 digest.  `goget` before hitting the network consults
  the cache.  Identical content from different URLs, protocols, or
  projects hits the cache exactly once.
- [ ] **`crate://` protocol** — `crate://serde@1.0.0` → fetches `.crate`
  from crates.io API.  `crate://serde@latest` → resolves latest version.
- [ ] **`gem://` protocol** — `gem://rails@7.2` → fetches `.gem` from
  RubyGems.org.
- [ ] **`npm://` protocol** — `npm://react@19` → fetches tarball from
  registry.npmjs.org.
- [ ] **`git://` protocol** — `git://github.com/user/repo@v1.0` → archive
  download via provider API.  `@tag`, `@branch`, `@commit` resolution.

### 🟡 High Priority

- [ ] **`oci://` protocol** — `oci://alpine:3.20` → pulls OCI
  image layers via registry v2 API, stores in CAS.  No daemon required —
  goget acts as a standalone OCI image fetcher.  Works against any
  OCI-compatible registry (Docker Hub, Quay.io, GHCR, …).  Complex
  protocol, but purely HTTP + JSON — stdlib covers it.
- [ ] **Lockfile import** — `goget crate:// --from-lock Cargo.lock`
  downloads and caches every dependency in one shot.  Same for
  `Gemfile.lock` and `package-lock.json`.

---

## v2.1.0 — Plugin system & extensibility

### 🔴 Supreme Priority

- [ ] **Plugin system** — protocol handlers as external binaries.
  Discovered from `~/.config/goget/plugins/`.  Line-based protocol
  over stdin/stdout or Unix socket.
- [ ] **`plugin://` protocol** — transparent proxy to external handlers.

### 🟡 High Priority

- [ ] **Plugin SDK** — Go library (`golang.org/x/` only) for writing plugins.
- [ ] **Plugin registry** — community-maintained index of known plugins.

---

## v2.2.0 — Distributed download mesh

Goget instances on the same LAN discover each other and share cached files.
mDNS/DNS-SD written from scratch on top of `golang.org/x/net/dns/dnsmessage`
and stdlib multicast UDP.

### 🔴 Supreme Priority

- [ ] **`ggtp://` protocol (goget transfer protocol)** — simple
  single-round-trip request for a file by SHA-256 from a peer.
- [ ] **LAN discovery (mDNS / DNS-SD)** — `_goget._tcp` announcements.
  Written from scratch using `golang.org/x/net/dns/dnsmessage` for
  message encoding and stdlib `net` for multicast UDP.

### 🟡 High Priority

- [ ] **Mesh-aware CAS** — query LAN peers for SHA-256 before downloading
  from the internet.
- [ ] **Peer authentication** — optional shared secret or mTLS between
  trusted nodes.

---

## v3.0.0 — goget-as-a-service

Long-lived daemon with REST API, TUI, and Web GUI.

### 🔴 Supreme Priority

- [ ] **Daemon mode** — `goget daemon` + REST API (localhost-only).
- [ ] **REST API** — `POST /downloads`, `GET /downloads/:id`, SSE progress.

### 🟡 High Priority

- [ ] **Web GUI** — Vue 3 + Vite + Tailwind CSS, daemon REST API + SSE.
- [ ] **Queue pause/resume/reorder** — full download queue management.

### 🟢 Medium Priority

- [ ] **S3 protocol** — as a plugin (see v2.1.0).  Signature v4 auth
  (HMAC-SHA256, all in stdlib).
- [ ] **Huawei OBS, Alibaba OSS, Tencent COS** — cloud storage plugins.
- [ ] **systemd unit** — for daemon / scheduled mode.
- [ ] **Download speed test / benchmark mode**.

---

## v3.1.0 — Streaming & real-time protocols

### 🟢 Medium Priority

- [ ] **HLS / MPEG-DASH** — download streamed video.
- [ ] **Icecast / SHOUTcast** — capture internet radio.
- [ ] **WebSocket capture** — record WebSocket messages (RFC 6455 framing
  from scratch on top of `net/http` hijack).

---

## 💡 Backlog (unscheduled)

Items needing more design, dependent on external factors, or deferred
until foundational libraries are written.

### Deferred — waiting for custom libraries

- [ ] **Brotli decompression** — RFC 7932.  Will be built as a standalone
  Go library first, then integrated into goget.  No third-party
  dependency — pure Go from scratch.
- [ ] **TUI (Terminal User Interface)** — interactive download manager.
  Will be built as a standalone Go TUI toolkit on top of `golang.org/x/term`,
  then integrated.  Layout engine, widgets, and event loop written from
  scratch.

### Performance

- [ ] **Memory-mapped I/O for large files** — `mmap` for checksum
  verification.
- [ ] **Zero-copy for `file://` and `data:` protocols** — `sendfile(2)` /
  `splice(2)` where available on Linux.
- [ ] **HTTP/3 (QUIC)** — when Go stdlib gains QUIC support.

### Protocols (rare / niche)

- [ ] **Rsync** — rolling checksum, delta transfer, wire protocol from scratch.
- [ ] **TFTP** — trivial file transfer, embedded devices and PXE.
- [ ] **NFS** — NFS v3/v4 client from scratch (ONC RPC/XDR, portmapper).

### Code Quality & Testing

- [ ] **Integration test matrix** — CI against real FTP, SFTP, WebDAV servers.
- [ ] **Coverage threshold increase** — raise from 40% to 60%.
- [ ] **Benchmark suite** — `go test -bench` in CI.
- [ ] **Fuzzing corpus in CI** — store and replay past crashes automatically.
- [ ] **`internal/log` migration** — migrate remaining ~140 `fmt.Fprintf` and
  `cli.Print*` callsites to `log.Infof`.
- [ ] **`unsafe` audit** — verify no `unsafe` usage outside build-tag-gated
  platform-specific code.

### Project Infrastructure

- [ ] **RPM spec file** — Fedora / RHEL / openEuler packaging.
- [ ] **FreeBSD port** — official FreeBSD ports tree entry.

### Documentation

- [ ] **Man page packaging** — already generated, needs installation via packaging.
- [ ] **Architecture decision records** — document design choices.
- [ ] **Protocol-specific guides** — one doc per protocol with examples.
- [ ] **Troubleshooting guide** — common errors and solutions.