From 1672698936698ebba4f29a8b677c4d590b30d84f Mon Sep 17 00:00:00 2001 From: Petr Date: Mon, 29 Jun 2026 12:30:28 +0200 Subject: [PATCH] chore: update BACKLOG with v1.1.0 progress and restated roadmap --- BACKLOG.md | 229 ++++++++++++++++++++++++++++++++++------------------- 1 file changed, 149 insertions(+), 80 deletions(-) diff --git a/BACKLOG.md b/BACKLOG.md index 0a8426f..63a59f8 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -9,35 +9,37 @@ move to `CHANGELOG.md` when the version ships. ## v1.1.0 ā€” Protocol completeness Finish the protocol layer so every supported protocol handles -recursive, parallel, resume, and shutdown correctly. +recursive, parallel, resume, and shutdown correctly. Also bring +certificate pinning and metadata debugging to all TLS protocols. ### šŸ”“ Supreme Priority -- [ ] **Graceful shutdown for all protocols** ā€” persist `.goget.meta` on +- [x] **Graceful shutdown for all protocols** ā€” persist `.goget.meta` on SIGINT for FTP single-file and SFTP parallel recursive paths. -- [ ] **WebDAV connection pooling** ā€” shared `http.Transport` across WebDAV - requests instead of a fresh transport per request (currently defeats HTTP - keep-alive, TLS session resumption, and HTTP/2 multiplexing). +- [ ] **`--pinned-cert` for all TLS protocols** ā€” extend certificate + pinning to FTPS, Gemini, and WebDAVS (currently HTTP/HTTPS only). + Also add SSH host-key pinning for SFTP as `--pinned-host-key`. +- [ ] **Unified worker pool** ā€” a single protocol-agnostic worker pool + used by FTP, SFTP, and WebDAV recursive-parallel paths. Removes + the three near-identical semaphore+WaitGroup implementations. ### šŸŸ” High Priority -- [ ] **WebDAV recursive parallel downloads** ā€” extend `--recursive-parallel` - to WebDAV using the shared transport. -- [ ] **SFTP recursive parallel downloads** ā€” parallel SFTP recursive with - session pooling. -- [ ] **FTP recursive parallel** ā€” connection pool for independent control - connections in FTP recursive mode. -- [ ] **`--dry-run` for WebDAV recursive** ā€” list matched entries without - downloading. +- [ ] **Gemini resume & metadata** ā€” persist `.goget.meta` during Gemini + downloads; `--resume` support. Currently Gemini has `Features: []` ā€” + it doesn't advertise resume, and the download loop doesn't save + progress. +- [ ] **Gopher resume & metadata** ā€” same as Gemini. Gopher `typeHTML` + and `typeTextFile` responses should save `.goget.meta` for resume. ### šŸŸ¢ Medium Priority -- [ ] **`--adjust-extension` for non-HTML protocols** ā€” apply `.html` +- [x] **`--adjust-extension` for non-HTML protocols** ā€” apply `.html` extension to text/html files from WebDAV, Gopher. -- [ ] **`--backup-converted` for WebDAV mirrors** ā€” keep `.orig` backups +- [x] **`--backup-converted` for WebDAV mirrors** ā€” keep `.orig` backups before link conversion. -- [ ] **Parallel recursive across all protocols** ā€” unified worker pool - independent of protocol handler. +- [ ] **`--show-metadata`** ā€” display `.goget.meta` contents for debugging + interrupted/resumed downloads. --- @@ -45,15 +47,6 @@ recursive, parallel, resume, and shutdown correctly. Close the gap with curl and wget, and improve scripting/automation UX. -### šŸŸ” High Priority - -- [ ] **`--create-dirs`** ā€” auto-create parent directories for `--output` - paths (curl `--create-dirs` / wget `--directory-prefix` equivalent). -- [ ] **Short flags** ā€” `-O`, `-o`, `-s`, `-v` aliases for curl/wget - muscle memory. `--output-document` alias for `--output`. -- [ ] **Brotli decompression** ā€” RFC 7932, written from scratch. - Currently only gzip/deflate/bzip2/LZW. - ### šŸŸ¢ Medium Priority - [ ] **`--progress=bar:force:noscroll`** ā€” curl-compatible progress options. @@ -61,21 +54,16 @@ Close the gap with curl and wget, and improve scripting/automation UX. of writing to disk. - [ ] **Bandwidth throttling per-domain** ā€” global rate limit only at present. - [ ] **MIME type detection for local files** ā€” determine Content-Type from - file extension when uploading. -- [ ] **`--cut-dirs` for recursive mode** ā€” strip directory components from - output paths. + file extension when uploading (`mime.TypeByExtension` from stdlib covers this). - [ ] **Notification hooks** ā€” `--on-error` with richer environment variables (exit code, error type, checksum). - [ ] **Retry with backoff visualization** ā€” show retry attempts in progress - bar (attempt 2/3, waiting 4s...). -- [ ] **Idempotent `--resume`** ā€” auto-detect whether resume is possible - without manual flag. -- [ ] **`--show-metadata`** ā€” display `.goget.meta` contents for debugging. + bar (attempt 2/3, waiting 4sā€¦). ### šŸ”µ Lower Priority - [ ] **URL glob expansion (`--glob`)** ā€” expand `[1-3]` and `{a,b}` patterns - in URLs. + in URLs (flag exists, logic not implemented). --- @@ -85,18 +73,20 @@ TLS, authentication, and verification improvements. ### šŸ”“ Supreme Priority -- [ ] **PGP replacement** ā€” `golang.org/x/crypto/openpgp` is frozen (archived - by Go team). Evaluate: inline a minimal subset, or wait for a community - `golang.org/x/` successor. +- [ ] **PGP replacement** ā€” `golang.org/x/crypto/openpgp` is frozen + (archived by Go team). Use `github.com/ProtonMail/go-crypto` + (BSD-3-Clause) as the replacement ā€” the only permitted third-party + dependency exception for the Go codebase. ### šŸŸ” High Priority -- [ ] **`--pinned-cert` for all TLS protocols** ā€” extend certificate pinning - to FTPS, SFTP, Gemini, and WebDAVS (currently HTTP/HTTPS only). - [ ] **SOCKS5 proxy for non-HTTP protocols** ā€” FTP and SFTP through SOCKS5 - proxy (currently HTTP/HTTPS only). -- [ ] **DNS-over-HTTPS (DoH)** ā€” optional DoH resolver. Must use only - `golang.org/x/` libraries. + proxy (currently HTTP/HTTPS and WebDAV only). `golang.org/x/net/proxy` + provides the SOCKS5 dialer; SFTP is straightforward, FTP needs + dual-connection handling. +- [ ] **DNS-over-HTTPS (DoH)** ā€” optional DoH resolver. Uses + `golang.org/x/net/dns/dnsmessage` for DNS wire format and `net/http` + for the HTTPS transport. ### šŸŸ¢ Medium Priority @@ -105,64 +95,146 @@ TLS, authentication, and verification improvements. --- -## v2.0.0 ā€” New protocols +## v2.0.0 ā€” Package registries & CAS cache -Cloud providers, additional network protocols, and long-lived services. +Turn goget from a file-in-URL-out downloader into a language-aware +package fetcher. The content-addressable cache deduplicates downloads +across protocols and projects ā€” a file is never downloaded twice. -### šŸ”µ Lower Priority +### šŸ”“ Supreme Priority -- [ ] **S3** ā€” generic S3-compatible protocol with signature v4 auth - (covers AWS S3, MinIO, and any S3-compatible storage). -- [ ] **Huawei OBS** ā€” Huawei Cloud Object Storage. -- [ ] **Alibaba OSS** ā€” Alibaba Cloud Object Storage Service. -- [ ] **Tencent COS** ā€” Tencent Cloud Object Storage. -- [ ] **Rsync** ā€” incremental file transfer. Written from scratch - (rolling checksum, delta transfer, rsync wire protocol). -- [ ] **TFTP** ā€” trivial file transfer, useful for embedded devices and PXE. -- [ ] **NFS** ā€” NFS v3/v4 client. Written from scratch - (ONC RPC/XDR, portmapper, mount protocol, NFS protocol). -- [ ] **goget-as-a-service** ā€” daemon mode with REST API for remote download - management. +- [ ] **Content-addressable cache (CAS)** ā€” every downloaded file is stored + under its SHA-256 digest. `goget` before hitting the network consults + the cache. Identical content from different URLs, protocols, or + projects hits the cache exactly once. +- [ ] **`crate://` protocol** ā€” `crate://serde@1.0.0` ā†’ fetches `.crate` + from crates.io API. `crate://serde@latest` ā†’ resolves latest version. +- [ ] **`gem://` protocol** ā€” `gem://rails@7.2` ā†’ fetches `.gem` from + RubyGems.org. +- [ ] **`npm://` protocol** ā€” `npm://react@19` ā†’ fetches tarball from + registry.npmjs.org. +- [ ] **`git://` protocol** ā€” `git://github.com/user/repo@v1.0` ā†’ archive + download via provider API. `@tag`, `@branch`, `@commit` resolution. + +### šŸŸ” High Priority + +- [ ] **`oci://` protocol** ā€” `oci://alpine:3.20` ā†’ pulls OCI + image layers via registry v2 API, stores in CAS. No daemon required ā€” + goget acts as a standalone OCI image fetcher. Works against any + OCI-compatible registry (Docker Hub, Quay.io, GHCR, ā€¦). Complex + protocol, but purely HTTP + JSON ā€” stdlib covers it. +- [ ] **Lockfile import** ā€” `goget crate:// --from-lock Cargo.lock` + downloads and caches every dependency in one shot. Same for + `Gemfile.lock` and `package-lock.json`. --- -## v2.1.0 ā€” Extensibility & UI +## v2.1.0 ā€” Plugin system & extensibility -### šŸ”µ Lower Priority +### šŸ”“ Supreme Priority -- [ ] **Plugin system** ā€” protocol handlers as external plugins. -- [ ] **TUI (Terminal User Interface)** ā€” interactive download manager with - live queue, progress, and log views. Keyboard navigation, color-coded - status, inline log viewer. -- [ ] **Web GUI** ā€” browser-based download manager (Vue 3 frontend, REST API, - live progress via SSE). -- [ ] **Download speed test / benchmark mode** ā€” measure maximum throughput - against a target server without saving the file. +- [ ] **Plugin system** ā€” protocol handlers as external binaries. + Discovered from `~/.config/goget/plugins/`. Line-based protocol + over stdin/stdout or Unix socket. +- [ ] **`plugin://` protocol** ā€” transparent proxy to external handlers. + +### šŸŸ” High Priority + +- [ ] **Plugin SDK** ā€” Go library (`golang.org/x/` only) for writing plugins. +- [ ] **Plugin registry** ā€” community-maintained index of known plugins. + +--- + +## v2.2.0 ā€” Distributed download mesh + +Goget instances on the same LAN discover each other and share cached files. +mDNS/DNS-SD written from scratch on top of `golang.org/x/net/dns/dnsmessage` +and stdlib multicast UDP. + +### šŸ”“ Supreme Priority + +- [ ] **`ggtp://` protocol (goget transfer protocol)** ā€” simple + single-round-trip request for a file by SHA-256 from a peer. +- [ ] **LAN discovery (mDNS / DNS-SD)** ā€” `_goget._tcp` announcements. + Written from scratch using `golang.org/x/net/dns/dnsmessage` for + message encoding and stdlib `net` for multicast UDP. + +### šŸŸ” High Priority + +- [ ] **Mesh-aware CAS** ā€” query LAN peers for SHA-256 before downloading + from the internet. +- [ ] **Peer authentication** ā€” optional shared secret or mTLS between + trusted nodes. + +--- + +## v3.0.0 ā€” goget-as-a-service + +Long-lived daemon with REST API, TUI, and Web GUI. + +### šŸ”“ Supreme Priority + +- [ ] **Daemon mode** ā€” `goget daemon` + REST API (localhost-only). +- [ ] **REST API** ā€” `POST /downloads`, `GET /downloads/:id`, SSE progress. + +### šŸŸ” High Priority + +- [ ] **Web GUI** ā€” Vue 3 + Vite + Tailwind CSS, daemon REST API + SSE. +- [ ] **Queue pause/resume/reorder** ā€” full download queue management. + +### šŸŸ¢ Medium Priority + +- [ ] **S3 protocol** ā€” as a plugin (see v2.1.0). Signature v4 auth + (HMAC-SHA256, all in stdlib). +- [ ] **Huawei OBS, Alibaba OSS, Tencent COS** ā€” cloud storage plugins. +- [ ] **systemd unit** ā€” for daemon / scheduled mode. +- [ ] **Download speed test / benchmark mode**. + +--- + +## v3.1.0 ā€” Streaming & real-time protocols + +### šŸŸ¢ Medium Priority + +- [ ] **HLS / MPEG-DASH** ā€” download streamed video. +- [ ] **Icecast / SHOUTcast** ā€” capture internet radio. +- [ ] **WebSocket capture** ā€” record WebSocket messages (RFC 6455 framing + from scratch on top of `net/http` hijack). --- ## šŸ’” Backlog (unscheduled) -Items that need more design, depend on external factors, or are -lower priority than the versioned roadmap. +Items needing more design, dependent on external factors, or deferred +until foundational libraries are written. + +### Deferred ā€” waiting for custom libraries + +- [ ] **Brotli decompression** ā€” RFC 7932. Will be built as a standalone + Go library first, then integrated into goget. No third-party + dependency ā€” pure Go from scratch. +- [ ] **TUI (Terminal User Interface)** ā€” interactive download manager. + Will be built as a standalone Go TUI toolkit on top of `golang.org/x/term`, + then integrated. Layout engine, widgets, and event loop written from + scratch. ### Performance - [ ] **Memory-mapped I/O for large files** ā€” `mmap` for checksum - verification of multi-gigabyte downloads. + verification. - [ ] **Zero-copy for `file://` and `data:` protocols** ā€” `sendfile(2)` / `splice(2)` where available on Linux. - [ ] **HTTP/3 (QUIC)** ā€” when Go stdlib gains QUIC support. -### Advanced UX +### Protocols (rare / niche) -- [ ] **Scheduling ā€” pause/resume queued downloads** ā€” ability to pause the - queue, reorder items, and resume later. +- [ ] **Rsync** ā€” rolling checksum, delta transfer, wire protocol from scratch. +- [ ] **TFTP** ā€” trivial file transfer, embedded devices and PXE. +- [ ] **NFS** ā€” NFS v3/v4 client from scratch (ONC RPC/XDR, portmapper). ### Code Quality & Testing -- [ ] **Integration test matrix** ā€” CI against real FTP, SFTP, and WebDAV - servers. +- [ ] **Integration test matrix** ā€” CI against real FTP, SFTP, WebDAV servers. - [ ] **Coverage threshold increase** ā€” raise from 40% to 60%. - [ ] **Benchmark suite** ā€” `go test -bench` in CI. - [ ] **Fuzzing corpus in CI** ā€” store and replay past crashes automatically. @@ -175,13 +247,10 @@ lower priority than the versioned roadmap. - [ ] **RPM spec file** ā€” Fedora / RHEL / openEuler packaging. - [ ] **FreeBSD port** ā€” official FreeBSD ports tree entry. -- [ ] **systemd unit** ā€” for daemon / scheduled mode. ### Documentation -- [ ] **Man page packaging** ā€” already generated by `--generate-man-page`, - needs installation via packaging. -- [ ] **Architecture decision records** ā€” document why specific choices were - made (no short flags, IPv6-first, long flags only, no global state). +- [ ] **Man page packaging** ā€” already generated, needs installation via packaging. +- [ ] **Architecture decision records** ā€” document design choices. - [ ] **Protocol-specific guides** ā€” one doc per protocol with examples. - [ ] **Troubleshooting guide** ā€” common errors and solutions.