petrbalvin/goget
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

chore: update BACKLOG with v1.1.0 progress and restated roadmap

Browse Source
This commit is contained in:
petrbalvin
2026-06-29 12:30:28 +02:00
parent f915299fa4
commit 1672698936
1 changed files with 149 additions and 80 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BACKLOG.md
+149 -80
View File
@@ -9,35 +9,37 @@ move to `CHANGELOG.md` when the version ships.
## v1.1.0 — Protocol completeness
Finish the protocol layer so every supported protocol handles
recursive, parallel, resume, and shutdown correctly.
recursive, parallel, resume, and shutdown correctly. Also bring
certificate pinning and metadata debugging to all TLS protocols.
### 🔴 Supreme Priority
- [ ] **Graceful shutdown for all protocols** — persist `.goget.meta` on
- [x] **Graceful shutdown for all protocols** — persist `.goget.meta` on
SIGINT for FTP single-file and SFTP parallel recursive paths.
- [ ] **WebDAV connection pooling** — shared `http.Transport` across WebDAV
requests instead of a fresh transport per request (currently defeats HTTP
keep-alive, TLS session resumption, and HTTP/2 multiplexing).
- [ ] **`--pinned-cert` for all TLS protocols** — extend certificate
pinning to FTPS, Gemini, and WebDAVS (currently HTTP/HTTPS only).
Also add SSH host-key pinning for SFTP as `--pinned-host-key`.
- [ ] **Unified worker pool** — a single protocol-agnostic worker pool
used by FTP, SFTP, and WebDAV recursive-parallel paths. Removes
the three near-identical semaphore+WaitGroup implementations.
### 🟡 High Priority
- [ ] **WebDAV recursive parallel downloads** — extend `--recursive-parallel`
to WebDAV using the shared transport.
- [ ] **SFTP recursive parallel downloads** — parallel SFTP recursive with
session pooling.
- [ ] **FTP recursive parallel** — connection pool for independent control
connections in FTP recursive mode.
- [ ] **`--dry-run` for WebDAV recursive** — list matched entries without
downloading.
- [ ] **Gemini resume & metadata** — persist `.goget.meta` during Gemini
downloads; `--resume` support. Currently Gemini has `Features: []` —
it doesn't advertise resume, and the download loop doesn't save
progress.
- [ ] **Gopher resume & metadata** — same as Gemini. Gopher `typeHTML`
and `typeTextFile` responses should save `.goget.meta` for resume.
### 🟢 Medium Priority
- [ ] **`--adjust-extension` for non-HTML protocols** — apply `.html`
- [x] **`--adjust-extension` for non-HTML protocols** — apply `.html`
extension to text/html files from WebDAV, Gopher.
- [ ] **`--backup-converted` for WebDAV mirrors** — keep `.orig` backups
- [x] **`--backup-converted` for WebDAV mirrors** — keep `.orig` backups
before link conversion.
- [ ] **Parallel recursive across all protocols** — unified worker pool
independent of protocol handler.
- [ ] **`--show-metadata`** — display `.goget.meta` contents for debugging
interrupted/resumed downloads.
---
@@ -45,15 +47,6 @@ recursive, parallel, resume, and shutdown correctly.
Close the gap with curl and wget, and improve scripting/automation UX.
### 🟡 High Priority
- [ ] **`--create-dirs`** — auto-create parent directories for `--output`
paths (curl `--create-dirs` / wget `--directory-prefix` equivalent).
- [ ] **Short flags** — `-O`, `-o`, `-s`, `-v` aliases for curl/wget
muscle memory. `--output-document` alias for `--output`.
- [ ] **Brotli decompression** — RFC 7932, written from scratch.
Currently only gzip/deflate/bzip2/LZW.
### 🟢 Medium Priority
- [ ] **`--progress=bar:force:noscroll`** — curl-compatible progress options.
@@ -61,21 +54,16 @@ Close the gap with curl and wget, and improve scripting/automation UX.
of writing to disk.
- [ ] **Bandwidth throttling per-domain** — global rate limit only at present.
- [ ] **MIME type detection for local files** — determine Content-Type from
file extension when uploading.
- [ ] **`--cut-dirs` for recursive mode** — strip directory components from
output paths.
file extension when uploading (`mime.TypeByExtension` from stdlib covers this).
- [ ] **Notification hooks** — `--on-error` with richer environment variables
(exit code, error type, checksum).
- [ ] **Retry with backoff visualization** — show retry attempts in progress
bar (attempt 2/3, waiting 4s...).
- [ ] **Idempotent `--resume`** — auto-detect whether resume is possible
without manual flag.
- [ ] **`--show-metadata`** — display `.goget.meta` contents for debugging.
bar (attempt 2/3, waiting 4s…).
### 🔵 Lower Priority
- [ ] **URL glob expansion (`--glob`)** — expand `[1-3]` and `{a,b}` patterns
in URLs.
in URLs (flag exists, logic not implemented).
---
@@ -85,18 +73,20 @@ TLS, authentication, and verification improvements.
### 🔴 Supreme Priority
- [ ] **PGP replacement** — `golang.org/x/crypto/openpgp` is frozen (archived
by Go team). Evaluate: inline a minimal subset, or wait for a community
`golang.org/x/` successor.
- [ ] **PGP replacement** — `golang.org/x/crypto/openpgp` is frozen
(archived by Go team). Use `github.com/ProtonMail/go-crypto`
(BSD-3-Clause) as the replacement — the only permitted third-party
dependency exception for the Go codebase.
### 🟡 High Priority
- [ ] **`--pinned-cert` for all TLS protocols** — extend certificate pinning
to FTPS, SFTP, Gemini, and WebDAVS (currently HTTP/HTTPS only).
- [ ] **SOCKS5 proxy for non-HTTP protocols** — FTP and SFTP through SOCKS5
proxy (currently HTTP/HTTPS only).
- [ ] **DNS-over-HTTPS (DoH)** — optional DoH resolver. Must use only
`golang.org/x/` libraries.
proxy (currently HTTP/HTTPS and WebDAV only). `golang.org/x/net/proxy`
provides the SOCKS5 dialer; SFTP is straightforward, FTP needs
dual-connection handling.
- [ ] **DNS-over-HTTPS (DoH)** — optional DoH resolver. Uses
`golang.org/x/net/dns/dnsmessage` for DNS wire format and `net/http`
for the HTTPS transport.
### 🟢 Medium Priority
@@ -105,64 +95,146 @@ TLS, authentication, and verification improvements.
---
## v2.0.0 — New protocols
## v2.0.0 — Package registries & CAS cache
Cloud providers, additional network protocols, and long-lived services.
Turn goget from a file-in-URL-out downloader into a language-aware
package fetcher. The content-addressable cache deduplicates downloads
across protocols and projects — a file is never downloaded twice.
### 🔵 Lower Priority
### 🔴 Supreme Priority
- [ ] **S3** — generic S3-compatible protocol with signature v4 auth
(covers AWS S3, MinIO, and any S3-compatible storage).
- [ ] **Huawei OBS** — Huawei Cloud Object Storage.
- [ ] **Alibaba OSS** — Alibaba Cloud Object Storage Service.
- [ ] **Tencent COS** — Tencent Cloud Object Storage.
- [ ] **Rsync** — incremental file transfer. Written from scratch
(rolling checksum, delta transfer, rsync wire protocol).
- [ ] **TFTP** — trivial file transfer, useful for embedded devices and PXE.
- [ ] **NFS** — NFS v3/v4 client. Written from scratch
(ONC RPC/XDR, portmapper, mount protocol, NFS protocol).
- [ ] **goget-as-a-service** — daemon mode with REST API for remote download
management.
- [ ] **Content-addressable cache (CAS)** — every downloaded file is stored
under its SHA-256 digest. `goget` before hitting the network consults
the cache. Identical content from different URLs, protocols, or
projects hits the cache exactly once.
- [ ] **`crate://` protocol** — `crate://serde@1.0.0` → fetches `.crate`
from crates.io API. `crate://serde@latest` → resolves latest version.
- [ ] **`gem://` protocol** — `gem://rails@7.2` → fetches `.gem` from
RubyGems.org.
- [ ] **`npm://` protocol** — `npm://react@19` → fetches tarball from
registry.npmjs.org.
- [ ] **`git://` protocol** — `git://github.com/user/repo@v1.0` → archive
download via provider API. `@tag`, `@branch`, `@commit` resolution.
### 🟡 High Priority
- [ ] **`oci://` protocol** — `oci://alpine:3.20` → pulls OCI
image layers via registry v2 API, stores in CAS. No daemon required —
goget acts as a standalone OCI image fetcher. Works against any
OCI-compatible registry (Docker Hub, Quay.io, GHCR, …). Complex
protocol, but purely HTTP + JSON — stdlib covers it.
- [ ] **Lockfile import** — `goget crate:// --from-lock Cargo.lock`
downloads and caches every dependency in one shot. Same for
`Gemfile.lock` and `package-lock.json`.
---
## v2.1.0 — Extensibility & UI
## v2.1.0 — Plugin system & extensibility
### 🔵 Lower Priority
### 🔴 Supreme Priority
- [ ] **Plugin system** — protocol handlers as external plugins.
- [ ] **TUI (Terminal User Interface)** — interactive download manager with
live queue, progress, and log views. Keyboard navigation, color-coded
status, inline log viewer.
- [ ] **Web GUI** — browser-based download manager (Vue 3 frontend, REST API,
live progress via SSE).
- [ ] **Download speed test / benchmark mode** — measure maximum throughput
against a target server without saving the file.
- [ ] **Plugin system** — protocol handlers as external binaries.
Discovered from `~/.config/goget/plugins/`. Line-based protocol
over stdin/stdout or Unix socket.
- [ ] **`plugin://` protocol** — transparent proxy to external handlers.
### 🟡 High Priority
- [ ] **Plugin SDK** — Go library (`golang.org/x/` only) for writing plugins.
- [ ] **Plugin registry** — community-maintained index of known plugins.
---
## v2.2.0 — Distributed download mesh
Goget instances on the same LAN discover each other and share cached files.
mDNS/DNS-SD written from scratch on top of `golang.org/x/net/dns/dnsmessage`
and stdlib multicast UDP.
### 🔴 Supreme Priority
- [ ] **`ggtp://` protocol (goget transfer protocol)** — simple
single-round-trip request for a file by SHA-256 from a peer.
- [ ] **LAN discovery (mDNS / DNS-SD)** — `_goget._tcp` announcements.
Written from scratch using `golang.org/x/net/dns/dnsmessage` for
message encoding and stdlib `net` for multicast UDP.
### 🟡 High Priority
- [ ] **Mesh-aware CAS** — query LAN peers for SHA-256 before downloading
from the internet.
- [ ] **Peer authentication** — optional shared secret or mTLS between
trusted nodes.
---
## v3.0.0 — goget-as-a-service
Long-lived daemon with REST API, TUI, and Web GUI.
### 🔴 Supreme Priority
- [ ] **Daemon mode** — `goget daemon` + REST API (localhost-only).
- [ ] **REST API** — `POST /downloads`, `GET /downloads/:id`, SSE progress.
### 🟡 High Priority
- [ ] **Web GUI** — Vue 3 + Vite + Tailwind CSS, daemon REST API + SSE.
- [ ] **Queue pause/resume/reorder** — full download queue management.
### 🟢 Medium Priority
- [ ] **S3 protocol** — as a plugin (see v2.1.0). Signature v4 auth
(HMAC-SHA256, all in stdlib).
- [ ] **Huawei OBS, Alibaba OSS, Tencent COS** — cloud storage plugins.
- [ ] **systemd unit** — for daemon / scheduled mode.
- [ ] **Download speed test / benchmark mode**.
---
## v3.1.0 — Streaming & real-time protocols
### 🟢 Medium Priority
- [ ] **HLS / MPEG-DASH** — download streamed video.
- [ ] **Icecast / SHOUTcast** — capture internet radio.
- [ ] **WebSocket capture** — record WebSocket messages (RFC 6455 framing
from scratch on top of `net/http` hijack).
---
## 💡 Backlog (unscheduled)
Items that need more design, depend on external factors, or are
lower priority than the versioned roadmap.
Items needing more design, dependent on external factors, or deferred
until foundational libraries are written.
### Deferred — waiting for custom libraries
- [ ] **Brotli decompression** — RFC 7932. Will be built as a standalone
Go library first, then integrated into goget. No third-party
dependency — pure Go from scratch.
- [ ] **TUI (Terminal User Interface)** — interactive download manager.
Will be built as a standalone Go TUI toolkit on top of `golang.org/x/term`,
then integrated. Layout engine, widgets, and event loop written from
scratch.
### Performance
- [ ] **Memory-mapped I/O for large files** — `mmap` for checksum
verification of multi-gigabyte downloads.
verification.
- [ ] **Zero-copy for `file://` and `data:` protocols** — `sendfile(2)` /
`splice(2)` where available on Linux.
- [ ] **HTTP/3 (QUIC)** — when Go stdlib gains QUIC support.
### Advanced UX
### Protocols (rare / niche)
- [ ] **Scheduling — pause/resume queued downloads** — ability to pause the
queue, reorder items, and resume later.
- [ ] **Rsync** — rolling checksum, delta transfer, wire protocol from scratch.
- [ ] **TFTP** — trivial file transfer, embedded devices and PXE.
- [ ] **NFS** — NFS v3/v4 client from scratch (ONC RPC/XDR, portmapper).
### Code Quality & Testing
- [ ] **Integration test matrix** — CI against real FTP, SFTP, and WebDAV
servers.
- [ ] **Integration test matrix** — CI against real FTP, SFTP, WebDAV servers.
- [ ] **Coverage threshold increase** — raise from 40% to 60%.
- [ ] **Benchmark suite** — `go test -bench` in CI.
- [ ] **Fuzzing corpus in CI** — store and replay past crashes automatically.
@@ -175,13 +247,10 @@ lower priority than the versioned roadmap.
- [ ] **RPM spec file** — Fedora / RHEL / openEuler packaging.
- [ ] **FreeBSD port** — official FreeBSD ports tree entry.
- [ ] **systemd unit** — for daemon / scheduled mode.
### Documentation
- [ ] **Man page packaging** — already generated by `--generate-man-page`,
needs installation via packaging.
- [ ] **Architecture decision records** — document why specific choices were
made (no short flags, IPv6-first, long flags only, no global state).
- [ ] **Man page packaging** — already generated, needs installation via packaging.
- [ ] **Architecture decision records** — document design choices.
- [ ] **Protocol-specific guides** — one doc per protocol with examples.
- [ ] **Troubleshooting guide** — common errors and solutions.