feat: initial goget release — modern IPv6-first download utility

2026-06-26 21:21:58 +02:00
commit 53db81e1f7
147 changed files with 45931 additions and 0 deletions
@@ -0,0 +1,304 @@
+# Contributing
+
+Thank you for considering contributing to **goget** — a modern IPv6-first
+download utility built on the Go standard library.
+
+## Development Setup
+
+### Requirements
+
+- Go 1.26+
+- Linux (amd64, arm64, riscv64, loong64) or FreeBSD (amd64, arm64, riscv64)
+
+### Quick Start
+
+```bash
+git clone https://codeberg.org/petrbalvin/goget.git
+cd goget
+just install        # go mod download
+just test           # go test -race -count=1 ./...
+just build          # go vet + gofmt check + build → bin/goget
+```
+
+### Running Tests
+
+```bash
+just test
+```
+
+The test suite (`*_test.go` files alongside the code) covers:
+
+- Protocol handlers (HTTP, FTP, SFTP, WebDAV, file, data, Gopher, Gemini).
+- Transport layer (IPv6-first dialer, proxy, retry, rate limiting).
+- Checksum verification, PGP signature handling, HSTS cache.
+- Recursive downloading, link rewriting, Metalink parsing.
+- Configuration loading, merging, and sanitization.
+- CLI flag parsing, output formatting, progress bar.
+
+### Linting
+
+```bash
+just build        # go vet + gofmt check (also part of CI)
+```
+
+CI runs both `go vet ./...` and a `gofmt -l` diff check. These are enforced —
+make sure they pass before submitting a pull request.
+
+## Code Style
+
+- Follow [Effective Go](https://go.dev/doc/effective_go) conventions.
+- Keep lines under 100 characters where practical.
+- Use `camelCase` for variables and functions, `PascalCase` for exported
+  symbols.
+- Prefix every file with a one-line comment describing its purpose.
+- Use build tags `//go:build linux || freebsd` on all source files.
+- Use long flags in CLI (`--header` not `-H`).
+- Prefer the Go standard library over external dependencies. External
+  dependencies are only allowed from `golang.org/x/` unless justified.
+- Error messages in English, lowercase, no trailing period, always with context
+  (`open config: permission denied`).
+- `gofmt` all code before committing.
+
+### Testing
+
+- Write tests alongside the code in `_test.go` files.
+- Use table-driven tests for multiple test cases.
+- Include both happy path and error path tests.
+- Name tests following Go convention: `TestFunctionName_Scenario`.
+- Use `testing.F` for fuzz tests in security-critical parsing code.
+
+## Commit Conventions
+
+We follow [Conventional Commits](https://www.conventionalcommits.org/).
+
+| Type       | Use                                          |
+|------------|----------------------------------------------|
+| `feat`     | New feature                                  |
+| `fix`      | Bug fix                                      |
+| `refactor` | Restructuring without behaviour change       |
+| `docs`     | Documentation                                |
+| `test`     | Adding or updating tests                     |
+| `chore`    | Maintenance, CI, tooling                     |
+| `perf`     | Performance improvement                      |
+
+```
+feat: add Gemini protocol support
+fix: handle timeout during parallel chunk downloads
+refactor: unify transport dialer configuration
+docs: add README quick start examples
+test: add edge cases for cron schedule parser
+chore: update golang.org/x/crypto
+```
+
+- English, lowercase subject, imperative mood (`add`, not `added`).
+- No trailing period, max 72 characters.
+- No version numbers in commit messages — versions belong to tags.
+
+## Pull Request Flow
+
+1. Create a feature branch from `development`.
+2. Make your changes with Conventional-Commits messages.
+3. Record your changes under the `## [development]` section at the top of
+   `CHANGELOG.md`, using the Keep a Changelog categories (`Added`, `Changed`,
+   `Fixed`, etc.).
+4. Ensure `just build` and `just test` pass.
+5. Add or update tests for your change.
+6. Update documentation if the public API, configuration, or behaviour changes:
+   - `README.md` — high-level overview.
+   - `docs/configuration.md` — config keys.
+   - `docs/api.md` — Go library API.
+   - `docs/cli-reference.md` — CLI flags.
+   - `docs/architecture.md` — structural changes.
+   - `docs/security.md` — anything touching auth, TLS, or verification.
+7. Open a pull request against `development`. Releases are cut by merging
+   `development` into `main` and pushing a `vX.Y.Z` tag — see
+   [Cutting a release](#cutting-a-release).
+
+## Project Structure
+
+```
+goget/
+├── cmd/
+│   └── goget/          # CLI entry point
+│       ├── main.go         # Core run loop, flag setup, orchestrator
+│       ├── commands.go     # Command handlers (download, upload, queue, config, etc.)
+│       ├── flags.go        # CLI flag definitions (struct + defineFlags)
+│       ├── completion.go   # Shell completion generators (bash, zsh, fish)
+│       ├── man_page.go     # Man page generation
+│       ├── schedule.go     # Cron and datetime scheduling
+│       └── verify.go       # Checksum and PGP verification
+├── internal/
+│   ├── archive/         # Archive extraction (tar, targz, tarbz2, zip)
+│   ├── auth/            # Authentication (basic, digest, OAuth 2.0, netrc)
+│   ├── cli/             # CLI utilities (help text, progress bar, color output)
+│   ├── compression/     # Decompression (gzip, bzip2, zlib, flate, lzw)
+│   ├── config/          # TOML config loading, saving, and merging
+│   ├── cookie/          # Netscape-format cookie jar
+│   ├── core/            # Core types (DownloadRequest, Result, Protocol interface, errors)
+│   ├── crypto/          # Checksum verification, PGP, certificate handling
+│   ├── format/          # Human-readable formatting (bytes, speed)
+│   ├── hsts/            # HSTS cache (RFC 6797)
+│   ├── log/             # Structured logger (text/JSON output, level filtering)
+│   ├── linkrewrite/     # HTML link rewriting for offline mirroring
+│   ├── metalink/        # Metalink (multi-source downloads)
+│   ├── mirror/          # Website mirroring logic
+│   ├── output/          # Output writing (atomic writes, resume metadata)
+│   ├── protocol/        # Protocol abstraction layer
+│   │   ├── http/        # HTTP/HTTPS client, parallel download, upload
+│   │   ├── ftp/         # FTP/FTPS client
+│   │   ├── sftp/        # SFTP client
+│   │   ├── file/        # Local file:// protocol
+│   │   ├── dataurl/     # data: URI protocol
+│   │   ├── gopher/      # Gopher protocol
+│   │   ├── gemini/      # Gemini protocol
+│   │   └── webdav/      # WebDAV protocol
+│   ├── queue/           # Download queue management
+│   ├── recursive/       # Recursive download (HTML parser, URL filter, crawler)
+│   ├── warc/            # WARC archiving writer
+│   └── transport/       # HTTP transport (dialer, proxy, TLS, retry, rate limiting)
+├── pkg/
+│   └── api/             # Public library API (Client, Download, Upload, Queue)
+└── docs/                # Architecture and design documentation
+```
+
+## Architecture Overview
+
+```mermaid
+graph TD
+    CLI[cmd/goget/ - CLI Entry]
+    Flags[Flags parsing]
+    Config[internal/config/]
+    Core[internal/core/ - Types & Protocol Interface]
+    Registry[internal/protocol/ - Registry]
+    HTTP[internal/protocol/http/]
+    FTP[internal/protocol/ftp/]
+    SFTP[internal/protocol/sftp/]
+    FILE[internal/protocol/file/]
+    DataURL[internal/protocol/dataurl/]
+    Gopher[internal/protocol/gopher/]
+    Gemini[internal/protocol/gemini/]
+    WebDAV[internal/protocol/webdav/]
+    Transport[internal/transport/ - Dialer, Proxy, TLS]
+    Output[internal/output/ - Atomic write, Resume]
+    Crypto[internal/crypto/ - Checksum, PGP]
+    Auth[internal/auth/]
+    Cookie[internal/cookie/]
+    HSTS[internal/hsts/]
+    Recursive[internal/recursive/]
+    Mirror[internal/mirror/]
+    Metalink[internal/metalink/]
+    QueueInt[internal/queue/]
+    Archive[internal/archive/]
+    API[pkg/api/ - Public Library]
+
+    CLI --> Flags
+    CLI --> Config
+    CLI --> Core
+    CLI --> Auth
+    CLI --> Crypto
+    CLI --> Recursive
+    CLI --> Mirror
+    CLI --> Metalink
+    CLI --> QueueInt
+    CLI --> Archive
+    Core --> Registry
+    Registry --> HTTP
+    Registry --> FTP
+    Registry --> SFTP
+    Registry --> FILE
+    Registry --> DataURL
+    Registry --> Gopher
+    Registry --> Gemini
+    Registry --> WebDAV
+    HTTP --> Transport
+    HTTP --> Cookie
+    HTTP --> HSTS
+    HTTP --> Output
+    API --> Core
+    API --> Registry
+```
+
+### Key Design Decisions
+
+- **Protocol abstraction** — Every network protocol implements `core.Protocol`
+  and registers with a global registry. The CLI and API resolve protocols by URL
+  scheme at runtime.
+- **IPv6-first** — The transport dialer prefers IPv6 addresses and falls back to
+  IPv4 only when no IPv6 address is available.
+- **Parallel downloads** — Files over 100 MB are automatically split into
+  chunks. Each chunk downloads via a separate connection with byte-range
+  requests.
+- **Atomic output** — Downloads write to a temporary file first, then atomically
+  rename on completion. Resume metadata is stored alongside the partial file.
+- **Config merging** — CLI flags override config file values, which override
+  defaults. The `config.Merge()` method applies CLI overrides after loading.
+- **No global state** — The CLI is the only singleton. Internal packages are
+  designed for testability with injectable dependencies.
+
+## Testing Strategy
+
+| Layer          | Approach                                                   | Tools               |
+|----------------|------------------------------------------------------------|---------------------|
+| Unit tests     | Table-driven tests for parsers, validators, handlers       | `testing` stdlib    |
+| Integration    | HTTP/HTTPS against local test servers, FTP against mock    | `net/http/httptest` |
+| Fuzz tests     | Random input for checksum parsing, Metalink, HSTS cache    | `testing.F`         |
+| Race detection | All tests run with `-race` in CI                           | `go test -race`     |
+
+## CI
+
+goget uses **Forgejo Actions**; both workflows live under `.forgejo/workflows/`
+and run on the `codeberg-small` / `codeberg-medium` runners.
+
+### Test — `.forgejo/workflows/test.yml`
+
+Triggered on push and pull requests targeting `development`.
+
+| Job   | What it does                                  |
+|-------|-----------------------------------------------|
+| `vet` | `go vet ./...` (must report zero warnings)    |
+| `test`| `go test -race -count=1 ./...` + coverage gate |
+| `fuzz`| `go test -fuzz` on every `FuzzXxx` function   |
+
+Run the same locally before opening a PR:
+
+```bash
+go vet ./...
+just test
+```
+
+### Release — `.forgejo/workflows/release.yml`
+
+Triggered by pushing a `v*` tag. It verifies the tag matches
+`internal/core.Version`, runs the quality gate (go vet + tests), builds static
+binaries for all 7 platform targets, extracts the matching `CHANGELOG.md`
+section, creates a Forgejo release via the REST API, and attaches the binaries.
+
+Requires a `FORGEJO_TOKEN` secret with permission to create releases.
+
+### Cutting a release
+
+1. Bump the `Version` var in `internal/core/version.go`.
+2. Rename `## [development]` to `## [X.Y.Z] — YYYY-MM-DD` at the top of
+   `CHANGELOG.md`, and add a fresh empty `## [development]` section above it.
+3. Ensure CI is green on `development`.
+4. Merge `development` into `main` (the only time `main` changes outside a tag).
+5. Tag and push:
+   ```bash
+   git tag -a vX.Y.Z -m "goget vX.Y.Z"
+   git push origin main
+   git push origin vX.Y.Z
+   ```
+6. The release workflow builds the binaries and publishes the release from the
+   CHANGELOG section.
+
+## Report a Bug
+
+Open an issue at <https://codeberg.org/petrbalvin/goget/issues> with:
+
+- Goget version (`goget --version`).
+- Operating system and architecture.
+- Command used and full output.
+- Expected vs actual behaviour.
+
+For **security issues**, please email **opensource@petrbalvin.org** rather than
+opening a public issue.