petrbalvin/goget
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity
Files
main
goget/CONTRIBUTING.md
T

12 KiB
Raw Permalink Blame History

Contributing

Thank you for considering contributing to goget — a modern IPv6-first download utility built on the Go standard library.

Development Setup

Requirements

  • Go 1.26+
  • Linux (amd64, arm64, riscv64, loong64) or FreeBSD (amd64, arm64, riscv64)

Quick Start

git clone https://codeberg.org/petrbalvin/goget.git
cd goget
just install        # go mod download
just test           # go test -race -count=1 ./...
just build          # go vet + gofmt check + build → bin/goget

Running Tests

just test

The test suite (*_test.go files alongside the code) covers:

  • Protocol handlers (HTTP, FTP, SFTP, WebDAV, file, data, Gopher, Gemini).
  • Transport layer (IPv6-first dialer, proxy, retry, rate limiting).
  • Checksum verification, PGP signature handling, HSTS cache.
  • Recursive downloading, link rewriting, Metalink parsing.
  • Configuration loading, merging, and sanitization.
  • CLI flag parsing, output formatting, progress bar.

Linting

just build        # go vet + gofmt check (also part of CI)

CI runs both go vet ./... and a gofmt -l diff check. These are enforced — make sure they pass before submitting a pull request.

Code Style

  • Follow Effective Go conventions.
  • Keep lines under 100 characters where practical.
  • Use camelCase for variables and functions, PascalCase for exported symbols.
  • Prefix every file with a one-line comment describing its purpose.
  • Use build tags //go:build linux || freebsd on all source files.
  • Use long flags in CLI (--header not -H).
  • Prefer the Go standard library over external dependencies. External dependencies are only allowed from golang.org/x/ unless justified.
  • Error messages in English, lowercase, no trailing period, always with context (open config: permission denied).
  • gofmt all code before committing.

Testing

  • Write tests alongside the code in _test.go files.
  • Use table-driven tests for multiple test cases.
  • Include both happy path and error path tests.
  • Name tests following Go convention: TestFunctionName_Scenario.
  • Use testing.F for fuzz tests in security-critical parsing code.

Commit Conventions

We follow Conventional Commits.

Type Use
feat New feature
fix Bug fix
refactor Restructuring without behaviour change
docs Documentation
test Adding or updating tests
chore Maintenance, CI, tooling
perf Performance improvement 
feat: add Gemini protocol support
fix: handle timeout during parallel chunk downloads
refactor: unify transport dialer configuration
docs: add README quick start examples
test: add edge cases for cron schedule parser
chore: update golang.org/x/crypto
  • English, lowercase subject, imperative mood (add, not added).
  • No trailing period, max 72 characters.
  • No version numbers in commit messages — versions belong to tags.

Pull Request Flow

  1. Create a feature branch from development.
  2. Make your changes with Conventional-Commits messages.
  3. Record your changes under the ## [development] section at the top of CHANGELOG.md, using the Keep a Changelog categories (Added, Changed, Fixed, etc.).
  4. Ensure just build and just test pass.
  5. Add or update tests for your change.
  6. Update documentation if the public API, configuration, or behaviour changes:
    • README.md — high-level overview.
    • docs/configuration.md — config keys.
    • docs/api.md — Go library API.
    • docs/cli-reference.md — CLI flags.
    • docs/architecture.md — structural changes.
    • docs/security.md — anything touching auth, TLS, or verification.
  7. Open a pull request against development. Releases are cut by merging development into main and pushing a vX.Y.Z tag — see Cutting a release.

Project Structure

goget/
├── cmd/
│   └── goget/              # CLI entry point
│       ├── main.go         # Core run loop, flag setup, orchestrator
│       ├── app.go          # Application wiring
│       ├── commands.go     # Command handlers (download, upload, queue, config, etc.)
│       ├── flags.go        # CLI flag definitions (struct + defineFlags)
│       ├── completion.go   # Shell completion generators (bash, zsh, fish)
│       ├── man_page.go     # Embedded manpage source and --generate-man-page
│       ├── setup.go        # One-time CLI setup helpers
│       ├── schedule.go     # Cron and datetime scheduling
│       └── verify.go       # Checksum and PGP verification
├── internal/
│   ├── archive/            # Archive extraction (tar, targz, tarbz2, zip)
│   ├── auth/               # Authentication (basic, digest, OAuth 2.0, netrc)
│   ├── cli/                # CLI utilities (help text, progress bar, color output)
│   ├── compression/        # Decompression (gzip, bzip2, zlib, flate, lzw)
│   ├── config/             # TOML config loading, saving, and merging
│   ├── cookie/             # Netscape-format cookie jar
│   ├── core/               # Core types (DownloadRequest, Result, Protocol interface, errors)
│   ├── crypto/             # Checksum verification, PGP, certificate handling
│   ├── format/             # Human-readable formatting (bytes, speed)
│   ├── hsts/               # HSTS cache (RFC 6797)
│   ├── linkrewrite/        # HTML link rewriting for offline mirroring
│   ├── log/                # Structured logger (text/JSON output, level filtering)
│   ├── metalink/           # Metalink (multi-source downloads)
│   ├── mirror/             # Website mirroring logic
│   ├── output/             # Output writing (atomic writes, resume metadata)
│   ├── pool/               # Unified recursive worker pool (FTP, SFTP, WebDAV)
│   ├── protocol/           # Protocol abstraction layer
│   │   ├── http/           # HTTP/HTTPS client, parallel download, upload
│   │   ├── ftp/            # FTP/FTPS client
│   │   ├── sftp/           # SFTP client
│   │   ├── file/           # Local file:// protocol
│   │   ├── dataurl/        # data: URI protocol
│   │   ├── gopher/         # Gopher protocol
│   │   ├── gemini/         # Gemini protocol
│   │   ├── webdav/         # WebDAV protocol
│   │   └── register/       # Single registration entry point
│   ├── queue/              # Download queue management
│   ├── recursive/          # Recursive download (HTML parser, URL filter, crawler)
│   ├── transport/          # HTTP transport (dialer, proxy, TLS, retry, rate limiting)
│   └── warc/               # WARC archiving writer
├── pkg/
│   └── api/                # Public library API (Client, Download, Upload, Queue)
└── docs/                   # Architecture and design documentation

Architecture Overview

graph TD
    CLI[cmd/goget/ - CLI Entry]
    Flags[Flags parsing]
    Config[internal/config/]
    Core[internal/core/ - Types & Protocol Interface]
    Registry[internal/protocol/ - Registry]
    HTTP[internal/protocol/http/]
    FTP[internal/protocol/ftp/]
    SFTP[internal/protocol/sftp/]
    FILE[internal/protocol/file/]
    DataURL[internal/protocol/dataurl/]
    Gopher[internal/protocol/gopher/]
    Gemini[internal/protocol/gemini/]
    WebDAV[internal/protocol/webdav/]
    Transport[internal/transport/ - Dialer, Proxy, TLS]
    Output[internal/output/ - Atomic write, Resume]
    Crypto[internal/crypto/ - Checksum, PGP]
    Auth[internal/auth/]
    Cookie[internal/cookie/]
    HSTS[internal/hsts/]
    Recursive[internal/recursive/]
    Mirror[internal/mirror/]
    Metalink[internal/metalink/]
    QueueInt[internal/queue/]
    Archive[internal/archive/]
    Pool[internal/pool/ - Unified worker pool]
    Log[internal/log/ - Structured logger]
    API[pkg/api/ - Public Library]

    CLI --> Flags
    CLI --> Config
    CLI --> Core
    CLI --> Log
    CLI --> Auth
    CLI --> Crypto
    CLI --> Recursive
    CLI --> Mirror
    CLI --> Metalink
    CLI --> QueueInt
    CLI --> Archive
    Core --> Registry
    Registry --> HTTP
    Registry --> FTP
    Registry --> SFTP
    Registry --> FILE
    Registry --> DataURL
    Registry --> Gopher
    Registry --> Gemini
    Registry --> WebDAV
    HTTP --> Transport
    HTTP --> Cookie
    HTTP --> HSTS
    HTTP --> Output
    FTP --> Pool
    SFTP --> Pool
    WebDAV --> Pool
    API --> Core
    API --> Registry

Key Design Decisions

  • Protocol abstraction — Every network protocol implements core.Protocol and registers with a global registry. The CLI and API resolve protocols by URL scheme at runtime.
  • IPv6-first — The transport dialer prefers IPv6 addresses and falls back to IPv4 only when no IPv6 address is available.
  • Parallel downloads — Files over 100 MB are automatically split into chunks. Each chunk downloads via a separate connection with byte-range requests.
  • Atomic output — Downloads write to a temporary file first, then atomically rename on completion. Resume metadata is stored alongside the partial file.
  • Config merging — CLI flags override config file values, which override defaults. The config.Merge() method applies CLI overrides after loading.
  • No global state — The CLI is the only singleton. Internal packages are designed for testability with injectable dependencies.

Testing Strategy

Layer Approach Tools
Unit tests Table-driven tests for parsers, validators, handlers testing stdlib
Integration HTTP/HTTPS against local test servers, FTP against mock net/http/httptest
Fuzz tests Random input for checksum parsing, Metalink, HSTS cache testing.F
Race detection All tests run with -race in CI go test -race

CI

goget uses Forgejo Actions; both workflows live under .forgejo/workflows/ and run on the codeberg-small / codeberg-medium runners.

Test — .forgejo/workflows/test.yml

Triggered on push and pull requests targeting development.

Job What it does
vet go vet ./... (must report zero warnings)
test go test -race -count=1 ./... + coverage gate
fuzz go test -fuzz on every FuzzXxx function

Run the same locally before opening a PR:

go vet ./...
just test

Release — .forgejo/workflows/release.yml

Triggered by pushing a v* tag. It verifies the tag matches internal/core.Version, runs the quality gate (go vet + tests), builds static binaries for all 7 platform targets, extracts the matching CHANGELOG.md section, creates a Forgejo release via the REST API, and attaches the binaries.

Requires a FORGEJO_TOKEN secret with permission to create releases.

Cutting a release

  1. Bump the Version var in internal/core/version.go.
  2. Rename ## [development] to ## [X.Y.Z] — YYYY-MM-DD at the top of CHANGELOG.md, and add a fresh empty ## [development] section above it.
  3. Ensure CI is green on development.
  4. Merge development into main (the only time main changes outside a tag).
  5. Tag and push: 
    git tag -a vX.Y.Z -m "goget vX.Y.Z"
git push origin main
git push origin vX.Y.Z
  6. The release workflow builds the binaries and publishes the release from the CHANGELOG section.

Report a Bug

Open an issue at https://codeberg.org/petrbalvin/goget/issues with:

  • Goget version (goget --version).
  • Operating system and architecture.
  • Command used and full output.
  • Expected vs actual behaviour.

For security issues, please email opensource@petrbalvin.org rather than opening a public issue.

Reference in New Issue View Git Blame Copy Permalink