feat: initial goget release — modern IPv6-first download utility
This commit is contained in:
@@ -0,0 +1,256 @@
|
||||
# Recursive Downloading & Website Mirroring
|
||||
|
||||
goget can recursively download linked pages and mirror entire websites for offline viewing. This document explains the two modes, their configuration, and the underlying architecture.
|
||||
|
||||
## Quick Comparison
|
||||
|
||||
| Feature | Recursive (`--recursive`) | Mirror (`--mirror`) |
|
||||
|---|---|---|
|
||||
| Depth limit | By default (`--max-depth N`) | Infinite |
|
||||
| Link conversion | Optional (`--convert-links`) | Automatic |
|
||||
| Page requisites | Manual (`--page-requisites`) | Automatic |
|
||||
| robots.txt | Respected by default | Respected by default |
|
||||
| Use case | Download a subtree of a site | Full offline archive |
|
||||
|
||||
## Recursive Mode
|
||||
|
||||
Recursive mode follows links from HTML pages and downloads linked resources.
|
||||
|
||||
```bash
|
||||
# Basic recursive download, depth 3
|
||||
goget --url https://example.com/docs/ --recursive --max-depth 3
|
||||
|
||||
# Only PDF files
|
||||
goget --recursive --accept "*.pdf" --url https://example.com/docs/
|
||||
|
||||
# Only images and HTML
|
||||
goget --recursive --accept "image/*,text/html" --url https://example.com/
|
||||
```
|
||||
|
||||
### Link Filtering
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
HTML[Parse HTML page]:::accent1
|
||||
Extract["Extract all links\n(a, img, link, script)"]:::accent1
|
||||
FilterByDomain{"Domain match?"}:::accent7
|
||||
FilterByPattern{"Accept pattern\nmatch?"}:::accent7
|
||||
FilterByDepth{"Depth ≤ max?"}:::accent7
|
||||
FilterByParent{"No-parent\ncheck?"}:::accent7
|
||||
ExcludeFilter{"Not in exclude\nlist?"}:::accent7
|
||||
Enqueue["Add to download queue"]:::accent1
|
||||
Skip["Skip"]:::accent4
|
||||
|
||||
HTML --> Extract --> FilterByDomain
|
||||
FilterByDomain -->|Yes| FilterByPattern
|
||||
FilterByDomain -->|No, --span-hosts| FilterByPattern
|
||||
FilterByDomain -->|No| Skip
|
||||
FilterByPattern -->|Yes| FilterByDepth
|
||||
FilterByPattern -->|No| Skip
|
||||
FilterByDepth -->|Yes| FilterByParent
|
||||
FilterByDepth -->|No| Skip
|
||||
FilterByParent -->|Pass| ExcludeFilter
|
||||
FilterByParent -->|Fail| Skip
|
||||
ExcludeFilter -->|Pass| Enqueue
|
||||
ExcludeFilter -->|Fail| Skip
|
||||
|
||||
classDef accent1 fill:#22C55E,stroke:#16A34A,color:#fff
|
||||
classDef accent4 fill:#EF4444,stroke:#DC2626,color:#fff
|
||||
classDef accent7 fill:#64748B,stroke:#475569,color:#fff
|
||||
```
|
||||
|
||||
### Accept Patterns
|
||||
|
||||
The `--accept` flag supports two types:
|
||||
|
||||
| Type | Example | Matches |
|
||||
|---|---|---|
|
||||
| **Glob** | `*.pdf` | Filenames ending in `.pdf` |
|
||||
| **Glob** | `*.html,*.css` | Multiple patterns (comma-separated) |
|
||||
| **MIME** | `text/html` | Exact MIME type |
|
||||
| **MIME** | `image/*` | Any MIME in the `image/` category |
|
||||
|
||||
### Domain Control
|
||||
|
||||
| Flag | Effect |
|
||||
|---|---|
|
||||
| (default) | Only follow links within the starting domain |
|
||||
| `--span-hosts` | Follow links to any domain |
|
||||
| `--domains a.com,b.com` | Restrict to specific domains |
|
||||
| `--follow-external` | Follow external links (aliases for `--span-hosts`) |
|
||||
| `--no-parent` | Don't ascendant above the starting URL path |
|
||||
| `--recursive-parallel` | `<N>` | Number of concurrent file downloads in recursive mode across all protocols (`0` = sequential). HTTP, WebDAV, FTP, and SFTP each get a worker pool bounded by a semaphore; subdirectory recursion propagates the limit. Connection pooling (FTP, SFTP) opens N independent control connections |
|
||||
|
||||
### Request Throttling
|
||||
|
||||
```bash
|
||||
# Fixed 2-second delay between requests
|
||||
goget --recursive --wait 2s --url https://example.com/docs/
|
||||
|
||||
# Randomize delay (0.5x – 1.5x of --wait)
|
||||
goget --recursive --wait 2s --random-wait --url https://example.com/docs/
|
||||
```
|
||||
|
||||
### Page Requisites
|
||||
|
||||
`--page-requisites` downloads CSS, JavaScript, and images needed to render each HTML page:
|
||||
|
||||
```bash
|
||||
goget --recursive --page-requisites --url https://example.com/page.html
|
||||
```
|
||||
|
||||
This parses `<link>`, `<script>`, `<img>`, `<source>`, and `<video>` tags and downloads their `src`/`href` targets.
|
||||
|
||||
## Mirror Mode
|
||||
|
||||
| `--dry-run` | — | List all files that would be downloaded in recursive/mirror mode without saving to disk |
|
||||
|
||||
`--mirror` is a shorthand for `--recursive --convert-links --page-requisites --infinite-depth`:
|
||||
|
||||
```bash
|
||||
# Full site mirror
|
||||
goget --url https://example.com --mirror --output ./mirror
|
||||
|
||||
# With link conversion for offline viewing
|
||||
goget --url https://example.com --mirror --convert-links --output ./mirror
|
||||
```
|
||||
|
||||
### Link Conversion
|
||||
|
||||
When `--convert-links` is enabled, HTML links are rewritten for local offline viewing:
|
||||
|
||||
```
|
||||
Before: <a href="https://example.com/about/">About</a>
|
||||
After: <a href="./about/index.html">About</a>
|
||||
|
||||
Before: <img src="https://example.com/img/logo.png">
|
||||
After: <img src="./img/logo.png">
|
||||
```
|
||||
|
||||
The link rewriter handles:
|
||||
- Absolute URLs → relative paths
|
||||
- Protocol-relative URLs (`//example.com/...`)
|
||||
- Root-relative URLs (`/about/`)
|
||||
- CSS `url()` references
|
||||
|
||||
### Robots.txt
|
||||
|
||||
goget respects `robots.txt` by default. Use `--no-robots` to ignore it:
|
||||
|
||||
```bash
|
||||
goget --mirror --no-robots --url https://example.com
|
||||
```
|
||||
|
||||
### Asset Control
|
||||
|
||||
```bash
|
||||
# Mirror but skip CSS/JS/images
|
||||
goget --mirror --no-mirror-assets --url https://example.com
|
||||
|
||||
# Only mirror specific file types
|
||||
goget --mirror --accept "*.html,*.jpg" --url https://example.com
|
||||
```
|
||||
|
||||
## Output Structure
|
||||
|
||||
Mirrored sites preserve the URL path structure:
|
||||
|
||||
```
|
||||
mirror/
|
||||
├── index.html
|
||||
├── about/
|
||||
│ └── index.html
|
||||
├── blog/
|
||||
│ ├── index.html
|
||||
│ └── post-1.html
|
||||
├── css/
|
||||
│ └── style.css
|
||||
└── img/
|
||||
└── logo.png
|
||||
```
|
||||
|
||||
Use `--cut-dirs N` to strip directory components:
|
||||
|
||||
```bash
|
||||
# Original: https://example.com/pub/docs/file.html
|
||||
# Without cut: ./mirror/pub/docs/file.html
|
||||
# With --cut-dirs 2: ./mirror/doc/file.html
|
||||
goget --mirror --cut-dirs 2 --url https://example.com/pub/docs/
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
### Crawler (`internal/recursive`)
|
||||
|
||||
```go
|
||||
type CrawlerConfig struct {
|
||||
MaxDepth int
|
||||
FollowExternal bool
|
||||
ExcludePatterns []string
|
||||
IncludePatterns []string
|
||||
AcceptPatterns []string
|
||||
NoParent bool
|
||||
PageRequisites bool
|
||||
ConvertLinks bool
|
||||
OutputDir string
|
||||
Delay time.Duration
|
||||
RandomWait bool
|
||||
MaxWorkers int // concurrent download workers (0 = sequential)
|
||||
}
|
||||
```
|
||||
|
||||
The crawler:
|
||||
1. Downloads an HTML page
|
||||
2. Parses it with `golang.org/x/net/html`
|
||||
3. Extracts all links (`a[href]`, `img[src]`, `link[href]`, `script[src]`)
|
||||
4. Filters against accept/reject patterns, domain rules, and depth limits
|
||||
5. Enqueues matching URLs for download
|
||||
6. Applies configurable delay between requests
|
||||
|
||||
### Mirror (`internal/mirror`)
|
||||
|
||||
The mirror wraps the crawler with additional configuration:
|
||||
- Infinite depth (or configurable via `MirrorConfig.MaxDepth`)
|
||||
- Automatic page requisites
|
||||
- Automatic link conversion
|
||||
- robots.txt compliance
|
||||
|
||||
### Link Rewriting (`internal/linkrewrite`)
|
||||
|
||||
After the download completes, links are rewritten in-place:
|
||||
1. Read each HTML file
|
||||
2. Parse with `golang.org/x/net/html`
|
||||
3. Identify all `href` and `src` attributes
|
||||
4. Convert absolute URLs to relative paths
|
||||
5. Write back the modified HTML (backup `.orig` if `--backup-converted`)
|
||||
|
||||
## Timestamping
|
||||
|
||||
`--timestamping` downloads a file only if the server copy is newer than the local copy:
|
||||
|
||||
```bash
|
||||
goget --timestamping --url https://example.com/file.zip
|
||||
```
|
||||
|
||||
goget compares the `Last-Modified` header against the local file's modification time.
|
||||
|
||||
## WARC Archiving
|
||||
|
||||
For legal and archival compliance, output can be written in WARC format:
|
||||
|
||||
```bash
|
||||
goget --warc-file archive.warc --url https://example.com/page.html
|
||||
```
|
||||
|
||||
Each downloaded resource gets a separate WARC record with:
|
||||
- Request metadata (URL, timestamp, headers)
|
||||
- Response metadata (status code, content type, headers)
|
||||
- Raw response body
|
||||
|
||||
## Performance Tips
|
||||
|
||||
- **Limit depth** — `--max-depth 3` prevents runaway recursion on large sites
|
||||
- **Filter aggressively** — Use `--accept "*.pdf,*.html"` and `--reject "*.zip"` to narrow scope
|
||||
- **Use delay** — `--wait 500ms` prevents server overload and IP blocks
|
||||
- **Limit domains** — `--domains example.com` prevents crawling external CDNs
|
||||
- **Skip assets** — `--no-mirror-assets` for text-only archives
|
||||
Reference in New Issue
Block a user