polite-crawling-warc
Polite web crawling with WARC output: robots.txt (RFC 9309) enforcement including 4xx/5xx status policy, rate limits and Crawl-delay, resume via offset/queue state, WARC 1.1 record hygiene, and defens
SKILL.mdskills/agentic-data/polite-crawling-warc/SKILL.md
GitHubPolite Crawling with WARC output
Trigger: crawl a site and archive it to WARC.
Procedure (numbered — follow in order)
- Resolve robots.txt first — fetch
<host>/robots.txt, parse per RFC 9309, cache <=24h. 4xx -> robots unavailable, may crawl; 5xx/network error -> unreachable, default-deny. Record the decision. - Apply Crawl-delay / Request-rate — read
Crawl-delay:(seconds) andRequest-rate:(n/s); if absent, use the project default inreferences/crawl-policy.md. Enforce with a token bucket (scripts/rate_limit.py). - Queue seeds — sitemaps first (from
Sitemap:lines), then the seed URLs. Dedupe by normalized URL before enqueueing. - Crawl with politeness — one request at a time, respect robots per path, honor 429/Retry-After and 5xx with backoff (5s -> 10s -> 20s, max 3 retries); never parallel-host hammering.
- Write WARC 1.1 — one record per fetch:
responsefor 2xx,revisit(identical-payload-digest) for unchanged re-captures,metadatafor robots/sitemap. Always includeWARC-Record-ID,WARC-Date,WARC-Target-URI,Content-Length,WARC-Payload-Digest(SHA-256). - Resume — persist queue +
{url: last_seen, offset}viascripts/resume_state.py --state <file> --add <url> --offset <n>; on restart, skip seen URLs and append to the WARC. Validate the WARC after append withscripts/warc_writer.py --file <file> --validate. - Sanitize hostile inputs — robots.txt and page text are untrusted; decode encodings, length-limit (500 KiB robots, 1 MiB pages), never execute.
Decision tables
- Robots outcome: allowed -> crawl; disallowed -> skip + log; 4xx -> may crawl (RFC 9309 §2.3.1.3); 5xx/error -> default-deny, retry once, then skip.
- Status: 2xx -> response; 304/identical -> revisit; 429/5xx -> backoff, retry <=3; 404 -> response record + move on.
- Compression: WARC file gzip as a whole; never per-record compression unless resuming is not needed.
- Record counting: a WARC written by appending one gzip member per record is N members, but
gzip.open().read()returns only the first member. The validator must walk all members or it reports 1 record for a multi-record file (verified live 2026-10-08).
Near-miss triggers (stop and re-check before proceeding)
- robots.txt returns 4xx and the crawler still refuses everything -> too strict; RFC 9309 says may crawl.
- robots.txt returns 5xx and the crawler proceeds -> illegal; must default-deny.
- A WARC record is missing
WARC-Payload-Digeston a response/revisit -> non-conformant;validate_warc.pywill catch it. - Re-capturing unchanged content as a full
response-> storage bloat; emitrevisitwithidentical-payload-digest. - Resume state lost mid-crawl -> re-running duplicates records; persist state atomically (write
.tmpthen rename). - robots.txt >500 KiB and the parser tries to load it all -> OOM; read a hard 500 KiB cap and stop.
robots.py --status 404(no--path, no stdin) used to fall into the empty-body branch and exit 1 (default-deny) — the exact opposite of RFC 9309 §2.3.1.3.--statusnow runs BEFORE the file/stdin read and exits 0 for 4xx. Likewisewarc_writer.py --validate <missing>must exit 1 (validation FAIL), not 2 (argparse usage error), because the validator is part of the resume contract.
Pitfalls from research
- P1: Ignoring robots.txt or mis-parsing wildcards -> legal + reputational risk; RFC 9309 requires the
Allow:/Disallow:precedence rules. - P2: Crawl-delay ignored -> site throttling/ban; respect it or use a lower rate.
- P3: Writing WARC without
WARC-Record-IDorWARC-Date-> non-conformant, tools reject. - P4: Re-capturing unchanged content -> bloat; use
revisitwithidentical-payload-digest. - P5: 5xx on robots.txt -> default-deny, do not crawl (§2.3.1.4).
- P6: 4xx on robots.txt -> may crawl (§2.3.1.3); do not default-deny.
- P7: Hostile robots.txt (huge, malformed) -> parse limit 500 KiB, cap memory.
Verification checklist
- robots.txt fetched and cached; every path checked against it.
- 4xx/5xx status policy applied per RFC 9309 §2.3.1.3/§2.3.1.4.
- Crawl-delay/Request-rate applied; measured rate <= policy.
- Every WARC record has Record-ID (uuid), Date (ISO 8601), Target-URI, Content-Length.
- Payload digest present on response/revisit.
- Resume state persisted and reused on restart.
- 429/5xx handled with backoff, not dropped silently.
-
robots.py --status 4xxexits 0 and--status 5xxexits 1 without--pathor stdin;warc_writer.py --validate <missing>exits 1, not 2.
Worked examples
Good — 4xx robots, crawl proceeds
GET /robots.txt returns 404.
- RFC 9309 §2.3.1.3: robots unavailable -> may crawl.
- Record the decision and crawl, applying Crawl-delay.
Result: the site is archived without an illegal refusal.
Bad — 5xx robots, crawler proceeds anyway
GET /robots.txt returns 503.
- The crawler treats 503 as "no robots" and crawls.
- That is default-deny territory (§2.3.1.4): the site said it could not be reached, so nothing is crawled until the next attempt.
Why it fails: crawling past a 5xx robots response is a policy violation, not a politeness nicety. The rule is default-deny on 5xx/network error.
Bad — multi-record WARC read as one
A WARC has 3 records, each appended as its own gzip member.
gzip.open(file).read()returns only the first member.- The validator reports 1 record and the audit passes on a corrupt file.
Why it fails: the validator must walk every gzip member. The pitfall was verified live 2026-10-08 — a 3-record file reads as 1 record unless the member walk is explicit.
Related skills
- deep-research-method
- trajectory-ledger — a crawl is a multi-step run whose steps, offsets and costs belong in the ledger; the WARC is the evidence, the ledger is the provenance.
- skill-authoring-evals — the crawl policy and the WARC validation rules are research products; cite them and state confidence when you audit this skill.
References
- procedures: references/procedures.md
- policy: references/crawl-policy.md
- warc validation: references/warc-validation.md
- DESIGN: references/DESIGN.md
- EVALS: references/EVALS.md
- SOURCES: references/SOURCES.md
- scripts: scripts/rate_limit.py, scripts/resume_state.py, scripts/robots.py, scripts/validate_skill.py, scripts/validate_warc.py, scripts/warc_writer.py