crawl-reachability
Reachability stage for AI crawlers. Use when pages are missing from AI results, appear empty to machines, or need sitemap and indexability checks. Discovers and samples pages, audits sitemap defects,
SKILL.mdskills/crawl-reachability/SKILL.md
GitHubCrawl & Reachability
The first gate in the pipeline, and the one most likely to make everything downstream moot. A page that cannot be retrieved, or that arrives without its content, cannot be selected as a source no matter how good the content is.
arXiv:2604.25707 §4.1 reports a 76.44% fetch success rate across pages that answer engines had already chosen to cite. Roughly one in four selected pages could not be retrieved. Reachability is not a hygiene item; it is a hard gate with a measured failure rate.
When to use
After url-validation clears the target. Run it standalone to answer "can a
crawler read this site at all" or "does this page render server-side".
Inputs
| Input | Required | Notes |
|---|---|---|
--workspace |
no | Run directory. Defaults to the most recent run. |
--max-pages |
no | Overrides the preflight page budget. |
--json |
no | Machine-readable output. |
Reads 01-url-validation.json for the target, scope, robots policy and budget.
Refuses to run if preflight has not cleared.
Procedure
- Discover. robots.txt
Sitemap:directives, then the conventional sitemap paths, expanding at most three sitemap indexes; fall back to a bounded same-scope homepage crawl when discovery yields fewer than 12 URLs. Probe forllms.txt. - Sample. Round-robin across page types so the sample spans product, pricing, article, docs, about and hub pages rather than 20 blog posts. This is what makes a later population claim defensible.
- Fetch, honouring robots.txt per URL and the crawl-delay, saving each page's HTML into the workspace for downstream skills. No JavaScript is executed — deliberately, because that is what a crawler sees.
- Audit the sitemap in depth (see the table below).
- Diagnose render faults per page.
- Check per-page reachability: status, canonical correctness,
X-Robots-Tag,noindex, soft-404s, redirect behaviour. - Write
02-crawl-reachability.jsonincluding the page inventory.
Run it:
python3 scripts/check_reachability.py [--workspace PATH] [--max-pages N] [--json]
Checks performed
Sitemap
| Check id | Detects | Severity | Claim level |
|---|---|---|---|
sitemap_missing |
No retrievable sitemap | high | 1 |
sitemap_not_declared |
Sitemap exists but robots.txt does not point to it | medium | 1 |
sitemap_malformed |
Bad namespace, unwrapped <loc>, over size limits |
medium | 1 |
sitemap_empty |
Zero <url> entries |
high | 1 |
sitemap_broken_entries |
Listed URLs returning 4xx/5xx | high/medium | 1 |
sitemap_redirect_heavy |
Over 30% of sampled entries redirect | medium | 1 |
sitemap_orphans |
Crawlable pages missing from the sitemap | high/medium | 1 |
sitemap_off_scope |
Entries outside the serving host | medium | 1 |
sitemap_scheme_mismatch |
http entries on an https site |
medium | 1 |
sitemap_bad_lastmod |
lastmod not a valid W3C datetime |
low | 1 |
sitemap_future_lastmod |
lastmod in the future |
medium | 1 |
sitemap_stale |
Median lastmod older than a year |
medium | 1 |
sitemap_uniform_lastmod |
Every entry shares one lastmod |
low | 1 |
Render faults
| Check id | Detects | Recoverable | Severity |
|---|---|---|---|
empty_shell |
Empty mount point, no server-rendered content | no | critical |
state_blob |
Content present only inside a client state payload | yes | high |
script_heavy_shell |
Script-dominated document, nothing recoverable | no | high |
thin_served_html |
Visible text under 5% of served bytes | no | high |
meta_only |
Metadata renders, body does not | no | medium |
lazy_content |
Primary content deferred behind loading placeholders | no | medium |
client_routing |
Hash-fragment routes; sub-pages have no own URL | no | high |
noscript_gap |
Framework present, no <noscript> fallback |
no | medium |
The state_blob / empty_shell distinction is the point of this section. Both
look identical to a browser-free reader, but the fixes differ by an order of
magnitude of effort: state_blob means the text already ships with the page
and needs hydrating into markup; empty_shell means it never left the server.
Page reachability
| Check id | Detects | Severity | Claim level |
|---|---|---|---|
page_unreachable |
Sampled pages returning errors | high | 1 |
noindex_on_public_page |
noindex on a page that should be indexed |
high | 1 |
x_robots_blocks |
X-Robots-Tag blocking indexing or AI agents |
high | 1 |
canonical_mismatch |
rel=canonical pointing elsewhere unexpectedly |
medium | 1 |
canonical_missing |
No rel=canonical on a site with URL variants |
low | 2 |
soft_404 |
200 response whose content says "not found" | medium | 2 |
slow_response |
Median response time over 3 seconds | medium | 2 |
Output
02-crawl-reachability.json, including the page inventory downstream skills
read:
{
"pages": [ { "url": "...", "page_type": "product", "status": 200, "html_path": "pages/003.html" } ],
"discovery": { "sitemap_found": true, "candidates": 214, "llms_txt": false },
"sitemap": { ...deep audit... },
"render": { "pages_with_blocking_faults": 3, "fault_counts": { ... } },
"findings": [ ... ]
}
Guardrails
Read-only. robots.txt is enforced per URL, not merely reported. Crawl-delay
is honoured. Requests are rate-limited and the page, byte and wall-clock
budgets are hard limits. Never authenticates, never submits a form, never
leaves the registrable domain.
Grounding
- Fetch success rate 76.44% across cited pages (§4.1) — reachability is a measured failure mode, not a theoretical one.
- Search-trigger rates 98.64% / 99.67% / 100% (§6) — retrieval is attempted on nearly every prompt, so a retrieval-time failure is a citation failure.
- Handout appendix C — content assembled after load is invisible to a reader that does not execute scripts.