crawl-reachability

Reachability stage for AI crawlers. Use when pages are missing from AI results, appear empty to machines, or need sitemap and indexability checks. Discovers and samples pages, audits sitemap defects,

SKILL.md

GitHub

Crawl & Reachability

The first gate in the pipeline, and the one most likely to make everything downstream moot. A page that cannot be retrieved, or that arrives without its content, cannot be selected as a source no matter how good the content is.

arXiv:2604.25707 §4.1 reports a 76.44% fetch success rate across pages that answer engines had already chosen to cite. Roughly one in four selected pages could not be retrieved. Reachability is not a hygiene item; it is a hard gate with a measured failure rate.

When to use

After url-validation clears the target. Run it standalone to answer "can a crawler read this site at all" or "does this page render server-side".

Inputs

Input Required Notes
--workspace no Run directory. Defaults to the most recent run.
--max-pages no Overrides the preflight page budget.
--json no Machine-readable output.

Reads 01-url-validation.json for the target, scope, robots policy and budget. Refuses to run if preflight has not cleared.

Procedure

  1. Discover. robots.txt Sitemap: directives, then the conventional sitemap paths, expanding at most three sitemap indexes; fall back to a bounded same-scope homepage crawl when discovery yields fewer than 12 URLs. Probe for llms.txt.
  2. Sample. Round-robin across page types so the sample spans product, pricing, article, docs, about and hub pages rather than 20 blog posts. This is what makes a later population claim defensible.
  3. Fetch, honouring robots.txt per URL and the crawl-delay, saving each page's HTML into the workspace for downstream skills. No JavaScript is executed — deliberately, because that is what a crawler sees.
  4. Audit the sitemap in depth (see the table below).
  5. Diagnose render faults per page.
  6. Check per-page reachability: status, canonical correctness, X-Robots-Tag, noindex, soft-404s, redirect behaviour.
  7. Write 02-crawl-reachability.json including the page inventory.

Run it:

python3 scripts/check_reachability.py [--workspace PATH] [--max-pages N] [--json]

Checks performed

Sitemap

Check id Detects Severity Claim level
sitemap_missing No retrievable sitemap high 1
sitemap_not_declared Sitemap exists but robots.txt does not point to it medium 1
sitemap_malformed Bad namespace, unwrapped <loc>, over size limits medium 1
sitemap_empty Zero <url> entries high 1
sitemap_broken_entries Listed URLs returning 4xx/5xx high/medium 1
sitemap_redirect_heavy Over 30% of sampled entries redirect medium 1
sitemap_orphans Crawlable pages missing from the sitemap high/medium 1
sitemap_off_scope Entries outside the serving host medium 1
sitemap_scheme_mismatch http entries on an https site medium 1
sitemap_bad_lastmod lastmod not a valid W3C datetime low 1
sitemap_future_lastmod lastmod in the future medium 1
sitemap_stale Median lastmod older than a year medium 1
sitemap_uniform_lastmod Every entry shares one lastmod low 1

Render faults

Check id Detects Recoverable Severity
empty_shell Empty mount point, no server-rendered content no critical
state_blob Content present only inside a client state payload yes high
script_heavy_shell Script-dominated document, nothing recoverable no high
thin_served_html Visible text under 5% of served bytes no high
meta_only Metadata renders, body does not no medium
lazy_content Primary content deferred behind loading placeholders no medium
client_routing Hash-fragment routes; sub-pages have no own URL no high
noscript_gap Framework present, no <noscript> fallback no medium

The state_blob / empty_shell distinction is the point of this section. Both look identical to a browser-free reader, but the fixes differ by an order of magnitude of effort: state_blob means the text already ships with the page and needs hydrating into markup; empty_shell means it never left the server.

Page reachability

Check id Detects Severity Claim level
page_unreachable Sampled pages returning errors high 1
noindex_on_public_page noindex on a page that should be indexed high 1
x_robots_blocks X-Robots-Tag blocking indexing or AI agents high 1
canonical_mismatch rel=canonical pointing elsewhere unexpectedly medium 1
canonical_missing No rel=canonical on a site with URL variants low 2
soft_404 200 response whose content says "not found" medium 2
slow_response Median response time over 3 seconds medium 2

Output

02-crawl-reachability.json, including the page inventory downstream skills read:

{
  "pages": [ { "url": "...", "page_type": "product", "status": 200, "html_path": "pages/003.html" } ],
  "discovery": { "sitemap_found": true, "candidates": 214, "llms_txt": false },
  "sitemap": { ...deep audit... },
  "render": { "pages_with_blocking_faults": 3, "fault_counts": { ... } },
  "findings": [ ... ]
}

Guardrails

Read-only. robots.txt is enforced per URL, not merely reported. Crawl-delay is honoured. Requests are rate-limited and the page, byte and wall-clock budgets are hard limits. Never authenticates, never submits a form, never leaves the registrable domain.

Grounding

  • Fetch success rate 76.44% across cited pages (§4.1) — reachability is a measured failure mode, not a theoretical one.
  • Search-trigger rates 98.64% / 99.67% / 100% (§6) — retrieval is attempted on nearly every prompt, so a retrieval-time failure is a citation failure.
  • Handout appendix C — content assembled after load is invisible to a reader that does not execute scripts.