Evidence Discovery (ev-discovery)

Different from pre-curation /ev-scout: scouting happens before topology mapping and casts a wide contextual net, including rivals and negative evidence. This Phase 0 is a later, optional evidence-development step for selected Stage-2 sources. Scout leads are unverified and non-scoring until the canonical verification and human approval sequence completes.

Phase 0 is optional. It searches for new, higher-quality Stage-2 evidence only when a maintainer declares a need.

Type-First Evidence Lake Contract (#1148)

The evidence lake is type-first. Discovery feeds Phase 1, which materializes evidence/by-type/<canonical-evidence-type>.md. Legacy evidence/tier_*.md files may still exist as coexistence artifacts, but they are not the semantic routing key.

Discovery Scope

Prioritize new evidence rows for:

• benchmark-result
• arxiv
• peer-review
• richer social-signal

Discovery may also identify better repo-own or github-stars-own sources, but it must not assign authoritative rank, tier, grade, or Trust Magnitude.

Benchmark candidate handoff

When discovery finds missing benchmark evidence, create a benchmark-source candidate record for Phase 2B review instead of appending a scoring benchmark-result row from a vendor page. The candidate should name the target skill, source URL, proposed benchmarkId, proposed generic applicability (appliesToGenericSkillRefs) when known, status: candidate, missing reproducibility fields, and notes. No discovered vendor page may become a scoring row until a human promotes the source in registry/benchmark-sources.json and approves /gaia-ingest-batch. A promoted registered benchmark source is still generic-applicable, non-scoring catalog metadata only until a named benchmark-result row satisfies reproducibility fields and approved provenance. Firecrawl-style vendor benchmark claims are candidate-only or registered-only until reproducibility fields, allowed provenance, and human approval exist.

Handoff to Phase 1

Append discovered raw rows to the appropriate source inputs with a canonical or normalizable evidence type. Phase 1 is responsible for normalizing aliases and materializing evidence/by-type/<type>.md. Do not hand-write generated by-type partitions as if they were source rows.

If one discovered peer-review source legitimately covers multiple named skills, hand Phase 1 a scratch multi-target packet manifest for evidence/scripts/peer_review_source_packets.py rather than cloning the same review into multiple intake mutations. The helper is peer-review-only, allows the same URL once per reviewed skill, rejects strength/scoring fields (trustNumber, grade, class, tier, level, stars, rank), and should write only temporary by-type output.

Preflight (#1788)

Firecrawl usability is not gated on the FIRECRAWL_API_KEY env var alone — the CLI can be authenticated via stored browser/CLI credentials independent of it. Check both before deciding to skip Phase 0:

firecrawl --status                 # Authenticated? credits remaining?
echo "${FIRECRAWL_API_KEY:+set}"   # non-empty output if the env var is set

Proceed with discovery if either check succeeds. Skip Phase 0 gracefully only when firecrawl --status reports unauthenticated and the env var is unset/empty.

Commands

Quote every URL and query — unquoted ?/&/glob characters expand under zsh, this repo's shell, and silently turn a single request into a file-glob error.

mkdir -p .firecrawl
firecrawl search "<query>" -o .firecrawl/result.json --json
firecrawl scrape '<url>' --json -o .firecrawl/scrape.json
gh api -X GET 'repos/<owner>/<repo>/git/trees/main' -f recursive=1

The gh api form above (path + -f recursive=1 instead of a literal ?recursive=1 in the URL) avoids the same zsh glob-expansion problem for GitHub tree listings.

Retry rule

Retry is a status-code rule, not a named-host list — a source is not "unresolved" just because it wasn't on someone's retry shortlist:

• On any 401, 402, 403, 429, or SSL/TLS failure from a direct fetch, retry once via firecrawl scrape before marking the source dead.
• Do not maintain or consult a list of specific domains to retry — apply the rule to every failing URL, every time.

Guardrails

• Requires Firecrawl usability per the Preflight above (firecrawl --status OR FIRECRAWL_API_KEY); skip gracefully only when both fail.
• Do not run as a broad web crawl without declared need.
• Do not mutate registry files.
• Do not append direct scoring benchmark-result rows from discovered vendor pages; route them through benchmark-source candidates and Phase 2B first.
• Do not treat generic-applicable registered benchmark sources as Trust Magnitude evidence; they are non-scoring until a named row satisfies reproducibility and approved provenance.
• Do not commit generated collector/by-type/report artifacts without a human gate.