interpret-e2e-result

Reads an e2e benchmark run log and explains what happened — which expected findings the agent recovered and which it missed (read from its final tree, not the judge's grade), what proof conclusion it

SKILL.md

GitHub

Interpret E2E Result

Narration: Read researcher_profile.narration_guidance from research.json if one exists in the working folder; otherwise default to concise narration.

Reads the files an e2e run produces and tells the user what happened in plain language, with pointers to the exact turn to read.

Each run writes three files to eval/runlogs/e2e/<test-id>/:

File Content
run-<ts>.json Structured result. Read only the harness facts from it — stop_reason, tool_calls[], narration[], usage, blocked_tree_reads[], blocked_context_calls[], error, compliance, guardrail_bypass_violations[]. Do NOT read judge_output, verdict, or outcome (see "Stay blind to the judge").
run-<ts>.final-tree.gedcomx.json The agent's final tree — the ground truth for what it recovered
run-<ts>.final-research.json The agent's final research.json, including the proof conclusion it wrote
  • When narration[] is absent (runs before 2026-08-03), read the run-<ts>.transcript.md sibling if present, otherwise tool_calls[] alone.

This skill reads files; it does not call any MCP tools.

Stay blind to the judge

Do not read or report the judge's grade for the run you are interpreting. That means all of:

  • judge_output — the per_finding labels, rationale, and proof_quality score;
  • the top-level verdict — this is the judge's genealogical conclusion verbatim;
  • the top-level outcome — the gate, which since the §8 detectors were demoted simply IS verdict, so reporting it leaks the verdict always rather than only when compliance is pass.

Those are the judge's grades: the thing the maintainer calibrates, and unnecessary for explaining what the agent did. The human grade comes first (/grade-e2e-run, blind) and must stay independent of the judge. Your report says which expected findings the tree recovered, which is what the grader labels, so it comes after the grade. If you also surface the judge's labels, a later re-grade is anchored and the calibration number is corrupted.

compliance and guardrail_bypass_violations are NOT judge output — read and report them freely (Step 2d). They are a deterministic harness check on whether the GPS guardrail skills ran, computed from the tool-call log and the final project files, and the grader does not score them.

This prohibition covers the run under interpretation, not the corpus. When Step 5 asks you to compare against a previously-passing run, identify that prior run from its committed .ann.json — a human's grade, which cannot leak the current run's judge output. Do not open a prior run's run-<ts>.json verdict to find one.

You have everything the user needs without the judge:

  • What the agent recovered / missed — compare the final tree to expected-findings.json yourself. The tree is ground truth; report presence or absence as facts, not as a verdict.
  • The proof conclusion — read what the agent wrote in final-research.json (proof_summaries). That's the agent's own claim, not a grade.
  • Why it stopped, what it did — stop_reason, tool_calls, blocked_tree_reads, blocked_context_calls, usage are harness facts, not judge output.

What to do

Ground every claim in the run-log files — do not invent specifics. You are reporting what happened, not reconstructing a plausible story. stop_reason, tool counts, and cost come from run-<ts>.json (harness fields only). What the agent found / missed comes from the final tree vs expected-findings.json. Anything more specific — which collections were searched, which records were found, which index confirmed a date — must come from tool_calls[].args/response_summary, narration[], or final-research.json (the agent's proof summaries and log entries name the actual sources, with ARKs — quote/cite those). If a specific source name isn't in any of those files, don't assert it; a plausible-sounding collection name you didn't actually read is a fabrication.

Step 1 — Locate the run log

If the user pointed at a specific run-<ts>.json, or a scratch_<ts>.json (the harness's name for a run the judge skipped), use it. Otherwise ask which fixture and which timestamp, or take the most recent if only one is present.

Grade first. When the run owes a grade, tell the user to run /grade-e2e-run first and stop. Do not produce the per-finding report. A run owes a grade when all of these hold: it is under eval/runlogs/e2e/ (a quarantine directory is not the corpus); it is a run-<ts>.json, not a scratch_ file; its run-<ts>.final-tree.gedcomx.json sibling is on disk; its top-level stop_reason (not usage.stop_reason) is not host_slept; and no run-<ts>.ann.json is on disk. These are the exemptions check_added_runlogs_graded applies in eval/harness/scripts/check_e2e_fixtures.py, read from the files on disk rather than from a commit, so a run that is not committed yet still owes its grade. Otherwise interpret as below, and also when the user says they will not grade this run or is mining a unit test from it.

A slept run. When the top-level stop_reason is host_slept, go straight to Step 3's host_slept entry. It is the whole interpretation, so skip every other step.

The fixture's expected-findings.json lives in eval/tests/e2e/<id>/. Read it alongside the run so you can compare expected vs found yourself.

Step 2 — Summarize what the agent recovered

Compare the final tree to each required: true finding in expected-findings.json and state, as facts, which the tree contains and which it doesn't. This is recall — did the tree end up with the stripped facts? Do not quote a judge verdict; describe what you see:

  • All required findings present — the run recovered the answer. Say so in a sentence, then describe the proof conclusion (Step 2b): a recovered answer with a thin or missing proof is worth flagging.
  • Some present — name which it recovered and which it missed; the misses are what to investigate (Step 4).
  • None present — the run recovered nothing. The agent stalled or went sideways; stop_reason usually explains.
  • No tree at all (run-<ts>.final-tree.gedcomx.json missing) — the agent crashed before producing tree state. Read the last narration entries and error for the crash.

For a fixture with negative findings (polarity: "avoid" in expected-findings.json), the correct outcome is the tree not asserting the wrong candidate — absent, or present only as a rejected hypothesis. If the tree asserts it, the agent over-claimed; call that out specifically — it's the failure that matters most.

Step 2b — Describe the agent's proof conclusion

Read the agent's proof_summaries from final-research.json and summarize the conclusion it wrote — the claim, the evidence it cites, and the confidence/tier it asserted. Report observable characteristics as facts (e.g. "rests on a single 1910 census entry," "notes an unresolved date conflict," "claims a proof_argument tier"). Do not score its soundness (1 / 2 / 3) — that is what /grade-e2e-run scores. If no proof summary was written, say so; it's not itself a failure, just nothing to describe.

Step 2c — Note any blocked tree-reads or context-writes

Check blocked_tree_reads in run-<ts>.json. The harness blocks person_read / person_search / person_ancestors during a run so the agent can't read the stripped answer off the live tree (it must research from records). Each entry is a denied attempt.

  • Empty — the agent didn't try to shortcut. Normal; say nothing.
  • Non-empty — the agent tried to read the tree but was blocked, so the answer, if recovered, was still earned from records. But flag it: a healthy /research flow shouldn't reach for person_read on the subject during an autonomous run. Repeated attempts may indicate the skill is leaning on tree-reading instead of records, which is worth a look at the /research primer.

Then check blocked_context_calls — the twin denied by a different guard. Every entry carries blocked_by: "context", but two different guards land here and blocked_by cannot tell them apart. Read the entry's tool:

  • research_append — an owned-section write: a caller wrote a section it does not own. This is the arm that actually fires; every entry in the committed corpus is one. Not evidence of a spawn failure, and the shipped plugin hook holds the same rule, so a user would hit it too. Do not assume the main thread: owner_denied has four rules — routed, out_of_lane, declaration and owned_field (a field routed on presence, project.status) — and out_of_lane fires for a named subagent reaching outside its own lanes. tool is research_append for all four, so it cannot tell them apart — read narration[] for the rule and tool_calls[] for the caller.
  • image_read — the one subagent-only tool, called on the main thread. No agent owns it, so this is the router reaching for a scan inline instead of calling image_transcribe, not a spawn failure. Not observed in the committed corpus. Older runs may also show extraction_append here, from before it left the subagent-only set.

Same shape as above ({tool, args, blocked_by}), and the denied attempt also shows up in narration[] with kind: "blocked".

  • Empty — normal; say nothing.
  • Non-empty — a call was denied and that work was not done. Flag it, and say which arm: an owned-section research_append is an ownership slip, not a spawn failure, so name the caller from tool_calls[] rather than assuming the main thread. A blocked image_read is the router reaching for a scan instead of calling image_transcribe; no agent owns it, so it is not a spawn failure. In an older run a blocked extraction_append did point at a record-extractor spawn failure. Read the matching narration[] turn: it shows what the router was trying to do, and for an owned-section write it names which rule fired.

Step 2d — Note any GPS guardrail bypasses

Check compliance and guardrail_bypass_violations[] in run-<ts>.json. This is a harness check, not a judge grade: it asks whether the agent actually invoked the GPS guardrail skills whose effects appear in the final project state — person-evidence, conflict-resolution, proof-conclusion and research-exhaustiveness. gps-mentor is not among them: it runs only when the user asks for a second opinion, so a run with no critique is not a violation.

  • compliance: "pass" — nothing to say.
  • compliance: "fail" — report each violation string as a fact. This is worth calling out even when the tree recovered everything: the agent reached the right answer by a route that skipped the doctrine, so the result is fine and the process is not. Say both plainly; they are separate findings and this skill is the only place the user will see the second one.
  • compliance absent (a run log written before this axis existed) — say nothing; it was never checked.
  • harness_schema_version 6 or below — a "was never successfully invoked" violation for an agent the run spawned typed and without error (an Agent/Task call whose subagent_type is that name and whose is_error is not true) is the retired Skill-only rule, not a bypass: today's detector credits that spawn. Check tool_calls for the spawn and say so instead of reporting it as a skipped step. An untyped or errored spawn is still a bypass.

Do not translate a compliance failure into a pass/fail judgement of the run overall — that is the outcome gate, which is judge-derived and off-limits.

Step 2e — Note whether the answer came from provided documents

If the fixture has a provided-documents/ folder (bundled external captures — Ancestry PDFs, Find A Grave pages the FS tools can't reach), check tool_calls for Read of those filenames. How the answer was obtained is part of the result, so say it plainly:

  • A finding came from a provided PDF — note it (e.g. "the burial came from the bundled Find A Grave capture, not a live FS record"). That's the intended path for external-only evidence, not a problem — but a reviewer should know which findings rested on bundled docs vs. live research, because only the live-research part reflects agent capability against FamilySearch.
  • A provided doc the run needed was never read — flag it: the agent may have missed the bundled evidence, which can explain a miss the tool calls otherwise make look like a search failure.

If the fixture has no provided-documents/, skip this step.

Step 3 — Explain the stop_reason

Translate stop_reason into something a researcher can act on:

  • completed — agent set project.status == "completed". Happy path, proof-conclusion fired. Look at the final tree to judge quality.
  • natural_end — SDK ended the conversation but the agent didn't declare done. Either the agent thought it was done without setting the flag (skill regression), or the conversation drifted to silence. Check the last few narration entries.
  • inactivity — agent went silent for the inactivity cap window (default ~10 min). It's stuck. tool_calls_before counts the calls that preceded an entry, so for the last call — index len(tool_calls) - 1 — the narration that followed it carries tool_calls_before == len(tool_calls). That entry is usually where the issue shows.
  • timeout — wall-clock cap fired (default 120 min; a fixture's caps can override). Either the question is too big for the cap or the agent looped. Skim the tail of tool_calls for repeating patterns.
  • tool_cap — agent hit the per-run tool-call cap (default 300). Almost always means looping. Read the last 20 tool calls; the loop shape is usually obvious.
  • cost_cap — spent past the cost threshold. Not an interruption — it is a post-hoc label (the cost-cap row of the stop-reason table in docs/specs/e2e-test-spec.md), so read usage.stop_reason to see how the run actually ended before calling it unfinished.
  • max_turns — SDK turn limit fired. Rare; usually means a conversational loop rather than a tool loop.
  • error — SDK or harness exception. Read result.error for the message and the last narration entries for the surrounding context.
  • host_slept — an environment failure, not a research result. The harness counted at least the inactivity cap's worth of host sleep (usage.counted_sleep_seconds), labeled the run host_slept and skipped the judge. The run is left out of outcome rates and owes no grade. Tell the researcher to commit its three files with no .ann.json, keep the machine awake (eval/README.md, "Keep the machine awake during a run") and re-run. The whole interpretation is this paragraph: do not report what the run recovered, and do not read its misses as agent failures.
  • mcp_unavailable — an environment failure, not a research result. The genealogy MCP tools were absent from the session, so the agent had no way to search anything. Tell the researcher to re-run the test — not to re-research the case, and to run make e2e-preflight for the server's own error text. You will not be handed a run log for this: such a run writes no files at all (e2e-test-spec.md §6, the retention rule under the stop_reason table), so it reaches a genealogist as terminal output. If you are asked to interpret one, the whole interpretation is the paragraph above — do not go looking for missing findings, and do not read the starting tree as the agent's output.

Step 4 — Compare expected vs found (when the tree is missing some findings)

For each entry in the fixture's expected-findings.json, look at the agent's run-<ts>.final-tree.gedcomx.json and decide:

  • Found — the agent's tree contains the expected person / relationship / fact, possibly with different wording or a different source path.
  • Missed — the agent didn't surface the finding at all. Worth asking why: did it search the right collections? Did it find the right candidate and then dismiss it? Did it never reach the right step?
  • Recorded off-shape — the agent found the right answer but put it somewhere the finding's shape doesn't expect (e.g. on a stub person rather than the principal). Diagnosis: finding shape, not agent capability.

For each missed finding, search narration[].text and tool_calls[].args/response_summary for the relevant person name or place. The agent often touched the right evidence but didn't conclude from it; that turn is the diagnostic moment.

Step 5 — Identify the most likely cause

First decide which situation you're in, because the causes differ:

First run of this fixture (no prior passing run to compare against). "Regression / drift / jitter" don't apply — there's no baseline. The useful questions are about this run:

  • It never researched — the agent stopped after setup (e.g. wrote a question, then stop_reason: natural_end with no record_search / fulltext_search in tool_calls). The GPS loop didn't advance. Pointer: tool counts (no FS search tools) + the last narration entry.
  • It ran out of budget — stop_reason is max_turns / timeout / tool_cap. It researched but didn't finish. Pointer: high turn/tool counts; check whether proof-conclusion was ever reached. (cost_cap is deliberately not in this list — it is applied after the run has already ended, so it never cut a run short.)
  • The evidence wasn't recoverable — it searched genuinely but the finding isn't findable from records (and isn't a provided-documents/ case). The fixture may be unsolvable as authored — a fixture problem, not an agent one.
  • Recorded off-shape — see Step 4 (finding-shape issue).

A previously-passing fixture now failing (you have a prior run to diff). These are the regression causes:

  • Agent reasoning regression — the agent took different decisions on the same evidence than a prior passing run. Pointer: diff tool_calls[] against the last passing run for the same fixture.
  • /research skill regression — the agent skipped a GPS step or picked the wrong sub-skill. Read the ordered sub-skills the agent ran from run-<ts>.json's tool_calls — the Skill entries' args.skill and the typed Agent/Task entries' args.subagent_type (strip a genealogy-research: prefix), in order: [tc['args'].get('skill') if tc['tool']=='Skill' else (tc['args'].get('subagent_type') or 'UNTYPED').split(':')[-1] for tc in tool_calls if tc['tool'] in ('Skill','Agent','Task')]. The orchestrator spawns every destination its routing table spells @plugin:<name> directly as an agent, so a Skill-only list misses them. If proof-conclusion never appears on either route, that's the smoking gun; an UNTYPED spawn standing where it should be is a bypass; if question-selection is missing where a gap needed one, that's another.
  • Sub-skill regression — the right sub-skill ran but produced worse output than before. Pointer: the relevant tool_calls block compared to the prior run.
  • FamilySearch data drift — same tool calls, different results. FS may have reindexed or a contributor may have edited the live tree. Pointer: tool_calls[].response_summary differs from the prior run. The agent isn't at fault. First compare harness_schema_version on both runs. Across the 1 → 2 boundary the capture format changed, not the data — roughly half of all captures differ there, so do not read a response_summary diff across it. The 4 → 5 boundary likewise moves response_summary, but only for image_transcribe: 5 preserves the full transcription where 4 truncated it at 500 chars, so a transcribe-call diff across it is the capture change, not FS drift. The 7 → 8 boundary does the same for Agent/Task calls: 8 keeps the subagent's full return where 7 cut it at 500 chars. The later tool_calls[].is_error boundary does not affect response_summary, so a drift comparison still holds across it; what it does move is compliance — no longer outcome — and the guardrail violation counts, which get stricter once the key is present (an errored call no longer counts as a successful invocation). That boundary is the commit main 4541a4c5, not cleanly a version — a 3 log has the key, a 2 log has it only if written after that commit.
  • Single-run jitter — Anthropic models are non-deterministic and this harness can't pin temperature=0. A single finding flipping recovered / not-recovered may just be variance. Recommend a re-run before drawing conclusions.

If the evidence isn't conclusive, say so — don't force a diagnosis.

Step 6 — Recommend the next action

Pick the cheapest next step that would actually resolve the ambiguity:

  • Re-run the fixture (cheap and decisive if you suspect jitter).
  • Diff against the last passing run log (cheap; commits make this easy).
  • Point the user at a specific turn: the tool_calls index i, and the narration entries with tool_calls_before == i + 1 (the prose that followed that call).
  • Update the fixture's README.md to record what shifted if it looks like FS data drift.
  • File a regression issue if the cause looks like an agent or sub-skill regression.

The run and its run-<ts>.ann.json land in the same PR; CI blocks a run that produced a tree without its .ann.json (a treeless or host_slept run is exempt).

What you do not do

  • Do not read or report the judge's grade for this run: judge_output (per_finding / rationale / proof_quality), the top-level verdict, or the outcome gate. compliance and guardrail_bypass_violations[] are harness facts and ARE reportable (Step 2d). Explain the run from the tree, the agent's proof conclusion, and the harness facts — never the judge's grades.
  • Do not edit fixtures or skills — interpretation only. If the diagnosis points at a fixture problem (e.g. a finding's description is too literal to match the tree), tell the user and suggest /author-e2e-fixture to revise.
  • Do not run the e2e harness yourself. The user runs the harness on the host where FS credentials and the compiled MCP server live; you only read the artifacts it produces.
  • Do not pad the output. A clean recovery gets a one-line summary; a run with misses gets the analysis above. Skip sections that don't apply.

Example

User: "Why did the <slug> run fail?"

You should:

  1. Read eval/runlogs/e2e/<slug>/run-<latest>.json (harness fields only — skip judge_output, verdict, and outcome) and check that run-<latest>.ann.json exists.
  2. Read eval/tests/e2e/<slug>/expected-findings.json and the final tree; the tree contains none of the required findings.
  3. See stop_reason: tool_cap.
  4. Skim the last 30 entries in tool_calls — agent is looping on place_search for "Augusta County" with three near-duplicate queries.
  5. Tell the user: "Recovered none of the required findings; stopped at the tool cap (300 calls). Agent looped on place_search from turn 47 onward — three near-duplicate queries for Augusta County. Likely cause: /research skill regression (place-disambiguation guidance is weaker than the last passing run). Recommend diffing tool_calls[] against the previous green run for this fixture before changing the skill."

Re-invocation behavior

Writes: nothing. This skill is read-only — it reads an e2e run log (harness fields only, never judge_output / verdict / outcome), the agent's final tree and research files, and the fixture's expected-findings.json, and explains the result in-session. It calls no MCP tools and edits no fixtures, skills, or project files.

On repeat invocation: safe to run as often as needed. Each call is a fresh read of the run-log artifacts.

Do not duplicate: N/A — no writes.