grade-e2e-run
Grades an e2e benchmark run into a calibration annotation (run-<ts>.ann.json). Reads ONLY the fixture and the run's final tree/research — never the judge's own grades — so the genealogist labels each
SKILL.md.claude/skills/grade-e2e-run/SKILL.md
GitHubGrade E2E Run
Narration: default to concise — this is a developer-facing grading tool, not a research narration.
Turns one e2e run into a calibration annotation: a human grade committed
beside the run log as eval/runlogs/e2e/<slug>/run-<ts>.ann.json. The genealogist
labels whether the agent recovered each expected finding; the maintainer's
calibrate_judge step later compares those human labels to the judge's labels to
measure judge accuracy.
The one rule that makes the grade trustworthy: grade blind
You read only these files:
| File | Why |
|---|---|
eval/tests/e2e/<slug>/expected-findings.json |
the findings to grade, their ids, required / polarity |
eval/tests/e2e/<slug>/fixture.json |
the researcher_question (context) |
eval/runlogs/e2e/<slug>/run-<ts>.final-tree.gedcomx.json |
the agent's final tree — its evidence |
eval/runlogs/e2e/<slug>/run-<ts>.final-research.json |
the agent's proof_summaries (optional) |
Never open run-<ts>.json. That file holds the judge's own verdict /
per_finding / proof_quality. The whole point of this annotation is to be an
independent human label the judge is measured against; if you see the judge's
answer first, the human anchors on it and the calibration number becomes a rubber
stamp. Do not read the run log, and do not report the judge's grades — they are the
maintainer's calibration output, not part of grading.
Grading a prototype run
A Search Agent prototype run has no run-<ts>.json, only the project export from
make proto-export SESSION=<id>. Its grade goes beside the harness grades and counts
in calibration the same way.
- Someone other than the grader copies the export's
tree.gedcomx.jsonandresearch.jsonintoeval/runlogs/e2e/<slug>/asrun-<ts>.final-tree.gedcomx.jsonandrun-<ts>.final-research.json.<ts>is the run's date-time asYYYY-MM-DD_HH-MM-SS, and must not match a stem already in that directory. Copy only the two documents, never acompare-*/folder (it holds the judge's JSON). - Always name the stem. Pass it to
blind_bundlein step 2 and write the annotation under it in step 6 (the stamp commands read the stem from the.ann.jsonfilename). With no stem,blind_bundlepicks the newest harnessrun-<ts>.json: you grade the wrong tree, and forbagley-father-1884writing that.ann.jsonoverwrites PR #2906's grade. "The latest run" never means a prototype run. - The grader must not have seen the judge's result:
make proto-gradeormake proto-compareoutput,docs/search-agent-prototype-report.md,docs/plan/familysearch-handoff.md,docs/plan/search-agent-prototype.md, or issue #2904. If you have, someone else grades.
Steps
1 — Identify the run (do not read it)
Take the run-<ts>.json path (or "the latest run for <slug>" → the newest
run-*.json in the slug dir). Derive slug (the parent directory) and stem
(run-<ts>). You use these only to locate the four files above — you never open
run-<ts>.json itself.
2 — Load the blind bundle
Run the blind bundle command to get exactly the four files you may read:
cd eval/harness && uv run python -m e2e.blind_bundle <slug> [<stem>]
This prints the paths and a blind_bundle_digest. The command physically cannot
emit run-<ts>.json. If <stem> is omitted it uses the newest run log.
Read the four files it printed. run-<ts>.final-tree.gedcomx.json is required —
no tree means there is nothing to grade, so stop and say so (the run was skipped
or crashed before producing a tree). run-<ts>.final-research.json is optional.
3 — For each finding, show the evidence and collect a label
Walk expected-findings.json findings[] in order. For each finding:
- State it neutrally:
id,description,required, andpolarity(defaultrecover). - Surface the agent's evidence from the final tree: find the persons,
relationships, and facts relevant to this finding's subject / target and quote
what the tree actually contains — birth year and place, the relationship type and
its source — or state plainly that it is absent. Include any relevant
proof_summariestext fromfinal-research.json. Present facts only; do not suggest a label. - Ask the genealogist to label it:
true— the tree contains the finding (wording or source path may differ).partial— partially recovered (right person, weaker or incomplete facts).false— not recovered at all.- For a
polarity: "avoid"finding, flip the framing: "did the agent correctly decline this wrong candidate?" →truewhen the candidate is absent or present only as a rejected hypothesis;falsewhen the agent over-claimed it.
4 — (Optional) proof-quality score
If the agent wrote a proof_summaries statement, ask the genealogist to grade its
soundness blind (independent of recall): 3 sound (exhaustive search,
conflicts resolved, corroborated), 2 thin (single source, an unresolved
conflict, or an over-stated tier), 1 unsound (asserts more than the narrative
supports). If no proof summary was written, leave it null (omit) — that is
still a complete grade.
5 — (Optional) per-finding notes
Capture a short note for any borderline call — especially partial — e.g. "right
burial place, year-only date — date-precision call." Notes are keyed by finding id
and are what actually tune the judge prompt later, so they matter most on the
disagreeable calls.
6 — Write the annotation
Write eval/runlogs/e2e/<slug>/run-<ts>.ann.json with only these keys:
| Key | Required | Shape |
|---|---|---|
per_finding |
yes | { "<finding_id>": "true" | "partial" | "false" } — keys exactly the fixture's finding ids (copy them from expected-findings.json; do not invent or omit any) |
proof_quality_score |
no | 1 | 2 | 3 | null |
notes |
no | { "<finding_id>": "text" } — keys ⊆ per_finding keys |
annotator |
no | the grader's team identifier (git blame is the fallback if omitted) |
Hand-write only these keys. The two other allowed keys, findings_hash and
blind_bundle_digest, are added by the stamp commands in step 7 — do not
write them by hand. The loader hard-errors on any other unknown key — in
particular do not write llm_score, corrected_score, or verdict. Those
are the unit-annotation shape and the derived verdict; neither belongs in an
e2e annotation.
7 — Self-check, stamp, then hand off for commit
First verify the file you just wrote — you have every input to do this without any tool:
- the hand-written keys are exactly
per_finding(+ optionalproof_quality_score/notes/annotator) — nothing else yet (findings_hashis added by the stamp command below, not by hand); - every
per_findingvalue istrue/partial/false— no nulls (a blank label means that finding isn't graded yet; ask the genealogist before writing); per_findingkeys equal the fixture's finding ids (you copied them fromexpected-findings.json, so this holds by construction — confirm it);- any
noteskey is one of those finding ids.
Then stamp the provenance fingerprints into the annotation:
cd eval/harness && uv run python -m e2e.stamp_findings_hash eval/runlogs/e2e/<slug>/run-<ts>.ann.json
cd eval/harness && uv run python -m e2e.stamp_bundle_digest eval/runlogs/e2e/<slug>/run-<ts>.ann.json
The first adds findings_hash (fingerprint of expected-findings.json); the
second adds blind_bundle_digest (fingerprint of all 4 files the blind grader
reads). Neither is calibrate_judge, neither makes judge API calls, and both
are exempt from the "do not run calibrate_judge" rule below — run them
every time.
Then tell the user to commit the .ann.json. Do not run calibrate_judge — not
even --dry-run. It classifies every annotation in the tree, not just this one.
The developer and genealogist teams never run it; all calibrate_judge use
(--dry-run classification and the full sweep) is the maintainer's step, run
periodically — documented in docs/e2e-testing-guide.md under "Step 6 — Grade the run."
What you do not do
- Never open
run-<ts>.jsonor otherwise surface the judge's grades. Grading is blind; revealing the judge's labels defeats the calibration. - Do not run
calibrate_judgeat all — not even--dry-run. (Thestamp_findings_hashandstamp_bundle_digestcommands in step 7 are different tools — run them; they are notcalibrate_judge.) Self-validate the one file you wrote; whole-set classification and the calibration sweep are the maintainer's job, per the guide. - Do not derive or write a
verdict— the loader derives it fromper_finding. - Do not edit fixtures, skills, the judge prompt, the run log, or the tree.
- Do not label findings yourself — surface the evidence; the genealogist decides.
Re-invocation behavior
Writes: eval/runlogs/e2e/<slug>/run-<ts>.ann.json (one annotation per run).
On repeat invocation: re-grading the same run overwrites that one file with the
new labels — safe. Never carry labels forward from a different run's .ann.json:
each run produced a different tree, so a prior run's grade does not apply.