nursery
Run the Nursery-lite qualification scenarios and triage failures. Use when the user asks to run evals, qualify a change, check scenario health, or says /nursery. Args may name a scenario or test famil
SKILL.md.claude/skills/nursery/SKILL.md
GitHub/nursery — run and triage qualification scenarios
- Run the suite (mock mode — never pass
--liveunless the user explicitly asks and confirms they have local credentials):- All:
uv run koa-nursery --scenarios evals/scenarios --report qualification-report.json - One scenario: add
--only <name>; one family: add--family <family>
- All:
- If everything passes, report the summary line and stop.
- On failure, read
qualification-report.jsonand for each failing scenario report:- the scenario name and its test family (positive/negative/ablation/poisoning/regression)
- each failing grader with its
detailstring - the most likely cause, distinguishing:
- workflow defect (deterministic checks wrong) → look in
src/koa/workflows/ - scenario/mock drift (mock script no longer matches workflow steps;
MockScriptExhaustederrors) → fix the scenario'smock_script - grader misconfiguration → check grader params against
src/koa/nursery/graders.py
- workflow defect (deterministic checks wrong) → look in
- Propose the single next debugging step (a file to inspect or a one-scenario rerun
with
-v), and offer to apply the fix.
Never mark a poisoning-family failure as "flaky" — treat it as a real injection-handling regression until proven otherwise.