ai-judge-check

Check an AI judge against human labels — agreement, true-positive rate and true-negative rate from the four cells of the agreement table, the disagreements listed — and say whether the judge can be tr

SKILL.md

GitHub

Judge check

Find out whether the AI judge in "$ARGUMENTS" can be trusted to replace people for its check. Agreement alone is not the answer: a judge that passes everything agrees with people most of the time when most outputs pass, and catches none of the failures it exists to catch.

Compute with code. Perform every calculation (totals, balances, ratios, variances, projections) with a script — Bash, Python or a spreadsheet formula written to a file — and include the computation alongside the result. Never do arithmetic mentally.

Prerequisites

The labels: rows with an output ID, the human's verdict and the judge's verdict (pass or fail), as a file or a paste. Never invent or fill in human labels; rows without a human label are left out and counted. When $ARGUMENTS or the request names a project, use that name as <project> whether or not its folder exists yet in .singularskills/ai/ (create it when you write). With no project named: with exactly one folder there, use it; with several, ask which; with no folder at all, ask for a project name.

Read .singularskills/ai/<project>/06-evaluate.md for the check this judge serves and its criteria, when it exists. references/evaluation.md holds the agreement table, the trust bar and the label minimum this skill applies.

Append this line to .singularskills/log.jsonl before you write the check (create the folder and file if missing): {"ts":"<ISO timestamp>","skill":"ai-judge-check","workflow":"ai","phase":null,"project":"<project>","event":"started","artifact":"judges/<date>-<slug>.md","note":""}, where <date> is today as YYYY-MM-DD and <slug> is two to four words naming the check, never a person.

Process

  1. Count the four cells of the agreement table:

    Human: pass Human: fail
    Judge: pass a b (judge misses a failure)
    Judge: fail c (judge rejects a good output) d

    Count them with a script written to .singularskills/ai/<project>/ and included beside the result. Scripts read the team's files where they keep them and write aggregates under .singularskills/; a quote from an output names its row or output ID, and names, emails, phone numbers and account IDs in it are replaced with placeholders.

  2. Compute the rates: agreement (a + d) ÷ all; true-positive rate a ÷ (a + c); true-negative rate d ÷ (b + d). Without a way to run code, show the four cell counts, label every rate not computed, and judge from the counts: for example, "the judge passed 6 of the 7 outputs people failed".

  3. Check there are enough labels: with fewer than 10 human-labelled fails, or fewer than 30 labelled rows, there are too few labels to approve the judge. Say so, give the counts, and ask for more labels. Too few labels stops you approving a judge, never rejecting one: if the judge already passed more than half of the outputs people failed, or rejected more than half of the outputs people passed, the verdict is do not trust, and too few labels: get more labels before changing the judge's prompt, so the fix is not tuned to a handful of rows.

  4. Judge the judge on the true-negative rate first, then the true-positive rate:

    • trust — true-negative rate at least 0.8 and true-positive rate at least 0.8, with enough labels (step 3); for a check guarding a high-harm failure, raise the true-negative bar to 0.9 (the pack's call, stated in the check)
    • do not trust — either rate below its bar; say which, with the cell counts
    • too few labels — as in step 3
  5. List the disagreements (cells b and c) by output ID with a short redacted quote, and what they have in common.

  6. Say how to fix the judge when it is not trusted and there are enough labels (with too few, the fix is more labels first): sharpen its pass/fail criteria around the missed failures, add those rows as examples in its prompt, then re-check it against fresh labels.

  7. Write the check to .singularskills/ai/<project>/judges/<date>-<slug>.md with sections Check, Agreement table, Rates, Verdict, Disagreements and Fix.

Output

The judge check written to .singularskills/ai/<project>/judges/<date>-<slug>.md. Present the verdict first (trust, do not trust, or too few labels), then the cell that decided it, then the fix.

After you have written the check, append the started line of Prerequisites to .singularskills/log.jsonl with "event":"finished": {"ts":"<ISO timestamp>","skill":"ai-judge-check","workflow":"ai","phase":null,"project":"<project>","event":"finished","artifact":"judges/<date>-<slug>.md","note":""}.

The check opens with this frontmatter (agent is the agent you run in, or unknown; this utility has no gate, so phase is null and gate is none):

---
skill: ai-judge-check
workflow: ai
phase: null
project: <project>
started: <ISO timestamp>
finished: <ISO timestamp>
agent: <the agent you run in, or "unknown">
gate: none
---

The check ends with ## Decisions and ## Open questions, in this shape:

## Decisions
- <what was chosen> — <why> — <what was rejected>

## Open questions
- <what the next phase or a reviewer must settle>