Evaluate

Find out how the AI feature in "$ARGUMENTS" actually behaves, from its real outputs, as references/evaluation.md describes.

Arguments. $ARGUMENTS may contain the project name and optional flags. Take the first token as <project> and treat --skip-checks as a flag; never include flags in file paths. All paths below use <project>. If --skip-checks is present, skip the prerequisite check and, before writing any artifact, append one line to .singularskills/log.jsonl: {"ts":"<ISO timestamp>","skill":"ai-evaluate","workflow":"ai","phase":6,"project":"<project>","event":"skipped","artifact":"06-evaluate.md","note":"<reason or \"none given\">"}.

Compute with code. Perform every calculation (totals, balances, ratios, variances, projections) with a script — Bash, Python or a spreadsheet formula written to a file — and include the computation alongside the result. Never do arithmetic mentally.

Prerequisites

Part of a workflow. This skill is phase 6 (Evaluate) of the AI Feature Lifecycle workflow (ai); cross-domain workflows run it too. First check .singularskills/log.jsonl for an open run for this project: a run-started line with "project":"<project>" and no later run-finished line for the same workflow and project. If one exists, the conductor is running you: continue here. Otherwise, if the user asked to start, run, continue or resume a process or workflow rather than this one step, do not continue here: invoke the workflow skill with the workflow the user asked for, or ai when they named none, followed by <project>; it runs this phase in order, asks before it writes and records the run. Otherwise continue here: a later phase runs on its own, behind its gate.

Run this skill's gate before writing: sh <this skill's folder>/scripts/gate.sh <project> <project root>, where <project root> is the folder that contains .singularskills/ (use . when you are already there). If it exits non-zero, stop and tell the user what it printed, unless --skip-checks was given (see Arguments). On a refusal, before you stop, append this line to .singularskills/log.jsonl (create the folder and file if missing): {"ts":"<ISO timestamp>","skill":"ai-evaluate","workflow":"ai","phase":6,"project":"<project>","event":"refused","artifact":"06-evaluate.md","note":"<what the gate printed>"}. Once the gate has passed, or under --skip-checks once you have appended the skipped line, append this line before you start work: {"ts":"<ISO timestamp>","skill":"ai-evaluate","workflow":"ai","phase":6,"project":"<project>","event":"started","artifact":"06-evaluate.md","note":""}.

Read .singularskills/ai/<project>/01-define.md for the success criteria and failure modes and .singularskills/ai/<project>/05-build.md for the initial test set. Read .singularskills/development/<project>/ when it exists, and name what was built; if nothing was built yet, say that this phase needs outputs from the built feature (or a prototype) and ask for them.

If .singularskills/ai/<project>/06-evaluate.md already exists, keep it as 06-evaluate-<date>.md (today's date as YYYY-MM-DD; if that name is taken, add -2, -3 and so on) before you write, and compare against it.

Process

1. Get the outputs: ask once, together with any other question, for a file of real outputs (about 100, with the input and an output ID per row), any user feedback, and the team's own notes per output if they have them.

2. Error analysis, first pass. Use the team's notes when given. Otherwise read every output and write a one-line note on each: what is wrong, or "fine". Then propose failure groups and count them with a script written to .singularskills/ai/<project>/ and included beside the result. Scripts read the team's outputs and logs where they keep them and write aggregates under .singularskills/; a quote from an output names its row or output ID, and names, emails, phone numbers and account IDs in it are replaced with placeholders.

3. Stop for confirmation. Show the proposed groups with their counts and two example rows each, and ask a person who knows the domain to confirm, merge, split or rename them. Under --skip-checks, or with no answer, write the groups as proposed and build no check from them: list them under Open questions as "groups not yet confirmed by a person".

4. Turn each confirmed group into a check, ranked by count and harm: a plain check (a rule code can test) for clear-cut failures, an AI judge with written pass/fail criteria for judgement calls. Fix-now items (a missing instruction, a wrong document) become fixes, not checks. List the new test rows in this artifact under Test set additions, for the next build to add to its test set.

5. Validate every AI judge with /ai-judge-check, against human labels the team supplies on the same outputs: agreement, true-positive rate and true-negative rate, as references/evaluation.md shows. Never invent human labels; with none, mark the judge not validated.

6. Report results per check: pass rate with the sample size, computed, or not measured when the check has not been run on outputs yet; the comparison with the previous evaluation when one was kept.

7. Write the artifact to .singularskills/ai/<project>/06-evaluate.md with sections:

   • Outputs — source, count, what was built
   • Notes — the per-output notes (row IDs, redacted quotes), or where the team's notes are
   • Failure groups — confirmed or proposed, with counts
   • Checks — per confirmed group: type, criteria, judge validation
   • Results — per check
   • Fixes — fix-now items and their owners
   • Test set additions — new test rows from the confirmed groups
   • Compared with previous — when one was kept

Output

The evaluation written to .singularskills/ai/<project>/06-evaluate.md. Present a summary to the user highlighting:

• The top failure groups and whether a person confirmed them
• The pass rate per check, with sample sizes
• Any judge that is not validated

Record the run. After you have written the artifact, append this line to .singularskills/log.jsonl (the started line of Prerequisites, with "event":"finished"): {"ts":"<ISO timestamp>","skill":"ai-evaluate","workflow":"ai","phase":6,"project":"<project>","event":"finished","artifact":"06-evaluate.md","note":""}.

Open the artifact with this frontmatter (agent is the agent you run in, or unknown; gate is passed when the gate ran clean and skipped under --skip-checks):

---
skill: ai-evaluate
workflow: ai
phase: 6
project: <project>
started: <ISO timestamp>
finished: <ISO timestamp>
agent: <the agent you run in, or "unknown">
gate: <passed|skipped>
---

End the artifact with ## Decisions and ## Open questions, in this shape:

## Decisions
- <what was chosen> — <why> — <what was rejected>

## Open questions
- <what the next phase or a reviewer must settle>