build-agent-evals
Build or improve an agent evaluation program with versioned datasets, evaluators, and release gates. Use when creating datasets, cases, evaluators, trajectory checks, durable-execution failure scenari
SKILL.mdskills/build-agent-evals/SKILL.md
GitHubBuild Agent Evals
Convert product claims and operational risks into reproducible evidence and release gates.
Workflow
- Extract testable claims from the Agent Spec. Include quality, safety, tool use, latency, cost, escalation, and recovery claims.
- Separate hard invariants from graded quality. A policy violation must not be averaged away by a high mean score.
- Build a dataset with happy paths, boundaries, adversarial inputs, historical regressions, tool failures, permission denials, and durable-execution cases.
- Attach stable case IDs, provenance, risk tags, capability tags, expected invariants, and dataset version.
- Prefer evaluators in this order:
- deterministic assertions;
- structured-output checks;
- tool and trajectory checks;
- task-specific graders;
- model-based judges for qualities that cannot be measured directly.
- Test the full execution path when it matters, including tool results and durable-execution recovery behavior, not only the final answer.
- Repeat stochastic cases and report pass rate, variance, and confidence—not one lucky run.
- Track quality beside latency, tokens, tool calls, retries, and estimated cost.
- Run evals and release checks through the project's own validation tooling or CI so provenance is normalized and gates fail closed; treat that output as release evidence.
- Establish a trusted baseline and fail CI on hard-gate violations or material regressions beyond an explicit tolerance.
- Sample production traces into reviewed datasets while redacting sensitive data and separating monitoring from release evidence.
Required Output
Produce versioned eval datasets, evaluator code, a runnable entry point, a machine-readable report, human-readable failure summaries, and documented release thresholds.
Read the eval standard before choosing metrics. Use the examples for case design, evaluator selection, and release gates.
Guardrails
- Do not use only happy-path examples.
- Do not rely solely on a model judge.
- Do not edit expected results merely to make a failing change pass.
- Do not compare model variants without controlling prompts, tools, datasets, and run counts.
- Do not log sensitive production content into eval datasets without a review and retention policy.
Completion Check
Confirm that failures identify the case, violated claim, evaluator, execution trace, model and prompt versions, cost, latency, and whether the failure blocks release.