Evaluation

Require task contract, candidate model revision, benchmark or acceptance set, backend, thresholds, data license, compute budget, and evaluation environment.

Workflow

1. Select task, benchmark, backend, metrics, slices, and leakage controls; pin model, dataset, code, and environment revisions.
2. Run a small smoke stage before scale, validate output schema, then execute the approved matrix.
3. Report aggregate and slice metrics, calibration/robustness, uncertainty, failures, missing values, and cost/latency.
4. For inspect-ai/lighteval-style LLM backends, produce an executable evaluation plan unless an equivalent registry operation exists; do not claim execution.
5. Return pass/fail thresholds and a go/no-go recommendation to ml-stack-deployment or ml-stack-audit.

Boundary and outputs

Never tune on the acceptance set, hide failed slices, or publish results without approval. Abstain when benchmark provenance, model revision, threshold, or uncertainty is missing. Next owner: ml-stack-deployment for a gated release, ml-stack-experiment for a controlled follow-up, or ml-stack-audit for independent reconciliation.

See benchmark matrix, smoke to scale, metrics and slices, robustness, and evaluation report.