model-evaluation
LLM evaluation methodology — side-by-side comparison, domain-specific benchmarking, regression detection for fine-tuned models. Trigger on: "evaluate model", "benchmark", "model comparison", "A/B test
SKILL.mdskills/model-evaluation/SKILL.md
GitHubModel evaluation
Define the intended workload and acceptance thresholds before collecting results. Include correctness, tool use, invalid inputs, safety boundaries, latency, throughput and cost where relevant.
- Freeze a representative held-out task set and rubric, with identifiers/hashes and independently verifiable expected results. Keep training and tuning examples separate; include edge cases and unfamiliar tasks.
- Record model revision, quantization, serving configuration, hardware, tokenizer, sampling and prompt/tool versions. Compare the same tasks under controlled conditions and disclose any differences.
- Repeat stochastic trials. Report sample size, variance/confidence intervals and failures/timeouts. A small smoke set is not statistical proof of superiority.
- Prefer deterministic scoring for executable answers. Check factual citations and dates against primary sources. Calibrate model judges against human labels, randomize answer order and audit disagreement/bias.
- Compare dimensions separately. Investigate regressions before deployment; aggregate scores can hide critical failures.
- Report observed token usage, latency distributions, resource use and provider receipts. Do not infer cost or speed from a model name. Use
token-miserfor paired usage receipts when appropriate.
Repetition or degeneration can have several causes: data, optimization, tokenizer/template, serving or sampling faults. Investigate rather than assuming excess epochs. There is no universal algebraic formula mapping data size or epochs to quality.
Publish reproducible commands/configuration, dataset provenance, exact candidate identity, results and limitations. Protect private evaluation content and do not claim unseen benchmark performance.