Model compare

Compare the models in "$ARGUMENTS" on the team's own results and recommend one.

Compute with code. Perform every calculation (totals, balances, ratios, variances, projections) with a script — Bash, Python or a spreadsheet formula written to a file — and include the computation alongside the result. Never do arithmetic mentally.

Prerequisites

The results: per model, per test case, the pass or fail verdict for each check, the cost and the latency, as a file or a paste. Never use benchmark scores or prices from memory; compare only what the team ran. When $ARGUMENTS or the request names a project, use that name as <project> whether or not its folder exists yet in .singularskills/ai/. With no project named: with exactly one folder there, use it; with several, ask which; with no folder at all, read none and continue.

Read .singularskills/ai/<project>/01-define.md for the budgets and .singularskills/ai/<project>/06-evaluate.md for the checks and the launch bar, when they exist.

Process

1. Compute per model with a script: pass rate per check with the number of cases, cost per use (mean and 90th percentile), and latency (median and 90th percentile). Work in a scratch location outside .singularskills/; this skill writes no file. Without a way to run code, show the raw counts per model and check, label rates not computed, and compare from the counts.

2. Check the differences are real: with fewer than about 50 cases per check, say a difference of a few points may be noise.

3. Compare against the budgets and the bar: which models meet every check's bar within the latency and cost budgets.

4. Recommend one, with the trade-off stated (for example, "B passes 4 points more on faithfulness for 3× the cost"), and what would change the choice.

Output

In chat: the recommendation first, then a table of models against checks, cost and latency, and the trade-off. This skill writes no file.