eval-compare
Compares the eval scores of two plugin folders, such as a plugin and a variant of it. Use when the user wants to know whether a change to a plugin raises or lowers its eval score.
SKILL.mdplugins/claudetto-lab/skills/eval-compare/SKILL.md
GitHub<task>
Compare the eval scores of two plugin folders for the user.
</task>
<procedure>
Each run costs money and takes minutes, so confirm the two folders with the user before the run.
Find the two plugin folders. Each one has a
.claude-plugin/plugin.jsonfile and anevals/folder. If the user names a git ref instead of a folder, ask for a folder, or tell the user to check out the ref in agit worktree, because the command reads folders and not refs.Run the comparison:
claudetto-eval-compare A BReplace
AandBwith the two folders. Addclaude plugin evaloptions after them when the user asks for them. The command adds--max-cost-usd 5for each run when the options set no bound. The command adds no--scaffoldand no--allow-tools, because they run the scripts and the tools of the suite as the user. If the cases need them, ask the user before you add them, and name the folders whose suites will run. The skill grants no tool, so Claude Code asks the user before each run. If a run stops at the cost bound, tell the user, and do not raise the bound without the user.Read the two result tables. Compare the
</procedure>WITHcolumn of table A with theWITHcolumn of table B, case by case. Do not compare theWITHOUTcolumn of the two tables as a plugin result, because it is the shared baseline. If the two baseline scores differ by more than the gap between the arms, say so, because the model varies between runs and the gap is then weak evidence.