marshal-benchmark
Compare AI coding strategies on a real task with Marshal: run the same goal through several configured clients (e.g. a premium model vs a cheaper one) and get a measured cost/latency/outcome compariso
SKILL.mdskills/marshal-benchmark/SKILL.md
GitHubBenchmarking routing strategies
Marshal's differentiator is proving what a routing choice costs, with measured numbers - not a fixed % claim. A strategy is a configured client (a backend + model + permission combo). This playbook runs one goal through N strategies and compares them.
Run it
list_clients- pick the strategies (client names) to compare, e.g. a premium worker vs a cheap one, both capable of the task.benchmark(goal, clients, task_id?, max_concurrency?)- runs the same goal through each client in parallel (each in its own worktree), then returns a comparison:strategies[]- per client:status,cost_usd,source,duration_ms, tokens,run_id.cheapest/fastest- the winning client, but only among strategies that succeeded and have a known cost. A strategy whose cost isunavailablenever wins "cheapest."
report(task_id)re-derives the same comparison later from the ledger (read-only) - reproducible and auditable.
Multi-workspace:
benchmarkandreportare scoped to one workspace, andtask_ids are per-workspace. If you benchmark withworkspace=<name>, pass the sameworkspacetoreport(task_id, workspace=<name>)- otherwise it reads the wrong repo's ledger and finds nothing.
Read it honestly
- Check
sourceon each row.native= the backend reported the cost;admin-api= the real charge read back from a provider usage API (e.g. EastRouter);unavailable= unknown (e.g. Cursor without the Admin API, or a token-only Codex run). Never read anunavailable$0as free. - The benchmark measures cost, latency, and outcome status - not correctness. A cheapest strategy
can still produce worse code. Review the diffs (
collect_run) before drawing conclusions. empty/failed/verify_failedstrategies still cost tokens; they appear in per-run cost but are excluded fromcheapest/fastest.
What to do with the result
- Route future work of this kind to the strategy with the best cost/quality trade-off (the cheapest one that also produces correct diffs).
- To keep a strategy's output,
integrateitsrun_id(see marshal-orchestrate). - The historical record of what each strategy cost - derived on read, never a stored guess - is the defensible asset. Build it up over real tasks rather than promising a universal savings number.