model-audit
Audit the ceh-* plugins against Claude's per-model prompting guides and open the results as a draft PR. Trigger on "/model-audit", "audit the plugins for the new model". User-invoked only: every audit
SKILL.md.claude/skills/model-audit/SKILL.md
GitHubModel audit
Run the audit scripts and hand the results over as one draft PR, so the user reviews that PR and
nothing else. The scripts, their arguments, and their state files are in
${CLAUDE_SKILL_DIR}/README.md.
$ARGUMENTS passes to audit.py unchanged.
Procedure
- Budget only. When
$ARGUMENTSisbudget, runpython "${CLAUDE_SKILL_DIR}/scripts/context_budget.py", reply with its table and the three plugins that cost the most, and stop: no model call, no PR. The table shows what each plugin's descriptions and context-injecting hooks add to every session, prompt, and subagent. - Detect. Check the working tree is clean apart from untracked files, then run
python "${CLAUDE_SKILL_DIR}/scripts/detect.py" --write. It prints JSON. Ifstaleis empty and--pluginswas not passed, report "nothing stale" and stop. Otherwise tell the user which plugins are stale and how many runs that makes: two per plugin with missing guides (/doctorplus the filter pass), plus one per pinned file with"stale": true. - Audit. Run
python "${CLAUDE_SKILL_DIR}/scripts/audit.py" $ARGUMENTSin the background and wait for it to exit. Read onlyaudits/<date>/SUMMARY.md. A non-zero exit means a plugin errored or failed validation: the summary says which. Carry on to the PR anyway, so the failure is reviewed with everything else. - Commit. Stage
${CLAUDE_SKILL_DIR}/assets/known-model-guides.json,audits/, andplugins/. Commit withchore(model-audit): audit reports for <date>plus the attribution line. If the pre-commit hook reformats files and aborts, stage them again and commit again. - Open the PR. Chain with
&&:git push -u origin HEAD && gh pr create --draft --title "chore(model-audit): audit <date>" --body-file <body>.<body>is a scratch copy ofSUMMARY.mdwith the PR attribution line appended. - Reply in the shape under Output.
Rules
- Do not read the per-plugin reports into this session. They are large, and the user reviews them in the PR.
- Do not bump versions or write the changelog. Which edits survive is the user's decision, and the summary's reviewer checklist covers both.
Output
Reply with the PR URL, the summary table, any errors or validation failures, and these next steps, trimmed to what the run needs: drop step 3 when nothing was tuned, steps 5 and 6 when no edit was kept.
- Review the reports. Read each
audits/<date>/<plugin>.mdand check that every kept change cites a guide that actually backs it.<plugin>.doctor.mdis the raw/doctoroutput it started from. - Keep or drop edits. With
--apply, revert the hunks underplugins/you reject. Without it, apply the ones you keep by hand before merging:tuning.jsonalready records the guides as checked, so the next run will not propose them again. - Accept or reject pinned tuning. For each accepted file, set
tuned-forto itsmodelin the plugin'stuning.json. For a rejected one, revert the edit and leavetuned-foralone:proposed-forstops the same proposal from coming back. - Mirror shared content. If a kept edit touches a section listed in
docs/CROSS_REFERENCES.md, apply it to every copy in the same PR. - Bump versions. Each plugin with a kept edit gets a PATCH bump in both its
plugin.jsonand.claude-plugin/marketplace.json, and itsdocs/PLUGIN_VERSIONS.mdrow. A plugin whose only change istuning.jsongets none. - Changelog. Add the PR's entry under the date it was opened, per the root
CLAUDE.mdVersioning section, with a### Plugin versionstable if anything was bumped. - Handle failures. For an errored plugin or a failed validation, fix the cause, then re-run
/model-audit --plugins <plugin-path>. A report that looks weak can be re-run with--model opus. - Verify and merge. Run
python tools/validate-plugins/validate.py, optionallyclaude plugin evalfrom a plugin root, tick the PR checklist, mark it ready, and merge with a merge commit per the rootCLAUDE.md"Landing a branch".
Stop conditions
- Working tree has tracked changes → stop and say so. Do not stash or discard them.
ghis not authenticated or the push fails → report the failure and leave the commit in place.