mutation-testing
Validates test quality beyond coverage: uses Claude subagents to generate intelligent mutations for source code, runs them via the project's native test runner (pytest, jest, vitest, mocha, go test),
SKILL.mdplugins/mutation-testing/skills/mutation-testing/SKILL.md
GitHubMutation Testing
Tests pass ≠ tests are useful. This skill uses Claude to generate meaningful mutations for source code, then runs the project's existing test suite against each one. A surviving mutant is a gap — a real bug the suite would miss.
What this skill does
- Discovers source files — the calling agent reads the project structure and selects files
- Detects test runner — the calling agent inspects project config and builds the test command
- Checks baseline — verifies the test suite passes before mutating anything
- Generates mutations via subagents — dispatches one Task per source file in parallel; each
subagent reads the file and returns a JSON array of mutations; results are merged into a single
mutations.jsonfile - Applies each mutation — writes the mutated file to disk, one at a time
- Runs the test suite — records
Killed(tests failed) vs.Survived(tests passed) - Restores each file — always restores the original before the next mutation
- Generates mutation-todos.md — table of survivors with rationale for each gap
- Persists to logbook — SQLite + JSONL under
./.logbooks/mutation-testing/ - Warns if score < 70% — exits with code 2
How to invoke this skill
This skill follows a two-phase protocol: generate then run.
Phase 1 — Generate mutations (caller's responsibility)
Before dispatching subagents, the calling agent must:
Discover source files by reading the project structure. Prefer
src/,lib/directories; skip test files (test_*.py,*_test.py,*.spec.*,*.test.*, paths withtest/ortests/), and skip non-source directories (.venv,.git,dist,build,node_modules,__pycache__,site-packages). Cap at 20 files.Detect the test runner by inspecting the project root, then build the full test command:
Signal Command pytest.ini,conftest.py, or[tool.pytest]inpyproject.tomlpython -m pytest --tb=no -q --no-headerjest.config.*or"jest"inpackage.jsondevDependenciesnpx jest --passWithNoTests --forceExit"vitest"inpackage.jsondevDependenciesnpx vitest run"mocha"inpackage.jsondevDependencies (no babel)npx mocha --recursive"mocha"+"@babel/register"inpackage.jsondevDependenciesnpx mocha --require @babel/register --recursiveAny *.gofile presentgo test ./..."test"script inpackage.jsonnpm testPass the full command via
--test-commandin Phase 2.
Then, for each source file to mutate, dispatch a subagent with this prompt template:
You are a mutation testing expert. Generate up to
{max_per_file}meaningful mutations for the following source file that will expose gaps in the test suite.Already-known open gaps (do NOT regenerate these — find new ones):
{known_gaps_for_file}File:
{rel_path}{line_numbered_source}Return a JSON array. Each element must be an object with these keys:
id(string) — unique ID, e.g."m1"mutatorName(string) — operator name, e.g."FlipGreaterThan"line(integer) — 1-based line numberoriginal_line(string) — complete original line, leading whitespace preservedmutated_line(string) — complete mutated line, leading whitespace preservedreplacement(string) — short change description, e.g."> → >="rationale(string) — why this exposes a test gapfile(string) — use exactly the path from the "File:" line above, with forward slashesTarget: conditional flips (
>→>=), boolean negation, arithmetic swaps, return value changes, string literal blanking, boundary shifts, argument removal/swap. Skip: comments, imports, type annotations, docstrings, blank lines, logging.
Collect all subagent outputs, merge into a single JSON array, and save to a temp file
(e.g. /tmp/mutations.json).
Tips:
- Dispatch subagents in parallel (one per file) for speed
- Pass
--model claude-sonnet-4-6as a logbook hint if using a cheaper model - Query prior open gaps first (from the logbook SQLite) and pass them per-file so subagents generate novel mutations rather than re-discovering known gaps
Phase 2 — Run the script
python plugins/mutation-testing/skills/mutation-testing/scripts/mutation_testing.py \
--mutations-file /tmp/mutations.json \
--test-command python -m pytest --tb=no -q
With options:
# Jest project
python ... --mutations-file /tmp/mutations.json \
--test-command npx jest --passWithNoTests --forceExit
# Custom threshold and timeout
python ... --mutations-file /tmp/mutations.json --threshold 80 --timeout 120 \
--test-command python -m pytest --tb=no -q
# Record which model was used for generation (metadata only)
python ... --mutations-file /tmp/mutations.json --model claude-sonnet-4-6 \
--test-command python -m pytest --tb=no -q
# Skip logbook writes
python ... --mutations-file /tmp/mutations.json --no-logbook \
--test-command python -m pytest --tb=no -q
Note:
--test-commandmust be the last flag — all remaining arguments after it are captured as the test command.
CLI flags
| Flag | Default | Description |
|---|---|---|
--mutations-file PATH |
required | JSON file of pre-generated mutations |
--test-command WORD … |
required | Full test command (must be last flag) |
--threshold |
70 |
Minimum acceptable mutation score |
--timeout |
60 |
Seconds allowed per test run |
--model |
none | Claude model used for generation — recorded in logbook as metadata (optional) |
--skip-baseline |
off | Skip the pre-mutation baseline pass check |
--no-logbook |
off | Skip SQLite/JSONL logbook writes |
Exit codes
| Code | Meaning |
|---|---|
0 |
Success — score ≥ threshold |
1 |
Setup, load, or baseline failure; or all mutations skipped |
2 |
Score below threshold |
Prerequisites
- Python 3.10+ (uses
match/caseandstr.removesuffix) - A working test suite (tests must pass before running mutation tests)
- At least one source file (discovered by the calling agent)
- No SDK or API key required — mutation generation is done by the calling Claude agent
Mutant statuses
| Status | Meaning |
|---|---|
Killed |
Tests detected the bug — good |
Survived |
No test caught the bug — test gap |
Error |
Test runner failed or errored |
Skipped |
Original line didn't match the expected text |
Score formula
score = Killed / (Killed + Survived + Errors) × 100
Skipped mutants are excluded from the denominator. If all mutants are skipped, the script exits 1 (setup issue, not a score failure).
| Score | Assessment |
|---|---|
| ≥ 90% | Strong test suite |
| 70–90% | Acceptable — review survivors |
| < 70% | Weak — prioritise mutation-todos.md |
Output artifacts
| Artifact | Location | Description |
|---|---|---|
mutation-todos.md |
repo root | Survivors table with rationale |
| SQLite ledger | ./.logbooks/mutation-testing/{slug}.sqlite |
Run history + survivors |
| JSONL trace | ./.logbooks/mutation-testing/{slug}.jsonl |
Append-only run log |
Logbook spec
Full schema, query examples, and export setup: mutation-runs.logbook.md.