mutation-testing

Validates test quality beyond coverage: uses Claude subagents to generate intelligent mutations for source code, runs them via the project's native test runner (pytest, jest, vitest, mocha, go test),

SKILL.md

GitHub

Mutation Testing

Tests pass ≠ tests are useful. This skill uses Claude to generate meaningful mutations for source code, then runs the project's existing test suite against each one. A surviving mutant is a gap — a real bug the suite would miss.

What this skill does

  1. Discovers source files — the calling agent reads the project structure and selects files
  2. Detects test runner — the calling agent inspects project config and builds the test command
  3. Checks baseline — verifies the test suite passes before mutating anything
  4. Generates mutations via subagents — dispatches one Task per source file in parallel; each subagent reads the file and returns a JSON array of mutations; results are merged into a single mutations.json file
  5. Applies each mutation — writes the mutated file to disk, one at a time
  6. Runs the test suite — records Killed (tests failed) vs. Survived (tests passed)
  7. Restores each file — always restores the original before the next mutation
  8. Generates mutation-todos.md — table of survivors with rationale for each gap
  9. Persists to logbook — SQLite + JSONL under ./.logbooks/mutation-testing/
  10. Warns if score < 70% — exits with code 2

How to invoke this skill

This skill follows a two-phase protocol: generate then run.

Phase 1 — Generate mutations (caller's responsibility)

Before dispatching subagents, the calling agent must:

  1. Discover source files by reading the project structure. Prefer src/, lib/ directories; skip test files (test_*.py, *_test.py, *.spec.*, *.test.*, paths with test/ or tests/), and skip non-source directories (.venv, .git, dist, build, node_modules, __pycache__, site-packages). Cap at 20 files.

  2. Detect the test runner by inspecting the project root, then build the full test command:

    Signal Command
    pytest.ini, conftest.py, or [tool.pytest] in pyproject.toml python -m pytest --tb=no -q --no-header
    jest.config.* or "jest" in package.json devDependencies npx jest --passWithNoTests --forceExit
    "vitest" in package.json devDependencies npx vitest run
    "mocha" in package.json devDependencies (no babel) npx mocha --recursive
    "mocha" + "@babel/register" in package.json devDependencies npx mocha --require @babel/register --recursive
    Any *.go file present go test ./...
    "test" script in package.json npm test

    Pass the full command via --test-command in Phase 2.

Then, for each source file to mutate, dispatch a subagent with this prompt template:

You are a mutation testing expert. Generate up to {max_per_file} meaningful mutations for the following source file that will expose gaps in the test suite.

Already-known open gaps (do NOT regenerate these — find new ones): {known_gaps_for_file}

File: {rel_path}

{line_numbered_source}

Return a JSON array. Each element must be an object with these keys:

  • id (string) — unique ID, e.g. "m1"
  • mutatorName (string) — operator name, e.g. "FlipGreaterThan"
  • line (integer) — 1-based line number
  • original_line (string) — complete original line, leading whitespace preserved
  • mutated_line (string) — complete mutated line, leading whitespace preserved
  • replacement (string) — short change description, e.g. "> → >="
  • rationale (string) — why this exposes a test gap
  • file (string) — use exactly the path from the "File:" line above, with forward slashes

Target: conditional flips (> → >=), boolean negation, arithmetic swaps, return value changes, string literal blanking, boundary shifts, argument removal/swap. Skip: comments, imports, type annotations, docstrings, blank lines, logging.

Collect all subagent outputs, merge into a single JSON array, and save to a temp file (e.g. /tmp/mutations.json).

Tips:

  • Dispatch subagents in parallel (one per file) for speed
  • Pass --model claude-sonnet-4-6 as a logbook hint if using a cheaper model
  • Query prior open gaps first (from the logbook SQLite) and pass them per-file so subagents generate novel mutations rather than re-discovering known gaps

Phase 2 — Run the script

python plugins/mutation-testing/skills/mutation-testing/scripts/mutation_testing.py \
  --mutations-file /tmp/mutations.json \
  --test-command python -m pytest --tb=no -q

With options:

# Jest project
python ... --mutations-file /tmp/mutations.json \
  --test-command npx jest --passWithNoTests --forceExit

# Custom threshold and timeout
python ... --mutations-file /tmp/mutations.json --threshold 80 --timeout 120 \
  --test-command python -m pytest --tb=no -q

# Record which model was used for generation (metadata only)
python ... --mutations-file /tmp/mutations.json --model claude-sonnet-4-6 \
  --test-command python -m pytest --tb=no -q

# Skip logbook writes
python ... --mutations-file /tmp/mutations.json --no-logbook \
  --test-command python -m pytest --tb=no -q

Note: --test-command must be the last flag — all remaining arguments after it are captured as the test command.

CLI flags

Flag Default Description
--mutations-file PATH required JSON file of pre-generated mutations
--test-command WORD … required Full test command (must be last flag)
--threshold 70 Minimum acceptable mutation score
--timeout 60 Seconds allowed per test run
--model none Claude model used for generation — recorded in logbook as metadata (optional)
--skip-baseline off Skip the pre-mutation baseline pass check
--no-logbook off Skip SQLite/JSONL logbook writes

Exit codes

Code Meaning
0 Success — score ≥ threshold
1 Setup, load, or baseline failure; or all mutations skipped
2 Score below threshold

Prerequisites

  • Python 3.10+ (uses match/case and str.removesuffix)
  • A working test suite (tests must pass before running mutation tests)
  • At least one source file (discovered by the calling agent)
  • No SDK or API key required — mutation generation is done by the calling Claude agent

Mutant statuses

Status Meaning
Killed Tests detected the bug — good
Survived No test caught the bug — test gap
Error Test runner failed or errored
Skipped Original line didn't match the expected text

Score formula

score = Killed / (Killed + Survived + Errors) × 100

Skipped mutants are excluded from the denominator. If all mutants are skipped, the script exits 1 (setup issue, not a score failure).

Score Assessment
≥ 90% Strong test suite
70–90% Acceptable — review survivors
< 70% Weak — prioritise mutation-todos.md

Output artifacts

Artifact Location Description
mutation-todos.md repo root Survivors table with rationale
SQLite ledger ./.logbooks/mutation-testing/{slug}.sqlite Run history + survivors
JSONL trace ./.logbooks/mutation-testing/{slug}.jsonl Append-only run log

Logbook spec

Full schema, query examples, and export setup: mutation-runs.logbook.md.