mutation-testing
Measure and improve test-suite strength with mutation testing: pick a tool (mutmut or cosmic-ray for Python, cargo-mutants for Rust, Stryker for JS/TS, PIT for Java), scope runs to changed code, inter
SKILL.mdskills/mutation-testing/SKILL.md
GitHubMutation Testing
A mutation tool makes one small change to production code (flip < to <=, delete a statement, replace a return value) and reruns the tests. If a test fails the mutant is killed; if all pass it survived, which means no test would notice that bug. Coverage says code ran; mutation testing says a test would fail if the code were wrong.
Boundaries: writing the tests themselves is unit-testing / test-driven-development; stronger input exploration is property-based-testing and fuzz-testing; contract tests across services are contract-testing.
Non-negotiable rules
- The baseline suite must pass, deterministically, before mutating. A flaky baseline makes survivors meaningless. Fix or quarantine flakes first.
- Mutation tools execute your tests many times and write mutated source to disk. Run on a clean git branch or copy (cargo-mutants and Stryker work in a temp copy by default;
mutmut applywrites into your tree, so commit first). Never run against code with side effects on real systems (databases, cloud, payments); tests must already be hermetic. - A survived mutant is a question, not a defect count. Do not weaken production code or add
no mutatepragmas just to raise the score; every exclusion needs a written reason. - Never chase 100%. Equivalent mutants (behavior unchanged) make it unreachable and the chase produces test-shaped noise. Gate on a threshold and on no surviving mutants in the lines you changed that are semantically meaningful.
- Do not edit tests to assert implementation details (private calls, exact log strings) only to kill a mutant; assert observable behavior.
Workflow
1. Decide whether and where
Good targets: pure logic with branching (pricing, validation, parsers, state transitions, permission checks). Poor targets: generated code, config, logging, thin wrappers, UI layout. Start with one module or the diff of one PR, never the whole repo first: cost is roughly (number of mutants) x (test time), so a 2-minute suite with 5,000 mutants is days unless tests are selected per mutant.
2. Pick the tool and run a bounded pass
| Language | Tool | Scoping / speed levers |
|---|---|---|
| Python | mutmut (3.x), cosmic-ray | mutmut: set source paths in [tool.mutmut] of pyproject.toml (or setup.cfg), also_copy for needed files, do_not_mutate; # pragma: no mutate per line. Commands include mutmut run, mutmut browse, mutmut apply (check mutmut --help for result viewing on your version). cosmic-ray: cosmic-ray init <config.toml> <session.sqlite>, then cosmic-ray exec ..., report with cr-report |
| Rust | cargo-mutants | cargo mutants; --list shows mutants without running; --in-diff changes.diff mutates only code touched by a diff (generate with git diff); --package, --file, --re / --exclude-re (regex filters on mutant names; run cargo mutants --help for your version); --shard k/n splits across CI jobs (all shards need identical args); --baseline=skip after a known-good run; --timeout. Results in mutants.out. Skip code with the #[mutants::skip] attribute (add the mutants crate) |
| JS/TS | Stryker | stryker.config.json: mutate globs (or file:lines ranges), testRunner, coverageAnalysis (perTest is the default in recent versions), thresholds (high 80 / low 60 / break null by default; set break to fail the build), incremental: true to reuse results, ignoreStatic, concurrency, reporters |
| Java | PIT (pitest) | Maven: mvn test-compile org.pitest:pitest-maven:mutationCoverage; targetClasses, targetTests, mutationThreshold (percent), withHistory, mutators, timeoutConstant (default 4000 ms) and timeoutFactor (default 1.25). HTML report in target/pit-reports/ |
Always confirm option names against the installed version's --help/docs; these tools change defaults between majors.
3. Classify every survivor (this is the real work)
For each surviving mutant, in the report diff, answer in order:
- Equivalent? Does the mutated program behave identically on all inputs? Examples:
i < ntoi != nin a loop that always exits atn; changing a cache-capacity hint; reordering independent statements; mutating a value that is later overwritten;x >= 0tox > 0wherex == 0is unreachable by invariant. Equivalence is undecidable in general; judge by reasoning, and record the justification. Mark and exclude with the tool's mechanism plus a comment. - Untested behavior? (most common) Write a test that fails on the mutant and passes on the original: a boundary value, the other branch, an exact return value instead of
is not None, an assertion on a side effect. - Dead or redundant code? If no input can distinguish it, simplify the code (delete the branch) instead of testing it.
- Test is too coarse? The test runs the code but asserts nothing relevant (smoke test). Strengthen the assertion.
- Timeout? Mutant introduced an infinite loop; tools usually count this as detected. Verify the timeout is sane; if a test genuinely hangs you have a missing test timeout.
- No coverage? No test executes that line at all; add a test or accept (and document) that the code is untested/unneeded.
4. Kill with intent
Mutant: if amount > limit became if amount >= limit and survived. Missing test: amount == limit. Write it:
def test_exactly_at_limit_is_allowed():
assert check_limit(amount=100, limit=100) is True # boundary pinned
Rerun only that mutant (mutmut: run for that name; cargo-mutants: --file or --re; Stryker: mutate line range). Confirm killed, and confirm the new test passes on the original code.
5. CI strategy
- PR gate: mutate only changed lines/files (cargo-mutants
--in-diff; Stryker--incrementalplus mutate globs; PITwithHistoryor SCM goals; mutmut scoped to changed modules). Setbreak/mutationThresholdfor new code slightly below your current score and ratchet up. - Nightly: broader run, sharded (
cargo mutants --shard k/nin a GitHub Actions matrix withfail-fast: false, archivemutants.out), results tracked as a trend. - Keep a time budget: abort at N minutes and report partial results rather than silently skipping.
- Never block merges on survivors in files with no behavior change (refactors, docs).
6. Report with evidence
Report: tool+version, scope, mutants generated, killed/survived/timeout/no-coverage counts, score = (killed + timeouts) / (generated minus unviable and minus mutants you have justified as equivalent); timeouts count as detected, no-coverage mutants count as not detected (state this convention, since tools differ), list of survivors with disposition (killed by new test X, equivalent because Y, deferred). A score alone is not evidence.
Worked example (Python)
def apply_discount(price: int, pct: int) -> int:
if pct < 0 or pct > 100:
raise ValueError("pct")
return price - price * pct // 100
Existing test: assert apply_discount(200, 50) == 100. Typical survivors: pct > 100 to pct >= 100 (no test at 100), pct < 0 to pct <= 0 (no test at 0), error-path message change (no test of exception), // to / (200*50 is exactly divisible; integer-vs-float result still == in Python, so only a type assertion or an indivisible case such as apply_discount(99, 33) exposes it).
Add tests at 0, 100, -1, 101 with pytest.raises(ValueError), and apply_discount(99, 33) == 67 (checking exact integer arithmetic: 99 - 32 = 67). Survivors now killed, except mutating the ValueError("pct") message. That survivor is not equivalent (behavior does change); label it an accepted survivor / unspecified behaviour, unless the message is part of the API, in which case assert it with match=. Record the decision either way.
Anti-patterns
- Running everything, waiting hours, then ignoring the report.
- Optimizing the percentage: adding
no mutatepragmas, deleting hard-to-kill code, or writing tests that mirror the implementation. - Treating timeouts as failures of the tool and discarding them.
- Using mutation score to compare teams or as a performance metric.
- Running on a flaky or order-dependent suite; parallelization then produces false survivors.
- Forgetting that mutation tools mutate the code under test, not the test code.
- Applying mutants (
mutmut apply) and leaving them in the working tree.
Verification checklist
- Baseline passes cleanly and deterministically
- Scope bounded (module or diff) and time budget stated
- Every survivor has a disposition: killed, equivalent (with reason), dead code removed, or deferred
- New tests fail against the mutant and pass on original code
- No source mutations left applied; no unexplained exclusion pragmas
- CI gate threshold and ratchet plan documented