Detection validation

Purple team answers one question, per technique: we executed it — did a detection fire? Everything else is commentary.

"We think we're covered" is not coverage. Coverage is a specific claim about specific techniques, backed by a recorded outcome for each. A map with blanks in the detection column is the same lie as a port scan that came back clean with zero coverage — a confident summary over an unanswered question.

When to Use

• Running a purple-team exercise, measuring detection coverage, validating that a SOC can see a technique, or turning 'we think we're covered' into evidence per technique.
• Use only within explicit authorization and the stated rules of engagement.

When NOT to Use

• Do not treat this skill as permission to access, test, or change a target.
• Do not convert missing evidence into a clean result or a stronger claim.

Every executed technique gets exactly one honest outcome

Outcome	Meaning
covered	executed, a detection fired
partial	fired weakly, or on only some variants
not-detected	executed, nothing fired — a real gap
not-detectable	passive technique with no observable (OSINT, reading public records) — an accepted gap, not a failure
unknown	executed but the outcome was never recorded — the one that poisons the report

The distinction that matters: not-detectable is honest (you looked, there is nothing to see, you accept it) while unknown is a blank wearing a confident coverage number. They must never be conflated.

Score the map

uv run {baseDir}/scripts/coverage_map.py map.json

Computes real coverage as covered+partial over everything executed that could be detected — passive not-detectable techniques are removed from the denominator. Exit 0 only when every executed technique has a known, non-gap outcome; 1 when any not-detected or unknown remains; 2 on bad or empty input. See references/coverage-map.md.

Validation record

For each technique, return the technique and variant, execution receipt, expected observable, telemetry-health evidence, alert evidence, one outcome from the table above, limitations, and an owner for any gap. Keep the execution receipt and alert receipt separate: the same timestamp or operator note cannot prove both sides of the join.

End with counts for covered, partial, not-detected, not-detectable, and unknown, plus the denominator used. Never publish only a percentage.

Pair with the attacker and the detector

This skill is the join between offense and defense:

• The offensive technique comes from the operator (or an attack simulation). If you cannot execute it, you cannot validate its detection — an untested technique is unknown, never assume covered.
• The detection that should fire is engineered and proven separately. A rule that passed its own true-positive test in isolation can still fail to fire in a live exercise against the real technique variant — that is exactly what this measures, and why the exercise is not redundant with the rule's unit test.

Rationalizations to reject

Thought	Reality
"We ran 40 techniques, we're covered."	Not until all 40 have a recorded outcome. Blanks are unknown, and unknown is not coverage.
"No alert fired, so the technique must be benign."	No alert fired means you did not detect it. That is a gap, not an all-clear.
"It's probably detected, the rule exists."	A rule existing is not a rule firing. The exercise exists because rules fail live that passed in isolation.
"Mark the passive stuff as covered so the number looks complete."	not-detectable is removed from the denominator, not counted as covered. Honest beats complete.
"We'll fill in the blanks later."	A blank is unknown now. Ship the map honest with the gaps visible, or re-run the technique.
"100% coverage."	Almost always means the un-run techniques were left off the map. Coverage is over what you executed, and the map must show what you did not.

References

• Coverage map format — the JSON the scorer consumes, the five outcomes, and how the denominator is computed