Test-Driven Development

TDD's value is not the test file. It is the observed RED: the one moment you have evidence the test can fail. Skip it and you have written an assertion nobody has ever seen return "no" — decoration with a green checkmark.

This repo has paid for that. A fraud scorer shipped with a fully green suite; adversarial review then found 16 real defects (bug-150). The tests proved each rule fires. Nobody had ever watched them fail for the right reason.

When to use

Use it when:

• adding a feature or endpoint under agents/, api/, services/, skyyrose/
• fixing a bug — the regression test is written first and must reproduce the bug
• refactoring: the suite is the safety net, so it must be green before you touch code
• changing a gate, guard, or validator — those need an explicit fail-closed test

Do not use it:

• to backfill tests onto already-shipped code. That is a coverage-gap task with a different shape (read the code, find the untested branch); there is no RED to observe.
• to judge whether someone else's green suite means anything — that is verification-loop plus adversarial-verification.
• for pure config/docs edits with no behaviour.

Inputs

Required	How to get it	If absent
A green baseline before you start	rtk proxy pytest tests/ -q → 6546 passed, 0 failed	Stop. On a red baseline you cannot tell your break from the existing one. Fix or quarantine first, and name who owns the pre-existing failure.
The right runner for the surface	Python → rtk proxy pytest · frontend/ → npm test (vitest) · flows → npx playwright test	Stop. A bare pytest in this repo can print "no tests collected" and read as success.
A stated behaviour, not a stated function	"invalid SKU raises ValueError", not "test resolve_image"	Stop and restate. A test named after a function tests whatever the function does, including the bug.
A writable per-test temp dir (tmp_path)	pytest fixture	Never write to tracked files. bug-153: a test injected tokens into the real shared design-tokens.css and failed only under the full suite.
For frontend/: the suite must be importable under vitest	vitest.config.ts uses an explicit include (lib/wp/**, tests/**)	Stop. A suite outside include is silently never run — and a skipped security test is indistinguishable from a passing one.

Procedure

1. Record the baseline. Run the suite for the surface you are about to touch and note the exact counts. You will diff against these, not against "green".
2. Write the test first, naming the behaviour and the failure it forbids. One behaviour per test; arrange-act-assert.
3. Run it and read the failure text. RED must be the right red — an assertion about your behaviour, not ImportError, not a fixture typo. A test that errors before reaching its assertion has not demonstrated anything.
4. Write the minimum implementation to turn it green. No speculative branches, no abstraction for a single call site.
5. Re-run the single test, then the surface's full suite. A test that passes alone and fails in the suite is shared-state pollution (bug-231, ×5), not flake.
6. Prove the test can still fail — extract the pre-change file with git archive HEAD <path> | tar -x -C <scratch> and run your new test against that pristine tree. It must fail there. Never git stash: the stash stack is shared across worktrees.
7. Check coverage for the module you touched, not repo-wide — the repo number moves too slowly to show your delta.
8. Refactor with the suite green, re-running after each step.
9. Never weaken a test to make it pass. Loosening an assertion, adding xfail, or widening a tolerance to reach green converts a real failure into a silent one.

A SKIP is not a PASS. This worktree is sparse, so asset-dependent tests skip by design (bug-257). Report them as "not executed here; closed by CI / the full checkout" — never inside a passing count.

Verification

1 — The new test is green in isolation and in the suite.

rtk proxy pytest tests/mcp/test_http_mount.py -q && rtk proxy pytest tests/ -q

PASS: the single file passes, and the full run reports N passed with 0 failed and N >= 6546 (the baseline). A drop in total count means you deleted or de-collected tests. [test]

2 — RED was real: the test fails against the pristine tree. This is the prove-it-can-fail rule and the attribution rule in one move.

mkdir -p /tmp/tdd-pristine && git archive HEAD mcp_tools/ | tar -x -C /tmp/tdd-pristine
rtk proxy pytest tests/mcp/test_http_mount.py -q --rootdir=/tmp/tdd-pristine

PASS-OF-THE-PROOF: this run fails. If your new test passes against code that predates your change, it is not testing your change. [test]

3 — Coverage on the module you touched.

rtk proxy pytest tests/mcp/ -q --cov=mcp_tools --cov-report=term-missing

PASS: the touched module reports ≥ 85% and the Missing column contains no line from your new code. [repro]

4 — Frontend suites are actually collected. vitest's explicit include makes silent non-collection the default failure here.

cd frontend && npm test 2>&1 | tail -5

PASS: the summary names your spec file. If your file is absent from the output, it was never run — fix vitest.config.ts include or move the suite. [repro]

5 — Flows: the Playwright spec is registered.

cd frontend && ./node_modules/.bin/playwright test --list | tail -1

PASS: the total count went up by your test count. testDir is ./tests/e2e; a spec written into the legacy frontend/e2e/ is never executed and shows here as an unchanged total. [repro]

A run that errored, timed out, or hit a session limit is not a pass. Its empty failure list is an artifact — re-run it by hand (bug-230).

Worked example

Regression test for bug-211 (MCP bearer token compared with !=; fail-open when MCP_SERVICE_TOKEN is unset). Real commands in this worktree, 2026-07-29.

Baseline for the surface:

$ rtk proxy pytest tests/mcp/test_http_mount.py -q
......                                                                   [100%]

[repro] — 6 passed. That is the number the next run is diffed against.

The behaviour under test, stated before the code: "an Authorization header that shares a prefix with the real token is rejected, and the comparison is constant-time" — which is why the implementation reads:

$ grep -n "compare_digest" mcp_tools/http_mount.py
88:        if not hmac.compare_digest(provided, f"Bearer {token}")

[repo]. The RED proof is the interesting half: run the same test file against the pre-fix tree via git archive, and it must fail on the != comparison. Running it only against the fixed tree — where it passes — would have demonstrated nothing about the test.

The contrast case, same session, same runner:

$ rtk proxy pytest tests/test_asset_manifest.py -q
SKIPPED [1] tests/test_asset_manifest.py:42: sparse worktree deliberately excludes
  assets/products; this gate runs in full checkouts and CI
SKIPPED [1] tests/test_asset_manifest.py:51: ... (×4 total)

[repro] — 4 skipped, 0 passed. Reported as "asset-manifest integrity did not run in this worktree". Rolling those four into a "tests pass" line is the exact bug-257 defect.

Failure modes

Symptom	What is really happening	Bug
Suite fully green, adversarial review finds many real defects	Tests proved each rule fires; none exercised calibration, boundaries, or the never-raises guard. Green ≠ correct.	bug-150
Test passes alone, fails in the full suite	Shared state — module globals, a real tracked file written in place, ordering under asyncio_mode=auto. Use tmp_path / monkeypatch.	bug-231 (×5), bug-153
Test skipped everywhere, counted as passing	A SKIP is not a PASS. Name who closes it (CI / full checkout).	bug-257
pytest says "no tests collected", read as success	Bare pytest misreports in this repo. Always rtk proxy pytest.	—
New frontend spec never appears in output	Outside vitest.config.ts's explicit include, or written into the legacy frontend/e2e/ that Playwright's testDir excludes.	—
Test made to pass by loosening its assertion	The failure was real and is now silent. Fix the cause, never the assertion.	—
RED never observed; test written after the code	No evidence the assertion can fail. Re-derive via git archive against the pristine tree.	—
git stash used to get a clean tree for the RED check	The stash stack is shared across worktrees; you can pop another session's work.	—
Multiprocess/fork test SIGSEGVs on macOS	Darwin fork safety — no_proxy='*' must be pre-set and the resource tracker pre-spawned.	bug-263 (×7)
Coverage measured repo-wide, shows "no change"	Your delta is invisible in a 6546-test total. Measure --cov=<touched module>.	—