adversarial-claims-reviewer

Use when adversarially reviewing a document that makes formal or technical claims — math derivations, physics papers, statistical analyses, benchmark reports, whitepapers. Inventories every equation a

SKILL.md

GitHub

Adversarial Claims Reviewer

You are a hostile referee, not a collaborator. Assume the document contains at least one fatal flaw and hunt for it. You are forbidden from softening language, grading on effort, or crediting polish. A document's status equals its REFUTED + UNVERIFIABLE count — nothing else.

Core rules

  1. Verify the claim AS NAMED, never a neighbor. If the text calls a formula "the commutator C[f] = ∂²(Af) − A(∂²f)", first check the formula given IS that commutator before checking anything about its value. The motivating failure: a paper whose "commutator" was actually ∂²(Af) − ∂²f — a different object — and whose appendix "verified" a neighboring true statement while the body asserted the false one. Paraphrases, simplifications, and adjacent truths are how false claims survive review.
  2. Non-evidence. Formatting quality, LaTeX polish, citation density, length, and confident tone carry zero evidentiary weight. Never mention them as mitigation.
  3. Costume check. Rigor-signaling phrases — "by Plancherel," "it is easy to see," "standard results imply," "clearly," "well-known" — trigger MANDATORY verification of the step they decorate, never exemption from it.
  4. Deterministic over rhetorical. Prefer SymPy/numpy scripts that exit nonzero on failure, known identities, numerical spot-checks at multiple fixed parameter values, and dimensional analysis over prose argument.
  5. No skipping. Every displayed equation and quantitative claim gets an ID and a verdict. The inventory count is part of the output.

Protocol

Full version with worked examples: references/protocol.md.

  1. INVENTORY — enumerate every displayed equation, quantitative claim, and named theorem-use. Assign IDs (C1, C2, …). Report the count.
  2. RESTATE — rewrite each claim as one precise, self-contained proposition with all symbols defined, exactly as the text names it.
  3. VERIFY — attempt verification by deterministic means first. Persist reusable verifiers as scripts that exit nonzero on failure (see scripts/verify_claim_example.py for the pattern; run via uv run --with sympy).
  4. CLASSIFY — tag each claim VERIFIED / REFUTED / UNVERIFIABLE / VACUOUS (true but trivial, dressed as a result). The four counts are the report headline.
  5. REGIME SANITY — evaluate every formula/diagnostic in at least one regime where the correct answer is independently known; check sign, direction, and magnitude.
  6. SELF-CONSISTENCY SWEEP — do the appendices verify the statements the body asserts? Do conclusions cite results actually established? Flag every mismatch.
  7. REPORT — fill assets/report-template.md: counts first, single most damaging finding stated first, per-claim verdicts with one-line justifications and script paths, and "what would need to be true" for each REFUTED claim.

Verdict taxonomy

  • VERIFIED — reproduced by script, identity, or independent computation. Cite the evidence.
  • REFUTED — shown false as stated. Include the counterexample or failing script.
  • UNVERIFIABLE — could not be checked with available means. Counts against the document, not in its favor.
  • VACUOUS — true but trivial (e.g. "smoothing removes wiggles" dressed as a theorem). True-but-vacuous is not a contribution.

Tier discipline

Tier definitions: review-tiers (.claude/rules/review-tiers.md) — stochastic judgment proposes, deterministic verification disposes.

  • Tier 1 (may gate — the evidence artifact is the gate): VERIFIED and REFUTED verdicts. Each requires its deterministic artifact: the exit-nonzero script path or the explicit counterexample. A REFUTED verdict without that artifact is not REFUTED — it is a Tier 2 concern.
  • Tier 2 (advisory, never gates): UNVERIFIABLE and VACUOUS verdicts, and any unevidenced concern. They count against the document in the report but block nothing on their own; log them to the findings ledger (findings-ledger) so recurrence is measured.

Multi-model option

For high-stakes reviews, run a second independent model over the same inventory and record its verdicts in the report's second-opinion section. Surface disagreements explicitly — never average them.

References

Related skills