ci-suite-audit
Audit registered CI test suites and recommend retention, split, nightly, quarantine, or turn-off verdicts based on runtime receipts, failure history, touch-set overlap, and prose/junk assertion analys
SKILL.mdskills/4-occasional/ci-suite-audit/SKILL.md
GitHubci-suite-audit — Test Suite Curation & Retention Triage Playbook
A structured, evidence-governed method for evaluating and curating test suites in large CI registries. It inspects runtime performance, defect and flake history, touch-set overlap, sibling coverage, and assertion quality (prose and junk checks) to classify test suites into actionable, defensible verdicts.
Purpose & Mission
The audit exists to reverse test registry expansion (#853: 207 → 419 in 41 days; the GH-854 CI stabilization objective) by proposing KEEP, NIGHTLY, MERGE, QUARANTINE, or TURN-OFF (TURN-OFF means unregister and add to EXEMPT or unpin from registry). The skill aggressively curates dead, redundant, prose-only, flake-prone, and unbacked test suites back down to a fast, reliable, regression-focused gate while fiercely protecting verified regression guards. Reports must explicitly state the total count and share of suites proposed to leave the PR gate.
The skill serves as the canonical curation method for full-suite audits (GH-854, GH-862).
Operating Rules & Core Constraints
- Manual Invocation Only: The skill is invoked by an operator for occasional audits (e.g. quarterly sweeps or after major CI churn windows). It is not automated, not scheduled, not wired into CI workflows, and does not run on pull requests.
- Text-Only Skill: Contains instructions and documentation only. No executable scripts live in the skill directory or under
utils/,scripts/, orbin/. Any one-off extraction or scoring code needed during an audit belongs in that run's evidence directory (TESTS-RESULTS/<date>+GH-<n>/). Scoring code and input tables (failure records, coverage maps, duplicate contracts) must derive every field from saved source artifacts. Hand-typed tables keyed by suite name and name-based verdict branches are forbidden. Code from a previous run may be reused only if it meets the same rule. - Target Branch Requirement: Run against
developmentor the active stabilization staging branch such asstaging/stabilize-2026-10; record the full 40-character target tip SHA. Another ref requires the operator to name it and the report to explain why. Never auditmain, which retains obsolete pre-freeze suites and stale registries. - Proposes Verdicts; Humans Decide: The skill generates structured recommendations with citations and confidence ratings. It never directly edits
validate.sh,test/,utils/ci-route.sh, or GitHub Actions workflows, never deletes files, and never opens pull requests. Changes to the registry or test files must be reviewed and approved by an operator per verdict class. - Restricted GitHub Writes: The skill's only permitted GitHub write operations are creating/updating its own dedicated report issue and appending per-turn decision comments on that issue (see Report Issue & Per-turn Comments). It never modifies or comments on any other issue or PR.
- Freeze & No New Tests Compliance (GH-831): The skill respects repository test freezes. It introduces no new gate machinery, no new runner scripts, and no new test arrays (
NIGHTLY_TESTSstays out of tree; candidates remain inTESTS). Quarantined suites use the existing unregister-and-exempt mechanism intest/gh306-registry-bidirectional.sh. - No Keep-by-Default (Honest Metrics): Anything not measured is
UNKNOWN; nothing is estimated silently. There is no default verdict. A row getsKEEPonly when D2 has an observed execution denominator $N > 0$ for that suite and D3/D5 produced outputs for it. Otherwise the verdict isINVESTIGATEand every unmeasured field isUNKNOWN. A suite must earnKEEPthrough verified regression catching, unique core contract guarding, or confirmed passing behavioral execution. - Multi-Source Failure Signals: Collect four sources in a stated window: (1) hosted full-registry job logs, (2) issues labelled
ci,stability, orbugthat name an exact suite, (3)fix:/hotfix:commits touchingtest/, and (4) committed receipts with a failingevent:"suite"record. Record the searched N and window for each source. Only executed runs contribute to a suite'sk of Ndenominator; issues and commits provide attribution, not extra runs. Uncommitted local logs are leads, not auditable failure counts. - Exact Entity Matching: Known flakes, defects, and tracking umbrellas must be matched by exact suite filename or canonical mapping. Loose prefix matching (e.g.
gh492matchinggh492-roadmap-state-sweep.shinstead ofgh492-idle-kill.sh) is strictly prohibited. - Log Evidence Preservation: Save each downloaded or extracted workflow log, run summary, or receipt into the evidence folder (
TESTS-RESULTS/<date>+GH-<n>/) the first time it is read.
Pre-Flight Calibration Step (Mandatory Gate)
Before scoring any suites in the active registry, the skill must execute a pre-flight calibration test against the 8 known bad suites turned off by operator decision #831 (their files remain on disk):
gh578-ci-optimize-skill.shgh778-review-code-skill.shgh798-status-skill.shgh779-radar-ci-health.shgh781-wam-radar-seed.shgh615-start-task-reinforce.shgh616-start-task-commensurate-envelope.shgh617-relay-xyz-commensurate-review.sh
Calibration Criteria:
- At least 7 of the 8 suites must evaluate as
TURN-OFForMERGEunder the skill's detector rules. gh798-status-skill.shmust be classified as a wording-only/prose suite (TURN-OFF).- Hard Stop: If the calibration step fails (fewer than 7 turn-offs or
gh798not classified as wording-only/TURN-OFF), the audit run is declared INVALID and must immediately halt without publishing or acting on any verdicts. - Score calibration rows with the same saved inputs and scoring pass as target rows; save them as
calibration.tsvand generate the report table from that file. Check the calibration verdicts before publishing any target verdict. A separate hand-assigned calibration table cannot pass this gate.
Unit & Data Access Modes
Unit of Analysis
The unit of curation is one entry in the TESTS=(...) array in validate.sh.
- Anything the suite executes via
bash,python3,node, or binary invocation belongs to its unit (e.g.gh436-merge-cleanup.shrunninggh436-merge-cleanup.py). - Sourced libraries (
test/_setup.sh,test/lib/*.sh) provide shared fixture and containment context. - Count the non-empty
TESTSentries directly fromvalidate.sh; do not compare only withvalidate.sh --list, which reads the same registry. Require the operator's expected count, stop on disagreement, and assertTESTS ∩ EXEMPT = ∅. RecordEXEMPTseparately, including the #831 eight, and score those eight only in calibration. Print the full target SHA and the SHA/date of the newest commit that changedvalidate.sh's registry. Subdirectory suites may have their own pins and need noEXEMPTentry. - Subdirectory Suites: Note that
test/gh306-registry-bidirectional.shEXEMPTonly governs top-leveltest/*.shfiles. Subdirectory suites (e.g.synthetic/*) that are turned off come out ofTESTSand their own registry pin (test/gh141-synthetic-registry.sh), not into gh306EXEMPT.
Access Modes
- In-Checkout Mode (Preferred):
- The operator invokes the skill inside an active, clean repository checkout on the target branch (e.g.
developmentor staging branch). - The skill does not pull or clone automatically.
- Uses local disk tools (
rg, file inspections, committed receipts) to inspect full source, helpers, and test definitions across all registered suites.
- The operator invokes the skill inside an active, clean repository checkout on the target branch (e.g.
- Connector-Only Mode (Fallback):
- Used when operating without a local workspace clone, reading registry files, receipts, and workflow logs via the GitHub connector.
- Full source reads are prioritized for: every non-KEEP candidate, the top 10 heavy suites by runtime, every
#853isolation member, and every suite with a failure in the analysis window. - Suites evaluated solely from run logs or labels are
INVESTIGATE(LOW) until their source is read. They cannot earnKEEPor a concrete removal verdict from those signals alone.
Data Limitations & Disclosures
Every audit report must explicitly disclose data boundaries:
- Committed Receipts: Committed validation receipts represent passing (
green) runs by construction; they provide accurate runtime distributions but no failure signal. - Hosted CI Logs: Historical failure logs are extracted from hosted
validate.shsummaryfailed:blocks. Log retention is subject to GitHub Actions artifact windows (typically 14–90 days). - Local Gate Receipts: Local gate runs (
validation.jsonl,ci-local.shlogs) supply historical failure signals not captured in hosted runs. - Label Coverage: Some test runs or shims may produce partial test label manifests. Record the measured labeled count; unlabeled suites are
UNKNOWNfor label-based heuristics. - Unmeasured Signals: Any signal, log, or receipt not directly observed or parsed is recorded as
UNKNOWN; it is never estimated silently.
Inputs
| Signal | Source | Collection Rule |
|---|---|---|
| Registry | validate.sh TESTS, gh306 EXEMPT |
Independently count non-empty TESTS entries and compare with the operator's expected count; assert disjointness with EXEMPT. Record full target and newest registry-change SHAs. |
| Current Tier | utils/ci-route.sh registry mapping |
Record Small (tier 1/2), Medium, or Large-only for each suite. |
| Runtime | TESTS-RESULTS/.../validation.jsonl (event:"suite", duration_ms) |
Compute median duration across at least 3 green full-gate receipts. Record runner host architecture. |
| Failure Tally | Hosted full-registry logs and committed failing suite receipts | Record each suite's executed k of N per run source, window, and run/receipt IDs. Mark unexecuted suites UNKNOWN; never report "never failed". Deduplicate the same run across sources. |
| Failure Attribution | Label-discovered issues and fix:/hotfix: commits touching test/ |
Record each source's searched N and window, then cite the exact issue or commit for each suite match. These are not run denominators. |
| Failure Cause | PRs/issues linked to failures; #853 tracking list | Classify failure mechanisms using the Failure Taxonomy. |
| Labels & Hints | PASS: / ok - output in full run logs |
Extract sub-check counts and keyword hints for overlap and prose checks. |
| Source Code | Suite file, executed scripts, sourced helpers | Full source read required for every non-KEEP recommendation. |
Known Flake Candidates to Seed
When initializing an audit, seed known non-deterministic candidates identified in prior windows:
gh610-claude-subscription.sh(intermittent across PR runs)gh123-lock-progress-bound.sh(timing and progress bounds)registry-lock-concurrency.sh(intermittent contention / race)- Discover every active #853 and #812 member by exact filename from the saved issue bodies, including
agent-chorus-bridge.sh,gh492-idle-kill.sh,gh620-skills-army-mini-sync.sh,gh777-inventory-ratchet.sh,gh674-merge-cleanup-hosted-lookup.sh, andgh436-merge-cleanup.sh. Name every discovered member, its issue, and its class in the report; unmeasured classes stayUNKNOWN.
Detectors (D1–D9)
D1: Runtime Profiling & Heavy Suite Leverage
- Measure median execution time in seconds, global runtime rank, and share of the median full gate. Use at least 3 green full-registry receipts whose registry matches the target count; list each receipt path and full tested SHA. Compute one median full-gate runtime denominator from those receipts and use
med_s / denominatorfor everypct_gatefield in the TSV and summary. If fewer than 3 qualify, runtime and gate-share fields areUNKNOWN; never hardcode runtime figures. - Heavy Suite: Rank ≤ 10 or consuming ≥ 1.0% of the total gate runtime.
- High-Leverage Heavy Suites: Special priority is given to measured heavy suites, including
gh251-validate-pytest-skip.sh,gh436-merge-cleanup.sh,gh549-work-events.sh, andmarathon-drive.shwhen their current rank qualifies. Investigate whether nested executions can be bounded, faster siblings exist, or candidate status for NIGHTLY applies. - Recompute rankings from fresh receipts after any suite trimming PR.
- Role: Runtime breaks ties and nominates NIGHTLY candidates. Runtime alone never justifies turning off a test.
D2: Failure History & Defect Attribution
- Classify all observed failures across the audit window using the Failure Taxonomy.
- Same-commit / same-SHA divergence (passing on one run, failing on another) serves as primary evidence of non-determinism (
flake). - Bind each run to the commit it actually tested. A hosted
wave-reconcilerun qualifies an integrated snapshot named in its log (Qualifying N landing(s) in integrated snapshot <sha>); itsheadShais the workflow's ref (a PR run's is the PR head) and must not be used. When the tested commit cannot be established, count the result but mark its SHA attributionunknown; it is not flake evidence. - Report each source separately: hosted and committed local receipts get per-suite executed
k of Nand run IDs; labelled issues and fix commits get searched N, date window, exact suite mapping, and source IDs. Show counts of suites with at least one observed failure and suites with no failure data. No matching issue or commit is evidence of zero failures.
D3: Touch-Set Overlap & Duplicate Analysis
- For each suite, statically analyze:
- Binaries and scripts executed (
bash <x>,python3 utils/py/<y>,bin/tick <verb>,node <z>). - Library files sourced (
test/_setup.sh,test/lib/*.sh). - Repository paths read or grepped.
- Repository paths written or modified.
- Binaries and scripts executed (
- Two suites overlap when their invoked target entry points overlap.
- Text similarity (e.g. 5-token shingle Jaccard ≥ 0.6) or label similarity is a secondary tiebreaker only after touch-sets overlap. Filename similarity (e.g.
gh155-phase3vsgh155-phase5) is not overlap if distinct subsystems are invoked. - When two suites share identical target entry points and duplicate contract assertions, nominate the redundant suite for
MERGE. - Generate duplicate candidates from saved, computed touch sets. Do not seed a hand-typed list of duplicate suite pairs.
D4: Sibling Coverage
- A suite is covered when a named sibling suite tests the same target entry points with an equal or superset set of behavioral assertions.
- Example:
synthetic/synthetic-pi-model-unset.shis covered bytest/pi-turn.sh(which asserts exit code 5, clean working tree, no commit, and uninvoked model binary).
D5: Prose & Non-Core Text Check
- Count assertions that inspect documentation and markdown files (
*.md,SKILL.md,docs/*,README,ROUTER.md,AGENTS.md) versus assertions that execute codebase scripts/binaries and verify behavioral contracts. - Executing code counts only when an assertion checks that code's behaviour. Running a script and then grepping a doc does not count.
- Exclude generated fixture files or runtime-emitted docs created inside a test sandbox (e.g. asserting
ESCALATION.mdwas created by an agent turn is behavioral). - Prose Ratio:
(doc-grep assertions) / (total assertions).- Ratio ≥ 0.6 or Pure Skill/Doc Text (GH-831): Candidate for
TURN-OFFif the suite merely asserts wording, markdown structure, or non-core skill text rather than runtime harness behavior. A suite at ≥ 0.6 that still has behavioral assertions no sibling covers (D4) isSPLIT, notTURN-OFF; name those assertions. - Ratio 0.2–0.6 (Mixed): Candidate for
SPLIT(keep behavioral checks; drop/move pure wording assertions). - Ratio < 0.2 (Behavioral): Retain on gate.
- Ratio ≥ 0.6 or Pure Skill/Doc Text (GH-831): Candidate for
D6: Junk Pattern Detection
Inspect individual assertions for anti-patterns:
- Exact String Fragility: Asserting exact prose strings that break on innocuous copy-edits but pass on broken logic.
- Duplicate Contract Calls: Repeating identical CLI invocations and flag checks across multiple independent suites without novel assertions.
- Stub Implementing Assertion: A test stub (e.g. mock
ghorgit) hardcodes the exact string the test subsequently asserts. - Private Call-Shape Checks: Asserting internal Python function names or private helper argument lists instead of public CLI behavior.
- Vacuous Negative Controls: Assertions that mutate a local copy or test fixture and grep the copy without exercising the actual code path (e.g.
gh798controls 8a/8b).
D7: Can-It-Fail Verification
- Before asserting that a test suite cannot fail, trace all sourced helpers and error traps.
- A suite sourcing
_setup.shthat callsfail()on error will exit 1 on failure even if the file concludes withexit 0.
D8: Fixed-at-HEAD Verification
- For every historical flake or failure, check whether a remediating commit already landed on the active branch (e.g.
gh649resolved bypwd -Pcanonical path resolution). - If fixed at HEAD, classify as
fixed-flake; cite the fixing commit and apply the ordinary KEEP evidence rule. Do not quarantine solely for a fixed historical failure.
D9: Four-Question Gate (OpenClaw)
For every evaluated suite:
- What contract or behavior does it protect? (CLI verb, concurrency invariant, data integrity, routing).
- What credible regression makes it fail? (State the failure scenario).
- Why doesn't existing coverage catch it? (Identify the unique boundary).
- Does it require a test-only seam in production code? (Reject artificial test-only hooks).
- Verdict Effect: A suite with no clear answer to Q1 or Q2 is a candidate for
TURN-OFForMERGE(it guards no identified behavior or failure mode). A suite with answers to Q1 and Q2 but no answer to Q3 is a candidate forMERGEinto its covering sibling.
Failure Taxonomy
| Class | Evidence Required | Verdict Effect |
|---|---|---|
regression-caught |
Failure directly caught a real bug, confirmed by a subsequent product code fix (e.g. gh436 in #812 caught by 0ae3452a/#794; gh496 caught race in #813/#818). |
Protected. Stays on the PR gate regardless of runtime. |
coupling |
Failure caused by unrelated inventory changes, doc rewordings, or count shifts. | If coupling is to prose/wording, run D5: at ratio ≥ 0.6 candidate for TURN-OFF. For fragile D6 assertions on core code, KEEP-FIX requires a cited assigned fix; otherwise INVESTIGATE. |
flake |
Same commit passed in another run; or error log cites timing bound/port race. | An ongoing unfixed flake blocking CI is QUARANTINE without an assigned fix, or KEEP-FIX with a cited assigned open fix. Historical divergence without recent red or an established fix is INVESTIGATE. |
host |
Failure caused by runner environment (macOS vs Linux paths, /tmp contention, host Python). |
KEEP-FIX with a cited assigned open fix; otherwise INVESTIGATE or QUARANTINE under the flake rule. The #853 umbrella alone is attribution, not assignment. |
fixed-flake |
Cause of failure was resolved by a landed commit at HEAD (D8). | Eligible for KEEP if the ordinary evidence bar is met. Cite fixing commit; do not quarantine solely for the fixed failure. |
unattributed |
Unexplained timeout or missing summary log. | INVESTIGATE if this is the only failure class; include the failure in the radar share. |
Verdicts, Decision Rules & Guardrails
| Verdict | Definition & Rule | Proposed Action (Requires Approval) |
|---|---|---|
| KEEP | Meets retention bar, catches regressions, or uniquely guards a core contract with passing behavioral receipts. | Retain in validate.sh TESTS. |
| KEEP-FIX | Retained suite with an active, assigned fix issue or PR open in the current window (requires its citation; an umbrella alone does not assign the fix). | Retain in TESTS; link the assigned fix. |
| NIGHTLY (candidate) | Heavy suite with a faster PR-time sibling covering its full target set (see NIGHTLY Rule). | Listed as candidate for future scheduled runs (#859). Remains in TESTS. |
| QUARANTINE | Flaky suite blocking CI with no fix landed at HEAD and no active fix lane assigned. | Move from TESTS to test/gh306-registry-bidirectional.sh EXEMPT with quarantine: <issue> reason. Keep file on disk. If multiple suites share root cause, recommend whack-a-mole. |
| SPLIT | Mixed suite (D5 prose ratio 0.2–0.6, or ≥ 0.6 with uncovered behavioral assertions) combining behavioral checks with prose greps. | Propose splitting: retain executable contract checks; drop or move wording greps. |
| MERGE | Redundant suite whose unique assertions are folded into a named keeper suite. | Propose folding assertions into keeper after red control; then turn off. |
| TURN-OFF | Obsolete suite (target removed), pure prose/skill-text suite (GH-831), or fully covered sibling with no unique assertions. | Remove from TESTS; add top-level suites to gh306 EXEMPT with reason, or remove subdirectory suites from their own registry pin (name that pin in restore). Keep the file on disk. |
| INVESTIGATE / UNKNOWN | Insufficient telemetry or unmeasured metrics. | Retain in TESTS pending further telemetry; never default to KEEP. |
Tier-Based Flake Rule (#802/#853)
A flaky suite in an active tier (the Small or Medium membership mapped by utils/ci-route.sh, including SUBSYSTEM_TESTS_small) receives KEEP-FIX only with a cited, assigned open fix issue or PR. Without one, an ongoing flake blocking CI is QUARANTINE under the ordinary rule. Historical same-SHA divergence with no recent red and no established fix is INVESTIGATE, not automatic KEEP or QUARANTINE. Outside those tiers, apply the same evidence rules rather than turning a suite off on membership alone.
The Retention Bar
The retention bar applies only after D3/D4 show that no other suite on the PR gate covers the same entry point. A covered suite is a MERGE or TURN-OFF candidate whatever category it falls in. When no covering sibling exists, a suite is eligible for KEEP under rule 7 if it independently guards:
- Package installation, bootstrapping, or migration logic.
- Concurrency, file locks, or driver lock invariants.
- Security, credential containment, or network egress boundaries.
- CLI contracts (exit codes, standard flags, stdout/stderr protocols).
- Data integrity, database schemas, or ledger transactions (
releases.db,tick). - Gate routing or CI test selection contracts (
ci-route.sh,gh308). - Source inspection when it is the only independent guard of a user-facing configuration key or path per D4.
Mantra: Static or slow is not a reason to delete; but redundant, dead, or unmeasured is never a reason to keep.
Sibling Deduplication Precedence
Sibling coverage and deduplication take precedence over the retention bar. Even if a suite tests a retention-bar contract, if another faster suite already guards that exact contract with equal or superset assertions (D4), the redundant duplicate is nominated for MERGE or TURN-OFF.
The NIGHTLY Rule
A heavy suite $H$ is a candidate for NIGHTLY only when all four conditions hold:
- $H$ is heavy (D1: rank ≤ 10 or ≥ 1.0% gate time) and has no
regression-caughtfailures in the audit window or issue history. - A named sibling suite $S$ remains on the PR gate, and $S$'s invoked target scripts/binaries are a superset of $H$'s invoked targets (D3). Static reads, greps, and written files do not count toward superset target invocations; only invoked target scripts/binaries count.
- $S$ is significantly faster (median duration of $S \le 20%$ of $H$) and has no open flakes.
- If $H$ guards a retention-bar contract, $S$ must guard that same contract.
PR Gate Yield Note: Candidates remain in TESTS today (#859 is pending), so a NIGHTLY verdict takes nothing off the PR gate until scheduled runner machinery lands. Reports must explicitly track the count of suites proposed to leave the PR gate separately.
Fallback: If no sibling qualifies, the suite stays on the PR gate; apply the ordinary KEEP evidence rule. Insufficient evidence yields INVESTIGATE, never automatic KEEP.
Core Guardrails
0 of N runsis never a reason to turn off or demote a test.- A
regression-caughtsuite is never proposed for NIGHTLY, QUARANTINE, or TURN-OFF. (e.g.gh436-merge-cleanup.shcaught the #812 regression; it remains on the PR gate). - An active #853 member is never turned off. It can be
KEEP-FIX,QUARANTINE, orINVESTIGATEunder the evidence rules. - Subdirectory Suites: Note that
test/gh306-registry-bidirectional.shEXEMPTonly governs top-leveltest/*.shfiles. Subdirectory suites (e.g.synthetic/*) that are turned off come out ofTESTSand their own registry pin (test/gh141-synthetic-registry.sh), not into gh306EXEMPT. - Check Pinned Suites: Before proposing
TURN-OFF, verify whether other suites assert the entry inTESTS(e.g.gh35-test-tiers.sh,gh365-driver-lane-registry.sh,gh141-synthetic-registry.sh,ci-workflow.sh,gh379-canary-uses-validate.sh, or release manifest suites). - No Unbacked Merges: If a proposed survivor for
MERGEorSPLITdoes not exist, mark the row asparked: no survivorrather than creating new suites under the freeze. - High Confidence Required: QUARANTINE, TURN-OFF, MERGE, SPLIT, and NIGHTLY require full source inspection and
HIGHorMEDconfidence.INVESTIGATEhas no confidence floor. - Observation Window: Approved turn-offs are moved to
EXEMPT(or removed from registry pin) for an observation window before anyone considers deleting a file (at least 14 days).
8-Point Acceptance Test for Audit Runs
Every completed audit run must pass this 8-point acceptance check before findings or report issues are accepted:
- Calibration Passed: Calibration against the 8 suites turned off by #831 passed (at least 7 of 8 scored
TURN-OFF/MERGE, andgh798scored wording-only/TURN-OFF). - Target Branch Pinned: Audit was executed strictly on
developmentor active stabilization staging branch (notmain). - Honest Metrics: All unmeasured suites, missing durations, or unobserved failure signals are recorded as
UNKNOWNorINVESTIGATE, with zero keep-by-default fallbacks. - Multi-Source Failures: Record the searched N and window for hosted runs, labelled issues, fix commits, and committed failing receipts; keep run denominators separate from attribution counts.
- Seeded Flakes Investigated:
gh610,gh123, andregistry-lock-concurrencyshow historical same-SHA divergence but no recent red since 2026-09-25. Classify themINVESTIGATEuntil a fix or continuing failure is established; do not inferKEEPfrom subsequent green runs. An ongoing unfixed flake blocking CI isQUARANTINE, orKEEP-FIXwith a cited assigned open fix. - Heavy Suites Profiled: High-leverage heavy suites (including
gh251,gh436,gh549,marathon-drive) are analyzed for nested runners and faster siblings. - Redundant Suites Merged: Duplicate suites with overlapping touch sets receive
MERGErecommendations naming a surviving keeper. - Sibling Skills Triggered: Sibling skill recommendations (
radarfor trunk-red clusters,whack-a-molefor shared root causes) fire accurately based on objective criteria.
Output Format
The audit produces a machine-readable tab-separated values (TSV) dataset and a markdown summary, saved to the evidence directory:
TESTS-RESULTS/<date>+GH-<issue>/ci-suite-audit.tsv
TSV Columns
suite tier_now med_s rank pct_gate fails (k of N) fail_class issues touch_set overlap_with covered_by prose_ratio junk_flags pins gate_q1_q3 verdict proposed_action evidence confidence source_read restore
evidence: Citesfile:line, job run ID, or GitHub issue/PR number.confidence:HIGHrequires source read and measured failure data; green receipts alone give at mostMED.LOWmeans material fields remain unmeasured;UNKNOWNmeans confidence cannot be assessed.restore: Shell command to restore or un-exempt the suite if needed.- Copy full 40-character SHAs from git or API output for commit links and report metadata; never reconstruct them. A short SHA is acceptable only in the issue title and dedupe marker.
Summary Markdown Layout
The summary report includes:
- Audit Metadata: Run date, full target and newest registry-change SHAs, ref/branch, expected and observed registry counts, analysis mode (in-checkout vs connector), each source's N/window/IDs, qualifying full-receipt paths and tested SHAs, and the one computed full-gate runtime denominator in seconds (or
UNKNOWN). Count suites with at least one observed failure and suites with no failure data. - Pre-Flight Calibration Results Table: Results for the 8 #831 calibration suites (
gh578,gh778,gh798,gh779,gh781,gh615,gh616,gh617) with prose ratios, verdicts, and pass assertion (≥7/8 turn-offs,gh798wording-only classified). - Detector Coverage Table (D1–D9): Evaluated counts vs the measured registry total for every detector, or count marked UNKNOWN.
- Verdict Breakdown Table: Tally of suites per verdict class (including INVESTIGATE and UNKNOWN) and count/percentage of suites proposed to leave the PR gate (QUARANTINE + TURN-OFF + MERGE).
- D3 Shared Entry-Point Clusters Table: Entry points invoked by $\ge 3$ suites, listing overlapping suites, overlap details, and nominated MERGE candidate or retention reason.
- Heavy Suites & NIGHTLY Evaluation Table: Top heavy suites by runtime, evaluating the 4 NIGHTLY conditions (1: heavy & no regression-caught, 2: superset sibling on PR gate, 3: sibling duration $\le 20%$, 4: sibling guards retention contract), nearest sibling, sibling median duration, and candidate verdict.
- SPLIT Ratio Band Verification Table: Verification that every SPLIT candidate's prose ratio falls within the 0.20–0.60 range, or is ≥ 0.60 with its uncovered behavioral assertions named (D5).
- Actionable Proposals Table: Itemized list of all non-KEEP candidates with suite name, tier, median duration, failure history, proposed action, evidence/citations, and mandatory
confidencecolumn. - Diagnostic & Remediation Reminders: Measured sibling skill trigger formulas (
radarshare $(unattributed + coupling)/red_runs \ge 25%$;whack-a-molecluster $\ge 3$ parallel-load/host races in #853).
Report Issue & Per-turn Comments
Report Issue Creation & Deduplication
Title Format:
ci-suite-audit: <audit date> report @ <registry SHA>(e.g.ci-suite-audit: 2026-10-08 report @ <short staging SHA>).Deduplication Marker: The report issue body begins with an HTML comment marker:
<!-- ci-suite-audit:<registry-sha>:<audit-date> -->Dedupe First: Before opening a new issue, look for an existing report in this order:
- The local record. When the skill creates a report issue, it writes the number and marker to
TESTS-RESULTS/<date>+GH-<issue>/report-issue.txt. A re-run first reads that file: it is the only check that is consistent immediately after creation. - A direct listing, matched locally on the marker:
gh issue list --state open --label ci --limit 200 --json number,title,body. Do not use--search.
Both GitHub reads are eventually consistent. The #862 practice runs opened duplicates when re-running 3 s after creation: first through search (#871/#872), then through listing (#873/#874). The local record closes that window, and a later session's listing covers re-runs minutes or days apart.
- Match found (same SHA or audit date): Update the existing issue body and post a comment with the delta. Never open a duplicate issue.
- Multiple open matches: Stop and request operator clarification.
- Closed match: Open a new issue referencing the previous closed report.
- The local record. When the skill creates a report issue, it writes the number and marker to
Labels: Apply
ciandstability, plusci-suite-auditif that label exists in the repository. Never apply theradarlabel.GitHub Size Limit (65,536 chars): If the full report exceeds GitHub's issue body limit, place the summary, verdict counts, non-KEEP rows, and reminders in the issue body. Post the full per-suite TSV/table across sequentially numbered issue comments (
table part k of n).Posting Fallback: Attempt issue creation via GitHub CLI (
gh issue create). If CLI is unavailable or unauthorized, create via GitHub connector tools. If running offline or without GitHub write permissions, write the formatted report to the evidence directory (TESTS-RESULTS/<date>+GH-<issue>/ISSUE.md) and alert the operator.
Per-Turn Comments
- In interactive audit sessions, after each turn where a decision is reached, post one structured comment detailing:
- Verdicts accepted, rejected, or overridden by the operator.
- Follow-up issues filed (only with explicit operator authorization).
- Open questions resolved.
- Post Only on Changes: Turns without decisions or changes generate no comment ("no change" comments are prohibited).
- Conclude each comment with the remaining open items. When all items are resolved, propose closing the issue. Close only upon operator confirmation.
Safety & Redaction
- Redact all access tokens, API keys, secret variables, and environment values.
- Strip local machine paths, replacing them with repository-relative paths (
skills/...,test/...).
Related Skills
The skill recommends sibling skills for broader coordination; it never invokes them autonomously.
- radar: The diagnostic sibling. Recommends running radar when CI failures reflect wider SDLC or process drift rather than isolated test defects:
- Trigger: $\ge 25%$ of red runs in the window are
unattributedorcoupling, or a trunk-red cluster appears (e.g.gh436andgh674red together across multiple runs, as in #812).
- Trigger: $\ge 25%$ of red runs in the window are
- whack-a-mole: The remediation sibling. Recommends running whack-a-mole when recurring test failures share a single root cause:
- Trigger: $\ge 3$ suites fail due to the same underlying mechanism (e.g. shared runner port race or
/tmpcollision). If an existing umbrella covers the pattern (such as #853 for test isolation), cross-reference that issue instead of opening a new one.
- Trigger: $\ge 3$ suites fail due to the same underlying mechanism (e.g. shared runner port race or
- ci-optimize: Pipeline architecture cross-link (unit: entire CI/CD pipeline;
ci-suite-auditunit: one test suite).
Sibling Reminder Block
Every audit report and issue concludes with a status block:
## Sibling Skill Recommendations
- **radar:** [trigger met: <evidence> | not triggered]
- **whack-a-mole:** [trigger met: <evidence (points to #853 if covered)> | not triggered]
Proposed Authoring Gate (Proposal for Operators)
For operator decision. Not active by default. Adapted from OpenClaw (MIT). Under the repository test freeze, no new test files may be added. When modifying or extending existing suites, apply this 4-question gate to every new assertion:
- What behavior or contract does this assertion protect? (Name the CLI verb, schema constraint, or isolation boundary).
- What credible regression makes this assertion fail? (Witness the failure under mutation or record a red control).
- Why don't existing assertions catch it? (Check existing suites with
rgbefore adding duplicate assertions). - Does it rely on artificial test-only hooks in production code? (Avoid adding flags or exports solely for testing).
Sources
Upstream credits, MIT license texts, and copyright notices are documented in NOTICE.