ci-debug
Debug failing CI pipelines, pre-push gates, hosted GitHub Actions workflows, and reproduce test failures in safe isolation. Applies debug-mantra ground-truth discipline, recon seam/blast radius mappin
SKILL.mdskills/2-daily/ci-debug/SKILL.md
GitHubci-debug — Governed CI/CD Diagnostic & Resolution Ladder
ci-debug is an end-to-end debugging and resolution ladder for failing CI/CD pipelines, local pre-push gates, and test suites. It coordinates three core disciplines:
/debug-mantra: Ground-truth extraction from raw CI artifacts, safe deterministic reproduction in a disposable full clone (GH-564), fail-path tracing, and disproof-first hypothesis falsification./recon: Seam, contract, and blast-radius mapping across callers, scripts, and harnesses before modifying code./ponytail(Sound Root-Cause, Never a Bandage): The least-mechanism, foundationally sound long-term resolution (stdlib/native first, single authority/resolver, zero duplicate subsystems, deleting dead code). Never masks symptoms with|| true, loose regexes, silenced assertions, or unmeasuredsleeps.
Related skill: sanity-check the claimed blocker
Recommend sanity-check when a CI failure's requirement value, user consequence, or need to block the current goal is unclear, or investigation stops producing evidence. Pass the actual failure and existing findings; use its disposition to choose repair now, deferral, or a concrete retirement proposal before sinking more effort into the machinery.
Reuse a current assessment for the same scope and evidence. A Fix now handoff
returns here at the relevant phase, without another sanity-check cycle or repeated
permission for an already-authorized repair. Deferral does not waive a required
gate, and this pointer does not authorize retirement or weaken safety controls.
In a flat installed collection, resolve sanity-check by skill name.
The 4-Phase Resolution Ladder
[1. Ground Truth & Repro (/debug-mantra)] ──► Raw logs/receipts, disposable clone (GH-564), isolate knobs, disproof-first
│
▼
[2. Seam & Blast Radius Recon (/recon)] ──► Identify failure archetype, map callers, contracts, build manifests
│
▼
[3. Sound Minimal Resolution (/ponytail)] ──► Root-cause fix (least mechanism, zero bandages), leave 1 runnable check
│
▼
[4. Governed Verification & Receipts] ──► Preflight (validate.sh), qualifying gate (ci-local.sh), hosted CI watch
Phase 1: Ground Truth & Isolation (/debug-mantra)
Establish primitive ground truth from raw artifacts before hypothesizing or modifying code.
1. Extract Raw Failure Evidence
Inspect the raw CI run or gate output directly. Never theorize from a summary:
# List recent hosted runs for the workflow or branch
gh run list --workflow ci.yml --limit 5
# Inspect exact failed step logs and error traces
gh run view <RUN_ID> --log-failed
# Or view local gate failure logs
cat .tick/gate-last.log 2>/dev/null || cat .git/pre-push.log 2>/dev/null
2. Provision Safe Full Clone Isolation (GH-564)
Never run test reproductions or mutation suites in the primary checkout or a linked worktree. Suites mutate .git/config, remotes, and ref locks:
# Provision a standalone disposable full clone outside the primary checkout.
# Compute the destination ONCE (two $(date +%s) calls can straddle a second) and stop on any failure,
# so a failed clone or cd can never leave you in the primary checkout with a suite about to run.
CLONE="/tmp/XYZ-forge-ci-debug-$$-$(date +%s)"
git clone . "$CLONE" && cd "$CLONE" && bash githooks/install.sh \
|| { echo "ci-debug: clone setup failed; stopping before any suite runs" >&2; exit 1; }
3. Reproduce Deterministically
Reproduce the exact command executed by CI:
- Local Gate Preflight:
./validate.sh(or targeted subsystem suite:bash test/ci-route.sh --subsystem <name>). - Hosted CI Command: Check
.github/workflows/ci.ymlfor exact step flags and environment variables. - Isolate Concurrency Knobs: If parallel execution (
./validate.sh, default cores/2) fails intermittently, run the failing suite in serial isolation (bash test/<suite>.sh). If it passes in serial, record the issue as resource contention / timing race, not a logic error.
4. Trace the Fail Path & Falsify Hypotheses
- Trace from the failing assertion back to the input state.
- Generate 2–3 ranked hypotheses. Identify the cleanest disproof for each.
- Run the disproof first: if a hypothesis fails to explain the failure path end-to-end, discard it immediately.
- Keep a running session breadcrumb ledger of ruled-out causes.
Phase 2: Seam & Blast Radius Recon (/recon)
Map the system boundaries before drafting a fix to ensure no unread code enters the blast radius.
1. Classify the Failure Archetype
| Archetype | Common Signatures in XYZ Forge | Diagnostic Seam |
|---|---|---|
| A. Static Guard & Lint | Frozen Bash twin guard (GH-308) failed, pipe-grep guard (GH-139) failed, mktemp trap guard (GH-177) triggered, PDDA frontmatter missing updated: |
test/gh308-frozen-twin-guard.sh, security-scan.sh, utils/pdda/pdda.sh run |
| B. Concurrency & Contention | SQLite database is locked, file lock timeout, generation trio mismatch (gh32-releases-app #558, gh53 #541) |
Lock paths in utils/py/, parallel worker count in validate.sh |
| C. Platform & Runner Delta | BSD vs GNU sed/awk/stat/date differences, macOS passing but canary-ubuntu failing on ubuntu-latest (#588/#590) |
Missing flags, non-portable bash syntax, path casing |
| D. Gate Provenance & Receipt Gap | wave-reconcile exit 6, --gate missing TESTS-RESULTS/ receipt (#546, #565, #592) |
check_provenance_receipts(), ci-local.sh execution |
| E. Reconciler / Ledger Drift | Legacy issue drift (GH-52 #584), missing DB table/column, roadmap sync error |
releases_app.py, wave_reconcile.py |
2. Map Seams and Blast Radius
- Trace callers, consumer scripts, and exported environment variables touching the failing seam.
- Check whether the seam has a single authoritative implementation (e.g. Python in
utils/py/vs legacy.shtwin). - Document unknowns: anything unverified by a direct source read is an unknown, not an assumption.
Phase 3: Sound Minimal Resolution (/ponytail — Structural Root-Cause)
Apply Ponytail to engineer the least-mechanism, foundationally sound resolution.
[!IMPORTANT] Ponytail is NOT a Bandage. Ponytail minimizes the mechanism required to solve the true root cause. It is never permission to mask symptoms, silence guards, or introduce fragile workarounds.
The Resolution Ladder:
- Does the failing code/machinery need to exist at all? Delete dead code, retired views, or obsolete wrappers instead of patching them.
- Standard library / Native platform feature covers it? Use built-in Python stdlib (
pathlib,sqlite3,subprocess) or POSIX shell constructs over custom helper layers. - Single Canonical Authority: Centralize lock resolution, path lookup, or DB operations into one canonical utility rather than duplicating logic across scripts.
- Frozen Twin Rule (GH-308): Behavior fixes belong in the authoritative Python twin under
utils/py/, NOT the frozen.shfallback. If an emergency edit to a.shtwin is truly warranted, add the mandatory commit trailer:Frozen-twin-exception: <path> — <reason> - No Static Comment Traps (SOP §3b): Static security guards read comments. Never write banned syntax (e.g. piped grep or credentials) inside explanatory comments.
- Leave One Runnable Red Control: Every non-trivial fix must leave behind a runnable test/assertion that proves the defect is resolved and fails when mutated. In XYZ-forge (
AGENTS.mdNo new tests, GH-831) that check lives in an existing suite or is recorded as a manual check underTESTS-RESULTS/, never a new suite.
Operational Containment Protocol (Secrets, Leakage & Incident Response)
If a defect, failed workflow, test artifact, or prompt transcript exposes API keys, service tokens, or private credentials:
- Priority 1 — Provider Revocation & Rotation First:
- Immediately revoke or rotate the compromised credential in the identity/cloud provider console or CLI before attempting git history manipulation.
- Priority 2 — Blast Radius Audit in Access Logs:
- Query provider audit and access logs for the compromised key's token ID during the exposure window to identify unauthorized accesses.
- Priority 3 — Preserve Sanitized Evidence:
- Never copy raw credentials, secret tokens, or decrypted payloads into issues, PR descriptions, or prompt logs. Replace all secret instances with deterministic redactions (
[REDACTED_API_KEY]).
- Never copy raw credentials, secret tokens, or decrypted payloads into issues, PR descriptions, or prompt logs. Replace all secret instances with deterministic redactions (
- Priority 4 — Explicitly Authorized History Scrubbing:
- History rewriting tools (
git-filter-repo/ BFG) require explicit operator confirmation. - Strictly comply with
WORKTREE-SAFETY.md: verify that no linked worktrees depend on the rewritten refs, take a full backup of.gitbeforehand, and coordinate ref updates across active clones.
- History rewriting tools (
Phase 4: Governed Verification & Receipts
- Preflight in Disposable Clone:
Ensure 100% of test suites pass cleanly../validate.sh - Qualifying Gate Run (Receipt Generation):
Verify that a valid, machine-readable JSONL receipt is written tobash ci-local.shTESTS-RESULTS/. - Commit & Push Boundary:
Push the task branch to GitHub (
githooks/pre-pushexecutes automatically). - Watch Hosted CI:
Confirm bothgh run watch $(gh run list --workflow ci.yml --limit 1 --json databaseId --jq '.[0].databaseId')smoke-gate-macosandcanary-ubuntujobs turn green.
Anti-Patterns: What NEVER to Do
- ❌ Blind CI Reruns: Running
gh run rerunrepeatedly hoping a flake disappears without a controlled baseline comparison. - ❌ Masking Failures: Adding
|| true, loosegrep -q, or vacuous assertions that pass on empty strings (AGENTS.md §6). - ❌ Running Mutation Suites in Primary Tree / Worktrees: Running
validate.shin the primary clone or linked worktrees corrupts.gitshared state (GH-564). - ❌ Symptom Bandages via Ponytail: Injecting arbitrary
sleeps or skipping safety guards under the guise of "simplicity." - ❌ Gate Bypasses as Merge Readiness: Using
git push --no-verifyorXYZ_SKIP_PREPUSH=1for anything other than operator-requested WIP drafts. - ❌ Editing Frozen Twins Silently: Modifying frozen
.shfiles without updating the authoritative Python implementation and providing explicit trailers (GH-308/GH-321).