ci-fleet-engineering
Use when adding, editing, or debugging ANY GitHub Actions workflow that mutates shared data, publishes an artifact, reports, or runs on a cron, and when diagnosing CI failures, cancelled runs, lock co
SKILL.md.claude/skills/ci-fleet-engineering/SKILL.md
GitHubCI Fleet Engineering
The full doctrine with incident write-ups: docs/CI-FLEET.md. The one-sentence
version: a green run must have done something, a red run must mean something,
and no run may destroy work — its own or another's.
When writing or editing a workflow
Mutates shared data? Use the compute/apply split —
docs/templates/split-writer-workflow-template.ymlis the copyable shape. The lock (one shared concurrency group,cancel-in-progress: false) is held ONLY by the short apply job. GitHub keeps ONE pending run/job per group and DESTROYS the older pending on a newer arrival — short holds are survival. Runpython3 tools/check_workflow_gates.pyafter touching any apply job.Long compute? The tool takes
--max-minutes, measured from PROCESS START, able to fire INSIDE one item, exiting 0 so publish still runs. Steptimeout-minutesis a continue-on-error BACKSTOP above it, judged by a verdict step: fired + output grew →::warning::; fired + nothing → fail.Restores: only a NON-EXISTENT release is a first run. Never
|| truea restore. Snapshot-and-check any shared SQLite withtools/sqlite_publish_guard.py.--clobber= delete-then-upload; never clobber-upload a file that can be rejected (2 GB cap → both assets vanish).Uploads wrap in
tools/gh_retry.sh(must-persist, fails loudly); dispatches intools/gh_dispatch.sh(fire-and-forget, exits 0).Never
cmd || true: capture the exit code and::warning::it.tool | tee logneedsset -o pipefail(or${PIPESTATUS[0]}) or a crash goes green.Inputs as env, never pasted.
${{ inputs.x }}insiderun:is text substitution: a quote closes the string, free text is injection. Map toenv:and read"$X". Optional flags areif [ -n "$X" ]; then ARGS+=(...); fi, never[ -n "$X" ] && ARGS+=(...): that returns 1 on empty input, and as a step's (or function's) last line it fails the step underbash -e.Every step that runs
ghhasGH_TOKENin scope (step, job or workflow env). A runner has no ambient credential; a publish step without it fails and skips everything after it.Reporters never fail on their findings. A dashboard, auditor or status check exits 0 whenever it read what it came to read, and puts findings in the step summary, a
::warning::, or an issue. A step in aREPORTERSworkflow that can exit non-zero carries# reporter-may-fail: <why>naming its OWN mechanics.Schedule for the lag. Crons run four to five hours late; when work must follow a release, trigger it with
workflow_run. A bot commit marked[skip ci]fires nopushworkflows at all, including deploys; chain those withworkflow_runand usepaths-ignoreinstead.A push retry rebases and resolves, never
pull --rebase || true. A conflict leaves the rebase half-done and every retry fails. Generated file: this run's copy wins. Hand-edited file: main's wins, or stop with an error. Abort a rebase that cannot continue (docs/CI-FLEET.md§3).Failures are remembered on the item, not the runner. A failure list on the runner dies with it, and the same poison items lead every run. Cap a download's total time, not only each read. Adopt an artifact already hosted before regenerating it.
Ingest hands off to the gate, and the gate publishes. Never let discovery publish new items before the audits that judge them have run. Give each source of a shared queue a fair share.
Standing guardians (enable once per repo)
.github/workflows/retry-infra-failures.yml+tools/retry_infra_failures.py— re-runs never-started runs and queue-displaced runs/apply-jobs. The gate is "zero steps ran," which is the no-side-effect proof; verify changes withDRY_RUN=1against real history..github/workflows/workflow-health.yml+tools/audit_workflow_health.py: daily judge of what each workflow's last SCHEDULED run PRODUCED (BROKEN, KILLED, FAILED, DROPPED, STALE, SILENT, DRAINED), each against its OWN cadence.NO_YIELD_LINEworkflows skip only the yield check; their failures still count. Self-cancelling workflows (cancel-in-progress: true) are not KILLED. The auditor never fails. Urgent findings (BROKEN, KILLED: the ones nothing else alerts) go to ONE self-closing issue viatools/report_workflow_health.py, opened or updated in place, closed when the fleet is clean.tools/check_workflow_gates.py: static checks, exit 1 on violation. Apply jobs fully gated; everyghstep hasGH_TOKEN; reporter workflows (REPORTERS) never exit non-zero without# reporter-may-fail:. Tests:tools/test_workflow_health.py,tools/test_workflow_health_gate.py.
When diagnosing
- Run duration includes QUEUE time — read job-level startedAt/completedAt before diagnosing slowness (a "673-minute run" was a 1-minute job).
- A cancelled run with zero steps was displaced, not stopped; one with steps was stopped by a human or timeout. Never blur these.
- A check that reports "nothing to do" suspiciously fast is a finding, not a completion — verify what was OFFERED to it (markers, filters) before believing the count. Timestamps + TTLs on every "verified" marker; record the SOURCE on every "already tried" marker; persist evidence, not verdicts.