CI Doctor

.gitlab-ci.yml is the most-churned file in this repo — CI firefighting is the largest recurring time cost. This skill turns the historical failure classes into a triage procedure so a red pipeline costs minutes, not an afternoon of push-and-watch.

The root .gitlab-ci.yml holds default, variables, workflow, stages, and an include: list; the jobs live in ci/<group>.yml. A YAML anchor (&x/*x) does not resolve across included files, so a template used from another file goes through extends: or !reference. Text checks read the whole pipeline through scripts/lib/gitlab-ci.mjs.

Project: futo-notes/futo-notes on gitlab.futo.org. The curl calls below use $GITLAB_TOKEN, which always works. glab is usually also present, but whether it can authenticate is per-machine and per-shell — see the auth gotcha in docs/agents/issue-tracker.md before blaming a read.

Step 1 — Get the facts (never guess from the commit diff alone)

API="https://gitlab.futo.org/api/v4/projects/futo-notes%2Ffuto-notes"
H='--header PRIVATE-TOKEN:'"$GITLAB_TOKEN"

# Latest pipelines for a ref (branch or tag)
curl -s $H "$API/pipelines?ref=main&per_page=5" | jq '.[] | {id, status, ref, sha: .sha[0:8], web_url}'

# Jobs in a pipeline — find the failures
curl -s $H "$API/pipelines/<PIPELINE_ID>/jobs?per_page=100" \
  | jq '.[] | select(.status=="failed" or .status=="canceled") | {id, name, stage, status, allow_failure}'

# The job log (the tail is where the truth is)
curl -s $H "$API/jobs/<JOB_ID>/trace" | tail -80

glab ci status / glab ci view work for quick looks; use the raw API when you need job traces or to script a watch loop.

For pipelines on the current branch, glab ci list -b <branch> may parse the branch as a date and fail. Use the encoded project path through glab api:

BRANCH=$(git branch --show-current)
glab api "projects/futo-notes%2Ffuto-notes/pipelines?ref=$BRANCH"

Read the log tail FIRST, then the failing job's script: in its ci/*.yml file, then the diff of the commit that broke it (git log --oneline -5 -- .gitlab-ci.yml ci/).

Step 2 — Classify against the known failure taxonomy

Match the symptom before inventing a novel diagnosis. These classes account for most historical red pipelines here:

Class	Symptom	Root cause pattern
First-contact tag job	A publish/release job fails on its first real run at tag time	Job only executes on tags, so it was never exercised: secrets not propagated (especially into nested VMs like Cirrus), cold caches hitting timeouts, wrong artifact lookup paths
CWD/path assumption	A step "succeeds" but its effect didn't happen; later step fails mysteriously	GitLab preserves cd across script lines; relative path resolved elsewhere; -f/`
Silent green	"Job succeeded" but nothing was published/uploaded	An error branch special-cased to exit 0
Missing release-gate wiring	A test failed but the release published anyway	New test job absent from release:gate.needs (deliberate exception: test:audit — see rule 3)
Artifact-path drift	No files to upload / downstream job can't find inputs	Workspace layout changed; artifacts:paths/lookup globs didn't
Cache death spiral	Retries keep timing out at the same wall	Caches upload only on_success, so every retry starts cold
Flake	Same job passes on retry with no change	Fixed timeout or exact-string assertion in a cross-platform test
Environment skew	Works locally, fails in CI image	Tool version differs (strip/binutils, glibc, NDK, Xcode) — check the job's image:/runner tag

Step 3 — Fix by the hard rules

Apply the rule for the class; do not just patch the symptom:

1. Absolute paths only in CI scripts — anchor on $CI_PROJECT_DIR. Never rely on a cd from an earlier line. After any destructive/cleanup line, verify the effect (ls, test -f) rather than trusting -f.
2. No silent green. Every error path exits non-zero. If a failure is genuinely acceptable, model it as allow_failure: true on the JOB (visible in the UI), never as exit 0 in the script. Assert outcomes: artifact exists, upload count > 0, store accepted the bundle.
3. Every new test job goes into release:gate.needs in the same commit. Publish jobs start as soon as their own needs succeed, so a red test job that is missing from that list does not hold anything back — that is how v1.4.0 twice came within one job of publishing past a failed test (the release:gate.needs comment in ci/publish.yml names both). Deliberate exception: test:audit is allow_failure: true and left out on purpose (docs/architecture-gates.md) — adding it would let a new advisory block a release.
4. Secrets into nested environments are passed explicitly (Cirrus VM, docker-in-docker). Never assume a CI variable propagates one level down.
5. Caches upload when: always so a timed-out job doesn't doom every retry to a cold start.
6. Flakes get root-caused — wait on conditions/events, not durations; don't assert exact user-facing strings. One commented timeout bump is the ceiling; a second bump on the same job means find the real cause.
7. Keep MR-blocking vs allow_failure/manual intent explicit in rules: — don't accidentally promote a manual job to blocking or vice versa.

Step 4 — Verify without burning a tag

• MR-scoped jobs: push a branch/MR and watch that pipeline.
• Tag-only jobs: never verify by tagging. Options, in order of preference:
  1. Run the job's script locally with the same inputs (most publish scripts are scripts/*.py|mjs|sh — they run outside CI).
  2. Push a temp branch with a one-commit rule override making the job run on that branch (rules: - if: $CI_COMMIT_BRANCH == "ci-dryrun-x"); revert before merge.
  3. For the updater path specifically: node scripts/release-build.mjs e2e rehearses build→sign→serve→verify end-to-end with localdev keys.
• After a failed tag: fix on main first. Then ask the user before any retag/new tag (publishing is an ask-first action). Repo convention favors bumping the patch version over deleting tags; release creation is retag-safe but partially-published artifacts from the failed run must be accounted for.

Watch loop after pushing a fix (jobs run 10–60 min; poll accordingly):

watch_pipeline() {
  curl -s $H "$API/pipelines/$1" | jq -r '"\(.status)  \(.web_url)"'
  curl -s $H "$API/pipelines/$1/jobs?per_page=100" \
    | jq -r '.[] | "\(.status | .[0:7])\t\(.stage)\t\(.name)"' | sort
}

Pre-tag checklist (run when asked for a "pre-tag check" or before /release tags)

• just check green locally
• Last main pipeline fully green (not "green with failed-but-allowed jobs you care about")
• .gitlab-ci.yml diff since the last tag reviewed against the rules above (git diff $(git describe --tags --abbrev=0) -- .gitlab-ci.yml ci/)
• Any NEW tag-only job since the last tag has been exercised (Step 4)
• release:gate.needs still lists every test job except the allow_failure reporter test:audit, non-blocking by design (grep -A30 'release:gate' ci/publish.yml)
• test:audit was yellow because it FOUND something, not because it never ran — its log has no AUDIT-DID-NOT-RUN line (allow_failure renders both the same)
• Updater artifacts: signing is the LAST touch on bytes (mesa patch / notarize / Authenticode all happen before .sig) — see docs/release/updater.md

Report format

State: pipeline + failing job(s) with links · the class from the taxonomy · root cause (quote the log line) · the fix and which hard rule it enforces · how it was verified (pipeline link or local run output) · any follow-up risk (e.g. "publish:ios remains unexercised until next tag").