ops-rollback
Revert AICP/fleet to a recorded last-known-good state — uses the rollback contract that `ops-deploy` Operation 4 records (commit SHA + image tags + profile name in `docs/DEPLOY-LOG.md` and `/tmp/prede
SKILL.md.claude/skills/ops-rollback/SKILL.md
GitHubops-rollback
The reverse-the-deploy skill. Reads the rollback contract that ops-deploy recorded, snapshots failed-state evidence, executes the reversion, verifies recovery, updates the deploy log. Distinct from ops-incident (broader incident process — rollback may be one of several tools an incident uses) and from ops-deploy (forward action with gates — this skill is its explicit reverse).
Trigger phrases (when to load this skill)
Load when the conversation matches any of:
- Direct verb: operator says "rollback", "revert the deploy", "go back to last good", "undo the release", "back out the change", "deploy was bad — revert".
- Within an incident:
ops-incidentOperation 1 identified that the active deploy is the cause; rollback is the chosen restoration mechanism. - Post-deploy regret: deploy succeeded smoke tests but is now showing problems in metrics — the operator wants to revert before the problem grows.
Do NOT load when:
- Multiple things are broken with unclear cause — load
ops-incidentfirst; rollback is one option there. - The current deploy is the right code; rollback would just regress to a different problem — load
ops-incidentto find a forward fix. - Operator wants to revert a single commit (not a deploy) — that's a
git revertworkflow, not this skill. - No prior deploy was recorded — there's nothing to roll back TO; load
ops-incidentfor restoration via other means. - The "rollback" is a config change (profile switch, env var revert) — load
config-deployfor surgical config reversion.
Operations
This skill has 4 named operations. Execute in strict order — preservation of evidence comes BEFORE reversion.
Operation 1: Identify the rollback target + verify it's good
Trigger: skill loaded; operator named the environment to roll back (or single-environment deployment is implied).
Process:
- Find the rollback contract recorded by the prior
ops-deploy:
Capture: previous commit SHA, previous image tags, previous profile name.ls /tmp/predeploy-*-<env>.txt 2>/dev/null tail -5 docs/DEPLOY-LOG.md # last successful deploys - If no contract exists (skill being run cold without a prior
ops-deploy):- Look for the last successful deploy in
docs/DEPLOY-LOG.mdto identify the SHA. - If no log either: identify last-known-good by
git log+ operator confirmation. Don't pick blindly.
- Look for the last successful deploy in
- Verify the rollback target is ACTUALLY good (don't roll back to another broken state):
If the target's deploy log entry saysgit log -1 --format="%H %s %ci" <target-sha> # confirm: this SHA was the deployed-and-smoked state, not a half-deployed stepstatus=ok, it's verified. Ifstatus=failedor no entry, escalate — operator must pick a different target. - Compute what specifically will change on rollback:
Surface this to operator: "rolling back N commits across M files; the changes you're undoing are: [list]". Get explicit confirmation before proceeding.git diff --stat <target-sha>..HEAD # files that will revert git log --oneline <target-sha>..HEAD # commits that will be undone - Identify migration / state implications — the data class of "irreversible":
- Any DB schema migration in the diff? (AICP has no SQL; fleet projects might.) If yes, rollback may strand data. Surface and pause.
- Any config that changed file paths, retention windows, encryption keys? Rolling back doesn't unmove files or undelete state.
- Any LocalAI Collection re-indexing? Rolling back doesn't undo that work but may make the new index incompatible.
Quality bar (Operation 1 done when):
- Rollback target SHA + image tags + profile name explicitly identified.
- Target verified as last-known-good (deploy log entry or operator confirmation).
- Diff/log preview shown to operator with commit list + file count.
- Operator explicitly confirmed (not implied) — rollback is destructive of forward work.
- State-irreversibility implications surfaced: migrations, file moves, re-indexes.
Operation 2: Preserve failed-state evidence
Trigger: Operation 1 confirmed; rollback is about to mutate state.
Process:
- Snapshot the failed-state — same shape as
ops-incidentOp1 (this skill is reusing that pattern):ts=$(date -u +%FT%TZ); mkdir -p /tmp/rollback-$ts docker compose ps > /tmp/rollback-$ts/compose-ps.txt docker compose logs --tail 500 > /tmp/rollback-$ts/logs.txt 2>&1 .venv/bin/aicp --check > /tmp/rollback-$ts/aicp-check.txt 2>&1 .venv/bin/aicp --dlq-status > /tmp/rollback-$ts/dlq.txt 2>&1 git log --oneline -10 > /tmp/rollback-$ts/recent-commits.txt git diff <target-sha>..HEAD > /tmp/rollback-$ts/diff-being-undone.diff - Capture the FAILING commit SHA (current HEAD before rollback) — needed for
ops-incidentto diagnose root cause AFTER service is restored:git rev-parse HEAD > /tmp/rollback-$ts/failed-sha.txt - If runtime state is at risk (e.g., DLQ entries, in-flight tasks): copy them to the snapshot dir before any restart:
cp -r ~/.aicp/dlq/$(date +%Y-%m-%d).jsonl /tmp/rollback-$ts/ 2>/dev/null cp .aicp/state.yaml /tmp/rollback-$ts/ 2>/dev/null
Quality bar (Operation 2 done when):
-
/tmp/rollback-<ts>/exists with logs + ps + check + dlq + recent-commits + diff. - Failed SHA captured to file (preserves identity through rollback).
- Runtime state copied if at risk.
- Snapshot completed BEFORE any rollback mutation.
Operation 3: Execute the rollback
Trigger: Operation 2 evidence preserved.
Process:
- Move the code:
Prefergit checkout <target-sha> # or for branch-tip rollback: git reset --hard <target-sha> # only if operator explicitly authorized destructive (rare; checkout is preferred)git checkout(detached HEAD) for safety; the operator can later branch off it.reset --hardonly on explicit authorization — it can lose unpushed commits in environments where this checkout could lose work. - Move the profile if rollback contract included one:
make profile-use PROFILE=<previous-profile> - Rebuild + restart services:
Usedocker compose build 2>&1 | tail -10 docker compose up -d 2>&1 | tail -10up -d(not--no-deps) — env vars and shared config likely changed alongside the rollback. - Watch logs for first 30 seconds — same pattern as
ops-deployOp2:docker compose logs -f --tail 50 --since 30s & sleep 30; kill $! - If rollback itself fails (rolled-back code doesn't boot): this is an emergency. Either roll FURTHER back (to an even earlier known-good), or invoke
ops-incidentbecause the system has no known-good.
Quality bar (Operation 3 done when):
- Code moved to target SHA (verified by
git rev-parse HEAD). - Profile activated if part of rollback contract.
- Compose services rebuilt and up.
- No service in
Restartingstate. - No FATAL or "Exited" entries in first-30s logs.
Operation 4: Smoke + log + handoff
Trigger: Operation 3 reverted state running.
Process:
- Smoke test the rolled-back system — same shape as
ops-deployOp3:
All services healthy + golden-path inference responds = rollback succeeded.docker compose ps --format json | jq -r '.[] | "\(.Name): \(.State) (\(.Health // "no-healthcheck"))"' .venv/bin/aicp --check 2>&1 | head -20 .venv/bin/aicp --backend local --prompt "say ok" 2>&1 | tail -5 - Update
docs/DEPLOY-LOG.mdwith the rollback entry:<ts> deploy=<env> sha=<target-sha> status=rollback by=<op> reverted-from=<failed-sha> reason="<one-line>" - Notify operator surfaces:
- ntfy:
ROLLBACK <env> reverted <failed-sha> → <target-sha> dur=<minutes> - Mission Control / standing-orders update if fleet was affected.
- ntfy:
- Hand off to
ops-incidentfor diagnosis:- The failed-state snapshot in
/tmp/rollback-<ts>/is the evidence. - The diff in
diff-being-undone.diffis the change set to investigate. - Tell operator explicitly: "service restored via rollback; load
ops-incidentto identify root cause and author durable fix before re-deploying."
- The failed-state snapshot in
Quality bar (Operation 4 done when):
- All services smoke-test green at the rolled-back state.
-
docs/DEPLOY-LOG.mdupdated with rollback entry naming both target and failed SHAs. - Notification sent (ntfy / MC / explicit "no notification surface configured").
-
ops-incidenthandoff explicitly stated to operator with snapshot path. - No half-rolled-back state remaining (no service still on the failed image, no profile still on failed config).
Gotchas (known failure modes — read before doing)
Gotcha 1: Rolling back to another broken state
Operator says "rollback to last week's deploy". Skill picks the SHA from a week ago — but that deploy was ALSO rolled back at the time, and the status=ok entry is the one from 9 days ago. Skill rolls back to a known-broken target; service stays broken; operator now has TWO incidents to debug.
The rule: Operation 1 step 3 verifies the target by reading docs/DEPLOY-LOG.md for status=ok. If the most recent entry for the target SHA is status=failed or status=rollback, that SHA is NOT a valid rollback target — keep walking back the log until a confirmed-good is found, or escalate to operator.
Gotcha 2: Rollback wipes runtime state
Failed deploy added a new field to a config; operator's runtime now references entries with that field. Rollback reverts the config; on restart, the runtime crashes parsing entries that the rolled-back code doesn't understand. Rollback "succeeded" code-wise but the system is now broken in a new way.
The rule: Operation 1 step 5 surfaces state-irreversibility implications BEFORE rollback. If runtime state references the new code's shape, rollback alone isn't sufficient — either migrate state back or accept data loss. Don't rollback in the dark.
Gotcha 3: Mutating before snapshotting
Same anti-pattern as ops-incident Gotcha 1: skill jumps to git checkout <target> without first capturing the failed-state. Logs that lived in the running container are gone after restart; the failed-state diff is unrecoverable; root cause is now a guessing game.
The rule: Operation 2 is non-negotiable. Snapshot to /tmp/rollback-<ts>/ BEFORE any mutation in Operation 3. The five-second cost preserves the entire evidence base for ops-incident to use.
Gotcha 4: git reset --hard loses unpushed forward work
Skill defaults to git reset --hard <target-sha>. But the operator had two un-pushed commits on the failed branch (a half-finished hotfix attempt that hadn't been committed cleanly). Reset destroys them; the operator's last 30 minutes of work are gone with no recovery.
The rule: Operation 3 step 1 prefers git checkout <sha> (detached HEAD). reset --hard is reserved for explicit operator authorization and is never the default. If the operator wants the branch tip moved, they say so AFTER the system is restored — restoration first, history surgery second.
Gotcha 5: Rollback declared successful, root cause unidentified
Service is back; deploy log shows the rollback; operator moves on. But nobody loaded ops-incident — root cause is unknown, and the same broken deploy will be re-attempted next week with the same failure mode.
The rule: Operation 4 step 4 makes the handoff EXPLICIT. The rollback skill's job ends with "service restored AND ops-incident invoked-or-explicitly-deferred". If operator defers ("we'll diagnose later"), the deferral is logged in docs/DEPLOY-LOG.md (status=rollback diag-deferred), so the next deploy attempt has a record that the cause is unknown.
Reference exemplars
The Extension Standards reference exemplars for skills are the second brain's model-builder and wiki-agent skills — see ~/devops-solutions-research-wiki/skills/. This skill follows their structure: trigger phrases, multiple named operations, per-operation Process + Quality bar, Gotchas with detection + rule + reasoning.
Domain context
This skill operates in the backend-ai-platform-python domain. AICP rollback is mostly straightforward — code+config+profile, no schema migrations to worry about — because there's no SQL database. Sister fleet projects (openfleet, dspd, nnrt) may have schema migrations; this skill flags them in Op1 step 5 and stops if rollback would strand data. The rollback contract relies on ops-deploy having recorded predeploy state correctly; if the skill is run cold (no contract), docs/DEPLOY-LOG.md is the fallback source of truth. AICP profile YAMLs are part of the rollback contract — a deploy that activated default -> personal is incomplete to roll back without also reverting the profile.
Related skills
| Skill | When | Why distinct |
|---|---|---|
| ops-deploy | Forward action with gates + recorded contract | Forward; this skill is its explicit reverse |
| ops-incident | Active incident response | Broader process; rollback is one mechanism this skill formalizes |
| config-deploy | Surgical config reversion (profile switch) | Just config; this skill reverts code+config+profile together |
| ops-maintenance | Routine upkeep | Preserves status quo; this skill is destructive of forward work |
| evolve-migrate | Cross-version foundation swap | Forward migration; this skill is reverse |
| incident-cycle | Compound incident → fix → prevention workflow | Compound; this skill is the rollback step within it |