kortix-rollback
How to roll Kortix PRODUCTION back to an older already-released version — the inverse of a release. Covers the one-dispatch rollback-prod.yml engine (--ref prod, shares the deploy-prod concurrency gro
SKILL.md.agents/skills/kortix-rollback/SKILL.md
GitHubKortix Rollback
The inverse of kortix-release. A release moves prod FORWARD onto new code; a
rollback re-points prod at an OLDER already-released vX.Y.Z, reusing that
release's prebuilt artifacts (zero rebuild). Use it when a shipped version is
breaking and the move is "go back now, fix forward later." The next forward
release still comes from staging, not directly from dev.
A rollback is temporary, never sticky. It rolls ECS services and promotes an old Vercel build. It never pushes to
prod, never touchesVERSIONor a branch, and never moves an image tag. The nextdeploy-prodrun supersedes it and returns prod to the version inprod:VERSION(see §6).
1. The one command
gh workflow run rollback-prod.yml --repo kortix-ai/CompanyOS --ref prod \
-f version=vX.Y.Z -f reason="<incident summary>" -f confirm="ROLLBACK PROD"
--ref prod is required: the job runs in environment: prod, which admits the
prod branch only. The run shares the deploy-prod concurrency group with
deploy-prod.yml, so a rollback dispatched during a release roll starts when
that roll ends. GitHub keeps one pending run per group: a rollback dispatched
while a deploy-prod run waits replaces the waiting run.
It plans each surface from the images that exist for X.Y.Z (a surface with no
:X.Y.Z image is skipped), then rolls in this order, one step after the other:
- api (ECS):
ecs-deploy.sh prod kortix/kortix-api:X.Y.Z --process-role all. The api tasks run the scheduled jobs. After that rollout is stable the script scaleskortix-prod-workerto 0. The worker is never rolled here. - gateway (ECS):
kortix/kortix-gateway:X.Y.Z. - frontend (ECS):
kortix/kortix-frontend:X.Y.Z, verified onprod-fe-ecs.kortix.com/api/health. - frontend (Vercel): promotes the production deployment whose commit message
names
release vX.Y.Z. Skipped withoutVERCEL_TOKEN.
A failed step stops the run: the surfaces before it are on X.Y.Z, the ones
after it are on the current version. Read the step log, not the summary table:
the summary prints the plan, not the result. Re-run with -f surfaces=<the rest>.
What it does not do:
- It moves no image tag.
:latest,:prodand:stablestay on the current version, so self-hosted boxes keep what they run. - It opens no pull request and merges nothing into
prod(the job hascontents: read). - It does not roll the database back (§4).
2. Per-surface mechanics
Backend (API, worker, gateway) and the ECS frontend run immutable images on
ECS Fargate. infra/scripts/ecs-deploy.sh registers a new task-definition
revision that points at the :X.Y.Z image and waits until the service is
stable. The image is frozen, so the roll is exact.
The Vercel frontend is built on the deploy-prod.yml runner and uploaded
prebuilt (infra/scripts/vercel-deploy-prebuilt.sh). Vercel keeps every
deployment's output, but there is no image tag. A Vercel rollback is a
promote of the old production deployment. The branch still holds the newer
frontend code.
3. What undoes a rollback
Nothing records that prod is rolled back: the state lives only in ECS and
Vercel. Each of these returns prod to the version in prod:VERSION:
- Any
deploy-prod.ymlrun. A push toprod(a release merge) or a manual dispatch (rebuild_desktop, a re-run) rolls worker, api, gateway and web toprod:VERSIONand switches the api back toKORTIX_PROCESS_ROLE=web. Do not dispatchdeploy-prod.ymlwhile prod is rolled back, unless you mean to roll forward. - Vercel never builds on a push. The Vercel project has no Git connection,
and
apps/web/vercel.jsonsetsgit.deploymentEnabled: false. kortix.com changes only throughdeploy-prod.yml(deploy-web-vercel), so a Vercel instant rollback of the frontend holds until the nextdeploy-prod.ymlrun.
Until one of these runs, the worker stays at 0 and the api runs the scheduled
jobs (KORTIX_PROCESS_ROLE=all). Nothing expires that state: plan the roll
forward.
4. Safety check BEFORE you roll — DB / state drift is the real blocker
The image-reuse + Vercel-promote mechanics work for any released version. What actually makes a rollback unsafe is forward state the old code can't handle — overwhelmingly the database.
Migrations in this repo are forward-only and additive by convention (ADD COLUMN/VALUE, CREATE … IF NOT EXISTS). Against additive migrations old code is
safe: the live schema is a strict superset of what the old version needs.
Never reverse a migration — that's what breaks things, not rolling code back.
The danger is a destructive migration between the target and current — a DROP COLUMN, RENAME, ALTER … TYPE, or a new NOT NULL. Old code hitting that
breaks. The plan step prints how many migrations prod has that vX.Y.Z lacks,
and a ::warning:: names the files that contain one of those statements. It
does not refuse: a person decides. Check it before you dispatch:
git diff vX.Y.Z..origin/prod -- packages/db/migrations
- All additive (
ADD …,CREATE … IF NOT EXISTS) → DB-safe to roll there. - Any
DROP/RENAME/ALTER … TYPE/ newNOT NULL→ stop and think.
Lesser, non-DB couplings (rare long tail): forward-written persisted state old code can't parse (queue/cache formats), or external contract changes (Stripe API version, webhook shapes).
Kortix Apps: static hosting and retention do not roll back
Two Apps changes ship in one release and survive a code rollback:
Static Apps go offline on an older api. A static App is served by the api from storage (
apps/api/src/apps/static-site.ts) and has noapp_runtimesrow. An api that predates that file requires a runtime, so every static App answers with a status page until the next promote.rollback-prod.ymlrefuses such an api target unlessstatic_apps_ackis exactlySTATIC APPS OFFLINE. Rolling onlygateway,frontendis not affected.Retired deployments stay retired. The first maintenance pass after the release retires every ready deployment beyond the active one and the newest
KORTIX_APPS_RETAINED_DEPLOYMENTS(default 5), 50 Apps per pass. It tears down their runtimes and images;reclaimAppArtifactsdeletes their archives 24 h later. A rollback cannot bring them back. Before the first release, count what it will retire:select app_id, count(*) - 6 as retired from kortix.app_deployments where status = 'ready' group by 1 having count(*) > 6;To defer it, set
KORTIX_APPS_RETAINED_DEPLOYMENTShigh on prod for that release and lower it on the next one.
Email: received mail waits for the roll-forward
From the release that stores each received email before the ack (KRTX-2235
step 3), the api puts Email on the integration.events.email job queue. An
older api never claims that queue, so emails stored before the rollback stay
queued until a later promote runs them; Resend got its 200 and does not
resend. Nothing is lost, only late. List what waits:
select job_key, created_at from kortix.job_queue
where queue = 'integration.events.email' and status = 'queued' order by created_at;
If a sender must not wait and the rollback target has the Resend route
(POST /v1/webhooks/email/resend, first shipped after v0.14.8), replay that
email from the Resend dashboard: the older api handles it, and the stored copy
is a no-op at the roll-forward (its session create already started, or its
follow-up key exists). A target without that route receives no Resend mail at
all; Resend's own retries (8 over about 31 h) then decide what arrives after
the roll-forward. Delete nothing.
5. Per-surface availability (the engine handles this)
You can't roll a surface BELOW where it existed. The gateway only entered prod
at ~v0.9.72 (image exists from 0.9.69), so there is no kortix-gateway:0.9.68.
The workflow checks each image (Docker Hub tags API) and skips any surface with
no :vX.Y.Z image rather than wedging ECS on ImagePullBackOff.
6. Resuming — a later release returns prod to LATEST
You do NOT manually undo a rollback. To resume, get the fix onto staging and
Promote to Production (see kortix-release). Its merge into prod runs
deploy-prod.yml, which rolls ECS to the new version, scales the worker back
up, and builds and deploys the latest Vercel frontend.
7. Gotchas
- Never roll an ECS service from a laptop while a deploy workflow can run (learnings ledger, 2026-11-01): the workflow's concurrency group does not see a local roll. Use this workflow.
- Verify after:
api.kortix.com/v1/health(version), ECS service health,prod-fe-ecs.kortix.com/api/health, and the live Vercel production deployment (/v9/projects/{id}→targets.production).