outbox-idempotency
Transactional outbox pattern for reliable event delivery: idempotency keys, ordered replay, poison-message handling, and leak tests that catch duplicates or lost events. Use when adding event-driven i
SKILL.mdskills/engineering/outbox-idempotency/SKILL.md
GitHubStandard: name/description/length rules — agentskills.io standard.
Outbox Idempotency Skill — Quick Reference (body <500 lines; full procedure in references/outbox-procedure.md)
When to use this skill (trigger)
Any of these fires it:
- Adding an event-driven integration (webhooks, CDC, message bus) that must not lose or duplicate events.
- A migration introduces an
outboxtable or anidempotency_keycolumn. - An incident report says "event delivered twice" or "event never arrived".
- Code review mentions
INSERT INTO outboxwithout a unique key or replay logic. - Designing retry semantics for a service that writes events and sends them.
Disambiguation
- Fires: event-driven integrations that must not lose/duplicate events, an outbox/idempotency_key migration, 'event delivered twice/never arrived', retry semantics for an event writer.
- Does NOT fire for: a plain CRUD endpoint with no events, or tenant isolation (use pg-rls-multitenant).
When NOT to use this skill (neighbour prompts — do not fire)
- Fire-and-forget logging or metrics where ordering does not matter and duplicates are harmless — there is no outbox to protect and no idempotency key is needed.
- A synchronous RPC that returns before any work is queued — the skill fires only when work outlives the request.
- "We already have retries" — retries are the reason the UNIQUE constraint exists; the skill fires when retries are mentioned alongside outbound writes.
Decision table — idempotency strategy
| Pattern | When to pick | Pitfall |
|---|---|---|
| UUID v7 idempotency key on every outbox row | Default; sortable by time; collision-safe | Key must be generated BEFORE the business transaction commits |
| Natural key (entity_id + version) | Immutable entities with monotonically increasing versions | Version gaps look like missing events if not handled |
| Hash of payload + sequence number | High-throughput streams where UUID cost matters | Hash collisions possible; pair with sequence check |
Near-miss triggers (act before the incident)
- A poller uses
LIMITwithoutORDER BY→ out-of-order delivery for ordering-dependent consumers. UPDATE sent_athappens BEFORE the publish ACK → duplicate publish on crash.DELETEappears anywhere nearpoison_outbox→ lost event with no audit trail.max_attemptsis absent or 0 → poison messages block the queue forever.- Outbox INSERT and business write are in separate transactions → lost events on connection drop.
- Replay reuses a fresh idempotency key → double-delivery even after the root cause is fixed.
Numbered procedure (summary; full recipe in references/outbox-procedure.md)
- Create outbox table —
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),idempotency_key text NOT NULL UNIQUE,payload jsonb NOT NULL,created_at timestamptz NOT NULL DEFAULT now(),sent_at timestamptz,attempts int NOT NULL DEFAULT 0,last_error text. - Write in same TX — INSERT outbox row + business data in ONE transaction; commit both or neither.
- Poller loop — SELECT ... WHERE sent_at IS NULL ORDER BY created_at LIMIT N FOR UPDATE SKIP LOCKED; publish; UPDATE sent_at = now() on success; increment attempts on failure.
- Poison handling — after max_attempts (default 5), move row to poison_outbox table with error snapshot; do NOT delete it.
- Replay — SELECT from poison_outbox ORDER BY created_at; re-publish only after root cause is fixed; keep original idempotency_key so dedup still works.
- Leak test harness —
scripts/check_outbox.pyvalidates schema presence, UNIQUE idempotency_key, poller ordering, poison threshold, and no lost events under concurrent publish. Run against Docker Postgres before merge; any FAIL blocks merge.
Worked examples
Good (one-transaction write):
BEGIN;
INSERT INTO accounts (id, balance) VALUES ('a1', 100);
INSERT INTO outbox (idempotency_key, payload)
VALUES ('acct-a1-debit-001', '{"event":"account.debitted","account":"a1"}');
COMMIT;
Bad (separate transactions — loses the event on connection drop):
INSERT INTO accounts (id, balance) VALUES ('a1', 100);
-- network drop here == event never sent, no row to recover
INSERT INTO outbox (idempotency_key, payload) VALUES (...);
Bad (poller without ordering — consumers that care about order break):
SELECT ... FROM outbox WHERE sent_at IS NULL LIMIT 50 FOR UPDATE SKIP LOCKED;
Pitfalls from research (real incidents)
- Out-of-order delivery when poller uses LIMIT without ORDER BY + SKIP LOCKED.
- Duplicate publishes when poller crashes between publish and UPDATE sent_at.
- Poison messages block the queue when max_attempts is absent.
- Lost events when business TX commits but outbox INSERT fails due to connection drop.
- Replaying without idempotency_key causes double-delivery even after fix.
Verification checklist (all must pass before merge)
- Outbox table has UNIQUE idempotency_key
- Business write + outbox INSERT are in ONE transaction
- Poller uses ORDER BY created_at + FOR UPDATE SKIP LOCKED
- Poison threshold moves rows after max_attempts
- Replay preserves original idempotency_key
- Leak test harness runs green against Docker Postgres
- No DELETE on poison_outbox rows
Verification
- Run the offline gate:
python3 scripts/check_outbox.py <sql-file>— must exit 0. - Run the self-test:
python3 scripts/check_outbox_test.py— must print ok. - Run the standard validator of the repository (from the repository root, not from this skill folder):
python3 scripts/validate_skills.py— must report OK. - Full procedure, version table, worked examples, and pitfalls:
references/outbox-procedure.md. - Eval prompts with expected behaviour and failure signs:
references/EVALS.md. - Primary sources with URLs and retrieval dates:
references/SOURCES.md.
Related skills
- acceptance-harness — the outbox leak test is an acceptance scenario with PASS evidence.
- honest-status — the outbox migration is a status item; mark it VERIFIED only when the leak test ran.
- problem-json-contracts — outbound failures are emitted as problem details.
- long-test-discipline — the leak test is a long test; run it in the foreground with a recorded seed and a resource cap.
- module-contract-first — the outbox schema and the idempotency contract are module contracts; snapshot them, block merge on drift.
- shared-tree-git — the outbox migration and the leak-test evidence are committed with a task-id prefix.