incident-responder

Runbook skill for failures: cron error, PA failure, cascade, gateway disconnect, semantic DB stale. Walks: detect → classify → diagnose → notify → log. Replaces ad-hoc failure handling. Triggers: "cro

SKILL.md

GitHub

Incident Responder

Standardizes failure handling. No more ad-hoc.

Classes

Class Trigger First action
cron-failure cron_health.py reports >26h gap Run cron manually, capture output
cron-cascade detect_cascade() returns true Suspect shared config; check Vertex proxy first
gateway-499 WA gateway 499 disconnect >10min docker restart ocana-litellm-proxy
semantic-stale check_sqlite_freshness() False Re-run memory_search ingest pipeline
pa-failure PA agent error reported DM owner; do NOT message PA's owner directly

Procedure (every class)

  1. Classify using table above
  2. Diagnose — collect logs (tools/error_tracker.py log <component> <msg>)
  3. Notify — DM the owner ONLY (not the affected PA's owner)
  4. Mitigate — execute the first action from the table
  5. Log — append daily-note: [HH:MM IL] INCIDENT <class>: <one-line summary>; resolved=<yes/no>
  6. Postmortem (if user-visible impact >5min) — create memory/incidents/YYYY-MM-DD-<class>.md

Anti-patterns

  • Do NOT loop trying the same fix twice
  • Do NOT tell the PA's owner there's a problem before the owner knows
  • Do NOT skip the daily-note log entry