incidents
Load this skill when responding to a production incident, writing a post-mortem, executing a hotfix, or classifying incident severity. Auto-load whenever a production issue is reported, a P1/P2/P3 cla
SKILL.mdarchive/ceh-ops/skills/incidents/SKILL.md
GitHubIncident Response
Severity Levels
| Level | Definition | Response Time |
|---|---|---|
| P1 | Production is down or data is corrupted | Immediate — all hands |
| P2 | Major feature broken, no workaround for users | < 1 hour |
| P3 | Feature degraded, workaround exists | < 1 business day |
Response Steps
- Detect — identify from monitoring alerts or user report
- Triage — classify severity, identify scope
- Mitigate — roll back if available; disable the feature if possible
- Fix — hotfix process (minimal scope, all CI must pass, staging still required)
- Post-mortem — written within 48 hours for P1/P2
Hotfix Process
- Branch:
fix/critical-<description>frommain - Scope: Minimal fix only — no unrelated changes
- Review: 1 approval minimum, fast-tracked
- CI: All checks must pass — do not skip CI under pressure. A broken hotfix is worse than a delayed one.
- Merge: Merge commit to
main(never squash — preserve the commits) - Tag: Bump PATCH version, apply tag
- Deploy: Staging → production (abbreviated but both still required)
Post-Mortem Format (Required for P1 and P2)
## Incident Post-Mortem: <Short Title>
**Date:** YYYY-MM-DD
**Severity:** P1 | P2
**Duration:** <how long production was impacted>
### What Happened
Brief timeline of events.
### Root Cause
The single underlying cause (not symptoms).
### Impact
What was broken. How many users/requests affected.
### Detection
How was the incident discovered? How long between impact and detection?
### Resolution
What fixed it?
### Prevention
What changes prevent recurrence?
- [ ] Action item (owner, due date)