post-mortem-author

Overview

Per google-sre-postmortem:

"A postmortem is 'a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring.'"

"Blameless postmortems are a tenet of SRE culture." (google-sre-postmortem)

"Writing a postmortem is not punishment - it is a learning opportunity for the entire company." (google-sre-postmortem)

The blameless framing is load-bearing. Per google-sre-postmortem, the document must "focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior" - assuming "everyone involved in an incident had good intentions and did the right thing with the information they had."

When to use

Per google-sre-postmortem, common triggers include:

"user-visible downtime, data loss, on-call interventions, extended resolution times, and monitoring failures."

Author a post-mortem after every such incident - not just sev-1. Lower-severity incidents accumulate context that prevents the sev-1.

Step 1 - Author the document

Copy the full section skeleton from references/post-mortem-document-template.md and fill every section. The required sections, in order:

1. Header - status, severity, authors, dates, reviewers.
2. Summary - 2-3 sentences: what, who, how long, what was done.
3. Impact - users affected, revenue (deferred vs lost), SLO debt, reputational.
4. Timeline - chronological events with UTC timestamps and a source per row.
5. Root cause - what happened in detail, with the system as the subject, never a person.
6. Contributing factors - every condition that allowed the incident; list all, since incidents rarely have one cause.
7. What went well - the positives; per google-sre-postmortem, post-mortems should call these out too.
8. Action items - the load-bearing section (Step 3).
9. Lessons learned and Postmortem trigger - what is known now, and which trigger criteria the incident met.

The worked example below fills this skeleton for a real SEV-2.

Step 2 - Blameless review

Per google-sre-postmortem: "Postmortems are not punishment."

Reviewers should:

• Reject blame language. "Bob deployed without testing" → "The pre-deploy testing didn't cover the null-metadata case."
• Focus on systems, not individuals. "Why did Bob fix this fast?" → "What enabled fast diagnosis? (Sentry stack trace + the team's incident-response training.)"
• Surface contributing factors openly. Multiple factors usually contribute; document all.

Step 3 - Action item discipline

Action items must have:

• Owner (one named person; not "the team").
• Due date (concrete; not "next sprint when we have time").
• Priority (P0/P1/P2/P3; ties to the action's importance).
• Success criterion (measurable; "X done" not "we'll think about X").

The action items are the post-mortem's value. Without them, the document is paperwork.

Step 4 - Approval + closure

The post-mortem isn't "done" until:

1. Reviewers (eng manager, SRE lead, Product) sign off.
2. Action items are added to the team's tracker (Jira / Linear / GitHub issues).
3. P1/P2 action items have a target completion before the next release.

The post-mortem is "closed" when all action items ship - typically 2-4 weeks. A 6-month-old open post-mortem is a process failure.

Step 5 - Storage

docs/postmortems/
├── INC-1234-stripe-webhook-2026-05-04.md
├── INC-1235-cache-invalidation-2026-05-12.md
├── INC-summary-2026-Q2.md       ← rollup
└── README.md

Markdown + git. Quarterly rollup identifies patterns:

## Q2 2026 incident summary

**Total incidents:** 12
**SEV-1:** 1
**SEV-2:** 6
**SEV-3:** 5

**Patterns:**
- 4 of 12 (33%) were "test gap" - the failing condition wasn't
  in the test suite. Action: invest in
  test-coverage-targeter
  + property-based testing.
- 3 of 12 (25%) involved canary metrics; 2 of those proceeded
  through canary gate. Action: review thresholds (per AI-2 from
  INC-1234).
- ...

Worked example - INC-1234 Stripe webhook failure (SEV-2)

The skeleton from Step 1, filled for a real incident.

Summary. A v1.4.5 deploy introduced a null-metadata crash in the Stripe webhook handler; ~12,400 customers (4.3% of MAU) hit failed checkout completions for 23 minutes until rollback.

Impact. ~$140,000 in delayed (not lost) orders; 32% of the monthly availability budget burned; 47 support tickets.

Timeline (excerpt).

Time (UTC)	Event	Source
14:00	Deploy of v1.4.5 to canary (5% traffic)	CD pipeline log
14:23	First Sentry alert: NullPointerException at WebhookHandler:42	Sentry
14:30	Canary window ends within thresholds; promoted to 100%	CD pipeline
14:42	PagerDuty SLO burn-rate alert; incident declared SEV-2	PagerDuty
14:58	Rollback complete; error rate returning to baseline	Datadog

Root cause (what, not who). The v1.4.5 handler added a path for Stripe's payment_intent.partially_funded event that called payment.metadata.get("internal_id"); for ~3% of events metadata was null, the exception was uncaught, the handler returned 500, and Stripe stopped retrying, so fulfillment never triggered. The canary stage saw the error rate rise (0.4% vs 0.3% baseline) but stayed under the 1.5x rollback threshold, so prod-canary-validator returned PROCEED with WARNING and the gate was acked.

Contributing factors. (1) test gap - no unit test for the null-metadata case; (2) canary threshold too lenient for a low baseline; (3) staging carries almost no Stripe webhook traffic, so the new event type was never exercised pre-deploy.

Action items.

ID	Action	Owner	Priority	Due	Success criterion
AI-1	Unit test for partially_funded with null metadata	Bob	P1	2 days	Test in WebhookHandlerTest.kt fails against the bug, passes after
AI-2	Tighten canary error-rate threshold 1.5x -> 1.3x	SRE	P2	1 sprint	canary-thresholds.yml updated; one normal canary passes
AI-3	Staging fixture covering all Stripe event types	Bob	P2	1 sprint	Staging "events by type" metric shows all types > 0

What went well. Sentry caught the regression at 14:23, well before the PagerDuty page; rollback finished in 7 minutes, inside RTO. Diagnosis came from the Sentry stack trace alone, with no production debugging.

Anti-patterns

Anti-pattern	Why it fails	Fix
Blame language	Defeats the blameless principle; team stops authoring post-mortems honestly.	Per Google SRE: focus on contributing causes, not individuals (Step 2).
Action items without owner / due date	Nobody acts; same incident recurs.	All four fields required (Step 3).
Skipping post-mortems for "small" incidents	Lower-severity context that prevents big incidents is missed.	Author per google-sre-postmortem trigger criteria (Step 1).
Post-mortem stored in private docs	Org learning capped at the team.	Public to org (per Google SRE pattern).
One-shot post-mortem with no follow-up	"Closed" but action items stale; recurrence likely.	Track action items in tracker (Step 4); post-mortem closed only when all done.
Post-mortem authored 2+ weeks after incident	Memory faded; details lost.	Author within 5 business days.

Limitations

• Honesty depends on culture. A team that fears blame will produce sanitized post-mortems. Leadership must reinforce blamelessness in word and action.
• Post-mortems don't prevent the next incident. They prevent the same class of incident. Pair with proactive practices (chaos testing, threat modeling).
• Time investment. A SEV-2 post-mortem typically takes 4-8 hours to author + 2-4 hours of review. Budget accordingly.

References

• gsp - Google SRE blameless post-mortem definition, triggers, blamelessness principle, learning-opportunity framing.
• prod-canary-validator - sibling: canary verdict; post-mortems often surface threshold tuning needs.