The postmortem that changes something
Most incident reviews produce a document and a ticket that never gets done. Here's the difference.
Most organizations run blameless postmortems. Most organizations also have the same class of incident repeatedly. Both of those things are true simultaneously and nobody finds it strange.
the failure pattern#
The typical postmortem:
- Timeline of what happened. Accurate, detailed, useful.
- Root cause. Usually a single technical fact.
- Action items. Five to twelve of them.
- Filed. Two action items get done. The rest age out.
Six months later, a similar incident, with a similar document.
Three things are wrong here.
problem one: "root cause" is singular#
Complex systems do not fail because of one thing. They fail because several conditions aligned, each of which was individually survivable.
The Cloudflare outage in November: a permissions change, a query that returned duplicates, a fixed buffer size, an error path that panicked instead of degrading, and a config pipeline that propagated globally in minutes. Remove any one and it does not happen, or it is much smaller.
Picking one and calling it "the root cause" means you fix one and leave four.
Better framing: contributing factors, plural, each with its own assessment of whether it is worth addressing. Some will not be — that is a legitimate decision if it is made explicitly.
problem two: action items without owners and dates are wishes#
An action item that says "improve monitoring for the config pipeline" with no owner, no date, and no definition of done is a sentence, not a plan.
The fix is unglamorous:
- Every action item has one named person. Not a team. A person.
- Every action item has a date. If nobody will commit to a date, it is not going to happen and you should delete it and say so.
- Every action item has a definition of done that someone else could verify.
- They go in the same backlog as feature work, prioritized against it. An action item in a separate "incident follow-up" list that is never sprint planned is a list of things that will not be done.
And the one that actually forces it: review the open action items at the start of the next postmortem. Nothing motivates completion like a room full of people looking at your undone item from last quarter's incident while discussing this quarter's similar one.
problem three: nobody asks about the near misses#
The incidents you review are the ones that broke through. For every one, there were several that did not — a bad deploy caught by a canary, a config error someone noticed in review, a query that would have taken down the database if it had run on Monday instead of Sunday.
Those contain the same information at a fraction of the cost, and almost nobody collects them.
Add a lightweight channel for it. "I nearly broke prod today, here is how." No document, no meeting, no blame. Just a note. The pattern that emerges over a quarter is more valuable than any individual postmortem.
the questions that produce useful findings#
Replace "what was the root cause" with:
- "What made this hard to detect?" Detection time is usually the largest component of impact and is the most improvable.
- "What made this hard to diagnose?" Usually missing observability. This produces the highest-value action items.
- "What made recovery slow?" Often a missing runbook, a missing kill switch, or a rollback that was not actually tested.
- "Who knew something that would have helped, and why did that not reach the responders?" This is an organizational question and it is frequently the real finding.
- "What did we do that helped?" Genuinely important. Practices that worked should be named so they get kept.
the blameless part, done correctly#
Blameless does not mean nobody made a mistake. It means the analysis focuses on why the mistake was possible and easy, rather than on the person.
"Alice deployed without running the tests" is blame and it is also useless. "The deploy path does not require tests to pass, and the shortcut that skips them is the fastest way to deploy" is the same fact stated in a way you can act on.
If the answer to "why did they do that" is "because the system made it easy and the correct path was hard," you have found something.
If the answer is genuinely "they were careless," you still fix the system, because the next person will also be careless eventually. Humans are the constant; the system is the variable.
— Dom, December 16, 2025