tech, developers, and the code underneath

issue 212· essay·

What a good incident channel looks like

The chat log is where the incident is actually run. Four conventions make it useful rather than a wall of noise.

Every incident runs in a chat channel. Almost nobody has thought about how that channel should work, so the default is thirty people speculating in parallel while two people try to fix something.

Four conventions fix most of it, and none of them need a tool.

1. one channel, created at declaration#

Not the team channel. Not a thread in the team channel. A dedicated channel per incident, named predictably: inc-2026-08-29-checkout-errors.

Why it matters: the incident becomes searchable as a unit, the timeline is the channel, and people who join late can read from the top instead of asking what happened. It also means the incident does not drown the normal channel, and the normal channel does not drown the incident.

Archive it afterwards rather than deleting it. It is the primary source for the postmortem.

2. named roles, stated out loud#

Three, at minimum, posted in the channel as the first message:

IC: @priya       — decides, delegates, does not debug
Comms: @marcus   — status page, stakeholders, customer support
Ops: @sam        — hands on keyboard

The incident commander not debugging is the rule people resist and the one that matters most. The moment the IC opens a terminal, nobody is tracking the whole picture, and the incident gets longer.

For a small incident one person can hold two roles. They should still say which ones, because the alternative is everyone assuming someone else is doing comms.

3. a pinned status message, edited in place#

One message, pinned, rewritten as things change:

STATUS 14:22 — Checkout failing for ~12% of users since 13:58.
Cause: unknown. Suspect the payments deploy at 13:55.
Now: rolling back that deploy (@sam), ETA 5 min.
Impact: card payments only, wallet payments unaffected.
Next update: 14:35

Five lines: what is broken, since when, what we think, what we are doing, when the next update comes.

This single convention removes most of the noise, because it answers the question that generates the noise — "what's the current state?" — without anyone having to ask. Everyone joining reads the pin instead of scrolling.

Always include the next-update time. It is what stops people asking for updates.

4. mark the speculation#

The most common way incidents go wrong is that a guess gets repeated until it becomes the working theory, and then twenty minutes go into the wrong system.

A one-word convention fixes it:

FACT: error rate went from 0.1% to 12% at 13:58:20
FACT: the payments deploy completed at 13:55:41
GUESS: the deploy caused it
ACTION: rolling back to confirm

Facts have evidence attached. Guesses are labelled as guesses. Actions say who is doing them.

It looks pedantic for about four minutes and then it saves the incident, because somebody reading the channel can tell the difference between what is known and what somebody said.

what to keep out#

Speculation from people who are not investigating. Well-meant and it fills the channel that responders are trying to read. If you are not on a role, watch.

"Is it fixed yet?" The pinned status has the next update time.

Root-cause analysis during the incident. Restore first. The good question at 14:22 is "what makes this stop", not "why did this happen". Why is a question for Thursday.

Blame, in any form, including jokes. It is in a permanent record that a person will read afterwards.

the handoff#

Long incidents cross shift boundaries and the handoff is where context dies. It needs to be explicit and in the channel:

HANDOFF 22:00 — @priya → @dan (IC)
Ruled out: deploy (rolled back, no change), CDN (unaffected regions also failing)
Current theory: connection pool exhaustion on the orders DB
In flight: @sam is capturing pg_stat_activity every 30s → thread above
Not yet tried: failover to replica
Customers: status page updated 21:40, support has the template

Ruled out, current theory, in flight, not tried, customer state. Five lines, and the incoming IC starts with the accumulated knowledge instead of rediscovering it.

why bother#

The channel is not a side effect of the incident. During the incident it is the coordination mechanism, and afterwards it is the only complete record of what was known and when.

A channel that follows these four conventions produces a postmortem that half writes itself, and — more importantly — an incident that ends sooner, because the people fixing it spent their attention on the system rather than on the chat.

get README in your inbox

One dispatch, no noise. Tech and developer news, plus the occasional long piece on the craft.

subscribe →