Run the incident before the incident

Every runbook that has never been executed is wrong. Not might be — is. The commands have changed, the dashboard moved, the person who wrote it left, and the system it describes has been modified fourteen times.

The only way to find out is to run it, and the only good time to run it is when nothing is actually broken.

what a game day is#

A scheduled exercise where you deliberately break something in a controlled way and have the on-call rotation respond as if it were real.

Two hours. A stated scenario. The people who would actually respond. Real tools, real runbooks, real dashboards. Someone taking notes on everything that did not work.

That is the whole practice. It is not chaos engineering in the automated-random sense, though that is a good adjacent practice. It is a rehearsal.

what you find, reliably#

I have never run one of these that did not find at least four of the following:

The runbook references something that does not exist. A dashboard that was renamed, a script in a repository that was archived, an alias nobody has.

Nobody has the access. The runbook says to restart the service. The on-call engineer does not have permission and does not know who does. This is the single most common finding.

The dashboard does not show the thing. You built the alert. You never built the view that tells you what to do about it.

The escalation path is a person, not a rotation. "Ask Sarah." Sarah is on vacation. Sarah left last year.

Nobody knows the customer impact. The service is degraded. Which customers? Which features? Nobody can answer, so nobody can decide how urgent it is.

Recovery has an undocumented step. The service restarts and does not work because a cache must be cleared first, which is knowledge that lives in one person's head.

The communication path is unclear. Who tells the customers? Who updates the status page? Who has the credentials for the status page?

Every one of those is cheap to fix on a Tuesday afternoon and extremely expensive to discover at 3 a.m.

the scenarios worth running#

Start with the boring ones. Exotic failure modes are fun and the common ones are what actually happens.

The database primary fails over. Does the application reconnect? How long? Does anything need a manual restart?

A dependency returns 500s. Not down — erroring. Does your circuit breaker work? Does your retry policy make it worse?

A dependency gets slow. Harder than down and much more common. Does your timeout fire? Do connections exhaust? Does the slowness propagate to callers?

Disk fills up. On the database host, on the log host, on the application host. This is the single most common self-inflicted outage.

A bad deploy. Deploy something broken to staging and time the rollback. Not the theoretical rollback — the actual one, executed by the actual on-call person.

Certificate expiration. Set one to expire in staging. Watch what happens. Most teams discover their monitoring does not cover this.

The person who knows is unavailable. Run a game day where the subject matter expert is explicitly not allowed to help. This finds the knowledge concentration problems that nothing else does.

how to run one that works#

Announce it. Do not surprise people. Surprise exercises generate resentment and teach people to distrust the process. Everyone should know it is a drill.

Staging first, production eventually. Staging finds most of the runbook problems. Production finds the ones that only exist because staging is not production, which are real and are the ones that matter most.

Have a stop condition. A named person who can call it off, and a defined way to revert whatever you broke.

Write down every friction point, including small ones. "It took four minutes to find the right dashboard" is a real finding.

Fix things within a week. A game day that produces a list nobody acts on is theater, and the second one will have lower attendance.

Do it quarterly. Systems change. A runbook validated a year ago is a runbook that has not been validated.

the cultural part#

The purpose is to find gaps in the system, not gaps in people.

If someone cannot resolve the scenario, that is a finding about documentation, tooling, or access — not about them. Say this explicitly before you start, and mean it, or people will optimize for looking competent rather than for surfacing problems.

The best outcome of a game day is a long list of things that went wrong, discovered by people who were not stressed, on a schedule, with time to fix them.