On-call is a design problem

Bad on-call is treated as a fact of life, a personal endurance test, or a staffing problem. It is none of those. It is a measurement of your system's design, delivered directly to your team's sleep.

what the pages are telling you#

Every page is a statement about the system.

A page that requires a human to restart something says the system cannot recover from a condition it will encounter again. That is automatable and the automation is usually a supervisor and a health check.

A page for a transient issue that resolved itself says your alert threshold is wrong, or your alert lacks a duration condition, or the underlying flakiness needs a retry with backoff.

A page where the runbook is "look at the dashboard and see if it is bad" is not a page. That is a dashboard, and it should be checked during business hours.

A page that only one person can resolve is a knowledge distribution failure and a single point of human failure. The fix is documentation and rotation of the work, not heroism.

A page at 3 a.m. for something that could have waited until 9 a.m. says nobody has classified alerts by urgency. Not everything that is wrong is urgent.

the audit#

Pull the last ninety days of pages. For each one:

• What was the user impact? (Often: none.)
• Could it have been automated?
• Could it have waited?
• Did the runbook exist and was it correct?
• What was the actual fix?

Then categorize:

Should not have paged — no user impact or no urgency. Delete the alert or downgrade it. This is usually a large fraction and deleting alerts is the highest value hour available to most teams.

Should have been automatic — the fix was mechanical. Automate it. If the runbook says "run this command," a computer can run that command.

Genuine incidents — real user impact requiring judgment. These are the ones on-call exists for, and there should not be many.

A healthy rotation is a small number of genuine incidents. If you are getting paged more than a couple of times per week, the problem is not that your systems are complex.

the design changes that reduce pages#

Make things self-healing. Restart on crash. Retry with exponential backoff and jitter. Circuit-break to a degraded mode. Shed load rather than falling over. Each of these converts a page into a metric.

Make degradation graceful and explicit. Decide in advance which features can be disabled. Feature flags that turn off the expensive path let a page become "turn off recommendations, investigate Monday."

Make everything reversible fast. A large fraction of incidents are caused by a deploy. If rollback is one command and takes ninety seconds, the incident is ninety seconds long. If it requires reversing a migration, it is four hours.

Add capacity headroom. Running at 85% utilization to save money means every spike is a page. Headroom is cheaper than the human cost, and much cheaper than the turnover.

the human side#

Pay for it. On-call is work performed outside working hours with real cost to the person's life. Compensate it explicitly — money or time off, and enough that the cost is visible to whoever decides whether to fix the alert noise.

Unpaid on-call means the cost is borne entirely by the person and is invisible to the organization, which guarantees it never improves.

Time off after a bad night. Non-negotiable, automatic, not something the person has to ask for.

Never one person. A primary and a secondary, always. The primary needs to be able to escalate without feeling like a failure.

Rotate wide. If only three people can be on-call, they will burn out and leave, and the knowledge leaves with them.

The person who was paged decides what gets fixed. Give the on-call engineer authority to prioritize the follow-up work. They have the information and the motivation, and nobody else has both.

the metric to track#

Pages per on-call shift, trended over quarters.

If it is flat or rising, nobody is fixing the causes. That is a management failure, not a resilience failure, and the outcome of ignoring it is attrition — which is more expensive than every fix combined.