The cost of a 500

A 0.1% error rate sounds like nothing. It is a rounding error on a dashboard, it is well inside almost any SLO, and it is the kind of number that gets a green tick in a weekly review.

Run the arithmetic on it and the picture is different.

the arithmetic#

Ten million requests a day at 0.1% is ten thousand failed requests. If a user session averages twenty requests, that is roughly five hundred sessions with at least one failure — every day.

But the failures are not evenly spread, and that is the part that matters. Errors cluster: by endpoint, by tenant, by client version, by region, by input shape. A 0.1% global rate is frequently one customer at 40% and everyone else at 0.001%.

For that customer, your service is broken. Your dashboard says 99.9%.

the slices that reveal it#

Aggregate error rate is nearly useless on its own. The same number means completely different things depending on the distribution beneath it, and you cannot tell which without slicing:

By customer or tenant. The single highest-value slice in any multi-tenant system. One broken integration hides perfectly in a global average.
By endpoint. A 0.1% average across forty endpoints can be 4% on the one that matters.
By client version. Errors concentrated in one mobile build means a bad release, not a server problem.
By region. Often a networking or dependency issue, not application logic.
By authenticated vs anonymous. Frequently a different code path entirely.

If your telemetry cannot answer "which customers saw errors in the last hour," that is the gap to close before adding any more dashboards.

the errors that are not 500s#

Counting HTTP status codes undercounts real failure, sometimes badly:

200 with an empty result because a downstream timed out and the code swallowed it. The most dangerous category — invisible to every status-based metric.
200 with a partial result. The page rendered, three of the eight widgets are missing.
Client-side failures that never reach your server at all: a bad bundle, a CSP violation, a network drop mid-request.
Slow enough to be a failure. A request that succeeds after 45 seconds has failed as far as the user is concerned, and it is counted as a success.
The retry that worked. The user saw a spinner for eight seconds. Your success rate is 100%.

The fix is measuring outcomes rather than responses. Did the order get placed? Did the message send? Those are answerable from your own data and they count the failures that status codes miss.

what a single failure actually costs#

Worth being concrete, because "0.1%" abstracts it away:

The user retries. If it fails again, some fraction leave.
Some fraction contact support. Each of those is real money and an engineer's attention.
Some fraction never come back, and you will not attribute that to today.
If it is a payment or a submission, someone may not know whether it went through — which generates duplicates, which generates a different class of problem.

None of that appears on the error-rate graph, which is why the graph being green is not the same as things being fine.

the practical program#

Alert on the slice, not the aggregate. "Any single tenant above 5% for ten minutes" catches things the global rate never will.

Sample real failures and read them. Not the counts — the actual requests, with their inputs and their traces. An hour a week reading real failed requests teaches you more about your system than any dashboard.

Put a request ID on the error page. So a user's complaint becomes a trace lookup rather than an investigation.

Track outcomes for your critical journeys. Checkout completed, message delivered, deploy finished. That number is the one to put in the weekly review, not the status-code ratio.

the framing that changes behaviour#

Stop saying "we're at 99.9%."

Start saying "about five hundred people had a broken session yesterday."

Same data. Completely different meeting.