The dashboard nobody looks at
Most dashboards are built during an incident and never opened again. A few are worth keeping, and they look different.
#reliability
23 pieces tagged reliability,
from March 10, 2025 to September 28, 2026.
The graph nobody drew
Your service dependency graph exists whether or not anyone has written it down. Writing it down takes an afternoon.
Timeouts: every one of them
A request crosses a dozen components with a timeout each, and almost nobody has ever added them up.
The queues you did not know you had
Every fixed-size resource is a queue. Most of them are unmonitored, and that is where latency hides.
What a good incident channel looks like
The chat log is where the incident is actually run. Four conventions make it useful rather than a wall of noise.
The cost of a 500
Error rates get tracked as a percentage. Users experience them as a specific person having a specific bad day.
Backpressure is the concept your system is missing
When a fast producer meets a slow consumer, something has to give. Deciding what, in advance, is the whole discipline.
Retries: a complete guide to not making it worse
The most common way a small incident becomes a large one is a retry policy written without thinking about aggregate behavior.
Configuration is the most under-designed part of your system
It has no type system, no tests, no review, and it causes a disproportionate share of outages.
Randomness, and the bugs you cannot reproduce
The bug that happens once a week and never in staging. A systematic approach to the class of problem everyone handles badly.
Distributed tracing that people actually use
Most tracing deployments produce beautiful waterfalls nobody opens. The difference is three implementation details.
The World Cup is the largest load test ever run
A month of synchronized global demand across three countries, sixteen cities, and every streaming platform at once.
everything else tagged #reliability 11
- The queue is the architectureessay
- Run the incident before the incidentessay
- Observability costs more than the thing it observesessay
- Rate limiting: the four algorithms and when each is wrongessay
- On-call is a design problemessay
- Your monitoring is measuring the wrong ninesessay
- The postmortem that changes somethingessay
- Cloudflare falls over because of a config filenews
- us-east-1 goes down and takes a large chunk of the internet with itnews
- Your logs are a product and you are shipping a bad oneessay
- In defense of the boring deployessay