Backpressure is the concept your system is missing
When a fast producer meets a slow consumer, something has to give. Deciding what, in advance, is the whole discipline.
#reliability
17 pieces tagged reliability,
from March 10, 2025 to July 3, 2026.
Retries: a complete guide to not making it worse
The most common way a small incident becomes a large one is a retry policy written without thinking about aggregate behavior.
Configuration is the most under-designed part of your system
It has no type system, no tests, no review, and it causes a disproportionate share of outages.
Randomness, and the bugs you cannot reproduce
The bug that happens once a week and never in staging. A systematic approach to the class of problem everyone handles badly.
Distributed tracing that people actually use
Most tracing deployments produce beautiful waterfalls nobody opens. The difference is three implementation details.
The World Cup is the largest load test ever run
A month of synchronized global demand across three countries, sixteen cities, and every streaming platform at once.
The queue is the architecture
Most scaling problems are solved by making something asynchronous. Most reliability problems are caused by doing it badly.
Run the incident before the incident
Game days, failure injection, and the specific reason your untested runbook is wrong.
Observability costs more than the thing it observes
At some point your telemetry bill exceeded your compute bill and nobody noticed. Here's how to fix it without going blind.
Rate limiting: the four algorithms and when each is wrong
Token bucket, leaky bucket, fixed window, sliding window. They fail differently and the differences matter.
On-call is a design problem
If your rotation is painful, that's information about your architecture, not about your people's resilience.
Your monitoring is measuring the wrong nines
Availability percentages are a compliance artifact. Here's what to measure if you want to know whether users are having a bad time.