Retries: a complete guide to not making it worse

Retries are the most commonly implemented and most commonly wrong piece of resilience engineering. A retry policy that seems obviously correct in isolation is frequently the mechanism that turns a brief degradation into a full outage.

the failure mode#

A downstream service slows down. Every caller times out. Every caller retries.

The downstream now receives double its normal traffic while already struggling. More requests time out. More retries. The load multiplies with every round.

The original problem might have been a five-second blip. The retry storm keeps the service down for twenty minutes, and it stays down after the original cause is resolved because the queued retries are still arriving.

This is metastable failure and retries are its most common cause.

the rules#

1. Only retry idempotent operations.

A retried non-idempotent operation charges the card twice. If you need to retry a write, make it idempotent first with an idempotency key.

2. Only retry retryable errors.

A 400 will be a 400 next time. Retrying it wastes a request and delays the error the caller needs to see.

The one people get wrong: 500 is ambiguous. It might be transient, it might be a deterministic bug. Retrying it once is usually reasonable; retrying it five times is usually pointless.

3. Exponential backoff with jitter. Always jitter.

Without jitter, all your clients retry at the same moments, and you have built a synchronized load generator.

def delay(attempt, base=0.1, cap=30):
    return random.uniform(0, min(cap, base * (2 ** attempt)))

That is full jitter, and it is the recommended default. It spreads retries across the whole interval, which is what you want. Half jitter — d/2 + random(0, d/2) — is a reasonable alternative when you want a guaranteed minimum delay.

The version without jitter is the one everyone writes first and it is the one that causes the storm.

4. Cap the attempts and cap the total time.

Three attempts, usually. And a total deadline — if the caller is going to give up after two seconds, retrying at three seconds is pure waste and it is load on a struggling service.

Propagate the deadline. If your caller has 500 ms left, your retry budget is 500 ms, not your configured default.

5. Do not retry at every layer.

This is the one that produces the shocking numbers. Three attempts at the HTTP client, three in the service wrapper, three in the caller, three at the gateway: 3⁴ = 81 requests for one logical call.

Retry at exactly one layer. Usually the outermost one that has the context to decide. Every other layer fails fast and propagates.

Audit this. Most systems that have grown organically retry at three or four layers and nobody knows.

6. Use a retry budget.

The refinement that actually prevents storms: cap retries as a fraction of total traffic, not per request.

if retries_in_window / requests_in_window > 0.1:
    do_not_retry()

When things are healthy, occasional retries are well under the budget and everything works. When things are broken, the budget is exhausted immediately and retries stop entirely — exactly when they would do the most harm.

This single mechanism converts retries from a failure amplifier into a bounded safety net, and it is not widely implemented.

7. Circuit break.

When a downstream is clearly failing, stop calling it. Fail fast, return a cached or degraded response, and probe occasionally to see if it has recovered.

Three states: closed (normal), open (failing fast), half-open (probing). The half-open state must allow only a trickle — if you send full traffic at a recovering service you will knock it over again.

the server side#

The other half, which is usually forgotten.

Send Retry-After on 429 and 503. Then clients that respect it retry at a time you chose rather than a time they chose.

Shed load rather than queueing it. A request that will time out anyway should be rejected immediately, not queued. Queueing under overload increases latency for everything without increasing throughput, and the requests you eventually serve have often already been abandoned.

Prioritize. Under load, serve health checks and critical paths, shed the rest. An unprioritized overload sheds randomly, which means your health checks fail and your orchestrator kills healthy instances.

the test#

Take your service. Make a downstream dependency return 503 for everything. Watch the request rate at the downstream.

If it goes up by more than a small factor, your retry configuration will cause an outage. You have just not had the trigger yet.