Backpressure is the concept your system is missing

Every system with a producer and a consumer eventually has a moment where the producer is faster. What happens next is either a design decision you made or an emergent behavior you discover during an incident.

the four options#

There are only four. Every system picks one, explicitly or by accident.

1. Buffer. Queue the excess.

Works for bursts. Fails for sustained overload, because a buffer is a delay, and an unbounded buffer is a memory leak with a friendly name. The failure mode is that memory grows until the process dies, taking the buffer with it — so you lose everything, at the worst possible moment.

2. Drop. Discard the excess.

Correct more often than people are comfortable with. Metrics, logs, telemetry, non-critical events — dropping 5% of samples under load is fine and dying is not.

The requirement: know that you dropped, and how much. Silent drops are how you get a dashboard that looks healthy while data is missing.

3. Block. Make the producer wait.

This is real backpressure. The consumer's slowness propagates upstream, the producer slows down, and the system reaches equilibrium at the consumer's rate.

Correct for internal pipelines where the producer can wait. Dangerous when the producer is a user-facing request handler, because now user requests are blocked on a background process.

4. Reject. Tell the producer no.

The right answer at a service boundary. A 429 or 503 returned in one millisecond is much better than a request that waits thirty seconds and then times out, because the caller can make a decision — retry later, degrade, or tell the user.

the wrong default#

Most systems buffer by default, unboundedly, without anyone deciding.

• An in-memory list that grows.
• A queue with no maximum length.
• A connection pool that queues waiters forever.
• A channel with a very large capacity, which is unbounded in practice.

The failure mode is always the same: latency grows, memory grows, and then the process dies. And the requests you were holding were abandoned by their callers ten seconds earlier, so all that work was for nothing.

A queue that is always full is not a buffer. It is a delay you cannot see.

the practical rules#

Every queue has a maximum size. Every one. Choose the number by asking: how long should a request be willing to wait? Multiply by the consumer's rate. That is your queue depth.

If you cannot answer that question, the queue is not designed.

Prefer rejecting to queueing at the edge. When a request arrives and the system is saturated, reject fast. The caller has a timeout; use it as your budget.

Propagate deadlines. If the caller has 200 ms left, every downstream operation should know that. Work performed after the caller has given up is pure waste, and under overload it is the majority of the work being done.

ctx, cancel := context.WithTimeout(ctx, remaining)
defer cancel()

Shed by priority, not randomly. Under load, serve health checks, serve authenticated users, serve the critical path. Shed background work, analytics, and prefetches. Random shedding means your health checks fail and your orchestrator kills healthy instances, which is a self-inflicted outage.

**Measure queue age, not depth.** Depth tells you how many. Age tells you how far behind. "The oldest item has been waiting four minutes" is actionable in a way that "there are 30,000 items" is not.

the pattern that ties it together#

Little's Law: L = λW. Items in the system equals arrival rate times time in system.

Rearranged: wait time equals queue length divided by service rate.

If your queue holds 10,000 items and you process 100 per second, the newest item waits 100 seconds. That is not a hypothetical — it is arithmetic, and it means your queue length choice is your latency choice, whether or not you framed it that way.

Pick the latency you can accept, multiply by the service rate, and that is your maximum queue length. Everything past it gets rejected.

the test#

Point a load generator at your service at three times its capacity. Watch:

• Does memory grow without bound?
• Does latency grow without bound?
• Does anything get rejected, or does everything just get slower?
• After you stop the load, how long until it recovers?

That last one is the important one. A system that takes twenty minutes to recover from a two-minute overload has a backpressure problem, and it will turn a small incident into a large one on a day you did not choose.