The queue is the architecture

The single most effective architectural move available to most systems is: take the slow thing out of the request path and put it in a queue.

It is also the move that introduces the most subtle failure modes, and the gap between "we added a queue" and "we added a queue correctly" is large.

what it buys#

Latency. The user gets a response when the work is accepted, not when it is done. A checkout that returns in 80 ms and sends the confirmation email asynchronously is a much better product than one that returns in 900 ms.

Absorbing spikes. A queue is a buffer. Traffic that would overwhelm a synchronous system accumulates and drains. This is the difference between a slow period and an outage.

Isolation. If the email provider is down, checkout still works. The messages accumulate and send later.

Retry for free. A failed message goes back on the queue. A failed synchronous call is a user-visible error.

what it costs#

Eventual consistency, everywhere. The user completed checkout and the confirmation has not arrived. The record exists and the search index does not have it. Every asynchronous boundary introduces a window where the system is inconsistent, and your UI has to be honest about it.

Debugging across the boundary. A synchronous stack trace tells you the whole story. An asynchronous failure requires correlating a producer, a broker, and a consumer, possibly hours apart.

Ordering. Most queues do not guarantee it, or guarantee it only within a partition. If your consumer must process events in order, that is a design constraint that reaches back into how you partition.

Duplicate delivery. Almost all queues are at-least-once. Your consumer will receive the same message twice. If that is not safe, you have a bug that appears under load, weeks after launch.

the rules#

1. Consumers must be idempotent. Non-negotiable.

At-least-once delivery means duplicates. The consumer must produce the same result whether it processes a message once or five times.

The usual implementation: a natural idempotency key, and a record of processed keys.

def handle(msg):
    key = msg.idempotency_key
    with tx():
        if already_processed(key):
            return
        do_the_work(msg)
        mark_processed(key)

The mark_processed must be in the same transaction as the work, or you have moved the race rather than eliminated it.

2. Every queue needs a dead letter queue, and someone must watch it.

A message that fails repeatedly must go somewhere. A DLQ nobody monitors is a place where data goes to be silently lost, which is worse than an error, because errors are visible.

Alert on DLQ depth. Not on it being non-zero — on it growing.

3. Retry with backoff and jitter, and cap the attempts.

Immediate retry on a failing downstream is a denial of service you are performing against yourself. Exponential backoff with jitter, a maximum attempt count, then the DLQ.

4. Monitor queue depth and age, not just throughput.

Throughput looks healthy right up until it does not. The metrics that tell you something is wrong:

• Depth — how many messages are waiting.
• Oldest message age — the most useful single metric. If it is growing, your consumers cannot keep up, and you know how far behind you are in time rather than in count.

5. Decide what happens when the queue is full.

It will be. Reject the producer, drop messages, or block? Each is right in different cases and the default is usually wrong for you. An unbounded queue is not a solution; it is a memory leak with extra steps.

6. Keep the payload small and the reference stable.

Put an ID in the message, not the whole object. The consumer fetches current state. This avoids stale data in the message and keeps the broker fast.

The exception: if you need the state as it was when the event occurred, put it in the message deliberately, and say so.

7. Version your message schema from day one.

Producers and consumers deploy independently. Old consumers will see new messages. Include a version field. Make additive changes only, or handle both shapes.

the thing that surprises people#

Queues do not reduce load. They defer it.

If your consumer processes 100 messages per second and you produce 150, you do not have a working system with a buffer. You have a system that is failing slowly, and the queue depth graph is a countdown.

A queue absorbs bursts. It does not fix a sustained capacity deficit, and the failure mode when you use it that way is a queue that grows for six hours and then an incident where you are simultaneously behind and unable to catch up.

Alert on the age, watch the trend, and size the consumers for the sustained rate.