The queues you did not know you had

You know about the message queue, because you chose it and it has a dashboard. The queues that hurt are the ones nobody named.

Anywhere a fixed-size resource is shared by more requests than it has capacity, there is a queue. It has a depth, a wait time, and a failure mode, and almost none of them are instrumented.

the inventory#

The connection pool. Twenty connections, forty concurrent requests: twenty requests are waiting. Pool wait time is the single most under-measured latency component in web applications, and it is invisible in a database query timer because the clock starts after the connection is acquired.

The thread pool or worker pool. Same shape. Requests queue for a worker, and the time spent waiting is not attributed to any handler.

The TCP accept backlog. The kernel holds connections your process has not accepted yet. Overflow silently drops them, and the client sees a timeout with no server-side trace at all.

The HTTP client's per-host connection limit. Most clients cap concurrent connections per destination. Exceed it and your requests queue in the client, before any network activity, invisible to server-side metrics on both ends.

The DNS resolver. A limited number of in-flight lookups with a cache that can stampede on expiry.

The disk queue. Storage devices have a queue depth. Exceed it and I/O waits.

The garbage collector. Not a queue exactly, but the same behaviour: work that accumulates and is paid in a burst.

Rate limiters. A limiter that delays rather than rejecting is a queue with an enforced service rate.

why this matters more than it sounds#

Little's Law: L = λW. Items in the system equals arrival rate times time in system. Rearranged, wait time is queue length over service rate.

That means your queue depth choice is a latency choice, whether or not you made it deliberately. A connection pool with a 30-second acquisition timeout is a promise that some requests will wait 30 seconds.

And these queues compose. A request waits for a worker, then waits for a connection, then waits for the disk. Each is modest; the sum is the p99 nobody can explain, because each layer's own metrics look fine.

how to find them#

Measure acquisition, not just use. Time the wait for the resource separately from the work done with it:

t0 = time.perf_counter()
with pool.acquire() as conn:
    t1 = time.perf_counter()
    result = conn.execute(query)
span.set_attribute("db.pool_wait_ms", (t1 - t0) * 1000)
span.set_attribute("db.query_ms", (time.perf_counter() - t1) * 1000)

Two numbers instead of one. The first one is the one you did not have, and it is frequently the larger.

Add up your span times. If the parent span is 400 ms and the children total 180 ms, the missing 220 ms is queueing somewhere. That gap is the most useful signal in a trace and almost nobody looks for it.

Check netstat -s for accept queue overflows. A non-zero and growing "listen queue overflowed" counter means you are dropping connections before your application sees them.

Load test past capacity deliberately. Push to 3× and watch which metric degrades first. That is your binding queue, and you will not find it at normal load.

the rule#

Every queue needs three things, and most have none:

1. A bounded size. Unbounded is a memory leak with a friendly name.
2. A wait metric — specifically the age of the oldest waiter, which tells you how far behind you are in time rather than in count.
3. A decision about overflow. Reject, drop, or block. Pick one deliberately, because the default is usually "queue forever," and queueing forever means serving requests whose callers gave up ten seconds ago.

Do that for the queues you chose. Then go find the six you did not.