Caching is the only optimization that reliably works

Most performance work is picking up pennies. Rewrite the hot loop, get 15%. Switch the serialization format, get 20%. Move to a faster language, get 3x and spend a quarter doing it.

Caching gets you 1000x, and it does it by not doing the work at all.

the hierarchy#

Every layer is faster than the one below and every layer has a different invalidation problem. Roughly:

The wins are largest at the top and the invalidation is hardest at the bottom. That trade-off is the entire discipline.

the part everyone skips#

Measure the hit rate. Not once — continuously, as a first-class metric with an alert.

A cache with a 50% hit rate is not half as good as one with 99%. It is dramatically worse than that, because the misses dominate the latency distribution:

99% hit rate:  0.99 × 1ms + 0.01 × 100ms  =  1.99 ms average
50% hit rate:  0.50 × 1ms + 0.50 × 100ms  = 50.5  ms average

25x worse. And your p99 is entirely determined by the miss path regardless of the hit rate, which is why profiling under normal load tells you nothing about it.

I have seen caches deployed with no hit rate metric, running at 20%, considered a success because latency improved slightly. Somebody added memory pressure, eviction went up, and the "improvement" evaporated silently.

the failure mode nobody plans for#

Cache stampede. A popular key expires. A thousand concurrent requests all miss simultaneously. All thousand hit the database. The database falls over. The cache never repopulates because nothing succeeds. Your outage is now self-sustaining.

Three fixes, use all of them:

1. Request coalescing. One request per key populates; the rest wait on it. singleflight in Go, Lock around the miss path elsewhere.
2. Jittered TTLs. Never expire a thousand keys at the same instant. Add random ±10%.
3. Serve stale while revalidating. Return the expired value immediately and refresh in the background. Almost always the right choice; almost nobody does it by default.

invalidation#

Phil Karlton's line about the two hard problems is quoted constantly and rarely acted on. The practical version:

Prefer TTL to explicit invalidation. A TTL is a bounded staleness guarantee you can reason about. Explicit invalidation is a distributed systems problem where every write path must know every cache key it affects, forever, including the ones added next year.

If you must invalidate explicitly, use key versioning. Do not delete — change the key.

user:1234:v7:profile

Bump the version on write. Old keys age out naturally. No delete storm, no race between invalidate and repopulate, no partially-invalidated state.

Never cache without a TTL. "It will be invalidated when it changes" is how you get a value from 2023 in production and no way to find out.

the one that is not optional#

Cache the expensive read-mostly thing closest to where it is used. If you do nothing else in a performance push, find the query that runs on every request, returns nearly-identical data, and takes 40 ms. Put it in memory with a 60-second TTL.

That single change is worth more than a month of algorithmic optimization on most web applications, and it takes an afternoon.

when not to cache#

Correctness-critical data with strict consistency requirements. Account balances, inventory at the point of sale, permission checks. The staleness window is a correctness bug, not a performance trade.

Cache the permission lookup if you must, with a short TTL and an explicit invalidation on revoke — and understand that you have accepted a window during which a revoked user still has access. Sometimes that is fine. Decide deliberately, and write down which one you chose.