The distribution is always the same: three dashboards carry all the traffic and forty have not been opened in a year.

why they accumulate#

Dashboards get created during incidents. Someone needs to see a specific correlation right now, builds a view, the incident ends, and the dashboard stays — specific to a problem that has been fixed, named something like "orders debug 2," owned by nobody.

Nothing removes them, so they accumulate until finding the useful one requires knowing its name in advance.

the three that earn their place#

The service dashboard. One per service, same layout for every service, so that anyone can read any service's dashboard without orientation. Rate, errors, duration, saturation. Six to eight panels, above the fold, no scrolling.

The consistency is the point. When every service looks the same, an engineer responding to an unfamiliar service already knows where to look.

The journey dashboard. One per critical user journey — checkout, signup, search. Not per service: per outcome, end to end, across every service involved.

This is the one most organisations lack, and it is the one that answers the question that matters during an incident: are users able to do the thing? Every service being green while checkout is broken is a common and confusing state, and only a journey view shows it.

The capacity dashboard. Resource headroom over weeks, not minutes. Reviewed on a schedule, not during incidents. This is the only one meant for planning rather than response.

what makes a panel useful#

A threshold drawn on it. A line with no reference is a shape. A line with "target: 200ms" drawn across it is information. Every latency and error panel should show what "fine" is.

Comparison to last week. Most values are meaningless in isolation. 4,000 requests per minute — is that high? The same panel with last week's line overlaid answers instantly.

A unit and a direction. Label it. State whether up is good. This sounds trivial and half of all panels fail it.

Percentiles, not averages, for anything latency-shaped. And show p50 next to p99: the gap between them is often the real signal.

Fewer panels. A dashboard with forty panels is a data dump. During an incident nobody scrolls. Eight panels that fit on a screen beat forty that do not.

the annotations that make it work#

The highest-value dashboard feature is the least used: deploy markers.

Vertical lines on every time series showing when a deploy happened. Most tools support it and most teams have not wired it up.

With them, "did this start after the deploy?" is answered by looking. Without them it is answered by opening another tab, finding the deploy log, and correlating timestamps by hand — during an incident, under pressure.

Add config changes and feature flag flips to the same annotation stream and you have covered the causes of most self-inflicted incidents.

the maintenance rule#

Delete dashboards nobody opens. Quarterly, using the view counts your tool already records. If someone misses one, they can rebuild it, and the rebuild will be better because it will reflect the current system.

Give every dashboard an owner and a purpose in its description. "Used by on-call to triage checkout failures" tells the next person whether to trust it. An unlabelled dashboard is one nobody dares delete and nobody quite believes.

the test#

During your next incident, notice which dashboard you actually opened first.

That is your real service dashboard. Whether it is the one you designated as such, and how much of what you needed was on it, is the most honest review of your monitoring you will ever get — and it costs nothing but paying attention for five minutes while you are already there.

The real list is usually fifteen things, and the difference between the two lists is where the surprising outages come from.

the full inventory#

For one service, one request path:

• Datastores. Primary, replicas, cache, search index, object storage.
• Internal services it calls synchronously.
• Third-party APIs — payments, email, geocoding, feature flags, auth.
• Infrastructure control planes. Service discovery, DNS, the secret manager, the config service. These are the ones nobody lists and they are the ones that take everything down.
• Identity. The token issuer, the JWKS endpoint it fetches keys from.
• Observability. If your telemetry agent blocks when the collector is down — and some do — it is in your request path.
• The container registry, at start-up. Fine until you need to scale during an incident and cannot pull an image.
• Certificate infrastructure, including OCSP and the ACME endpoint.
• NTP. Rarely, spectacularly.

The pattern: the dependencies people list are the ones they call. The ones that hurt are the ones they need in order to start, authenticate, or scale.

the two questions per dependency#

For each item, two answers, written down:

What happens when it is slow? Slow is worse than down and much more common. Down fails fast; slow holds your connections, exhausts your pool, and turns one dependency's bad day into your outage.

What happens when it is unavailable? Three possible answers, and only one is a decision:

• We fail. Fine, if the dependency is genuinely essential — a database for a write path.
• We degrade. Serve stale cache, hide the feature, use a default. This is the answer for most non-essential dependencies and it requires code that usually does not exist yet.
• We do not know. This is the real answer for most dependencies, and it is the finding.

the classification that matters#

Sort every dependency into two buckets:

Hard — the request genuinely cannot be served without it. Should be a very short list. Each one caps your maximum availability at its own.

Soft — the request can be served in degraded form. Every soft dependency needs a timeout, a fallback, and a circuit breaker, or it is a hard dependency that nobody has admitted to.

The arithmetic is the reason this matters: five hard dependencies at 99.9% each gives you at best 99.5%, before any failure of your own. If your availability target is higher than that, some of those dependencies have to become soft, and that is an engineering project rather than a target you can declare.

the feature flag trap#

Worth naming specifically because it catches good teams. Feature flag services are adopted as a safety mechanism — a way to turn things off when they go wrong.

Then the flag SDK becomes a hard dependency on the request path, and when the flag service has an outage your service does too. The tool you adopted to make incidents smaller has made one bigger.

The fix is standard for this whole category and rarely applied: cache flag values locally, evaluate from the cache, refresh in the background, and ship a default in the binary. Then the flag service being down means flags are stale, which is survivable.

how to draw it#

Do not buy anything. Open a file:

orders-api
  HARD  postgres-primary        write path; no fallback
  HARD  auth-jwks               cached 1h, so 1h of grace
  SOFT  redis                   cache; miss → primary, 3x latency
  SOFT  inventory-svc           timeout 300ms → show "check availability"
  SOFT  pricing-svc             timeout 200ms → list price, no promo
  SOFT  flags-svc               local cache + baked defaults
  BOOT  vault                   secrets at start; running pods unaffected
  BOOT  registry                image pull; blocks scale-up only

Twenty minutes per service. The BOOT category — needed to start but not to serve — is the one that catches people out during incidents, because those dependencies are invisible right up until you need to replace a pod.

Then, for every SOFT line, check that the fallback described actually exists in code. That check is where the real findings are, and it is usually not the timeout that is missing but the fallback behind it.

The result is systems where the client gives up at 10 seconds, the server keeps working for 60, and the database holds a lock for 300.

the inventory#

For a single HTTP request, in rough order:

Two things stand out. "Often none" appears five times — most application-level timeouts are unset by default. And the values are not coordinated with each other in any way.

the rule that fixes most of it#

Timeouts must decrease as you go deeper.

client 10s
  └─ load balancer 9s
       └─ application 8s
            └─ downstream call 3s (with 1 retry → 6s worst case)
                 └─ connection acquire 1s
                      └─ database statement 2s

Each layer must be shorter than its caller, with room for retries. If an inner layer can outlast its caller, the caller gives up while the work continues — which means you are burning capacity on results nobody will receive. Under load, that is most of your capacity.

deadline propagation#

The better version of the rule: do not configure each timeout independently. Pass the deadline down.

// caller has 10s; every downstream inherits what remains
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()

// 3s into the request, this call gets at most 7s, automatically
resp, err := downstream.Fetch(ctx, id)

Go's context, gRPC's deadlines and equivalents elsewhere exist for this. Once you propagate deadlines, a slow first step automatically shortens the budget for later ones, and work never continues past the point where anyone is waiting.

This is the single highest-value change available in most service codebases, and it is usually a few days of threading a parameter through.

the ones people forget#

Lock wait timeouts. A transaction waiting on a row lock with no timeout waits forever. Set lock_timeout in Postgres, innodb_lock_wait_timeout in MySQL.

DDL statements. A migration that cannot get its lock will queue behind a long transaction — and everything else queues behind it. SET lock_timeout = '3s' before DDL is the line that prevents a large share of migration incidents.

Idle-in-transaction. A connection that opened a transaction and went away holds locks and blocks vacuum indefinitely. idle_in_transaction_session_timeout is the safety net.

Client-side connect vs. read. These are different, and most libraries let you set them separately. A short connect timeout with a long read timeout is usually what you want.

DNS. Rarely configured, occasionally the whole problem.

write them down#

One table, in the repository, listing every timeout in the request path with its current value and where it is configured.

The exercise takes an afternoon and reliably finds at least one inversion — an inner layer waiting longer than the outer one — and at least two places where the value is a framework default nobody chose.

That document then becomes something you can review when latency changes, rather than a set of numbers scattered across six config files and three languages.

the failure mode this prevents#

Without coordinated timeouts, a slow dependency does not degrade your service — it exhausts it. Requests pile up waiting on something their callers abandoned long ago, connections stay held, the pool empties, and healthy requests start failing for want of a connection.

That is a full outage caused by one slow downstream, and the difference between it and a brief latency blip is entirely whether the numbers were coordinated.

Anywhere a fixed-size resource is shared by more requests than it has capacity, there is a queue. It has a depth, a wait time, and a failure mode, and almost none of them are instrumented.

the inventory#

The connection pool. Twenty connections, forty concurrent requests: twenty requests are waiting. Pool wait time is the single most under-measured latency component in web applications, and it is invisible in a database query timer because the clock starts after the connection is acquired.

The thread pool or worker pool. Same shape. Requests queue for a worker, and the time spent waiting is not attributed to any handler.

The TCP accept backlog. The kernel holds connections your process has not accepted yet. Overflow silently drops them, and the client sees a timeout with no server-side trace at all.

The HTTP client's per-host connection limit. Most clients cap concurrent connections per destination. Exceed it and your requests queue in the client, before any network activity, invisible to server-side metrics on both ends.

The DNS resolver. A limited number of in-flight lookups with a cache that can stampede on expiry.

The disk queue. Storage devices have a queue depth. Exceed it and I/O waits.

The garbage collector. Not a queue exactly, but the same behaviour: work that accumulates and is paid in a burst.

Rate limiters. A limiter that delays rather than rejecting is a queue with an enforced service rate.

why this matters more than it sounds#

Little's Law: L = λW. Items in the system equals arrival rate times time in system. Rearranged, wait time is queue length over service rate.

That means your queue depth choice is a latency choice, whether or not you made it deliberately. A connection pool with a 30-second acquisition timeout is a promise that some requests will wait 30 seconds.

And these queues compose. A request waits for a worker, then waits for a connection, then waits for the disk. Each is modest; the sum is the p99 nobody can explain, because each layer's own metrics look fine.

how to find them#

Measure acquisition, not just use. Time the wait for the resource separately from the work done with it:

t0 = time.perf_counter()
with pool.acquire() as conn:
    t1 = time.perf_counter()
    result = conn.execute(query)
span.set_attribute("db.pool_wait_ms", (t1 - t0) * 1000)
span.set_attribute("db.query_ms", (time.perf_counter() - t1) * 1000)

Two numbers instead of one. The first one is the one you did not have, and it is frequently the larger.

Add up your span times. If the parent span is 400 ms and the children total 180 ms, the missing 220 ms is queueing somewhere. That gap is the most useful signal in a trace and almost nobody looks for it.

Check netstat -s for accept queue overflows. A non-zero and growing "listen queue overflowed" counter means you are dropping connections before your application sees them.

Load test past capacity deliberately. Push to 3× and watch which metric degrades first. That is your binding queue, and you will not find it at normal load.

the rule#

Every queue needs three things, and most have none:

1. A bounded size. Unbounded is a memory leak with a friendly name.
2. A wait metric — specifically the age of the oldest waiter, which tells you how far behind you are in time rather than in count.
3. A decision about overflow. Reject, drop, or block. Pick one deliberately, because the default is usually "queue forever," and queueing forever means serving requests whose callers gave up ten seconds ago.

Do that for the queues you chose. Then go find the six you did not.

Four conventions fix most of it, and none of them need a tool.

1. one channel, created at declaration#

Not the team channel. Not a thread in the team channel. A dedicated channel per incident, named predictably: inc-2026-08-29-checkout-errors.

Why it matters: the incident becomes searchable as a unit, the timeline is the channel, and people who join late can read from the top instead of asking what happened. It also means the incident does not drown the normal channel, and the normal channel does not drown the incident.

Archive it afterwards rather than deleting it. It is the primary source for the postmortem.

2. named roles, stated out loud#

Three, at minimum, posted in the channel as the first message:

IC: @priya       — decides, delegates, does not debug
Comms: @marcus   — status page, stakeholders, customer support
Ops: @sam        — hands on keyboard

The incident commander not debugging is the rule people resist and the one that matters most. The moment the IC opens a terminal, nobody is tracking the whole picture, and the incident gets longer.

For a small incident one person can hold two roles. They should still say which ones, because the alternative is everyone assuming someone else is doing comms.

3. a pinned status message, edited in place#

One message, pinned, rewritten as things change:

STATUS 14:22 — Checkout failing for ~12% of users since 13:58.
Cause: unknown. Suspect the payments deploy at 13:55.
Now: rolling back that deploy (@sam), ETA 5 min.
Impact: card payments only, wallet payments unaffected.
Next update: 14:35

Five lines: what is broken, since when, what we think, what we are doing, when the next update comes.

This single convention removes most of the noise, because it answers the question that generates the noise — "what's the current state?" — without anyone having to ask. Everyone joining reads the pin instead of scrolling.

Always include the next-update time. It is what stops people asking for updates.

4. mark the speculation#

The most common way incidents go wrong is that a guess gets repeated until it becomes the working theory, and then twenty minutes go into the wrong system.

A one-word convention fixes it:

FACT: error rate went from 0.1% to 12% at 13:58:20
FACT: the payments deploy completed at 13:55:41
GUESS: the deploy caused it
ACTION: rolling back to confirm

Facts have evidence attached. Guesses are labelled as guesses. Actions say who is doing them.

It looks pedantic for about four minutes and then it saves the incident, because somebody reading the channel can tell the difference between what is known and what somebody said.

what to keep out#

Speculation from people who are not investigating. Well-meant and it fills the channel that responders are trying to read. If you are not on a role, watch.

"Is it fixed yet?" The pinned status has the next update time.

Root-cause analysis during the incident. Restore first. The good question at 14:22 is "what makes this stop", not "why did this happen". Why is a question for Thursday.

Blame, in any form, including jokes. It is in a permanent record that a person will read afterwards.

the handoff#

Long incidents cross shift boundaries and the handoff is where context dies. It needs to be explicit and in the channel:

HANDOFF 22:00 — @priya → @dan (IC)
Ruled out: deploy (rolled back, no change), CDN (unaffected regions also failing)
Current theory: connection pool exhaustion on the orders DB
In flight: @sam is capturing pg_stat_activity every 30s → thread above
Not yet tried: failover to replica
Customers: status page updated 21:40, support has the template

Ruled out, current theory, in flight, not tried, customer state. Five lines, and the incoming IC starts with the accumulated knowledge instead of rediscovering it.

why bother#

The channel is not a side effect of the incident. During the incident it is the coordination mechanism, and afterwards it is the only complete record of what was known and when.

A channel that follows these four conventions produces a postmortem that half writes itself, and — more importantly — an incident that ends sooner, because the people fixing it spent their attention on the system rather than on the chat.

Run the arithmetic on it and the picture is different.

the arithmetic#

Ten million requests a day at 0.1% is ten thousand failed requests. If a user session averages twenty requests, that is roughly five hundred sessions with at least one failure — every day.

But the failures are not evenly spread, and that is the part that matters. Errors cluster: by endpoint, by tenant, by client version, by region, by input shape. A 0.1% global rate is frequently one customer at 40% and everyone else at 0.001%.

For that customer, your service is broken. Your dashboard says 99.9%.

the slices that reveal it#

Aggregate error rate is nearly useless on its own. The same number means completely different things depending on the distribution beneath it, and you cannot tell which without slicing:

• By customer or tenant. The single highest-value slice in any multi-tenant system. One broken integration hides perfectly in a global average.
• By endpoint. A 0.1% average across forty endpoints can be 4% on the one that matters.
• By client version. Errors concentrated in one mobile build means a bad release, not a server problem.
• By region. Often a networking or dependency issue, not application logic.
• By authenticated vs anonymous. Frequently a different code path entirely.

If your telemetry cannot answer "which customers saw errors in the last hour," that is the gap to close before adding any more dashboards.

the errors that are not 500s#

Counting HTTP status codes undercounts real failure, sometimes badly:

• 200 with an empty result because a downstream timed out and the code swallowed it. The most dangerous category — invisible to every status-based metric.
• 200 with a partial result. The page rendered, three of the eight widgets are missing.
• Client-side failures that never reach your server at all: a bad bundle, a CSP violation, a network drop mid-request.
• Slow enough to be a failure. A request that succeeds after 45 seconds has failed as far as the user is concerned, and it is counted as a success.
• The retry that worked. The user saw a spinner for eight seconds. Your success rate is 100%.

The fix is measuring outcomes rather than responses. Did the order get placed? Did the message send? Those are answerable from your own data and they count the failures that status codes miss.

what a single failure actually costs#

Worth being concrete, because "0.1%" abstracts it away:

• The user retries. If it fails again, some fraction leave.
• Some fraction contact support. Each of those is real money and an engineer's attention.
• Some fraction never come back, and you will not attribute that to today.
• If it is a payment or a submission, someone may not know whether it went through — which generates duplicates, which generates a different class of problem.

None of that appears on the error-rate graph, which is why the graph being green is not the same as things being fine.

the practical program#

Alert on the slice, not the aggregate. "Any single tenant above 5% for ten minutes" catches things the global rate never will.

Sample real failures and read them. Not the counts — the actual requests, with their inputs and their traces. An hour a week reading real failed requests teaches you more about your system than any dashboard.

Put a request ID on the error page. So a user's complaint becomes a trace lookup rather than an investigation.

Track outcomes for your critical journeys. Checkout completed, message delivered, deploy finished. That number is the one to put in the weekly review, not the status-code ratio.

the framing that changes behaviour#

Stop saying "we're at 99.9%."

Start saying "about five hundred people had a broken session yesterday."

Same data. Completely different meeting.

the four options#

There are only four. Every system picks one, explicitly or by accident.

1. Buffer. Queue the excess.

Works for bursts. Fails for sustained overload, because a buffer is a delay, and an unbounded buffer is a memory leak with a friendly name. The failure mode is that memory grows until the process dies, taking the buffer with it — so you lose everything, at the worst possible moment.

2. Drop. Discard the excess.

Correct more often than people are comfortable with. Metrics, logs, telemetry, non-critical events — dropping 5% of samples under load is fine and dying is not.

The requirement: know that you dropped, and how much. Silent drops are how you get a dashboard that looks healthy while data is missing.

3. Block. Make the producer wait.

This is real backpressure. The consumer's slowness propagates upstream, the producer slows down, and the system reaches equilibrium at the consumer's rate.

Correct for internal pipelines where the producer can wait. Dangerous when the producer is a user-facing request handler, because now user requests are blocked on a background process.

4. Reject. Tell the producer no.

The right answer at a service boundary. A 429 or 503 returned in one millisecond is much better than a request that waits thirty seconds and then times out, because the caller can make a decision — retry later, degrade, or tell the user.

the wrong default#

Most systems buffer by default, unboundedly, without anyone deciding.

• An in-memory list that grows.
• A queue with no maximum length.
• A connection pool that queues waiters forever.
• A channel with a very large capacity, which is unbounded in practice.

The failure mode is always the same: latency grows, memory grows, and then the process dies. And the requests you were holding were abandoned by their callers ten seconds earlier, so all that work was for nothing.

A queue that is always full is not a buffer. It is a delay you cannot see.

the practical rules#

Every queue has a maximum size. Every one. Choose the number by asking: how long should a request be willing to wait? Multiply by the consumer's rate. That is your queue depth.

If you cannot answer that question, the queue is not designed.

Prefer rejecting to queueing at the edge. When a request arrives and the system is saturated, reject fast. The caller has a timeout; use it as your budget.

Propagate deadlines. If the caller has 200 ms left, every downstream operation should know that. Work performed after the caller has given up is pure waste, and under overload it is the majority of the work being done.

ctx, cancel := context.WithTimeout(ctx, remaining)
defer cancel()

Shed by priority, not randomly. Under load, serve health checks, serve authenticated users, serve the critical path. Shed background work, analytics, and prefetches. Random shedding means your health checks fail and your orchestrator kills healthy instances, which is a self-inflicted outage.

**Measure queue age, not depth.** Depth tells you how many. Age tells you how far behind. "The oldest item has been waiting four minutes" is actionable in a way that "there are 30,000 items" is not.

the pattern that ties it together#

Little's Law: L = λW. Items in the system equals arrival rate times time in system.

Rearranged: wait time equals queue length divided by service rate.

If your queue holds 10,000 items and you process 100 per second, the newest item waits 100 seconds. That is not a hypothetical — it is arithmetic, and it means your queue length choice is your latency choice, whether or not you framed it that way.

Pick the latency you can accept, multiply by the service rate, and that is your maximum queue length. Everything past it gets rejected.

the test#

Point a load generator at your service at three times its capacity. Watch:

• Does memory grow without bound?
• Does latency grow without bound?
• Does anything get rejected, or does everything just get slower?
• After you stop the load, how long until it recovers?

That last one is the important one. A system that takes twenty minutes to recover from a two-minute overload has a backpressure problem, and it will turn a small incident into a large one on a day you did not choose.

the failure mode#

A downstream service slows down. Every caller times out. Every caller retries.

The downstream now receives double its normal traffic while already struggling. More requests time out. More retries. The load multiplies with every round.

The original problem might have been a five-second blip. The retry storm keeps the service down for twenty minutes, and it stays down after the original cause is resolved because the queued retries are still arriving.

This is metastable failure and retries are its most common cause.

the rules#

1. Only retry idempotent operations.

A retried non-idempotent operation charges the card twice. If you need to retry a write, make it idempotent first with an idempotency key.

2. Only retry retryable errors.

A 400 will be a 400 next time. Retrying it wastes a request and delays the error the caller needs to see.

The one people get wrong: 500 is ambiguous. It might be transient, it might be a deterministic bug. Retrying it once is usually reasonable; retrying it five times is usually pointless.

3. Exponential backoff with jitter. Always jitter.

Without jitter, all your clients retry at the same moments, and you have built a synchronized load generator.

def delay(attempt, base=0.1, cap=30):
    return random.uniform(0, min(cap, base * (2 ** attempt)))

That is full jitter, and it is the recommended default. It spreads retries across the whole interval, which is what you want. Half jitter — d/2 + random(0, d/2) — is a reasonable alternative when you want a guaranteed minimum delay.

The version without jitter is the one everyone writes first and it is the one that causes the storm.

4. Cap the attempts and cap the total time.

Three attempts, usually. And a total deadline — if the caller is going to give up after two seconds, retrying at three seconds is pure waste and it is load on a struggling service.

Propagate the deadline. If your caller has 500 ms left, your retry budget is 500 ms, not your configured default.

5. Do not retry at every layer.

This is the one that produces the shocking numbers. Three attempts at the HTTP client, three in the service wrapper, three in the caller, three at the gateway: 3⁴ = 81 requests for one logical call.

Retry at exactly one layer. Usually the outermost one that has the context to decide. Every other layer fails fast and propagates.

Audit this. Most systems that have grown organically retry at three or four layers and nobody knows.

6. Use a retry budget.

The refinement that actually prevents storms: cap retries as a fraction of total traffic, not per request.

if retries_in_window / requests_in_window > 0.1:
    do_not_retry()

When things are healthy, occasional retries are well under the budget and everything works. When things are broken, the budget is exhausted immediately and retries stop entirely — exactly when they would do the most harm.

This single mechanism converts retries from a failure amplifier into a bounded safety net, and it is not widely implemented.

7. Circuit break.

When a downstream is clearly failing, stop calling it. Fail fast, return a cached or degraded response, and probe occasionally to see if it has recovered.

Three states: closed (normal), open (failing fast), half-open (probing). The half-open state must allow only a trickle — if you send full traffic at a recovering service you will knock it over again.

the server side#

The other half, which is usually forgotten.

Send Retry-After on 429 and 503. Then clients that respect it retry at a time you chose rather than a time they chose.

Shed load rather than queueing it. A request that will time out anyway should be rejected immediately, not queued. Queueing under overload increases latency for everything without increasing throughput, and the requests you eventually serve have often already been abandoned.

Prioritize. Under load, serve health checks and critical paths, shed the rest. An unprioritized overload sheds randomly, which means your health checks fail and your orchestrator kills healthy instances.

the test#

Take your service. Make a downstream dependency return 503 for everything. Watch the request rate at the downstream.

If it goes up by more than a small factor, your retry configuration will cause an outage. You have just not had the trigger yet.

A config file with a wrong value. A feature flag flipped. A generated file that doubled in size. A DNS record. A permission change.

Configuration receives a fraction of the engineering rigor that code does, and it has comparable power to break things.

why it goes wrong#

No type system. A YAML file will happily contain timeout: 30s where the code expects an integer, or enabled: "false" which is a truthy string.

No tests. Nobody writes a test for their config.

No review, or perfunctory review. A config change is "just a value" and gets approved in ten seconds.

Deployed differently from code. Frequently faster, frequently without staging, frequently without a canary. The safety mechanisms built for code deployment routinely do not cover config.

Environment drift. Staging and production differ in ways nobody has enumerated, so staging validates a configuration that is not production's.

No rollback story. "What was this value before" is often unanswerable.

the fixes#

1. Parse and validate at startup, not at use.

The worst failure mode is a config error that manifests three hours later when a rarely-used code path reads a malformed value.

Validate everything at boot. Fail loudly and immediately if anything is wrong.

class Settings(BaseModel):
    database_url: PostgresDsn
    timeout_ms: int = Field(gt=0, le=60_000)
    max_connections: int = Field(ge=1, le=1000)
    feature_new_checkout: bool = False

settings = Settings(**load_config())   # raises at startup, with a clear message

An application that will not start is far better than one that starts and behaves wrongly.

2. Types, with a schema.

Whatever your language, there is a library that turns untyped config into a validated typed object. Use it. This eliminates the entire category of string-that-should-be-a-number bugs.

3. Config changes go through the same pipeline as code.

Version control. Review. Staging. Canary. Rollback.

The argument against is that config changes need to be fast, especially for incident response. That is a real need and the answer is a small explicitly-defined set of emergency levers — kill switches, rate limits — with a fast path, and everything else on the normal pipeline.

Not "all config is fast" because that is how config takes down your service.

4. Validate generated configuration.

If a config file is produced by a program, that program can be wrong. A size check against the previous version, a schema check, a sanity check on record count.

This is the specific failure that has caused several high-profile outages: a generated file changed unexpectedly and propagated globally before anyone looked at it.

5. Make the environment explicit and diff it.

You should be able to answer "how does staging differ from production" with a command. If you cannot, staging is not validating production.

config-diff staging production

6. Log the effective configuration at startup.

Not the file — the resolved values after defaults, overrides, and environment variables are applied. Redact secrets. This is the single most useful thing for debugging "it works on my machine," because the effective config is frequently not what anyone thinks it is.

7. Feature flags need the same discipline as code.

Owner. Expiry. Test both branches. Log the flag state on every event. A flag flip is a production change and should be treated as one.

the hierarchy that works#

Most systems end up with layered configuration and the layering should be explicit and simple:

defaults in code
  ← config file
    ← environment variables
      ← command line flags

Later overrides earlier. Log which layer each effective value came from when debugging.

Keep the layers few. Systems with six overlapping sources of configuration — defaults, file, environment, a service, a database table, a flag system — produce values nobody can trace. Each layer you add makes "why is this value what it is" harder to answer.

secrets#

Not the same thing as configuration and should not be in the same place.

• Never in the config file. Never in version control. Never in the image.
• A secret manager, injected at runtime.
• Rotated on a schedule, and the rotation must be tested.
• Never logged. Redact by default at the logger, not at each call site — because somebody will forget.

the general point#

Configuration is the input to your program that is most likely to be wrong and least likely to be checked.

Every rigor you apply to code — types, validation, tests, review, staged rollout, rollback — applies to configuration, and applying it costs a day of setup.

The reason it does not happen is that config does not feel like code. It is data, and data feels safe.

It is not data. It is the arguments to your program, and passing wrong arguments to a program is how programs go wrong.

Most engineers approach these by staring at code and hoping. There is a better method.

the sources of nondeterminism#

There is a finite list. Work through it.

Concurrency. Two things running at once with insufficient ordering. The largest category by far. Includes: unsynchronized shared state, check-then-act races, lost updates, and the classic where two requests both check "does this exist" and both create it.

Time. Anything that depends on wall clock: timeouts, expiry, scheduled work, date boundaries. These fail at midnight, at month end, on the DST transition, on leap day, and when NTP adjusts the clock backward.

Ordering. Hash map iteration order, filesystem directory order, unordered message delivery, parallel test execution. Code that accidentally depends on an order that is not guaranteed works until it does not.

External state. A cache that is sometimes warm. A connection that is sometimes pooled. A DNS response that sometimes returns a different IP. A downstream that is sometimes slow.

Resource exhaustion. Memory pressure changes GC timing, which changes interleaving. Connection pool exhaustion changes behavior under load. File descriptor limits. These make other latent bugs appear.

Actual randomness. UUIDs, load balancing, sampling, retries with jitter, partitioning by hash.

Uninitialized memory or undefined behavior, in languages that permit it.

the method#

1. Instrument before you theorize.

You cannot reproduce it, so you must capture it. Add logging around the suspicious area — not "entering function," but the actual values, the timing, the thread or task identity, the state.

The instinct is to avoid adding logging to production. Add it. A bug you cannot reproduce is a bug you must observe in the wild, and that requires observation.

2. Find the correlation.

You have occurrences. What do they have in common?

• Time of day. (Points at scheduled work, or peak load.)
• Specific tenant or user. (Points at data-dependent behavior.)
• Specific host or region.
• Specific client version.
• Load level.
• Whether some other event happened first.

This is why wide structured events matter. If every request logs its context, the correlation is a query. Without it, you are guessing.

3. Make it more likely.

Once you have a hypothesis, try to increase the failure rate:

• Suspect a race? Add a sleep in the window you think is unprotected. If the failure rate goes from 0.1% to 90%, you found it.
• Suspect ordering? Randomize the order deliberately. Run tests with randomized seeds and shuffled execution.
• Suspect load-related? Load test with the specific pattern.
• Suspect time? Set the clock. Run at 23:59:59. Run on 29 February.

Making a rare bug common is the single most effective technique in this whole category, and it is underused because it feels like making things worse.

4. Add an assertion.

If you believe an invariant holds, assert it. In production, with an alert.

assert order.total == sum(i.price * i.qty for i in order.items), \
    f"order {order.id} total mismatch: {order.total}"

You will find out that it does not hold, and you will find out with the context attached rather than downstream where the symptom appears.

The window between "the invariant broke" and "someone noticed the symptom" is where the information lives, and assertions collapse it to zero.

5. Bisect it.

If it started recently, git bisect still works on statistical failures — you just need a test script that runs the operation many times and reports failure if the rate exceeds a threshold. Slower than a deterministic bisect and far better than reading diffs.

the prevention#

Make things deterministic where you can. Inject the clock rather than calling it. Inject the random source with a seed. Sort collections before iterating where order matters. Deterministic systems have bugs you can reproduce, which means bugs you can fix.

Run tests in randomized order, with a printed seed. If a test only passes in a specific order, it has a hidden dependency, and that dependency is a bug in your production code more often than people assume.

Use your language's race detector. Go's -race, thread sanitizer for C and C++, Java's concurrency tooling. These find real races that have never yet manifested. Run them in CI.

Prefer immutability. A value that cannot change cannot be changed concurrently. This eliminates the largest category of nondeterminism structurally.

Log the identifiers. Trace ID, request ID, tenant. Nine tenths of the difficulty in these investigations is being unable to correlate events you already recorded.

the hard truth#

Some of these take weeks. A concurrency bug that occurs once per million requests in a system with a subtle ordering dependency is genuinely difficult, and no method makes it easy.

The method makes it tractable: instead of staring and hoping, you have a list of candidate sources, a way to gather evidence, and a way to raise the failure rate until it is reproducible.

That is the difference between a bug you eventually fix and one that sits in the backlog for two years labeled "cannot reproduce."

Three details separate the two outcomes.

detail one: the trace ID must be everywhere#

A trace is only useful if you can find it. Which means the trace ID must appear:

• In every log line, so you can pivot from a log to the trace.
• In the response headers, so a client can report it.
• On the error page, so a user can paste it into a support ticket.
• In your error tracking, so an exception links to its trace.
• In the support tool, so a support engineer can hand it to an engineer.

The workflow that makes tracing valuable: customer reports a problem → support gets the trace ID → engineer opens exactly that request and sees everything.

Without the ID in those places, the workflow is: customer reports a problem → engineer tries to guess which of eight million traces it was.

That single difference determines whether the investment pays off, and it is a plumbing problem rather than a tracing problem.

detail two: spans need attributes, not just timing#

A span that says "database query, 45 ms" is nearly useless. Which query? Which table? How many rows? Was it cached?

with tracer.start_as_current_span("db.query") as span:
    span.set_attribute("db.system", "postgresql")
    span.set_attribute("db.operation", "select")
    span.set_attribute("db.table", "orders")
    span.set_attribute("db.rows_returned", len(rows))
    span.set_attribute("app.tenant_id", tenant)
    span.set_attribute("app.cache_hit", cached)

The attributes are what let you ask questions the waterfall cannot answer: is this slow for one tenant, is it slow when the cache misses, is it slow when the result set is large.

Put your business identifiers on the root span. Tenant, user, plan tier, feature flags, client version. Then you can filter traces by them, which is how you find the pattern rather than the instance.

High cardinality is correct here. This is not a metrics system.

detail three: sample intelligently, keep the interesting ones#

Full sampling is expensive and mostly wasteful — the successful, fast, boring requests are identical to each other.

Tail-based sampling makes the decision after the trace completes, when you know whether it was interesting:

• 100% of traces with an error.
• 100% of traces above a latency threshold.
• A small percentage of everything else, for baseline comparison.
• 100% of traces for a specific customer, when you are debugging that customer.

That last one is worth building explicitly. A flag that says "capture everything for this tenant for the next hour" turns an unreproducible customer report into a solvable problem, and it is the single most useful debugging feature you can add to a multi-tenant system.

the things that waste effort#

Instrumenting everything. Auto-instrumentation gives you spans for every framework operation, most of which are noise. A trace with four hundred spans is not more informative than one with twenty; it is less, because the signal is buried.

Instrument the boundaries — service calls, database queries, external APIs, queue operations — and add manual spans only for genuinely expensive internal operations.

Perfect propagation. You will have services that drop the context: a message queue without header support, a third-party integration, a legacy component. Do not block the rollout on 100% coverage. A trace with a gap is still much better than no trace.

Building dashboards from traces. Traces answer "what happened to this request." Metrics answer "what is happening to all requests." Using traces for aggregate views is expensive and slow. Use both, for what each is good at.

the question tracing answers that nothing else does#

Not "is the system slow" — metrics tell you that, cheaper.

Not "what error happened" — logs tell you that.

**"Why was this specific request slow, and what was different about it?"**

That is the question. It is the question you have during an incident, when one customer is affected and the aggregate metrics look fine. And it is unanswerable without tracing.

If your tracing setup does not make that question easy to answer in under a minute, the setup is the problem, not the concept.

the minimum viable version#

If you are starting from nothing:

1. Adopt OpenTelemetry. It is the standard, the instrumentation libraries are broad, and it keeps you portable across backends.
2. Propagate context across every service boundary.
3. Put the trace ID in every log line and every response header.
4. Add business identifiers to the root span.
5. Tail-sample: all errors, all slow, 1% of the rest.

That is a week of work and it covers most of the value. Everything beyond it is refinement.

From an engineering perspective this is the most demanding recurring event in consumer computing, and almost nothing about how it works gets written up.

the shape of the demand#

Synchronized, not smooth. Streaming traffic for a scheduled match is a step function. Millions of concurrent sessions establish within a two-minute window around kickoff, and the ones that fail to establish are a product failure with no recovery — the user missed the start.

Multi-peak within a session. A goal produces a spike in social traffic, in betting platforms, in messaging, in news sites, and in the streams of people switching from another match. These arrive within seconds of each other and are correlated across completely unrelated companies.

Correlated across the industry. This is the part that is genuinely unusual. Every CDN, every mobile network, every payment processor, and every messaging platform experiences the peak simultaneously. There is no borrowing capacity from a quiet neighbor, because there is no quiet neighbor.

Multi-region with different profiles. Matches in three countries across several time zones means the traffic profile shifts across the tournament in ways that capacity planning must anticipate.

what breaks, historically#

Payment processing at scale. Betting and merchandise platforms see enormous transaction spikes at specific moments. Payment processors are a shared dependency and their capacity is a hard constraint nobody downstream controls.

Mobile networks in venue areas. Sixty thousand people in one place, all trying to upload video. This is a well-understood problem with an expensive solution (temporary cell capacity) and it still degrades.

Authentication systems. Everyone logs in at once. Login is frequently the least scaled part of a streaming stack because it is not on the hot path during normal operation.

The thundering herd on recovery. Something fails, comes back, and every client reconnects simultaneously — causing a second failure. Retry jitter is the single most important line of code in a system like this and it is routinely absent.

Ad insertion. Server-side ad insertion personalizes streams per viewer while keeping segments cacheable. It is a hard constraint and it is a disproportionate source of live-stream incidents.

what the good operators do#

Pre-scale, do not autoscale. Autoscaling responds to load after it arrives. Instance startup is measured in tens of seconds; the demand arrives in one. For a known event at a known time, capacity is provisioned in advance and the autoscaler is a safety net, not the mechanism.

Load shed by feature, not by user. When capacity is short, disable the recommendations, the comments, the statistics overlay — keep the video. A degraded experience for everyone beats a perfect experience for 80% and nothing for the rest.

Multi-CDN with active steering. Not for capacity alone — for the fact that any single CDN will have a bad region on a given day. A client-side or DNS-level steering layer that measures real performance and shifts traffic is the difference between a degraded minute and an outage.

Rehearse. Group stage matches are the rehearsal for the knockout rounds. The teams that treat early matches as production load tests, with instrumentation and a retrospective after each one, are the ones that survive the final.

A war room with authority. Not a monitoring dashboard — a room with the people who can make decisions, including the decision to turn features off, without an approval chain.

the generalizable lesson#

The interesting property here is demand you cannot smooth, shed, or refuse.

Most systems get to spread load over time, queue it, or degrade gracefully by making users wait. None of those work when the product is a live event — a queue means the user misses the goal.

What is left is: provision for peak, make every layer redundant, degrade by feature rather than by user, and rehearse.

That is expensive, unglamorous, and the only thing that works. It is worth remembering the next time someone proposes autoscaling as the answer to a spike that arrives faster than a machine can boot.

Some capacity you have to already own.

It is also the move that introduces the most subtle failure modes, and the gap between "we added a queue" and "we added a queue correctly" is large.

what it buys#

Latency. The user gets a response when the work is accepted, not when it is done. A checkout that returns in 80 ms and sends the confirmation email asynchronously is a much better product than one that returns in 900 ms.

Absorbing spikes. A queue is a buffer. Traffic that would overwhelm a synchronous system accumulates and drains. This is the difference between a slow period and an outage.

Isolation. If the email provider is down, checkout still works. The messages accumulate and send later.

Retry for free. A failed message goes back on the queue. A failed synchronous call is a user-visible error.

what it costs#

Eventual consistency, everywhere. The user completed checkout and the confirmation has not arrived. The record exists and the search index does not have it. Every asynchronous boundary introduces a window where the system is inconsistent, and your UI has to be honest about it.

Debugging across the boundary. A synchronous stack trace tells you the whole story. An asynchronous failure requires correlating a producer, a broker, and a consumer, possibly hours apart.

Ordering. Most queues do not guarantee it, or guarantee it only within a partition. If your consumer must process events in order, that is a design constraint that reaches back into how you partition.

Duplicate delivery. Almost all queues are at-least-once. Your consumer will receive the same message twice. If that is not safe, you have a bug that appears under load, weeks after launch.

the rules#

1. Consumers must be idempotent. Non-negotiable.

At-least-once delivery means duplicates. The consumer must produce the same result whether it processes a message once or five times.

The usual implementation: a natural idempotency key, and a record of processed keys.

def handle(msg):
    key = msg.idempotency_key
    with tx():
        if already_processed(key):
            return
        do_the_work(msg)
        mark_processed(key)

The mark_processed must be in the same transaction as the work, or you have moved the race rather than eliminated it.

2. Every queue needs a dead letter queue, and someone must watch it.

A message that fails repeatedly must go somewhere. A DLQ nobody monitors is a place where data goes to be silently lost, which is worse than an error, because errors are visible.

Alert on DLQ depth. Not on it being non-zero — on it growing.

3. Retry with backoff and jitter, and cap the attempts.

Immediate retry on a failing downstream is a denial of service you are performing against yourself. Exponential backoff with jitter, a maximum attempt count, then the DLQ.

4. Monitor queue depth and age, not just throughput.

Throughput looks healthy right up until it does not. The metrics that tell you something is wrong:

• Depth — how many messages are waiting.
• Oldest message age — the most useful single metric. If it is growing, your consumers cannot keep up, and you know how far behind you are in time rather than in count.

5. Decide what happens when the queue is full.

It will be. Reject the producer, drop messages, or block? Each is right in different cases and the default is usually wrong for you. An unbounded queue is not a solution; it is a memory leak with extra steps.

6. Keep the payload small and the reference stable.

Put an ID in the message, not the whole object. The consumer fetches current state. This avoids stale data in the message and keeps the broker fast.

The exception: if you need the state as it was when the event occurred, put it in the message deliberately, and say so.

7. Version your message schema from day one.

Producers and consumers deploy independently. Old consumers will see new messages. Include a version field. Make additive changes only, or handle both shapes.

the thing that surprises people#

Queues do not reduce load. They defer it.

If your consumer processes 100 messages per second and you produce 150, you do not have a working system with a buffer. You have a system that is failing slowly, and the queue depth graph is a countdown.

A queue absorbs bursts. It does not fix a sustained capacity deficit, and the failure mode when you use it that way is a queue that grows for six hours and then an incident where you are simultaneously behind and unable to catch up.

Alert on the age, watch the trend, and size the consumers for the sustained rate.

The only way to find out is to run it, and the only good time to run it is when nothing is actually broken.

what a game day is#

A scheduled exercise where you deliberately break something in a controlled way and have the on-call rotation respond as if it were real.

Two hours. A stated scenario. The people who would actually respond. Real tools, real runbooks, real dashboards. Someone taking notes on everything that did not work.

That is the whole practice. It is not chaos engineering in the automated-random sense, though that is a good adjacent practice. It is a rehearsal.

what you find, reliably#

I have never run one of these that did not find at least four of the following:

The runbook references something that does not exist. A dashboard that was renamed, a script in a repository that was archived, an alias nobody has.

Nobody has the access. The runbook says to restart the service. The on-call engineer does not have permission and does not know who does. This is the single most common finding.

The dashboard does not show the thing. You built the alert. You never built the view that tells you what to do about it.

The escalation path is a person, not a rotation. "Ask Sarah." Sarah is on vacation. Sarah left last year.

Nobody knows the customer impact. The service is degraded. Which customers? Which features? Nobody can answer, so nobody can decide how urgent it is.

Recovery has an undocumented step. The service restarts and does not work because a cache must be cleared first, which is knowledge that lives in one person's head.

The communication path is unclear. Who tells the customers? Who updates the status page? Who has the credentials for the status page?

Every one of those is cheap to fix on a Tuesday afternoon and extremely expensive to discover at 3 a.m.

the scenarios worth running#

Start with the boring ones. Exotic failure modes are fun and the common ones are what actually happens.

The database primary fails over. Does the application reconnect? How long? Does anything need a manual restart?

A dependency returns 500s. Not down — erroring. Does your circuit breaker work? Does your retry policy make it worse?

A dependency gets slow. Harder than down and much more common. Does your timeout fire? Do connections exhaust? Does the slowness propagate to callers?

Disk fills up. On the database host, on the log host, on the application host. This is the single most common self-inflicted outage.

A bad deploy. Deploy something broken to staging and time the rollback. Not the theoretical rollback — the actual one, executed by the actual on-call person.

Certificate expiration. Set one to expire in staging. Watch what happens. Most teams discover their monitoring does not cover this.

The person who knows is unavailable. Run a game day where the subject matter expert is explicitly not allowed to help. This finds the knowledge concentration problems that nothing else does.

how to run one that works#

Announce it. Do not surprise people. Surprise exercises generate resentment and teach people to distrust the process. Everyone should know it is a drill.

Staging first, production eventually. Staging finds most of the runbook problems. Production finds the ones that only exist because staging is not production, which are real and are the ones that matter most.

Have a stop condition. A named person who can call it off, and a defined way to revert whatever you broke.

Write down every friction point, including small ones. "It took four minutes to find the right dashboard" is a real finding.

Fix things within a week. A game day that produces a list nobody acts on is theater, and the second one will have lower attendance.

Do it quarterly. Systems change. A runbook validated a year ago is a runbook that has not been validated.

the cultural part#

The purpose is to find gaps in the system, not gaps in people.

If someone cannot resolve the scenario, that is a finding about documentation, tooling, or access — not about them. Say this explicitly before you start, and mean it, or people will optimize for looking competent rather than for surfacing problems.

The best outcome of a game day is a long list of things that went wrong, discovered by people who were not stressed, on a schedule, with time to fix them.

how it happens#

Logs at debug level in production, because someone turned it on during an incident in 2023 and nobody turned it off.

Every metric at every dimension. A counter with five labels, each with a hundred values, is ten billion time series. Cardinality multiplies and the pricing follows.

100% trace sampling because sampling felt like giving something up.

Retention set to the maximum because nobody knew what to pick and the default was generous.

Duplicate pipelines. Logs going to two vendors during a migration that never completed.

Each decision was locally reasonable. The bill is the sum.

the reduction, in order of return#

1. Sample traces intelligently. You do not need every trace. You need:

• All traces that errored.
• All traces above a latency threshold.
• A small percentage of the rest, for baseline.

Tail-based sampling — decide after the trace completes, when you know whether it is interesting — typically cuts volume by 90%+ while keeping essentially all diagnostic value. This is the single largest win available.

2. Fix your cardinality. Find the metrics with the most series. There will be one or two that dominate.

The usual culprits: user ID as a metric label, URL path with IDs in it (/user/12345/profile instead of /user/:id/profile), and error messages as labels.

Metrics are for aggregates. Events are for specifics. A user ID belongs in a structured event where it costs one field, not in a metric label where it multiplies the series count.

3. Drop the logs you never query. Audit what you actually search. Most organizations find that a large fraction of log volume is from a handful of noisy sources nobody has ever looked at — health check logs, framework debug output, successful-request logs that duplicate a metric.

Route those to cheap object storage rather than an indexed store, or drop them.

4. Tier your retention. You do not need 90 days of everything.

• Metrics: long retention, they are small when aggregated.
• Traces: short, days. You investigate recent things.
• Logs: short and hot for search, long and cold in object storage for compliance.

5. Aggregate at the source. Emit a histogram, not a thousand individual timing events. Most agents can pre-aggregate and it moves cost from the vendor to your own process, where it is much cheaper.

what not to cut#

Be careful here, because the failure mode of aggressive cost reduction is finding out during an incident that you deleted the thing you needed.

Keep all errors. Never sample errors. They are rare and they are the whole point.

Keep the request-level events for your critical journeys. One wide structured event per request, with all the context, is the highest value telemetry you have per byte. It is cheaper than the interior logging it replaces.

Keep enough cardinality to slice by customer and region. Aggregate-only metrics hide the failures that matter — "one tenant is completely broken" looks identical to "everyone is slightly slower" in a global average.

the structural fix#

Move from logs to events. The expensive pattern is many log lines per request, each a string, indexed for full-text search.

The cheap pattern is one structured event per unit of work, with everything you know attached as fields, queried by field rather than by text.

Fewer records, more information per record, dramatically cheaper to store and query. This is the change that actually fixes the cost problem rather than trimming it.

Own your pipeline. An open telemetry collector between your services and your vendor lets you filter, sample, aggregate, and route without changing application code, and without being locked into one destination's pricing model.

That is worth setting up before your bill is a problem, because doing it under cost pressure means making decisions in a hurry.

the question to ask quarterly#

For each telemetry stream: when did we last use this to answer a question?

Anything nobody has queried in a quarter is a candidate for deletion or cold storage. Run the audit. The answer is usually uncomfortable and the savings are usually large.

fixed window#

Count requests per fixed interval. Reset the counter at the boundary.

key: user:1234:minute:2026-04-06T14:23
incr, expire 60s, reject if > limit

Pros: trivial to implement, one counter, minimal memory.

Cons: the boundary problem. A client can send the full limit at 14:23:59 and the full limit again at 14:24:00 — double the intended rate, in one second, legally.

Use when: the limit is generous relative to the burst you can absorb, and simplicity matters more than precision. Most internal services.

sliding window log#

Store a timestamp per request. Count the ones inside the window.

Pros: exactly correct. No boundary artifacts.

Cons: memory proportional to the limit times the number of clients. A limit of 10,000 per hour per user across a million users is a lot of timestamps.

Use when: limits are small, precision matters, and client count is bounded. API keys with strict quotas.

sliding window counter#

The practical compromise. Keep counters for the current and previous window, interpolate based on how far into the current window you are.

def allowed(now, limit, window):
    cur_start = now - (now % window)
    elapsed = (now - cur_start) / window
    estimate = prev_count * (1 - elapsed) + cur_count
    return estimate < limit

Pros: two counters per client, no boundary problem in practice, cheap.

Cons: an approximation. Can be slightly wrong at the edges under a very uneven request distribution.

Use when: this is the default. It is what most production rate limiters actually do and it is almost always the right choice.

token bucket#

A bucket holds tokens, refilled at a constant rate up to a maximum. Each request consumes one. Empty bucket, request rejected.

def allowed(bucket, now, rate, capacity):
    bucket.tokens = min(capacity, bucket.tokens + (now - bucket.last) * rate)
    bucket.last = now
    if bucket.tokens >= 1:
        bucket.tokens -= 1
        return True
    return False

Pros: allows bursts up to the bucket capacity while enforcing a long-run average. This matches how real clients behave — idle, then a burst of activity — much better than a strict window.

Cons: two parameters instead of one, and people set them without thinking about what the burst allowance means.

Use when: you want to allow legitimate bursts. Most user-facing APIs. This is the second-best default after sliding window counter and is better when burstiness is expected.

leaky bucket#

A queue drained at a constant rate. Requests enter the queue; overflow is rejected.

Pros: output rate is perfectly smooth, which is what you want when you are protecting a downstream system that cannot handle bursts at all.

Cons: adds latency — requests wait in the queue. Not appropriate for interactive traffic.

Use when: you are shaping traffic to a fixed-capacity downstream, like a third-party API with a hard rate limit, or a legacy system that falls over above a threshold.

the parts everyone gets wrong#

Rate limiting the wrong key. Limiting by IP breaks for users behind NAT and does nothing against a distributed attacker. Limit by authenticated identity where you have one, and by IP only as a coarse pre-auth defense.

Not telling the client anything useful. A 429 with no information is hostile. Send the standard headers:

RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 42
Retry-After: 42

A client that knows when to retry will retry then. A client that does not will hammer you.

No jitter on the client side. If a thousand clients are told to retry in 42 seconds, they all retry at exactly the same moment. Always jitter, and say so in your API documentation.

Rate limiting at the wrong layer. Limiting at the application means the request already consumed a connection, a thread, and possibly a database query. For abuse protection, limit at the edge. For fairness between legitimate users, limit in the application where you know who they are.

Ignoring cost variance. Not all requests are equal. A search that scans a million rows and a health check are both "one request." Weight by cost, or rate-limit expensive endpoints separately, or you will limit the wrong thing.

Global counters in a distributed system. Exact global rate limiting requires coordination on every request, which is a latency and availability problem. The usual answer is per-node limits with the total divided across nodes, accepting some imprecision, or a shared store with local caching and periodic reconciliation.

Decide which imprecision you can live with, deliberately, rather than discovering it during an incident.

the one-line recommendation#

Sliding window counter for fairness, token bucket where bursts are legitimate, limit by authenticated identity, always send Retry-After, always jitter.

what the pages are telling you#

Every page is a statement about the system.

A page that requires a human to restart something says the system cannot recover from a condition it will encounter again. That is automatable and the automation is usually a supervisor and a health check.

A page for a transient issue that resolved itself says your alert threshold is wrong, or your alert lacks a duration condition, or the underlying flakiness needs a retry with backoff.

A page where the runbook is "look at the dashboard and see if it is bad" is not a page. That is a dashboard, and it should be checked during business hours.

A page that only one person can resolve is a knowledge distribution failure and a single point of human failure. The fix is documentation and rotation of the work, not heroism.

A page at 3 a.m. for something that could have waited until 9 a.m. says nobody has classified alerts by urgency. Not everything that is wrong is urgent.

the audit#

Pull the last ninety days of pages. For each one:

• What was the user impact? (Often: none.)
• Could it have been automated?
• Could it have waited?
• Did the runbook exist and was it correct?
• What was the actual fix?

Then categorize:

Should not have paged — no user impact or no urgency. Delete the alert or downgrade it. This is usually a large fraction and deleting alerts is the highest value hour available to most teams.

Should have been automatic — the fix was mechanical. Automate it. If the runbook says "run this command," a computer can run that command.

Genuine incidents — real user impact requiring judgment. These are the ones on-call exists for, and there should not be many.

A healthy rotation is a small number of genuine incidents. If you are getting paged more than a couple of times per week, the problem is not that your systems are complex.

the design changes that reduce pages#

Make things self-healing. Restart on crash. Retry with exponential backoff and jitter. Circuit-break to a degraded mode. Shed load rather than falling over. Each of these converts a page into a metric.

Make degradation graceful and explicit. Decide in advance which features can be disabled. Feature flags that turn off the expensive path let a page become "turn off recommendations, investigate Monday."

Make everything reversible fast. A large fraction of incidents are caused by a deploy. If rollback is one command and takes ninety seconds, the incident is ninety seconds long. If it requires reversing a migration, it is four hours.

Add capacity headroom. Running at 85% utilization to save money means every spike is a page. Headroom is cheaper than the human cost, and much cheaper than the turnover.

the human side#

Pay for it. On-call is work performed outside working hours with real cost to the person's life. Compensate it explicitly — money or time off, and enough that the cost is visible to whoever decides whether to fix the alert noise.

Unpaid on-call means the cost is borne entirely by the person and is invisible to the organization, which guarantees it never improves.

Time off after a bad night. Non-negotiable, automatic, not something the person has to ask for.

Never one person. A primary and a secondary, always. The primary needs to be able to escalate without feeling like a failure.

Rotate wide. If only three people can be on-call, they will burn out and leave, and the knowledge leaves with them.

The person who was paged decides what gets fixed. Give the on-call engineer authority to prioritize the follow-up work. They have the information and the motivation, and nobody else has both.

the metric to track#

Pages per on-call shift, trended over quarters.

If it is flat or rising, nobody is fixing the causes. That is a management failure, not a resilience failure, and the outcome of ignoring it is attrition — which is more expensive than every fix combined.

what "available" usually means#

For most teams it means: a health check endpoint returned 200 within a timeout, from a monitoring location, on a schedule.

That measures whether a process is running. It does not measure whether anyone can do anything.

Real user-visible failures that a health check will not catch:

• The API returns 200 with an empty result set because a downstream cache is cold.
• Requests succeed at p50 and take 30 seconds at p95, and the p95 is entirely one large customer.
• Login works, but only for users whose sessions were created before the deploy.
• Everything works except in one region, which is 8% of traffic and 40% of revenue.
• Writes succeed and are silently not persisted because a queue consumer is wedged.

Every one of those is a full outage for the affected users and none of them move a health-check-based availability number.

what to measure instead#

Measure user journeys, not endpoints. Pick the three to five things users come to your product to do. Checkout. Send a message. Run a query. Deploy. Instrument those end to end, and define success as "the journey completed correctly," not "each service returned 200."

Measure from the client where you can. Server-side metrics cannot see DNS failures, TLS problems, CDN issues, or the user's terrible network. A meaningful fraction of real user-visible failure is invisible from inside your datacenter.

Use success rate, not uptime. The fraction of requests that succeeded, over a window. This handles partial failure correctly, which uptime does not. A service serving 60% of requests successfully is not "up," and a binary metric says it is.

Slice by everything. Aggregate metrics hide the failures that matter. The same success rate can mean "everyone has a slightly bad time" or "one tenant is completely broken," and those require completely different responses.

Slice by: customer, region, client version, endpoint, feature flag. If you cannot slice by these, that is your observability gap.

Set a latency threshold and count violations. "p99 latency" as a number is hard to alert on and easy to game. "Percentage of requests slower than 2 seconds" is a success rate, comparable across time, and directly meaningful. If more than 1% of checkouts take longer than 2 seconds, something is wrong, and you can say that in a sentence a product manager understands.

the SLO framing, without ceremony#

You do not need a formal SLO program. You need three numbers per critical journey:

1. What fraction of attempts must succeed. 99.9%, say.
2. What counts as success. Completed, correct, within a latency bound.
3. Over what window. 28 days, rolling.

The difference between your target and 100% is your error budget. When you are burning it fast, stop shipping and fix reliability. When you have plenty, ship faster and take more risk.

That is the entire value of the framework: it converts "should we prioritize reliability or features" from an argument into an arithmetic question. Everything else in the SLO literature is optional.

alerting on the right thing#

Alert on symptoms, not causes. "Checkout success rate below 99%" pages someone. "CPU above 80%" does not, because high CPU is sometimes fine and low CPU with a broken service is not.

Alert on burn rate, not instantaneous values. A single failed request should not page anyone. Burning a month's error budget in an hour should page everyone. Multi-window burn-rate alerting is the single best improvement most teams can make to their paging, and it dramatically reduces false pages.

Every page should have an action. If the runbook is "look at it and see if it recovers," it is not a page. It is a dashboard.

the test#

Take your last five user-reported incidents. For each, ask: did an alert fire before the user reported it?

If the answer is mostly no, your monitoring is measuring the wrong things, and no amount of additional dashboards will fix that. The gap is not visibility. It is that you are watching the system instead of watching the users.

the failure pattern#

The typical postmortem:

1. Timeline of what happened. Accurate, detailed, useful.
2. Root cause. Usually a single technical fact.
3. Action items. Five to twelve of them.
4. Filed. Two action items get done. The rest age out.

Six months later, a similar incident, with a similar document.

Three things are wrong here.

problem one: "root cause" is singular#

Complex systems do not fail because of one thing. They fail because several conditions aligned, each of which was individually survivable.

The Cloudflare outage in November: a permissions change, a query that returned duplicates, a fixed buffer size, an error path that panicked instead of degrading, and a config pipeline that propagated globally in minutes. Remove any one and it does not happen, or it is much smaller.

Picking one and calling it "the root cause" means you fix one and leave four.

Better framing: contributing factors, plural, each with its own assessment of whether it is worth addressing. Some will not be — that is a legitimate decision if it is made explicitly.

problem two: action items without owners and dates are wishes#

An action item that says "improve monitoring for the config pipeline" with no owner, no date, and no definition of done is a sentence, not a plan.

The fix is unglamorous:

• Every action item has one named person. Not a team. A person.
• Every action item has a date. If nobody will commit to a date, it is not going to happen and you should delete it and say so.
• Every action item has a definition of done that someone else could verify.
• They go in the same backlog as feature work, prioritized against it. An action item in a separate "incident follow-up" list that is never sprint planned is a list of things that will not be done.

And the one that actually forces it: review the open action items at the start of the next postmortem. Nothing motivates completion like a room full of people looking at your undone item from last quarter's incident while discussing this quarter's similar one.

problem three: nobody asks about the near misses#

The incidents you review are the ones that broke through. For every one, there were several that did not — a bad deploy caught by a canary, a config error someone noticed in review, a query that would have taken down the database if it had run on Monday instead of Sunday.

Those contain the same information at a fraction of the cost, and almost nobody collects them.

Add a lightweight channel for it. "I nearly broke prod today, here is how." No document, no meeting, no blame. Just a note. The pattern that emerges over a quarter is more valuable than any individual postmortem.

the questions that produce useful findings#

Replace "what was the root cause" with:

• "What made this hard to detect?" Detection time is usually the largest component of impact and is the most improvable.
• "What made this hard to diagnose?" Usually missing observability. This produces the highest-value action items.
• "What made recovery slow?" Often a missing runbook, a missing kill switch, or a rollback that was not actually tested.
• "Who knew something that would have helped, and why did that not reach the responders?" This is an organizational question and it is frequently the real finding.
• "What did we do that helped?" Genuinely important. Practices that worked should be named so they get kept.

the blameless part, done correctly#

Blameless does not mean nobody made a mistake. It means the analysis focuses on why the mistake was possible and easy, rather than on the person.

"Alice deployed without running the tests" is blame and it is also useless. "The deploy path does not require tests to pass, and the shortcut that skips them is the fastest way to deploy" is the same fact stated in a way you can act on.

If the answer to "why did they do that" is "because the system made it easy and the correct path was hard," you have found something.

If the answer is genuinely "they were careless," you still fix the system, because the next person will also be careless eventually. Humans are the constant; the system is the variable.

The chain of events is a small masterpiece of the genre.

what happened#

A database permissions change caused a query that generates a Bot Management feature configuration file to return duplicate rows. The query had been returning one row per feature; after the permissions change it returned rows from multiple underlying schemas.

The generated file therefore roughly doubled in size.

The proxy that consumes this file preallocates a fixed-size buffer sized against a limit of 200 features — comfortably above the ~60 actually in use. The doubled file exceeded that limit.

The Rust code handling this hit an unrecoverable error path and the proxy panicked rather than degrading. Because the configuration file propagates network-wide every few minutes, the failure propagated network-wide within minutes.

Recovery was complicated by the fact that the bad file kept regenerating and redeploying.

the lessons, which are old#

Configuration is code and needs the same rigor. This was not a code deploy. It was a data change that propagated to production automatically with no staging, no canary, and no validation. Config deployment pipelines are consistently held to a lower standard than code deployment pipelines, and config causes a large share of major outages.

Fixed-size limits need to fail soft. The 200-feature limit was reasonable. Panicking when exceeded was not. The correct behavior for a proxy encountering an oversized config is to log loudly, alert, and continue with the previous known-good version.

Validate generated artifacts before propagation. A size check, a schema check, a sanity check on row count against the previous version — any of these would have caught it. Generated files should be validated as rigorously as user input, because the generator can be wrong.

Blast radius follows deployment speed. Config that propagates globally in minutes is a feature until it propagates a bad config globally in minutes. Staged rollout applies to configuration too.

Have a kill switch for automated pipelines. Much of the recovery time went to stopping the thing that kept redeploying the bad file. Every automated deployment path needs a way to stop it that does not require fixing the underlying problem first.

the Rust note#

This will be used as an argument about Rust, and it should not be.

The panic was a deliberate choice at that call site — the code used a construct that terminates on error rather than propagating it. That is a design decision about error handling, available in any language. In C the equivalent code would have written past the buffer, which is worse.

The actual lesson is about where you choose to make errors fatal. In a proxy handling live traffic, almost nothing should be fatal. Degrade, alert, continue. "Fail fast" is good advice for a batch job and bad advice for a load balancer.

the credit due#

Cloudflare published a detailed technical postmortem within a day, named the specific code path, and did not hide behind "an issue with a third-party provider."

That is how it should be done and it is rarer than it should be. A company that publishes real postmortems earns more trust than one that never has visible incidents, because the second one is not telling you about them.

AWS's public summary attributes the trigger to a latent race condition in the automation that manages DynamoDB's DNS records, which resulted in an empty record set for a regional endpoint and no automatic recovery path.

the cascade#

The failure sequence is the interesting part.

DynamoDB's endpoint became unresolvable. That alone would be bad. What made it a regional event is that an enormous number of AWS's own services use DynamoDB internally. The EC2 instance launch path, IAM's control plane, Lambda's invocation machinery, and dozens of others depend on it.

So the failure propagated: DynamoDB down means new EC2 instances cannot launch, which means autoscaling cannot replace failing capacity, which means load shedding, which means more failures. Network Load Balancer health checks destabilized. The recovery itself was slowed by the backlog of queued work that had accumulated.

This is textbook metastable failure: a system that is stable under normal load and stable under no load, but which, once pushed past a threshold, sustains its own failure through retry amplification and queue buildup even after the original trigger is fixed.

why us-east-1#

It is the oldest region, the largest, and the default in approximately every tutorial ever written. Several global AWS control planes are homed there — IAM, CloudFront configuration, Route 53's control plane, and others. That means a us-east-1 event has global blast radius even for customers with no resources in the region.

That architecture is a historical artifact. It is also extremely hard to change now, which is a lesson about early decisions in systems that grow.

the honest customer takeaway#

The reflexive response is "multi-region." Before you spend a year on that, do the arithmetic.

Multi-region active-active is genuinely hard. Data consistency across regions, failover testing that actually works, doubled infrastructure cost, and a substantially more complex system that fails in new ways. Many organizations that attempt it end up with a system that is less reliable overall because the complexity introduces more failure modes than the regional risk it removes.

The dependency you cannot escape. If your multi-region architecture depends on a global control plane that lives in us-east-1, you did not achieve independence. Check this specifically. A lot of people discovered it yesterday.

What is actually worth doing, in order:

1. Know your dependencies. Most teams cannot enumerate what their service requires to start. Write it down. The exercise is revealing.
2. Static stability. Design so existing capacity keeps serving when the control plane is unavailable. If your service needs to call an API to keep running, it will stop when that API stops. Cache aggressively, fail open where safe, and do not require a control plane call on the request path.
3. Graceful degradation. Decide in advance which features can be turned off. A checkout that works without recommendations is much better than a site that is down.
4. Exponential backoff with jitter, and circuit breakers. Retry storms are what turns an incident into an outage. This is the single highest-leverage code change available.
5. Multi-region for the tier that genuinely warrants it. Which is usually not everything.

the industry-level observation#

A meaningful fraction of the world's software depends on a small number of regions operated by a small number of companies. That concentration produces excellent reliability most of the time and correlated failure occasionally.

There is no individual fix. Every company independently choosing the most reliable provider produces exactly this concentration. It is a collective action problem, and the only actors who can address it are regulators thinking about systemic risk, who are — belatedly — starting to.

2025-06-02 14:23:11 INFO  Processing request

Processing which request? For whom? Started when? Finished when? Did it succeed? This line costs money to produce, store, index, and retain, and it conveys approximately zero bits of information.

Multiply by four hundred million a day.

the mental model that fixes this#

Stop thinking of logs as a narrative of what your program did. Think of them as structured events about units of work, emitted once, with everything you know attached.

The canonical form: one event per request, per job, per message consumed. Wide. Structured. Emitted at the end when you know how it went.

{
  "event": "http_request",
  "trace_id": "4bf92f3577b34da6",
  "method": "POST",
  "route": "/api/orders",
  "status": 201,
  "duration_ms": 143,
  "user_id": "u_8812",
  "org_id": "org_44",
  "db_queries": 7,
  "db_ms": 89,
  "cache_hits": 3,
  "cache_misses": 1,
  "upstream_ms": 22,
  "region": "us-east-1",
  "version": "2025.06.02-a1b2c3d",
  "feature_flags": ["new_checkout", "fast_path"]
}

One line. Now you can answer questions you did not think to ask when you wrote it:

• Which org has the slowest p99 on this route?
• Are requests with new_checkout enabled making more database queries?
• Did latency change after the deploy of a1b2c3d?
• Is the cache miss rate correlated with the slow requests, or is that coincidence?

None of those are answerable by grepping "Processing request."

the rules#

Log at the boundary of a unit of work, not at every step inside it. Interior logging is what tracing is for. If you need step-level visibility, emit spans, not log lines.

Never log a string you have to parse later. "user " + id + " failed" means somebody writes a regex. Put id in a field.

Attach identity to everything. Trace ID, user, org, request ID. The single most common debugging failure is having the information and being unable to correlate it.

High cardinality is the point. The advice to avoid high-cardinality fields comes from metrics systems, where cardinality multiplies storage. Events are not metrics. user_id in an event is exactly what makes it useful, and any observability tool that cannot handle it is the wrong tool.

Log the decision, not the branch. Not "entering fast path". Rather "fast_path": true on the one event.

Errors get context, not just stack traces. A stack trace tells you where. The fields tell you which input, which user, which version, which flag combination. Where is the easy part.

what this costs#

Fewer lines, each larger. In practice, storage usually goes down, because you delete the interior noise. Query performance improves dramatically because you are filtering structured fields rather than doing full-text search.

The real cost is discipline at write time, and one afternoon to set up a logger that makes the structured path the easy path. If emitting a field requires more typing than emitting a string, people will emit strings.

the test#

Pick an incident from the last quarter. Ask: could I have diagnosed that from logs alone, without adding new logging and redeploying?

If the answer is no — and it usually is — that is the gap. The information you needed was in the process's memory at the time and you did not write it down.

Write it down.

Deployment is a solved problem that organizations keep re-opening because the solution is boring and boring does not get anyone promoted.

the properties that matter#

Strip away the tooling debates and a good deploy has four properties.

It is the same every time. Not "mostly the same, except for the database migration, and except on Fridays, and except when Sarah does it." The same. If there is a manual step, it is in the pipeline as an approval gate, not in someone's head.

It is reversible in under a minute. This is the one people get wrong. They build elaborate progressive rollouts and then discover that rolling back requires reverting a migration that dropped a column. Reversibility is a design constraint on your changes, not a feature of your deploy tool.

Failure is detected without a human looking. If the way you find out about a bad deploy is a customer email, you do not have a deployment process, you have a deployment ritual.

It happens often enough to be unremarkable. The failure rate of a deploy scales superlinearly with the amount of change in it. Ten small deploys are dramatically safer than one deploy containing ten changes, and this is the single most robust finding in the entire delivery-metrics literature.

the expand-contract discipline#

Most rollback pain is schema pain. The fix is a discipline, not a tool:

1. Expand. Add the new column, nullable. Deploy. The old code ignores it.
2. Migrate. Backfill. Dual-write from application code. Deploy. Both shapes work.
3. Switch. Read from the new column. Deploy. Still reversible — the old column is intact.
4. Contract. Drop the old column, days or weeks later, once you are certain.

Four deploys instead of one. Every intermediate state is reversible. It feels slow and it is the reason you do not have a 3 a.m. incident where the rollback made things worse.

flags are not free#

Feature flags decouple deploy from release and that is genuinely valuable. They also multiply your state space: n flags means 2^n possible configurations, and you are testing approximately one of them.

Rules that keep this manageable:

• Every flag gets an owner and an expiry date at creation.
• A flag that has been at 100% for a month is deleted, not left "just in case."
• Flags never nest. If flag A only matters when flag B is on, you have made something unreasonable.
• Kill-switch flags are a separate category with separate rules and they can live forever.

The failure mode is flag debt: a codebase with two hundred flags where nobody knows which combinations are actually exercised in production. That is worse than no flags, because it looks like safety.

the actually unfashionable opinion#

Most teams do not need Kubernetes to deploy well, and many teams running Kubernetes deploy worse than they did before, because the platform's complexity consumed the attention that used to go into the process.

If your deploys are fast, boring, and reversible, your infrastructure is correct regardless of what it is. If they are not, no amount of platform will fix it, because the problem is that nobody has treated the deploy as a product with a user.

The user is your team at 2 a.m. Design for them.