Randomness, and the bugs you cannot reproduce

The worst bugs are the ones that happen sometimes. They resist the standard debugging loop entirely, because the loop requires reproduction and reproduction is exactly what you do not have.

Most engineers approach these by staring at code and hoping. There is a better method.

the sources of nondeterminism#

There is a finite list. Work through it.

Concurrency. Two things running at once with insufficient ordering. The largest category by far. Includes: unsynchronized shared state, check-then-act races, lost updates, and the classic where two requests both check "does this exist" and both create it.

Time. Anything that depends on wall clock: timeouts, expiry, scheduled work, date boundaries. These fail at midnight, at month end, on the DST transition, on leap day, and when NTP adjusts the clock backward.

Ordering. Hash map iteration order, filesystem directory order, unordered message delivery, parallel test execution. Code that accidentally depends on an order that is not guaranteed works until it does not.

External state. A cache that is sometimes warm. A connection that is sometimes pooled. A DNS response that sometimes returns a different IP. A downstream that is sometimes slow.

Resource exhaustion. Memory pressure changes GC timing, which changes interleaving. Connection pool exhaustion changes behavior under load. File descriptor limits. These make other latent bugs appear.

Actual randomness. UUIDs, load balancing, sampling, retries with jitter, partitioning by hash.

Uninitialized memory or undefined behavior, in languages that permit it.

the method#

1. Instrument before you theorize.

You cannot reproduce it, so you must capture it. Add logging around the suspicious area — not "entering function," but the actual values, the timing, the thread or task identity, the state.

The instinct is to avoid adding logging to production. Add it. A bug you cannot reproduce is a bug you must observe in the wild, and that requires observation.

2. Find the correlation.

You have occurrences. What do they have in common?

• Time of day. (Points at scheduled work, or peak load.)
• Specific tenant or user. (Points at data-dependent behavior.)
• Specific host or region.
• Specific client version.
• Load level.
• Whether some other event happened first.

This is why wide structured events matter. If every request logs its context, the correlation is a query. Without it, you are guessing.

3. Make it more likely.

Once you have a hypothesis, try to increase the failure rate:

• Suspect a race? Add a sleep in the window you think is unprotected. If the failure rate goes from 0.1% to 90%, you found it.
• Suspect ordering? Randomize the order deliberately. Run tests with randomized seeds and shuffled execution.
• Suspect load-related? Load test with the specific pattern.
• Suspect time? Set the clock. Run at 23:59:59. Run on 29 February.

Making a rare bug common is the single most effective technique in this whole category, and it is underused because it feels like making things worse.

4. Add an assertion.

If you believe an invariant holds, assert it. In production, with an alert.

assert order.total == sum(i.price * i.qty for i in order.items), \
    f"order {order.id} total mismatch: {order.total}"

You will find out that it does not hold, and you will find out with the context attached rather than downstream where the symptom appears.

The window between "the invariant broke" and "someone noticed the symptom" is where the information lives, and assertions collapse it to zero.

5. Bisect it.

If it started recently, git bisect still works on statistical failures — you just need a test script that runs the operation many times and reports failure if the rate exceeds a threshold. Slower than a deterministic bisect and far better than reading diffs.

the prevention#

Make things deterministic where you can. Inject the clock rather than calling it. Inject the random source with a seed. Sort collections before iterating where order matters. Deterministic systems have bugs you can reproduce, which means bugs you can fix.

Run tests in randomized order, with a printed seed. If a test only passes in a specific order, it has a hidden dependency, and that dependency is a bug in your production code more often than people assume.

Use your language's race detector. Go's -race, thread sanitizer for C and C++, Java's concurrency tooling. These find real races that have never yet manifested. Run them in CI.

Prefer immutability. A value that cannot change cannot be changed concurrently. This eliminates the largest category of nondeterminism structurally.

Log the identifiers. Trace ID, request ID, tenant. Nine tenths of the difficulty in these investigations is being unable to correlate events you already recorded.

the hard truth#

Some of these take weeks. A concurrency bug that occurs once per million requests in a system with a subtle ordering dependency is genuinely difficult, and no method makes it easy.

The method makes it tractable: instead of staring and hoping, you have a list of candidate sources, a way to gather evidence, and a way to raise the failure rate until it is reproducible.

That is the difference between a bug you eventually fix and one that sits in the backlog for two years labeled "cannot reproduce."