Here is where the time actually goes, roughly in order of how often each is the dominant cause.

1. everything talks to a real database#

The most common cause by a wide margin. Each test sets up a database, inserts fixtures, runs, and tears down.

Fixes, in order of return:

Roll back instead of truncating. Wrap each test in a transaction and roll it back. Orders of magnitude faster than deleting rows, and it needs no cleanup code.

Share the schema, not the data. Create the schema once per run, not per test.

Use a tmpfs for the test database. The data does not need to survive; durable writes are pure cost. On Postgres, a data directory in memory plus fsync=off and synchronous_commit=off is dramatically faster and completely inappropriate for anything but tests.

Move the logic out of the database's reach. The deepest fix: if business logic is a pure function of its inputs, its tests do not need a database at all. Tests that are hard to write without infrastructure are telling you about your design.

2. the suite is serial#

Most runners parallelise and most projects have not turned it on, usually because it exposed shared state once and someone reverted it.

That shared state is a bug. Tests that pass only in a particular order have an ordering dependency, and ordering dependencies in tests usually mirror a real one in the code.

Turn on parallelism, fix what breaks, and you will find at least one genuine issue.

3. sleeps#

time.sleep(2)   # wait for the worker to pick it up

Every one of these is pure latency, and they are always tuned to the slowest machine anyone has run on. Fifty of them is a hundred seconds of doing nothing.

Replace with polling on the actual condition:

wait_until(lambda: job.reload().status == "done", timeout=5)

Same worst case, typically a hundredth of the elapsed time, and it fails with a useful message instead of a mysterious assertion.

4. the fixtures are enormous#

A test that needs one user builds an organisation, three teams, forty users and a year of history, because it reuses the "standard" fixture.

Build the minimum. Factories with sensible defaults, overridden per test, beat shared fixture files that grow to serve every case.

5. it is doing real network I/O#

Tests hitting real HTTP endpoints — even internal ones, even mock servers over a socket — pay connection setup and scheduling per call.

Intercept at the client layer rather than the network layer. Most languages have a way to stub the HTTP client in-process, and it is both faster and more deterministic than a local server.

The exception is contract tests, which exist to catch exactly what stubs hide. Keep those, run them separately, and do not let them into the fast suite.

6. everything is an end-to-end test#

Browser-driving tests are two to three orders of magnitude slower than unit tests. A suite made mostly of them is slow no matter what you do.

The ratio that works: a large number of fast unit tests, a moderate number of integration tests around real boundaries, and a small number of end-to-end tests covering the handful of journeys that must never break.

Inverting that ratio is the most expensive testing mistake a team can make, and it is usually the result of not trusting the lower layers rather than a deliberate choice.

the measurement first#

Before fixing anything, get the distribution:

pytest --durations=25
go test ./... -json | jq -r 'select(.Action=="pass") | "\(.Elapsed) \(.Test)"' | sort -rn | head -25

Almost always, a small number of tests dominate. Fixing the slowest twenty is usually the whole job, and it is an afternoon rather than a project.

the target#

Under 10 seconds for the unit suite, run on every save. Under two minutes for everything that gates a merge.

The ten-second number is not arbitrary — it is roughly the threshold past which people stop running tests as they work and start batching. Once that happens you have lost the feedback loop, and the suite's speed stops being a convenience question and starts being a quality one.

Anywhere a fixed-size resource is shared by more requests than it has capacity, there is a queue. It has a depth, a wait time, and a failure mode, and almost none of them are instrumented.

the inventory#

The connection pool. Twenty connections, forty concurrent requests: twenty requests are waiting. Pool wait time is the single most under-measured latency component in web applications, and it is invisible in a database query timer because the clock starts after the connection is acquired.

The thread pool or worker pool. Same shape. Requests queue for a worker, and the time spent waiting is not attributed to any handler.

The TCP accept backlog. The kernel holds connections your process has not accepted yet. Overflow silently drops them, and the client sees a timeout with no server-side trace at all.

The HTTP client's per-host connection limit. Most clients cap concurrent connections per destination. Exceed it and your requests queue in the client, before any network activity, invisible to server-side metrics on both ends.

The DNS resolver. A limited number of in-flight lookups with a cache that can stampede on expiry.

The disk queue. Storage devices have a queue depth. Exceed it and I/O waits.

The garbage collector. Not a queue exactly, but the same behaviour: work that accumulates and is paid in a burst.

Rate limiters. A limiter that delays rather than rejecting is a queue with an enforced service rate.

why this matters more than it sounds#

Little's Law: L = λW. Items in the system equals arrival rate times time in system. Rearranged, wait time is queue length over service rate.

That means your queue depth choice is a latency choice, whether or not you made it deliberately. A connection pool with a 30-second acquisition timeout is a promise that some requests will wait 30 seconds.

And these queues compose. A request waits for a worker, then waits for a connection, then waits for the disk. Each is modest; the sum is the p99 nobody can explain, because each layer's own metrics look fine.

how to find them#

Measure acquisition, not just use. Time the wait for the resource separately from the work done with it:

t0 = time.perf_counter()
with pool.acquire() as conn:
    t1 = time.perf_counter()
    result = conn.execute(query)
span.set_attribute("db.pool_wait_ms", (t1 - t0) * 1000)
span.set_attribute("db.query_ms", (time.perf_counter() - t1) * 1000)

Two numbers instead of one. The first one is the one you did not have, and it is frequently the larger.

Add up your span times. If the parent span is 400 ms and the children total 180 ms, the missing 220 ms is queueing somewhere. That gap is the most useful signal in a trace and almost nobody looks for it.

Check netstat -s for accept queue overflows. A non-zero and growing "listen queue overflowed" counter means you are dropping connections before your application sees them.

Load test past capacity deliberately. Push to 3× and watch which metric degrades first. That is your binding queue, and you will not find it at normal load.

the rule#

Every queue needs three things, and most have none:

1. A bounded size. Unbounded is a memory leak with a friendly name.
2. A wait metric — specifically the age of the oldest waiter, which tells you how far behind you are in time rather than in count.
3. A decision about overflow. Reject, drop, or block. Pick one deliberately, because the default is usually "queue forever," and queueing forever means serving requests whose callers gave up ten seconds ago.

Do that for the queues you chose. Then go find the six you did not.

It is not. There are four rules and four shapes.

the four rules#

The x-axis is not time. This is the rule everyone gets wrong. Left-to-right is alphabetical or arbitrary — it is not chronological. A frame on the left did not happen before a frame on the right.

Width is total time. A frame's width is the proportion of samples that included it. Wide means expensive. That is the entire message.

Height is stack depth. A frame sitting on another means it was called by it. Tall is not bad; tall just means deep call stacks.

Colour is usually meaningless. In most tools it is random, chosen to make adjacent frames distinguishable. Do not read anything into it unless the tool explicitly says otherwise — some use it for language or module.

That is it. Wide is expensive, stacked means called-by, colour is decoration.

the four shapes#

A wide plateau near the top. One function, doing real work, dominating. This is the good case: a clear, single hotspot. Optimise that function or call it less.

A wide plateau near the bottom, narrowing above. The time is spread across many children. There is no single hotspot; the cost is the whole subtree. Look one level up — usually the answer is "call this subtree fewer times" rather than "make some leaf faster".

A staircase. Deep, narrow, repeating structure. Often recursion, sometimes a framework's middleware chain. Each frame is cheap; the depth is the cost. Look for a way to shorten the chain rather than to optimise any frame in it.

Many thin spikes with nothing dominant. Death by a thousand cuts, or your profile is too short. Check the sample count first — a profile of two hundred samples looks like this regardless of the workload. If the sample count is healthy and it still looks like this, the program is genuinely uniform and your wins are architectural rather than local.

the practical workflow#

Profile the thing you care about, under load that resembles production. A profile of a cold process running a synthetic benchmark measures start-up and your benchmark harness.

Profile long enough. Seconds, not milliseconds. You are sampling; you need samples.

Look at the widest frame you did not expect. Not the widest frame — the widest surprising one. Everyone's profile is dominated by something obvious and irreducible. The win is in the frame that has no business being 8% of your runtime.

Check for time you cannot see. A flame graph of CPU samples shows CPU. If your program spends most of its wall clock waiting on a socket, an on-CPU profile will be nearly empty and entirely misleading. That is what off-CPU profiling and tracing are for, and confusing the two is the most common way people conclude a profiler is lying to them.

differential flame graphs#

Underused and excellent. Profile before, profile after, render the difference — frames that got wider are coloured one way, narrower the other.

This turns "did my change help?" from an argument about noisy averages into a picture. It is also the fastest way to find a performance regression introduced between two releases: profile both, diff, look at what turned red.

the honest caveat#

A flame graph tells you where time went. It does not tell you whether that time was necessary, and it will happily show you a perfectly optimised hot loop that should not have been called at all.

The largest performance wins are almost always "stop doing this work" rather than "do this work faster," and no profiler can suggest that. It can only show you where to point the question.

The only mechanism I have seen reliably prevent this is a budget: a number, agreed in advance, that fails the build when exceeded.

why "we should keep it fast" does not work#

Because "fast" has no threshold, so there is never a moment where a specific change is the problem.

Every individual addition is defensible. The library is 12 KB and saves a week. The query is 8 ms and enables a feature. The middleware is 3 ms and improves security.

Ten of those and your page is 400 ms slower, and no single change was wrong. There was no point at which anyone could say no, because the comparison was always "this change versus nothing" rather than "this change versus the budget."

A budget changes the comparison. Now the question is "what are you willing to remove to make room for this," which is a real conversation.

setting the numbers#

Derive them from user-facing outcomes, not from what you currently have.

For a web frontend:

For an API:

Per-layer budgets are the underrated part. A single end-to-end number tells you something regressed. Per-layer budgets tell you where.

total request budget: 200 ms
  auth:        10 ms
  validation:   5 ms
  database:    80 ms
  business:    40 ms
  serialize:   15 ms
  overhead:    50 ms

When the database layer goes to 120 ms, that specific budget fails, and the person who changed the query knows immediately rather than in a month when someone investigates a general slowdown.

This is borrowed directly from game development, where per-subsystem frame budgets have been standard practice for decades.

enforcement#

The budget must fail something, or it is a wish.

In CI, on every pull request:

- name: bundle size
  run: npx size-limit          # fails if over the configured budget

- name: performance test
  run: k6 run --threshold 'http_req_duration{p(99)}<500' load.js

Report the delta on the pull request. "This change adds 8 KB to the main bundle (142 KB → 150 KB, budget 170 KB)." Visible, in context, at the moment of decision.

Allow overrides with a written reason. Not blocking forever — blocking until somebody says why. The record of overrides is itself useful; if you are overriding every week, the budget is wrong or the system is losing.

the budget for query count#

The single most useful backend budget, and the least common.

Count database queries per request in tests. Assert a maximum.

with assert_max_queries(10):
    client.get("/api/orders")

This catches N+1 queries at the moment they are introduced, which is the only cheap time to catch them. An N+1 that reaches production is found weeks later by someone investigating a slow endpoint, and by then it is embedded in an ORM relationship that four other things depend on.

Almost every framework has a way to do this, and almost nobody does.

what happens when you exceed it#

The conversation that a budget forces, in order:

1. Can we make the new thing cheaper? Lazy load it, defer it, make the query better.
2. Can we remove something else? This is the valuable one. It surfaces the feature nobody uses and the library that is doing 5% of what it costs.
3. Should the budget change? Sometimes yes, deliberately, with a reason recorded. A budget that never changes is a budget that will be ignored.
4. Do we not ship this? Rare and it should be available.

Any of those is better than the default, which is that the change lands and the system is permanently slower.

the thing to measure first#

Before setting a budget, get the current numbers and the distribution. p50, p75, p95, p99. On real user hardware and real networks, not on a developer laptop on office wifi.

Then set the budget at roughly where you are, and ratchet it down over time rather than setting an aspirational number you fail immediately.

A budget you exceed on day one gets disabled on day two.

why connections are expensive#

In Postgres specifically, each connection is a separate operating system process with its own memory. A few megabytes of baseline, plus work memory for sorting and hashing, plus its share of shared buffer access.

Four hundred connections means four hundred processes. The scheduler is context switching between them, they are contending for the same locks and buffers, and the memory is largely wasted because most of them are idle.

The counterintuitive result, which has been measured many times: throughput frequently goes down as connection count goes up, past a fairly low threshold.

More connections does not mean more concurrency. It means more contention.

the actual number#

A widely used starting formula:

connections = (core_count × 2) + effective_spindle_count

For an 8-core machine with SSD storage, that is somewhere around 16 to 20.

That number seems shockingly low to people running pools of 100 or more. It is correct, and the reasoning is straightforward: a query is either using CPU or waiting on I/O. You need enough connections to keep the cores busy and to have some work queued behind I/O waits. Past that, additional connections are queued at the database instead of queued in your pool, and queueing at the database is worse because it consumes resources.

Test it. Take your load test, run it at pool sizes of 10, 20, 40, 80, and 160, and plot throughput and p99 latency. The curve rises, flattens, and then degrades. Most people are on the degrading side and have never looked.

the pooler layers#

Three, and they do different things.

Application-side pool. In-process, reuses connections across requests. Every ORM and database driver has one. This is the minimum.

External pooler — PgBouncer, pgcat, or a cloud provider's equivalent. Sits between your application and the database and multiplexes many client connections onto few server connections.

This is what you need when you have many application instances. Twenty instances with a pool of 20 each is 400 connections to the database, even if each instance is mostly idle. A pooler collapses that to the number the database actually wants.

The database's own limit. max_connections. Set it lower than you think — it is a safety valve, and setting it high does not make the database faster, it makes the failure mode worse.

pooling modes, and the one that bites#

External poolers have modes and choosing wrong causes subtle correctness bugs.

Session pooling. A client gets a server connection for the duration of its session. Safe, and provides little multiplexing benefit.

Transaction pooling. A server connection is assigned per transaction and returned after commit. This is where the big multiplexing win is, and it is what most people want.

The catch: anything that depends on session state breaks.

• Prepared statements (unless the pooler supports them explicitly, which newer ones do)
• SET at the session level
• Session-level advisory locks
• LISTEN/NOTIFY
• Temporary tables
• WITH HOLD cursors

If your ORM uses server-side prepared statements by default — many do — you must either disable them or use a pooler that handles them. This is the single most common transaction-pooling problem and it manifests as intermittent errors under load, which is a miserable thing to debug.

Statement pooling. A connection per statement. Maximum multiplexing, breaks multi-statement transactions. Almost never what you want.

the settings that matter#

Beyond size:

Connection timeout. How long a request waits for a connection before failing. This should be short — a few seconds. A request waiting thirty seconds for a connection has already been abandoned by its caller.

Idle timeout. Return connections to the database when not in use. Important with many application instances.

Max lifetime. Recycle connections periodically. This handles the case where a database failover happens and your pool is holding connections to the old primary — without a max lifetime, some pools hold those indefinitely.

Validation. Test a connection before handing it out, or handle the failure on first use. Something has to deal with connections that died while idle.

the diagnostic#

SELECT state, count(*), max(now() - state_change) AS longest
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY state;

What you are looking for:

• Many idle in transaction — the worst state. A transaction is open and doing nothing, holding locks and preventing vacuum. This is an application bug: a transaction opened and not committed, usually because of an early return or an exception path.
• Many idle — pool is oversized. Harmless but wasteful.
• Many active with long durations — queries are slow; the pool is not the problem.

idle in transaction is the one to alert on. It is always a bug and it causes outages that look like database problems and are application problems.

the summary#

Size your pool from a load test, not from a blog post. It is smaller than you think. Use an external pooler in transaction mode if you have many application instances, and check your prepared statement behavior when you do.

Size matters for real reasons: pull time on cold start, registry cost, deployment speed when you are scaling out under load, and attack surface — every package in the image is something that can have a CVE you have to answer for.

the six causes#

1. You shipped the build toolchain.

The compiler, the headers, the package manager cache, the source tree, the test fixtures. All of it needed to build, none of it needed to run.

Multi-stage builds fix this completely:

FROM golang:1.24 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /app ./cmd/server

FROM gcr.io/distroless/static-debian12
COPY --from=build /app /app
ENTRYPOINT ["/app"]

Final image: the binary, plus CA certificates and timezone data. Tens of megabytes.

2. You started from a full distribution image.

FROM ubuntu is roughly 80 MB before you install anything, and it includes a package manager, a shell, and a hundred utilities you will never invoke.

The ladder, from largest to smallest:

• Full distribution — 80 MB+
• -slim variants — 30–80 MB
• Alpine — 5–10 MB, with musl libc, which will occasionally surprise you
• Distroless — just the runtime, no shell, no package manager
• scratch — nothing at all, for static binaries

The Alpine caveat, since it bites people: musl's allocator and DNS resolver behave differently from glibc's. Python performance in particular can be significantly worse, and some binary wheels do not exist for musl. Test rather than assume.

3. Your layers are ordered wrong.

Each instruction creates a layer. A layer is invalidated when it or anything before it changes.

# bad — any source change reinstalls every dependency
COPY . .
RUN npm ci

# good — dependencies are cached until the lockfile changes
COPY package.json package-lock.json ./
RUN npm ci
COPY . .

This does not shrink the final image but it dramatically speeds up builds, which is usually what people actually care about.

4. You deleted things in a later layer.

RUN apt-get install -y build-essential   # layer 1: +400 MB
RUN apt-get remove -y build-essential    # layer 2: marks deleted, image unchanged

Layers are additive. Deleting a file in a later layer hides it and does not remove it. The bytes are still in the image and still transferred on pull.

Everything must happen in one RUN:

RUN apt-get update \
 && apt-get install -y --no-install-recommends build-essential \
 && make \
 && apt-get purge -y build-essential \
 && apt-get autoremove -y \
 && rm -rf /var/lib/apt/lists/*

Better: use a multi-stage build and do not install the toolchain in the final image at all.

5. You have no .dockerignore.

COPY . . copies .git, node_modules, build artifacts, test fixtures, and your local .env.

.git
node_modules
dist
*.log
.env*
**/__pycache__
coverage

The .git directory alone is frequently hundreds of megabytes on a mature repository, and it is in a lot of images.

6. Your dependencies are enormous.

Sometimes it is genuinely the dependencies — machine learning stacks with CUDA libraries are legitimately multiple gigabytes.

Check whether you need the GPU variant. torch with CUDA is roughly 2.5 GB; the CPU build is a fraction of that. If you are serving on CPU, you are shipping GPU libraries for nothing.

finding out where it went#

docker history --no-trunc <image>       # size per layer

Or use a layer inspection tool that shows you which files are in which layer and how much space is wasted. Ten minutes with one of those tells you exactly what to fix.

the security dimension#

Every package in the image is potential CVE surface, and your scanner will report all of them regardless of whether the code is reachable.

A distroless image has almost nothing to report, which means the reports you do get are signal rather than noise. That is worth more than the size reduction — a vulnerability report with three entries gets read; one with four hundred does not.

The trade-off: no shell means you cannot docker exec in to debug. Use ephemeral debug containers that attach to the running pod's namespaces instead, which is a better practice anyway because it means your production image is not a debugging toolkit.

the target#

• Compiled language, static binary: under 30 MB.
• Interpreted with dependencies: under 200 MB.
• Anything over a gigabyte without a machine learning stack: something is wrong and it is one of the six above.

the layers where it pays#

HTTP responses. Everyone does this and most do it badly. Check yours:

curl -sI -H 'Accept-Encoding: br, gzip' https://example.com/api/data | grep -i content-encoding

If that returns nothing, you are shipping uncompressed JSON, and JSON compresses extraordinarily well — commonly 80–90% for typical API responses, because it is mostly repeated key names.

Brotli beats gzip by roughly 15–20% on text at comparable CPU cost, and is supported everywhere. Use it for static assets at maximum level (precomputed, so the CPU cost is paid once) and at a moderate level for dynamic responses.

Database storage. Most modern databases support per-table or per-column compression. On a table of text or JSON, it frequently halves the storage — which also halves the I/O, which means more of the working set fits in memory, which is where the real win is.

The CPU cost of decompression is almost always smaller than the I/O cost you avoided.

Logs and telemetry. Log shipping is often a meaningful fraction of internal network traffic and vendor cost. Compressed batching typically reduces it by an order of magnitude, and the batching itself reduces request overhead.

Backups and object storage. Storage is cheap and it is not free, and egress definitely is not. Compression at rest is a direct cost reduction with no downside for cold data.

Container images. Zstandard-compressed layers pull faster than gzip, which matters for cold starts and for anything that scales by launching new instances.

Inter-service traffic. gRPC and Protocol Buffers are already compact. If you are sending JSON between services — and most people are — compressing it is a large win that costs one configuration line.

which algorithm#

The default answer is zstd. It has a level parameter spanning from faster-than-lz4 to nearly-as-small-as-xz, decompression is fast at every level, and it has dictionary support.

the dictionary trick#

The most underused feature in compression, and the one with the biggest payoff for small messages.

Compression works by finding repetition. A 200-byte JSON message has almost no internal repetition, so compression barely helps — sometimes it makes it bigger.

But across many messages, there is enormous repetition: the same field names, the same enum values, the same URL prefixes.

A shared dictionary trained on representative samples gives the compressor that repetition up front:

zstd --train samples/*.json -o dict.zst

Then compress each message against the dictionary. Small-message ratios that were 1.1× become 3× or better. For any system moving many small similar messages — event streams, queue payloads, cache values — this is a large and nearly free win.

when not to compress#

Already-compressed data. Images, video, audio, archives. You will spend CPU to make it slightly larger.

Very small payloads without a dictionary. Under a few hundred bytes, compression overhead can exceed the savings.

When you are CPU-bound and not bandwidth-bound. Measure. This is rarer than people assume but it does happen.

Encrypted data, compressed after encryption. Pointless — ciphertext is incompressible. And compressing before encryption can leak information about the plaintext through the ciphertext length, which is the class of attack that includes CRIME and BREACH. If you compress and encrypt, know why it is safe in your context.

the ten-minute audit#

1. Check that HTTP responses are compressed, including API responses, not just HTML.
2. Check that Brotli is enabled, not just gzip.
3. Check your log shipping compresses and batches.
4. Check whether your largest database tables support compression and whether it is on.
5. If you move many small similar messages, train a dictionary.

That is an afternoon and it routinely produces a larger improvement than a month of application-level optimization, at a fraction of the risk.

The reason it does not happen is that nobody gets promoted for enabling Brotli.

1. You are not caching, or your cache never hits#

By far the most common. Not "we have no cache" — teams have caches. The caches do not hit.

Cache keys that include a timestamp, a branch name, a commit SHA, or anything else that changes every run will produce a 0% hit rate while looking like a working cache. Worse than nothing, because you also pay the upload.

Measure the hit rate first. Most CI systems report it and almost nobody looks. If it is not above 80% on a typical branch build, fix the key before you do anything else.

The key should be a hash of the inputs that actually determine the output: the lockfile for dependencies, the source files for a compilation unit. Nothing else.

2. You are rebuilding things that did not change#

If your build runs everything on every commit, you are paying for the whole repository to verify a one-line change in one module.

The fix is a build graph that knows what depends on what and only rebuilds the affected subtree. This is what Bazel, Buck, Nx, Turborepo, and Gradle's configuration cache exist for.

The cost is real — adopting a graph-aware build system is a project, and for a small repository it is not worth it. The threshold is roughly when your full build exceeds five minutes and most changes touch one small part.

3. Your tests are serial, or parallel badly#

Two separate failures.

Serial: you have 4,000 tests running one after another on one core. Almost every test framework has a parallel mode and it is frequently off by default because parallel tests expose shared state.

If your tests fail when run in parallel, that is a bug in the tests — shared fixtures, a common database, an order dependency — and fixing it is worth doing regardless of speed.

Parallel badly: you split tests into eight shards by file count, and one shard takes six minutes while the others take ninety seconds. Your suite takes six minutes.

Split by historical duration, not by count. Most CI systems support this and it frequently halves the wall clock for free.

4. You are doing full clones#

- uses: actions/checkout@v4
  with:
    fetch-depth: 1

A repository with a long history and large binary files can take minutes to clone. Almost no build step needs history. If one does — a changelog generator, a version-from-tags scheme — give that one job the full clone and shallow-clone everything else.

5. Your runners are too small#

The least satisfying answer and frequently the correct one.

Engineer time is expensive. A larger runner that halves your build time pays for itself immediately at any reasonable team size, and the arithmetic is easy to do:

build minutes/day × engineers × wait fraction × hourly cost
    vs.
extra runner cost

The extra runner is almost always cheaper. Teams resist this because compute cost is a visible line item and engineer waiting is not.

6. You are compiling in the container build#

If your Dockerfile runs npm install and then a compile, and you are not using BuildKit cache mounts and multi-stage builds properly, you are redoing work on every image build.

# cache mount survives across builds
RUN --mount=type=cache,target=/root/.npm \
    npm ci

Layer ordering matters too: copy the lockfile and install before copying source, so a source change does not invalidate the dependency layer. This is well known and consistently gotten wrong.

the measurement#

Before fixing anything, get the breakdown:

queue wait     ← runner capacity or concurrency limits
checkout       ← clone depth, LFS
dependency install ← caching
build          ← incrementality
test           ← parallelism
publish        ← usually fine

Most teams find that one stage is 60% of the time and they had assumed it was a different one.

the target#

Under ten minutes for the check that gates merge. Under two minutes if you can get there.

The reason for the ten-minute line is behavioral: past it, people stop waiting, start batching, and the whole delivery process degrades in ways that are hard to attribute back to the build.

Not everything has to be in that ten minutes. Split the blocking check from the exhaustive one. Lint, typecheck, and unit tests gate the merge. Integration, e2e, and the full platform matrix run after and page someone on failure.

That single split is usually the largest available improvement and it requires no new tooling.

Here is what the rest of us should be stealing.

the frame budget#

A game running at 60 frames per second has 16.6 milliseconds to do everything: input, simulation, physics, animation, culling, draw call submission, audio, and whatever the game is actually about.

Not on average. Every frame. A frame that takes 20 ms is a visible stutter, and players notice a single dropped frame.

That constraint produces a discipline the rest of software largely lacks:

Budgets are allocated per subsystem, in advance. Physics gets 3 ms. AI gets 2 ms. If a system exceeds its budget, that is a bug with an owner, not a "performance improvement opportunity for next quarter."

Compare to how most web services handle latency: an aspirational p99 target, no per-component budget, and no mechanism that fails when a component regresses.

Steal this. Give each layer of your request path a latency budget. Assert on it in tests. A change that pushes the database layer from 20 ms to 40 ms should fail CI, not show up on a dashboard a month later.

Worst case matters more than average. Nobody in games optimizes mean frame time. They optimize the 99.9th percentile, because that is what the player experiences as jank.

The web equivalent is obvious and widely ignored.

determinism as a feature#

Multiplayer games with lockstep networking require that the same inputs produce bit-identical outputs on every machine. That is a much stronger constraint than "correct," and achieving it teaches you where nondeterminism actually hides:

• Floating point differences across compilers and architectures.
• Iteration order over hash containers.
• Time-based logic.
• Uninitialized memory.
• Anything threaded without a strict ordering.

Every one of those is a source of flaky tests and unreproducible bugs in ordinary software, and most engineers have never had to hunt them systematically.

If your test suite is flaky, the game industry solved that problem decades ago and the answer is: make the system deterministic, then the flakiness is a bug you can find.

data-oriented design#

The insight: modern CPUs are enormously fast and memory is enormously slow, so performance is dominated by cache behavior rather than instruction count.

Which means the layout of your data matters more than the cleverness of your algorithm, at least until the constant factors stop dominating.

Arrays of structs versus structs of arrays. Contiguous memory over pointer chasing. Processing in batches over per-object virtual dispatch.

This applies directly to any data-processing code and most developers have never been taught to think about it. If you have a hot loop over a collection of objects and it is slower than it should be, the layout is usually why.

shipping to a fixed target#

Console games ship to hardware that does not change for seven years. You cannot tell the player to buy more RAM. You cannot autoscale.

That produces genuine engineering rather than procurement: measure, budget, optimize, cut scope, ship.

There is a lesson here for cloud-native development, where "we will scale it" has become a substitute for making things efficient. The teams that treat their resource envelope as fixed produce dramatically more efficient systems, and the constraint is a gift.

the thing games do worse#

To be fair: game codebases are frequently a mess, crunch culture is a genuine industry problem, testing practices are often weaker than in other software, and "ship it and patch it" has become normalized in a way that undermines a lot of the above.

Take the engineering discipline. Do not take the working conditions.

the talks to look for#

The technical postmortems are the good ones — a team explaining what went wrong in a shipped product, in detail, publicly. The industry has a stronger culture of this than almost any other software domain, and the material is excellent regardless of whether you care about games.

I have published misleading benchmarks myself, without intending to. That is the normal case — the incentives select for flattering methodology long before anyone decides to deceive.

the questions to ask, in order#

1. What was the baseline configured like?

The single most common distortion. The new thing is tuned by experts who built it. The baseline is at defaults, or configured by someone who wanted a specific result.

Look for: was the comparison system's cache warmed? Its connection pool sized? Its JIT given time? Its indexes created? If the writeup does not say, assume the answer is no.

2. What is the workload, and is it yours?

A benchmark showing 10× on sequential scans tells you nothing if your workload is point lookups. A benchmark on 100-byte values tells you nothing about 100 KB values.

The best benchmark is the one you run on your own data. Every published number is a hypothesis about your workload, not a measurement of it.

3. Is it a throughput number or a latency number, and which do you care about?

These trade off. A system can achieve enormous throughput by batching, which destroys latency. A system can achieve excellent latency by not batching, which caps throughput.

Numbers presented without both are presenting the flattering one.

4. What percentile?

An average latency is nearly useless. Users experience the tail. A system with a 5 ms average and a 4-second p99 is a bad system, and its average looks great.

If a benchmark reports only mean latency, that is a signal about the p99.

5. How long did it run?

Short benchmarks miss: garbage collection, compaction, cache eviction, log rotation, connection churn, thermal throttling, and every periodic background process.

A 30-second benchmark measures the best 30 seconds. Anything under ten minutes on a stateful system is measuring the warm-up.

6. How many runs, and what was the variance?

A single run is an anecdote. If there is no error bar, the difference between the two bars may be noise, and you cannot tell.

7. What hardware, and was it the same?

Different instance types, different storage, different network. Cloud instances of the same type vary meaningfully between individual instances — noisy neighbors, different underlying CPU steppings.

8. Who ran it?

Vendor benchmarks favor the vendor. Not usually through dishonesty — through a thousand small decisions about what to measure, made by people who know their own system's strengths.

Independent benchmarks are better and rarer. A benchmark run by the losing vendor's competitor is data; a benchmark where the losing vendor was invited to tune their configuration is much better data.

the specific tricks#

Comparing against a version from two years ago. Technically accurate, completely useless.

Choosing a metric where you win. If a system reports "requests per second per core," ask what happens to total requests per second.

The unlabeled log scale. Makes a 15% difference look like a chasm.

The truncated y-axis. Same trick, more common.

Different consistency levels. Comparing an eventually-consistent write to a synchronously-replicated one is not a comparison. This is endemic in database benchmarking.

Excluding the slow path. "Cached read performance" where the competitor's cache was not warm.

Cost normalization that ignores the cost. "Performance per dollar" using list price when nobody pays list price.

how to publish an honest one#

If you are producing benchmarks:

• Publish the exact configuration for both systems, and the commands.
• Ask the other project to review your configuration before you publish. If they say your baseline is untuned, fix it.
• Report p50, p95, p99, and max. Report variance across runs.
• Say what workload this represents and what it does not.
• Publish the harness so people can reproduce it.
• State the version numbers and the date.

That is more work and it is the difference between marketing and engineering.

the practical rule#

For anything you are actually going to depend on: run it yourself, on your data, for at least an hour, with the tail latency recorded.

Published benchmarks tell you which systems are worth testing. They do not tell you which one to use, and treating them as if they do is how you end up migrating twice.

what it does#

Historically Postgres read from disk synchronously: a backend process issues a read, blocks, gets the page, continues. On a spinning disk that was fine — the disk was the bottleneck and you could not do better. On modern NVMe, where the device can service many requests in parallel and a single-threaded synchronous reader leaves most of the device idle, it wasted a great deal of available throughput.

Postgres 18 adds an I/O method abstraction:

io_method = sync          # the old behavior
io_method = worker        # dedicated I/O worker processes (default)
io_method = io_uring      # Linux io_uring, if built with support

Sequential scans, bitmap heap scans, and vacuum use it. Other paths do not yet; this is an incremental rollout across releases.

what people are seeing#

The consistent pattern from the reports I trust:

Large sequential scans on fast NVMe: substantial improvement. Analytical queries scanning a large table are the clearest win. This is exactly what you would predict — the workload was read-parallel and the code was not.

Vacuum: meaningfully faster. This matters more than it sounds. Vacuum being slow is the root of a large class of Postgres operational problems, and anything that makes it finish sooner reduces bloat pressure.

OLTP with a good cache hit rate: little change. If your working set is in shared buffers, you were not doing I/O, so making I/O faster does nothing. Most transactional workloads are in this category and should not expect much.

Cloud block storage: mixed. Network-attached storage has enough latency and enough of its own queueing that the gains are smaller and less predictable than on local NVMe. Test rather than assume.

the tuning notes#

Two settings matter and the defaults are conservative:

io_combine_limit = 128kB        # how much adjacent I/O to merge into one request
effective_io_concurrency = 16   # how many concurrent reads to issue

effective_io_concurrency has historically been set based on spindle count, which is meaningless on NVMe. For a modern SSD, values well above the old defaults are appropriate. Measure — the curve flattens and then degrades, and where it turns depends on your device.

If you are on Linux with a recent kernel and can build with io_uring support, it generally outperforms the worker method by a modest margin. The worker method is the safe default and it is genuinely good.

the other things in 18 worth knowing#

Skip scan for B-tree indexes. A multi-column index on (a, b) can now be used for a query filtering only on b, when a has low cardinality. This eliminates a category of redundant index that people have been maintaining for years.

uuidv7() built in. Time-ordered UUIDs, natively. If you use UUIDs as primary keys, v7 dramatically improves index locality compared to v4, which scatters inserts across the whole B-tree. This is a real performance issue that a lot of applications have and do not know about.

OAuth authentication support, which matters for anyone trying to eliminate static database passwords.

Faster major-version upgrades, with statistics preserved across pg_upgrade. Previously you finished an upgrade with no statistics and a database that planned terribly until you ran ANALYZE across everything. That was a genuine outage risk and it is fixed.

should you upgrade#

Yes, on the normal schedule — after a point release or two, with a tested rollback, having read the release notes for the incompatibilities.

The async I/O is the headline and for most transactional workloads the statistics preservation on upgrade and uuidv7() will matter more day to day.

As always with Postgres: the release is well-tested, the migration is usually boring, and the people who get burned are the ones who skipped four major versions and tried to do it in one jump.

why linking matters#

For a large Rust binary, linking is frequently the dominant cost of an incremental build. You change one line, the compiler recompiles one crate quickly, and then the linker spends several seconds stitching together a hundred megabytes of object files and debug information.

GNU ld is old, single-threaded in the parts that matter, and was designed for a different era of binary sizes. lld is parallel, substantially faster, and has been production-ready for years — it is the default on several other platforms already.

Reported improvements vary by project, with the largest gains on debug builds of large dependency trees. Two to five times faster linking is a common range. If your edit-compile-run loop is four seconds and linking was two of them, you feel this every single time.

how to check#

cargo build --timings

Opens an HTML report showing where time went, including link time as a distinct phase. Most people have never looked at this and are surprised by what it says.

If you were already opting into lld or mold via .cargo/config.toml, nothing changes for you. If you were not — which is most people, because the configuration was obscure and required knowing it existed — you get the improvement for free on upgrade.

the general lesson#

Defaults are the most impactful thing a toolchain ships.

A faster linker has been available for years. Anyone could have configured it. The instructions were in a blog post. Almost nobody did, because it required knowing the option existed, knowing it was safe, and caring enough to edit a config file.

Changing the default delivers the improvement to everyone at once. That is worth more than a decade of documentation.

This generalizes. If you maintain a tool and you find yourself writing documentation explaining how to enable a better behavior, ask whether the better behavior should be the default. The answer is usually yes, and the reason it is not is usually caution about breaking a small number of unusual setups — which is a real concern that should be handled with a flag to opt out, not a flag to opt in.

the rest of 1.90#

• Cargo workspace publishing — cargo publish --workspace publishes multiple interdependent crates in the correct order. Anyone maintaining a multi-crate project has written a shell script for this. Now you can delete it.
• Demoted target tiers for several less-used platforms.
• x86_64-apple-darwin demoted to tier 2 with host tools still provided, which reflects where Apple hardware has actually gone.
• The usual const stabilizations and API additions.

the trend line#

Rust's reputation for slow compilation is the single most cited objection to adopting it, and it has been steadily addressed for years: incremental compilation, parallel front-end work, the new trait solver, better caching, and now the linker.

Compile times are meaningfully better than they were three years ago and still slower than Go. That gap is partly fundamental — monomorphization and the amount of optimization Rust does are not free — and partly still addressable.

Anyone who tried Rust in 2021 and bounced off the build times should try again. The numbers are different now.

Caching gets you 1000x, and it does it by not doing the work at all.

the hierarchy#

Every layer is faster than the one below and every layer has a different invalidation problem. Roughly:

The wins are largest at the top and the invalidation is hardest at the bottom. That trade-off is the entire discipline.

the part everyone skips#

Measure the hit rate. Not once — continuously, as a first-class metric with an alert.

A cache with a 50% hit rate is not half as good as one with 99%. It is dramatically worse than that, because the misses dominate the latency distribution:

99% hit rate:  0.99 × 1ms + 0.01 × 100ms  =  1.99 ms average
50% hit rate:  0.50 × 1ms + 0.50 × 100ms  = 50.5  ms average

25x worse. And your p99 is entirely determined by the miss path regardless of the hit rate, which is why profiling under normal load tells you nothing about it.

I have seen caches deployed with no hit rate metric, running at 20%, considered a success because latency improved slightly. Somebody added memory pressure, eviction went up, and the "improvement" evaporated silently.

the failure mode nobody plans for#

Cache stampede. A popular key expires. A thousand concurrent requests all miss simultaneously. All thousand hit the database. The database falls over. The cache never repopulates because nothing succeeds. Your outage is now self-sustaining.

Three fixes, use all of them:

1. Request coalescing. One request per key populates; the rest wait on it. singleflight in Go, Lock around the miss path elsewhere.
2. Jittered TTLs. Never expire a thousand keys at the same instant. Add random ±10%.
3. Serve stale while revalidating. Return the expired value immediately and refresh in the background. Almost always the right choice; almost nobody does it by default.

invalidation#

Phil Karlton's line about the two hard problems is quoted constantly and rarely acted on. The practical version:

Prefer TTL to explicit invalidation. A TTL is a bounded staleness guarantee you can reason about. Explicit invalidation is a distributed systems problem where every write path must know every cache key it affects, forever, including the ones added next year.

If you must invalidate explicitly, use key versioning. Do not delete — change the key.

user:1234:v7:profile

Bump the version on write. Old keys age out naturally. No delete storm, no race between invalidate and repopulate, no partially-invalidated state.

Never cache without a TTL. "It will be invalidated when it changes" is how you get a value from 2023 in production and no way to find out.

the one that is not optional#

Cache the expensive read-mostly thing closest to where it is used. If you do nothing else in a performance push, find the query that runs on every request, returns nearly-identical data, and takes 40 ms. Put it in memory with a 60-second TTL.

That single change is worth more than a month of algorithmic optimization on most web applications, and it takes an afternoon.

when not to cache#

Correctness-critical data with strict consistency requirements. Account balances, inventory at the point of sale, permission checks. The staleness window is a correctness bug, not a performance trade.

Cache the permission lookup if you must, with a short TTL and an explicit invalidation on revoke — and understand that you have accepted a window during which a revoked user still has access. Sometimes that is fine. Decide deliberately, and write down which one you chose.

fn frobnicate<const N: usize>(arr: [u8; N]) -> [u8; N] { /* ... */ }

let data = [1, 2, 3, 4];
let out: [u8; _] = frobnicate(data);   // N inferred as 4

Previously you either wrote the number, which duplicated information the compiler already had, or restructured to avoid needing it. Small change, removes a real papercut, and it composes well with the growing amount of const-generic code in the ecosystem.

the SIMD stabilizations#

A large batch of x86 intrinsics stabilized, including AVX-512 families. This matters for a specific and growing population: people writing hand-tuned kernels in Rust for compression, encoding, cryptography, and increasingly ML inference.

Before this, using these intrinsics required nightly. Which meant that a production library wanting AVX-512 paths had to either require nightly — a non-starter for most consumers — or drop to C via FFI.

That was a genuine competitive disadvantage against C++ for numerical work and it is now mostly gone.

The pattern for using them safely:

if is_x86_feature_detected!("avx512f") {
    unsafe { fast_path_avx512(data) }
} else if is_x86_feature_detected!("avx2") {
    unsafe { fast_path_avx2(data) }
} else {
    scalar_path(data)
}

Runtime detection, multiple implementations, scalar fallback. Tedious and correct. std::simd remains unstable for the portable abstraction, which is the thing most people actually want and which continues to take a long time.

also in the release#

• unsafe attributes get more coverage, continuing the 2024 edition direction.
• Result::flatten stabilizes. Small and frequently wanted.
• Cargo now supports --compile-time-deps for building only what is needed for procedural macros and build scripts, which speeds up some tooling workflows meaningfully.
• i686 targets raise their baseline CPU requirement, which will affect approximately nobody and will affect them a lot.

the const generics arc, for context#

Const generics were stabilized in a minimal form in Rust 1.51, four years ago. Since then the feature has been growing capability incrementally: first only integers, then inference improvements, then better error messages, now this.

The full feature — const generic expressions, where you can write [T; N * 2] — is still unstable and has been "coming soon" for a long time. The blocker is that arbitrary const expressions in type position require deciding when two type-level expressions are equal, which is undecidable in general and requires drawing a careful line.

Rust's approach to this has consistently been: ship the decidable subset, leave the hard part open, do not paint yourself into a corner. It is slow and it has not produced an unsound type system, which is more than several languages with similar features can say.

The launch also included a "Heavy" tier that runs multiple agents in parallel and selects among their answers, and a $300/month subscription for it.

reading the numbers correctly#

The headline HLE figure comes from the Heavy configuration with tools enabled. That is a legitimate configuration and it is not comparable to a single-sample number from a competitor, which is how it was presented in most coverage.

There are at least four distinct things being reported as "the score":

1. Single sample, no tools.
2. Single sample, with tools (search, code execution).
3. Consensus of N samples, no tools.
4. Multi-agent parallel with tools and selection.

Number four can be five or ten times the cost of number one. Comparing across these without stating which is being used is not a small methodological quibble. It is the difference between "our model is better" and "we spent more money at inference time."

To be clear: xAI disclosed their configurations. The disclosure was in the livestream and the fine print. The number that traveled was the big one, with no configuration attached, and that is now just how model launches work.

the parallel-agents technique#

Worth taking seriously on its own merits, independent of the marketing.

Running N independent attempts and selecting the best is a well-established test-time scaling method, and it works better than one long chain for a specific reason: independent samples have independent errors, so selection can filter them, while a single chain compounds its errors with no mechanism to recover.

The hard part is selection. If you can verify — the tests pass, the proof checks, the code compiles — selection is easy and this technique is enormously powerful. If you cannot verify, you need a judge, and the judge has the same failure modes as the generator.

This is why the technique works so much better on math and code than on open-ended reasoning. Verifiability is the whole game.

The practical version for your own work: if you have a verifier, sample multiple times and filter. You do not need a special model tier for this. Three samples through a cheap model with a test-suite check will frequently outperform one sample through an expensive one, for less money.

the other thing#

Grok's public-facing behavior in the weeks before this launch included a series of incidents that xAI attributed to a system prompt change. I am not going to recount them; they are well documented and they were bad.

The engineering lesson worth extracting: a system prompt is production configuration. It should be version controlled, reviewed, tested against an adversarial eval suite, and rolled out gradually. Treating it as a text box someone can edit is how you get an incident that makes international news.

If your product has a system prompt in a config file that anyone can change without review, fix that this week.

the state of play#

Grok 4 is a competitive frontier model. xAI went from founded to frontier in roughly two years, which remains the most remarkable thing about the company and is mostly a story about capital and urgency rather than research insight.

The models are converging. The differentiation is moving to price, latency, tooling, and trust, and xAI's position on that last one is self-inflicted.

The headline finding: past a certain complexity, accuracy collapses to zero, and counterintuitively the models reduce their reasoning effort as problems get harder, despite having budget remaining.

The paper is good, the reaction was overheated in both directions, and the rebuttals are worth reading alongside it.

what the paper found#

Three regimes:

1. Low complexity — standard models match or beat reasoning models. The extra thinking is wasted and sometimes harmful, because the model overthinks its way past a correct early answer.
2. Medium complexity — reasoning models win clearly. This is the regime the benchmarks live in.
3. High complexity — both collapse to zero accuracy.

The reduced-effort finding at high complexity is the genuinely interesting one. The models emit fewer reasoning tokens on harder problems, which is exactly backwards, and suggests something like learned giving-up rather than a compute limit.

the rebuttals#

Several, and they land differently.

The output length objection. Tower of Hanoi with N disks requires 2^N − 1 moves. At N=15 that is 32,767 moves. If the model must enumerate every move in its output, it hits the token limit before it hits a reasoning limit. Several researchers showed models explicitly stating they would not enumerate all moves due to length — and being scored as failures. That is measuring output capacity, not reasoning.

Ask instead for a program that generates the solution and the models do fine. That is a meaningful distinction: knowing the algorithm versus executing it by hand.

The unsolvable instances objection. Some river-crossing configurations in the evaluation set have no solution. Models were penalized for failing to solve them, which is not a reasoning failure, it is a benchmark bug.

The framing objection. "Reasoning models cannot reason" was the headline everywhere. The paper does not claim that. It claims a specific scaling limitation on a specific class of problem.

what survives#

After the corrections, the real finding is narrower and still important:

Current reasoning models do not reliably execute long deterministic procedures. They can identify the right algorithm and fail to carry it out over many steps. Error rates compound; there is no self-correction mechanism strong enough to catch accumulated drift over hundreds of steps.

That is a genuine limitation with direct practical consequences. If your task requires exact multi-step execution — a data migration, a complex refactor across many files, a financial calculation — the model should be writing code that does it, not doing it token by token.

the practical rule#

Use the model to produce the procedure. Use a computer to execute it.

This is not a workaround, it is the correct architecture. Deterministic execution is what computers are for. A model that writes a correct script and runs it is strictly better than a model that simulates the script in its head, and it is verifiable, repeatable, and debuggable.

Every production agent architecture I have seen work well converges on this. Every one that tries to do arithmetic in the reasoning trace eventually produces a number that is wrong in a way nobody catches.

the meta-lesson#

A paper from a large company with a competitive interest, on a contested topic, with a provocative title, will be read as a position statement regardless of its contents. The authors probably knew that.

Read the methodology section. The methodology section is where papers are true or false.

Within forty-eight hours the release turned into a story about benchmark integrity instead.

what happened#

Maverick posted a very strong score on LMArena, the human-preference leaderboard. It then emerged that the model evaluated on the arena was an "experimental chat version" tuned for conversationality — not the checkpoint released to the public. LMArena updated its policies and published the disputed comparison. Meta's response was that experimental variants are normal and the arena version was labeled.

Both of those things can be true and the outcome is still bad, because the number that traveled was attached to a model nobody could download.

why this keeps happening#

Leaderboards are the only shared vocabulary the field has, and they are being asked to carry weight they cannot bear.

• Human preference arenas measure whether people like the answer. That correlates with quality and also with formatting, length, confidence, and sycophancy. A model tuned to be agreeable climbs.
• Static benchmarks leak into training data. Every popular benchmark is on the internet and every frontier model has read the internet. Contamination is not always deliberate and it is essentially always present.
• Vendor-run evaluations use vendor-chosen settings. Consensus-of-64 for yours, single-sample for theirs. Both numbers are real and the comparison is not.

There is no fix that survives contact with commercial incentives. The only durable answer is that you have to run your own evaluation on your own task.

the 10 million token claim#

Scout's context window is stated at 10M tokens, achieved through an interleaved attention scheme without positional embeddings in some layers, plus inference-time temperature scaling on attention. It was trained on far shorter sequences and generalizes upward.

Take this as an upper bound on what the architecture accepts, not on what it usefully processes. Independent long-context evaluations found substantial degradation well before that number. That is not unique to Llama 4 — it is true of every long-context claim — but 10M is a big enough number that the gap between accepted and useful is enormous.

what is actually good here#

The MoE architecture with 17B active parameters is a real efficiency story. Maverick's quality-per-active-parameter is strong, and active parameters are what determine inference cost. A model you can serve at 17B economics with much better quality than a 17B dense model is a genuinely useful thing to have.

Native multimodality — images in the base model rather than bolted on via an adapter — is also the right architecture and is where everyone is heading.

The license is still not open source. "Llama community license" has an MAU threshold and naming requirements. Call it open weights, which it is, and do not call it open source, which it is not.

the practical advice#

Build an eval harness with fifty examples from your actual domain. Run every candidate model against it. Store the results in your repo next to your tests.

It will take a day and it will make you immune to this entire genre of news.