tech, developers, and the code underneath

issue 139· essay·

Why your build is slow

Six causes, in order of how often they're the actual problem, with the fix for each.

Build time is the tax you pay on every single change, and most teams have never measured where it goes. Here are the causes, roughly ordered by how often I find each one to be the actual bottleneck.

1. You are not caching, or your cache never hits#

By far the most common. Not "we have no cache" — teams have caches. The caches do not hit.

Cache keys that include a timestamp, a branch name, a commit SHA, or anything else that changes every run will produce a 0% hit rate while looking like a working cache. Worse than nothing, because you also pay the upload.

Measure the hit rate first. Most CI systems report it and almost nobody looks. If it is not above 80% on a typical branch build, fix the key before you do anything else.

The key should be a hash of the inputs that actually determine the output: the lockfile for dependencies, the source files for a compilation unit. Nothing else.

2. You are rebuilding things that did not change#

If your build runs everything on every commit, you are paying for the whole repository to verify a one-line change in one module.

The fix is a build graph that knows what depends on what and only rebuilds the affected subtree. This is what Bazel, Buck, Nx, Turborepo, and Gradle's configuration cache exist for.

The cost is real — adopting a graph-aware build system is a project, and for a small repository it is not worth it. The threshold is roughly when your full build exceeds five minutes and most changes touch one small part.

3. Your tests are serial, or parallel badly#

Two separate failures.

Serial: you have 4,000 tests running one after another on one core. Almost every test framework has a parallel mode and it is frequently off by default because parallel tests expose shared state.

If your tests fail when run in parallel, that is a bug in the tests — shared fixtures, a common database, an order dependency — and fixing it is worth doing regardless of speed.

Parallel badly: you split tests into eight shards by file count, and one shard takes six minutes while the others take ninety seconds. Your suite takes six minutes.

Split by historical duration, not by count. Most CI systems support this and it frequently halves the wall clock for free.

4. You are doing full clones#

yaml
- uses: actions/checkout@v4
  with:
    fetch-depth: 1

A repository with a long history and large binary files can take minutes to clone. Almost no build step needs history. If one does — a changelog generator, a version-from-tags scheme — give that one job the full clone and shallow-clone everything else.

5. Your runners are too small#

The least satisfying answer and frequently the correct one.

Engineer time is expensive. A larger runner that halves your build time pays for itself immediately at any reasonable team size, and the arithmetic is easy to do:

build minutes/day × engineers × wait fraction × hourly cost
    vs.
extra runner cost

The extra runner is almost always cheaper. Teams resist this because compute cost is a visible line item and engineer waiting is not.

6. You are compiling in the container build#

If your Dockerfile runs npm install and then a compile, and you are not using BuildKit cache mounts and multi-stage builds properly, you are redoing work on every image build.

dockerfile
# cache mount survives across builds
RUN --mount=type=cache,target=/root/.npm \
    npm ci

Layer ordering matters too: copy the lockfile and install before copying source, so a source change does not invalidate the dependency layer. This is well known and consistently gotten wrong.

the measurement#

Before fixing anything, get the breakdown:

queue wait     ← runner capacity or concurrency limits
checkout       ← clone depth, LFS
dependency install ← caching
build          ← incrementality
test           ← parallelism
publish        ← usually fine

Most teams find that one stage is 60% of the time and they had assumed it was a different one.

the target#

Under ten minutes for the check that gates merge. Under two minutes if you can get there.

The reason for the ten-minute line is behavioral: past it, people stop waiting, start batching, and the whole delivery process degrades in ways that are hard to attribute back to the build.

Not everything has to be in that ten minutes. Split the blocking check from the exhaustive one. Lint, typecheck, and unit tests gate the merge. Integration, e2e, and the full platform matrix run after and page someone on failure.

That single split is usually the largest available improvement and it requires no new tooling.

get README in your inbox

One dispatch, no noise. Tech and developer news, plus the occasional long piece on the craft.

subscribe →