Compression is underrated

Compression trades CPU for bytes. On modern hardware, where CPU is abundant and bandwidth is the constraint at nearly every layer, that trade is favorable far more often than people apply it.

the layers where it pays#

HTTP responses. Everyone does this and most do it badly. Check yours:

bash

curl -sI -H 'Accept-Encoding: br, gzip' https://example.com/api/data | grep -i content-encoding

If that returns nothing, you are shipping uncompressed JSON, and JSON compresses extraordinarily well — commonly 80–90% for typical API responses, because it is mostly repeated key names.

Brotli beats gzip by roughly 15–20% on text at comparable CPU cost, and is supported everywhere. Use it for static assets at maximum level (precomputed, so the CPU cost is paid once) and at a moderate level for dynamic responses.

Database storage. Most modern databases support per-table or per-column compression. On a table of text or JSON, it frequently halves the storage — which also halves the I/O, which means more of the working set fits in memory, which is where the real win is.

The CPU cost of decompression is almost always smaller than the I/O cost you avoided.

Logs and telemetry. Log shipping is often a meaningful fraction of internal network traffic and vendor cost. Compressed batching typically reduces it by an order of magnitude, and the batching itself reduces request overhead.

Backups and object storage. Storage is cheap and it is not free, and egress definitely is not. Compression at rest is a direct cost reduction with no downside for cold data.

Container images. Zstandard-compressed layers pull faster than gzip, which matters for cold starts and for anything that scales by launching new instances.

Inter-service traffic. gRPC and Protocol Buffers are already compact. If you are sending JSON between services — and most people are — compressing it is a large win that costs one configuration line.

which algorithm#

algorithm	use for
zstd	almost everything. Wide speed/ratio range, fast decompression.
brotli	HTTP text, especially static assets at max level.
gzip	compatibility fallback. Never the best choice, always supported.
lz4	when speed dominates entirely. In-memory, hot paths, real-time.
xz / lzma	archives you compress once and rarely read. Slow, small.

The default answer is zstd. It has a level parameter spanning from faster-than-lz4 to nearly-as-small-as-xz, decompression is fast at every level, and it has dictionary support.

the dictionary trick#

The most underused feature in compression, and the one with the biggest payoff for small messages.

Compression works by finding repetition. A 200-byte JSON message has almost no internal repetition, so compression barely helps — sometimes it makes it bigger.

But across many messages, there is enormous repetition: the same field names, the same enum values, the same URL prefixes.

A shared dictionary trained on representative samples gives the compressor that repetition up front:

bash

zstd --train samples/*.json -o dict.zst

Then compress each message against the dictionary. Small-message ratios that were 1.1× become 3× or better. For any system moving many small similar messages — event streams, queue payloads, cache values — this is a large and nearly free win.

when not to compress#

Already-compressed data. Images, video, audio, archives. You will spend CPU to make it slightly larger.

Very small payloads without a dictionary. Under a few hundred bytes, compression overhead can exceed the savings.

When you are CPU-bound and not bandwidth-bound. Measure. This is rarer than people assume but it does happen.

Encrypted data, compressed after encryption. Pointless — ciphertext is incompressible. And compressing before encryption can leak information about the plaintext through the ciphertext length, which is the class of attack that includes CRIME and BREACH. If you compress and encrypt, know why it is safe in your context.

the ten-minute audit#

Check that HTTP responses are compressed, including API responses, not just HTML.
Check that Brotli is enabled, not just gzip.
Check your log shipping compresses and batches.
Check whether your largest database tables support compression and whether it is on.
If you move many small similar messages, train a dictionary.

That is an afternoon and it routinely produces a larger improvement than a month of application-level optimization, at a fraction of the risk.

The reason it does not happen is that nobody gets promoted for enabling Brotli.