How to read a benchmark

Every vendor publishes benchmarks. Most of them are technically accurate and misleading, and the techniques are consistent enough to enumerate.

I have published misleading benchmarks myself, without intending to. That is the normal case — the incentives select for flattering methodology long before anyone decides to deceive.

the questions to ask, in order#

1. What was the baseline configured like?

The single most common distortion. The new thing is tuned by experts who built it. The baseline is at defaults, or configured by someone who wanted a specific result.

Look for: was the comparison system's cache warmed? Its connection pool sized? Its JIT given time? Its indexes created? If the writeup does not say, assume the answer is no.

2. What is the workload, and is it yours?

A benchmark showing 10× on sequential scans tells you nothing if your workload is point lookups. A benchmark on 100-byte values tells you nothing about 100 KB values.

The best benchmark is the one you run on your own data. Every published number is a hypothesis about your workload, not a measurement of it.

3. Is it a throughput number or a latency number, and which do you care about?

These trade off. A system can achieve enormous throughput by batching, which destroys latency. A system can achieve excellent latency by not batching, which caps throughput.

Numbers presented without both are presenting the flattering one.

4. What percentile?

An average latency is nearly useless. Users experience the tail. A system with a 5 ms average and a 4-second p99 is a bad system, and its average looks great.

If a benchmark reports only mean latency, that is a signal about the p99.

5. How long did it run?

Short benchmarks miss: garbage collection, compaction, cache eviction, log rotation, connection churn, thermal throttling, and every periodic background process.

A 30-second benchmark measures the best 30 seconds. Anything under ten minutes on a stateful system is measuring the warm-up.

6. How many runs, and what was the variance?

A single run is an anecdote. If there is no error bar, the difference between the two bars may be noise, and you cannot tell.

7. What hardware, and was it the same?

Different instance types, different storage, different network. Cloud instances of the same type vary meaningfully between individual instances — noisy neighbors, different underlying CPU steppings.

8. Who ran it?

Vendor benchmarks favor the vendor. Not usually through dishonesty — through a thousand small decisions about what to measure, made by people who know their own system's strengths.

Independent benchmarks are better and rarer. A benchmark run by the losing vendor's competitor is data; a benchmark where the losing vendor was invited to tune their configuration is much better data.

the specific tricks#

Comparing against a version from two years ago. Technically accurate, completely useless.

Choosing a metric where you win. If a system reports "requests per second per core," ask what happens to total requests per second.

The unlabeled log scale. Makes a 15% difference look like a chasm.

The truncated y-axis. Same trick, more common.

Different consistency levels. Comparing an eventually-consistent write to a synchronously-replicated one is not a comparison. This is endemic in database benchmarking.

Excluding the slow path. "Cached read performance" where the competitor's cache was not warm.

Cost normalization that ignores the cost. "Performance per dollar" using list price when nobody pays list price.

how to publish an honest one#

If you are producing benchmarks:

• Publish the exact configuration for both systems, and the commands.
• Ask the other project to review your configuration before you publish. If they say your baseline is untuned, fix it.
• Report p50, p95, p99, and max. Report variance across runs.
• Say what workload this represents and what it does not.
• Publish the harness so people can reproduce it.
• State the version numbers and the date.

That is more work and it is the difference between marketing and engineering.

the practical rule#

For anything you are actually going to depend on: run it yourself, on your data, for at least an hour, with the tail latency recorded.

Published benchmarks tell you which systems are worth testing. They do not tell you which one to use, and treating them as if they do is how you end up migrating twice.