Reading a flame graph

A flame graph answers one question extremely well: where is the time going? Most people who look at one have never been told how to read it, so they squint at a colourful pile of rectangles and conclude that profiling is hard.

It is not. There are four rules and four shapes.

the four rules#

The x-axis is not time. This is the rule everyone gets wrong. Left-to-right is alphabetical or arbitrary — it is not chronological. A frame on the left did not happen before a frame on the right.

Width is total time. A frame's width is the proportion of samples that included it. Wide means expensive. That is the entire message.

Height is stack depth. A frame sitting on another means it was called by it. Tall is not bad; tall just means deep call stacks.

Colour is usually meaningless. In most tools it is random, chosen to make adjacent frames distinguishable. Do not read anything into it unless the tool explicitly says otherwise — some use it for language or module.

That is it. Wide is expensive, stacked means called-by, colour is decoration.

the four shapes#

A wide plateau near the top. One function, doing real work, dominating. This is the good case: a clear, single hotspot. Optimise that function or call it less.

A wide plateau near the bottom, narrowing above. The time is spread across many children. There is no single hotspot; the cost is the whole subtree. Look one level up — usually the answer is "call this subtree fewer times" rather than "make some leaf faster".

A staircase. Deep, narrow, repeating structure. Often recursion, sometimes a framework's middleware chain. Each frame is cheap; the depth is the cost. Look for a way to shorten the chain rather than to optimise any frame in it.

Many thin spikes with nothing dominant. Death by a thousand cuts, or your profile is too short. Check the sample count first — a profile of two hundred samples looks like this regardless of the workload. If the sample count is healthy and it still looks like this, the program is genuinely uniform and your wins are architectural rather than local.

the practical workflow#

Profile the thing you care about, under load that resembles production. A profile of a cold process running a synthetic benchmark measures start-up and your benchmark harness.

Profile long enough. Seconds, not milliseconds. You are sampling; you need samples.

Look at the widest frame you did not expect. Not the widest frame — the widest surprising one. Everyone's profile is dominated by something obvious and irreducible. The win is in the frame that has no business being 8% of your runtime.

Check for time you cannot see. A flame graph of CPU samples shows CPU. If your program spends most of its wall clock waiting on a socket, an on-CPU profile will be nearly empty and entirely misleading. That is what off-CPU profiling and tracing are for, and confusing the two is the most common way people conclude a profiler is lying to them.

differential flame graphs#

Underused and excellent. Profile before, profile after, render the difference — frames that got wider are coloured one way, narrower the other.

This turns "did my change help?" from an argument about noisy averages into a picture. It is also the fastest way to find a performance regression introduced between two releases: profile both, diff, look at what turned red.

the honest caveat#

A flame graph tells you where time went. It does not tell you whether that time was necessary, and it will happily show you a perfectly optimised hot loop that should not have been called at all.

The largest performance wins are almost always "stop doing this work" rather than "do this work faster," and no profiler can suggest that. It can only show you where to point the question.