Distributed tracing that people actually use

Distributed tracing is the right answer to "what happened to this request across nine services." Most implementations produce a system that is technically correct and that nobody opens during an incident.

Three details separate the two outcomes.

detail one: the trace ID must be everywhere#

A trace is only useful if you can find it. Which means the trace ID must appear:

In every log line, so you can pivot from a log to the trace.
In the response headers, so a client can report it.
On the error page, so a user can paste it into a support ticket.
In your error tracking, so an exception links to its trace.
In the support tool, so a support engineer can hand it to an engineer.

The workflow that makes tracing valuable: customer reports a problem → support gets the trace ID → engineer opens exactly that request and sees everything.

Without the ID in those places, the workflow is: customer reports a problem → engineer tries to guess which of eight million traces it was.

That single difference determines whether the investment pays off, and it is a plumbing problem rather than a tracing problem.

detail two: spans need attributes, not just timing#

A span that says "database query, 45 ms" is nearly useless. Which query? Which table? How many rows? Was it cached?

python

with tracer.start_as_current_span("db.query") as span:
    span.set_attribute("db.system", "postgresql")
    span.set_attribute("db.operation", "select")
    span.set_attribute("db.table", "orders")
    span.set_attribute("db.rows_returned", len(rows))
    span.set_attribute("app.tenant_id", tenant)
    span.set_attribute("app.cache_hit", cached)

The attributes are what let you ask questions the waterfall cannot answer: is this slow for one tenant, is it slow when the cache misses, is it slow when the result set is large.

Put your business identifiers on the root span. Tenant, user, plan tier, feature flags, client version. Then you can filter traces by them, which is how you find the pattern rather than the instance.

High cardinality is correct here. This is not a metrics system.

detail three: sample intelligently, keep the interesting ones#

Full sampling is expensive and mostly wasteful — the successful, fast, boring requests are identical to each other.

Tail-based sampling makes the decision after the trace completes, when you know whether it was interesting:

100% of traces with an error.
100% of traces above a latency threshold.
A small percentage of everything else, for baseline comparison.
100% of traces for a specific customer, when you are debugging that customer.

That last one is worth building explicitly. A flag that says "capture everything for this tenant for the next hour" turns an unreproducible customer report into a solvable problem, and it is the single most useful debugging feature you can add to a multi-tenant system.

the things that waste effort#

Instrumenting everything. Auto-instrumentation gives you spans for every framework operation, most of which are noise. A trace with four hundred spans is not more informative than one with twenty; it is less, because the signal is buried.

Instrument the boundaries — service calls, database queries, external APIs, queue operations — and add manual spans only for genuinely expensive internal operations.

Perfect propagation. You will have services that drop the context: a message queue without header support, a third-party integration, a legacy component. Do not block the rollout on 100% coverage. A trace with a gap is still much better than no trace.

Building dashboards from traces. Traces answer "what happened to this request." Metrics answer "what is happening to all requests." Using traces for aggregate views is expensive and slow. Use both, for what each is good at.

the question tracing answers that nothing else does#

Not "is the system slow" — metrics tell you that, cheaper.

Not "what error happened" — logs tell you that.

**"Why was this specific request slow, and what was different about it?"**

That is the question. It is the question you have during an incident, when one customer is affected and the aggregate metrics look fine. And it is unanswerable without tracing.

If your tracing setup does not make that question easy to answer in under a minute, the setup is the problem, not the concept.

the minimum viable version#

If you are starting from nothing:

Adopt OpenTelemetry. It is the standard, the instrumentation libraries are broad, and it keeps you portable across backends.
Propagate context across every service boundary.
Put the trace ID in every log line and every response header.
Add business identifiers to the root span.
Tail-sample: all errors, all slow, 1% of the rest.

That is a week of work and it covers most of the value. Everything beyond it is refinement.