Observability costs more than the thing it observes

A meaningful number of organizations now spend more on observability tooling than on the compute being observed. That is not automatically wrong — visibility has real value — but it is almost never a decision anyone made deliberately.

how it happens#

Logs at debug level in production, because someone turned it on during an incident in 2023 and nobody turned it off.

Every metric at every dimension. A counter with five labels, each with a hundred values, is ten billion time series. Cardinality multiplies and the pricing follows.

100% trace sampling because sampling felt like giving something up.

Retention set to the maximum because nobody knew what to pick and the default was generous.

Duplicate pipelines. Logs going to two vendors during a migration that never completed.

Each decision was locally reasonable. The bill is the sum.

the reduction, in order of return#

1. Sample traces intelligently. You do not need every trace. You need:

• All traces that errored.
• All traces above a latency threshold.
• A small percentage of the rest, for baseline.

Tail-based sampling — decide after the trace completes, when you know whether it is interesting — typically cuts volume by 90%+ while keeping essentially all diagnostic value. This is the single largest win available.

2. Fix your cardinality. Find the metrics with the most series. There will be one or two that dominate.

The usual culprits: user ID as a metric label, URL path with IDs in it (/user/12345/profile instead of /user/:id/profile), and error messages as labels.

Metrics are for aggregates. Events are for specifics. A user ID belongs in a structured event where it costs one field, not in a metric label where it multiplies the series count.

3. Drop the logs you never query. Audit what you actually search. Most organizations find that a large fraction of log volume is from a handful of noisy sources nobody has ever looked at — health check logs, framework debug output, successful-request logs that duplicate a metric.

Route those to cheap object storage rather than an indexed store, or drop them.

4. Tier your retention. You do not need 90 days of everything.

• Metrics: long retention, they are small when aggregated.
• Traces: short, days. You investigate recent things.
• Logs: short and hot for search, long and cold in object storage for compliance.

5. Aggregate at the source. Emit a histogram, not a thousand individual timing events. Most agents can pre-aggregate and it moves cost from the vendor to your own process, where it is much cheaper.

what not to cut#

Be careful here, because the failure mode of aggressive cost reduction is finding out during an incident that you deleted the thing you needed.

Keep all errors. Never sample errors. They are rare and they are the whole point.

Keep the request-level events for your critical journeys. One wide structured event per request, with all the context, is the highest value telemetry you have per byte. It is cheaper than the interior logging it replaces.

Keep enough cardinality to slice by customer and region. Aggregate-only metrics hide the failures that matter — "one tenant is completely broken" looks identical to "everyone is slightly slower" in a global average.

the structural fix#

Move from logs to events. The expensive pattern is many log lines per request, each a string, indexed for full-text search.

The cheap pattern is one structured event per unit of work, with everything you know attached as fields, queried by field rather than by text.

Fewer records, more information per record, dramatically cheaper to store and query. This is the change that actually fixes the cost problem rather than trimming it.

Own your pipeline. An open telemetry collector between your services and your vendor lets you filter, sample, aggregate, and route without changing application code, and without being locked into one destination's pricing model.

That is worth setting up before your bill is a problem, because doing it under cost pressure means making decisions in a hurry.

the question to ask quarterly#

For each telemetry stream: when did we last use this to answer a question?

Anything nobody has queried in a quarter is a candidate for deletion or cold storage. Run the audit. The answer is usually uncomfortable and the savings are usually large.