how it happens#

Logs at debug level in production, because someone turned it on during an incident in 2023 and nobody turned it off.

Every metric at every dimension. A counter with five labels, each with a hundred values, is ten billion time series. Cardinality multiplies and the pricing follows.

100% trace sampling because sampling felt like giving something up.

Retention set to the maximum because nobody knew what to pick and the default was generous.

Duplicate pipelines. Logs going to two vendors during a migration that never completed.

Each decision was locally reasonable. The bill is the sum.

the reduction, in order of return#

1. Sample traces intelligently. You do not need every trace. You need:

• All traces that errored.
• All traces above a latency threshold.
• A small percentage of the rest, for baseline.

Tail-based sampling — decide after the trace completes, when you know whether it is interesting — typically cuts volume by 90%+ while keeping essentially all diagnostic value. This is the single largest win available.

2. Fix your cardinality. Find the metrics with the most series. There will be one or two that dominate.

The usual culprits: user ID as a metric label, URL path with IDs in it (/user/12345/profile instead of /user/:id/profile), and error messages as labels.

Metrics are for aggregates. Events are for specifics. A user ID belongs in a structured event where it costs one field, not in a metric label where it multiplies the series count.

3. Drop the logs you never query. Audit what you actually search. Most organizations find that a large fraction of log volume is from a handful of noisy sources nobody has ever looked at — health check logs, framework debug output, successful-request logs that duplicate a metric.

Route those to cheap object storage rather than an indexed store, or drop them.

4. Tier your retention. You do not need 90 days of everything.

• Metrics: long retention, they are small when aggregated.
• Traces: short, days. You investigate recent things.
• Logs: short and hot for search, long and cold in object storage for compliance.

5. Aggregate at the source. Emit a histogram, not a thousand individual timing events. Most agents can pre-aggregate and it moves cost from the vendor to your own process, where it is much cheaper.

what not to cut#

Be careful here, because the failure mode of aggressive cost reduction is finding out during an incident that you deleted the thing you needed.

Keep all errors. Never sample errors. They are rare and they are the whole point.

Keep the request-level events for your critical journeys. One wide structured event per request, with all the context, is the highest value telemetry you have per byte. It is cheaper than the interior logging it replaces.

Keep enough cardinality to slice by customer and region. Aggregate-only metrics hide the failures that matter — "one tenant is completely broken" looks identical to "everyone is slightly slower" in a global average.

the structural fix#

Move from logs to events. The expensive pattern is many log lines per request, each a string, indexed for full-text search.

The cheap pattern is one structured event per unit of work, with everything you know attached as fields, queried by field rather than by text.

Fewer records, more information per record, dramatically cheaper to store and query. This is the change that actually fixes the cost problem rather than trimming it.

Own your pipeline. An open telemetry collector between your services and your vendor lets you filter, sample, aggregate, and route without changing application code, and without being locked into one destination's pricing model.

That is worth setting up before your bill is a problem, because doing it under cost pressure means making decisions in a hurry.

the question to ask quarterly#

For each telemetry stream: when did we last use this to answer a question?

Anything nobody has queried in a quarter is a candidate for deletion or cold storage. Run the audit. The answer is usually uncomfortable and the savings are usually large.

the shape of it#

Track any capability benchmark across model sizes over time and the pattern is consistent: the frontier improves steadily, and the small-model tier improves faster. The gap between "the best model available" and "a model that costs 2% as much" has been compressing.

The mechanisms are known:

• Distillation. Training small models on the outputs of large ones transfers a surprising amount of capability. The recipes are public.
• Better data. Curated, synthetic, and filtered training data improves small models disproportionately, because they have less capacity to waste on noise.
• Mixture of experts. Total parameters for knowledge, active parameters for cost. The economics of a 30B-total/3B-active model are close to a 3B dense model, and the quality is much closer to a 30B dense one.
• Reasoning post-training. RL on verifiable rewards works on small models. A small model that thinks can outperform a large model that does not, on the tasks where thinking helps.

what it means in practice#

Go through a typical production AI workload and categorize the calls:

The first column is most of the volume. The second is most of the value per call and a small fraction of the calls.

If you are running everything through a frontier model, you are likely spending a large multiple of what you need to, and you probably have not measured which calls actually need it.

the architecture#

                  ┌─→ small model ──→ verifier ──→ ok? → done
request → route ──┤                              └→ no  ─┐
                  └─→ frontier model ←──────────────────┘

Three components and each matters:

The router. Simpler than people build. Input length, detected task type, and a handful of keywords gets you most of the way. A small model as a classifier works too. Do not build a sophisticated router before you have measured that a simple one is insufficient.

The verifier. This is the part that gets skipped and it is what makes the architecture safe. Small models fail confidently. A cheap check — schema validation, a range assertion, running the generated code, a second model asked "is this answer plausible" — catches the failures that would otherwise reach production silently.

The escalation path. When verification fails, retry with the frontier model. Track the rate. If it climbs, something changed.

the instrumentation that matters#

Three metrics, on a dashboard:

1. Escalation rate. The fraction of requests that fall through to the expensive path. This is the health metric for the whole design.
2. Cost per successful task, not per token. Tokens are an implementation detail; tasks are the unit you care about.
3. Quality on your eval set, by tier. Run both tiers against the same evaluation regularly. When a new small model ships, you will know within an hour whether you can move work down.

the second-order effect#

When inference is nearly free, you use more of it.

Things that were not worth a model call become worth it: normalizing an input, double-checking an output, generating three candidates and picking one, enriching a record, summarizing an intermediate step.

The systems being built now use dramatically more model calls per user action than the systems from two years ago, at lower total cost. That is Jevons operating on schedule, and it is why "our AI costs went down per token and up in total" is the normal experience.

the thing to watch#

The interesting question is not whether small models keep improving. They will.

It is whether the frontier keeps being worth its premium. If the gap on practically-relevant tasks keeps compressing, the frontier tier's addressable workload shrinks to a narrow band of genuinely hard problems.

That is a much smaller business than the one being priced today, and it is the scenario that ought to worry the labs more than competition does.

The headline is not the benchmark. It is what the price-performance point makes economically viable.

the pattern across the industry#

Every major provider now ships roughly the same ladder:

The interesting fact is that the small tier's capability is rising faster than the frontier's. Today's small model is roughly where the frontier was eighteen months ago, and it costs about two percent as much.

For anyone doing volume work, that is the number that matters. Not "how smart is the best model" but "how cheap is the model that is good enough for this task."

what it enables#

Agent fleets. If a small model can handle subtasks reliably, an orchestrator using a frontier model can dispatch twenty parallel workers using a small one. Total cost stays reasonable; throughput multiplies. This architecture was uneconomic a year ago and is now obvious.

Real-time interaction. Latency matters more than capability for anything a human is waiting on. A fast model that is 90% as good and 5× faster wins on almost any interactive surface.

Processing everything instead of sampling. Classification, extraction, routing, and enrichment over an entire corpus rather than a sample. At small-model prices, "run it on all of it" becomes the default rather than a budget question.

Pipeline stages that were not worth it. Adding a model call to normalize an input, or to double-check an output, or to summarize an intermediate result — each of these was a cost decision at frontier prices and is now free enough to just do.

the routing architecture#

This is the shape that production systems are converging on:

request → cheap classifier → simple?  → small model → done
                           → complex? → frontier model → done
                           → ambiguous? → small model → verifier → escalate if low confidence

Most requests take the cheap path. The expensive model handles the tail. Overall cost is dominated by the common case, which is cheap.

Two implementation notes that matter:

The classifier can be the small model itself, or often a much simpler heuristic. Do not over-engineer the router — a regex and an input-length check gets you surprisingly far.

Instrument the escalation rate. If it is climbing, either your traffic mix changed or your small-model prompts have degraded. That metric is the health check for the whole architecture.

the caution#

Small models fail differently than large ones. They do not fail less on the tasks they handle — they fail more confidently on the ones just outside their envelope.

A large model asked something beyond its ability will often hedge. A small model will answer, fluently and wrongly.

So: verify the output of small models where correctness matters. A cheap verifier — schema validation, a test run, a range check, a second model asked to critique — costs almost nothing and catches the category of failure that will otherwise reach production silently.

The cost savings are real. They come with a verification obligation, and the teams that skip it will find out in a quarter.

the structural problem#

A subscription product with unbounded variable cost per user is a bet that usage distribution stays roughly log-normal. Most users are light, some are heavy, the average works out.

Agentic AI tools break that assumption in a specific way: the heaviest users are not 10x the median, they are 1000x. An engineer running parallel agents continuously during work hours consumes a genuinely different order of magnitude than someone asking a few questions a day.

At that spread, flat-rate pricing does not work. There is no price that is both attractive to the median user and non-catastrophic for the top percentile. You either subsidize the heavy users from the light ones — which works until the heavy users are a larger share — or you introduce limits.

Everyone in this category is going to hit this. Most already have.

the part that is avoidable#

The economics are not the failure. The communication is.

Here is the pattern that generates anger, which I have now watched play out at four companies:

1. Launch with generous or unstated limits.
2. Users build workflows around the observed capacity.
3. Limits tighten, often announced after users notice.
4. Users discover the limit by hitting it mid-task.
5. The error message does not say when it resets or how much was used.

Every step after the first is a choice.

what good looks like#

Publish the meter. If there is a limit, show consumption against it, continuously, before it is hit. Every cloud provider learned this a decade ago. An unmetered limit is a trap regardless of how generous it is.

Give the number in the units the user thinks in. "You have used 60% of your weekly allowance" is useful. "Rate limit exceeded" is not. Neither is a token count, because nobody has intuition for tokens.

Announce changes before they take effect, with a date. People will be annoyed. They will be much less annoyed than if they find out at 4 p.m. on a deadline.

Degrade, do not cut off. Falling back to a cheaper model with a notice is almost always better than a hard stop. The user's task completes; they learn about the limit; nobody loses work.

Make the expensive thing visible while it is happening. If a request is going to consume a large share of the budget, say so before running it. Users make reasonable decisions when they can see the cost.

the pricing shape that actually fits#

My read is that this category converges on hybrid: a subscription that covers a defined baseline, plus metered usage above it, with a spend cap the user controls.

That is how cloud infrastructure priced itself, after a decade of the same argument, and for the same reason — the underlying cost is genuinely variable and pretending otherwise breaks in both directions. Flat pricing means either the vendor eats unbounded cost or the user hits a wall.

The version I want as a user: a dial that says "spend up to $X this month," a meter that shows where I am, and no surprises. That is not complicated, and it is strange that a category built by extremely sophisticated companies has mostly not shipped it.

the vendor's side, fairly#

Serving these workloads is genuinely expensive and the costs are not well-predictable even to the vendor. Nobody had usage data for agentic coding tools two years ago because they did not exist.

Getting the pricing wrong initially is forgivable. Adjusting it is necessary. Doing it without a meter, without notice, and with an error message that tells the user nothing — that is the part that is just bad product work.

The company's own framing is unusually candid: this is the last of the non-chain-of-thought line, it is better at the things large pretrained models are better at — world knowledge, writing quality, fewer hallucinations, something they describe as improved "EQ" — and it is not designed to compete on math and coding benchmarks against reasoning models.

That framing is correct and it is also an obituary.

what the scaling curve says now#

For most of 2020 to 2023, the answer to "how do we get a better model" was "make it bigger." Scaling laws were the field's organizing principle. GPT-4.5 is what you get when you keep pulling that lever with 2024-era techniques, and what you get is: meaningfully better at some qualitative things, not competitive on the benchmarks people actually optimize for, and expensive enough that the economics are hostile.

Meanwhile, test-time compute — spending inference tokens on reasoning — produces larger gains on hard problems for far less capital. The lever moved.

This does not mean pretraining scale is dead. Reasoning models are built on pretrained base models and a better base makes a better reasoner. It means the marginal dollar goes to post-training and inference compute rather than to another pretraining order of magnitude.

where the big model is actually better#

Worth being fair to it, because the discourse is going to flatten this into "GPT-4.5 flopped."

Large non-reasoning models are genuinely better at:

• Writing that sounds like a person. Reasoning models often produce prose that reads like a report. This one does not.
• Broad factual recall. More parameters means more memorized world.
• Fewer confident fabrications on knowledge questions, per OpenAI's own hallucination evaluations.
• Following a conversation with implicit context over many turns.

If your product is a writing tool, a support agent, or anything where tone and recall matter more than multi-step logic, a bigger base model is the right choice and always was.

the pricing problem#

The price is the story for most developers. At those rates, the set of applications where GPT-4.5 is the correct economic choice is small. It is a model for cases where quality dominates cost, which is a real category and a narrow one.

Expect this to be the pattern going forward: a small number of very expensive models for the top of the quality curve, a large middle of cost-effective workhorses, and cheap small models absorbing everything routine. Route accordingly, and instrument your routing so you know what you are actually spending per feature.

the historical note#

Someone will write a retrospective in a few years that treats February 2025 as the moment the pure-scale thesis visibly stopped being the main event. They will be oversimplifying, because these transitions are always gradual and the narrative is always cleaner in hindsight.

But they will not be wrong about the direction.