Small models ate the middle

The most consequential trend in applied AI over the last eighteen months is not at the frontier. It is that the bottom of the market got good enough for most work.

the shape of it#

Track any capability benchmark across model sizes over time and the pattern is consistent: the frontier improves steadily, and the small-model tier improves faster. The gap between "the best model available" and "a model that costs 2% as much" has been compressing.

The mechanisms are known:

• Distillation. Training small models on the outputs of large ones transfers a surprising amount of capability. The recipes are public.
• Better data. Curated, synthetic, and filtered training data improves small models disproportionately, because they have less capacity to waste on noise.
• Mixture of experts. Total parameters for knowledge, active parameters for cost. The economics of a 30B-total/3B-active model are close to a 3B dense model, and the quality is much closer to a 30B dense one.
• Reasoning post-training. RL on verifiable rewards works on small models. A small model that thinks can outperform a large model that does not, on the tasks where thinking helps.

what it means in practice#

Go through a typical production AI workload and categorize the calls:

The first column is most of the volume. The second is most of the value per call and a small fraction of the calls.

If you are running everything through a frontier model, you are likely spending a large multiple of what you need to, and you probably have not measured which calls actually need it.

the architecture#

                  ┌─→ small model ──→ verifier ──→ ok? → done
request → route ──┤                              └→ no  ─┐
                  └─→ frontier model ←──────────────────┘

Three components and each matters:

The router. Simpler than people build. Input length, detected task type, and a handful of keywords gets you most of the way. A small model as a classifier works too. Do not build a sophisticated router before you have measured that a simple one is insufficient.

The verifier. This is the part that gets skipped and it is what makes the architecture safe. Small models fail confidently. A cheap check — schema validation, a range assertion, running the generated code, a second model asked "is this answer plausible" — catches the failures that would otherwise reach production silently.

The escalation path. When verification fails, retry with the frontier model. Track the rate. If it climbs, something changed.

the instrumentation that matters#

Three metrics, on a dashboard:

1. Escalation rate. The fraction of requests that fall through to the expensive path. This is the health metric for the whole design.
2. Cost per successful task, not per token. Tokens are an implementation detail; tasks are the unit you care about.
3. Quality on your eval set, by tier. Run both tiers against the same evaluation regularly. When a new small model ships, you will know within an hour whether you can move work down.

the second-order effect#

When inference is nearly free, you use more of it.

Things that were not worth a model call become worth it: normalizing an input, double-checking an output, generating three candidates and picking one, enriching a record, summarizing an intermediate step.

The systems being built now use dramatically more model calls per user action than the systems from two years ago, at lower total cost. That is Jevons operating on schedule, and it is why "our AI costs went down per token and up in total" is the normal experience.

the thing to watch#

The interesting question is not whether small models keep improving. They will.

It is whether the frontier keeps being worth its premium. If the gap on practically-relevant tasks keeps compressing, the frontier tier's addressable workload shrinks to a narrow band of genuinely hard problems.

That is a much smaller business than the one being priced today, and it is the scenario that ought to worry the labs more than competition does.