o3 and o4-mini put tools inside the reasoning loop
The models can now search, run Python, and look at images while they think. That's an architecture change, not a feature.
OpenAI released o3 and o4-mini this week. The benchmark numbers are strong. The architectural change is more interesting: these models can call tools during the reasoning process rather than before or after it.
why that ordering matters#
Previously the loop looked like this: the model thinks, decides to call a tool, stops, the tool runs, the result comes back, the model starts a new turn. The reasoning chain is broken at each tool boundary.
Now the tool call happens inside the chain of thought. The model can search the web, read the result, keep reasoning, run some Python to check a calculation, notice the result contradicts its assumption, back up, and try something else — all within a single response.
That is a qualitatively different capability. It turns the model from something that reasons about static context into something that can investigate.
The most striking demonstration is image manipulation during reasoning: the model can crop, rotate, and zoom into a picture as part of working out what it shows. Give it a photograph of a whiteboard at an angle and it will straighten and enlarge the region it needs.
the practical impact#
For agent builders this collapses a lot of orchestration you used to write yourself. The ReAct-style loop — think, act, observe, repeat — was scaffolding around a model that could not do it natively. Increasingly you do not need the scaffolding.
That is worth planning for. If your product's differentiation is an agent framework that manages tool-calling loops, the model is going to absorb that layer. This has happened repeatedly: function calling absorbed the output-parsing libraries, structured outputs absorbed the JSON-repair libraries, and now in-context tool use is absorbing the loop.
The durable layer is not orchestration. It is your data, your evaluations, your domain constraints, and your interface.
the cost shape#
Reasoning models with in-loop tool use have wildly variable cost per request. A simple question is cheap. A hard question where the model searches nine times and runs Python four times is not.
Two consequences:
- Your unit economics need a distribution, not an average. The p99 request can be twenty times the median. If you priced on the median you have a problem.
- You need a timeout and a budget cap, enforced by you, not by hope. A runaway reasoning loop on a pathological input is a real failure mode.
o4-mini specifically#
The cost-efficiency story. Very strong performance on math and coding relative to its price, and high rate limits. For a lot of production workloads this is the correct default, with o3 reserved for the requests that genuinely need it.
The routing logic between them is worth building explicitly. A simple classifier on the incoming request — or even a heuristic on input length and detected task type — that picks the model will save more money than any prompt optimization.
the caveat everyone should say out loud#
Higher capability on reasoning benchmarks does not mean fewer hallucinations. In some evaluations these models produce more confident claims because they have reasoned their way to them. A wrong answer with a fifteen-step justification is harder to catch than a wrong answer without one.
Verification does not get easier as the models get smarter. It gets harder.
— Dom, April 17, 2025