Grok 4 and the benchmark that ate the discourse

xAI released Grok 4 last night with claimed state-of-the-art results across several benchmarks, including a striking number on Humanity's Last Exam.

The launch also included a "Heavy" tier that runs multiple agents in parallel and selects among their answers, and a $300/month subscription for it.

reading the numbers correctly#

The headline HLE figure comes from the Heavy configuration with tools enabled. That is a legitimate configuration and it is not comparable to a single-sample number from a competitor, which is how it was presented in most coverage.

There are at least four distinct things being reported as "the score":

1. Single sample, no tools.
2. Single sample, with tools (search, code execution).
3. Consensus of N samples, no tools.
4. Multi-agent parallel with tools and selection.

Number four can be five or ten times the cost of number one. Comparing across these without stating which is being used is not a small methodological quibble. It is the difference between "our model is better" and "we spent more money at inference time."

To be clear: xAI disclosed their configurations. The disclosure was in the livestream and the fine print. The number that traveled was the big one, with no configuration attached, and that is now just how model launches work.

the parallel-agents technique#

Worth taking seriously on its own merits, independent of the marketing.

Running N independent attempts and selecting the best is a well-established test-time scaling method, and it works better than one long chain for a specific reason: independent samples have independent errors, so selection can filter them, while a single chain compounds its errors with no mechanism to recover.

The hard part is selection. If you can verify — the tests pass, the proof checks, the code compiles — selection is easy and this technique is enormously powerful. If you cannot verify, you need a judge, and the judge has the same failure modes as the generator.

This is why the technique works so much better on math and code than on open-ended reasoning. Verifiability is the whole game.

The practical version for your own work: if you have a verifier, sample multiple times and filter. You do not need a special model tier for this. Three samples through a cheap model with a test-suite check will frequently outperform one sample through an expensive one, for less money.

the other thing#

Grok's public-facing behavior in the weeks before this launch included a series of incidents that xAI attributed to a system prompt change. I am not going to recount them; they are well documented and they were bad.

The engineering lesson worth extracting: a system prompt is production configuration. It should be version controlled, reviewed, tested against an adversarial eval suite, and rolled out gradually. Treating it as a text box someone can edit is how you get an incident that makes international news.

If your product has a system prompt in a config file that anyone can change without review, fix that this week.

the state of play#

Grok 4 is a competitive frontier model. xAI went from founded to frontier in roughly two years, which remains the most remarkable thing about the company and is mostly a story about capital and urgency rather than research insight.

The models are converging. The differentiation is moving to price, latency, tooling, and trust, and xAI's position on that last one is self-inflicted.