Llama 4 arrives, and the leaderboard problem gets a name

Meta released Llama 4 over the weekend: Scout (17B active parameters, 16 experts, a claimed 10-million-token context window) and Maverick (17B active, 128 experts), both mixture-of-experts, both released under the Llama community license. A larger Behemoth was described as still training.

Within forty-eight hours the release turned into a story about benchmark integrity instead.

what happened#

Maverick posted a very strong score on LMArena, the human-preference leaderboard. It then emerged that the model evaluated on the arena was an "experimental chat version" tuned for conversationality — not the checkpoint released to the public. LMArena updated its policies and published the disputed comparison. Meta's response was that experimental variants are normal and the arena version was labeled.

Both of those things can be true and the outcome is still bad, because the number that traveled was attached to a model nobody could download.

why this keeps happening#

Leaderboards are the only shared vocabulary the field has, and they are being asked to carry weight they cannot bear.

• Human preference arenas measure whether people like the answer. That correlates with quality and also with formatting, length, confidence, and sycophancy. A model tuned to be agreeable climbs.
• Static benchmarks leak into training data. Every popular benchmark is on the internet and every frontier model has read the internet. Contamination is not always deliberate and it is essentially always present.
• Vendor-run evaluations use vendor-chosen settings. Consensus-of-64 for yours, single-sample for theirs. Both numbers are real and the comparison is not.

There is no fix that survives contact with commercial incentives. The only durable answer is that you have to run your own evaluation on your own task.

the 10 million token claim#

Scout's context window is stated at 10M tokens, achieved through an interleaved attention scheme without positional embeddings in some layers, plus inference-time temperature scaling on attention. It was trained on far shorter sequences and generalizes upward.

Take this as an upper bound on what the architecture accepts, not on what it usefully processes. Independent long-context evaluations found substantial degradation well before that number. That is not unique to Llama 4 — it is true of every long-context claim — but 10M is a big enough number that the gap between accepted and useful is enormous.

what is actually good here#

The MoE architecture with 17B active parameters is a real efficiency story. Maverick's quality-per-active-parameter is strong, and active parameters are what determine inference cost. A model you can serve at 17B economics with much better quality than a 17B dense model is a genuinely useful thing to have.

Native multimodality — images in the base model rather than bolted on via an adapter — is also the right architecture and is where everyone is heading.

The license is still not open source. "Llama community license" has an MAU threshold and naming requirements. Call it open weights, which it is, and do not call it open source, which it is not.

the practical advice#

Build an eval harness with fifty examples from your actual domain. Run every candidate model against it. Store the results in your repo next to your tests.

It will take a day and it will make you immune to this entire genre of news.