Model evaluation for people who ship

Every few weeks a new model ships and someone asks whether you should switch. Without an evaluation harness, answering that takes a week of impressions and you will get it wrong. With one, it takes an hour.

This is the highest-return day of engineering available to anyone building on models, and a surprising number of teams have not done it.

what it is not#

Not MMLU. Not a leaderboard. Not "vibes after twenty prompts."

Public benchmarks tell you which models are worth testing. They do not predict performance on your task, because your task is not in them and because contamination is universal.

what it is#

Fifty to two hundred examples from your actual production traffic, with expected outputs or a grading rubric, run automatically, producing a number.

evals/
  cases/
    001-refund-request.json
    002-ambiguous-address.json
    ...
  run.py
  results/
    2026-05-27-model-a.json

Each case:

{
  "id": "042",
  "input": { "ticket": "my order never arrived and I want my money back" },
  "expect": { "category": "refund", "urgency": "high", "needs_human": false },
  "notes": "the word 'never' should not trigger the fraud path"
}

That is it. The whole thing is a test suite where the assertions are fuzzier.

where the cases come from#

Your production failures. This is the best source by a wide margin. Every time the system gets something wrong, that becomes a case. Your eval set grows into a precise map of your problem's difficulty.

Set this up as a workflow: a thumbs-down in the product, or a support escalation, creates a candidate case that someone reviews and adds.

Your edge cases. The weird inputs. The empty ones. The ones in another language. The adversarial ones. The ones with an injection attempt.

A stratified sample of normal traffic. So you notice when a change breaks the common case while fixing an edge case.

Cases with no correct answer. Where the right behavior is to refuse, escalate, or ask a clarifying question. Models are frequently bad at this and it is rarely tested.

grading#

Three approaches, and you will use all three.

Exact or structural match. For classification, extraction, and structured output. Cheap, deterministic, unambiguous. Use it wherever you can.

Programmatic checks. For generated code: does it compile, do the tests pass. For SQL: does it run, does it return the right shape. This is the strongest form of grading and it is available more often than people realize — if you can verify mechanically, do.

Model-as-judge. For open-ended output. A second model grades against a rubric.

Use it carefully:

• Write a specific rubric, not "is this good." Score each dimension separately.
• Validate the judge against human ratings on a sample. If the judge disagrees with you, the judge is wrong and the rubric needs work.
• Use a different model than the one being evaluated, or at minimum be aware of self-preference bias, which is well documented and large.

the metrics#

Accuracy on your set, obviously.

Cost per case. Tokens in and out, at current prices. Track this — a model that is 2% better and 4× the cost is usually the wrong choice.

Latency, at percentiles. p50 and p95. Reasoning models have high variance and the average hides it.

Failure mode distribution. Not just how many wrong — how wrong. A model that fails by refusing is very different from one that fails by confidently fabricating, and the aggregate score treats them identically.

the workflow#

python evals/run.py --model model-a --model model-b --parallel 8

Output a table. Commit the results. Diff them across runs.

Run it:

• When any model ships. Within an hour of the announcement, you know.
• When you change a prompt. Prompt changes are code changes with no type system and no compiler; the eval is your only regression check.
• On a schedule. Providers update models behind stable names. Behavior drifts. You want to know from your dashboard, not from your support queue.

That last one catches something most teams never notice: the model you deployed against is not the model serving your traffic today.

the one that matters most#

Version your prompts and store the eval result with them.

A prompt is production configuration. It should be in version control, reviewed, and associated with a measured quality number. "Someone edited the prompt and something got worse three weeks ago" is a debugging session that should not be possible.

the payoff#

Once this exists:

• Model migrations are an afternoon.
• Prompt changes are safe to make.
• You can argue about model choice with data instead of anecdotes.
• You detect provider-side drift.
• Onboarding a new engineer to the AI parts of your system means handing them the eval set, which is the best available documentation of what the system is supposed to do.

It is a day of work. It is the single highest-leverage day available in this space and it has been for three years.