Here is where the time actually goes, roughly in order of how often each is the dominant cause.

1. everything talks to a real database#

The most common cause by a wide margin. Each test sets up a database, inserts fixtures, runs, and tears down.

Fixes, in order of return:

Roll back instead of truncating. Wrap each test in a transaction and roll it back. Orders of magnitude faster than deleting rows, and it needs no cleanup code.

Share the schema, not the data. Create the schema once per run, not per test.

Use a tmpfs for the test database. The data does not need to survive; durable writes are pure cost. On Postgres, a data directory in memory plus fsync=off and synchronous_commit=off is dramatically faster and completely inappropriate for anything but tests.

Move the logic out of the database's reach. The deepest fix: if business logic is a pure function of its inputs, its tests do not need a database at all. Tests that are hard to write without infrastructure are telling you about your design.

2. the suite is serial#

Most runners parallelise and most projects have not turned it on, usually because it exposed shared state once and someone reverted it.

That shared state is a bug. Tests that pass only in a particular order have an ordering dependency, and ordering dependencies in tests usually mirror a real one in the code.

Turn on parallelism, fix what breaks, and you will find at least one genuine issue.

3. sleeps#

time.sleep(2)   # wait for the worker to pick it up

Every one of these is pure latency, and they are always tuned to the slowest machine anyone has run on. Fifty of them is a hundred seconds of doing nothing.

Replace with polling on the actual condition:

wait_until(lambda: job.reload().status == "done", timeout=5)

Same worst case, typically a hundredth of the elapsed time, and it fails with a useful message instead of a mysterious assertion.

4. the fixtures are enormous#

A test that needs one user builds an organisation, three teams, forty users and a year of history, because it reuses the "standard" fixture.

Build the minimum. Factories with sensible defaults, overridden per test, beat shared fixture files that grow to serve every case.

5. it is doing real network I/O#

Tests hitting real HTTP endpoints — even internal ones, even mock servers over a socket — pay connection setup and scheduling per call.

Intercept at the client layer rather than the network layer. Most languages have a way to stub the HTTP client in-process, and it is both faster and more deterministic than a local server.

The exception is contract tests, which exist to catch exactly what stubs hide. Keep those, run them separately, and do not let them into the fast suite.

6. everything is an end-to-end test#

Browser-driving tests are two to three orders of magnitude slower than unit tests. A suite made mostly of them is slow no matter what you do.

The ratio that works: a large number of fast unit tests, a moderate number of integration tests around real boundaries, and a small number of end-to-end tests covering the handful of journeys that must never break.

Inverting that ratio is the most expensive testing mistake a team can make, and it is usually the result of not trusting the lower layers rather than a deliberate choice.

the measurement first#

Before fixing anything, get the distribution:

pytest --durations=25
go test ./... -json | jq -r 'select(.Action=="pass") | "\(.Elapsed) \(.Test)"' | sort -rn | head -25

Almost always, a small number of tests dominate. Fixing the slowest twenty is usually the whole job, and it is an afternoon rather than a project.

the target#

Under 10 seconds for the unit suite, run on every save. Under two minutes for everything that gates a merge.

The ten-second number is not arbitrary — it is roughly the threshold past which people stop running tests as they work and start batching. Once that happens you have lost the feedback loop, and the suite's speed stops being a convenience question and starts being a quality one.

The technique has been around since QuickCheck in 1999 and has stayed niche. It fits the current moment unusually well, and it is worth another look.

the idea#

Instead of "for this input, expect this output," you state a property that should hold for all inputs, and the framework generates hundreds of inputs trying to break it.

from hypothesis import given, strategies as st

@given(st.lists(st.integers()))
def test_sort_is_idempotent(xs):
    assert sorted(sorted(xs)) == sorted(xs)

@given(st.lists(st.integers()))
def test_sort_preserves_elements(xs):
    assert sorted(xs).count == xs.count   # same multiset

@given(st.text())
def test_roundtrip(s):
    assert decode(encode(s)) == s

The framework generates empty lists, single elements, lists with duplicates, huge values, negative numbers, and — critically — when it finds a failure, it shrinks the input to the minimal case that still fails.

That shrinking is what makes the technique practical. A failure on a 400-element list is not actionable. The same failure shrunk to [0, 0] tells you exactly what is wrong.

the properties that are actually useful#

The hard part is not the tooling. It is identifying properties, and there is a small catalogue that covers most cases.

Round trip. decode(encode(x)) == x. Serialization, compression, parsing, encryption. This one property catches an enormous number of real bugs and applies almost everywhere.

Invariants. Something that is always true regardless of operations. A balanced tree stays balanced. A sorted list stays sorted. Account balances sum to the same total across a transfer. A cache never returns stale data past its TTL.

Oracle. Compare against a simpler, slower, obviously-correct implementation. You have an optimized version; write the naive one, and assert they agree. This is extremely powerful and underused — it is how you test the fast path against the version you can reason about.

Idempotence. f(f(x)) == f(x). Normalization, deduplication, and — importantly — any API operation that claims to be idempotent. If you have idempotency keys, this is how you actually verify them.

Commutativity and associativity. Order should not matter. Merge operations, set operations, CRDT merges.

Metamorphic relations. When you cannot state the correct output, state how the output should change. Adding an item to a cart should increase the total by that item's price. Searching for a more specific query should return a subset. Sorting descending should be the reverse of sorting ascending.

This last category is the one that unlocks property testing for business logic, where there is no obvious oracle.

where it fits best#

Parsers and serializers. Round trip, always.

Data structures. Invariants after every operation sequence.

State machines. Generate random valid operation sequences, assert the invariants hold throughout. This finds ordering bugs that no hand-written test would.

Anything with an obvious naive implementation. Oracle testing.

Financial and unit arithmetic. Rounding, currency, conversions. The edge cases are numerous and boring, which is exactly what a generator is for.

Concurrent code, with a framework that generates interleavings. This is the hardest category to test any other way.

where it does not fit#

UI. The properties are aesthetic.

Anything where the correct output requires human judgment.

Integration tests against real systems. Generation implies many runs, and many runs against a real database is slow.

Code where you cannot state a property. If you genuinely cannot, that may itself be information about the design — code with no statable invariants is code with no contract.

the practical advice#

Do not replace example tests. Keep them. They document intent and they are the fastest way to communicate what a function is for. Add properties alongside.

Start with round-trip properties. Easiest to state, highest hit rate for real bugs.

Save the failing seed. When a property fails, the framework gives you the counterexample. Add it as a regression example test so it is checked deterministically forever.

Bound the input space. Unbounded generation produces absurd inputs and slow tests. Constrain to realistic ranges: strings up to a reasonable length, numbers in a plausible domain.

Run more cases in CI than locally. A hundred examples locally for fast feedback, a thousand in CI, ten thousand in a nightly run.

why now#

Two reasons this fits the current moment.

Verification is the bottleneck. Property testing is verification you write once and that checks a large input space forever. That is exactly the leverage the moment calls for.

Stating a property is a task worth doing carefully by hand, and generating the implementation is not. Writing "the total must always equal the sum of line items" requires understanding the domain; nothing else in the test does.

That is a good division of labor, and it points at where human effort should concentrate: on specifying what must be true, and letting everything else be derived.

That was an accurate assessment of the tools at the time. The tools are substantially different now and the assessment has not updated.

what changed#

Flow-sensitive analysis became standard. Older linters matched patterns in the syntax tree. Modern analyzers track values through the control flow graph, which means they can tell that a variable was checked for null on line 12 and therefore is not null on line 40. That single capability eliminates the largest source of false positives.

Language servers made it interactive. A warning in your editor while you type is a different product from a report generated in CI. You fix it in context, in seconds, instead of triaging a list a week later.

Type systems absorbed much of it. A lot of what static analysis used to catch is now caught by the compiler in a typed language, for free, with no false positives at all.

The defaults got sane. Modern tools ship with a curated recommended set rather than everything enabled. clippy, ruff, biome, staticcheck and their peers are opinionated about what is worth reporting.

They got fast. Analyzers written in compiled languages run over a large codebase in seconds. Speed matters more than people credit — a check that takes two minutes gets run in CI, and a check that takes two seconds gets run on every save, which is where it actually changes behavior.

what to actually run#

A fast linter with a good default set, on save, in the editor. ruff for Python, clippy for Rust, biome or eslint for JavaScript, staticcheck for Go.

A typechecker in strict mode, in CI. This is the highest-value item on the list for a gradually-typed language, and the non-strict modes permit exactly the holes that make the guarantees unreliable.

A security-focused analyzer if you handle untrusted input. Taint tracking — does data from a request reach a SQL query, a shell command, or a template without sanitization — is the specific capability worth having, and it is a genuinely different analysis from ordinary linting.

A dependency scanner, ranked by reachability if your tooling supports it. A critical CVE in code you never call is lower priority than a medium in your request path, and a scanner that cannot tell you which is which produces a queue nobody reads.

the adoption sequence#

Turning on a full rule set against an existing codebase produces thousands of warnings and gets the tool disabled. The sequence that works:

1. Run it in report-only mode. Get the number. Do not fix anything yet.

2. Enable a small subset that has near-zero false positives and fix those. Usually: unused variables, unreachable code, obviously wrong comparisons, missing awaits. Twenty rules, not four hundred.

3. Make it blocking for new and changed code only. Most tools support this, either natively or through a diff-aware wrapper. This is the key move — the existing violations do not block anyone, and the codebase stops getting worse immediately.

4. Burn down the backlog opportunistically. When you touch a file, fix its warnings. No cleanup sprint, no dedicated project.

5. Add rules gradually, one at a time, each with a burn-down.

Steps three and four are where most adoptions succeed or fail. A tool that blocks the whole team on a pre-existing backlog gets turned off; one that only blocks new violations is uncontroversial.

the rules worth arguing about#

Some checks are genuinely contested and you should decide deliberately rather than accepting the default:

Cyclomatic complexity limits. Sometimes a function is legitimately complex because the domain is. A hard limit produces artificially split functions that are harder to read, not easier.

Line length. Real disagreement, formatter should handle it, not worth a rule.

Naming conventions. Worth enforcing, and pick your convention rather than the tool's default if they differ.

Anything with more than a few percent false positives. A rule that is wrong one time in ten trains people to ignore it, and that habit generalizes to the rules that are right.

the honest limits#

Static analysis finds a specific class of bug: local, syntactic, pattern-matchable. It does not find logic errors, wrong business rules, race conditions in most cases, or performance problems.

It is not a substitute for tests, review, or thought. It is a way to spend zero human attention on the errors that do not require human attention, which frees attention for the ones that do.

That framing — attention allocation rather than bug finding — is the one that makes it worth the setup.

the new reason it matters#

Machine-generated code has a characteristic error profile: plausible, syntactically valid, and wrong in specific recurring ways. Unchecked errors, missing awaits, resource leaks, off-by-one in boundary conditions.

Those are exactly the errors static analysis is good at. Running a strict analyzer over generated code is verification you get for free, and it is one of the few places where the verification bottleneck has an automated answer.

Turn it on.

This is the highest-return day of engineering available to anyone building on models, and a surprising number of teams have not done it.

what it is not#

Not MMLU. Not a leaderboard. Not "vibes after twenty prompts."

Public benchmarks tell you which models are worth testing. They do not predict performance on your task, because your task is not in them and because contamination is universal.

what it is#

Fifty to two hundred examples from your actual production traffic, with expected outputs or a grading rubric, run automatically, producing a number.

evals/
  cases/
    001-refund-request.json
    002-ambiguous-address.json
    ...
  run.py
  results/
    2026-05-27-model-a.json

Each case:

{
  "id": "042",
  "input": { "ticket": "my order never arrived and I want my money back" },
  "expect": { "category": "refund", "urgency": "high", "needs_human": false },
  "notes": "the word 'never' should not trigger the fraud path"
}

That is it. The whole thing is a test suite where the assertions are fuzzier.

where the cases come from#

Your production failures. This is the best source by a wide margin. Every time the system gets something wrong, that becomes a case. Your eval set grows into a precise map of your problem's difficulty.

Set this up as a workflow: a thumbs-down in the product, or a support escalation, creates a candidate case that someone reviews and adds.

Your edge cases. The weird inputs. The empty ones. The ones in another language. The adversarial ones. The ones with an injection attempt.

A stratified sample of normal traffic. So you notice when a change breaks the common case while fixing an edge case.

Cases with no correct answer. Where the right behavior is to refuse, escalate, or ask a clarifying question. Models are frequently bad at this and it is rarely tested.

grading#

Three approaches, and you will use all three.

Exact or structural match. For classification, extraction, and structured output. Cheap, deterministic, unambiguous. Use it wherever you can.

Programmatic checks. For generated code: does it compile, do the tests pass. For SQL: does it run, does it return the right shape. This is the strongest form of grading and it is available more often than people realize — if you can verify mechanically, do.

Model-as-judge. For open-ended output. A second model grades against a rubric.

Use it carefully:

• Write a specific rubric, not "is this good." Score each dimension separately.
• Validate the judge against human ratings on a sample. If the judge disagrees with you, the judge is wrong and the rubric needs work.
• Use a different model than the one being evaluated, or at minimum be aware of self-preference bias, which is well documented and large.

the metrics#

Accuracy on your set, obviously.

Cost per case. Tokens in and out, at current prices. Track this — a model that is 2% better and 4× the cost is usually the wrong choice.

Latency, at percentiles. p50 and p95. Reasoning models have high variance and the average hides it.

Failure mode distribution. Not just how many wrong — how wrong. A model that fails by refusing is very different from one that fails by confidently fabricating, and the aggregate score treats them identically.

the workflow#

python evals/run.py --model model-a --model model-b --parallel 8

Output a table. Commit the results. Diff them across runs.

Run it:

• When any model ships. Within an hour of the announcement, you know.
• When you change a prompt. Prompt changes are code changes with no type system and no compiler; the eval is your only regression check.
• On a schedule. Providers update models behind stable names. Behavior drifts. You want to know from your dashboard, not from your support queue.

That last one catches something most teams never notice: the model you deployed against is not the model serving your traffic today.

the one that matters most#

Version your prompts and store the eval result with them.

A prompt is production configuration. It should be in version control, reviewed, and associated with a measured quality number. "Someone edited the prompt and something got worse three weeks ago" is a debugging session that should not be possible.

the payoff#

Once this exists:

• Model migrations are an afternoon.
• Prompt changes are safe to make.
• You can argue about model choice with data instead of anecdotes.
• You detect provider-side drift.
• Onboarding a new engineer to the AI parts of your system means handing them the eval set, which is the best available documentation of what the system is supposed to do.

It is a day of work. It is the single highest-leverage day available in this space and it has been for three years.

The only way to find out is to run it, and the only good time to run it is when nothing is actually broken.

what a game day is#

A scheduled exercise where you deliberately break something in a controlled way and have the on-call rotation respond as if it were real.

Two hours. A stated scenario. The people who would actually respond. Real tools, real runbooks, real dashboards. Someone taking notes on everything that did not work.

That is the whole practice. It is not chaos engineering in the automated-random sense, though that is a good adjacent practice. It is a rehearsal.

what you find, reliably#

I have never run one of these that did not find at least four of the following:

The runbook references something that does not exist. A dashboard that was renamed, a script in a repository that was archived, an alias nobody has.

Nobody has the access. The runbook says to restart the service. The on-call engineer does not have permission and does not know who does. This is the single most common finding.

The dashboard does not show the thing. You built the alert. You never built the view that tells you what to do about it.

The escalation path is a person, not a rotation. "Ask Sarah." Sarah is on vacation. Sarah left last year.

Nobody knows the customer impact. The service is degraded. Which customers? Which features? Nobody can answer, so nobody can decide how urgent it is.

Recovery has an undocumented step. The service restarts and does not work because a cache must be cleared first, which is knowledge that lives in one person's head.

The communication path is unclear. Who tells the customers? Who updates the status page? Who has the credentials for the status page?

Every one of those is cheap to fix on a Tuesday afternoon and extremely expensive to discover at 3 a.m.

the scenarios worth running#

Start with the boring ones. Exotic failure modes are fun and the common ones are what actually happens.

The database primary fails over. Does the application reconnect? How long? Does anything need a manual restart?

A dependency returns 500s. Not down — erroring. Does your circuit breaker work? Does your retry policy make it worse?

A dependency gets slow. Harder than down and much more common. Does your timeout fire? Do connections exhaust? Does the slowness propagate to callers?

Disk fills up. On the database host, on the log host, on the application host. This is the single most common self-inflicted outage.

A bad deploy. Deploy something broken to staging and time the rollback. Not the theoretical rollback — the actual one, executed by the actual on-call person.

Certificate expiration. Set one to expire in staging. Watch what happens. Most teams discover their monitoring does not cover this.

The person who knows is unavailable. Run a game day where the subject matter expert is explicitly not allowed to help. This finds the knowledge concentration problems that nothing else does.

how to run one that works#

Announce it. Do not surprise people. Surprise exercises generate resentment and teach people to distrust the process. Everyone should know it is a drill.

Staging first, production eventually. Staging finds most of the runbook problems. Production finds the ones that only exist because staging is not production, which are real and are the ones that matter most.

Have a stop condition. A named person who can call it off, and a defined way to revert whatever you broke.

Write down every friction point, including small ones. "It took four minutes to find the right dashboard" is a real finding.

Fix things within a week. A game day that produces a list nobody acts on is theater, and the second one will have lower attendance.

Do it quarterly. Systems change. A runbook validated a year ago is a runbook that has not been validated.

the cultural part#

The purpose is to find gaps in the system, not gaps in people.

If someone cannot resolve the scenario, that is a finding about documentation, tooling, or access — not about them. Say this explicitly before you start, and mean it, or people will optimize for looking competent rather than for surfacing problems.

The best outcome of a game day is a long list of things that went wrong, discovered by people who were not stressed, on a schedule, with time to fix them.

what automated tooling catches#

Roughly a third to a half of real accessibility issues are detectable by a linter or a test:

• Images without alternative text.
• Form inputs without associated labels.
• Insufficient color contrast.
• Missing document language.
• Heading levels that skip.
• Buttons and links with no accessible name.
• ARIA attributes that are invalid or applied to the wrong role.
• Interactive elements not reachable by keyboard.
• Duplicate or missing landmark regions.

Every one of those is a fixed rule with a fixed check, and every one of them is a real barrier for someone.

Getting the automatable third fixed is not the whole job and it is a much better place than most applications are.

the setup#

// component test
import { axe } from 'vitest-axe';

test('checkout form has no automatically detectable a11y violations', async () => {
  const { container } = render(<CheckoutForm />);
  expect(await axe(container)).toHaveNoViolations();
});

Plus a linter for the static cases:

{ "extends": ["plugin:jsx-a11y/recommended"] }

Plus a crawl of key pages in CI against the built site.

That is an afternoon of setup. It will find violations on day one — every codebase has them — so run it in report-only mode first, fix the backlog, then make it blocking. Making it blocking with a hundred existing violations gets it disabled within a week.

what tooling cannot catch#

Being honest about the limits matters, because "we run axe" becomes a claim of compliance that it does not support.

**Whether the alt text is good.** alt="image" passes. It is useless. A screenshot of a chart needs a description of what the chart shows, not the word "chart."

Whether the reading order makes sense. CSS can visually reorder content while the DOM order stays wrong, and a screen reader follows the DOM.

Whether focus management works. Open a modal — where does focus go? Close it — does it return? Navigate to a new route — is focus reset and announced? These are the most common real-world failures in single-page applications and no automated check catches them.

Whether dynamic content is announced. A live region that updates too often is worse than one that does not update at all.

Whether it is actually usable. A page can pass every automated check and be completely impractical to navigate.

the manual checks worth doing routinely#

Three, each a few minutes, on every significant feature:

1. Unplug the mouse. Navigate the entire flow with Tab, Shift+Tab, Enter, Space, arrows, and Escape. Can you complete the task? Can you always see where focus is? Does focus ever get trapped, or jump somewhere unexpected?

This single test finds more real problems than any automated tool.

2. Zoom to 400%. Browser zoom, at a 1280px viewport. Does content reflow, or does it require horizontal scrolling? Is anything cut off? This is a WCAG requirement and it is failed constantly.

3. Turn on a screen reader for five minutes. VoiceOver on macOS (Cmd+F5), NVDA on Windows. You will be bad at it and that is fine — you are not evaluating your skill, you are listening to whether your page announces anything coherent.

Most developers who do this once are permanently changed by it, because the experience of hearing your own interface read aloud as "button, button, button, link, clickable" is clarifying.

the framing that gets it prioritized#

Not "it is the right thing to do." That is true and it does not survive a prioritization meeting.

It is a legal requirement in many jurisdictions, with an enforcement trend that is going up rather than down, and in the EU the European Accessibility Act now applies to a broad category of consumer-facing digital services.

It is a larger market than most product managers assume. Roughly one in six people has a disability. Not all of those affect software use; enough do.

It overlaps with quality generally. Semantic HTML, keyboard support, and clear focus states make interfaces better for everyone. Keyboard shortcuts are an accessibility feature that power users love.

It is much cheaper to build in than to retrofit. An audit that finds two hundred issues in a shipped product is a quarter of remediation work. Catching them at the component level costs minutes.

the single highest-value practice#

Use semantic HTML.

A <button> is focusable, keyboard-activatable, announced as a button, and works with every assistive technology, for free. A <div onclick> requires you to add a role, a tabindex, key handlers for Enter and Space, and focus styles — and you will get one of them wrong.

The overwhelming majority of accessibility problems in modern web applications come from reimplementing native elements badly. Use the element. It already works.

They also multiply your state space by two for every flag you add, and almost nobody accounts for that.

the arithmetic#

Twenty boolean flags is 2^20 possible configurations — over a million. Your test suite exercises the default configuration. Your staging environment exercises one other. Production is running dozens simultaneously, because different flags are on for different user segments.

Which means: most of the configurations your users are running have never been executed anywhere before.

Most of the time this is fine, because most flags are independent. The failures come from the ones that are not, and you find out about those from a support ticket.

the failure modes#

Interaction bugs. Flag A changes the data format. Flag B reads that data. Both work alone. Together, one of them is reading a shape it does not expect. This is the classic and it is very hard to catch because neither change is wrong.

Flag debt. A flag that has been at 100% for a year is still in the code, with its dead branch, and nobody remembers whether it can be removed. The dead branch does not compile-error and does not test-fail; it just sits there being wrong.

The stale branch. Once a flag is at 100%, the disabled path stops being exercised. Six months later someone flips it as a rollback and discovers the old path no longer works with the current schema. Your kill switch is broken and you find out during an incident.

Configuration drift. Flags set differently in staging and production means staging tests a configuration nobody runs.

the rules that keep this manageable#

Every flag has an owner and an expiry date at creation. Not a suggestion — a required field. A flag that has passed its expiry shows up in a report and somebody has to either extend it with a reason or delete it.

Flags are deleted, not left at 100%. The cleanup is part of the work, not a follow-up ticket. A flag rollout is not done when it reaches 100%; it is done when the flag and the dead branch are gone.

Flags never nest. If flag A only means something when flag B is on, you have created a configuration nobody can reason about. Combine them into one flag with three states, or restructure.

Kill switches are a separate category with separate rules. They live forever, they are documented, and they are tested on a schedule — a quarterly exercise where you flip each one in staging and verify it works. An untested kill switch is not a kill switch.

Test both branches. Your test suite should run the critical path with each significant flag both on and off. Not the full combinatorial space — that is impossible — but each flag independently against the default configuration. That catches the "old path rotted" failure, which is the expensive one.

the taxonomy that helps#

Four kinds of flag, with different lifecycles:

The first two must expire. The last two must be documented and tested. Conflating them is how you get a thousand flags and no idea which matter.

the observability part#

Every event you log should carry the flag configuration that produced it.

{ "event": "checkout", "status": 500, "flags": ["new_pricing", "fast_path"] }

Without this, a bug that only affects one flag combination is undiagnosable — you see errors, you cannot correlate them to anything, and you spend a day guessing. With it, the correlation is a single query.

This is a fifteen-minute change and it is the single highest-value thing you can do if you use flags at all.

the counterargument#

Some teams respond to all of this by using fewer flags and doing more trunk-based deployment with fast rollback.

That is a legitimate position and it works when your rollback is genuinely fast and your changes are genuinely reversible. Flags are a tool for when they are not — when a change is expensive to revert, when you need per-segment control, or when release timing must be decoupled from deployment for business reasons.

Use them for those. Do not use them because they feel safer, because a flag you do not test is not safety, it is the appearance of it.

Generation got roughly two orders of magnitude cheaper. Verification did not get cheaper at all.

Everything else — the review bottleneck, the arguments about junior hiring, the sudden importance of tests, the unease people cannot articulate about agent-written code — is downstream of that asymmetry.

why verification did not get cheaper#

You might expect it to. Surely a model can check code as well as it can write it?

It can, sort of, and not in the way that matters. Three problems:

The checker has the same blind spots as the writer. A model that does not know your system produced code with an assumption baked in. The same model reviewing it holds the same assumption and does not see it. Independent errors can be filtered by voting; correlated errors cannot.

Verification requires knowing what correct means. That is a property of your domain, your users, your obligations — not of the code. A model can verify that code does what it says. It cannot verify that what it says is what you needed, because that information was never written down anywhere.

Confidence is decoupled from correctness. Generated code is fluent. Fluency is the signal humans use to assess competence, and it has been severed from correctness. Reviewing text that reads well but is subtly wrong is much harder than reviewing text that reads badly, and everyone underestimates this.

what this changes#

Tests stop being a chore and become the primary artifact. If verification is the constraint, then anything that makes verification automatic is worth enormous investment. A test suite is executable verification. It was always valuable; it is now the thing that determines your throughput.

The practical implication: write tests first, review those carefully, and let the implementation be cheap. Invert the effort. The specification is the part you have to get right; the code is increasingly a derived artifact.

Type systems get a promotion. Every constraint the compiler can check is a constraint you do not have to verify by reading. Languages with expressive type systems and strong static guarantees are worth more than they were, because they convert human verification into machine verification.

I have watched teams that were ambivalent about strict TypeScript become evangelists in the space of a year, and the reason is always the same: it catches the class of error that agent-written code produces most often.

Small diffs become non-negotiable. Verification cost scales superlinearly with diff size, because the number of interactions you have to reason about grows faster than the lines. A machine can produce a 2,000-line change effortlessly. Accepting it is not a favor to anyone.

Property-based testing gets its moment. If you can state an invariant, you can verify a very large input space cheaply. Invariants are exactly the kind of thing that a human should specify and a machine should check. This technique has been niche for twenty years and it is the right shape for this moment.

Observability becomes a verification tool. If you cannot verify everything before deploy, you verify in production: feature flags, canaries, and metrics that detect wrongness quickly. Fast detection is a substitute for perfect pre-verification, and it is often a better investment.

the skill that matters#

The engineers getting genuine leverage out of these tools all share one trait, and it is not prompting.

It is that they can look at a plausible-looking piece of code and say "that retry loop will hammer the upstream on a 429" or "that will deadlock under concurrent writes" or "that assumes the list is sorted and nothing sorts it."

That skill comes from having debugged those exact failures. It does not come from reading about them.

Which produces the uncomfortable question everybody is circling: the traditional way to acquire it was to write a lot of code badly and then fix it. If that apprenticeship is being automated away, what replaces it?

the answer I have, which is partial#

Deliberately do the hard part yourself sometimes.

Not out of nostalgia. Because the judgment is the product, and the judgment is built by doing. If you delegate every debugging session, you will be worse at debugging in two years, and debugging is the thing you are being paid for now.

Pick the gnarliest bug of the week and do it by hand. Read the code the agent wrote in the module you own, all of it, once a month. Write the tricky concurrency code yourself and let the machine do the CRUD.

That is not a workflow recommendation for efficiency. It is a training regimen, and treating it as one is the honest framing.

That signal is available for free, it is extremely reliable, and most teams respond to it by making the tests worse instead of the design better.

the tells#

You need six mocks to test one function. The function has six dependencies. That is the actual problem. The mocks are the messenger.

You are testing private methods. The class is doing more than one thing and you know it, which is why the interesting behavior is not reachable from the public interface. Extract the thing you want to test into its own unit with its own public interface.

Setup is longer than the assertion. The unit under test requires a large amount of world to exist before it does anything. That is coupling, expressed in lines of setup code.

You cannot test it without a database. Sometimes true and unavoidable. Usually it means business logic and persistence are interleaved, and separating them would make both testable and both clearer.

The test breaks whenever you refactor. You are asserting on implementation rather than behavior. Either the test is bad, or the unit has no meaningful behavioral contract, which is worse.

You cannot describe what the test verifies in one sentence. Then neither can the code.

the mechanism#

Tests are the first consumer of your interface. They are the only place, before production, where you have to actually use the thing you built rather than look at it.

Everything that is awkward about consuming your interface shows up in the test as friction: too many parameters, unclear ordering, hidden global state, temporal coupling between calls, error conditions that cannot be triggered deliberately.

This is why "test-first" works for people it works for, and it is not because of any ritual. It is because writing the usage before the implementation forces you to design the interface from the caller's perspective, which is the only perspective that matters.

You do not have to write tests first to get this. You have to notice when the test is hard and treat that as data.

the common wrong response#

Team hits testing friction. Team adds a mocking framework with more power. Now you can mock static methods, patch constructors, intercept module loading, and rewrite the world.

The friction goes away. The design problem does not. It is now permanently invisible, encased in tooling, and the test suite has become a second implementation of the system that must be maintained in parallel with the first.

Powerful mocking frameworks are a tool for testing legacy code you cannot change. Reaching for one in new code is choosing not to hear the signal.

what good looks like#

def test_order_total_applies_bulk_discount():
    order = Order(items=[Item(price=10_00, qty=100)])
    assert order.total() == 900_00

No mocks. No setup. No database. One sentence describes it. It breaks if and only if the discount rule changes.

That test is easy to write because Order.total is a pure function of its data. Making it a pure function of its data was the design decision. The test just confirmed it was a good one.

the one exception worth stating#

Sometimes the system is genuinely complex and the tests are genuinely complicated, because the domain is. Distributed consensus, a compiler backend, a scheduler. Difficulty is not always a design smell.

The distinguishing question: is the test complicated because the problem is hard, or because getting to the problem is hard? Complexity in the assertions is fine. Complexity in the setup is the smell.