The difference is entirely in the setup.

write the question down first#

A spike without a written question becomes exploration, and exploration has no end condition.

The question has to be answerable and specific:

• Bad: "look into whether we can use X"
• Good: "can X ingest 50k events/second on a single node with our event shape, and what does its failure behaviour look like when the disk fills?"

The second one tells you when you are finished, what to measure, and what would count as a no. The first one runs until someone gets bored.

state the decision it feeds#

A spike exists to unblock a decision. Name the decision explicitly: we will use X for the ingest path, or we will stay with Y and shard it.

If you cannot name the decision, you are not running a spike, you are learning something — which is fine and should be called that, because it has a different budget.

set the box, and mean it#

Two days is the usual right answer. Long enough to get past setup, short enough that being wrong is cheap.

The rule that makes it work: when the box ends, you stop and report, even if you are nearly there. "Nearly there" is where spikes go to become three-week projects. If the answer genuinely needs more time, that is itself a finding — report it and ask for a second box with a narrower question.

the output is a document, not a branch#

This is the part teams skip. The deliverable is one page:

## Question
Can X ingest 50k events/s on one node with our event shape?

## Answer
Yes for the steady state, no for our burst profile.
32k/s sustained, 61k/s for ~40s before the write buffer backs up.

## Evidence
Harness: spike/x-ingest/ (throwaway). Ran on c7g.4xlarge, 20 min,
production event sample from 2026-08-10. Numbers: spike/x-ingest/results.md

## What surprised me
Backpressure is silent — it drops rather than erroring, and the drop
counter is only in a debug endpoint. That is a production risk with X
regardless of throughput.

## Recommendation
Stay with Y and shard. Revisit X if their 3.x backpressure work lands.

## What I did not test
Multi-node, recovery after disk-full, or the managed offering.

Six sections, twenty minutes to write, and it is still useful in a year when somebody asks why the decision went the way it did. The "what I did not test" section is the one that keeps the document honest.

throw the code away#

Spike code is written to answer a question, not to be maintained. It has no tests, no error handling, hard-coded credentials and a main function that does everything.

That is correct — the speed comes from those omissions. The failure is letting it become the foundation of the real implementation, at which point you have shipped a prototype and will spend a year discovering what it does not handle.

Put it in a directory named spike/, or a branch you never merge. Say in the document where it is. Then write the real thing properly, informed by what you learned.

The strongest signal that a team's spikes are working: the spike code never ships.

when a spike is the wrong tool#

• When the answer is already written down. Read the documentation, the changelog, the issue tracker. A surprising amount of "we need to try it" is answerable in an hour of reading.
• When the question is about production behaviour under real load. A spike cannot tell you that. A canary can.
• When the decision is reversible and cheap. Then just pick one. A spike to choose between two libraries you could swap in a day costs more than being wrong.

Spikes are for expensive, hard-to-reverse decisions where the deciding information does not exist yet. That is a narrower set than it feels like, and it is exactly where two days is a bargain.

That gap is where most delivery confusion lives, and it is cheap to close.

the honest definition#

A change is done when the outcome it was meant to produce is observable, and the team would find out if it stopped.

Not "the code is merged." Merged code that is not deployed is inventory. Deployed code that nobody looked at is a hypothesis.

That framing sounds demanding and mostly is not, because for a large share of work the observation is trivial — the feature is behind a flag, the flag is on for the team, someone clicked it. The point is that someone did.

the checklist that actually helps#

Most definition-of-done checklists fail because they list activities rather than properties. "Tests written" is an activity. "The change is covered by a test that would fail if it regressed" is a property, and it is checkable.

The version I have seen work is short:

• The change does what the ticket asked, verified by someone other than the author — a test, a reviewer, or the person who asked for it.
• It is in production, or in a deliberate queue with a named release date.
• A regression would be caught by a test, a metric, or an alert. If none of those apply, that is a decision, not an oversight.
• Its failure mode is known. Someone can say what happens when it breaks.
• It is documented where the next person will look — which is usually the code, sometimes the runbook, rarely the wiki.
• The flag, the branch, and the dead code are cleaned up, or ticketed with a date.

Six lines. If a change satisfies them, it is done in a way that survives the author going on holiday.

the two failure modes#

Done-done-done. Teams that discover the ambiguity often respond by inventing stages: "dev done", "QA done", "really done". This makes the confusion explicit without removing it, and adds ceremony. The fix is one definition, not three adjectives.

Definition as theatre. A twenty-item checklist copied from a blog post, pasted into the wiki, referenced never. If your definition of done is not enforced by something — a pull request template, a merge check, a question someone actually asks in standup — it is decoration.

The test: pick a ticket closed last week and walk the list. If it fails two items and nobody noticed, the definition is not operating.

the part about flags and branches#

The most common way "done" quietly is not: the feature ships behind a flag at 10%, works, and then nobody finishes the rollout. Three months later the flag is still at 10%, the old code path is still there, and the ticket has been closed since March.

That is not done. It is half-deployed with the cleanup unfunded, and it is how codebases accumulate the dual implementations that make everything else harder.

Make the rollout part of the ticket, not a follow-up. A change is not finished when it is enabled for some users; it is finished when the decision has been made either way and the losing path is deleted.

why it is worth the argument#

A shared definition of done is what makes "how much is left" answerable. Without it, the burn-down is measuring something nobody agrees on, estimates are uncomparable between people, and "almost done" means whatever the speaker wants.

It costs one conversation and a paragraph in the repository. It is the cheapest process improvement available to most teams and it is skipped because it sounds like process rather than engineering.

The result is consistently: an architecture overview that is meaningless without context, a list of tools with no explanation of why, and no answer to any question a new person actually has.

what a new engineer actually needs, in order#

Day one: get something running.

Not understanding. Running. A new engineer who has the application running locally on day one is in a completely different position from one who is still fighting a dependency on day three.

This section should be a numbered list of commands that work. Tested. On a clean machine. Recently.

## get it running

1. Install prerequisites:  `brew bundle`   (or the equivalent — see below)
2. Copy the environment:   `cp .env.example .env`
3. Get secrets:            `./scripts/fetch-dev-secrets` (needs VPN)
4. Start dependencies:     `docker compose up -d`
5. Migrate:                `./scripts/db-reset`
6. Run:                    `pnpm dev`

You should see the app at http://localhost:5173 with seeded data.
Log in with dev@example.com / password.

If step 3 fails with "unauthorized", you need to be added to the
dev-secrets group — ask in #eng-onboarding.

That last paragraph — the anticipated failure with its resolution — is the part that separates a document that works from one that does not. Every step that has ever failed for anyone should have a note.

Day two: make a change and see it.

A guided first change. Something trivial and real: change a label, add a field, fix a typo in a template. All the way through: edit, test, review, merge, deploy.

The point is not the change. It is that they have now executed the entire delivery pipeline once, so every subsequent change is a variation on something they have done.

Day three to five: the map.

Now the architecture overview, and now it means something, because they have seen the system run.

Keep it to a page. What are the major pieces, what does each do, how do they talk. A diagram. Where the code for each piece lives.

Not a complete description. A map, at the resolution of "which building do I go to."

Week two: the why.

The decisions that are not obvious from the code. Why the database is structured that way. Why there is a queue between those two services. Why that module is frozen. Why the obvious approach to X does not work.

This is the highest-value and least-written documentation in any organization, because it exists only in the heads of people who were there. When they leave, it is gone, and the next person spends a year rediscovering it — usually by proposing the obvious approach and being told no.

the sections everyone forgets#

The glossary. Every organization has jargon: internal product names, acronyms, words used with a specific local meaning. A new person hears twenty of these in their first week and cannot ask about all of them without feeling stupid.

Write them down. This is a thirty-minute task with an outsized payoff.

Who to ask about what. Not the org chart. "Payments: ask Priya. Deploy pipeline: #platform-help. Anything about the legacy importer: Marcus, and be warned it is complicated."

The things that will surprise you. The test suite that fails on the first run until you seed a fixture. The service that takes four minutes to start. The one flaky test everyone knows about. The staging environment that resets on Sundays.

Every codebase has these. Writing them down converts "this is broken and I do not want to admit I cannot fix it" into "oh, that is expected."

What not to touch. Frozen modules, generated files, anything requiring a specific review.

the maintenance mechanism#

Onboarding docs rot faster than any other documentation, because the people who would notice the errors are the ones who no longer read it.

The fix: the last person onboarded owns it.

Every new engineer's first task is to follow the document, fix everything that was wrong, and add the things they had to ask about. Then they own it until the next person arrives.

This works because their frustration is fresh and their perspective is exactly the target reader's. It is the only mechanism I have seen that keeps these documents accurate.

the test#

Hand it to a new engineer and do not help them.

Time how long until they have the application running. Note every question they had to ask. Each question is a gap, with a measured cost.

Then fix them, and repeat with the next person.

Most teams have never done this and would be surprised by the result.

Here is an attempt at what the role actually is, based on watching people do it well.

what it is not#

Not "a senior engineer who has been there longer." Time in seat produces a senior engineer with more context, which is valuable and is not this.

Not "the best coder." The highest-output individual contributor is a valuable role and it is a different one. Staff engineers frequently write less code than seniors.

Not "a manager who did not want to manage." The scope is comparable to a manager's; the mechanism is entirely different.

what it is#

The concise version: a staff engineer is responsible for the technical success of work that spans more than one team, without having authority over those teams.

Everything distinctive about the role follows from "without authority." You cannot assign work. You cannot approve headcount. You cannot make a decision stick by deciding it. Every outcome has to be achieved through information, credibility, and persuasion.

the actual activities#

Finding the problem nobody owns. The most common form of staff-level impact. Every organization has problems that fall between teams: the shared library nobody maintains, the integration that fails and each side thinks is the other's, the performance issue that is caused by the interaction of three services.

Nobody owns these, so nobody fixes them, so they persist for years. Identifying one, proving it matters, and getting it fixed is a substantial contribution and it usually requires a person who is not on any of the teams involved.

Making a decision that spans teams. Which of three approaches, when each team has a preference and none of them can see the whole picture. This is where the "no authority" constraint bites hardest: the decision has to be made in a way that the people affected accept, which means the reasoning has to be visible and the objections have to be genuinely addressed.

Writing the document that ends the argument. A recurring disagreement that resurfaces every quarter because nobody wrote down the resolution. One good document — with the options, the trade-offs, the decision, and the reasoning — can end a multi-year debate.

Being the person who read the whole system. Most engineers know their team's code. Somebody needs to know how it all fits together, including the parts nobody has touched in three years. That knowledge is what makes the cross-cutting problems visible.

Raising the floor. Not by writing better code. By making the good pattern easy: a library that removes a class of bug, a template that encodes the right defaults, a lint rule that prevents the mistake, documentation that means nobody has to ask. Impact through leverage rather than output.

Mentoring, specifically on judgment. Not "how do I use this API." "Should we build this at all," "how do I disagree with my manager about a technical decision," "how do I tell if this design will be a problem in a year."

Saying no with a reason, at a level where it lands. Frequently the most valuable thing a staff engineer does, and the reason it requires seniority is that the no has to come with an alternative and with credibility behind it.

the day, concretely#

A real week looks roughly like:

• 20% writing code, usually the hard or risky part of something, or a prototype that settles an argument.
• 25% writing documents — designs, decisions, analyses, postmortems.
• 25% in conversations — reviews, one-on-ones, arguing about designs, being asked "does this seem right to you."
• 15% reading — code, incident reports, other people's designs, the thing everyone is complaining about.
• 15% on whatever is currently on fire.

The proportion of coding is the part that surprises people moving into the role, and the discomfort of not shipping visible code is the most common reason people bounce out of it.

how to tell if someone is good at it#

Do things get decided? Not "do they have opinions." Do arguments they are involved in reach a resolution that holds.

Do other engineers get better? Look at the people around them over a year.

Do they work on things nobody asked them to? The highest-value staff work is usually self-directed, because if it were obvious and assigned it would already have an owner.

Are they trusted by people who disagree with them? This is the real test. Someone who is only trusted by people who already agree has influence, not credibility.

the failure modes#

Becoming an architecture astronaut. Designs, diagrams, opinions, no contact with running code. The credibility runs out within about a year, and it is very hard to get back.

Becoming a very expensive senior engineer. Doing excellent work with single-team scope. Comfortable, valuable, and not the job.

Spreading too thin. Involved in twelve things, effective in none. The scope is tempting and the constraint is real: two or three significant efforts at a time is the realistic maximum.

Losing the ability to build. The role requires enough hands-on work to stay credible and calibrated. An engineer who has not shipped in a year is guessing.

the failure pattern#

1. Platform team forms to reduce duplicated infrastructure work.
2. They build an abstraction over the cloud provider.
3. The abstraction covers 80% of cases well.
4. The remaining 20% is impossible, and the escape hatch is either absent or punished.
5. Product teams work around the platform.
6. Platform team responds by mandating the platform.
7. Everyone is unhappy and the platform is now a tax.

Every step follows from the previous one. The root is step four.

the four properties of platforms that work#

1. An escape hatch that is not punished.

The platform covers the common case. It cannot cover every case, and pretending otherwise is what breaks trust.

There must be a supported path for "I need something the platform does not do," and taking that path must not require an exception process, a meeting, or an apologetic Slack message.

The best platforms make the escape hatch cheap and then compete on being better than it. The worst make it forbidden, which does not eliminate the need — it drives it underground.

2. Adoption is voluntary, at least at first.

A platform that teams choose is a platform that is good. A platform teams are required to use never gets the feedback that would make it good, because the feedback mechanism — people leaving — has been disabled.

If you cannot get voluntary adoption, that is information. Mandating it does not fix the underlying problem; it hides it and converts a product problem into a political one.

Mandate later, when it is genuinely better, and the mandate will be uncontroversial because everyone already uses it.

3. The abstraction leaks deliberately, not accidentally.

Every abstraction leaks. The question is whether you planned for it.

A good platform lets you drop a level when you need to: use the paved path for the deployment, and reach the underlying resource directly when you need something specific. A bad one hides the underlying system entirely, so that when it fails you cannot debug it and neither can the platform team, because now there are two systems to understand.

Concretely: if your platform generates infrastructure configuration, let people see it. If it wraps a cloud API, let people access the underlying resource. If it runs their container, give them the logs from the actual runtime, not a filtered view.

4. The platform team is measured on adoption and satisfaction, not on compliance.

If the platform team's metric is "percentage of services on the platform," they will optimize for mandating it.

If the metric is "would you use this if you had a choice," they will optimize for making it good.

Ask that question quarterly, anonymously, and publish the answer.

the specific things that generate resentment#

Slow escape. A team needs something the platform does not support. The answer is "file a request, we will look at it next quarter." Their deadline is Friday.

Breaking changes without migration paths. The platform is infrastructure. Break it and every team stops. Platform teams frequently hold themselves to a lower compatibility standard than they would accept from a vendor.

Opaque failures. The deploy failed. The error is a platform-internal message. The product engineer cannot debug it and must escalate, which means waiting.

Being a gate rather than a service. A platform that must approve things is a bureaucracy. A platform that makes the right thing easy is infrastructure.

Solving the platform team's problems. Standardization is valuable to the platform team and is not automatically valuable to product teams. If the pitch for a migration is "this makes our lives easier," expect a cool reception.

the framing that works#

You are building a product. Your users are engineers. They have alternatives.

That framing produces the right behaviors automatically: user research before building, documentation that assumes nothing, onboarding that works, support that responds, and a roadmap driven by what users need rather than by architectural preference.

The platform teams I have seen work best behave exactly like a startup selling to a skeptical market, and they say so out loud.

The ones that fail behave like an internal standards body, and they are usually correct about the standards and wrong about how to get them adopted.

the measurement that matters#

Time from "a new engineer joins" to "their code is running in production."

That single number captures most of what a platform is for, it is measurable, and it is the thing product teams actually care about. If it is going down, the platform is working, regardless of what the adoption dashboard says.

The first part will recover. The second part is a mistake, and it is the kind of mistake that is invisible for four years and then extremely expensive.

the argument for not hiring juniors#

Stated honestly, because it is not stupid:

A junior engineer's first-year output is mostly small, well-specified tasks — the CRUD endpoint, the test coverage, the bug in a file someone pointed them at. That work is now substantially automatable. Meanwhile the junior requires mentoring time from senior engineers, which is the scarcest resource.

So the ROI on a junior looks worse than it did. That reasoning is coherent.

why it is wrong#

Seniors come from juniors. There is no other supply. An organization that hires only seniors is free-riding on other organizations' training, and if everyone does it, the pipeline empties. This is a classic collective action failure and the industry is walking into it with open eyes.

The judgment that makes seniors valuable comes from doing the work. The ability to look at plausible code and know it is wrong comes from having written the wrong version and debugged it at 3 a.m. You cannot read your way to it and you cannot prompt your way to it.

If the apprenticeship stops, the next generation of senior engineers does not exist, and the current one retires.

Juniors are better at the new tools. Consistently, in my experience. They have no prior workflow to defend and they explore. A team of only senior engineers adopts new tooling slowly and grudgingly.

Mentoring makes seniors better. The engineer who has to explain why a design is wrong understands it better afterward. Teams with no juniors lose that forcing function and get sloppier about articulating their own reasoning.

what actually has to change#

The old model — hire a junior, give them small tickets for a year, gradually increase scope — does not work as well when the small tickets are automated. The model has to change, not the hiring.

Start them on reading, not writing. Give a new engineer a real system and a week to understand and explain it. Have them write the architecture document that does not exist. This builds the skill that actually matters now and it produces something useful.

Give them debugging, not features. Debugging is the skill that generalizes, that AI is least reliable at, and that cannot be learned from a course. Pair them on incidents. Give them the flaky test nobody wants.

Make them review agent output. Reviewing machine-generated code with a senior engineer walking through what is wrong with it is an extraordinarily efficient teaching mechanism. You get a stream of plausible-but-flawed code, which is exactly the training material you want and which used to be expensive to produce.

Require them to write the tests first. Specifying behavior before implementing teaches design, and it is the part of the workflow that has become more important rather than less.

Do not let them delegate the hard part. For the first year, some things get done by hand, deliberately, because the point is the learning rather than the output. Say this out loud so it does not feel like an arbitrary restriction.

the hiring signal that works now#

Traditional junior screens — implement this algorithm, complete this take-home — are substantially defeated and were never good predictors anyway.

What works better:

A code review exercise. Give them a pull request with three problems. Watch what they find and how they talk about it. This is the job.

A debugging exercise on a real repository. Failing test, thirty minutes, any tools they want including AI. Watch the process, not the outcome. Do they read the error? Form a hypothesis? Check it? Notice when the model's suggestion is wrong?

A conversation about something they built. Follow-up questions until you hit the edge of their understanding. Where that edge sits, and how they handle reaching it, tells you almost everything.

the case to make internally#

If you are arguing for junior headcount:

The cost of a junior is roughly a senior's partial attention for a year plus a below-market salary. The cost of a senior hire in three years, in a market where nobody trained anyone, is going to be considerably higher than it is now.

Every organization that stopped training in 2009 spent 2013 through 2016 paying enormous premiums for the engineers who had been trained elsewhere. It is the same trade and it is being made again.

The organizations that keep training through this period will have a meaningful advantage in five years, and it will be very hard to catch up to them quickly.

That understates it by a large factor, and the reason is the same reason context switching is expensive for a CPU.

the fragmentation cost#

Deep engineering work requires holding a system in your head: the call graph, the invariants, the specific thing you were about to check, the four hypotheses you had narrowed to two.

Building that state takes time — call it twenty to forty minutes for nontrivial work. Losing it takes one interruption.

So a meeting at 2 p.m. does not cost thirty minutes. It costs:

• The twenty minutes before it, where you cannot start anything substantial because you will be interrupted.
• The thirty minutes of the meeting.
• The twenty to forty minutes after it, rebuilding the state you lost.

That is roughly ninety minutes for a thirty-minute meeting, per person, and it is the optimistic case.

Worse: a meeting placed in the middle of a morning does not remove ninety minutes from a four-hour block. It removes the block. Two ninety-minute fragments are not equivalent to one three-hour stretch, because the hardest work requires depth that ninety minutes cannot reach.

the schedule shapes that work#

Meeting-free days. Two full days a week with no recurring meetings. Not "try to keep them clear" — a calendar policy. This is the single highest-impact scheduling change available and it is nearly free.

Meetings at the edges. Cluster them at the start or end of the day. A day with three meetings from 9 to 11 and nothing after is dramatically more productive than one with three meetings at 10, 1, and 3.

Default to 25 and 50 minutes. Not 30 and 60. The buffer prevents the cascade where every meeting starts late and the person coming from the previous one misses the first five minutes.

Async by default for status. Standup is a meeting to exchange information that could be a written update. The argument for synchronous standup is that it surfaces blockers, which it does, and a written update with a "blocked on" field surfaces them too, in a searchable form, without costing everyone their morning.

when meetings are actually right#

They are, frequently, and the anti-meeting position overcorrects.

High-bandwidth disagreement. Two people who disagree about a design will resolve it in twenty minutes of conversation and in nine days of comment threads. Real-time is enormously better for anything with back-and-forth.

Ambiguity resolution. When nobody is sure what the problem even is, a conversation converges much faster than writing, because you can ask the clarifying question immediately.

Anything with emotional content. Feedback, conflict, bad news. Writing is the wrong medium and using it is usually avoidance.

Building relationships. Real, unmeasurable, and the reason fully-async organizations struggle in ways they cannot diagnose.

The rule: meet for things that require interaction. Write for things that require information.

the tests#

Before scheduling, three questions:

What decision will be made? If none, this is a status update. Write it.

Who must be there for that decision? The answer is usually two to four people. Everyone else can read the outcome.

What would happen if we did not have it? If the answer is "nothing," you have found a recurring meeting that outlived its purpose. Every organization has several.

the audit worth doing#

Take a recurring meeting and cancel it for a month. Not "make it optional" — cancel it.

If someone notices and asks for it back, with a reason, reinstate it. Most of the time nobody notices, which tells you what you needed to know.

This works because recurring meetings are created for a reason that expires and never re-evaluated, and the social cost of proposing cancellation is high enough that nobody does it. A trial cancellation removes the social cost.

the part managers should hear#

Your calendar is not your team's calendar. A manager's day is legitimately made of meetings — that is the job, and thirty-minute chunks are the natural unit.

An engineer's day is not, and scheduling as if it were is the single most common way well-meaning managers destroy their team's output while looking at metrics that do not show it.

Protect the blocks. It is most of what schedule-level management can do.

Both failures come from the same mistake: treating "no" as a verdict rather than as the opening of a conversation about trade-offs.

what the request actually is#

When someone asks you to build something, they are not asking for the thing. They are asking for an outcome, and the thing is their guess at how to get it.

That means the highest-value response is frequently not yes or no. It is a better guess.

"You are asking for a real-time dashboard. What decision are you going to make from it? If it is 'should we page someone,' an alert is better than a dashboard and it is a day of work instead of a month."

That is a no to the request and a yes to the outcome, and nobody experiences it as obstruction.

Ask what the outcome is before you evaluate the request. Half the time the request evaporates.

the four honest noes#

"Not this, that." The alternative approach that achieves the same outcome for less. This is the best one and it requires actually understanding the problem.

"Yes, and here is what it displaces." Not a no. A statement about capacity, which is factual and is the other person's decision to make.

"I can do that in this cycle. It means the API migration slips to next quarter. Which do you want?"

This is enormously more effective than "we do not have time," because it hands the prioritization decision to the person whose job it is, with the information they need.

"Yes, after X." A sequencing objection. "We can build this on top of the new data model. Building it on the old one means we build it twice."

"No, and here is the risk I am flagging." The real no, reserved for correctness, security, legal, or ethical problems. Use it rarely so that it lands when you do.

"I am not going to implement this the way it is specified because it stores plaintext credentials. I will implement it with a token exchange, which takes three extra days. If that is unacceptable, I want the decision documented and made by someone who can accept the risk."

That is a hard sentence to say and it is the sentence the job sometimes requires.

the ones that do not work#

"That is not possible." Almost always false, and the person will find someone who says it is. Say "that would take six months" instead, which is the real constraint and is checkable.

"That is a bad idea." Without an alternative, this is just friction.

Silence. Not responding is a no that damages trust, because it looks like you did not care rather than that you disagreed.

"Sure" followed by not doing it. The worst one. It destroys your reliability, which is the only currency you actually have.

the timing#

Say no early. The cost of a no rises with every day of planning that assumed a yes.

An objection raised in the design review is a discussion. The same objection raised two weeks before launch is a crisis, and people will remember that you could have said it earlier — correctly.

If you have doubts, voice them while they are cheap.

the part about capital#

Every no spends something. Every yes earns something. If you never say yes, your noes stop landing, because you have become the person who says no.

Deliver reliably on what you agree to. That is what makes the refusal credible when it matters. Engineers who are trusted to ship get enormous latitude to push back, and engineers who are not, do not — regardless of whether they are right.

the hardest case#

Sometimes you are overruled on something you believe is wrong, and it is not a correctness or ethics issue — it is a judgment call and someone with the authority made a different one.

Disagree and commit is the right practice here and it is genuinely hard. Say your piece once, clearly, in writing. Then build the thing well.

Do not build it badly to prove a point. Do not relitigate it in every standup. Do not say "I told you so" if it goes wrong — the written record already said it, and gloating costs you the ability to be listened to next time.

You will be wrong about some of these. That is the actual reason to commit gracefully: you are not always right, and a culture where disagreement is followed by good-faith execution is one where being wrong is survivable for everyone, including you.

what the pages are telling you#

Every page is a statement about the system.

A page that requires a human to restart something says the system cannot recover from a condition it will encounter again. That is automatable and the automation is usually a supervisor and a health check.

A page for a transient issue that resolved itself says your alert threshold is wrong, or your alert lacks a duration condition, or the underlying flakiness needs a retry with backoff.

A page where the runbook is "look at the dashboard and see if it is bad" is not a page. That is a dashboard, and it should be checked during business hours.

A page that only one person can resolve is a knowledge distribution failure and a single point of human failure. The fix is documentation and rotation of the work, not heroism.

A page at 3 a.m. for something that could have waited until 9 a.m. says nobody has classified alerts by urgency. Not everything that is wrong is urgent.

the audit#

Pull the last ninety days of pages. For each one:

• What was the user impact? (Often: none.)
• Could it have been automated?
• Could it have waited?
• Did the runbook exist and was it correct?
• What was the actual fix?

Then categorize:

Should not have paged — no user impact or no urgency. Delete the alert or downgrade it. This is usually a large fraction and deleting alerts is the highest value hour available to most teams.

Should have been automatic — the fix was mechanical. Automate it. If the runbook says "run this command," a computer can run that command.

Genuine incidents — real user impact requiring judgment. These are the ones on-call exists for, and there should not be many.

A healthy rotation is a small number of genuine incidents. If you are getting paged more than a couple of times per week, the problem is not that your systems are complex.

the design changes that reduce pages#

Make things self-healing. Restart on crash. Retry with exponential backoff and jitter. Circuit-break to a degraded mode. Shed load rather than falling over. Each of these converts a page into a metric.

Make degradation graceful and explicit. Decide in advance which features can be disabled. Feature flags that turn off the expensive path let a page become "turn off recommendations, investigate Monday."

Make everything reversible fast. A large fraction of incidents are caused by a deploy. If rollback is one command and takes ninety seconds, the incident is ninety seconds long. If it requires reversing a migration, it is four hours.

Add capacity headroom. Running at 85% utilization to save money means every spike is a page. Headroom is cheaper than the human cost, and much cheaper than the turnover.

the human side#

Pay for it. On-call is work performed outside working hours with real cost to the person's life. Compensate it explicitly — money or time off, and enough that the cost is visible to whoever decides whether to fix the alert noise.

Unpaid on-call means the cost is borne entirely by the person and is invisible to the organization, which guarantees it never improves.

Time off after a bad night. Non-negotiable, automatic, not something the person has to ask for.

Never one person. A primary and a secondary, always. The primary needs to be able to escalate without feeling like a failure.

Rotate wide. If only three people can be on-call, they will burn out and leave, and the knowledge leaves with them.

The person who was paged decides what gets fixed. Give the on-call engineer authority to prioritize the follow-up work. They have the information and the motivation, and nobody else has both.

the metric to track#

Pages per on-call shift, trended over quarters.

If it is flat or rising, nobody is fixing the causes. That is a management failure, not a resilience failure, and the outcome of ignoring it is attrition — which is more expensive than every fix combined.

That pause is the entire problem.

the four kinds, and why mixing them fails#

The taxonomy that fixed documentation for me — and it is not mine, it is the Diátaxis framework — is that there are four distinct kinds and they have incompatible goals.

Tutorials teach a beginner by having them do something that works. The goal is a successful experience, not completeness. A tutorial that mentions every option has failed. It should be prescriptive, opinionated, and it should work exactly as written, every time.

How-to guides help someone accomplish a specific task they already understand. "How to configure TLS." Goal-oriented, assumes competence, skips explanation.

Reference describes the machinery precisely and completely. Every parameter, every return value, every error. Boring by design. Generated where possible.

Explanation provides understanding. Why is it designed this way? What are the trade-offs? What is the mental model? This is the kind that is almost always missing and the kind that most reduces support burden.

Most documentation fails because it tries to be all four at once. A tutorial that stops to explain architecture loses the beginner. A reference page with a narrative is hard to scan. Separate them, label them, and each one gets better.

what "staff it like a product" means#

One named owner. Not "the team." A person whose review includes it.

A budget in the sprint. Documentation work sized and scheduled alongside features, not appended to the end of a ticket where it gets cut.

Metrics. Support tickets that a doc would have prevented. Search queries with no results. Time-to-first-successful-request for a new user. Page-level feedback. Every one of these is measurable and almost nobody measures them.

A definition of done that includes it. A feature is not shipped until it is documented. This is either enforced or it is a slogan; there is no middle.

the practices that actually move the needle#

Docs live with the code. Same repository, same pull request, same review. Documentation in a separate wiki drifts within one quarter, guaranteed, without exception.

Test the examples. Every code sample in your docs should be extracted and run in CI. Broken examples are worse than no examples — they destroy trust in the whole document, and every set of docs has them, because they were correct when written.

Write the failure cases. The single highest-value section in any documentation is "common errors and what they mean." This is what Stack Overflow existed to provide and it is the thing your docs almost certainly lack.

Go read your support queue. Every recurring question is a documentation gap with a measured frequency attached.

Date and version everything. "This page describes v4.2, last updated 2026-01-15." Undated documentation is untrustworthy documentation, because the reader cannot tell whether it is current.

Make the first example work. The single most common documentation failure: the quickstart does not run. Someone changed a default, renamed a parameter, required a new config field. Test the quickstart in CI, on a clean environment, on every release.

the argument that gets budget#

Documentation is deflection. Every question answered by a doc is a question not asked of an engineer.

Count your support load. Estimate the fraction that is documentation-shaped — "how do I," "what does this error mean," "does it support." In most organizations it is more than half.

Now price that in engineer-hours. That is your documentation ROI, and it is usually large enough to fund a technical writer, which is the actual right answer and which almost nobody does.

the new reason it matters#

Your documentation is now also a model's training data and a model's retrieval corpus.

When a developer asks an assistant about your library, the answer is synthesized from your docs. If your docs are wrong, incomplete, or ambiguous, the assistant confidently produces wrong code, and the user blames your library.

You have less control over how your project is explained than you did three years ago, and the only lever you have is the quality of the source material.

That is a strange new incentive and it is the strongest argument for good documentation that has ever existed.

the failure pattern#

The typical postmortem:

1. Timeline of what happened. Accurate, detailed, useful.
2. Root cause. Usually a single technical fact.
3. Action items. Five to twelve of them.
4. Filed. Two action items get done. The rest age out.

Six months later, a similar incident, with a similar document.

Three things are wrong here.

problem one: "root cause" is singular#

Complex systems do not fail because of one thing. They fail because several conditions aligned, each of which was individually survivable.

The Cloudflare outage in November: a permissions change, a query that returned duplicates, a fixed buffer size, an error path that panicked instead of degrading, and a config pipeline that propagated globally in minutes. Remove any one and it does not happen, or it is much smaller.

Picking one and calling it "the root cause" means you fix one and leave four.

Better framing: contributing factors, plural, each with its own assessment of whether it is worth addressing. Some will not be — that is a legitimate decision if it is made explicitly.

problem two: action items without owners and dates are wishes#

An action item that says "improve monitoring for the config pipeline" with no owner, no date, and no definition of done is a sentence, not a plan.

The fix is unglamorous:

• Every action item has one named person. Not a team. A person.
• Every action item has a date. If nobody will commit to a date, it is not going to happen and you should delete it and say so.
• Every action item has a definition of done that someone else could verify.
• They go in the same backlog as feature work, prioritized against it. An action item in a separate "incident follow-up" list that is never sprint planned is a list of things that will not be done.

And the one that actually forces it: review the open action items at the start of the next postmortem. Nothing motivates completion like a room full of people looking at your undone item from last quarter's incident while discussing this quarter's similar one.

problem three: nobody asks about the near misses#

The incidents you review are the ones that broke through. For every one, there were several that did not — a bad deploy caught by a canary, a config error someone noticed in review, a query that would have taken down the database if it had run on Monday instead of Sunday.

Those contain the same information at a fraction of the cost, and almost nobody collects them.

Add a lightweight channel for it. "I nearly broke prod today, here is how." No document, no meeting, no blame. Just a note. The pattern that emerges over a quarter is more valuable than any individual postmortem.

the questions that produce useful findings#

Replace "what was the root cause" with:

• "What made this hard to detect?" Detection time is usually the largest component of impact and is the most improvable.
• "What made this hard to diagnose?" Usually missing observability. This produces the highest-value action items.
• "What made recovery slow?" Often a missing runbook, a missing kill switch, or a rollback that was not actually tested.
• "Who knew something that would have helped, and why did that not reach the responders?" This is an organizational question and it is frequently the real finding.
• "What did we do that helped?" Genuinely important. Practices that worked should be named so they get kept.

the blameless part, done correctly#

Blameless does not mean nobody made a mistake. It means the analysis focuses on why the mistake was possible and easy, rather than on the person.

"Alice deployed without running the tests" is blame and it is also useless. "The deploy path does not require tests to pass, and the shortcut that skips them is the fastest way to deploy" is the same fact stated in a way you can act on.

If the answer to "why did they do that" is "because the system made it easy and the correct path was hard," you have found something.

If the answer is genuinely "they were careless," you still fix the system, because the next person will also be careless eventually. Humans are the constant; the system is the variable.

• Brainteasers measured whether you had heard the brainteaser. Replaced by algorithms.
• Algorithm puzzles measured whether you had ground a puzzle site. Replaced, partially, by take-homes.
• Take-homes measured how much unpaid time you had, which selects against people with children and second jobs. Replaced, partially, by pairing.
• Pairing is the best of them and it measures your comfort being watched, which is correlated with experience and confidence, which is correlated with demographics.

Now there is a new one, and it is genuinely novel.

the current problem#

A candidate with a model available can pass most remote technical screens. Not "can cheat" — can pass, because the tasks we set are exactly the tasks these tools are good at.

The industry response has been mostly bad. Return-to-office for interviews. Proctoring software. Increasingly hostile monitoring. All of it makes the experience worse for honest candidates and is defeated by anyone determined.

The response has been bad because the question was framed wrong. The question is not "how do we stop candidates using AI." It is "what are we actually trying to measure, given that the job now includes using these tools."

what the job actually is#

If you hired someone today, their work would involve:

• Understanding an existing system well enough to change it safely.
• Deciding what to build, which is mostly deciding what not to build.
• Using AI tools effectively, including knowing when the output is wrong.
• Communicating a technical decision to people who will be affected by it.
• Debugging something under time pressure with incomplete information.
• Reviewing someone else's code — increasingly, a machine's — and catching the problem.

Not one of those is measured by "implement an LRU cache in forty-five minutes."

interviews that measure the real thing#

Code review. Give them a 300-line pull request with three deliberate problems: one obvious bug, one subtle design issue, one thing that is fine but looks wrong. Ask them to review it.

This is excellent. It is exactly the job, AI does not obviously help, and the conversation about the third item — where they explain why the suspicious thing is actually correct — tells you more about their judgment than any implementation task.

Debugging a real system. Give them a repository, a failing test, and thirty minutes. Let them use whatever tools they want, including models. Watch how they narrow it down. Do they read the error? Do they form a hypothesis? Do they check it? Do they notice when the model's suggestion is wrong?

Watching someone debug with AI assistance is a much better signal than watching them code without it, because it is the actual work.

Design discussion on their own past work. "Tell me about a system you built. What would you change?" The follow-up questions are where the signal is. People who genuinely understood their system can answer six levels deep. People who did not, cannot, and it becomes clear quickly.

A short, paid, scoped project. Four hours, paid at a real rate, on something close to the actual work. This is the highest-signal option and the least scalable, and it is worth it for senior roles.

what to stop doing#

Stop asking people to implement data structures from memory. They will not do this in the job, and if they need one they will look it up, correctly.

Stop the six-round loop. Every round is a coin flip with a false-negative rate. Six rounds does not make the signal six times better; it makes the process long enough that good candidates take another offer.

Stop pretending the whiteboard measures anything but whiteboard performance.

the thing nobody wants to hear#

Interviews have a low ceiling on signal. The correlation between interview performance and job performance is weak in every study anyone has run.

The highest-signal thing is working with someone. Everything else is a proxy. Which argues for: shorter processes, more willingness to take a chance, and robust ways to correct the mistake — a real probation practice, honest early feedback, and the organizational nerve to act on it.

That is a harder cultural change than redesigning the interview loop, which is why everyone redesigns the interview loop instead.

The methodologies keep failing because they are solving the wrong problem.

what is actually happening#

When a manager asks "how long will this take," they are usually not asking for a prediction. They are asking one of these:

• Can I promise this to a customer for Q1?
• Should we do this or the other thing?
• Do I need to hire?
• Is this a two-week thing or a two-quarter thing, because those go in different plans?

Every one of those is answerable. None of them requires a precise duration. And a precise duration, given confidently, actively harms all four — because it gets recorded, propagated, and treated as a commitment by people three steps removed who never saw the assumptions.

The engineer knows the estimate is uncertain. The number arrives downstream stripped of that uncertainty. That is the failure, and it is a communication failure, not a forecasting one.

what to say instead#

Give a range with an explicit confidence.

"Two to six weeks. I'd take even money on three. The spread is because I don't know yet whether the legacy import path can be reused — if it can, it's two weeks; if I have to rewrite it, it's five or six."

That sentence contains everything the asker needs: a planning number, a worst case, and the specific unknown that determines which. It is also honest, which the single number was not.

Name the decisive uncertainty. Almost every estimate has one thing that dominates the variance. Say what it is. That converts "the engineer is being vague" into "there is a specific question we could go answer," which is actionable.

Offer to reduce the uncertainty.

"Give me two days to spike the import path and I'll come back with a much tighter number."

Two days to turn a 3× range into a 1.3× range is almost always worth it, and proposing it changes the conversation from negotiation to problem-solving.

Separate the estimate from the commitment. "It will probably take three weeks" and "I commit to delivering it in three weeks" are different statements, and conflating them is where trust gets destroyed. If someone wants a commitment, they are asking for buffer, and you should say what buffer you need.

why estimates are actually wrong#

Not because engineers are optimistic — though they are. The dominant causes, in my experience, in order:

1. Unknown unknowns in existing code. The thing you have to modify does something you did not expect. This is the biggest one by a wide margin and it scales with the age and size of the codebase.
2. Work that is not the work. Code review latency, CI flakes, environment problems, a dependency upgrade you did not plan, the meeting.
3. Scope discovered during implementation. "Also it has to work for the enterprise tier" arrives in week two.
4. Interruption. An estimate of three focused days is a lie if the person has two days of meetings and an on-call rotation.

Note that only the first is about the difficulty of the task. Most estimation error is organizational, which is why individual estimation skill improves results much less than people expect.

the thing that actually works#

Track your own history. Not for a methodology — for calibration.

Write down the estimate. Write down the actual. Do it for three months. You will discover you have a personal multiplier, it is remarkably stable, and it is probably between 1.5 and 3.

Then apply it. Silently. That single practice does more than any team-level process, and it requires no one else's participation.

the part managers should read#

If you punish an engineer for a missed estimate, you will get padded estimates forever, and you will have destroyed the information content of the number.

If you ask "what would make you more confident," you will get real answers and a team that tells you when something is going wrong early, which is the only thing that actually helps.

You cannot have both. Pick.

The memo is better written and more reasonable than the headlines about it. It is also going to be misapplied everywhere it gets copied, and it is going to get copied everywhere.

the defensible core#

Two claims in the memo are correct.

"Try the AI first" is good engineering hygiene. Before you build the internal tool, before you write the script, before you file the ticket — spend ten minutes seeing whether an existing model handles it. Often it does. The cost of checking is near zero and the expected value is high.

Stagnation is a choice. A team that has not changed how it works in two years during a period of rapid tooling change is not being careful, it is being incurious. That is a fair thing to name.

the failure mode#

"Prove AI cannot do it" is an unfalsifiable standard, and unfalsifiable standards in an organization become political tools.

You cannot prove a negative about a capability that changes monthly. Any manager who wants to deny headcount now has an infinitely flexible reason, and any manager who wants to grant it will produce a document explaining why AI cannot do it. The document is not evidence. It is theater. Every organization that has ever required a justification memo for headcount has produced a genre of justification-memo prose, and this is just the newest style.

The real question — "what is the highest-leverage use of an additional engineer" — was always the question. Adding an AI framing does not make it easier to answer, it makes it easier to obscure.

the second-order effect nobody plans for#

If teams are evaluated on AI usage, teams will use AI, including in places where it is worse. This is Goodhart's law with a new coat of paint. You will get:

• Code review comments generated by a model that read fine and check nothing.
• Documentation nobody reads, generated because generating it is cheap.
• Test suites with impressive coverage numbers and no assertions that would fail.
• Postmortems written by a model that faithfully summarize the incident and identify no real cause.

All of that is measurable AI adoption. None of it is value.

what a better version looks like#

If you want the outcome the memo is aiming at, ask for the outcome directly:

• Cycle time, not tool adoption. If the team ships faster with the same quality, they are using their tools well, and it does not matter which ones.
• Toil reduction as a named goal, with a quarterly review of what got automated. That surfaces the same opportunities without the unfalsifiable test.
• A budget and permission, not a mandate. The blocker in most organizations is not enthusiasm, it is that the good tools are not approved and the data policy is unclear. Fix that and adoption happens without a memo.

the part I actually agree with#

The memo says learning to use these tools well is now part of the job. That is true and it is not controversial in the way people are treating it.

Getting good output from a model is a skill — knowing what context to provide, how to decompose a task, when the answer is wrong in a way that looks right. It is closer to being a good technical lead than to typing a search query. People who have developed it are meaningfully more effective, and people who dismissed the whole category in 2023 and never revisited it are falling behind in a way that is going to be uncomfortable to talk about at review time.

That is worth saying out loud. It just does not require a headcount policy.