The distribution is always the same: three dashboards carry all the traffic and forty have not been opened in a year.

why they accumulate#

Dashboards get created during incidents. Someone needs to see a specific correlation right now, builds a view, the incident ends, and the dashboard stays — specific to a problem that has been fixed, named something like "orders debug 2," owned by nobody.

Nothing removes them, so they accumulate until finding the useful one requires knowing its name in advance.

the three that earn their place#

The service dashboard. One per service, same layout for every service, so that anyone can read any service's dashboard without orientation. Rate, errors, duration, saturation. Six to eight panels, above the fold, no scrolling.

The consistency is the point. When every service looks the same, an engineer responding to an unfamiliar service already knows where to look.

The journey dashboard. One per critical user journey — checkout, signup, search. Not per service: per outcome, end to end, across every service involved.

This is the one most organisations lack, and it is the one that answers the question that matters during an incident: are users able to do the thing? Every service being green while checkout is broken is a common and confusing state, and only a journey view shows it.

The capacity dashboard. Resource headroom over weeks, not minutes. Reviewed on a schedule, not during incidents. This is the only one meant for planning rather than response.

what makes a panel useful#

A threshold drawn on it. A line with no reference is a shape. A line with "target: 200ms" drawn across it is information. Every latency and error panel should show what "fine" is.

Comparison to last week. Most values are meaningless in isolation. 4,000 requests per minute — is that high? The same panel with last week's line overlaid answers instantly.

A unit and a direction. Label it. State whether up is good. This sounds trivial and half of all panels fail it.

Percentiles, not averages, for anything latency-shaped. And show p50 next to p99: the gap between them is often the real signal.

Fewer panels. A dashboard with forty panels is a data dump. During an incident nobody scrolls. Eight panels that fit on a screen beat forty that do not.

the annotations that make it work#

The highest-value dashboard feature is the least used: deploy markers.

Vertical lines on every time series showing when a deploy happened. Most tools support it and most teams have not wired it up.

With them, "did this start after the deploy?" is answered by looking. Without them it is answered by opening another tab, finding the deploy log, and correlating timestamps by hand — during an incident, under pressure.

Add config changes and feature flag flips to the same annotation stream and you have covered the causes of most self-inflicted incidents.

the maintenance rule#

Delete dashboards nobody opens. Quarterly, using the view counts your tool already records. If someone misses one, they can rebuild it, and the rebuild will be better because it will reflect the current system.

Give every dashboard an owner and a purpose in its description. "Used by on-call to triage checkout failures" tells the next person whether to trust it. An unlabelled dashboard is one nobody dares delete and nobody quite believes.

the test#

During your next incident, notice which dashboard you actually opened first.

That is your real service dashboard. Whether it is the one you designated as such, and how much of what you needed was on it, is the most honest review of your monitoring you will ever get — and it costs nothing but paying attention for five minutes while you are already there.

Here is the tedious work, in a manageable order.

the audit, in one afternoon#

For each service, answer four questions. Write the answers down.

1. What identity does it run as? Not "a service account" — which one. A surprising number of services run as something shared, or as a human's credentials that were used for the initial deploy and never replaced.

2. What can that identity do? Enumerate the actual permissions. In most cloud platforms this is one API call. The answer is usually much broader than anyone expects, because permissions accumulate and nothing removes them.

3. What does it actually use? This is the gap. Cloud providers log every authorised call — CloudTrail, audit logs, equivalents elsewhere. Ninety days of logs tell you precisely which permissions were exercised.

4. What is the delta? Everything granted and never used is a candidate for removal, and that set is typically most of the grant.

That fourth number is the finding. In every audit I have seen, a service uses somewhere between 5% and 20% of what it is permitted to do.

the specific over-grants to look for#

Wildcards. s3:* on *. Almost always the result of "it wasn't working so I broadened it until it did," which is a debugging technique that becomes a permanent security posture.

Write access where only read is used. Extremely common for anything reading configuration or reference data.

Access to everything of a type. A service that needs one bucket having access to all buckets. Scope to the resource, and where possible to a prefix within it.

Permissions for a feature that was removed. The code went; the grant stayed.

Human roles used by machines. A deploy pipeline running as a role designed for an engineer's console access — which usually includes the ability to change permissions, and that is the one that turns a compromise into a takeover.

iam:* or its equivalent. The permission to grant permissions. Anything with this is effectively an administrator regardless of what else it has. Treat it as its own category and audit it separately.

the direction to move#

Short-lived over long-lived. Workload identity — the container proves what it is and receives a token that expires in minutes — removes the stealable credential entirely. This is the single highest-value change on this list, and it is architectural rather than a matter of tightening a policy.

Scoped over broad. One bucket and one prefix, not the account.

Read over write. Split the identity if the same service does both, so the read path cannot write.

Deny at the boundary. A service in a network that cannot reach the internet cannot exfiltrate, whatever its IAM permissions say. Network policy and identity policy are independent layers and both matter.

the thing that makes it stick#

An audit is a snapshot. Permissions creep back the next time something does not work at 6 p.m. on a Friday.

Two mechanisms hold the line:

Permissions in code, reviewed like code. If a grant requires a pull request, the broad one gets a comment. If it is a console click, it does not.

A scheduled review. Quarterly, using the same used-versus-granted delta. Twenty minutes per service, and it catches both the emergency grant nobody reverted and the feature that was deleted.

the honest framing for a sceptical audience#

Least privilege does not prevent compromise. It bounds what a compromise costs.

That is the argument to make when someone asks why it is worth the effort: the question is not whether a credential will leak — a dependency, a laptop, a log file, a misconfigured bucket, eventually one will. The question is whether the answer to "what could they do with it" is "read one bucket" or "anything."

Those two answers are the difference between an incident report and a breach notification, and the work that separates them is an afternoon per service.

RFCs are dense but they are not difficult, and they follow conventions that make them scannable once you know them.

the keywords are load-bearing#

RFC 2119 defines a small vocabulary, and in a specification these words are technical terms, not English:

• MUST / REQUIRED / SHALL — absolute requirement. Violating it makes your implementation non-conforming.
• MUST NOT / SHALL NOT — absolute prohibition.
• SHOULD / RECOMMENDED — there may be valid reasons to ignore this, but understand the implications first. In practice: everyone does it, and if you do not, something will break eventually.
• SHOULD NOT — same, inverted.
• MAY / OPTIONAL — genuinely optional. Critically: an implementation that does not do it must interoperate with one that does, and vice versa.

When reading, look for the capitalised words first. They are the actual requirements; everything around them is explanation. Skimming an RFC by jumping between MUSTs is a legitimate and efficient way to read one.

the structure is consistent#

Abstract — one paragraph. Read it to decide whether you want this document.

Introduction / Terminology — read the terminology. Specifications define ordinary-looking words precisely, and misreading one is the most common source of implementation bugs.

The body — the actual protocol.

Security Considerations — mandatory in every RFC and consistently the most interesting section. It is where the authors say what they could not fix, what attacks apply, and what implementers get wrong. If you read one section, read this one.

IANA Considerations — registries. Where you find the canonical list of registered values, which is often the thing you actually needed.

ABNF grammar — usually an appendix. The precise syntax, in a formal grammar. If you are writing a parser, this is authoritative and the prose is not.

the things that trip people up#

Updated and obsoleted. An RFC is never edited. It is replaced. Always check the header for "Obsoleted by" — implementing an obsolete specification is a common and embarrassing mistake, and HTTP/1.1 alone has been re-specified several times.

Errata. Published RFCs have errata filed against them. Check them; some are substantive corrections.

Not all RFCs are standards. The series includes Informational, Experimental, Best Current Practice, Historic, and April Fools jokes. The status is in the header. "It's an RFC" is not the same as "it's a standard."

The grammar wins over the prose. Where the ABNF and the description disagree, the ABNF is normative. This is the single most useful thing to know when a specification appears ambiguous.

the reading strategy#

For a specification you need to implement:

1. Abstract, to confirm it is the right document.
2. Check the header for obsoletions, then the errata.
3. Terminology section, properly.
4. Security Considerations — early, not last. It tells you what the hard parts are before you have written anything.
5. Skim for MUST and MUST NOT to get the shape of the requirements.
6. Then the relevant sections in detail, with the ABNF beside you.

That is maybe ninety minutes for a substantial specification, and it is dramatically cheaper than the alternative, which is discovering the requirements one interoperability bug at a time.

why bother#

Because the specification is the only source that is actually authoritative. Blog posts, Stack Overflow answers and library documentation are all someone's interpretation, and interpretations drift.

When two implementations disagree — and they will — the specification is what settles it. Being the person on the team who can read one and say "we are wrong, section 4.2 says the header is case-insensitive" is a genuinely useful thing to be, and it costs one afternoon of learning the conventions.

This is worse than either finishing or never starting, and it is the default outcome unless something prevents it.

why it costs more than both#

Two systems to maintain. Every change lands twice, or lands in one and silently diverges in the other.

Two sets of bugs, plus a third set caused by the interaction.

Nobody knows which is authoritative. New engineers ask; the answer is "it depends."

The benefit never arrives. The reason for the migration — delete the old thing, get the performance, simplify the model — is only realised at 100%. At 70% you have paid the full cost and collected none of the return.

It gets harder over time. The remaining 30% is the hard 30%: the weird integrations, the customer with the bespoke arrangement, the code nobody understands. And it gets harder as the people who understood the original migration leave.

why it happens#

Not laziness. The incentives genuinely point this way:

The easy 70% delivers most of the visible benefit. The graph goes up, the demo works, the announcement is made. The remaining 30% has no visible reward.

The hard cases are hard for a reason. They were skipped because someone did not know how to handle them, and that has not changed.

Priorities move. The migration was urgent in Q1. In Q3 there is a launch.

Nobody owns the finish. The person who drove it moved on, and completion was never anyone's explicit goal.

the mechanisms that actually work#

Name the deletion, not the migration. The project is not "migrate to the new pricing service." It is "delete the old pricing service." The deliverable is the deletion, and the migration is how you get there. This one rewording changes what people track and what counts as done.

Set a date, publicly, at the start. Not "by end of year" — a date, in the plan, with the deletion as the milestone. Dates without deletions slip silently; a date attached to "and then this code is gone" is checkable.

Make the old path visibly worse. Log a warning on every use. Add latency — genuinely, deliberately. Put a banner in the internal tool. The old path being comfortable is why nobody leaves it.

Count the stragglers on a dashboard. "Requests still on the legacy path" as a number that goes down, reviewed weekly. Anything not measured stalls at whatever level nobody notices.

Do the hard cases first. Backwards from the usual instinct and correct. The easy 70% will always be doable; the hard 30% is what determines whether the project is possible at all. Finding out in month one that a case cannot be migrated is a much better outcome than finding out in month nine.

Budget the finish before starting. If you cannot fund the last 30%, do not start. A migration you cannot complete is worse than the system you have.

the decision worth making explicitly#

Sometimes finishing genuinely is not worth it. The remaining cases are rare, the old path works, and the effort is better spent elsewhere.

That is a legitimate call — but it has to be made, written down, and the state made permanent rather than provisional:

The legacy importer stays for CSV uploads from the four enterprise customers using it. New integrations use the API. We are not migrating them; the importer is now a supported, frozen component with an owner, not a migration in progress. Reviewed 2026-09-21, revisit 2028.

That is a completely different thing from a stalled migration, even though the code looks identical. One is a decision with an owner. The other is a mess nobody has admitted to.

The cost of the second one is not the code. It is that every engineer who touches that area has to reconstruct which way things are supposed to be going, and nobody can tell them.

Worth twenty minutes up front.

the options#

Auto-increment integer. Compact, fast, perfect index locality, human-readable in logs.

The problems are real. It leaks volume — /orders/48213 tells a competitor how many orders you have taken. It requires a round trip to the database to learn the ID, which blocks client-side generation and batching. And it makes merging data from two systems a genuine ordeal, because both start at 1.

UUIDv4. Random, globally unique, generatable anywhere without coordination, leaks nothing.

The cost is index behaviour. Random values scatter inserts across the entire B-tree, which destroys cache locality, inflates the index, and causes page splits on every insert. On a large, write-heavy table this is a genuine and measurable problem, not a theoretical one.

UUIDv7. Time-ordered UUID: a millisecond timestamp prefix, then randomness. Keeps global uniqueness and client-side generation, restores insert locality because new rows land at the end of the index.

This is the default I would now recommend for most new systems. It fixes the one serious problem with v4 while keeping everything that made v4 attractive. Postgres has uuidv7() built in; most languages have a library.

The trade: it leaks creation time. Usually fine, occasionally not.

ULID / KSUID and friends. Same idea as v7 — sortable, time-prefixed — with a more compact text encoding. Fine choices. UUIDv7 has the advantage of being a standard with native database support, which matters more than encoding length.

Natural keys. Email, ISBN, SKU. Almost always a mistake as a primary key, because "naturally unique and never changes" turns out to be false: people change email addresses, and standards get revised. Use them as unique constraints, not as the identifier everything else points at.

the two-identifier pattern#

Frequently the right answer is not to choose:

• Internal key: bigint, auto-increment. Used for foreign keys and joins. Compact, fast, never exposed.
• External ID: UUIDv7 or a prefixed string. Used in URLs, APIs, logs, support conversations. Unique index on it.

You get join performance and index locality internally, and no information leakage externally. The cost is one extra column and remembering which is which.

prefix your external IDs#

If you expose identifiers, prefix them by type:

cus_01J9F3K2M4N5P6Q7R8S9T0V1W2
ord_01J9F3K8X1Y2Z3A4B5C6D7E8F9

This is a small thing with an outsized payoff:

• A support engineer can tell what an ID refers to without asking.
• Passing a customer ID where an order ID belongs becomes detectable, and can be rejected at the API boundary rather than producing a confusing not-found.
• Logs and error reports become self-describing.
• You can rotate the encoding later without ambiguity.

Several well-run APIs do this and it is consistently one of the things developers say they appreciate about them.

the practical rules#

Never expose auto-increment integers publicly. Volume leakage plus trivial enumeration.

Do not use UUIDv4 as a clustered primary key on a table that will get large and write-heavy. If you already have, v7 for new tables and consider whether the old one needs a migration — usually it does not, but measure the index bloat before deciding.

Store UUIDs as a native uuid type, not as text. Sixteen bytes versus thirty-six, and correct comparison semantics.

Decide the case and format once, write it down, and validate at the boundary. Half your system accepting hyphenated lowercase and the other half accepting bare uppercase is a bug that surfaces years later in one integration.

Never reuse an ID. Ever. Deleted means gone; the identifier is retired with it. Reuse turns a stale reference from a clean not-found into silent corruption pointing at the wrong record.

Here is where the time actually goes, roughly in order of how often each is the dominant cause.

1. everything talks to a real database#

The most common cause by a wide margin. Each test sets up a database, inserts fixtures, runs, and tears down.

Fixes, in order of return:

Roll back instead of truncating. Wrap each test in a transaction and roll it back. Orders of magnitude faster than deleting rows, and it needs no cleanup code.

Share the schema, not the data. Create the schema once per run, not per test.

Use a tmpfs for the test database. The data does not need to survive; durable writes are pure cost. On Postgres, a data directory in memory plus fsync=off and synchronous_commit=off is dramatically faster and completely inappropriate for anything but tests.

Move the logic out of the database's reach. The deepest fix: if business logic is a pure function of its inputs, its tests do not need a database at all. Tests that are hard to write without infrastructure are telling you about your design.

2. the suite is serial#

Most runners parallelise and most projects have not turned it on, usually because it exposed shared state once and someone reverted it.

That shared state is a bug. Tests that pass only in a particular order have an ordering dependency, and ordering dependencies in tests usually mirror a real one in the code.

Turn on parallelism, fix what breaks, and you will find at least one genuine issue.

3. sleeps#

time.sleep(2)   # wait for the worker to pick it up

Every one of these is pure latency, and they are always tuned to the slowest machine anyone has run on. Fifty of them is a hundred seconds of doing nothing.

Replace with polling on the actual condition:

wait_until(lambda: job.reload().status == "done", timeout=5)

Same worst case, typically a hundredth of the elapsed time, and it fails with a useful message instead of a mysterious assertion.

4. the fixtures are enormous#

A test that needs one user builds an organisation, three teams, forty users and a year of history, because it reuses the "standard" fixture.

Build the minimum. Factories with sensible defaults, overridden per test, beat shared fixture files that grow to serve every case.

5. it is doing real network I/O#

Tests hitting real HTTP endpoints — even internal ones, even mock servers over a socket — pay connection setup and scheduling per call.

Intercept at the client layer rather than the network layer. Most languages have a way to stub the HTTP client in-process, and it is both faster and more deterministic than a local server.

The exception is contract tests, which exist to catch exactly what stubs hide. Keep those, run them separately, and do not let them into the fast suite.

6. everything is an end-to-end test#

Browser-driving tests are two to three orders of magnitude slower than unit tests. A suite made mostly of them is slow no matter what you do.

The ratio that works: a large number of fast unit tests, a moderate number of integration tests around real boundaries, and a small number of end-to-end tests covering the handful of journeys that must never break.

Inverting that ratio is the most expensive testing mistake a team can make, and it is usually the result of not trusting the lower layers rather than a deliberate choice.

the measurement first#

Before fixing anything, get the distribution:

pytest --durations=25
go test ./... -json | jq -r 'select(.Action=="pass") | "\(.Elapsed) \(.Test)"' | sort -rn | head -25

Almost always, a small number of tests dominate. Fixing the slowest twenty is usually the whole job, and it is an afternoon rather than a project.

the target#

Under 10 seconds for the unit suite, run on every save. Under two minutes for everything that gates a merge.

The ten-second number is not arbitrary — it is roughly the threshold past which people stop running tests as they work and start batching. Once that happens you have lost the feedback loop, and the suite's speed stops being a convenience question and starts being a quality one.

The real list is usually fifteen things, and the difference between the two lists is where the surprising outages come from.

the full inventory#

For one service, one request path:

• Datastores. Primary, replicas, cache, search index, object storage.
• Internal services it calls synchronously.
• Third-party APIs — payments, email, geocoding, feature flags, auth.
• Infrastructure control planes. Service discovery, DNS, the secret manager, the config service. These are the ones nobody lists and they are the ones that take everything down.
• Identity. The token issuer, the JWKS endpoint it fetches keys from.
• Observability. If your telemetry agent blocks when the collector is down — and some do — it is in your request path.
• The container registry, at start-up. Fine until you need to scale during an incident and cannot pull an image.
• Certificate infrastructure, including OCSP and the ACME endpoint.
• NTP. Rarely, spectacularly.

The pattern: the dependencies people list are the ones they call. The ones that hurt are the ones they need in order to start, authenticate, or scale.

the two questions per dependency#

For each item, two answers, written down:

What happens when it is slow? Slow is worse than down and much more common. Down fails fast; slow holds your connections, exhausts your pool, and turns one dependency's bad day into your outage.

What happens when it is unavailable? Three possible answers, and only one is a decision:

• We fail. Fine, if the dependency is genuinely essential — a database for a write path.
• We degrade. Serve stale cache, hide the feature, use a default. This is the answer for most non-essential dependencies and it requires code that usually does not exist yet.
• We do not know. This is the real answer for most dependencies, and it is the finding.

the classification that matters#

Sort every dependency into two buckets:

Hard — the request genuinely cannot be served without it. Should be a very short list. Each one caps your maximum availability at its own.

Soft — the request can be served in degraded form. Every soft dependency needs a timeout, a fallback, and a circuit breaker, or it is a hard dependency that nobody has admitted to.

The arithmetic is the reason this matters: five hard dependencies at 99.9% each gives you at best 99.5%, before any failure of your own. If your availability target is higher than that, some of those dependencies have to become soft, and that is an engineering project rather than a target you can declare.

the feature flag trap#

Worth naming specifically because it catches good teams. Feature flag services are adopted as a safety mechanism — a way to turn things off when they go wrong.

Then the flag SDK becomes a hard dependency on the request path, and when the flag service has an outage your service does too. The tool you adopted to make incidents smaller has made one bigger.

The fix is standard for this whole category and rarely applied: cache flag values locally, evaluate from the cache, refresh in the background, and ship a default in the binary. Then the flag service being down means flags are stale, which is survivable.

how to draw it#

Do not buy anything. Open a file:

orders-api
  HARD  postgres-primary        write path; no fallback
  HARD  auth-jwks               cached 1h, so 1h of grace
  SOFT  redis                   cache; miss → primary, 3x latency
  SOFT  inventory-svc           timeout 300ms → show "check availability"
  SOFT  pricing-svc             timeout 200ms → list price, no promo
  SOFT  flags-svc               local cache + baked defaults
  BOOT  vault                   secrets at start; running pods unaffected
  BOOT  registry                image pull; blocks scale-up only

Twenty minutes per service. The BOOT category — needed to start but not to serve — is the one that catches people out during incidents, because those dependencies are invisible right up until you need to replace a pod.

Then, for every SOFT line, check that the fallback described actually exists in code. That check is where the real findings are, and it is usually not the timeout that is missing but the fallback behind it.

The result is systems where the client gives up at 10 seconds, the server keeps working for 60, and the database holds a lock for 300.

the inventory#

For a single HTTP request, in rough order:

Two things stand out. "Often none" appears five times — most application-level timeouts are unset by default. And the values are not coordinated with each other in any way.

the rule that fixes most of it#

Timeouts must decrease as you go deeper.

client 10s
  └─ load balancer 9s
       └─ application 8s
            └─ downstream call 3s (with 1 retry → 6s worst case)
                 └─ connection acquire 1s
                      └─ database statement 2s

Each layer must be shorter than its caller, with room for retries. If an inner layer can outlast its caller, the caller gives up while the work continues — which means you are burning capacity on results nobody will receive. Under load, that is most of your capacity.

deadline propagation#

The better version of the rule: do not configure each timeout independently. Pass the deadline down.

// caller has 10s; every downstream inherits what remains
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()

// 3s into the request, this call gets at most 7s, automatically
resp, err := downstream.Fetch(ctx, id)

Go's context, gRPC's deadlines and equivalents elsewhere exist for this. Once you propagate deadlines, a slow first step automatically shortens the budget for later ones, and work never continues past the point where anyone is waiting.

This is the single highest-value change available in most service codebases, and it is usually a few days of threading a parameter through.

the ones people forget#

Lock wait timeouts. A transaction waiting on a row lock with no timeout waits forever. Set lock_timeout in Postgres, innodb_lock_wait_timeout in MySQL.

DDL statements. A migration that cannot get its lock will queue behind a long transaction — and everything else queues behind it. SET lock_timeout = '3s' before DDL is the line that prevents a large share of migration incidents.

Idle-in-transaction. A connection that opened a transaction and went away holds locks and blocks vacuum indefinitely. idle_in_transaction_session_timeout is the safety net.

Client-side connect vs. read. These are different, and most libraries let you set them separately. A short connect timeout with a long read timeout is usually what you want.

DNS. Rarely configured, occasionally the whole problem.

write them down#

One table, in the repository, listing every timeout in the request path with its current value and where it is configured.

The exercise takes an afternoon and reliably finds at least one inversion — an inner layer waiting longer than the outer one — and at least two places where the value is a framework default nobody chose.

That document then becomes something you can review when latency changes, rather than a set of numbers scattered across six config files and three languages.

the failure mode this prevents#

Without coordinated timeouts, a slow dependency does not degrade your service — it exhausts it. Requests pile up waiting on something their callers abandoned long ago, connections stay held, the pool empties, and healthy requests start failing for want of a connection.

That is a full outage caused by one slow downstream, and the difference between it and a brief latency blip is entirely whether the numbers were coordinated.

The fix is a text file. It costs nothing and it is the difference between an investigation and a random walk.

what goes in it#

14:02  Symptom: 502s on /api/orders, ~4% of requests, started ~13:40.
       Not correlated with deploy (last one 11:20).

14:08  FACT: only POST. GET is clean. (checked: 30min of access logs)
14:15  FACT: all failures have Content-Length > 64KB.
       → hypothesis: body size limit somewhere
14:22  Checked nginx client_max_body_size = 10m. Not it.
14:31  Checked the app's own limit — 100MB. Not it.
14:40  FACT: failures all hit pod-7 and pod-11 (of 12). Others clean.
       → hypothesis abandoned: not size, size just correlates with
         which client sends big bodies, and that client is sticky
14:52  pod-7 and pod-11 started 13:38. Two minutes before symptoms.
       → new hypothesis: bad config in the new pods
15:01  CONFIRMED: those pods have UPSTREAM_TIMEOUT unset → defaulting to 1s

An hour of work, ten lines. Note what it captures: facts with how they were checked, hypotheses explicitly raised and explicitly abandoned, and the moment a correlation turned out to be a red herring.

why it works#

It stops you re-checking things. The single biggest time sink in a long debugging session is doing the same check twice because you cannot remember the result.

It forces hypotheses to be explicit. Written down, "it's a size limit" is a claim you can test and discard. Held in your head, it quietly steers every subsequent action.

It separates fact from guess. The most common way a debugging session goes wrong is a guess getting promoted to a fact through repetition. Labelling them differently prevents it.

It makes handoff possible. When you hit the end of your day, or the end of your knowledge, the notebook is the handoff. Without it the next person starts from zero.

It becomes the postmortem. The timeline is already written, with the dead ends included — which are the most instructive part and the first thing lost to memory.

the three prompts#

When stuck, the notebook gives you somewhere to answer these:

"What changed?" Deploys, config, data volume, dependency versions, traffic shape, time of day. Most bugs in a previously-working system are caused by a change, and enumerating them beats staring at code.

"What do I actually know?" Read back your own FACT lines. Frequently the answer is that you know much less than you thought, and one of your assumptions was never verified.

"What would prove me wrong?" Not what would confirm the theory — what would kill it. If you cannot name that, the theory is not yet a hypothesis.

the format does not matter#

A scratch file, a comment thread on the ticket, a channel where you narrate to yourself. What matters is that it is written, timestamped, and distinguishes what you checked from what you suspect.

The one thing that does not work is keeping it in your head, because the whole point is that your head is where the confusion is.

the compounding version#

Keep the notebooks. A directory of them, named by date and symptom.

Six months in, you have a searchable record of how this system actually fails, which is knowledge that otherwise exists only as a vague feeling in whoever was on call. When the same class of bug returns — and it does — the search takes a minute and the fix takes five.

That is the cheapest institutional memory available to an engineering team, and it is built one text file at a time.

These correlate less than we would like, and the gap is where a lot of otherwise good software fails.

what users actually experience#

Not your service boundaries. Not your consistency model. Not the fact that you handle a partition correctly.

• How many steps to do the common thing.
• Whether it responds instantly or after a spinner.
• Whether an error tells them what to do.
• Whether it does the same thing twice in a row.
• Whether it remembers what they told it last time.

Every one of those is an interface property, and every one is achievable on top of an ugly implementation — or destroyed on top of a beautiful one.

the trade that is usually made backwards#

There is a real tension between internal cleanliness and external simplicity, and it comes up constantly:

• The clean data model exposes three concepts where users think in one.
• The correct API makes the caller specify things they do not care about.
• The properly-separated services mean the UI has to make four calls and handle each failing independently.
• The general solution has eleven configuration options; the specific one would have had zero.

The default resolution is to protect the internals and push the complexity outward, because the internals are what engineers look at and defend in review.

That is backwards. The interface is used far more times than the implementation is read. Complexity at the boundary is multiplied by every user and every call; complexity inside is paid by the people who chose it.

what this looks like in practice#

Collapse concepts at the boundary. If users think of one thing and your model has three, expose one and do the mapping internally. Yes, that is a lossy abstraction and you will occasionally have to break it. That is the correct place to put the pain.

Default everything. Every required parameter is a decision you have forced on someone who has less context than you. Make it optional with a sensible default, and let the people who genuinely need control find the option.

Make the common path one step. Count the actions for the thing 90% of users do 90% of the time. If it is more than two, that is the work.

Absorb the failure. If your architecture means four things can fail independently, the interface should not surface four independent failures. Retry, degrade, or present one coherent state.

Make it fast where they are waiting. A local read, an optimistic update, a cached response. Users cannot tell the difference between "fast because it is well-engineered" and "fast because you cheated" — and neither can anyone else.

the counterweight, honestly#

This is not an argument for shipping a nice facade over a broken system. The internals are what make the interface keep working — under load, at the edges, after a year of changes. Software that is lovely to use and impossible to change dies just as reliably, only slower.

The claim is narrower: when the two genuinely conflict, and they do, the interface should usually win — because it is what the software is for, and because interface decisions are much more expensive to reverse than internal ones. You can rewrite the storage layer. You cannot easily take back a concept you taught a million users.

the test#

Watch someone use your software for the first time without helping them.

Count the moments they hesitate. Each one is a place where your model and theirs diverged, and no amount of internal quality closes that gap.

That exercise is uncomfortable, takes twenty minutes, and consistently produces a better backlog than any amount of architectural discussion.

Those internal interfaces then last longer than the public ones, because nothing external forces them to change and nothing internal makes changing them worth the trouble.

why they matter more than they look#

They are the seams your architecture is made of. A module's public functions are its contract, whether or not you called it a contract.

Everyone reads them. A public API is read by customers. An internal one is read by every engineer who joins, forever.

They shape what is easy. An interface that makes the right thing awkward guarantees people do the wrong thing, and then you have a pattern.

They are hard to change once used. Not because of compatibility, but because finding and updating every caller is work nobody has budgeted.

what to actually apply#

Not the full public-API apparatus. Four things:

1. Name it from the caller's side.

The best test is to write the call before the implementation:

# would a caller guess this?
orders.pending_for(customer_id, since=last_sync)

# or this?
OrderService.get_orders_by_customer_with_status_filter(customer_id, "PENDING", last_sync)

The first reads like the sentence the caller was thinking. The second reads like the implementation leaked into the name.

2. Make the wrong call hard to write.

# a caller can get this wrong and never know
def transfer(from_account, to_account, amount): ...
transfer(b, a, 500)   # silently backwards

# a caller cannot
def transfer(*, source: AccountId, dest: AccountId, cents: int): ...

Keyword-only arguments, distinct types for things that must not be swapped, and units in the name. Every one of these converts a runtime bug into a compile or call-site error.

3. Return something that cannot be misread.

None for "not found" and None for "error" and None for "empty" is three different meanings on one value. A Result, an explicit exception, or distinct return types cost a few lines and remove a category of caller bug.

4. Document the failure modes, not the happy path.

def reserve(sku: str, qty: int) -> Reservation:
    """Reserve stock, or raise.

    Raises:
        OutOfStock: qty exceeds available. Callers should offer backorder.
        SkuUnknown: not a real SKU — this is a bug, not a user error.
        LockTimeout: transient, safe to retry with backoff.
    """

The signature already said what it takes and returns. What it cannot say is which errors are retryable and which are programmer error — and that is exactly what a caller needs.

the deprecation problem#

Internal APIs never get deprecated because there is always something more urgent, and the old one keeps working.

The mechanism that helps is embarrassingly simple: make the old path noisy in development.

def get_orders_by_customer(*args, **kwargs):
    warnings.warn(
        "get_orders_by_customer is replaced by orders.pending_for(); "
        "see ADR-14. Removal targeted 2026-12.",
        DeprecationWarning, stacklevel=2,
    )
    return orders.pending_for(*args, **kwargs)

Warnings in test output that name the replacement and a date get acted on. Silent compatibility shims live forever.

the boundary that actually needs versioning#

Most internal interfaces do not need versions — you can find every caller and change them together. That is the entire advantage of being internal and you should use it rather than building machinery to avoid it.

The exception is any interface crossing a deploy boundary: service to service, or anything consumed by a client you do not deploy simultaneously. There, old and new run at the same time by definition, and you need the same expand-contract discipline as a schema change — add the new field, support both, migrate callers, remove the old.

The mistake is applying that machinery to an in-process module boundary where a single commit could have updated everything.

the one-line version#

Design internal interfaces from the call site, make the wrong call unwriteable, document the errors, and change them freely while you still can — because the window where you can find every caller closes quietly, and nobody notices until they need it open.

Shell is a fine language for gluing programs together. It is a terrible language for doing it safely unless you tell it to be, and telling it to be takes six lines.

the preamble#

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

What each one prevents:

set -e — exit on any command that fails. Without it, a script continues merrily after cd /nonexistent and then runs the rest of its commands in the wrong directory. This is how a cleanup script deletes the wrong thing.

set -u — error on an undefined variable. Without it, rm -rf "$BUILD_DIR/" with an unset BUILD_DIR expands to rm -rf /. This has happened to real people, at real companies, more than once.

set -o pipefail — a pipeline fails if any stage fails, not just the last one. Without it, curl bad-url | jq . succeeds, because jq was happy with the empty input.

IFS=$'\n\t' — stop splitting on spaces. This is what makes filenames with spaces stop being a source of bugs.

Four lines. They convert an entire category of silent wrong behaviour into loud failure.

the next four things#

Quote every expansion. "$var", not $var. Always, including inside [[ ]] where it usually does not matter, because "usually" is not a rule anyone remembers correctly.

Use "${var:?message}" for required inputs.

: "${DATABASE_URL:?DATABASE_URL is required}"

One line, fails immediately with a useful message rather than three steps later with a confusing one.

Make it idempotent. A script that is safe to run twice is a script that is safe to run at all. mkdir -p, rm -f, check-before-create. The second run is the one that happens during an incident when nobody is sure whether the first one worked.

Clean up with trap.

tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT

Now the temporary directory is removed whether the script succeeds, fails, or is interrupted.

the usability part#

A script that other people run needs the same courtesy as any other interface:

usage() {
  cat <<'EOF'
Usage: deploy.sh ENVIRONMENT [--dry-run]

  ENVIRONMENT   staging | production
  --dry-run     print what would happen, change nothing

Requires: awscli >= 2, jq. Reads DEPLOY_ROLE from the environment.
EOF
}
[[ $# -eq 0 || "${1:-}" == "-h" ]] && { usage; exit 0; }

Add a dry-run mode to anything destructive. It costs one conditional and it is the difference between a script people trust and a script people read three times before running.

Echo what you are about to do. set -x is the crude version and it is better than silence. A script that prints "deleting 4 objects from s3://bucket/prefix/" before doing it lets a human catch the mistake.

when to stop using shell#

Shell is right for: calling other programs in sequence, moving files, gluing a pipeline together. Under about a hundred lines.

Switch to a real language when you need:

• Data structures. Bash arrays are a trap and associative arrays are worse.
• Any arithmetic beyond counting.
• Error handling with recovery, rather than exit-on-failure.
• Parsing anything structured. If you are pulling JSON apart with sed, stop.
• Tests. You can test shell, and almost nobody does, which tells you something.

Python or Go for anything past that line. The rewrite is an hour and it pays back the first time somebody has to change it.

the check that costs nothing#

shellcheck scripts/*.sh

Run it in CI. It catches unquoted expansions, useless cat, subshell variable scoping, and roughly a dozen other things that produce silent wrong behaviour.

Every shell script I have ever run it against had at least one real finding. It takes five minutes to add and it is the highest-value linting available for the least-linted language in most repositories.

Run the arithmetic on it and the picture is different.

the arithmetic#

Ten million requests a day at 0.1% is ten thousand failed requests. If a user session averages twenty requests, that is roughly five hundred sessions with at least one failure — every day.

But the failures are not evenly spread, and that is the part that matters. Errors cluster: by endpoint, by tenant, by client version, by region, by input shape. A 0.1% global rate is frequently one customer at 40% and everyone else at 0.001%.

For that customer, your service is broken. Your dashboard says 99.9%.

the slices that reveal it#

Aggregate error rate is nearly useless on its own. The same number means completely different things depending on the distribution beneath it, and you cannot tell which without slicing:

• By customer or tenant. The single highest-value slice in any multi-tenant system. One broken integration hides perfectly in a global average.
• By endpoint. A 0.1% average across forty endpoints can be 4% on the one that matters.
• By client version. Errors concentrated in one mobile build means a bad release, not a server problem.
• By region. Often a networking or dependency issue, not application logic.
• By authenticated vs anonymous. Frequently a different code path entirely.

If your telemetry cannot answer "which customers saw errors in the last hour," that is the gap to close before adding any more dashboards.

the errors that are not 500s#

Counting HTTP status codes undercounts real failure, sometimes badly:

• 200 with an empty result because a downstream timed out and the code swallowed it. The most dangerous category — invisible to every status-based metric.
• 200 with a partial result. The page rendered, three of the eight widgets are missing.
• Client-side failures that never reach your server at all: a bad bundle, a CSP violation, a network drop mid-request.
• Slow enough to be a failure. A request that succeeds after 45 seconds has failed as far as the user is concerned, and it is counted as a success.
• The retry that worked. The user saw a spinner for eight seconds. Your success rate is 100%.

The fix is measuring outcomes rather than responses. Did the order get placed? Did the message send? Those are answerable from your own data and they count the failures that status codes miss.

what a single failure actually costs#

Worth being concrete, because "0.1%" abstracts it away:

• The user retries. If it fails again, some fraction leave.
• Some fraction contact support. Each of those is real money and an engineer's attention.
• Some fraction never come back, and you will not attribute that to today.
• If it is a payment or a submission, someone may not know whether it went through — which generates duplicates, which generates a different class of problem.

None of that appears on the error-rate graph, which is why the graph being green is not the same as things being fine.

the practical program#

Alert on the slice, not the aggregate. "Any single tenant above 5% for ten minutes" catches things the global rate never will.

Sample real failures and read them. Not the counts — the actual requests, with their inputs and their traces. An hour a week reading real failed requests teaches you more about your system than any dashboard.

Put a request ID on the error page. So a user's complaint becomes a trace lookup rather than an investigation.

Track outcomes for your critical journeys. Checkout completed, message delivered, deploy finished. That number is the one to put in the weekly review, not the status-code ratio.

the framing that changes behaviour#

Stop saying "we're at 99.9%."

Start saying "about five hundred people had a broken session yesterday."

Same data. Completely different meeting.

The difference is entirely in the setup.

write the question down first#

A spike without a written question becomes exploration, and exploration has no end condition.

The question has to be answerable and specific:

• Bad: "look into whether we can use X"
• Good: "can X ingest 50k events/second on a single node with our event shape, and what does its failure behaviour look like when the disk fills?"

The second one tells you when you are finished, what to measure, and what would count as a no. The first one runs until someone gets bored.

state the decision it feeds#

A spike exists to unblock a decision. Name the decision explicitly: we will use X for the ingest path, or we will stay with Y and shard it.

If you cannot name the decision, you are not running a spike, you are learning something — which is fine and should be called that, because it has a different budget.

set the box, and mean it#

Two days is the usual right answer. Long enough to get past setup, short enough that being wrong is cheap.

The rule that makes it work: when the box ends, you stop and report, even if you are nearly there. "Nearly there" is where spikes go to become three-week projects. If the answer genuinely needs more time, that is itself a finding — report it and ask for a second box with a narrower question.

the output is a document, not a branch#

This is the part teams skip. The deliverable is one page:

## Question
Can X ingest 50k events/s on one node with our event shape?

## Answer
Yes for the steady state, no for our burst profile.
32k/s sustained, 61k/s for ~40s before the write buffer backs up.

## Evidence
Harness: spike/x-ingest/ (throwaway). Ran on c7g.4xlarge, 20 min,
production event sample from 2026-08-10. Numbers: spike/x-ingest/results.md

## What surprised me
Backpressure is silent — it drops rather than erroring, and the drop
counter is only in a debug endpoint. That is a production risk with X
regardless of throughput.

## Recommendation
Stay with Y and shard. Revisit X if their 3.x backpressure work lands.

## What I did not test
Multi-node, recovery after disk-full, or the managed offering.

Six sections, twenty minutes to write, and it is still useful in a year when somebody asks why the decision went the way it did. The "what I did not test" section is the one that keeps the document honest.

throw the code away#

Spike code is written to answer a question, not to be maintained. It has no tests, no error handling, hard-coded credentials and a main function that does everything.

That is correct — the speed comes from those omissions. The failure is letting it become the foundation of the real implementation, at which point you have shipped a prototype and will spend a year discovering what it does not handle.

Put it in a directory named spike/, or a branch you never merge. Say in the document where it is. Then write the real thing properly, informed by what you learned.

The strongest signal that a team's spikes are working: the spike code never ships.

when a spike is the wrong tool#

• When the answer is already written down. Read the documentation, the changelog, the issue tracker. A surprising amount of "we need to try it" is answerable in an hour of reading.
• When the question is about production behaviour under real load. A spike cannot tell you that. A canary can.
• When the decision is reversible and cheap. Then just pick one. A spike to choose between two libraries you could swap in a day costs more than being wrong.

Spikes are for expensive, hard-to-reverse decisions where the deciding information does not exist yet. That is a narrower set than it feels like, and it is exactly where two days is a bargain.

It has held up for a decade. It is worth restating because a decade is long enough that people have started applying it wrongly, in a specific and predictable way.

why it works#

Known failure modes. The value of a boring technology is not that it is good. It is that when it breaks at 3 a.m., the error message has been pasted into a public forum by someone who then explained the fix.

A hiring pool that already knows it. Every unusual choice is a thing you must teach every new engineer, forever.

Operational surface you can reason about. Boring things have runbooks, monitoring integrations, backup tooling, and a decade of accumulated operational wisdom.

Someone else finds the bugs. A widely-deployed system has been run at scales you will never reach, by people who reported what broke.

the amendment#

Here is what the original argument does not say, and what a decade has taught: boring is a property of a technology at a moment in time, and it moves in both directions.

Things become boring. Things also stop being boring — a project loses its maintainers, its community fragments, its corporate sponsor reorganises, the ecosystem moves and it does not.

So "choose boring" is not a decision you make once. It is a property you have to re-check, and the failure mode nobody plans for is the technology that was the safe choice in 2019 and is now a liability that everyone forgot to reconsider.

The practical version: once a year, for each load-bearing dependency, ask — is this still boring? Is it still maintained, still hired for, still the thing a new engineer would expect? A "yes" costs a minute. A "no" is the most valuable thing you will learn that quarter.

how to tell whether something is boring now#

Not by age. Some old things are unmaintained; some five-year-old things are completely settled.

The signals that actually matter:

• Multiple independent maintainers, ideally from more than one employer.
• A release in the last six months, and a changelog that shows maintenance rather than churn.
• You can hire for it. Search jobs, not GitHub stars.
• Operational documentation written by users, not just by the vendor.
• A migration path off it, documented by people who took it. A technology nobody has successfully left is a technology you cannot leave.
• Boring failure modes. Can you name what happens when it runs out of memory, loses its network, or gets a corrupt config? If nobody has written that down, it is not boring yet.

where to spend the tokens#

The argument's real content is that innovation tokens are scarce, and the scarcity is not about technology risk. It is about attention.

Every unusual choice consumes attention: to learn it, to operate it, to debug it, to explain it. Attention is the actual constraint on an engineering team, and it is far more limited than the budget.

So spend the tokens where the unusual choice is the product. If your differentiator is a query engine, be adventurous about storage and utterly boring about everything else. If your differentiator is a workflow, use the most conventional stack in existence and put all the attention into the workflow.

The teams that get this wrong are almost never wrong about one big choice. They are wrong about six small ones, each individually defensible, that together consumed all the attention that should have gone into the thing customers pay for.

the counterweight#

Taken too far this becomes an argument for never learning anything, and that is its own failure. A team that made every choice in 2016 and never revisited one is not disciplined, it is stuck — and the technology it chose has probably stopped being boring in the meantime.

The synthesis: be adventurous in one place at a time, deliberately, with a written reason. Be boring everywhere else. Re-check annually.

That is a harder discipline than either "always use the new thing" or "never use the new thing," and it is the only one that survives a decade.

It is also a permanent commitment that somebody will still be paying for in eight years. Here is the conversation worth having first.

what a service costs, in full#

Not the code. The code is the cheap part.

• A repository, a build, a deploy pipeline, a rollback path.
• A place to run, sized, with capacity headroom.
• Monitoring, alerting, dashboards, an on-call runbook.
• Secrets, credentials, and their rotation.
• Dependency upgrades and security patching, forever.
• A place in the request path that can now time out, and the circuit breaker and retry policy that implies.
• Documentation, and an owner who is still at the company.
• One more thing a new engineer must learn about.

That list is the same whether the service is four hundred lines or forty thousand. The overhead is fixed, which is why small services are the ones whose economics are worst.

the questions, in order#

1. Could this be a module?

The default answer for new functionality is a well-bounded module in something that already exists. It gets all of the above for free.

The follow-up that matters: if this were a module with an enforced boundary, what would we lose? If the honest answer is "nothing, it would just feel less tidy," write the module.

2. What is the deployment argument?

The strongest reason to extract is that a team needs to release on its own cadence and currently cannot. That is real and it scales with organisation size.

But check it: are they actually blocked, or do they merely deploy together? Two teams that could deploy independently and choose not to have a process problem, not an architecture problem, and a new service will not fix it.

3. What is the scaling argument, with numbers?

"It might need to scale differently" is not an argument. "This component is CPU-bound and spiky while the rest is I/O-bound and steady, and we are currently provisioning for the peak of both" is.

If you cannot state the resource profile that differs, there is no scaling argument.

4. Where does the data live?

The question that sinks most extractions. If the new service needs the same tables the old one does, you have not split anything — you have created two writers to one database, which is worse than one writer, and you now have to coordinate migrations across two deploy cycles.

A service that does not own its data is a distributed monolith with extra latency.

5. What happens when it is down?

If the answer is "the main flow breaks," you have added a failure mode and bought nothing in return. Fault isolation is not free; it is the result of deliberate timeouts, fallbacks and degradation, and if you are going to build those anyway you could have built them around a module.

6. Who owns it in two years?

Name the team. Not the person — the team. Services outlive their authors, and an ownerless service is the one that runs three major versions behind until it is a security incident.

the cases where the answer is yes#

Being fair, because the reflex against splitting can be as unexamined as the reflex toward it.

• A genuinely different resource profile, with numbers.
• A compliance or data-residency boundary that must be enforced structurally.
• A component that has to be written in a different language for a real reason.
• A team that is demonstrably blocked on someone else's release cadence.
• Something with wildly different availability requirements — a batch job that can be down for an hour next to a checkout path that cannot.

In every one of those, the service is buying something specific that a module cannot.

the reversibility test#

Before you create it, ask: if this turns out to be wrong, how do we merge it back?

If the answer is "we would not, we would just live with it," you are making a one-way decision on a hunch. Those deserve more scrutiny than they usually get, and the moment before the repository exists is the last time the scrutiny is free.

It is not. There are four rules and four shapes.

the four rules#

The x-axis is not time. This is the rule everyone gets wrong. Left-to-right is alphabetical or arbitrary — it is not chronological. A frame on the left did not happen before a frame on the right.

Width is total time. A frame's width is the proportion of samples that included it. Wide means expensive. That is the entire message.

Height is stack depth. A frame sitting on another means it was called by it. Tall is not bad; tall just means deep call stacks.

Colour is usually meaningless. In most tools it is random, chosen to make adjacent frames distinguishable. Do not read anything into it unless the tool explicitly says otherwise — some use it for language or module.

That is it. Wide is expensive, stacked means called-by, colour is decoration.

the four shapes#

A wide plateau near the top. One function, doing real work, dominating. This is the good case: a clear, single hotspot. Optimise that function or call it less.

A wide plateau near the bottom, narrowing above. The time is spread across many children. There is no single hotspot; the cost is the whole subtree. Look one level up — usually the answer is "call this subtree fewer times" rather than "make some leaf faster".

A staircase. Deep, narrow, repeating structure. Often recursion, sometimes a framework's middleware chain. Each frame is cheap; the depth is the cost. Look for a way to shorten the chain rather than to optimise any frame in it.

Many thin spikes with nothing dominant. Death by a thousand cuts, or your profile is too short. Check the sample count first — a profile of two hundred samples looks like this regardless of the workload. If the sample count is healthy and it still looks like this, the program is genuinely uniform and your wins are architectural rather than local.

the practical workflow#

Profile the thing you care about, under load that resembles production. A profile of a cold process running a synthetic benchmark measures start-up and your benchmark harness.

Profile long enough. Seconds, not milliseconds. You are sampling; you need samples.

Look at the widest frame you did not expect. Not the widest frame — the widest surprising one. Everyone's profile is dominated by something obvious and irreducible. The win is in the frame that has no business being 8% of your runtime.

Check for time you cannot see. A flame graph of CPU samples shows CPU. If your program spends most of its wall clock waiting on a socket, an on-CPU profile will be nearly empty and entirely misleading. That is what off-CPU profiling and tracing are for, and confusing the two is the most common way people conclude a profiler is lying to them.

differential flame graphs#

Underused and excellent. Profile before, profile after, render the difference — frames that got wider are coloured one way, narrower the other.

This turns "did my change help?" from an argument about noisy averages into a picture. It is also the fastest way to find a performance regression introduced between two releases: profile both, diff, look at what turned red.

the honest caveat#

A flame graph tells you where time went. It does not tell you whether that time was necessary, and it will happily show you a perfectly optimised hot loop that should not have been called at all.

The largest performance wins are almost always "stop doing this work" rather than "do this work faster," and no profiler can suggest that. It can only show you where to point the question.

That gap is where most delivery confusion lives, and it is cheap to close.

the honest definition#

A change is done when the outcome it was meant to produce is observable, and the team would find out if it stopped.

Not "the code is merged." Merged code that is not deployed is inventory. Deployed code that nobody looked at is a hypothesis.

That framing sounds demanding and mostly is not, because for a large share of work the observation is trivial — the feature is behind a flag, the flag is on for the team, someone clicked it. The point is that someone did.

the checklist that actually helps#

Most definition-of-done checklists fail because they list activities rather than properties. "Tests written" is an activity. "The change is covered by a test that would fail if it regressed" is a property, and it is checkable.

The version I have seen work is short:

• The change does what the ticket asked, verified by someone other than the author — a test, a reviewer, or the person who asked for it.
• It is in production, or in a deliberate queue with a named release date.
• A regression would be caught by a test, a metric, or an alert. If none of those apply, that is a decision, not an oversight.
• Its failure mode is known. Someone can say what happens when it breaks.
• It is documented where the next person will look — which is usually the code, sometimes the runbook, rarely the wiki.
• The flag, the branch, and the dead code are cleaned up, or ticketed with a date.

Six lines. If a change satisfies them, it is done in a way that survives the author going on holiday.

the two failure modes#

Done-done-done. Teams that discover the ambiguity often respond by inventing stages: "dev done", "QA done", "really done". This makes the confusion explicit without removing it, and adds ceremony. The fix is one definition, not three adjectives.

Definition as theatre. A twenty-item checklist copied from a blog post, pasted into the wiki, referenced never. If your definition of done is not enforced by something — a pull request template, a merge check, a question someone actually asks in standup — it is decoration.

The test: pick a ticket closed last week and walk the list. If it fails two items and nobody noticed, the definition is not operating.

the part about flags and branches#

The most common way "done" quietly is not: the feature ships behind a flag at 10%, works, and then nobody finishes the rollout. Three months later the flag is still at 10%, the old code path is still there, and the ticket has been closed since March.

That is not done. It is half-deployed with the cleanup unfunded, and it is how codebases accumulate the dual implementations that make everything else harder.

Make the rollout part of the ticket, not a follow-up. A change is not finished when it is enabled for some users; it is finished when the decision has been made either way and the losing path is deleted.

why it is worth the argument#

A shared definition of done is what makes "how much is left" answerable. Without it, the burn-down is measuring something nobody agrees on, estimates are uncomparable between people, and "almost done" means whatever the speaker wants.

It costs one conversation and a paragraph in the repository. It is the cheapest process improvement available to most teams and it is skipped because it sounds like process rather than engineering.

Everything else is decoration that will drift out of sync.

the four that earn their place#

The constraint that came from outside.

# The vendor's API rejects batches over 500 even though their docs say 1000.
# Confirmed with their support, ticket #48812, March 2026.
BATCH_SIZE = 500

Nothing in the code can tell you this. Without it, someone raises the number back to 1000 in eighteen months, ships it, and spends a day debugging.

The alternative that was tried and failed.

// Tried sync.Map here first. It was slower for this access pattern — writes
// dominate and the keys are hot, which is the case sync.Map is worst at.
// Benchmark: bench/cache_test.go, BenchmarkHotKeys.

This is the highest-value comment there is, because it stops the next person repeating your experiment. It converts a day of their work into a paragraph.

The non-obvious ordering.

// Must run before initAuth(): the session store reads the feature flag cache,
// which is populated here. Reversing these fails silently — the user gets the
// default flag set rather than an error.

"Fails silently" is the part that matters. A comment marking a trap is worth more than one marking a happy path.

The deliberate weirdness.

// Deliberately not using the iterator: this is on the hot path and the
// bounds checks cost ~8% on the profile. See PERF-221.

Without this, a well-meaning reviewer "cleans it up" and quietly regresses performance. Deliberate ugliness that isn't labelled reads as accidental ugliness, and accidental ugliness gets fixed.

the ones that do not#

Restating the line. // increment counter above counter += 1. This is the canonical example and it is still everywhere.

Section headers in a long function. // ---- validation ---- is a request for an extract function refactor wearing a disguise.

Commented-out code. Delete it. Git has it. A block of dead code with no note is a question nobody can answer: is this a work in progress, a rollback, or something someone forgot?

Changelogs in the file header. // 2019-03-04 JS: added retry logic. That is what the commit log is for, and it is accurate there.

Comments that duplicate the type. // returns a list of user IDs above fn user_ids() -> Vec<UserId>. The signature already said it, and the signature cannot go stale.

the decay problem#

Every comment is a claim that is not checked by anything. The code changes; the comment does not; now it is actively misleading, and a misleading comment is worse than no comment because it is trusted.

Three things reduce the decay:

Put it as close to the fact as possible. A comment above the line it describes survives longer than one at the top of the file describing behaviour forty lines down.

Prefer things that are checked. A well-named function, a type, a test with a descriptive name, an assertion — all of these carry the same information and break when they become false. Reach for those first; comment only what none of them can express.

Reference something durable. A ticket, a benchmark file, a commit hash, a support ticket number, a link to the vendor's documentation. Then a reader who doubts the comment can check it rather than guess.

the test#

Before writing a comment, ask: could I encode this in a name, a type, or a test instead?

If yes, do that — it will be checked, and the comment will not.

If no — if the information genuinely lives outside the code, in a vendor's behaviour or a decision someone made or an experiment that failed — write it down. That is exactly the case comments exist for, and those comments will still be earning their keep in a decade.

what it did to my engineering#

It forced me to actually understand things. You can hold a fuzzy model of a technology in your head indefinitely and it feels like knowledge. The moment you try to explain it in a paragraph, the fuzziness becomes visible.

I have abandoned drafts because I discovered, four hundred words in, that I did not understand the thing well enough to write about it. Every one of those was more educational than the pieces I finished.

It made me check things. Writing "X is faster than Y" in public means someone will ask for numbers. Knowing that in advance changes how you form the belief in the first place.

It made me notice my own patterns. Reading two hundred pieces of my own writing back, the recurring themes are obvious to me now and were invisible while writing: verification over generation, boring over clever, measurement over intuition, and a persistent suspicion of anything that requires you to trust rather than check.

I did not set out with a thesis. It assembled itself.

It taught me to be wrong in public, which is a skill and is uncomfortable and is the only way to find out you were wrong quickly.

what I have been most wrong about#

I graded myself in December and the pattern held for the following eight months.

I am reliably right about technical trajectories and reliably wrong about adoption.

Local models got good; people did not switch, because hosted models got cheap faster than I expected. RAG got less necessary; the infrastructure repositioned instead of dying, which I have now failed to predict three separate times. Registry security controls were obviously needed; they arrived after the incident rather than before.

The lesson I keep relearning: the technology is the easy part to forecast, and the technology was never the hard part. Human and organizational behavior is where the uncertainty lives and where the consequences land.

I under-predict inertia and over-predict rationality. Almost every wrong call has that shape.

the thing about writing news#

Two hundred pieces, roughly half of them about things that happened in a specific week.

Reading them back, the ones that held up are almost never the ones that reported the event. They are the ones that used the event to explain a mechanism.

Nobody needs my summary of what a company announced. They can read the announcement. What is worth writing is: why does this shape of thing keep happening, and what does it imply for what you should do on Monday.

The news is a prompt. The mechanism is the article.

I did not know that when I started and it took about forty pieces to figure out.

on the archive#

Everything published here is still at its original URL. Nothing has been quietly edited, renamed, or removed. Corrections are marked in place.

That is a deliberate choice and it costs something — there are pieces I would write differently now, and a few I think are wrong.

Leaving them up is the point. A publication that silently revises its history is not a record, it is a marketing surface. The value of an archive is that it shows what someone thought at the time, including when that was wrong, and you can only get that by not touching it.

If you want to know whether to trust a technical writer, check whether their old pieces still exist and whether the wrong ones were corrected in the open.

on the format#

Gray background. Monospace headings. One column. No popups, no cookie banner, no newsletter modal, no autoplaying anything, no third-party JavaScript on any page.

This is not minimalism as an aesthetic. It is that every one of those things was added to a website to serve the publisher at the reader's expense, and the cumulative effect has made reading on the web genuinely unpleasant.

A page should render before you notice it loading. Text is the interface. Nothing moves unless the reader moved it.

Those are not hard constraints to meet. Almost nobody meets them, and the reason is never technical.

the next two hundred#

Same beat. Tech, developers, and the code underneath. News where the news teaches something, essays where the news does not.

More on the verification problem, because it is the defining engineering question of this period and it is nowhere near resolved. More on the craft, because the craft is what survives the tooling cycles. Fewer pieces about model launches, because they have stopped being informative.

Thanks for reading. Corrections are always welcome and get priority over everything else.

— Dom

what stuck#

Mechanical refactors at scale. The clearest win, by a wide margin. Renaming a concept across four hundred files, migrating a deprecated API, converting a pattern used everywhere. Verifiable, tedious, and exactly what the tools are good at.

The important second-order effect: refactors that were too expensive to do now happen. A codebase where cross-cutting cleanup is affordable is a meaningfully better codebase, and that is a permanent improvement rather than a productivity number.

Working in unfamiliar territory. A language you do not know, a framework you have not used, an API you have never touched. The median output in an unfamiliar domain is better than your first attempt, and reading it teaches you the idioms.

This is the use case I would defend most strongly and it is discussed least.

Test generation from a specification. Not "write tests for this function" — that produces tests that assert the implementation. But "here is the behavior, write tests that verify it" works well and it inverts the effort in the right direction.

Investigation. Reading logs, bisecting history, tracing a call path, summarizing a large diff. Parallelizable, cheap, and it saves the expensive resource, which is your attention.

Repository-level instruction files. AGENTS.md and its equivalents became standard practice, and the discipline of writing down how your project actually works improved documentation for humans as a side effect.

what did not stick#

Fully autonomous feature development. The demo works. The real version produces a plausible implementation of a subtly different feature, because the requirements that live in someone's head were never written down.

Agent fleets at high concurrency. The generation scales; the review does not. Two to three concurrent agents with one reviewer turned out to be the practical limit, and the constraint is entirely on the human side.

Orchestration frameworks. Absorbed into the models, as function-calling libraries and JSON-repair libraries were before them. The durable layer was never orchestration.

"Just describe it and it builds." For anything with design decisions, the description that is precise enough to produce the right result is approximately as long as the code, and writing it is the same work.

what actually changed about the job#

Review is the bottleneck, permanently. Generation got roughly two orders of magnitude cheaper. Verification got no cheaper at all. Everything downstream follows from that asymmetry and nothing in two years has changed it.

Tests became the primary artifact. If the implementation is cheap and verification is expensive, effort moves to specification. The teams getting the most out of these tools are the ones with strong test suites, and the correlation is not subtle.

Type systems got a promotion. Every constraint the compiler checks is verification you do not perform by reading. Teams that were ambivalent about strict typing became evangelists, and the reason is always the same: it catches the class of error generated code produces most.

Small diffs became non-negotiable. A machine can produce two thousand lines effortlessly. Accepting it is not a favor to anyone.

The skill that separates people is judgment, not speed. It always was. It is now the only thing, and the gap between engineers who can tell when output is wrong and engineers who cannot is much more visible than it was.

the thing still unresolved#

The apprenticeship problem.

The judgment that makes a senior engineer valuable was acquired by writing a lot of code badly and then debugging it. That work is being automated. Nobody has a replacement for how the next generation acquires it, and junior hiring contracted sharply during exactly the period when the training mechanism was being removed.

I have written about this several times and I still do not have an answer beyond: deliberately do the hard part yourself sometimes, review generated code carefully as a learning exercise, and hire juniors anyway.

That is a partial answer to a structural problem and I am not satisfied with it.

the honest summary, two years in#

These tools are genuinely useful and the useful envelope is narrower and more specific than either the enthusiasts or the skeptics claimed.

They are excellent at bounded, verifiable, tedious work. They are unreliable at anything requiring judgment about what should be built. They multiply output and do not multiply throughput, because throughput is limited by review.

The engineers getting the most from them are the ones who were already good at specifying problems precisely and at telling when something is wrong. That is not a new skill and it was never evenly distributed.

Which is roughly what every previous tooling revolution did: raised the floor, moved the bottleneck, and made expertise more valuable rather than less.

Deleting code is the only refactoring that is unambiguously good. It cannot introduce a bug in the deleted code, because there is no deleted code. It reduces build time, test time, cognitive load, security surface, and the probability that someone reads the wrong thing.

And nobody schedules it.

what accumulates#

Dead code. Never called. Nobody noticed, because nothing fails when unused code exists.

Features nobody uses. Built for a customer who churned, an experiment that ended, a requirement that changed. Still there, still tested, still maintained, still appearing in every search result.

Abandoned abstractions. Someone built a plugin system for the second plugin, which was never written. Now every call goes through a registry that has one entry.

Configuration for conditions that no longer occur. A flag for a migration that completed in 2023.

Compatibility shims. For a version nobody runs, an API that was removed, a browser that no longer exists.

Tests for deleted behavior. Still running, still slow, testing something that cannot happen.

Vendored copies. Of a library that is now a real dependency.

Commented-out code. Always. Delete it. Git remembers.

how to find it#

Coverage, over a long window. Run coverage in production if your language supports it, or over your full integration suite. Anything at zero across a month is a candidate.

Be careful: zero coverage does not prove dead. It might be an error path, a rare branch, or something exercised only in a region you did not sample. Verify before deleting.

Static analysis for unreachable code. Most linters find unreferenced functions within a module. Cross-module dead code is harder and several tools do it.

Feature flags at 100% for over a quarter. The disabled branch is dead code with a switch on it.

Endpoints with no traffic. Log every route. Anything with zero requests in ninety days is a candidate. This is trivially easy to check and almost nobody does.

Deprecation warnings nobody triggers. If you have been logging a deprecation warning for a year and it has never fired, the deprecated thing is unused.

git log on the file. Anything untouched for three years in an actively developed codebase is either perfect or forgotten.

how to do it safely#

Log before you delete. For anything you are not certain about, add logging and wait. A month of zero calls is strong evidence.

def old_thing(x):
    log.warning("old_thing called", stack=traceback.format_stack())
    return new_thing(x)

Delete in a separate commit from any other change. A deletion mixed with a refactor is unreviewable and un-revertable.

Delete the tests too. Tests for deleted code are the most common thing left behind and they will confuse the next person enormously.

Do not comment it out. Do not move it to an old/ directory. Do not keep it "just in case." Git has it. If you genuinely might need it, note the commit hash in the deletion's commit message.

Do it in batches, by area. One area, fully cleaned, is better than a thousand scattered deletions that are impossible to review.

the resistance you will meet#

"What if we need it?" You will not. And if you do, it is in git. In many years I have never seen a team need to recover deleted code that they could not recover.

"Someone might be using it." Measure. That is what the logging is for. Guessing in either direction is worse than checking.

"It works, why touch it?" Because it costs. Every line is read by every person who greps this file, is compiled on every build, is scanned by every security tool, and is a possible place for a future bug.

"That is not a priority." Correct, and it never will be, which is why it has to be scheduled rather than prioritized. Ten percent of one sprint, quarterly, with a line count as the deliverable.

the framing that gets it done#

Make it a competition. Track lines removed. Celebrate the largest deletion of the quarter.

This is slightly silly and it works, because it inverts the default incentive. Engineers are implicitly rewarded for adding — features shipped, code written — and never for removing, even though removing is frequently worth more.

Naming it, tracking it, and praising it is enough to change the behavior.

the number#

A large fraction of most mature codebases is dead or effectively dead. I have never audited one where it was under 10%, and I have seen 40%.

Every line of that is being read, compiled, tested, scanned, and maintained, at a cost nobody has ever measured.

Go find some.

The only mechanism I have seen reliably prevent this is a budget: a number, agreed in advance, that fails the build when exceeded.

why "we should keep it fast" does not work#

Because "fast" has no threshold, so there is never a moment where a specific change is the problem.

Every individual addition is defensible. The library is 12 KB and saves a week. The query is 8 ms and enables a feature. The middleware is 3 ms and improves security.

Ten of those and your page is 400 ms slower, and no single change was wrong. There was no point at which anyone could say no, because the comparison was always "this change versus nothing" rather than "this change versus the budget."

A budget changes the comparison. Now the question is "what are you willing to remove to make room for this," which is a real conversation.

setting the numbers#

Derive them from user-facing outcomes, not from what you currently have.

For a web frontend:

For an API:

Per-layer budgets are the underrated part. A single end-to-end number tells you something regressed. Per-layer budgets tell you where.

total request budget: 200 ms
  auth:        10 ms
  validation:   5 ms
  database:    80 ms
  business:    40 ms
  serialize:   15 ms
  overhead:    50 ms

When the database layer goes to 120 ms, that specific budget fails, and the person who changed the query knows immediately rather than in a month when someone investigates a general slowdown.

This is borrowed directly from game development, where per-subsystem frame budgets have been standard practice for decades.

enforcement#

The budget must fail something, or it is a wish.

In CI, on every pull request:

- name: bundle size
  run: npx size-limit          # fails if over the configured budget

- name: performance test
  run: k6 run --threshold 'http_req_duration{p(99)}<500' load.js

Report the delta on the pull request. "This change adds 8 KB to the main bundle (142 KB → 150 KB, budget 170 KB)." Visible, in context, at the moment of decision.

Allow overrides with a written reason. Not blocking forever — blocking until somebody says why. The record of overrides is itself useful; if you are overriding every week, the budget is wrong or the system is losing.

the budget for query count#

The single most useful backend budget, and the least common.

Count database queries per request in tests. Assert a maximum.

with assert_max_queries(10):
    client.get("/api/orders")

This catches N+1 queries at the moment they are introduced, which is the only cheap time to catch them. An N+1 that reaches production is found weeks later by someone investigating a slow endpoint, and by then it is embedded in an ORM relationship that four other things depend on.

Almost every framework has a way to do this, and almost nobody does.

what happens when you exceed it#

The conversation that a budget forces, in order:

1. Can we make the new thing cheaper? Lazy load it, defer it, make the query better.
2. Can we remove something else? This is the valuable one. It surfaces the feature nobody uses and the library that is doing 5% of what it costs.
3. Should the budget change? Sometimes yes, deliberately, with a reason recorded. A budget that never changes is a budget that will be ignored.
4. Do we not ship this? Rare and it should be available.

Any of those is better than the default, which is that the change lands and the system is permanently slower.

the thing to measure first#

Before setting a budget, get the current numbers and the distribution. p50, p75, p95, p99. On real user hardware and real networks, not on a developer laptop on office wifi.

Then set the budget at roughly where you are, and ratchet it down over time rather than setting an aspirational number you fail immediately.

A budget you exceed on day one gets disabled on day two.

Here is the order I go in, and what each thing tells you.

1. can I run it? (15 minutes)#

Clone it. Follow the README. Start a timer.

This is the single most informative test and most people skip it in favor of reading code.

• Under 10 minutes to a running application: the team cares about developer experience, and probably about a lot of other things.
• An hour, with several undocumented steps: onboarding costs a week and every new hire pays it.
• You cannot get it running: the only people who can work on this are the ones who already have it working. This is a serious risk and it is invisible from the outside.

Note every step that failed. That list is the health assessment.

2. the test suite (10 minutes)#

Run it. Then look at it.

Does it pass? On a clean checkout, first try. If not, that tells you the team has normalized a red build.

How long? Under two minutes is excellent. Over ten and people have stopped running it locally.

What is the ratio of assertions to setup? Read three test files. If setup dominates, the code is heavily coupled.

Are there tests for the error paths? Almost nobody writes these and their presence is a strong positive signal.

Is there a skip or xfail graveyard? Count them. A pile of disabled tests means the suite has been losing a slow argument with reality.

3. the shape of the repository (5 minutes)#

tokei .            # lines by language
git log --oneline | wc -l
git log --format='%an' | sort | uniq -c | sort -rn | head

Contributor concentration. If one person wrote 80% of it and they left, that is the largest risk in the codebase, larger than anything technical.

Language sprawl. Four languages in a small project usually means four sets of tooling and nobody who understands all of it.

Directory structure. Does it reflect the domain or the framework? models/, views/, controllers/ tells you nothing about what the software does. billing/, inventory/, shipping/ tells you everything.

4. the largest files (5 minutes)#

find . -name '*.py' -not -path '*/.venv/*' | xargs wc -l | sort -rn | head -20

Every codebase has a few enormous files. Open the biggest one.

• Is it generated? Fine, ignore it.
• Is it a god object? A 4,000-line service class is where all the complexity accumulated and where all the bugs live.
• When was it last changed? git log -1 on it. If it is huge and changes weekly, that is the hot spot, and any work you do will touch it.

5. dependencies (5 minutes)#

How many? Compare against similar projects. An unusual number in either direction is worth understanding.

How old? Anything more than two major versions behind is an upgrade project waiting for you.

Anything abandoned? Check the largest ones for last release date. A critical dependency with no release in three years is a fork you have not made yet.

Anything surprising? A cryptography library nobody has heard of. A vendored copy of something. A dependency on a specific fork.

6. the commit history (10 minutes)#

git log --oneline -50

Message quality. "fix", "wip", "asdf" versus real descriptions. This is a direct readout of the engineering culture and it is remarkably predictive.

Commit size. Are they atomic, or is every commit a 3,000-line dump?

Is there review? Merge commits from pull requests, or direct pushes to the main branch?

Cadence. Steady, or bursts separated by silence?

7. the things that are missing (5 minutes)#

Frequently the most informative part.

• No CI configuration. Nothing is checked automatically.
• No linter or formatter config. Every file is a different style and every review argues about it.
• No CONTRIBUTING.md or equivalent on a project with multiple contributors.
• No changelog. Nobody knows what changed between versions.
• No architecture documentation of any kind. The system exists only in people's heads.
• No .env.example. You cannot configure it without asking someone.

8. one real feature, end to end (10 minutes)#

Pick a user-visible feature and trace it from the entry point to the database.

This is the highest-value ten minutes. You learn: how many layers, how much indirection, where the business logic lives, whether the abstractions are consistent, and whether you could add a similar feature without asking anyone.

If you cannot follow it in ten minutes, neither can anyone else, and every change will be expensive.

the summary judgment#

After an hour I can usually say:

• How long until a new engineer is productive. (From step 1 and 8.)
• Whether changes are safe. (From step 2.)
• Where the risk is concentrated. (From steps 3 and 4.)
• What the team values. (From steps 6 and 7.)

None of that requires understanding what the software does. It is all structural, and structure is what determines the cost of working in something.

A real changelog is a small amount of work with an outsized return.

what it is for#

Deciding whether to upgrade. The single most common reason someone reads a changelog. They are on version 3.2, version 3.7 exists, and they want to know whether it is worth the risk.

Knowing what will break. The most important information you can provide, and the thing auto-generated release notes are worst at.

Debugging. "This started failing after we upgraded" — a good changelog turns that into "here is the change that caused it" in thirty seconds.

Finding out what exists. People discover features by reading changelogs. This is a real and underrated distribution channel for your own work.

why generated release notes are not enough#

A list of merged pull request titles has three problems.

It is written for the wrong audience. "Refactor connection handling" means something to the maintainer and nothing to the user. What changed for them?

It has no hierarchy. A breaking change and a typo fix appear as sibling bullets of equal weight.

It has no migration guidance. "Remove deprecated parse() method" tells you something broke. It does not tell you what to do about it.

the format#

Keep a Changelog is the established convention and it is good. The structure:

## [4.2.0] - 2026-07-22

### Breaking
- `Client.connect()` no longer accepts a positional timeout.
  Pass `timeout=` as a keyword.
      # before
      client.connect(host, 30)
      # after
      client.connect(host, timeout=30)

### Added
- `Client.ping()` for health checks without a full round trip (#412)
- Support for Unix domain sockets via `unix://` URLs (#398)

### Fixed
- Connections leaked when the handshake timed out (#405).
  If you saw file descriptor exhaustion under load, this was it.

### Deprecated
- `Client.legacy_mode` — will be removed in 5.0. Use `compatibility=`.

### Security
- Fixed a case where credentials could appear in debug logs (GHSA-xxxx-xxxx).
  Affects 4.0.0–4.1.3. Rotate credentials if debug logging was enabled.

the rules that make it useful#

Breaking changes first, always. That is what people are scanning for. Do not bury them under twelve feature bullets.

Include the migration. A breaking change without "do this instead" makes the reader open your source code. Two lines of before-and-after saves everyone time.

Describe the user-visible effect, not the implementation. Not "refactored the retry logic." Rather: "retries now use exponential backoff with jitter; if you relied on the previous fixed 1-second interval, set retry_delay=1.0."

Say who is affected. "If you use X, this changes for you. Otherwise nothing changes." Most readers can then stop reading, which is a service.

Link to the issue or pull request for anyone who wants detail. The changelog is a summary, not a substitute.

Date every release, in ISO format. Version numbers alone do not tell you whether you are two months or three years behind.

Write it as you go, not at release time. An Unreleased section at the top that each pull request adds to. Reconstructing a changelog from git history at release time is miserable and it is why changelogs get skipped.

the security section specifically#

If you fix a security issue, say so, with:

• Which versions are affected.
• What the impact is.
• Whether any action beyond upgrading is required.

That last one is the part that gets omitted and it is critical. "Upgrade to 4.2.0" is insufficient if credentials may have been exposed — the user also needs to rotate them, and they will not know unless you say so.

for internal projects#

The same file works for internal services, and the audience is your future self and the person who takes over the service.

The most valuable internal changelog entries are the ones that record a decision:

## 2026-07-14
- Switched from polling to webhooks for order status. Polling was
  costing ~40 requests/second against the vendor's rate limit and
  we were getting throttled during peaks.

Six months later, when someone asks why there is webhook infrastructure, the answer is in one place.

the return on investment#

Ten minutes per release. In exchange:

• Fewer support questions.
• Faster upgrades by your users, which means fewer people on old versions you have to support.
• Fewer surprised users after a breaking change.
• A record you can search when debugging.

There are not many ten-minute tasks with that profile.