This is worse than either finishing or never starting, and it is the default outcome unless something prevents it.

why it costs more than both#

Two systems to maintain. Every change lands twice, or lands in one and silently diverges in the other.

Two sets of bugs, plus a third set caused by the interaction.

Nobody knows which is authoritative. New engineers ask; the answer is "it depends."

The benefit never arrives. The reason for the migration — delete the old thing, get the performance, simplify the model — is only realised at 100%. At 70% you have paid the full cost and collected none of the return.

It gets harder over time. The remaining 30% is the hard 30%: the weird integrations, the customer with the bespoke arrangement, the code nobody understands. And it gets harder as the people who understood the original migration leave.

why it happens#

Not laziness. The incentives genuinely point this way:

The easy 70% delivers most of the visible benefit. The graph goes up, the demo works, the announcement is made. The remaining 30% has no visible reward.

The hard cases are hard for a reason. They were skipped because someone did not know how to handle them, and that has not changed.

Priorities move. The migration was urgent in Q1. In Q3 there is a launch.

Nobody owns the finish. The person who drove it moved on, and completion was never anyone's explicit goal.

the mechanisms that actually work#

Name the deletion, not the migration. The project is not "migrate to the new pricing service." It is "delete the old pricing service." The deliverable is the deletion, and the migration is how you get there. This one rewording changes what people track and what counts as done.

Set a date, publicly, at the start. Not "by end of year" — a date, in the plan, with the deletion as the milestone. Dates without deletions slip silently; a date attached to "and then this code is gone" is checkable.

Make the old path visibly worse. Log a warning on every use. Add latency — genuinely, deliberately. Put a banner in the internal tool. The old path being comfortable is why nobody leaves it.

Count the stragglers on a dashboard. "Requests still on the legacy path" as a number that goes down, reviewed weekly. Anything not measured stalls at whatever level nobody notices.

Do the hard cases first. Backwards from the usual instinct and correct. The easy 70% will always be doable; the hard 30% is what determines whether the project is possible at all. Finding out in month one that a case cannot be migrated is a much better outcome than finding out in month nine.

Budget the finish before starting. If you cannot fund the last 30%, do not start. A migration you cannot complete is worse than the system you have.

the decision worth making explicitly#

Sometimes finishing genuinely is not worth it. The remaining cases are rare, the old path works, and the effort is better spent elsewhere.

That is a legitimate call — but it has to be made, written down, and the state made permanent rather than provisional:

The legacy importer stays for CSV uploads from the four enterprise customers using it. New integrations use the API. We are not migrating them; the importer is now a supported, frozen component with an owner, not a migration in progress. Reviewed 2026-09-21, revisit 2028.

That is a completely different thing from a stalled migration, even though the code looks identical. One is a decision with an owner. The other is a mess nobody has admitted to.

The cost of the second one is not the code. It is that every engineer who touches that area has to reconstruct which way things are supposed to be going, and nobody can tell them.

Worth twenty minutes up front.

the options#

Auto-increment integer. Compact, fast, perfect index locality, human-readable in logs.

The problems are real. It leaks volume — /orders/48213 tells a competitor how many orders you have taken. It requires a round trip to the database to learn the ID, which blocks client-side generation and batching. And it makes merging data from two systems a genuine ordeal, because both start at 1.

UUIDv4. Random, globally unique, generatable anywhere without coordination, leaks nothing.

The cost is index behaviour. Random values scatter inserts across the entire B-tree, which destroys cache locality, inflates the index, and causes page splits on every insert. On a large, write-heavy table this is a genuine and measurable problem, not a theoretical one.

UUIDv7. Time-ordered UUID: a millisecond timestamp prefix, then randomness. Keeps global uniqueness and client-side generation, restores insert locality because new rows land at the end of the index.

This is the default I would now recommend for most new systems. It fixes the one serious problem with v4 while keeping everything that made v4 attractive. Postgres has uuidv7() built in; most languages have a library.

The trade: it leaks creation time. Usually fine, occasionally not.

ULID / KSUID and friends. Same idea as v7 — sortable, time-prefixed — with a more compact text encoding. Fine choices. UUIDv7 has the advantage of being a standard with native database support, which matters more than encoding length.

Natural keys. Email, ISBN, SKU. Almost always a mistake as a primary key, because "naturally unique and never changes" turns out to be false: people change email addresses, and standards get revised. Use them as unique constraints, not as the identifier everything else points at.

the two-identifier pattern#

Frequently the right answer is not to choose:

• Internal key: bigint, auto-increment. Used for foreign keys and joins. Compact, fast, never exposed.
• External ID: UUIDv7 or a prefixed string. Used in URLs, APIs, logs, support conversations. Unique index on it.

You get join performance and index locality internally, and no information leakage externally. The cost is one extra column and remembering which is which.

prefix your external IDs#

If you expose identifiers, prefix them by type:

cus_01J9F3K2M4N5P6Q7R8S9T0V1W2
ord_01J9F3K8X1Y2Z3A4B5C6D7E8F9

This is a small thing with an outsized payoff:

• A support engineer can tell what an ID refers to without asking.
• Passing a customer ID where an order ID belongs becomes detectable, and can be rejected at the API boundary rather than producing a confusing not-found.
• Logs and error reports become self-describing.
• You can rotate the encoding later without ambiguity.

Several well-run APIs do this and it is consistently one of the things developers say they appreciate about them.

the practical rules#

Never expose auto-increment integers publicly. Volume leakage plus trivial enumeration.

Do not use UUIDv4 as a clustered primary key on a table that will get large and write-heavy. If you already have, v7 for new tables and consider whether the old one needs a migration — usually it does not, but measure the index bloat before deciding.

Store UUIDs as a native uuid type, not as text. Sixteen bytes versus thirty-six, and correct comparison semantics.

Decide the case and format once, write it down, and validate at the boundary. Half your system accepting hyphenated lowercase and the other half accepting bare uppercase is a bug that surfaces years later in one integration.

Never reuse an ID. Ever. Deleted means gone; the identifier is retired with it. Reuse turns a stale reference from a clean not-found into silent corruption pointing at the wrong record.

The real list is usually fifteen things, and the difference between the two lists is where the surprising outages come from.

the full inventory#

For one service, one request path:

• Datastores. Primary, replicas, cache, search index, object storage.
• Internal services it calls synchronously.
• Third-party APIs — payments, email, geocoding, feature flags, auth.
• Infrastructure control planes. Service discovery, DNS, the secret manager, the config service. These are the ones nobody lists and they are the ones that take everything down.
• Identity. The token issuer, the JWKS endpoint it fetches keys from.
• Observability. If your telemetry agent blocks when the collector is down — and some do — it is in your request path.
• The container registry, at start-up. Fine until you need to scale during an incident and cannot pull an image.
• Certificate infrastructure, including OCSP and the ACME endpoint.
• NTP. Rarely, spectacularly.

The pattern: the dependencies people list are the ones they call. The ones that hurt are the ones they need in order to start, authenticate, or scale.

the two questions per dependency#

For each item, two answers, written down:

What happens when it is slow? Slow is worse than down and much more common. Down fails fast; slow holds your connections, exhausts your pool, and turns one dependency's bad day into your outage.

What happens when it is unavailable? Three possible answers, and only one is a decision:

• We fail. Fine, if the dependency is genuinely essential — a database for a write path.
• We degrade. Serve stale cache, hide the feature, use a default. This is the answer for most non-essential dependencies and it requires code that usually does not exist yet.
• We do not know. This is the real answer for most dependencies, and it is the finding.

the classification that matters#

Sort every dependency into two buckets:

Hard — the request genuinely cannot be served without it. Should be a very short list. Each one caps your maximum availability at its own.

Soft — the request can be served in degraded form. Every soft dependency needs a timeout, a fallback, and a circuit breaker, or it is a hard dependency that nobody has admitted to.

The arithmetic is the reason this matters: five hard dependencies at 99.9% each gives you at best 99.5%, before any failure of your own. If your availability target is higher than that, some of those dependencies have to become soft, and that is an engineering project rather than a target you can declare.

the feature flag trap#

Worth naming specifically because it catches good teams. Feature flag services are adopted as a safety mechanism — a way to turn things off when they go wrong.

Then the flag SDK becomes a hard dependency on the request path, and when the flag service has an outage your service does too. The tool you adopted to make incidents smaller has made one bigger.

The fix is standard for this whole category and rarely applied: cache flag values locally, evaluate from the cache, refresh in the background, and ship a default in the binary. Then the flag service being down means flags are stale, which is survivable.

how to draw it#

Do not buy anything. Open a file:

orders-api
  HARD  postgres-primary        write path; no fallback
  HARD  auth-jwks               cached 1h, so 1h of grace
  SOFT  redis                   cache; miss → primary, 3x latency
  SOFT  inventory-svc           timeout 300ms → show "check availability"
  SOFT  pricing-svc             timeout 200ms → list price, no promo
  SOFT  flags-svc               local cache + baked defaults
  BOOT  vault                   secrets at start; running pods unaffected
  BOOT  registry                image pull; blocks scale-up only

Twenty minutes per service. The BOOT category — needed to start but not to serve — is the one that catches people out during incidents, because those dependencies are invisible right up until you need to replace a pod.

Then, for every SOFT line, check that the fallback described actually exists in code. That check is where the real findings are, and it is usually not the timeout that is missing but the fallback behind it.

The result is systems where the client gives up at 10 seconds, the server keeps working for 60, and the database holds a lock for 300.

the inventory#

For a single HTTP request, in rough order:

Two things stand out. "Often none" appears five times — most application-level timeouts are unset by default. And the values are not coordinated with each other in any way.

the rule that fixes most of it#

Timeouts must decrease as you go deeper.

client 10s
  └─ load balancer 9s
       └─ application 8s
            └─ downstream call 3s (with 1 retry → 6s worst case)
                 └─ connection acquire 1s
                      └─ database statement 2s

Each layer must be shorter than its caller, with room for retries. If an inner layer can outlast its caller, the caller gives up while the work continues — which means you are burning capacity on results nobody will receive. Under load, that is most of your capacity.

deadline propagation#

The better version of the rule: do not configure each timeout independently. Pass the deadline down.

// caller has 10s; every downstream inherits what remains
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()

// 3s into the request, this call gets at most 7s, automatically
resp, err := downstream.Fetch(ctx, id)

Go's context, gRPC's deadlines and equivalents elsewhere exist for this. Once you propagate deadlines, a slow first step automatically shortens the budget for later ones, and work never continues past the point where anyone is waiting.

This is the single highest-value change available in most service codebases, and it is usually a few days of threading a parameter through.

the ones people forget#

Lock wait timeouts. A transaction waiting on a row lock with no timeout waits forever. Set lock_timeout in Postgres, innodb_lock_wait_timeout in MySQL.

DDL statements. A migration that cannot get its lock will queue behind a long transaction — and everything else queues behind it. SET lock_timeout = '3s' before DDL is the line that prevents a large share of migration incidents.

Idle-in-transaction. A connection that opened a transaction and went away holds locks and blocks vacuum indefinitely. idle_in_transaction_session_timeout is the safety net.

Client-side connect vs. read. These are different, and most libraries let you set them separately. A short connect timeout with a long read timeout is usually what you want.

DNS. Rarely configured, occasionally the whole problem.

write them down#

One table, in the repository, listing every timeout in the request path with its current value and where it is configured.

The exercise takes an afternoon and reliably finds at least one inversion — an inner layer waiting longer than the outer one — and at least two places where the value is a framework default nobody chose.

That document then becomes something you can review when latency changes, rather than a set of numbers scattered across six config files and three languages.

the failure mode this prevents#

Without coordinated timeouts, a slow dependency does not degrade your service — it exhausts it. Requests pile up waiting on something their callers abandoned long ago, connections stay held, the pool empties, and healthy requests start failing for want of a connection.

That is a full outage caused by one slow downstream, and the difference between it and a brief latency blip is entirely whether the numbers were coordinated.

Anywhere a fixed-size resource is shared by more requests than it has capacity, there is a queue. It has a depth, a wait time, and a failure mode, and almost none of them are instrumented.

the inventory#

The connection pool. Twenty connections, forty concurrent requests: twenty requests are waiting. Pool wait time is the single most under-measured latency component in web applications, and it is invisible in a database query timer because the clock starts after the connection is acquired.

The thread pool or worker pool. Same shape. Requests queue for a worker, and the time spent waiting is not attributed to any handler.

The TCP accept backlog. The kernel holds connections your process has not accepted yet. Overflow silently drops them, and the client sees a timeout with no server-side trace at all.

The HTTP client's per-host connection limit. Most clients cap concurrent connections per destination. Exceed it and your requests queue in the client, before any network activity, invisible to server-side metrics on both ends.

The DNS resolver. A limited number of in-flight lookups with a cache that can stampede on expiry.

The disk queue. Storage devices have a queue depth. Exceed it and I/O waits.

The garbage collector. Not a queue exactly, but the same behaviour: work that accumulates and is paid in a burst.

Rate limiters. A limiter that delays rather than rejecting is a queue with an enforced service rate.

why this matters more than it sounds#

Little's Law: L = λW. Items in the system equals arrival rate times time in system. Rearranged, wait time is queue length over service rate.

That means your queue depth choice is a latency choice, whether or not you made it deliberately. A connection pool with a 30-second acquisition timeout is a promise that some requests will wait 30 seconds.

And these queues compose. A request waits for a worker, then waits for a connection, then waits for the disk. Each is modest; the sum is the p99 nobody can explain, because each layer's own metrics look fine.

how to find them#

Measure acquisition, not just use. Time the wait for the resource separately from the work done with it:

t0 = time.perf_counter()
with pool.acquire() as conn:
    t1 = time.perf_counter()
    result = conn.execute(query)
span.set_attribute("db.pool_wait_ms", (t1 - t0) * 1000)
span.set_attribute("db.query_ms", (time.perf_counter() - t1) * 1000)

Two numbers instead of one. The first one is the one you did not have, and it is frequently the larger.

Add up your span times. If the parent span is 400 ms and the children total 180 ms, the missing 220 ms is queueing somewhere. That gap is the most useful signal in a trace and almost nobody looks for it.

Check netstat -s for accept queue overflows. A non-zero and growing "listen queue overflowed" counter means you are dropping connections before your application sees them.

Load test past capacity deliberately. Push to 3× and watch which metric degrades first. That is your binding queue, and you will not find it at normal load.

the rule#

Every queue needs three things, and most have none:

1. A bounded size. Unbounded is a memory leak with a friendly name.
2. A wait metric — specifically the age of the oldest waiter, which tells you how far behind you are in time rather than in count.
3. A decision about overflow. Reject, drop, or block. Pick one deliberately, because the default is usually "queue forever," and queueing forever means serving requests whose callers gave up ten seconds ago.

Do that for the queues you chose. Then go find the six you did not.

Those internal interfaces then last longer than the public ones, because nothing external forces them to change and nothing internal makes changing them worth the trouble.

why they matter more than they look#

They are the seams your architecture is made of. A module's public functions are its contract, whether or not you called it a contract.

Everyone reads them. A public API is read by customers. An internal one is read by every engineer who joins, forever.

They shape what is easy. An interface that makes the right thing awkward guarantees people do the wrong thing, and then you have a pattern.

They are hard to change once used. Not because of compatibility, but because finding and updating every caller is work nobody has budgeted.

what to actually apply#

Not the full public-API apparatus. Four things:

1. Name it from the caller's side.

The best test is to write the call before the implementation:

# would a caller guess this?
orders.pending_for(customer_id, since=last_sync)

# or this?
OrderService.get_orders_by_customer_with_status_filter(customer_id, "PENDING", last_sync)

The first reads like the sentence the caller was thinking. The second reads like the implementation leaked into the name.

2. Make the wrong call hard to write.

# a caller can get this wrong and never know
def transfer(from_account, to_account, amount): ...
transfer(b, a, 500)   # silently backwards

# a caller cannot
def transfer(*, source: AccountId, dest: AccountId, cents: int): ...

Keyword-only arguments, distinct types for things that must not be swapped, and units in the name. Every one of these converts a runtime bug into a compile or call-site error.

3. Return something that cannot be misread.

None for "not found" and None for "error" and None for "empty" is three different meanings on one value. A Result, an explicit exception, or distinct return types cost a few lines and remove a category of caller bug.

4. Document the failure modes, not the happy path.

def reserve(sku: str, qty: int) -> Reservation:
    """Reserve stock, or raise.

    Raises:
        OutOfStock: qty exceeds available. Callers should offer backorder.
        SkuUnknown: not a real SKU — this is a bug, not a user error.
        LockTimeout: transient, safe to retry with backoff.
    """

The signature already said what it takes and returns. What it cannot say is which errors are retryable and which are programmer error — and that is exactly what a caller needs.

the deprecation problem#

Internal APIs never get deprecated because there is always something more urgent, and the old one keeps working.

The mechanism that helps is embarrassingly simple: make the old path noisy in development.

def get_orders_by_customer(*args, **kwargs):
    warnings.warn(
        "get_orders_by_customer is replaced by orders.pending_for(); "
        "see ADR-14. Removal targeted 2026-12.",
        DeprecationWarning, stacklevel=2,
    )
    return orders.pending_for(*args, **kwargs)

Warnings in test output that name the replacement and a date get acted on. Silent compatibility shims live forever.

the boundary that actually needs versioning#

Most internal interfaces do not need versions — you can find every caller and change them together. That is the entire advantage of being internal and you should use it rather than building machinery to avoid it.

The exception is any interface crossing a deploy boundary: service to service, or anything consumed by a client you do not deploy simultaneously. There, old and new run at the same time by definition, and you need the same expand-contract discipline as a schema change — add the new field, support both, migrate callers, remove the old.

The mistake is applying that machinery to an in-process module boundary where a single commit could have updated everything.

the one-line version#

Design internal interfaces from the call site, make the wrong call unwriteable, document the errors, and change them freely while you still can — because the window where you can find every caller closes quietly, and nobody notices until they need it open.

It has held up for a decade. It is worth restating because a decade is long enough that people have started applying it wrongly, in a specific and predictable way.

why it works#

Known failure modes. The value of a boring technology is not that it is good. It is that when it breaks at 3 a.m., the error message has been pasted into a public forum by someone who then explained the fix.

A hiring pool that already knows it. Every unusual choice is a thing you must teach every new engineer, forever.

Operational surface you can reason about. Boring things have runbooks, monitoring integrations, backup tooling, and a decade of accumulated operational wisdom.

Someone else finds the bugs. A widely-deployed system has been run at scales you will never reach, by people who reported what broke.

the amendment#

Here is what the original argument does not say, and what a decade has taught: boring is a property of a technology at a moment in time, and it moves in both directions.

Things become boring. Things also stop being boring — a project loses its maintainers, its community fragments, its corporate sponsor reorganises, the ecosystem moves and it does not.

So "choose boring" is not a decision you make once. It is a property you have to re-check, and the failure mode nobody plans for is the technology that was the safe choice in 2019 and is now a liability that everyone forgot to reconsider.

The practical version: once a year, for each load-bearing dependency, ask — is this still boring? Is it still maintained, still hired for, still the thing a new engineer would expect? A "yes" costs a minute. A "no" is the most valuable thing you will learn that quarter.

how to tell whether something is boring now#

Not by age. Some old things are unmaintained; some five-year-old things are completely settled.

The signals that actually matter:

• Multiple independent maintainers, ideally from more than one employer.
• A release in the last six months, and a changelog that shows maintenance rather than churn.
• You can hire for it. Search jobs, not GitHub stars.
• Operational documentation written by users, not just by the vendor.
• A migration path off it, documented by people who took it. A technology nobody has successfully left is a technology you cannot leave.
• Boring failure modes. Can you name what happens when it runs out of memory, loses its network, or gets a corrupt config? If nobody has written that down, it is not boring yet.

where to spend the tokens#

The argument's real content is that innovation tokens are scarce, and the scarcity is not about technology risk. It is about attention.

Every unusual choice consumes attention: to learn it, to operate it, to debug it, to explain it. Attention is the actual constraint on an engineering team, and it is far more limited than the budget.

So spend the tokens where the unusual choice is the product. If your differentiator is a query engine, be adventurous about storage and utterly boring about everything else. If your differentiator is a workflow, use the most conventional stack in existence and put all the attention into the workflow.

The teams that get this wrong are almost never wrong about one big choice. They are wrong about six small ones, each individually defensible, that together consumed all the attention that should have gone into the thing customers pay for.

the counterweight#

Taken too far this becomes an argument for never learning anything, and that is its own failure. A team that made every choice in 2016 and never revisited one is not disciplined, it is stuck — and the technology it chose has probably stopped being boring in the meantime.

The synthesis: be adventurous in one place at a time, deliberately, with a written reason. Be boring everywhere else. Re-check annually.

That is a harder discipline than either "always use the new thing" or "never use the new thing," and it is the only one that survives a decade.

It is also a permanent commitment that somebody will still be paying for in eight years. Here is the conversation worth having first.

what a service costs, in full#

Not the code. The code is the cheap part.

• A repository, a build, a deploy pipeline, a rollback path.
• A place to run, sized, with capacity headroom.
• Monitoring, alerting, dashboards, an on-call runbook.
• Secrets, credentials, and their rotation.
• Dependency upgrades and security patching, forever.
• A place in the request path that can now time out, and the circuit breaker and retry policy that implies.
• Documentation, and an owner who is still at the company.
• One more thing a new engineer must learn about.

That list is the same whether the service is four hundred lines or forty thousand. The overhead is fixed, which is why small services are the ones whose economics are worst.

the questions, in order#

1. Could this be a module?

The default answer for new functionality is a well-bounded module in something that already exists. It gets all of the above for free.

The follow-up that matters: if this were a module with an enforced boundary, what would we lose? If the honest answer is "nothing, it would just feel less tidy," write the module.

2. What is the deployment argument?

The strongest reason to extract is that a team needs to release on its own cadence and currently cannot. That is real and it scales with organisation size.

But check it: are they actually blocked, or do they merely deploy together? Two teams that could deploy independently and choose not to have a process problem, not an architecture problem, and a new service will not fix it.

3. What is the scaling argument, with numbers?

"It might need to scale differently" is not an argument. "This component is CPU-bound and spiky while the rest is I/O-bound and steady, and we are currently provisioning for the peak of both" is.

If you cannot state the resource profile that differs, there is no scaling argument.

4. Where does the data live?

The question that sinks most extractions. If the new service needs the same tables the old one does, you have not split anything — you have created two writers to one database, which is worse than one writer, and you now have to coordinate migrations across two deploy cycles.

A service that does not own its data is a distributed monolith with extra latency.

5. What happens when it is down?

If the answer is "the main flow breaks," you have added a failure mode and bought nothing in return. Fault isolation is not free; it is the result of deliberate timeouts, fallbacks and degradation, and if you are going to build those anyway you could have built them around a module.

6. Who owns it in two years?

Name the team. Not the person — the team. Services outlive their authors, and an ownerless service is the one that runs three major versions behind until it is a security incident.

the cases where the answer is yes#

Being fair, because the reflex against splitting can be as unexamined as the reflex toward it.

• A genuinely different resource profile, with numbers.
• A compliance or data-residency boundary that must be enforced structurally.
• A component that has to be written in a different language for a real reason.
• A team that is demonstrably blocked on someone else's release cadence.
• Something with wildly different availability requirements — a batch job that can be down for an hour next to a checkout path that cannot.

In every one of those, the service is buying something specific that a module cannot.

the reversibility test#

Before you create it, ask: if this turns out to be wrong, how do we merge it back?

If the answer is "we would not, we would just live with it," you are making a one-way decision on a hunch. Those deserve more scrutiny than they usually get, and the moment before the repository exists is the last time the scrutiny is free.

Here is a way to actually decide.

the cost of lock-in#

Switching cost. How much engineering time to move off, if you had to.

Pricing power. A vendor who knows you cannot leave prices accordingly. This is real and it is usually the largest ongoing cost.

Capability ceiling. You are limited to what they support, on their timeline.

Correlated risk. They have an outage, you have an outage. They change their terms, you comply. They get acquired and sunset the product, you migrate on their schedule.

the cost of avoiding lock-in#

This is the half that gets ignored, and it is frequently larger.

The abstraction layer itself. Code to write, maintain, test, and debug through. It is a permanent tax and it makes stack traces longer.

Lowest common denominator. Your abstraction can only expose what all candidate providers support. You give up the features that made the good option good.

The abstraction is usually wrong anyway. It was designed against one provider's model. When you actually try to swap, you discover the abstraction encoded assumptions that do not hold, and you rewrite it.

Delayed value. Time spent on portability is time not spent on the product.

the framework#

For each dependency, estimate:

1. Switching cost — engineer-weeks to move.
2. Probability you switch in the next three years.
3. Cost of the abstraction — engineer-weeks now, plus ongoing drag.

Then: if switching_cost × probability < abstraction_cost, do not abstract.

The numbers are rough. The exercise still clarifies, because it forces you to state the probability out loud, and stated probabilities are usually much lower than the implied ones people are acting on.

the categories, worked through#

Object storage. Switching cost: low. The S3 API is a de facto standard and every provider implements it. Probability: moderate — people do move for pricing.

Verdict: use the S3 API, do not abstract further. The API is already the abstraction.

Compute. Switching cost: moderate if containerized, high if you use provider-specific serverless. Probability: low.

Verdict: containerize — which is good practice anyway — and use whatever managed service you want. Do not build a compute abstraction layer.

Relational database. Switching cost: high. Probability: low.

Verdict: use the database's features. Teams that avoid stored procedures, database-specific types, and advanced indexing to stay portable are giving up real capability for an event that will not happen. Postgres-specific SQL is fine. You are not going to migrate to a different engine, and if you do, the SQL dialect will be the smallest part of the pain.

Managed queues, streams, and similar. Switching cost: moderate. The semantics differ enough between providers that a thin abstraction genuinely helps.

Verdict: a thin interface — publish, subscribe, ack — is worth it. Not a full abstraction; a boundary.

Authentication. Switching cost: very high — you have to migrate user credentials, sessions, and integrations. Probability: low, but the consequences of being stuck are severe.

Verdict: use standard protocols. OIDC and SAML are the abstraction. A provider that supports them is replaceable in principle; one with a proprietary SDK is not.

AI model providers. Switching cost: low if you kept the interface thin. Probability: high — the market is moving fast and the right choice changes quarterly.

Verdict: definitely abstract. This is the clearest case on the list. A thin interface plus an eval harness in your own repository makes model changes an afternoon. Teams that did this in 2024 have been switching providers casually ever since.

Observability. Switching cost: moderate to high — instrumentation is everywhere in your code. Probability: moderate, usually driven by cost.

Verdict: OpenTelemetry. The instrumentation is vendor-neutral, the collector handles routing, and swapping backends is a configuration change.

the pattern#

Abstract where the switching probability is high and the abstraction is cheap. AI providers, observability backends, object storage.

Do not abstract where switching is unlikely and the abstraction costs you real capability. Databases, compute platforms, managed services you chose for their specific features.

Use standard protocols wherever they exist. OIDC, S3, OpenTelemetry, SQL. A standard is an abstraction someone else maintains, and it is always cheaper than yours.

the thing that actually protects you#

Not an abstraction layer. Your data, in a format you can export, and a documented process for leaving.

Ask, before adopting anything: can I get all my data out, in a usable format, without their cooperation? If yes, you have real optionality regardless of how coupled your code is, because the expensive part of a migration is never the code — it is the data.

If the answer is no, that is a much bigger red flag than any API coupling, and it is the question almost nobody asks during procurement.

why connections are expensive#

In Postgres specifically, each connection is a separate operating system process with its own memory. A few megabytes of baseline, plus work memory for sorting and hashing, plus its share of shared buffer access.

Four hundred connections means four hundred processes. The scheduler is context switching between them, they are contending for the same locks and buffers, and the memory is largely wasted because most of them are idle.

The counterintuitive result, which has been measured many times: throughput frequently goes down as connection count goes up, past a fairly low threshold.

More connections does not mean more concurrency. It means more contention.

the actual number#

A widely used starting formula:

connections = (core_count × 2) + effective_spindle_count

For an 8-core machine with SSD storage, that is somewhere around 16 to 20.

That number seems shockingly low to people running pools of 100 or more. It is correct, and the reasoning is straightforward: a query is either using CPU or waiting on I/O. You need enough connections to keep the cores busy and to have some work queued behind I/O waits. Past that, additional connections are queued at the database instead of queued in your pool, and queueing at the database is worse because it consumes resources.

Test it. Take your load test, run it at pool sizes of 10, 20, 40, 80, and 160, and plot throughput and p99 latency. The curve rises, flattens, and then degrades. Most people are on the degrading side and have never looked.

the pooler layers#

Three, and they do different things.

Application-side pool. In-process, reuses connections across requests. Every ORM and database driver has one. This is the minimum.

External pooler — PgBouncer, pgcat, or a cloud provider's equivalent. Sits between your application and the database and multiplexes many client connections onto few server connections.

This is what you need when you have many application instances. Twenty instances with a pool of 20 each is 400 connections to the database, even if each instance is mostly idle. A pooler collapses that to the number the database actually wants.

The database's own limit. max_connections. Set it lower than you think — it is a safety valve, and setting it high does not make the database faster, it makes the failure mode worse.

pooling modes, and the one that bites#

External poolers have modes and choosing wrong causes subtle correctness bugs.

Session pooling. A client gets a server connection for the duration of its session. Safe, and provides little multiplexing benefit.

Transaction pooling. A server connection is assigned per transaction and returned after commit. This is where the big multiplexing win is, and it is what most people want.

The catch: anything that depends on session state breaks.

• Prepared statements (unless the pooler supports them explicitly, which newer ones do)
• SET at the session level
• Session-level advisory locks
• LISTEN/NOTIFY
• Temporary tables
• WITH HOLD cursors

If your ORM uses server-side prepared statements by default — many do — you must either disable them or use a pooler that handles them. This is the single most common transaction-pooling problem and it manifests as intermittent errors under load, which is a miserable thing to debug.

Statement pooling. A connection per statement. Maximum multiplexing, breaks multi-statement transactions. Almost never what you want.

the settings that matter#

Beyond size:

Connection timeout. How long a request waits for a connection before failing. This should be short — a few seconds. A request waiting thirty seconds for a connection has already been abandoned by its caller.

Idle timeout. Return connections to the database when not in use. Important with many application instances.

Max lifetime. Recycle connections periodically. This handles the case where a database failover happens and your pool is holding connections to the old primary — without a max lifetime, some pools hold those indefinitely.

Validation. Test a connection before handing it out, or handle the failure on first use. Something has to deal with connections that died while idle.

the diagnostic#

SELECT state, count(*), max(now() - state_change) AS longest
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY state;

What you are looking for:

• Many idle in transaction — the worst state. A transaction is open and doing nothing, holding locks and preventing vacuum. This is an application bug: a transaction opened and not committed, usually because of an early return or an exception path.
• Many idle — pool is oversized. Harmless but wasteful.
• Many active with long durations — queries are slow; the pool is not the problem.

idle in transaction is the one to alert on. It is always a bug and it causes outages that look like database problems and are application problems.

the summary#

Size your pool from a load test, not from a blog post. It is smaller than you think. Use an external pooler in transaction mode if you have many application instances, and check your prepared statement behavior when you do.

the four options#

There are only four. Every system picks one, explicitly or by accident.

1. Buffer. Queue the excess.

Works for bursts. Fails for sustained overload, because a buffer is a delay, and an unbounded buffer is a memory leak with a friendly name. The failure mode is that memory grows until the process dies, taking the buffer with it — so you lose everything, at the worst possible moment.

2. Drop. Discard the excess.

Correct more often than people are comfortable with. Metrics, logs, telemetry, non-critical events — dropping 5% of samples under load is fine and dying is not.

The requirement: know that you dropped, and how much. Silent drops are how you get a dashboard that looks healthy while data is missing.

3. Block. Make the producer wait.

This is real backpressure. The consumer's slowness propagates upstream, the producer slows down, and the system reaches equilibrium at the consumer's rate.

Correct for internal pipelines where the producer can wait. Dangerous when the producer is a user-facing request handler, because now user requests are blocked on a background process.

4. Reject. Tell the producer no.

The right answer at a service boundary. A 429 or 503 returned in one millisecond is much better than a request that waits thirty seconds and then times out, because the caller can make a decision — retry later, degrade, or tell the user.

the wrong default#

Most systems buffer by default, unboundedly, without anyone deciding.

• An in-memory list that grows.
• A queue with no maximum length.
• A connection pool that queues waiters forever.
• A channel with a very large capacity, which is unbounded in practice.

The failure mode is always the same: latency grows, memory grows, and then the process dies. And the requests you were holding were abandoned by their callers ten seconds earlier, so all that work was for nothing.

A queue that is always full is not a buffer. It is a delay you cannot see.

the practical rules#

Every queue has a maximum size. Every one. Choose the number by asking: how long should a request be willing to wait? Multiply by the consumer's rate. That is your queue depth.

If you cannot answer that question, the queue is not designed.

Prefer rejecting to queueing at the edge. When a request arrives and the system is saturated, reject fast. The caller has a timeout; use it as your budget.

Propagate deadlines. If the caller has 200 ms left, every downstream operation should know that. Work performed after the caller has given up is pure waste, and under overload it is the majority of the work being done.

ctx, cancel := context.WithTimeout(ctx, remaining)
defer cancel()

Shed by priority, not randomly. Under load, serve health checks, serve authenticated users, serve the critical path. Shed background work, analytics, and prefetches. Random shedding means your health checks fail and your orchestrator kills healthy instances, which is a self-inflicted outage.

**Measure queue age, not depth.** Depth tells you how many. Age tells you how far behind. "The oldest item has been waiting four minutes" is actionable in a way that "there are 30,000 items" is not.

the pattern that ties it together#

Little's Law: L = λW. Items in the system equals arrival rate times time in system.

Rearranged: wait time equals queue length divided by service rate.

If your queue holds 10,000 items and you process 100 per second, the newest item waits 100 seconds. That is not a hypothetical — it is arithmetic, and it means your queue length choice is your latency choice, whether or not you framed it that way.

Pick the latency you can accept, multiply by the service rate, and that is your maximum queue length. Everything past it gets rejected.

the test#

Point a load generator at your service at three times its capacity. Watch:

• Does memory grow without bound?
• Does latency grow without bound?
• Does anything get rejected, or does everything just get slower?
• After you stop the load, how long until it recovers?

That last one is the important one. A system that takes twenty minutes to recover from a two-minute overload has a backpressure problem, and it will turn a small incident into a large one on a day you did not choose.

the actual trade#

Native gives you: full platform capability, best performance, platform-idiomatic interface, immediate access to new OS features, and the best debugging tools.

It costs you: two or three codebases, two or three teams, features implemented multiple times, and behavior that diverges over years in ways nobody tracks.

Cross-platform gives you: one codebase, one team, features implemented once, consistent behavior.

It costs you: a framework layer between you and the platform, a lag before new OS features are available, worse debugging when the problem is in the bridge, and an interface that is either non-idiomatic on every platform or requires platform-specific work anyway.

the thing the arguments miss#

The largest cost of cross-platform is not performance. It is the escape hatch.

Everything is fine until you need something the framework does not support. Then you are writing platform-specific native code, plus a bridge, plus a fallback, plus tests for all three — and you have the complexity of native development plus the framework.

This happens. It always happens. The question is how often, and that depends entirely on what your app does.

Low escape-hatch pressure: content, commerce, forms, dashboards, CRUD, most business applications. These use the standard widget set and standard capabilities. Cross-platform works well.

High escape-hatch pressure: camera and media processing, background execution, Bluetooth and hardware peripherals, complex custom rendering, deep platform integration (widgets, shortcuts, app extensions), anything real-time.

For high-pressure apps, cross-platform frequently costs more than native, because you pay for the framework and then write the native code anyway.

Be honest about which one you are before choosing. Most teams that regret their choice were high-pressure and assessed themselves as low.

the options, honestly#

React Native. Mature, large ecosystem, native widgets. The new architecture removed the old asynchronous bridge, which was the main performance complaint. Best choice if your team is already React.

Flutter. Renders its own widgets, which means true visual consistency and a non-native feel that some users notice and most do not. Excellent performance for custom interfaces. Dart is a real adoption cost for a team that does not know it.

Kotlin Multiplatform. Share business logic, write native UI. This is the approach I find most defensible: the logic layer — networking, models, validation, persistence — is where duplication is most wasteful and least visible to users, and the UI layer is where platform idiom matters most.

Requires two UI implementations, which is the point rather than a limitation.

Web technologies in a wrapper. Fastest to build if you have web engineers, and the platform feel is the weakest. Fine for content-heavy applications, poor for anything interaction-heavy.

Native. Still correct for a large category, and the category is smaller than native advocates believe.

the organizational question that actually decides it#

Do you have or can you hire two platform teams?

If yes, native is viable and gives you the best result.

If no — and for most companies below a certain size the answer is no — the choice is between cross-platform and shipping on one platform. Framed that way, the decision is usually easy.

What is your feature velocity?

If you ship a large feature monthly, implementing it twice is a permanent 2× cost on your most expensive activity. If you ship quarterly, the duplication matters less.

How much does platform idiom matter to your users?

For a consumer app competing on polish: a lot. For an internal tool: nothing. For a B2B product where the buyer is not the user: less than you think.

the hybrid that most people should consider#

Share the logic, write the UI natively.

The business logic — API clients, data models, validation, offline storage, sync, analytics — is genuinely identical across platforms and duplicating it produces bugs that exist on one platform and not the other, which are the worst bugs to diagnose.

The UI is where platform conventions matter, where users notice, and where the framework abstraction costs the most.

This is more work than full cross-platform and less than full native, and it puts the sharing where the value is.

the thing I would tell someone deciding#

Prototype the hardest thing first.

Not the login screen. The thing you are worried about — the camera flow, the background sync, the complex list, the offline behavior. Build that on your candidate stack, in a week.

You will learn more from that week than from any amount of comparison, and you will learn it while changing your mind is still cheap.

The teams that regret their choice almost always chose based on a comparison article, built the easy part first, and discovered the hard part in month five.

the failure mode#

A downstream service slows down. Every caller times out. Every caller retries.

The downstream now receives double its normal traffic while already struggling. More requests time out. More retries. The load multiplies with every round.

The original problem might have been a five-second blip. The retry storm keeps the service down for twenty minutes, and it stays down after the original cause is resolved because the queued retries are still arriving.

This is metastable failure and retries are its most common cause.

the rules#

1. Only retry idempotent operations.

A retried non-idempotent operation charges the card twice. If you need to retry a write, make it idempotent first with an idempotency key.

2. Only retry retryable errors.

A 400 will be a 400 next time. Retrying it wastes a request and delays the error the caller needs to see.

The one people get wrong: 500 is ambiguous. It might be transient, it might be a deterministic bug. Retrying it once is usually reasonable; retrying it five times is usually pointless.

3. Exponential backoff with jitter. Always jitter.

Without jitter, all your clients retry at the same moments, and you have built a synchronized load generator.

def delay(attempt, base=0.1, cap=30):
    return random.uniform(0, min(cap, base * (2 ** attempt)))

That is full jitter, and it is the recommended default. It spreads retries across the whole interval, which is what you want. Half jitter — d/2 + random(0, d/2) — is a reasonable alternative when you want a guaranteed minimum delay.

The version without jitter is the one everyone writes first and it is the one that causes the storm.

4. Cap the attempts and cap the total time.

Three attempts, usually. And a total deadline — if the caller is going to give up after two seconds, retrying at three seconds is pure waste and it is load on a struggling service.

Propagate the deadline. If your caller has 500 ms left, your retry budget is 500 ms, not your configured default.

5. Do not retry at every layer.

This is the one that produces the shocking numbers. Three attempts at the HTTP client, three in the service wrapper, three in the caller, three at the gateway: 3⁴ = 81 requests for one logical call.

Retry at exactly one layer. Usually the outermost one that has the context to decide. Every other layer fails fast and propagates.

Audit this. Most systems that have grown organically retry at three or four layers and nobody knows.

6. Use a retry budget.

The refinement that actually prevents storms: cap retries as a fraction of total traffic, not per request.

if retries_in_window / requests_in_window > 0.1:
    do_not_retry()

When things are healthy, occasional retries are well under the budget and everything works. When things are broken, the budget is exhausted immediately and retries stop entirely — exactly when they would do the most harm.

This single mechanism converts retries from a failure amplifier into a bounded safety net, and it is not widely implemented.

7. Circuit break.

When a downstream is clearly failing, stop calling it. Fail fast, return a cached or degraded response, and probe occasionally to see if it has recovered.

Three states: closed (normal), open (failing fast), half-open (probing). The half-open state must allow only a trickle — if you send full traffic at a recovering service you will knock it over again.

the server side#

The other half, which is usually forgotten.

Send Retry-After on 429 and 503. Then clients that respect it retry at a time you chose rather than a time they chose.

Shed load rather than queueing it. A request that will time out anyway should be rejected immediately, not queued. Queueing under overload increases latency for everything without increasing throughput, and the requests you eventually serve have often already been abandoned.

Prioritize. Under load, serve health checks and critical paths, shed the rest. An unprioritized overload sheds randomly, which means your health checks fail and your orchestrator kills healthy instances.

the test#

Take your service. Make a downstream dependency return 503 for everything. Watch the request rate at the downstream.

If it goes up by more than a small factor, your retry configuration will cause an outage. You have just not had the trigger yet.

A config file with a wrong value. A feature flag flipped. A generated file that doubled in size. A DNS record. A permission change.

Configuration receives a fraction of the engineering rigor that code does, and it has comparable power to break things.

why it goes wrong#

No type system. A YAML file will happily contain timeout: 30s where the code expects an integer, or enabled: "false" which is a truthy string.

No tests. Nobody writes a test for their config.

No review, or perfunctory review. A config change is "just a value" and gets approved in ten seconds.

Deployed differently from code. Frequently faster, frequently without staging, frequently without a canary. The safety mechanisms built for code deployment routinely do not cover config.

Environment drift. Staging and production differ in ways nobody has enumerated, so staging validates a configuration that is not production's.

No rollback story. "What was this value before" is often unanswerable.

the fixes#

1. Parse and validate at startup, not at use.

The worst failure mode is a config error that manifests three hours later when a rarely-used code path reads a malformed value.

Validate everything at boot. Fail loudly and immediately if anything is wrong.

class Settings(BaseModel):
    database_url: PostgresDsn
    timeout_ms: int = Field(gt=0, le=60_000)
    max_connections: int = Field(ge=1, le=1000)
    feature_new_checkout: bool = False

settings = Settings(**load_config())   # raises at startup, with a clear message

An application that will not start is far better than one that starts and behaves wrongly.

2. Types, with a schema.

Whatever your language, there is a library that turns untyped config into a validated typed object. Use it. This eliminates the entire category of string-that-should-be-a-number bugs.

3. Config changes go through the same pipeline as code.

Version control. Review. Staging. Canary. Rollback.

The argument against is that config changes need to be fast, especially for incident response. That is a real need and the answer is a small explicitly-defined set of emergency levers — kill switches, rate limits — with a fast path, and everything else on the normal pipeline.

Not "all config is fast" because that is how config takes down your service.

4. Validate generated configuration.

If a config file is produced by a program, that program can be wrong. A size check against the previous version, a schema check, a sanity check on record count.

This is the specific failure that has caused several high-profile outages: a generated file changed unexpectedly and propagated globally before anyone looked at it.

5. Make the environment explicit and diff it.

You should be able to answer "how does staging differ from production" with a command. If you cannot, staging is not validating production.

config-diff staging production

6. Log the effective configuration at startup.

Not the file — the resolved values after defaults, overrides, and environment variables are applied. Redact secrets. This is the single most useful thing for debugging "it works on my machine," because the effective config is frequently not what anyone thinks it is.

7. Feature flags need the same discipline as code.

Owner. Expiry. Test both branches. Log the flag state on every event. A flag flip is a production change and should be treated as one.

the hierarchy that works#

Most systems end up with layered configuration and the layering should be explicit and simple:

defaults in code
  ← config file
    ← environment variables
      ← command line flags

Later overrides earlier. Log which layer each effective value came from when debugging.

Keep the layers few. Systems with six overlapping sources of configuration — defaults, file, environment, a service, a database table, a flag system — produce values nobody can trace. Each layer you add makes "why is this value what it is" harder to answer.

secrets#

Not the same thing as configuration and should not be in the same place.

• Never in the config file. Never in version control. Never in the image.
• A secret manager, injected at runtime.
• Rotated on a schedule, and the rotation must be tested.
• Never logged. Redact by default at the logger, not at each call site — because somebody will forget.

the general point#

Configuration is the input to your program that is most likely to be wrong and least likely to be checked.

Every rigor you apply to code — types, validation, tests, review, staged rollout, rollback — applies to configuration, and applying it costs a day of setup.

The reason it does not happen is that config does not feel like code. It is data, and data feels safe.

It is not data. It is the arguments to your program, and passing wrong arguments to a program is how programs go wrong.

It has exceptions, and knowing which situation you are in matters more than knowing the rule.

why rewrites usually fail#

The old system's behavior is undocumented and load-bearing. Every strange conditional in a decade-old codebase is a bug report somebody filed. You will not find them by reading the code, because the code does not say why. You will find them by shipping the rewrite and having those bugs re-reported.

The rewrite has no users, so it gets no feedback. The old system is being exercised by real traffic continuously. The new one is exercised by your test suite, which encodes what you think it should do.

Feature parity is a moving target. The old system keeps getting features, because the business does not stop. You are chasing a target that recedes, and the chase consumes the time you budgeted for the rewrite.

Nobody can justify continuing past month six. The rewrite has produced nothing users can see. The pressure to redirect the team to visible work is enormous and usually wins, leaving you with two systems.

The knowledge is in the people, not the code. And the people who knew have frequently left, which is often why the rewrite was proposed.

the three cases where it is right#

1. The platform is dead.

The framework is unmaintained, the language runtime is past end-of-life and getting CVEs, the vendor discontinued the product, the hardware is unobtainable.

This is not a preference — you are being forced. Rewrite, and be grateful you found out before the security incident.

2. The domain model is fundamentally wrong.

Not "the code is messy." The core abstractions do not correspond to reality, and every feature requires working around them.

The test: can you name a specific business capability that is impossible, not just awkward, in the current model? "We cannot support customers with more than one billing entity, and the fix requires changing what a customer is."

If you cannot name one, you have messy code, not a wrong model, and messy code is fixed by refactoring.

3. The system is small enough that the rewrite is short.

If the whole thing is three weeks of work, the calculus changes entirely. The risk of a three-week rewrite is bounded. Just do it.

The received wisdom is about large systems, and people apply it to small ones where it does not hold.

how to do it when you must#

Strangler fig, never big bang.

Put a routing layer in front of the old system. Move one endpoint at a time to the new implementation. Route traffic gradually. When everything has moved, delete the old system.

             ┌─→ new service (endpoints A, B)
client → router
             └─→ legacy monolith (everything else)

Properties this gives you:

• Value ships continuously. Every migrated endpoint is a delivered improvement.
• Risk is bounded per endpoint. If one goes wrong, route it back.
• The project survives leadership changes, because it is producing visible progress the whole time.
• You learn the old system's real behavior incrementally, at the point where you have to reimplement it.

Run both and compare. For a while, send traffic to both implementations, return the old one's response, and log the differences. This is the single most effective technique for discovering undocumented behavior, and it finds things no amount of code reading would have.

old = legacy.compute(req)
try:
    new = rewritten.compute(req)
    if new != old:
        log.warn("divergence", request=req, old=old, new=new)
except Exception as e:
    log.error("new path failed", error=e, request=req)
return old      # old is still authoritative

Run that for weeks. The divergence log is your actual specification.

Freeze the old system's features. If the business keeps adding to it, you will never catch up. This requires an explicit organizational decision and it is usually the hardest part.

Set a deletion date and mean it. The worst outcome is two systems forever. If the migration stalls at 70%, you have doubled your maintenance surface permanently.

the question to ask first#

Before any rewrite: what specifically will be better, and how will you know?

If the answer is "the code will be cleaner," that is not a business outcome and the project will lose its funding to something that is.

If the answer is "features in this area will take one week instead of three, and here are the last five that took three," you have a case.

The rewrites that succeed are the ones where somebody could state the payback in a sentence. The ones that fail are the ones motivated by taste, and taste is not wrong — it is just not fundable.

The physics is real. The applicability is narrower than the marketing, and the reason is data.

the physics#

Light in fiber travels roughly 200,000 km/s. New York to London and back is about 55 ms of pure propagation, before any processing. Add TLS handshakes, TCP setup, and real-world routing that is not a great circle, and a cross-Atlantic round trip is frequently 100 ms or more.

Running code 20 ms from the user instead of 120 ms is a genuine improvement and it is not achievable any other way.

the problem#

Your data is not at the edge. It is in a database, in one region, and if your edge function needs it, you have moved the compute closer to the user and left the round trip in place — plus added a hop.

user → edge (5ms) → origin database (120ms) → edge → user

That is slower than the user talking to the origin directly, because you added a hop to a path that was always dominated by the database call.

This is the single most common edge computing mistake and it is easy to make, because the architecture diagram looks right.

what edge is genuinely good for#

Anything that needs no origin data:

• Redirects and rewrites. URL normalization, locale routing, legacy path mapping.
• Authentication token validation. A signed JWT can be verified with a public key at the edge, and an invalid request never reaches your origin. This is a real win — you reject bad traffic at the perimeter.
• A/B test assignment. Deterministic hash of a cookie into a bucket. No state required.
• Header manipulation. Security headers, CORS, feature policy.
• Bot filtering and rate limiting. Reject at the edge, before the request costs you anything.
• Personalization of cached content. Fetch the cached page, inject the user's name from a cookie, return. The expensive part stays cached.

Anything where the data is genuinely replicated to the edge:

Several platforms now offer edge-replicated key-value and SQL storage. If your data is small, read-heavy, and tolerant of replication lag — configuration, feature flags, product catalogs, translations — this works well and the latency win is real.

The constraints are real too: writes go to a primary, replication is eventual, and storage per location is limited.

what edge is bad for#

Anything write-heavy. Writes need coordination. Coordination needs a primary. The primary is in one place.

Anything requiring strong consistency. By definition, this needs coordination, which needs round trips, which is what you were trying to avoid.

Anything with a large working set. You cannot replicate a terabyte to three hundred locations.

Anything computationally heavy. Edge runtimes have tight CPU and memory limits. They are designed for milliseconds of work per request.

Anything that needs a specific runtime. Most edge platforms run a constrained JavaScript or WebAssembly environment. Your Python dependency with a C extension is not going there.

the architecture that works#

Layered, with each layer doing what it is good at:

edge      → auth check, rate limit, routing, cached content, header work
regional  → application logic, caching, session state
origin    → the database, the writes, the truth

Most requests are answered at the edge from cache. Some go to a regional application tier. Few reach the origin.

That is a CDN with programmability, which is what edge computing actually is, and framing it that way produces much better decisions than framing it as "serverless everywhere."

the thing to measure first#

Before adopting any of this: where does your latency actually go?

Break down a typical request:

• DNS
• TLS handshake
• Network round trip
• Time to first byte at origin
• Origin processing
• Database time within that
• Response transfer

If origin processing is 400 ms and network is 40 ms, moving compute to the edge addresses 40 ms of a 440 ms problem. Fix the 400 first.

This is the most common reason edge adoption disappoints: it was applied to a latency problem that was not a network problem.

the honest summary#

Edge is a very good CDN with programmability, and that is genuinely valuable — it lets you do real work at the perimeter that used to require an origin request.

It is not a general application platform, and the platforms selling it as one are selling the constraint as a feature.

Use it for the perimeter. Keep your data where it can be consistent.

This is not an edge case. It is the normal operating condition of every distributed system, and idempotency is what makes it survivable.

the fundamental problem#

You send a request. The connection times out.

Did it succeed?

You do not know. There are three possibilities:

1. The request never arrived. Retry is correct.
2. The request arrived and failed. Retry is correct.
3. The request arrived, succeeded, and the response was lost. Retry charges the card twice.

You cannot distinguish these from the client. This is not a limitation of your tooling; it is a theorem about asynchronous networks. There is no protocol that resolves it.

The only resolution is to make retrying safe.

what idempotency means#

An operation is idempotent if performing it multiple times has the same effect as performing it once.

Naturally idempotent:

• PUT /user/123 {"name": "Dom"} — set to a value.
• DELETE /user/123 — the second one finds nothing to do.
• SET balance = 100 — absolute assignment.

Not idempotent:

• POST /orders — creates a new one each time.
• UPDATE accounts SET balance = balance + 100 — relative change.
• Sending an email.
• Incrementing a counter.

the pattern#

For operations that are not naturally idempotent, the client supplies a key and the server remembers it.

POST /payments
Idempotency-Key: 8f14e45f-ea5c-4b0d-9c1a-2f7e3d4a5b6c

{"amount_cents": 4200, "currency": "usd", "source": "card_x"}

Server side:

def create_payment(key, request):
    with transaction():
        existing = lookup(key)
        if existing:
            if existing.request_hash != hash(request):
                raise Conflict("key reused with different parameters")
            return existing.response      # replay, do not re-execute

        result = charge_the_card(request)
        store(key, hash(request), result)
        return result

Four details that matter and are usually missed:

Store the result, not just the key. A retry should return the original response, not a "already processed" error. The client's retry should look like a successful first attempt.

Hash the request. If the same key arrives with different parameters, that is a client bug and you should say so rather than silently returning the wrong result.

Same transaction. The lookup, the work, and the store must be atomic. Otherwise two concurrent retries both find nothing and both execute.

Expire the keys. Days, not forever. Storage is not free and a key from a year ago is not going to be retried.

where else this applies#

Message consumers. At-least-once delivery guarantees duplicates. Same pattern: an idempotency key on the message, a record of processed keys, both in one transaction.

Webhooks you send. Include an event ID. Your receivers will need it, and if you do not provide one they will invent something worse.

Webhooks you receive. Assume duplicates. Every major provider retries and several will send the same event twice under normal operation.

Deployment and provisioning. "Create this resource" should succeed if it already exists in the right state. This is why declarative infrastructure tools work and imperative scripts do not.

Migrations and backfills. They get interrupted. They must be safe to re-run from the beginning.

the design advice#

Prefer absolute over relative. SET balance = 100 is idempotent. ADD 100 is not. When you have the choice, take the absolute form.

Where you cannot — and you often cannot, because concurrent updates need relative operations — use a version or a compare-and-set:

UPDATE accounts SET balance = balance + 100, version = version + 1
WHERE id = $1 AND version = $2;

Zero rows updated means someone else got there first, and you know it.

Let the client generate the key. The client knows whether this is a new request or a retry. The server cannot tell. A server-generated key defeats the entire purpose.

Document it. If your API supports idempotency keys, say so prominently, say how long they are honored, and say what happens on key reuse with different parameters. Consumers who do not know about the mechanism will not use it, and then they will double-charge someone and it will be your incident too.

why this is the one to internalize#

Most distributed systems failures are one of: a duplicate, a message lost, or an out-of-order arrival.

Idempotency makes duplicates safe. Retries make loss survivable — and retries require idempotency to be safe. Which leaves ordering, which is a narrower problem that fewer systems actually have.

So: one property, correctly implemented, defuses the majority of what goes wrong. That is a very good ratio, and it is why this is the thing to get right before anything else.

It is also the move that introduces the most subtle failure modes, and the gap between "we added a queue" and "we added a queue correctly" is large.

what it buys#

Latency. The user gets a response when the work is accepted, not when it is done. A checkout that returns in 80 ms and sends the confirmation email asynchronously is a much better product than one that returns in 900 ms.

Absorbing spikes. A queue is a buffer. Traffic that would overwhelm a synchronous system accumulates and drains. This is the difference between a slow period and an outage.

Isolation. If the email provider is down, checkout still works. The messages accumulate and send later.

Retry for free. A failed message goes back on the queue. A failed synchronous call is a user-visible error.

what it costs#

Eventual consistency, everywhere. The user completed checkout and the confirmation has not arrived. The record exists and the search index does not have it. Every asynchronous boundary introduces a window where the system is inconsistent, and your UI has to be honest about it.

Debugging across the boundary. A synchronous stack trace tells you the whole story. An asynchronous failure requires correlating a producer, a broker, and a consumer, possibly hours apart.

Ordering. Most queues do not guarantee it, or guarantee it only within a partition. If your consumer must process events in order, that is a design constraint that reaches back into how you partition.

Duplicate delivery. Almost all queues are at-least-once. Your consumer will receive the same message twice. If that is not safe, you have a bug that appears under load, weeks after launch.

the rules#

1. Consumers must be idempotent. Non-negotiable.

At-least-once delivery means duplicates. The consumer must produce the same result whether it processes a message once or five times.

The usual implementation: a natural idempotency key, and a record of processed keys.

def handle(msg):
    key = msg.idempotency_key
    with tx():
        if already_processed(key):
            return
        do_the_work(msg)
        mark_processed(key)

The mark_processed must be in the same transaction as the work, or you have moved the race rather than eliminated it.

2. Every queue needs a dead letter queue, and someone must watch it.

A message that fails repeatedly must go somewhere. A DLQ nobody monitors is a place where data goes to be silently lost, which is worse than an error, because errors are visible.

Alert on DLQ depth. Not on it being non-zero — on it growing.

3. Retry with backoff and jitter, and cap the attempts.

Immediate retry on a failing downstream is a denial of service you are performing against yourself. Exponential backoff with jitter, a maximum attempt count, then the DLQ.

4. Monitor queue depth and age, not just throughput.

Throughput looks healthy right up until it does not. The metrics that tell you something is wrong:

• Depth — how many messages are waiting.
• Oldest message age — the most useful single metric. If it is growing, your consumers cannot keep up, and you know how far behind you are in time rather than in count.

5. Decide what happens when the queue is full.

It will be. Reject the producer, drop messages, or block? Each is right in different cases and the default is usually wrong for you. An unbounded queue is not a solution; it is a memory leak with extra steps.

6. Keep the payload small and the reference stable.

Put an ID in the message, not the whole object. The consumer fetches current state. This avoids stale data in the message and keeps the broker fast.

The exception: if you need the state as it was when the event occurred, put it in the message deliberately, and say so.

7. Version your message schema from day one.

Producers and consumers deploy independently. Old consumers will see new messages. Include a version field. Make additive changes only, or handle both shapes.

the thing that surprises people#

Queues do not reduce load. They defer it.

If your consumer processes 100 messages per second and you produce 150, you do not have a working system with a buffer. You have a system that is failing slowly, and the queue depth graph is a countdown.

A queue absorbs bursts. It does not fix a sustained capacity deficit, and the failure mode when you use it that way is a queue that grows for six hours and then an incident where you are simultaneously behind and unable to catch up.

Alert on the age, watch the trend, and size the consumers for the sustained rate.

the levels#

Level 0: LIKE '%query%'.

Works for tiny datasets. No ranking, no stemming, no typo tolerance, and a full table scan on every query.

Fine for an admin tool with a thousand rows. Not fine for anything a customer touches.

Level 1: your database's full-text search.

Postgres tsvector, MySQL full-text, SQLite FTS5. You get stemming, stop words, ranking, and index support.

ALTER TABLE articles ADD COLUMN search tsvector
  GENERATED ALWAYS AS (
    setweight(to_tsvector('english', coalesce(title,'')), 'A') ||
    setweight(to_tsvector('english', coalesce(body,'')), 'B')
  ) STORED;

CREATE INDEX ON articles USING GIN (search);

SELECT id, title, ts_rank(search, q) AS rank
FROM articles, websearch_to_tsquery('english', $1) q
WHERE search @@ q
ORDER BY rank DESC LIMIT 20;

The setweight calls are the part people miss: a match in the title should outrank a match in the body, and without weighting it does not.

This handles most applications. If you have under a few million documents and your users search for terms that appear in them, stop here. One system, no synchronization problem, joins to your relational data.

Level 2: a dedicated search engine.

Elasticsearch, OpenSearch, Typesense, Meilisearch. You get: typo tolerance, faceting, synonyms, custom analyzers, distributed scaling, and much better relevance tuning.

The cost is a second system with a synchronization problem. Your search index is now eventually consistent with your database, and every write path must update both. That inconsistency will produce bugs — a deleted item still appearing in results is the classic — and handling it correctly is real work.

Level 3: hybrid semantic search.

Vector embeddings alongside keyword search, combined with reciprocal rank fusion or a learned reranker.

This handles the case where the user's words are not the document's words. "How do I cancel" should find "Terminating your subscription."

Important: hybrid, not pure vector. Pure semantic search is bad at exact matches — product codes, error numbers, names, function names — precisely the queries where users are most certain about what they want and least tolerant of a wrong answer.

what makes search actually good#

The engine is the easy part. Relevance is the hard part, and it is mostly not about the algorithm.

Weight your fields. Title beats body. Exact phrase beats individual terms. Recent beats old, for content where recency matters.

Use behavioral signals. What users clicked on for similar queries is the strongest relevance signal available, and it requires logging queries and clicks from day one. Retrofitting this means starting your data collection from zero.

Handle the empty result. "No results for X" is a failure. Show something: did-you-mean, related content, popular items, a way to browse. An empty page is where users leave.

Handle the head queries manually. A small number of queries make up a large share of volume. Look at your top hundred, check what they return, and pin the correct answer where it is wrong. This is unglamorous, takes an afternoon, and improves perceived quality more than any algorithmic change.

Log everything. Query, result count, position clicked, whether anything was clicked. The searches that return nothing and the searches where nobody clicks are your improvement backlog, delivered for free.

the instrumentation that matters#

Three metrics:

1. Zero-result rate. Should be low. Every zero-result query is a user who did not find what they wanted.
2. Click-through rate, and the position clicked. If people consistently click the fifth result, your ranking is wrong.
3. Query refinement rate. Users who search, then immediately search again with different words, did not find it the first time.

Most teams have none of these and are tuning relevance by intuition.

the recommendation#

Start at level 1. Postgres full-text search with weighted fields covers more applications than people expect, and it does not introduce a synchronization problem.

Move to level 2 when you have a specific complaint you cannot fix — typo tolerance, faceting at scale, or performance. Move to level 3 when you have evidence that users search for concepts rather than terms.

And whatever level you are at: log the queries. That data is the input to every future improvement and you cannot get it retroactively.

Three distinct things wear the label. They have different causes, different costs, and different correct responses.

1. deliberate debt#

You knew the better solution. You chose the faster one on purpose, for a reason.

"We are hard-coding this to hit the deadline. We will parameterize it in Q3."

This is the original meaning of the metaphor and it is a legitimate engineering tool. Borrowing against future effort to ship now is frequently correct, especially when you are not yet sure the thing will survive.

The failure is not taking on the debt. It is not recording it.

What makes deliberate debt manageable:

• A comment at the site, explaining the trade-off and the condition that would trigger the fix.
• A ticket, linked from the comment.
• A named condition: "when we have more than five customers on this path" is much better than "later."

# DEBT: hardcoded to the US tax table to ship for the March launch.
# Parameterize when we take our first non-US customer. See ENG-4417.
TAX_RATE = 0.0725

That comment costs thirty seconds and it is the difference between debt and mystery.

2. accidental debt#

You did not know better at the time. The requirements changed. The library you chose turned out to be wrong. The abstraction fit the problem you had and not the one you have now.

This is the largest category and it is not anyone's fault. It is the natural consequence of building things under uncertainty.

The response is refactoring as part of ordinary work, not as a project.

The practice that works: when you touch a file, leave it slightly better. Not a rewrite — rename the confusing variable, extract the function that is doing two things, delete the dead branch. Small, continuous, in the same commit as the feature.

Refactoring projects — a quarter dedicated to cleanup — mostly fail. They are unfunded after the first month, they conflict with in-flight work, and they produce large risky changes with no user-visible benefit to justify them.

Continuous small improvement compounds and is nearly invisible in the process. That is the whole trick.

3. structural debt#

The architecture is wrong for what the system now does.

The monolith needs to be split, or the microservices need to be merged. The data model does not represent the domain. The synchronous design cannot support the scale. The framework choice from 2018 is blocking everything.

This is the expensive category and it is qualitatively different from the other two, because you cannot fix it incrementally without a plan. Small improvements to a wrong structure make the wrong structure more entrenched.

The response is a real project, with a real justification, sized honestly.

That requires:

• A specific cost statement. Not "the architecture is bad." "Every new feature in this area takes three times as long as an equivalent feature elsewhere, and here are the last four examples with dates."
• A specific benefit. What becomes possible or fast afterward.
• A path with intermediate value. A twelve-month rewrite with no deliverable until month twelve will be cancelled in month seven. Structure it so each phase ships something.
• A strangler pattern, not a rewrite. Build the new alongside the old, migrate incrementally, delete the old. Big-bang rewrites have a well-documented failure rate and the reason is always the same: the old system's behavior includes a decade of undocumented edge cases nobody enumerated.

how to talk about it#

The conversation that fails: "we need to spend time on technical debt."

The conversation that works: "the last four features in the billing area each took about three weeks. Equivalent features elsewhere take one. The difference is the data model, specifically that subscriptions and invoices share a table. Fixing it is about six weeks and would bring billing features back to normal velocity. We have eleven billing features on the roadmap."

The second version has: a measurement, a cause, a cost, and a payback period. It is a business case, and business cases get funded.

The first version is a complaint, and complaints do not.

the thing nobody says#

Some debt should never be paid.

Code in a system that will be retired, or that nobody has changed in three years, or that works and has no pending requirements — leave it. Ugly code that is stable and unmodified costs you nothing. It is not debt if you never pay interest on it.

The question is never "is this code good." It is "is this code costing us anything." A lot of what gets called technical debt is code that offends someone's taste in a module nobody touches, and refactoring it is a hobby, not engineering.

Here is where the line actually is now, with the reasoning rather than the conclusion, because the line keeps moving and the reasoning does not.

the four variables#

Whether to retrieve or to stuff the context depends on four things:

Corpus size. If everything fits, stuff it. "Fits" now means a large document set — a codebase, a product's full documentation, a year of one team's tickets. It does not mean an enterprise's entire document store.

Query pattern. If you ask many questions against the same corpus, caching the prefix makes stuffing cheap. If every query touches a different slice of a large corpus, retrieval wins.

Latency budget. Prefill on a very large context takes real time even when cached. If you need sub-second responses, a small retrieved context is faster.

Citation requirement. If you must show which source supported a claim, retrieval gives you that structurally. Stuffing requires trusting the model's attribution, which is less reliable than people assume.

the decision, concretely#

Stuff the context when: the corpus is under a few hundred thousand tokens, you query it repeatedly (so caching applies), and you want the model to see relationships between distant parts.

The clearest example remains a codebase. Retrieval over code performs worse than whole-repository context because code's meaning lives in the relationships between files, and chunking destroys exactly that.

Retrieve when: the corpus is large, queries are diverse, latency matters, or you need citations.

Do both when: you have a large corpus with a hot subset. Stuff the hot subset — the style guide, the schema, the core documents — and retrieve from the tail.

what actually changed: retrieval became a tool#

The important architectural shift is not about size. It is that retrieval moved from a preprocessing step to a tool the model calls.

Old shape:

query → embed → search → rerank → stuff top-k → generate

One retrieval, before generation, with k fixed by you.

New shape:

query → model reasons → calls search tool → reads results
      → reasons → searches again with a better query → reads
      → generates answer with citations

The model decides what to search for, evaluates whether the results answered the question, and searches again with a refined query if not.

This is dramatically better and it is better for a specific reason: the first query is usually not the right query. A user asks about "the timeout issue." The right search is for the specific component's retry configuration, which you only know to search for after reading something else.

A fixed one-shot retrieval cannot do that. An agent with a search tool can.

what this means for your pipeline#

Embeddings matter less. When the model can iterate, a mediocre first retrieval is recoverable. When you had one shot, embedding quality was everything.

Keyword search came back. Hybrid search — BM25 plus vectors — consistently outperforms pure vector search, and for a model that can iterate, plain keyword search is often sufficient. Exact terms matter: error codes, function names, product names. Embeddings are bad at exact match and always were.

If you built a pure-vector pipeline in 2023, adding keyword search is probably your biggest available quality improvement.

Chunking matters less, and differently. Instead of chunking for retrieval, store documents whole and retrieve whole documents when they fit. Chunk only what is too large, and chunk on structural boundaries — sections, functions, headings — rather than by token count.

Metadata filtering matters more. The model can specify constraints: this project, this date range, this author. Filtering is cheap, precise, and it is frequently what the query actually needed. Make sure your index supports it.

the practical setup#

For most applications:

1. Postgres with pgvector plus full-text search. One system, hybrid search, metadata filters, and joins to your relational data.
2. Expose search as a tool, not as a preprocessing step. Let the model iterate.
3. Return whole documents where they fit; chunk on structure where they do not.
4. Cache the stable prefix — the instructions, the schema, the core reference material.
5. Measure retrieval quality separately from answer quality. If the answer is wrong, you need to know whether the right document was retrieved. Most teams cannot answer that and debug blind.

That last one is the single most useful piece of instrumentation in a retrieval system and almost nobody has it.

the thing I got wrong#

I said the RAG infrastructure category was solving a temporary problem. The capability got absorbed, as predicted. The infrastructure repositioned rather than disappearing, which I did not predict.

That is the third time I have watched this exact pattern and failed to apply the lesson. Infrastructure around a model limitation rarely dies. It moves to whatever the model still cannot do, which is usually one layer out.

the options#

URL versioning. /v1/users, /v2/users.

Explicit, visible, cacheable, easy to route. Anyone can see which version a request targets by reading the URL, which matters more than it sounds — for debugging, for logs, for support conversations.

The cost: it encourages big-bang versions. /v2 implies everything changed, so teams batch breaking changes into a major version, and consumers face a large migration instead of several small ones.

Header versioning. Accept: application/vnd.example.v2+json or a custom header.

Cleaner URLs, more RESTful in a purist sense, allows finer granularity.

The cost: invisible. You cannot paste a URL and know what it returns. Caching requires Vary handling that intermediaries get wrong. Debugging is harder. In practice most teams that choose this regret the invisibility.

Date-based versioning. API-Version: 2026-04-15.

The consumer pins to a date. Every breaking change gets a new date. Consumers upgrade by changing the date and reading the changelog for the intervening period.

This is what several of the better-run APIs do, and it is my recommendation. The advantage is that breaking changes can be small and frequent rather than large and rare, and each one is individually documented.

The cost: you must maintain compatibility shims for every version you support, which is real engineering work and requires discipline about how long you support them.

No versioning: only additive changes. Never break anything. Add fields, never remove or change them. Add endpoints, never change existing ones.

This works, is the least work for consumers, and is genuinely achievable for many APIs. The cost is accumulating cruft — deprecated fields that must be populated forever, endpoints nobody should use but that cannot be removed.

what actually counts as breaking#

Less obvious than it looks. Breaking:

• Removing a field or an endpoint.
• Renaming anything.
• Changing a type — even integer to string for an ID, which people do.
• Adding a required request field.
• Changing validation to be stricter.
• Changing an error code for an existing condition.
• Changing default values.
• Changing pagination behavior.
• Changing the order of an array that consumers might depend on.

Non-breaking, usually:

• Adding an optional request field.
• Adding a response field. Usually. A consumer with strict schema validation that rejects unknown fields will break, which is why your documentation should state explicitly that clients must tolerate unknown fields.

That last one is worth putting in writing early. "Clients MUST ignore unrecognized fields" in your API documentation, from day one, converts a whole category of future breaking change into a non-breaking one.

the deprecation process that works#

1. Announce, with a date, in the changelog and in the response headers.

Deprecation: Sat, 15 Apr 2026 00:00:00 GMT
Sunset: Wed, 15 Oct 2026 00:00:00 GMT
Link: <https://docs.example.com/migrate/v2>; rel="deprecation"

1. Measure who is still using it. You need per-consumer usage metrics on the

deprecated path or you are guessing. This is the step everyone skips and it is what makes the rest possible.

1. Contact the remaining users directly. Not a blog post. An email to the

specific integrations still calling it.

1. Brownouts. Before the sunset, disable the endpoint for short windows —

an hour, then a day — with clear error messages. This surfaces the consumers who ignored every notice, while they still have time to fix it.

1. Sunset, with a clear error explaining what to do.

The brownout step is the one that separates deprecations that work from deprecations that get postponed four times.

the internal API exception#

For an API consumed only inside your organization, versioning is usually over-engineering. You can find every consumer and change them.

Use a shared schema, break things when needed, coordinate the change, move on. The versioning machinery exists because you cannot coordinate with strangers, and if you can coordinate, skip it.

The failure mode is treating an internal API as external — building versioning, deprecation policies, and compatibility shims for three consumers you could have just updated.

the recommendation#

Date-based versioning for a public API, with a documented support window and a changelog that lists every change with its date.

Additive-only for an internal API, with a shared schema and the discipline to coordinate when you must break something.

And in both cases: state in your documentation, prominently, that clients must ignore unknown fields. It costs nothing now and saves a version bump later.

It is also, for a majority of the teams running it, a substantial amount of complexity in exchange for benefits they do not receive, and saying so is still mildly heretical.

the problem it actually solves#

Kubernetes was built for: many services, many teams, heterogeneous workloads, on a fleet of machines, where you want bin-packing efficiency and declarative self-healing, and where the platform is operated by people whose job that is.

If you have all of those, it is the right answer and there is no close second.

what it costs#

A permanent learning tax. Pods, deployments, services, ingresses, configmaps, secrets, persistent volume claims, storage classes, service accounts, roles, network policies, resource quotas, and a YAML dialect for each. Every engineer who deploys anything must learn a meaningful fraction of it.

Operational surface. Control plane upgrades, node upgrades, CNI plugin, CSI driver, ingress controller, cert manager, metrics server, log shipper. Each is a component that can break and that must be upgraded on someone else's schedule.

Debugging distance. "Why is my service not reachable" has a dozen possible answers across five layers, and diagnosing it requires understanding all of them.

Cost, frequently. A managed control plane plus nodes sized for the platform's own overhead plus the observability stack it needs is often more than the equivalent capacity on simpler infrastructure.

Resume-driven adoption. This is real and worth naming. Kubernetes on your CV is worth money. That is a genuine incentive pointed away from the simplest solution that works.

the honest threshold#

You probably want Kubernetes if:

• More than roughly fifteen to twenty distinct services, deployed independently.
• More than a handful of teams that need to deploy without coordinating.
• You have someone whose job includes operating the platform, not as a side task.
• Genuinely heterogeneous workloads with different scaling characteristics.
• Multi-tenancy requirements with real isolation needs.

You probably do not if:

• Under ten services.
• One or two teams.
• Nobody owns the platform.
• Traffic is predictable.
• You are running one application with a database.

what to do below the threshold#

The options are better than they were, and all of them are boring:

A platform-as-a-service. Push code, it runs. This is the correct answer for a very large number of applications and the reason people avoid it is usually aesthetic.

Containers on a managed container service without the orchestrator — the various "run this container, scale it, load balance it" products every cloud offers. You get containers, autoscaling, and rolling deploys without the platform.

A couple of servers and a process manager. Systemd units, a reverse proxy with automatic certificates, and a deploy script. This runs an enormous amount of traffic, is trivially debuggable, and every engineer already understands it.

Docker Compose on one machine. For staging, for internal tools, for anything where a single host is enough. Unfashionable, works.

the migration-path argument#

"We will need Kubernetes eventually, so we should start now."

This is the most common justification and it is usually wrong, for two reasons.

First, the complexity cost is paid every day from now until then, and "eventually" frequently never arrives.

Second, containerizing your application is the actual hard part of a future migration, and you can do that without an orchestrator. A containerized application running under a simple process manager can move to Kubernetes later in a couple of weeks.

Build the container. Skip the platform until you have the problem it solves.

the position I will defend#

The teams I have seen most successfully run Kubernetes are large organizations with dedicated platform teams, where it is genuinely the right tool.

The teams I have seen most damaged by it are small ones where a single engineer set it up, that engineer left, and the remaining team is operating a system nobody understands and is afraid to touch.

That second failure is common, expensive, and entirely predictable from the staffing at adoption time. If nobody's job is going to be operating the platform, you should not have a platform.

fixed window#

Count requests per fixed interval. Reset the counter at the boundary.

key: user:1234:minute:2026-04-06T14:23
incr, expire 60s, reject if > limit

Pros: trivial to implement, one counter, minimal memory.

Cons: the boundary problem. A client can send the full limit at 14:23:59 and the full limit again at 14:24:00 — double the intended rate, in one second, legally.

Use when: the limit is generous relative to the burst you can absorb, and simplicity matters more than precision. Most internal services.

sliding window log#

Store a timestamp per request. Count the ones inside the window.

Pros: exactly correct. No boundary artifacts.

Cons: memory proportional to the limit times the number of clients. A limit of 10,000 per hour per user across a million users is a lot of timestamps.

Use when: limits are small, precision matters, and client count is bounded. API keys with strict quotas.

sliding window counter#

The practical compromise. Keep counters for the current and previous window, interpolate based on how far into the current window you are.

def allowed(now, limit, window):
    cur_start = now - (now % window)
    elapsed = (now - cur_start) / window
    estimate = prev_count * (1 - elapsed) + cur_count
    return estimate < limit

Pros: two counters per client, no boundary problem in practice, cheap.

Cons: an approximation. Can be slightly wrong at the edges under a very uneven request distribution.

Use when: this is the default. It is what most production rate limiters actually do and it is almost always the right choice.

token bucket#

A bucket holds tokens, refilled at a constant rate up to a maximum. Each request consumes one. Empty bucket, request rejected.

def allowed(bucket, now, rate, capacity):
    bucket.tokens = min(capacity, bucket.tokens + (now - bucket.last) * rate)
    bucket.last = now
    if bucket.tokens >= 1:
        bucket.tokens -= 1
        return True
    return False

Pros: allows bursts up to the bucket capacity while enforcing a long-run average. This matches how real clients behave — idle, then a burst of activity — much better than a strict window.

Cons: two parameters instead of one, and people set them without thinking about what the burst allowance means.

Use when: you want to allow legitimate bursts. Most user-facing APIs. This is the second-best default after sliding window counter and is better when burstiness is expected.

leaky bucket#

A queue drained at a constant rate. Requests enter the queue; overflow is rejected.

Pros: output rate is perfectly smooth, which is what you want when you are protecting a downstream system that cannot handle bursts at all.

Cons: adds latency — requests wait in the queue. Not appropriate for interactive traffic.

Use when: you are shaping traffic to a fixed-capacity downstream, like a third-party API with a hard rate limit, or a legacy system that falls over above a threshold.

the parts everyone gets wrong#

Rate limiting the wrong key. Limiting by IP breaks for users behind NAT and does nothing against a distributed attacker. Limit by authenticated identity where you have one, and by IP only as a coarse pre-auth defense.

Not telling the client anything useful. A 429 with no information is hostile. Send the standard headers:

RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 42
Retry-After: 42

A client that knows when to retry will retry then. A client that does not will hammer you.

No jitter on the client side. If a thousand clients are told to retry in 42 seconds, they all retry at exactly the same moment. Always jitter, and say so in your API documentation.

Rate limiting at the wrong layer. Limiting at the application means the request already consumed a connection, a thread, and possibly a database query. For abuse protection, limit at the edge. For fairness between legitimate users, limit in the application where you know who they are.

Ignoring cost variance. Not all requests are equal. A search that scans a million rows and a health check are both "one request." Weight by cost, or rate-limit expensive endpoints separately, or you will limit the wrong thing.

Global counters in a distributed system. Exact global rate limiting requires coordination on every request, which is a latency and availability problem. The usual answer is per-node limits with the total divided across nodes, accepting some imprecision, or a shared store with local caching and periodic reconciliation.

Decide which imprecision you can live with, deliberately, rather than discovering it during an incident.

the one-line recommendation#

Sliding window counter for fairness, token bucket where bursts are legitimate, limit by authenticated identity, always send Retry-After, always jitter.

This is not a controversial position anymore, which is itself notable, because a decade ago it was.

what it absorbed#

The reason the argument ended is that Postgres kept adding the things people left it for:

JSON. jsonb with indexing, operators, and path queries. The document database use case — schemaless-ish data with nested structure — is handled well enough that the specialized option is rarely worth a second system.

Full-text search. Built in, with ranking, stemming, and index support. Not as good as a dedicated search engine at large scale or with complex relevance requirements. Good enough for the search box on most applications, and one fewer system to operate.

Vector search. pgvector with HNSW indexing. Again: not the best available at extreme scale, and entirely adequate for most retrieval workloads, with the enormous advantage that your vectors sit next to your metadata and you can filter and join in one query.

That last point is underrated. A dedicated vector database that cannot join to your relational data means you do the join in application code, badly.

Time series. Partitioning, BRIN indexes, and extensions cover a lot of the use case.

Geospatial. PostGIS remains the best geospatial implementation in any database, period.

Queues. SELECT ... FOR UPDATE SKIP LOCKED gives you a correct, transactional job queue in about twenty lines. For anything under high-thousands of jobs per second — which is most systems — you do not need a dedicated queue, and the transactional property (enqueue in the same transaction as the state change) removes an entire class of consistency bug.

the operational reality#

One system to operate. Every additional datastore is a backup strategy, a monitoring setup, an upgrade path, a failure mode, an on-call runbook, and a set of consistency questions between it and everything else.

The cost of a second datastore is not the license. It is the permanent operational surface, and teams consistently underestimate it by a large factor.

Transactions across your data. If your users and your documents and your embeddings are in one database, a change to all three is one transaction. Split across three systems, it is a distributed consistency problem you now own.

when it is genuinely wrong#

Being honest about this matters, because "just use Postgres" as a reflex is the same failure mode as any other reflex.

Extreme write throughput on a single logical dataset. Postgres scales writes vertically, well, up to a point. Past that point you need sharding, and Postgres's sharding story is real but less mature than systems designed for it from the start. If you genuinely need millions of writes per second, look elsewhere.

Analytical queries over very large datasets. Postgres is a row store. Scanning a billion rows to compute an aggregate is what column stores are for, and the difference is orders of magnitude. Use a column store — several integrate cleanly with Postgres.

Global multi-region with low write latency everywhere. Postgres replication is primary-based. If you need writes accepted in multiple regions with low latency, that is a distributed database problem and Postgres is not one.

Extremely high-volume caching. Redis exists and is better at being a cache. This is a legitimate second system.

Embedded / on-device. SQLite. Different problem, different answer.

the thing to actually do#

Start with Postgres. Put everything in it. Measure.

When something is genuinely the bottleneck — and you will know, because you will have the metrics — extract that one thing to a specialized system with a clear reason.

That order matters. Teams that start with five specialized systems on the theory that they will need them end up operating five systems for a workload one would have handled, and the complexity is permanent.

the version note#

Whatever you are running, be less than two major versions behind. Postgres's release quality is high, the upgrades are usually boring, and the performance improvements between versions are substantial and free.

The people who have bad Postgres upgrade experiences are the ones who skipped four versions and tried to do it in one jump during an outage.