Shell is a fine language for gluing programs together. It is a terrible language for doing it safely unless you tell it to be, and telling it to be takes six lines.

the preamble#

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

What each one prevents:

set -e — exit on any command that fails. Without it, a script continues merrily after cd /nonexistent and then runs the rest of its commands in the wrong directory. This is how a cleanup script deletes the wrong thing.

set -u — error on an undefined variable. Without it, rm -rf "$BUILD_DIR/" with an unset BUILD_DIR expands to rm -rf /. This has happened to real people, at real companies, more than once.

set -o pipefail — a pipeline fails if any stage fails, not just the last one. Without it, curl bad-url | jq . succeeds, because jq was happy with the empty input.

IFS=$'\n\t' — stop splitting on spaces. This is what makes filenames with spaces stop being a source of bugs.

Four lines. They convert an entire category of silent wrong behaviour into loud failure.

the next four things#

Quote every expansion. "$var", not $var. Always, including inside [[ ]] where it usually does not matter, because "usually" is not a rule anyone remembers correctly.

Use "${var:?message}" for required inputs.

: "${DATABASE_URL:?DATABASE_URL is required}"

One line, fails immediately with a useful message rather than three steps later with a confusing one.

Make it idempotent. A script that is safe to run twice is a script that is safe to run at all. mkdir -p, rm -f, check-before-create. The second run is the one that happens during an incident when nobody is sure whether the first one worked.

Clean up with trap.

tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT

Now the temporary directory is removed whether the script succeeds, fails, or is interrupted.

the usability part#

A script that other people run needs the same courtesy as any other interface:

usage() {
  cat <<'EOF'
Usage: deploy.sh ENVIRONMENT [--dry-run]

  ENVIRONMENT   staging | production
  --dry-run     print what would happen, change nothing

Requires: awscli >= 2, jq. Reads DEPLOY_ROLE from the environment.
EOF
}
[[ $# -eq 0 || "${1:-}" == "-h" ]] && { usage; exit 0; }

Add a dry-run mode to anything destructive. It costs one conditional and it is the difference between a script people trust and a script people read three times before running.

Echo what you are about to do. set -x is the crude version and it is better than silence. A script that prints "deleting 4 objects from s3://bucket/prefix/" before doing it lets a human catch the mistake.

when to stop using shell#

Shell is right for: calling other programs in sequence, moving files, gluing a pipeline together. Under about a hundred lines.

Switch to a real language when you need:

• Data structures. Bash arrays are a trap and associative arrays are worse.
• Any arithmetic beyond counting.
• Error handling with recovery, rather than exit-on-failure.
• Parsing anything structured. If you are pulling JSON apart with sed, stop.
• Tests. You can test shell, and almost nobody does, which tells you something.

Python or Go for anything past that line. The rewrite is an hour and it pays back the first time somebody has to change it.

the check that costs nothing#

shellcheck scripts/*.sh

Run it in CI. It catches unquoted expansions, useless cat, subshell variable scoping, and roughly a dozen other things that produce silent wrong behaviour.

Every shell script I have ever run it against had at least one real finding. It takes five minutes to add and it is the highest-value linting available for the least-linted language in most repositories.

Here is a way to actually decide.

the cost of lock-in#

Switching cost. How much engineering time to move off, if you had to.

Pricing power. A vendor who knows you cannot leave prices accordingly. This is real and it is usually the largest ongoing cost.

Capability ceiling. You are limited to what they support, on their timeline.

Correlated risk. They have an outage, you have an outage. They change their terms, you comply. They get acquired and sunset the product, you migrate on their schedule.

the cost of avoiding lock-in#

This is the half that gets ignored, and it is frequently larger.

The abstraction layer itself. Code to write, maintain, test, and debug through. It is a permanent tax and it makes stack traces longer.

Lowest common denominator. Your abstraction can only expose what all candidate providers support. You give up the features that made the good option good.

The abstraction is usually wrong anyway. It was designed against one provider's model. When you actually try to swap, you discover the abstraction encoded assumptions that do not hold, and you rewrite it.

Delayed value. Time spent on portability is time not spent on the product.

the framework#

For each dependency, estimate:

1. Switching cost — engineer-weeks to move.
2. Probability you switch in the next three years.
3. Cost of the abstraction — engineer-weeks now, plus ongoing drag.

Then: if switching_cost × probability < abstraction_cost, do not abstract.

The numbers are rough. The exercise still clarifies, because it forces you to state the probability out loud, and stated probabilities are usually much lower than the implied ones people are acting on.

the categories, worked through#

Object storage. Switching cost: low. The S3 API is a de facto standard and every provider implements it. Probability: moderate — people do move for pricing.

Verdict: use the S3 API, do not abstract further. The API is already the abstraction.

Compute. Switching cost: moderate if containerized, high if you use provider-specific serverless. Probability: low.

Verdict: containerize — which is good practice anyway — and use whatever managed service you want. Do not build a compute abstraction layer.

Relational database. Switching cost: high. Probability: low.

Verdict: use the database's features. Teams that avoid stored procedures, database-specific types, and advanced indexing to stay portable are giving up real capability for an event that will not happen. Postgres-specific SQL is fine. You are not going to migrate to a different engine, and if you do, the SQL dialect will be the smallest part of the pain.

Managed queues, streams, and similar. Switching cost: moderate. The semantics differ enough between providers that a thin abstraction genuinely helps.

Verdict: a thin interface — publish, subscribe, ack — is worth it. Not a full abstraction; a boundary.

Authentication. Switching cost: very high — you have to migrate user credentials, sessions, and integrations. Probability: low, but the consequences of being stuck are severe.

Verdict: use standard protocols. OIDC and SAML are the abstraction. A provider that supports them is replaceable in principle; one with a proprietary SDK is not.

AI model providers. Switching cost: low if you kept the interface thin. Probability: high — the market is moving fast and the right choice changes quarterly.

Verdict: definitely abstract. This is the clearest case on the list. A thin interface plus an eval harness in your own repository makes model changes an afternoon. Teams that did this in 2024 have been switching providers casually ever since.

Observability. Switching cost: moderate to high — instrumentation is everywhere in your code. Probability: moderate, usually driven by cost.

Verdict: OpenTelemetry. The instrumentation is vendor-neutral, the collector handles routing, and swapping backends is a configuration change.

the pattern#

Abstract where the switching probability is high and the abstraction is cheap. AI providers, observability backends, object storage.

Do not abstract where switching is unlikely and the abstraction costs you real capability. Databases, compute platforms, managed services you chose for their specific features.

Use standard protocols wherever they exist. OIDC, S3, OpenTelemetry, SQL. A standard is an abstraction someone else maintains, and it is always cheaper than yours.

the thing that actually protects you#

Not an abstraction layer. Your data, in a format you can export, and a documented process for leaving.

Ask, before adopting anything: can I get all my data out, in a usable format, without their cooperation? If yes, you have real optionality regardless of how coupled your code is, because the expensive part of a migration is never the code — it is the data.

If the answer is no, that is a much bigger red flag than any API coupling, and it is the question almost nobody asks during procurement.

Size matters for real reasons: pull time on cold start, registry cost, deployment speed when you are scaling out under load, and attack surface — every package in the image is something that can have a CVE you have to answer for.

the six causes#

1. You shipped the build toolchain.

The compiler, the headers, the package manager cache, the source tree, the test fixtures. All of it needed to build, none of it needed to run.

Multi-stage builds fix this completely:

FROM golang:1.24 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /app ./cmd/server

FROM gcr.io/distroless/static-debian12
COPY --from=build /app /app
ENTRYPOINT ["/app"]

Final image: the binary, plus CA certificates and timezone data. Tens of megabytes.

2. You started from a full distribution image.

FROM ubuntu is roughly 80 MB before you install anything, and it includes a package manager, a shell, and a hundred utilities you will never invoke.

The ladder, from largest to smallest:

• Full distribution — 80 MB+
• -slim variants — 30–80 MB
• Alpine — 5–10 MB, with musl libc, which will occasionally surprise you
• Distroless — just the runtime, no shell, no package manager
• scratch — nothing at all, for static binaries

The Alpine caveat, since it bites people: musl's allocator and DNS resolver behave differently from glibc's. Python performance in particular can be significantly worse, and some binary wheels do not exist for musl. Test rather than assume.

3. Your layers are ordered wrong.

Each instruction creates a layer. A layer is invalidated when it or anything before it changes.

# bad — any source change reinstalls every dependency
COPY . .
RUN npm ci

# good — dependencies are cached until the lockfile changes
COPY package.json package-lock.json ./
RUN npm ci
COPY . .

This does not shrink the final image but it dramatically speeds up builds, which is usually what people actually care about.

4. You deleted things in a later layer.

RUN apt-get install -y build-essential   # layer 1: +400 MB
RUN apt-get remove -y build-essential    # layer 2: marks deleted, image unchanged

Layers are additive. Deleting a file in a later layer hides it and does not remove it. The bytes are still in the image and still transferred on pull.

Everything must happen in one RUN:

RUN apt-get update \
 && apt-get install -y --no-install-recommends build-essential \
 && make \
 && apt-get purge -y build-essential \
 && apt-get autoremove -y \
 && rm -rf /var/lib/apt/lists/*

Better: use a multi-stage build and do not install the toolchain in the final image at all.

5. You have no .dockerignore.

COPY . . copies .git, node_modules, build artifacts, test fixtures, and your local .env.

.git
node_modules
dist
*.log
.env*
**/__pycache__
coverage

The .git directory alone is frequently hundreds of megabytes on a mature repository, and it is in a lot of images.

6. Your dependencies are enormous.

Sometimes it is genuinely the dependencies — machine learning stacks with CUDA libraries are legitimately multiple gigabytes.

Check whether you need the GPU variant. torch with CUDA is roughly 2.5 GB; the CPU build is a fraction of that. If you are serving on CPU, you are shipping GPU libraries for nothing.

finding out where it went#

docker history --no-trunc <image>       # size per layer

Or use a layer inspection tool that shows you which files are in which layer and how much space is wasted. Ten minutes with one of those tells you exactly what to fix.

the security dimension#

Every package in the image is potential CVE surface, and your scanner will report all of them regardless of whether the code is reachable.

A distroless image has almost nothing to report, which means the reports you do get are signal rather than noise. That is worth more than the size reduction — a vulnerability report with three entries gets read; one with four hundred does not.

The trade-off: no shell means you cannot docker exec in to debug. Use ephemeral debug containers that attach to the running pod's namespaces instead, which is a better practice anyway because it means your production image is not a debugging toolkit.

the target#

• Compiled language, static binary: under 30 MB.
• Interpreted with dependencies: under 200 MB.
• Anything over a gigabyte without a machine learning stack: something is wrong and it is one of the six above.

Here is the pattern that works, and the specific things that go wrong.

the locks that kill you#

The failure mode is almost always a lock held longer than expected, blocking every query behind it, until connections exhaust and the application falls over.

In Postgres, these are safe (metadata-only, fast):

• ADD COLUMN with no default, or with a non-volatile default (since PG 11).
• DROP COLUMN (marks it dead, does not rewrite).
• ADD CONSTRAINT ... NOT VALID, then VALIDATE CONSTRAINT separately.
• CREATE INDEX CONCURRENTLY.
• Renaming things.

These rewrite the table and hold an exclusive lock for the duration:

• ALTER COLUMN TYPE (most of the time).
• ADD COLUMN with a volatile default.
• SET NOT NULL directly (use a CHECK constraint validated separately, then convert).

The one everyone hits: CREATE INDEX without CONCURRENTLY blocks writes for the entire build. On a large table that is minutes to hours. CONCURRENTLY takes longer and does not block, and it can fail — leaving an invalid index you must drop and retry.

The one nobody expects: even a fast ALTER TABLE must acquire an exclusive lock, and it will queue behind any long-running transaction. Then everything queues behind it. A migration that takes 5 ms can cause a 10-minute outage if a reporting query is holding a lock.

Set lock_timeout before every DDL statement:

SET lock_timeout = '3s';
ALTER TABLE orders ADD COLUMN region text;

If it cannot get the lock in three seconds, it fails and you retry, rather than queueing behind something and taking the site down. This one line prevents a large share of migration incidents.

expand and contract, always#

Never change a column in place. Four deploys:

1. Expand. Add the new column, nullable. Deploy. Old code ignores it.

2. Backfill and dual-write. Application writes both columns. Backfill existing rows in batches. Deploy.

-- in a loop, with a pause between batches
UPDATE orders SET region = derive_region(country)
WHERE region IS NULL AND id BETWEEN $1 AND $2;

Batch size in the thousands. Sleep between batches. Monitor replication lag and back off if it grows — a backfill that outruns replication is how you take down your read replicas.

3. Switch reads. Read from the new column. Deploy. Still fully reversible, the old column is intact and still being written.

4. Contract. Stop writing the old column, deploy, and drop it days later.

Four deploys instead of one. Every intermediate state works with both the old and new code, so any deploy can be rolled back independently.

the backfill rules#

Idempotent. It will be interrupted. It must be safe to re-run.

Resumable. Track progress. A backfill that starts over from the beginning after a failure will never finish on a large table.

Rate limited. Not "as fast as possible." A backfill competing with production traffic for I/O is a self-inflicted incident. Watch replication lag, watch p99 latency, and throttle.

Observable. Log progress. "Backfilled 4.2M of 180M rows, ETA 6h" lets someone make a decision. Silence does not.

Kill-switchable. You need to be able to stop it immediately without deploying. A flag it checks between batches.

testing it#

On a copy of production, at production size. A migration tested on a 10,000-row development database tells you the syntax is right and nothing about the runtime.

With concurrent load. Locks only matter under contention. Run the migration against a restored copy while replaying production traffic.

With the rollback. Actually run it. A rollback plan that has never been executed is a hypothesis.

the checklist#

Before any production migration on a large table:

• [ ] Tested on production-sized data with concurrent load
• [ ] lock_timeout set on every DDL statement
• [ ] Expand-contract, not in-place
• [ ] Backfill is batched, rate-limited, resumable, and killable
• [ ] Rollback tested
• [ ] Replication lag monitored with a threshold to pause at
• [ ] Someone is watching, with the kill switch, for the duration
• [ ] Not on a Friday

That last one is not superstition. It is that the people who understand the change should be available for the two days after it, and on Friday they are not.

It is also, for a majority of the teams running it, a substantial amount of complexity in exchange for benefits they do not receive, and saying so is still mildly heretical.

the problem it actually solves#

Kubernetes was built for: many services, many teams, heterogeneous workloads, on a fleet of machines, where you want bin-packing efficiency and declarative self-healing, and where the platform is operated by people whose job that is.

If you have all of those, it is the right answer and there is no close second.

what it costs#

A permanent learning tax. Pods, deployments, services, ingresses, configmaps, secrets, persistent volume claims, storage classes, service accounts, roles, network policies, resource quotas, and a YAML dialect for each. Every engineer who deploys anything must learn a meaningful fraction of it.

Operational surface. Control plane upgrades, node upgrades, CNI plugin, CSI driver, ingress controller, cert manager, metrics server, log shipper. Each is a component that can break and that must be upgraded on someone else's schedule.

Debugging distance. "Why is my service not reachable" has a dozen possible answers across five layers, and diagnosing it requires understanding all of them.

Cost, frequently. A managed control plane plus nodes sized for the platform's own overhead plus the observability stack it needs is often more than the equivalent capacity on simpler infrastructure.

Resume-driven adoption. This is real and worth naming. Kubernetes on your CV is worth money. That is a genuine incentive pointed away from the simplest solution that works.

the honest threshold#

You probably want Kubernetes if:

• More than roughly fifteen to twenty distinct services, deployed independently.
• More than a handful of teams that need to deploy without coordinating.
• You have someone whose job includes operating the platform, not as a side task.
• Genuinely heterogeneous workloads with different scaling characteristics.
• Multi-tenancy requirements with real isolation needs.

You probably do not if:

• Under ten services.
• One or two teams.
• Nobody owns the platform.
• Traffic is predictable.
• You are running one application with a database.

what to do below the threshold#

The options are better than they were, and all of them are boring:

A platform-as-a-service. Push code, it runs. This is the correct answer for a very large number of applications and the reason people avoid it is usually aesthetic.

Containers on a managed container service without the orchestrator — the various "run this container, scale it, load balance it" products every cloud offers. You get containers, autoscaling, and rolling deploys without the platform.

A couple of servers and a process manager. Systemd units, a reverse proxy with automatic certificates, and a deploy script. This runs an enormous amount of traffic, is trivially debuggable, and every engineer already understands it.

Docker Compose on one machine. For staging, for internal tools, for anything where a single host is enough. Unfashionable, works.

the migration-path argument#

"We will need Kubernetes eventually, so we should start now."

This is the most common justification and it is usually wrong, for two reasons.

First, the complexity cost is paid every day from now until then, and "eventually" frequently never arrives.

Second, containerizing your application is the actual hard part of a future migration, and you can do that without an orchestrator. A containerized application running under a simple process manager can move to Kubernetes later in a couple of weeks.

Build the container. Skip the platform until you have the problem it solves.

the position I will defend#

The teams I have seen most successfully run Kubernetes are large organizations with dedicated platform teams, where it is genuinely the right tool.

The teams I have seen most damaged by it are small ones where a single engineer set it up, that engineer left, and the remaining team is operating a system nobody understands and is afraid to touch.

That second failure is common, expensive, and entirely predictable from the staffing at adoption time. If nobody's job is going to be operating the platform, you should not have a platform.

What did not get settled is how to do it without producing something everyone hates. Here is what a decade of watching this has produced.

the state file is the whole problem#

Declarative infrastructure tools work by comparing three things: your configuration, a recorded state, and reality. Every hard problem comes from those three disagreeing.

State drift. Someone changed something by hand. Now state says one thing and reality says another, and the next apply will either revert their change or fail confusingly.

The fix is process, not tooling: nobody has write access to production infrastructure except the pipeline. Break-glass access exists, is audited, and triggers a drift check afterward. Without this, drift is continuous and IaC becomes theater.

State locking. Two applies at once corrupt state. Every backend supports locking. Verify yours is actually enabled — this is a default that people leave off and discover during an incident.

State as a blast radius. One giant state file means every apply touches everything and a corrupted state loses everything. Split by lifecycle and blast radius: networking, data stores, and applications should not share state.

Rule of thumb: if a plan takes more than a couple of minutes, your state is too big.

modules that became frameworks#

The pattern is universal. A team writes a module to wrap a resource with sensible defaults. It grows parameters. It grows conditionals. Three years later it has forty inputs, nested conditional logic, and nobody can tell what it creates without running a plan.

At that point the abstraction is costing more than the duplication it prevented.

The guidance that holds:

• Wrap only when there is real shared policy — tagging, naming, security baselines. Not to save typing.
• Modules should be shallow. A module that takes one resource and adds organizational defaults is good. A module that composes twelve resources with conditional branches is a framework, and frameworks need maintainers.
• Prefer duplication over the wrong abstraction, more so here than in application code, because infrastructure changes less and the cost of a bad abstraction is higher.
• No conditionals that change what resources exist. count = var.enabled ? 1 : 0 is fine once. Nested versions of it produce configurations nobody can reason about.

the practice that separates teams#

Plan output is reviewed as part of the pull request.

Not "run apply and see." The plan — the exact set of creates, changes, and destroys — posted to the pull request, read by a human, before merge.

Teams that do this catch the accidental destroy, the security group opened to the world, the database replacement that would have caused an outage. Teams that do not find out during apply.

This is a one-day CI setup and it is the single highest-value practice in the whole category.

The corollary: pay attention to every destroy in a plan. Most infrastructure incidents caused by IaC are a resource being replaced when the author expected it to be updated in place. Some attribute changes force replacement. The plan tells you. Nobody reads it.

the things to keep out#

Secrets. Never in the configuration, never in state. State files contain resource attributes in plaintext, including some you would not expect. Use a secret manager and reference it, and encrypt the state backend regardless.

Application deployment. Infrastructure tools are bad at deploying applications — they are declarative and convergent, and deployment is imperative and sequenced. Use them to provision the platform; use something else to deploy onto it.

Anything with a fast change cadence. If a value changes weekly, it should be configuration read at runtime, not infrastructure applied through a pipeline.

the fork question#

The OpenTofu fork means there is now a genuinely open-source implementation with a foundation behind it, alongside the commercial one. Both work. The configuration language is largely compatible.

The practical guidance: this is a lower-stakes decision than it feels. The lock-in is in your modules and your provider usage, not in the binary. Pick based on your licensing requirements and your appetite for the respective governance models, and know that migrating between them is a smaller project than it sounds.

the thing I would tell a team starting today#

Start smaller than you think. One state file per environment per major system. Plain resources before modules. Plan review in CI from day one. No manual changes, ever, enforced by permissions.

The teams that hate their infrastructure code all made the same mistake: they built an abstraction layer before they understood the domain, and then they were maintaining two things.

Several things changed and the answer is now more nuanced than the reflex suggests.

what changed in favor of self-hosting#

Machines got enormous. A single server you can rent for a few hundred dollars a month has more cores, more memory, and dramatically more I/O than a rack of hardware from 2012. A very large number of applications fit on one machine with room to spare.

The tooling got good. Configuration management, container runtimes, reverse proxies with automatic certificates, backup tooling. What required a team a decade ago requires a competent person and a weekend.

Cloud egress pricing did not fall. Compute prices came down. Bandwidth pricing at the major clouds is still a large multiple of what it costs, and for bandwidth-heavy applications it dominates the bill.

Managed service prices are high relative to the alternative. A managed database costs several times what the equivalent instance costs, for operational convenience that is real and is not always worth the multiple.

what changed against it#

Security expectations rose. Patching, hardening, monitoring, incident response. Self-hosting means you own all of it, and the threat environment is worse than it was.

Compliance frameworks assume cloud controls. SOC 2, ISO 27001, and their relatives are achievable self-hosted and the evidence collection is more work.

The talent assumption inverted. A decade ago every team had someone who knew Linux systems administration. Now a lot of teams do not, and hiring for it is harder than hiring for cloud skills.

the honest decision framework#

Self-host when:

• Your workload is steady rather than spiky. Cloud's core value proposition is elasticity, and you are paying for elasticity you do not use.
• Bandwidth is a large share of your bill.
• You have or can hire operational competence.
• Data locality or sovereignty is a requirement.
• You are at a scale where the cloud premium is a meaningful number — which starts lower than most people assume.

Use the cloud when:

• Traffic is spiky or unpredictable.
• You are early and optimizing for speed of iteration over unit economics.
• You need global presence and do not want to operate it.
• Your team's time is better spent on the product, which for most early-stage companies it is.
• Compliance requirements are easier to satisfy with a provider's attestations.

The hybrid that most people should consider: run the steady baseline on owned or rented hardware, burst to cloud for peaks, keep object storage and CDN with a provider. This captures most of the cost advantage without giving up elasticity where it matters.

the middle option nobody talks about#

Between "hyperscaler" and "rack in a colo" there is a large market of dedicated server providers and mid-size clouds: a real machine, in a real datacenter, with network and power handled, for a monthly fee.

You get root, predictable performance without noisy neighbors, and bandwidth allowances that are not priced as a profit center. You do not get managed databases, autoscaling, or a hundred adjacent services.

For a very large number of applications this is the correct answer and it is under-considered because the discourse is binary.

the operational minimum#

If you self-host, these are non-negotiable:

• Automated, tested restores. Not backups — restores. A backup you have never restored is a hypothesis. Test it quarterly, on a schedule, with a timer running.
• Unattended security updates, at least for the OS.
• Monitoring with alerting that reaches a human. Disk full is the most common self-hosted outage and it is entirely preventable.
• Infrastructure as code. The machine must be reproducible. If rebuilding it requires someone's memory, you have a single point of failure that is a person.
• A documented runbook for the failures you expect: disk, certificate expiration, service crash, host failure.

That is a weekend of setup and a few hours a month. If nobody on the team will own those hours, use the cloud — that is a legitimate reason and it is the actual deciding factor more often than cost is.

the thing that changed my mind#

I used to treat "we run our own servers" as a red flag. I now treat "we are on the cloud and have never modeled the alternative" as an equal one.

Both are defaults applied without analysis. The analysis takes an afternoon and the answer is frequently not what the reflex says.

They also multiply your state space by two for every flag you add, and almost nobody accounts for that.

the arithmetic#

Twenty boolean flags is 2^20 possible configurations — over a million. Your test suite exercises the default configuration. Your staging environment exercises one other. Production is running dozens simultaneously, because different flags are on for different user segments.

Which means: most of the configurations your users are running have never been executed anywhere before.

Most of the time this is fine, because most flags are independent. The failures come from the ones that are not, and you find out about those from a support ticket.

the failure modes#

Interaction bugs. Flag A changes the data format. Flag B reads that data. Both work alone. Together, one of them is reading a shape it does not expect. This is the classic and it is very hard to catch because neither change is wrong.

Flag debt. A flag that has been at 100% for a year is still in the code, with its dead branch, and nobody remembers whether it can be removed. The dead branch does not compile-error and does not test-fail; it just sits there being wrong.

The stale branch. Once a flag is at 100%, the disabled path stops being exercised. Six months later someone flips it as a rollback and discovers the old path no longer works with the current schema. Your kill switch is broken and you find out during an incident.

Configuration drift. Flags set differently in staging and production means staging tests a configuration nobody runs.

the rules that keep this manageable#

Every flag has an owner and an expiry date at creation. Not a suggestion — a required field. A flag that has passed its expiry shows up in a report and somebody has to either extend it with a reason or delete it.

Flags are deleted, not left at 100%. The cleanup is part of the work, not a follow-up ticket. A flag rollout is not done when it reaches 100%; it is done when the flag and the dead branch are gone.

Flags never nest. If flag A only means something when flag B is on, you have created a configuration nobody can reason about. Combine them into one flag with three states, or restructure.

Kill switches are a separate category with separate rules. They live forever, they are documented, and they are tested on a schedule — a quarterly exercise where you flip each one in staging and verify it works. An untested kill switch is not a kill switch.

Test both branches. Your test suite should run the critical path with each significant flag both on and off. Not the full combinatorial space — that is impossible — but each flag independently against the default configuration. That catches the "old path rotted" failure, which is the expensive one.

the taxonomy that helps#

Four kinds of flag, with different lifecycles:

The first two must expire. The last two must be documented and tested. Conflating them is how you get a thousand flags and no idea which matter.

the observability part#

Every event you log should carry the flag configuration that produced it.

{ "event": "checkout", "status": 500, "flags": ["new_pricing", "fast_path"] }

Without this, a bug that only affects one flag combination is undiagnosable — you see errors, you cannot correlate them to anything, and you spend a day guessing. With it, the correlation is a single query.

This is a fifteen-minute change and it is the single highest-value thing you can do if you use flags at all.

the counterargument#

Some teams respond to all of this by using fewer flags and doing more trunk-based deployment with fast rollback.

That is a legitimate position and it works when your rollback is genuinely fast and your changes are genuinely reversible. Flags are a tool for when they are not — when a change is expensive to revert, when you need per-segment control, or when release timing must be decoupled from deployment for business reasons.

Use them for those. Do not use them because they feel safer, because a flag you do not test is not safety, it is the appearance of it.

silicon#

Amazon continues pushing Trainium and Inferentia as the alternative to buying Nvidia. The pitch is price-performance for customers who can tolerate a different software stack.

The honest state of it: the hardware is competitive on paper, the software ecosystem is meaningfully behind CUDA, and the gap is closing slowly. If your workload runs through PyTorch with standard operations, the port is manageable. If you have custom kernels, it is a project.

The strategic point is the same for every hyperscaler: reducing dependency on a single supplier with enormous pricing power. Whether the customer benefit materializes depends entirely on whether the savings get passed through.

the agent announcements#

Every cloud vendor is now shipping agent infrastructure: runtimes, memory services, gateways for tool access, identity for agents, observability for agent traces.

This category is real. Running agents in production has genuine infrastructure requirements that are different from running services:

• Long-lived sessions with state that outlives a request.
• Non-deterministic execution paths that make traditional tracing awkward.
• Cost per invocation that varies by orders of magnitude.
• Identity and permission scoping for a thing acting on a user's behalf.
• Human approval gates in the middle of automated flows.

Those are real problems and the tooling is early everywhere. Evaluate on whether it solves a problem you actually have, not on whether the demo was good.

the pattern I would push back on#

Every vendor's agent framework wants to be the place your orchestration lives. That is a lock-in position, and orchestration is the layer most likely to be absorbed by the models themselves — it has been happening steadily for two years.

Keep your orchestration portable. Use the managed pieces for the genuinely hard infrastructure — identity, secure tool access, session storage — and keep the logic in your own code.

the underrated announcements#

The ones nobody writes about and everybody uses:

• Incremental improvements to S3 consistency and performance.
• Networking latency reductions.
• Cost management tooling that is slightly less bad.
• Database engine version updates.

These are worth more to most organizations than any AI announcement, and they get four slides between two hours of agent demos.

the meta-observation#

Cloud conferences have gotten less interesting, and that is good. It means the platform is mature. The exciting years of a platform are the years when fundamental things are missing.

The interesting question for AWS is not what they announced. It is whether the operational excellence that justified the premium is still there after a year that included a major regional outage. That is an execution question and it does not get answered at a conference.

what to actually do with this#

Skip the keynote. Read the "what's new" feed filtered to the services you actually use. Look for the deprecations, which are the announcements that will cost you time and which are never on stage.

And check your bill. The single highest-value hour available to most engineering organizations is someone competent looking at the AWS bill line by line, and almost nobody does it.

AWS's public summary attributes the trigger to a latent race condition in the automation that manages DynamoDB's DNS records, which resulted in an empty record set for a regional endpoint and no automatic recovery path.

the cascade#

The failure sequence is the interesting part.

DynamoDB's endpoint became unresolvable. That alone would be bad. What made it a regional event is that an enormous number of AWS's own services use DynamoDB internally. The EC2 instance launch path, IAM's control plane, Lambda's invocation machinery, and dozens of others depend on it.

So the failure propagated: DynamoDB down means new EC2 instances cannot launch, which means autoscaling cannot replace failing capacity, which means load shedding, which means more failures. Network Load Balancer health checks destabilized. The recovery itself was slowed by the backlog of queued work that had accumulated.

This is textbook metastable failure: a system that is stable under normal load and stable under no load, but which, once pushed past a threshold, sustains its own failure through retry amplification and queue buildup even after the original trigger is fixed.

why us-east-1#

It is the oldest region, the largest, and the default in approximately every tutorial ever written. Several global AWS control planes are homed there — IAM, CloudFront configuration, Route 53's control plane, and others. That means a us-east-1 event has global blast radius even for customers with no resources in the region.

That architecture is a historical artifact. It is also extremely hard to change now, which is a lesson about early decisions in systems that grow.

the honest customer takeaway#

The reflexive response is "multi-region." Before you spend a year on that, do the arithmetic.

Multi-region active-active is genuinely hard. Data consistency across regions, failover testing that actually works, doubled infrastructure cost, and a substantially more complex system that fails in new ways. Many organizations that attempt it end up with a system that is less reliable overall because the complexity introduces more failure modes than the regional risk it removes.

The dependency you cannot escape. If your multi-region architecture depends on a global control plane that lives in us-east-1, you did not achieve independence. Check this specifically. A lot of people discovered it yesterday.

What is actually worth doing, in order:

1. Know your dependencies. Most teams cannot enumerate what their service requires to start. Write it down. The exercise is revealing.
2. Static stability. Design so existing capacity keeps serving when the control plane is unavailable. If your service needs to call an API to keep running, it will stop when that API stops. Cache aggressively, fail open where safe, and do not require a control plane call on the request path.
3. Graceful degradation. Decide in advance which features can be turned off. A checkout that works without recommendations is much better than a site that is down.
4. Exponential backoff with jitter, and circuit breakers. Retry storms are what turns an incident into an outage. This is the single highest-leverage code change available.
5. Multi-region for the tier that genuinely warrants it. Which is usually not everything.

the industry-level observation#

A meaningful fraction of the world's software depends on a small number of regions operated by a small number of companies. That concentration produces excellent reliability most of the time and correlated failure occasionally.

There is no individual fix. Every company independently choosing the most reliable provider produces exactly this concentration. It is a collective action problem, and the only actors who can address it are regulators thinking about systemic risk, who are — belatedly — starting to.

Deployment is a solved problem that organizations keep re-opening because the solution is boring and boring does not get anyone promoted.

the properties that matter#

Strip away the tooling debates and a good deploy has four properties.

It is the same every time. Not "mostly the same, except for the database migration, and except on Fridays, and except when Sarah does it." The same. If there is a manual step, it is in the pipeline as an approval gate, not in someone's head.

It is reversible in under a minute. This is the one people get wrong. They build elaborate progressive rollouts and then discover that rolling back requires reverting a migration that dropped a column. Reversibility is a design constraint on your changes, not a feature of your deploy tool.

Failure is detected without a human looking. If the way you find out about a bad deploy is a customer email, you do not have a deployment process, you have a deployment ritual.

It happens often enough to be unremarkable. The failure rate of a deploy scales superlinearly with the amount of change in it. Ten small deploys are dramatically safer than one deploy containing ten changes, and this is the single most robust finding in the entire delivery-metrics literature.

the expand-contract discipline#

Most rollback pain is schema pain. The fix is a discipline, not a tool:

1. Expand. Add the new column, nullable. Deploy. The old code ignores it.
2. Migrate. Backfill. Dual-write from application code. Deploy. Both shapes work.
3. Switch. Read from the new column. Deploy. Still reversible — the old column is intact.
4. Contract. Drop the old column, days or weeks later, once you are certain.

Four deploys instead of one. Every intermediate state is reversible. It feels slow and it is the reason you do not have a 3 a.m. incident where the rollback made things worse.

flags are not free#

Feature flags decouple deploy from release and that is genuinely valuable. They also multiply your state space: n flags means 2^n possible configurations, and you are testing approximately one of them.

Rules that keep this manageable:

• Every flag gets an owner and an expiry date at creation.
• A flag that has been at 100% for a month is deleted, not left "just in case."
• Flags never nest. If flag A only matters when flag B is on, you have made something unreasonable.
• Kill-switch flags are a separate category with separate rules and they can live forever.

The failure mode is flag debt: a codebase with two hundred flags where nobody knows which combinations are actually exercised in production. That is worse than no flags, because it looks like safety.

the actually unfashionable opinion#

Most teams do not need Kubernetes to deploy well, and many teams running Kubernetes deploy worse than they did before, because the platform's complexity consumed the attention that used to go into the process.

If your deploys are fast, boring, and reversible, your infrastructure is correct regardless of what it is. If they are not, no amount of platform will fix it, because the problem is that nobody has treated the deploy as a product with a user.

The user is your team at 2 a.m. Design for them.

Call it T. If T is two minutes, your team behaves one way. If T is thirty minutes, it behaves completely differently, and the difference is not "things take twenty-eight minutes longer."

what actually happens as T grows#

Below ~2 minutes, you wait. You keep the change in your head, you see the result, you fix it. The loop is tight enough that debugging is interactive.

Between 2 and 10 minutes, you context switch. You go read something else. When the result comes back you have to page the change back in. The reload cost is real and it is roughly proportional to how complex the change was — which means it is worst exactly when you can least afford it.

Past 10 minutes, behavior changes qualitatively. People start batching. They stop pushing small commits because the feedback is too expensive per commit, so they push bigger ones, which are harder to review and more likely to fail, which makes each failure more expensive to diagnose. It is a doom loop and it is entirely emergent from one number.

Past 30 minutes, people route around CI. They test locally in ways that diverge from CI, they merge on green-ish, they add [skip ci], and eventually somebody proposes a nightly build as the "real" signal, at which point you have given up.

the diagnostic#

Before optimizing, measure the right thing. Not average pipeline duration — p90 time-to-signal for the check that actually blocks merge. Most teams measure the wrong thing here and optimize a stage nobody was waiting on.

Then break it down:

queue wait      ← runners are saturated or your concurrency limit is wrong
checkout        ← you're doing a full clone of a repo with 200k commits
dependency install ← almost always the biggest fixable chunk
build           ← is anything cached? really?
test            ← is it parallel? is it parallel *well*?

the fixes, roughly in order of return#

1. Cache dependencies properly. Not "we have a cache step." Verify the hit rate. A cache that misses on every branch because the key includes the branch name is worse than no cache, because it also uploads.
2. Shallow clone. fetch-depth: 1 unless you genuinely need history. If you need history for one job, do it in one job.
3. Split the blocking check from the exhaustive check. Lint, typecheck, and unit tests block merge. Integration, e2e, and the full matrix run after, and page someone if they fail. Not everything needs to gate.
4. Parallelize by timing, not by file count. Splitting tests evenly by count gives you one shard that takes four times as long as the others. Split by historical duration.
5. Kill the flaky tests. A test that fails 2% of the time in a suite of 50 parallel jobs means your pipeline fails constantly. Quarantine them the day they are identified. A quarantined test is a bug ticket; a flaky test in the blocking path is a tax on everyone forever.
6. Bigger runners. This is the least intellectually satisfying fix and often the best one. Engineer time costs more than compute. Do the arithmetic before you spend a sprint on a clever solution.

the cultural part#

Someone has to own T. Not "the platform team should look at CI sometime." A named person, a dashboard, and a number that goes in the same review as uptime.

Because the alternative is that it degrades a little every sprint — one more test, one more dependency, one more required check — and nobody notices until it is thirty minutes and everybody has quietly stopped trusting it.