Infrastructure as code, ten years of lessons

Infrastructure as code won the argument. Nobody clicks through a console to provision production anymore, or admits to it.

What did not get settled is how to do it without producing something everyone hates. Here is what a decade of watching this has produced.

the state file is the whole problem#

Declarative infrastructure tools work by comparing three things: your configuration, a recorded state, and reality. Every hard problem comes from those three disagreeing.

State drift. Someone changed something by hand. Now state says one thing and reality says another, and the next apply will either revert their change or fail confusingly.

The fix is process, not tooling: nobody has write access to production infrastructure except the pipeline. Break-glass access exists, is audited, and triggers a drift check afterward. Without this, drift is continuous and IaC becomes theater.

State locking. Two applies at once corrupt state. Every backend supports locking. Verify yours is actually enabled — this is a default that people leave off and discover during an incident.

State as a blast radius. One giant state file means every apply touches everything and a corrupted state loses everything. Split by lifecycle and blast radius: networking, data stores, and applications should not share state.

Rule of thumb: if a plan takes more than a couple of minutes, your state is too big.

modules that became frameworks#

The pattern is universal. A team writes a module to wrap a resource with sensible defaults. It grows parameters. It grows conditionals. Three years later it has forty inputs, nested conditional logic, and nobody can tell what it creates without running a plan.

At that point the abstraction is costing more than the duplication it prevented.

The guidance that holds:

• Wrap only when there is real shared policy — tagging, naming, security baselines. Not to save typing.
• Modules should be shallow. A module that takes one resource and adds organizational defaults is good. A module that composes twelve resources with conditional branches is a framework, and frameworks need maintainers.
• Prefer duplication over the wrong abstraction, more so here than in application code, because infrastructure changes less and the cost of a bad abstraction is higher.
• No conditionals that change what resources exist. count = var.enabled ? 1 : 0 is fine once. Nested versions of it produce configurations nobody can reason about.

the practice that separates teams#

Plan output is reviewed as part of the pull request.

Not "run apply and see." The plan — the exact set of creates, changes, and destroys — posted to the pull request, read by a human, before merge.

Teams that do this catch the accidental destroy, the security group opened to the world, the database replacement that would have caused an outage. Teams that do not find out during apply.

This is a one-day CI setup and it is the single highest-value practice in the whole category.

The corollary: pay attention to every destroy in a plan. Most infrastructure incidents caused by IaC are a resource being replaced when the author expected it to be updated in place. Some attribute changes force replacement. The plan tells you. Nobody reads it.

the things to keep out#

Secrets. Never in the configuration, never in state. State files contain resource attributes in plaintext, including some you would not expect. Use a secret manager and reference it, and encrypt the state backend regardless.

Application deployment. Infrastructure tools are bad at deploying applications — they are declarative and convergent, and deployment is imperative and sequenced. Use them to provision the platform; use something else to deploy onto it.

Anything with a fast change cadence. If a value changes weekly, it should be configuration read at runtime, not infrastructure applied through a pipeline.

the fork question#

The OpenTofu fork means there is now a genuinely open-source implementation with a foundation behind it, alongside the commercial one. Both work. The configuration language is largely compatible.

The practical guidance: this is a lower-stakes decision than it feels. The lock-in is in your modules and your provider usage, not in the binary. Pick based on your licensing requirements and your appetite for the respective governance models, and know that migrating between them is a smaller project than it sounds.

the thing I would tell a team starting today#

Start smaller than you think. One state file per environment per major system. Plain resources before modules. Plan review in CI from day one. No manual changes, ever, enforced by permissions.

The teams that hate their infrastructure code all made the same mistake: they built an abstraction layer before they understood the domain, and then they were maintaining two things.