Immutable infrastructure, and the replacement nobody planned for

TL;DR: Replace rather than modify, and the environment becomes reproducible, drift-resistant and rollback-friendly. The trap is that the provider decides which attribute changes force replacement, not you, so a one-line diff can destroy a database. Read the plan, not the diff.

The symptom this explains

An engineer opens a pull request that changes one line: a version string in a launch template, or a parameter on a database. Two reviewers approve it, because a one-line diff is a one-line diff. The apply runs, and a live database is destroyed and recreated empty.

Nobody was careless. The diff really was one line, and everything about what that line would do was stated somewhere neither reviewer looked, which is the plan rather than the code. The rest of this page is about why that gap exists and which changes fall into it.

Why replacement is the better default

A server patched in place for three years is a unique artifact whose state is the sum of every command anyone has ever run on it, which means nobody can rebuild it and a problem on that host may not reproduce anywhere else. That is the snowflake. The entire reason infrastructure as code exists is to stop producing them.

Replacement limits accumulated host mutations, but reproducibility still needs pinned inputs and a repeatable build. Drift can occur before replacement, so detection remains useful. Promoting the tested image by digest preserves artifact identity; restoring an old image is a rollback only while configuration and durable state remain compatible.

The part that surprises people

The provider decides what forces replacement, not you and not your intent. In Terraform, changing an EC2 instance’s AMI or a subnet’s CIDR can require replacement. Database changes need closer inspection: RDS supports renaming a DB instance identifier in place, with endpoint and availability implications, while other attributes may require replacement. Check the plan produced by the pinned provider version. AWS documents RDS renaming; a name change is not proof of a destroy-and-create operation. The configuration diff can be one line. The plan says must be replaced, in a section people skim.

This is why the review object is the plan and not the code. A generated or hand-written one-line change reads as trivial and states its full intent only in the plan output. A gate that requires explicit acknowledgement for any plan containing a destroy is a useful control in an IaC pipeline, with implementation and review costs.

The lifecycle levers

Three arguments change the behaviour, and each is a decision rather than a workaround:

• create_before_destroy builds the replacement before removing the original, which is how you replace something serving traffic. It requires that both can exist at once, so a resource with a globally unique name needs that handled.
• prevent_destroy rejects a destroy or replacement while the lifecycle rule remains in the resource configuration. Removing the entire resource block also removes that protection; use provider-side deletion protection and reviewed plans for critical data. Terraform’s lifecycle reference explicitly documents this limit. Correct for a production database, and it will one day block a legitimate change, at which point somebody removes it in a hurry. Use it where an accidental destroy is unrecoverable.
• ignore_changes stops Terraform reconciling a specific attribute, which is how you coexist with another system that legitimately owns a field. It is also how drift becomes invisible, so each use needs a comment saying who owns that attribute instead.

Where immutability stops being free

Stateful resources are the boundary. Replacing a stateless instance is routine, whereas replacing a database means moving the data, and there is no lifecycle argument that makes that free or fast. So the honest architecture separates the two layers and treats them differently: immutable compute sitting over durable state, with the state layer changed through migrations.

It is worth pricing that, because "not free or fast" is the kind of phrase a plan review skates over. Take a 500 GB database that has to be replaced with create_before_destroy. The data has to be copied to the new instance before the old one goes away, and at a sustained 100 MB/s that copy is 500,000 / 100 = 5,000 seconds, or about 1.4 hours, during which both instances exist and both are billed. Then there is a cutover, which needs writes stopped or reconciled, because anything written to the old instance after the copy started is not on the new one. So the change costs roughly an hour and a half of doubled cost, a write freeze or a replication catch-up, and a connection-string change that every client has to pick up.

That figure assumes a copy rate you should measure rather than accept, and it deliberately ignores index rebuilds and verification, which usually dominate. The reason to do the arithmetic at all is that the answer decides the shape of the change: an hour and a half is a maintenance window, and if the same calculation on your data comes out at fourteen hours it is a migration project instead, and finding that out during the apply is the worst possible time.

Long-lived caches pull the same way. So do licence-bound hosts, and anything carrying an identity another system depends on, a fixed IP or a specific hostname that somebody wrote into a firewall rule two years ago. Naming those exceptions out loud, instead of pretending the whole estate is cattle, is what turns this from a slogan into something a team can actually apply.

Read the plan for the words that name the risk

Read every plan for the words must be replaced. Treat their presence as a different class of change entirely, one that earns a second approver, a maintenance window, or a create-before-destroy path worked out before anyone runs apply. The plan can expose a destructive operation before it happens; review it together with dependencies, data recovery and the expected service interruption.

Self-check

A pull request changes one line: the engine_version on an RDS instance, from one minor version to the next. The plan output contains a section reading must be replaced, and the instance holds 800 GB of production data. Say what you do before this is approved, what create_before_destroy would and would not solve here, and whether prevent_destroy on that resource would have caught it. Then answer a second question: does replacing a VM from a pinned image guarantee a reproducible environment? Work through both before reading on.

The first thing to do is not a decision but a check, because the plan tells you that it will replace and not why. Find the attribute forcing it, since a minor engine version upgrade is normally an in-place operation and a plan proposing replacement usually means something else in the diff, or a provider version change, is the real cause. If the forcing attribute turns out to be incidental, the answer is to fix the configuration rather than to plan a replacement at all. That is the most common good outcome here and it is only available to someone who read the plan.

If replacement really is required, create_before_destroy changes the failure mode rather than the cost. It gets you a working instance before the old one is removed, which matters, and it does not move the data for you: 800 GB at a measured 100 MB/s is 800,000 / 100 = 8,000 seconds, about 2.2 hours of copying with both instances running and billed, plus a cutover during which writes have to be stopped or reconciled. It also needs both resources to coexist, so any globally unique identifier has to be handled explicitly.

prevent_destroy would have caught this one, and it is worth being exact about what that means. It fails the plan while the lifecycle rule is present in the resource configuration, so the apply stops rather than the database being lost. It does not protect against removal of the whole resource block, which takes the rule away with it, which is why provider-side deletion protection belongs underneath it as a second layer that your configuration cannot revoke.

On the second question: no, and for a reason that is easy to state and easy to forget. A pinned image fixes what is baked into the image. It does not fix the configuration injected at boot, the packages a startup script installs from an upstream that has moved, the dependencies the instance resolves when it starts, or whether the durable state it attaches to is compatible with the new version. Pinning by digest rather than by tag closes the first gap only. Reproducibility is a property of the whole set of inputs, and replacement is the mechanism that makes controlling them possible rather than the thing that controls them.

Next: state and drift explains why the plan knows things the code does not, and the drift detection question puts plan review on a schedule so a surprise like this is found on a Tuesday.