Terraform — Production Errors, Mistakes and Recovery

The Terraform errors you meet in production (locks, drift, unknown values, cycles, forced replacement, providers, state and backends) and the mistakes behind most outages, with documented recovery steps and guardrails.

Overview

Terraform rarely fails because Terraform is broken. It fails because the state no longer matches reality (a killed run left a lock, someone clicked in the console, a state file was lost), because a plan was computed against the wrong inputs (wrong workspace, wrong account, a stale plan file), or because a reviewer did not read the one line that said must be replaced. The tool does exactly what the plan says, so the plan is where incidents are prevented.

Two published incidents show the pattern. On July 7, 2023, GitLab.com was down for web and API traffic from 16:25 to 18:42 UTC after an out-of-sync Terraform plan, prepared three weeks earlier, was executed against production (GitLab incident review). In March 2026, Alexey Grigorev described how, after moving to a new computer without his Terraform state, an AI coding agent ran terraform destroy and removed the DataTalks.Club course platform's production infrastructure, including the RDS database and its automated snapshots; AWS restored a snapshot exactly 24 hours after the database was deleted (his write-up).

Symptom to first command

Error acquiring the state lock — Read the Lock Info block, then match Who (user@host) to a CI runner or a person — State locks
Plan wants to change things nobody changed — terraform plan -refresh-only — Drift and manual changes
Invalid count argument or Invalid for_each argument — Find the resource attribute the expression depends on — Plan-time errors
Error: Cycle — terraform graph -type=plan -draw-cycles — Plan-time errors
A database or cluster shows must be replaced — Find the attribute marked # forces replacement — Unexpected replacement
Failed to install provider, checksum or constraint errors — terraform init -upgrade only after reading the constraint — Providers and lock file
No valid credential sources found — Check which identity and account the provider resolved — Providers and lock file
Backend initialization required — terraform init -migrate-state or -reconfigure, chosen on purpose — State and backends
Resources gone after an apply nobody meant to run — Lock the workspace or stop the pipeline — Mistakes

What you will be able to do

Recover a stuck state lock safely and explain why force-unlock is dangerous.
Tell drift from a configuration change and reconcile it with a refresh-only plan.
Fix unknown-value errors in count and for_each, and break dependency cycles.
Read a plan for must be replaced and choose between create_before_destroy, ignore_changes, moved and removed.
Diagnose provider, lock file, credential and rate-limit failures.
Restore state from S3 versioning, GCS object versioning or HCP Terraform state versions.
Run the first hour after an accidental destroy, and name the guardrails that stop it happening again.

Key concepts

State: The record that binds addresses in configuration to real objects. Terraform plans by comparing configuration, state and what the provider reports.
State lock: A backend lock taken for every operation that could write state, so two runs cannot write at once. Released at the end of the run, or manually with force-unlock and the lock ID.
Drift: Real infrastructure that no longer matches state or configuration, usually from console or other-tool changes. Reviewed with terraform plan -refresh-only.
Forces replacement: A change to an argument that cannot be updated in place, so the plan destroys and recreates the object (destroy first, unless create_before_destroy is set).
moved block: Terraform 1.1+ configuration that renames a resource address in state during plan, so a refactor does not destroy and recreate the object.
removed block: Terraform 1.7+ configuration that removes a resource from state; with lifecycle destroy = false the real object is kept.
import block: Terraform 1.5+ configuration that brings an existing object under management through a reviewed plan; -generate-config-out can write the resource block.
Dependency lock file: .terraform.lock.hcl records provider versions and checksums (providers only, not modules) and belongs in version control.
Saved plan: A plan written with -out and applied as a file without a prompt. Terraform refuses it as stale if state changed after it was created.
Policy as code: Sentinel or OPA policy sets in HCP Terraform that check a plan after it is created and before apply, with advisory or mandatory enforcement levels.

When to use it

A Terraform run in CI or on a laptop failed with a lock, provider, credential, count, for_each or cycle error and you need the safe fix.
A plan shows changes nobody made in code, or shows must be replaced on a stateful resource.
Someone deleted a resource block, ran destroy, applied in the wrong workspace or account, or edited state, and you need to recover.
You are designing a Terraform pipeline for production and need approvals, policies and state protection.
You are preparing for a DevOps, SRE or platform interview with infrastructure-as-code incident questions.

Interview questions

What happens if you delete a resource block from a .tf file without running terraform destroy on it first?
Someone ran terraform apply against the production workspace by mistake and it destroyed resources. What do you do in the first hour, and how do you prevent it from happening again?
Terraform says the state is locked and the CI run that held the lock was killed. How do you safely recover?
A routine change shows that a production database must be replaced. What do you do?
terraform plan fails with Invalid for_each argument because keys cannot be determined until apply. How do you fix it properly?
Someone opened a security group rule in the console during an incident. How do you reconcile it with Terraform?
How do you rename resources or move them into a module without destroying anything?
CI fails after a teammate upgraded a provider on a Mac, with a checksum error. What happened and how do you fix it?
How would you design a production Terraform pipeline so a stale or unreviewed plan can never be applied?
Your Terraform state contains database passwords. What is the risk and what do you change?
How do you move an S3 backend from DynamoDB locking to S3 native locking, or to a new bucket, without losing state?