The first drift alert lands in a channel someone actually reads. The second arrives next morning saying almost the same thing: an autoscaling group’s desired count moved, a tag appeared that nobody wrote, a load balancer picked up an attribute the provider set on its own. By week three the channel is muted. By month two a security group opens SSH to the world, and that message scrolls past in the same wall of text as the noise.<\/p>\n\n\n\n
That is the failure worth writing about. Drift itself is normal. Somebody fixes production at 2am, a managed service rotates a value, a policy engine stamps a tag. The part that breaks is the alert, and it breaks quietly, because a muted channel looks identical to a quiet one.<\/p>\n\n\n\n
Switching on Terraform drift detection takes one flag. Building Terraform drift detection that engineers still read six months later means deciding what not<\/em> to send: which plan mode you run, what you classify as noise, how you route by blast radius, and whether the message names who made the change. A detector firing daily with the same content is worse than none, because no detector leaves you knowingly blind while a muted one leaves you blind with a green dashboard. Every section below asks the same question differently: does this message change what a human does in the next ten minutes?<\/p>\n\n\n\n This is the biggest source of false drift, and it catches almost everyone building their first check. A normal Refresh-only mode narrows the question. HashiCorp documents it as a plan whose goal is only to update state and root module outputs to match changes made to remote objects outside of Terraform. That is the actual definition of drift.<\/p>\n\n\n\n Two flags there matter more than they look. The trade-off, stated plainly: refresh-only will not tell you that merged code has not shipped, and it will not find resources created outside Terraform, because it can only refresh what is already in state. I run both modes and route them separately. Refresh-only feeds the alert. Normal plan feeds a weekly digest tied to the repository, where unapplied changes are a delivery problem rather than a security one.<\/p>\n\n\n\n Most recurring drift falls into a few families. Sorting yours takes an hour and decides most of your routing rules.<\/p>\n\n\n\n A JSON policy document comes back with keys reordered. An identifier returns in different case. An empty block reads back as null instead of an empty map. Terraform sees a difference because it compares a stored representation against an API response, and that round trip is not always lossless. Fix these in the configuration where you can, because a phantom diff that follows you into every plan also hides real diffs on the same resource.<\/p>\n\n\n\n The classic is autoscaling capacity. Your config says two instances, a scaling policy moved it to five during a spike, and the next plan proposes taking it back to two. Applying that kills three instances mid-spike. The drift is real; reverting it is the wrong move.<\/p>\n\n\n\n Understand what you bought. Organization tag policies, cost allocation tooling, backup schedulers and compliance agents all add tags. The AWS provider exposes both the tags you set and the effective set including defaults, and ignoring the effective set silences all tag drift at once. Convenient, and slightly dangerous if you use tags for access control, because tag-based policy conditions stop being watched. The same logic covers values the platform rotates or defaults on its own, like a managed secret version: they drift on a schedule that has nothing to do with your team, and belong in a digest.<\/p>\n\n\n\n The goal here is not a clean plan. A clean plan is a side effect. The goal is a decision per family: see it immediately, see it weekly, or never see it again.<\/p>\n\n\n\n “Drift detected: 12 resources changed” makes a human open a terminal before they know whether to care, and that friction is where attention goes to die. Split the output by what each resource can do if it is wrong:<\/p>\n\n\n\n To split them you need structure, not text. Terraform’s machine-readable output emits a message for each resource that changed outside Terraform, carrying the resource address and the action it would take.<\/p>\n\n\n\n Check the exact field names inside that object against the machine-readable output reference for your Terraform version before building matching logic on them. Keep the severity mapping in the repository next to the code it describes. A resource type moving from digest to page should be a pull request, not something changed in a monitoring UI at midnight.<\/p>\n\n\n\n Terraform tells you what changed. It has no idea who changed it, and that gap is where the investigation time goes. Half the drift you find is a legitimate incident fix nobody wrote down. If the alert names the actor, the response is a two-word reply in the thread instead of twenty minutes of archaeology. On AWS the audit trail can be queried by resource:<\/p>\n\n\n\n CloudTrail’s console event history only reaches back a limited window, so a weekly check may be hunting an event that has aged out; a trail delivering to S3, or CloudTrail Lake, covers the gap. Azure exposes the equivalent through Activity Logs, Google Cloud through Cloud Audit Logs. Wiring the lookup into the alert body does more for adoption than any amount of formatting.<\/p>\n\n\n\n There are three answers to a drift finding, and picking the wrong one can be worse than the drift.<\/p>\n\n\n\n The trap is automating option one. Auto-approving a remediation apply sounds like the natural endpoint of a drift pipeline, and it is fine right up until the drifted attribute is one that forces replacement. Then “restore the desired state” means destroying and recreating a live resource, unattended, at whatever hour the schedule runs. If you automate anyway, gate it hard: an allowlist of resource types, a check that the planned action is an in-place update rather than a replace, and a notification either way.<\/p>\n\n\n\n The nastiest failure mode is not a noisy check. It is a check reporting “no drift” because it never ran the plan successfully.<\/p>\n\n\n\n With Note that the failure branch notifies too. Exit code 1 deserves the same urgency as drift, because a broken detector and a clean estate produce identical silence.<\/p>\n\n\n\n Drift checks run unattended, so nobody notices when a role trust policy changes, an OIDC subject claim stops matching after a repository rename, or a long-lived key gets rotated out. Add a deadman check: if the job has not reported within its expected interval, that absence is the alert. Same reasoning as heartbeat monitoring for backups.<\/p>\n\n\n\n Bump a provider and a new attribute default can appear across every resource of a type. Overnight the check reports hundreds of changed resources and whoever is on call assumes the tool is broken, which it sort of is. Commit the dependency lock file, pin the provider version, run the drift job on the same Terraform version as your apply pipeline, and treat provider upgrades as a reviewed change rather than something a bot merges.<\/p>\n\n\n\n Refresh queries the API for every resource in state. Large workspaces get slow, then overlap their own schedule, then time out. The durable fix is splitting state along boundaries that already exist in your team structure. Terraform does offer targeting to narrow a plan, but it is documented as an exceptional-use option and prints a warning for good reason, so treat it as a stopgap rather than the design.<\/p>\n\n\n\n A scheduled job in GitHub Actions or GitLab CI, or a systemd timer on a small always-on box from a provider like Contabo or InterServer, covers a handful of workspaces perfectly well and costs almost nothing. I would not reach for a platform before that stops working. What eventually stops working is the surrounding machinery: credentials for a dozen accounts across providers, concurrency against real applies, and retaining history so you can see that the same resource has drifted repeatedly this quarter, which is a process problem no single alert reveals.<\/p>\n\n\n\n That is where HCP Terraform’s health assessments, or a platform like Spacelift, Scalr or env0, starts earning its fee, since drift detection becomes a workspace setting with history and notifications attached rather than code you maintain. Two honest caveats: HCP Terraform gates health assessments behind its paid editions and requires remote or agent execution, so it is not a drop-in for local-state workflows, and any platform will happily send you the same undifferentiated wall of drift the homegrown script did. Severity tiering is a decision about your infrastructure that no vendor can make for you. Buying the runner does not buy the judgement.<\/p>\n\n\n\n A normal plan compares configuration, state and reality, so it flags genuine drift and any committed change that has not been applied. Refresh-only compares state against reality only, which matches what most people mean by drift. Use refresh-only for the alert and normal plan for a delivery-focused report.<\/p>\n\n\n\n No. Plan operations do not write state. Only applying a refresh-only plan updates the state file to match the drifted remote objects, and even then it makes no changes to infrastructure. That is why a detection job can safely skip the state lock.<\/p>\n\n\n\n Tie the interval to how long you would tolerate not knowing. Daily for production and anything holding customer data, weekly for lower environments, less often for slow-moving workspaces where refresh is expensive. Frequency matters less than whether the message gets read.<\/p>\n\n\n\n Not with a plan. Terraform refreshes only what is already in state, so a resource it has never seen is invisible. You need a cloud-side inventory such as AWS Config, Azure Resource Graph or Cloud Asset Inventory, compared against your state, and import blocks to bring the results under management. Worth knowing that driftctl, the open-source tool most often recommended here, has moved into maintenance mode.<\/p>\n\n\n\n Yes, and that is the point of using it. It stops Terraform proposing updates for that attribute regardless of what changed it, so a deliberate suppression and a malicious change look identical afterwards. Scope it to specific attributes, never whole resources you still care about, and document the owner.<\/p>\n\n\n\n Almost always a provider or Terraform version change rather than something happening in your cloud account. A new provider release can add or reshape an attribute and every resource of that type reports a difference at once. Compare the versions used by the drift runner and the apply pipeline before investigating the infrastructure.<\/p>\n\n\n\n Terraform drift detection is not a monitoring problem, it is an attention problem. The flag that finds drift took an afternoon. Everything determining whether anyone still reads the output next quarter is judgement: refresh-only rather than normal plan, noise families sorted and suppressed deliberately, severity by blast radius rather than resource count, an actor name in the message, and an alert when the detector itself goes dark.<\/p>\n\n\n\n Judge it by one test: when a message lands in that channel, does anyone open it? If the honest answer is no, your detection is working and your alert is not.<\/p>\n\n\n\n This is common work once a Terraform estate grows past one team. Things I can help with:<\/p>\n\n\n\n If you want a second opinion, send me a week of your drift output or the workflow file that produces it, and I will tell you what I would cut first.<\/p>\n\n\n\nPick the right plan mode or you will alert on your own backlog<\/h2>\n\n\n\n
terraform plan<\/code> compares three things: your configuration, your state, and the real infrastructure. It returns a non-zero exit code when any of them disagree, so it fires for real drift and just as loudly for a pull request that merged this morning and has not been applied. Your drift channel becomes a notification feed for your own deployment backlog, and people learn that most messages are work in progress.<\/p>\n\n\n\nterraform init -input=false\n\nterraform plan -refresh-only -detailed-exitcode -lock=false -input=false -no-color\n\n# exit 0 state matches the real infrastructure\n# exit 1 the check itself failed\n# exit 2 something changed outside Terraform<\/code><\/pre>\n\n\n\n-detailed-exitcode<\/code> is what makes the result machine-readable at all: without it a successful plan returns zero whether or not anything changed, so your job passes forever. And -lock=false<\/code> is safe here specifically because a plan never writes state. Only terraform apply -refresh-only<\/code> updates the state file. A read-only detector that grabs the state lock will collide with real applies, and the loser of that race is the engineer trying to ship.<\/p>\n\n\n\nClassify the noise before you route anything<\/h2>\n\n\n\n
Provider normalization<\/h3>\n\n\n\n
Attributes another system legitimately owns<\/h3>\n\n\n\n
resource \"aws_autoscaling_group\" \"web\" {\n min_size = 2\n max_size = 10\n desired_capacity = 2\n\n lifecycle {\n # Owned by the scaling policy after creation.\n # Reverting this during a spike removes live capacity.\n ignore_changes = [desired_capacity]\n }\n}<\/code><\/pre>\n\n\n\nignore_changes<\/code> stops Terraform proposing an update for that attribute, and it applies only after creation, not on the first apply. It also delegates the attribute permanently. If someone sets desired capacity to zero at 3am, your check will not raise a hand. Fair trade for capacity. Terrible trade for an ingress rule.<\/p>\n\n\n\nTags stamped by something else<\/h3>\n\n\n\n
Route by blast radius, not by count<\/h2>\n\n\n\n
\n
terraform plan -refresh-only -no-color -json \n | jq -c 'select(.type == \"resource_drift\") | .change'<\/code><\/pre>\n\n\n\nMake the alert answer “who did this”<\/h2>\n\n\n\n
aws cloudtrail lookup-events \n --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0123456789abcdef0<\/code><\/pre>\n\n\n\nDecide the remediation path before you automate it<\/h2>\n\n\n\n
\n
Troubleshooting a drift job that quietly stopped working<\/h2>\n\n\n\n
Exit code 2 kills your script<\/h3>\n\n\n\n
set -e<\/code>, a non-zero exit terminates the script. Exit code 2 means drift, which is the case you care about, so a naive script dies exactly when it has something to report. Capture the code without letting the shell act on it:<\/p>\n\n\n\nset -euo pipefail\n\nterraform plan -refresh-only -detailed-exitcode -lock=false \n -input=false -no-color > plan.txt 2>&1 && code=0 || code=$?\n\ncase \"$code\" in\n 0) echo \"no drift\" ;;\n 2) notify_drift plan.txt ;;\n *) notify_broken plan.txt; exit 1 ;;\nesac<\/code><\/pre>\n\n\n\nCredentials that rot without complaining<\/h3>\n\n\n\n
A provider upgrade that drifts everything<\/h3>\n\n\n\n
Refresh times that outgrow the schedule<\/h3>\n\n\n\n
When to stop building this and buy it<\/h2>\n\n\n\n
\n\n\n\nCommon mistakes<\/h2>\n\n\n\n
\n
Best practices worth the effort<\/h2>\n\n\n\n
\n
Terraform drift detection FAQ<\/h2>\n\n\n\n
What is the difference between a normal plan and refresh-only?<\/h3>\n\n\n\n
Does a refresh-only plan modify my state file?<\/h3>\n\n\n\n
How often should Terraform drift detection run?<\/h3>\n\n\n\n
How do I find resources created outside Terraform?<\/h3>\n\n\n\n
Does ignore_changes hide drift I actually care about?<\/h3>\n\n\n\n
Why did drift suddenly appear across hundreds of resources?<\/h3>\n\n\n\n
The one thing worth keeping<\/h2>\n\n\n\n
\n\n\n\nNeed help making your drift alerts worth reading?<\/h2>\n\n\n\n
\n