That is the failure worth writing about. Drift itself is normal. Somebody fixes production at 2am, a managed service rotates a value, a policy engine stamps a tag. The part that breaks is the alert, and it breaks quietly, because a muted channel looks identical to a quiet one.

Switching on Terraform drift detection takes one flag. Building Terraform drift detection that engineers still read six months later means deciding what not to send: which plan mode you run, what you classify as noise, how you route by blast radius, and whether the message names who made the change. A detector firing daily with the same content is worse than none, because no detector leaves you knowingly blind while a muted one leaves you blind with a green dashboard. Every section below asks the same question differently: does this message change what a human does in the next ten minutes?

Pick the right plan mode or you will alert on your own backlog

This is the biggest source of false drift, and it catches almost everyone building their first check. A normal terraform plan compares three things: your configuration, your state, and the real infrastructure. It returns a non-zero exit code when any of them disagree, so it fires for real drift and just as loudly for a pull request that merged this morning and has not been applied. Your drift channel becomes a notification feed for your own deployment backlog, and people learn that most messages are work in progress.

Refresh-only mode narrows the question. HashiCorp documents it as a plan whose goal is only to update state and root module outputs to match changes made to remote objects outside of Terraform. That is the actual definition of drift.

terraform init -input=false

terraform plan -refresh-only -detailed-exitcode -lock=false -input=false -no-color

# exit 0  state matches the real infrastructure
# exit 1  the check itself failed
# exit 2  something changed outside Terraform

Two flags there matter more than they look. -detailed-exitcode is what makes the result machine-readable at all: without it a successful plan returns zero whether or not anything changed, so your job passes forever. And -lock=false is safe here specifically because a plan never writes state. Only terraform apply -refresh-only updates the state file. A read-only detector that grabs the state lock will collide with real applies, and the loser of that race is the engineer trying to ship.

The trade-off, stated plainly: refresh-only will not tell you that merged code has not shipped, and it will not find resources created outside Terraform, because it can only refresh what is already in state. I run both modes and route them separately. Refresh-only feeds the alert. Normal plan feeds a weekly digest tied to the repository, where unapplied changes are a delivery problem rather than a security one.

Classify the noise before you route anything

Most recurring drift falls into a few families. Sorting yours takes an hour and decides most of your routing rules.

Provider normalization

A JSON policy document comes back with keys reordered. An identifier returns in different case. An empty block reads back as null instead of an empty map. Terraform sees a difference because it compares a stored representation against an API response, and that round trip is not always lossless. Fix these in the configuration where you can, because a phantom diff that follows you into every plan also hides real diffs on the same resource.

Attributes another system legitimately owns

The classic is autoscaling capacity. Your config says two instances, a scaling policy moved it to five during a spike, and the next plan proposes taking it back to two. Applying that kills three instances mid-spike. The drift is real; reverting it is the wrong move.

resource "aws_autoscaling_group" "web" {
  min_size         = 2
  max_size         = 10
  desired_capacity = 2

  lifecycle {
    # Owned by the scaling policy after creation.
    # Reverting this during a spike removes live capacity.
    ignore_changes = [desired_capacity]
  }
}

Understand what you bought. ignore_changes stops Terraform proposing an update for that attribute, and it applies only after creation, not on the first apply. It also delegates the attribute permanently. If someone sets desired capacity to zero at 3am, your check will not raise a hand. Fair trade for capacity. Terrible trade for an ingress rule.

Tags stamped by something else

Organization tag policies, cost allocation tooling, backup schedulers and compliance agents all add tags. The AWS provider exposes both the tags you set and the effective set including defaults, and ignoring the effective set silences all tag drift at once. Convenient, and slightly dangerous if you use tags for access control, because tag-based policy conditions stop being watched. The same logic covers values the platform rotates or defaults on its own, like a managed secret version: they drift on a schedule that has nothing to do with your team, and belong in a digest.

The goal here is not a clean plan. A clean plan is a side effect. The goal is a decision per family: see it immediately, see it weekly, or never see it again.

Route by blast radius, not by count

“Drift detected: 12 resources changed” makes a human open a terminal before they know whether to care, and that friction is where attention goes to die. Split the output by what each resource can do if it is wrong:

• Wake someone. Anything governing exposure or access: security groups and firewall rules, IAM policies and roles, key policies, public access blocks, bucket policies, DNS records pointing at production.
• Review today. Instance sizes, scaling bounds, database parameters, retention settings, anything that costs money or changes capacity.
• Weekly digest. Tags, descriptions, names, and every family you classified as platform noise.

To split them you need structure, not text. Terraform’s machine-readable output emits a message for each resource that changed outside Terraform, carrying the resource address and the action it would take.

terraform plan -refresh-only -no-color -json 
  | jq -c 'select(.type == "resource_drift") | .change'

Check the exact field names inside that object against the machine-readable output reference for your Terraform version before building matching logic on them. Keep the severity mapping in the repository next to the code it describes. A resource type moving from digest to page should be a pull request, not something changed in a monitoring UI at midnight.

Make the alert answer “who did this”

Terraform tells you what changed. It has no idea who changed it, and that gap is where the investigation time goes. Half the drift you find is a legitimate incident fix nobody wrote down. If the alert names the actor, the response is a two-word reply in the thread instead of twenty minutes of archaeology. On AWS the audit trail can be queried by resource:

aws cloudtrail lookup-events 
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0123456789abcdef0

CloudTrail’s console event history only reaches back a limited window, so a weekly check may be hunting an event that has aged out; a trail delivering to S3, or CloudTrail Lake, covers the gap. Azure exposes the equivalent through Activity Logs, Google Cloud through Cloud Audit Logs. Wiring the lookup into the alert body does more for adoption than any amount of formatting.

Decide the remediation path before you automate it

There are three answers to a drift finding, and picking the wrong one can be worse than the drift.

1. Revert. Apply the configuration and push reality back. Right when the change was unauthorized or wrong.
2. Adopt. Change the code to match what is running. Right when the change was correct and the code was stale, which happens more than people admit.
3. Tolerate. Suppress the attribute deliberately, with a comment and an owner. Right when another system owns it.

The trap is automating option one. Auto-approving a remediation apply sounds like the natural endpoint of a drift pipeline, and it is fine right up until the drifted attribute is one that forces replacement. Then “restore the desired state” means destroying and recreating a live resource, unattended, at whatever hour the schedule runs. If you automate anyway, gate it hard: an allowlist of resource types, a check that the planned action is an in-place update rather than a replace, and a notification either way.

Troubleshooting a drift job that quietly stopped working

The nastiest failure mode is not a noisy check. It is a check reporting “no drift” because it never ran the plan successfully.

Exit code 2 kills your script

With set -e, a non-zero exit terminates the script. Exit code 2 means drift, which is the case you care about, so a naive script dies exactly when it has something to report. Capture the code without letting the shell act on it:

set -euo pipefail

terraform plan -refresh-only -detailed-exitcode -lock=false 
  -input=false -no-color > plan.txt 2>&1 && code=0 || code=$?

case "$code" in
  0) echo "no drift" ;;
  2) notify_drift plan.txt ;;
  *) notify_broken plan.txt; exit 1 ;;
esac

Note that the failure branch notifies too. Exit code 1 deserves the same urgency as drift, because a broken detector and a clean estate produce identical silence.

Credentials that rot without complaining

Drift checks run unattended, so nobody notices when a role trust policy changes, an OIDC subject claim stops matching after a repository rename, or a long-lived key gets rotated out. Add a deadman check: if the job has not reported within its expected interval, that absence is the alert. Same reasoning as heartbeat monitoring for backups.

A provider upgrade that drifts everything

Bump a provider and a new attribute default can appear across every resource of a type. Overnight the check reports hundreds of changed resources and whoever is on call assumes the tool is broken, which it sort of is. Commit the dependency lock file, pin the provider version, run the drift job on the same Terraform version as your apply pipeline, and treat provider upgrades as a reviewed change rather than something a bot merges.

Refresh times that outgrow the schedule

Refresh queries the API for every resource in state. Large workspaces get slow, then overlap their own schedule, then time out. The durable fix is splitting state along boundaries that already exist in your team structure. Terraform does offer targeting to narrow a plan, but it is documented as an exceptional-use option and prints a warning for good reason, so treat it as a stopgap rather than the design.

When to stop building this and buy it

A scheduled job in GitHub Actions or GitLab CI, or a systemd timer on a small always-on box from a provider like Contabo or InterServer, covers a handful of workspaces perfectly well and costs almost nothing. I would not reach for a platform before that stops working. What eventually stops working is the surrounding machinery: credentials for a dozen accounts across providers, concurrency against real applies, and retaining history so you can see that the same resource has drifted repeatedly this quarter, which is a process problem no single alert reveals.

That is where HCP Terraform’s health assessments, or a platform like Spacelift, Scalr or env0, starts earning its fee, since drift detection becomes a workspace setting with history and notifications attached rather than code you maintain. Two honest caveats: HCP Terraform gates health assessments behind its paid editions and requires remote or agent execution, so it is not a drop-in for local-state workflows, and any platform will happily send you the same undifferentiated wall of drift the homegrown script did. Severity tiering is a decision about your infrastructure that no vendor can make for you. Buying the runner does not buy the judgement.

Common mistakes

• Using normal plan mode for the alert. You notify yourself about unapplied work and teach people the channel is noise.
• Alerting on a count instead of a resource. “Nine resources changed” makes someone open a terminal to find out if they should care.
• Treating exit code 1 as a non-event. A detector that errors is indistinguishable from an estate with no drift.
• Taking the state lock for a read-only check. Monitoring should not be able to block a deployment.
• Auto-approving remediation across all resource types. One force-replacement attribute and the fix becomes the outage.

Best practices worth the effort

• Run refresh-only for drift and normal plan for unapplied changes, routed to different destinations.
• Keep the severity mapping in the repository, version controlled and reviewed like code.
• Give every suppression a comment naming the system that owns the attribute and a date to revisit.
• Alert on the detector failing and on the detector going quiet, not only on findings.
• Review recurring drift monthly. The same resource drifting repeatedly is a process gap, not an infrastructure one.

Terraform drift detection FAQ

What is the difference between a normal plan and refresh-only?

A normal plan compares configuration, state and reality, so it flags genuine drift and any committed change that has not been applied. Refresh-only compares state against reality only, which matches what most people mean by drift. Use refresh-only for the alert and normal plan for a delivery-focused report.

Does a refresh-only plan modify my state file?

No. Plan operations do not write state. Only applying a refresh-only plan updates the state file to match the drifted remote objects, and even then it makes no changes to infrastructure. That is why a detection job can safely skip the state lock.

How often should Terraform drift detection run?

Tie the interval to how long you would tolerate not knowing. Daily for production and anything holding customer data, weekly for lower environments, less often for slow-moving workspaces where refresh is expensive. Frequency matters less than whether the message gets read.

How do I find resources created outside Terraform?

Not with a plan. Terraform refreshes only what is already in state, so a resource it has never seen is invisible. You need a cloud-side inventory such as AWS Config, Azure Resource Graph or Cloud Asset Inventory, compared against your state, and import blocks to bring the results under management. Worth knowing that driftctl, the open-source tool most often recommended here, has moved into maintenance mode.

Does ignore_changes hide drift I actually care about?

Yes, and that is the point of using it. It stops Terraform proposing updates for that attribute regardless of what changed it, so a deliberate suppression and a malicious change look identical afterwards. Scope it to specific attributes, never whole resources you still care about, and document the owner.

Why did drift suddenly appear across hundreds of resources?

Almost always a provider or Terraform version change rather than something happening in your cloud account. A new provider release can add or reshape an attribute and every resource of that type reports a difference at once. Compare the versions used by the drift runner and the apply pipeline before investigating the infrastructure.

The one thing worth keeping

Terraform drift detection is not a monitoring problem, it is an attention problem. The flag that finds drift took an afternoon. Everything determining whether anyone still reads the output next quarter is judgement: refresh-only rather than normal plan, noise families sorted and suppressed deliberately, severity by blast radius rather than resource count, an actor name in the message, and an alert when the detector itself goes dark.

Judge it by one test: when a message lands in that channel, does anyone open it? If the honest answer is no, your detection is working and your alert is not.

Need help making your drift alerts worth reading?

This is common work once a Terraform estate grows past one team. Things I can help with:

• Auditing an existing drift job and finding the false positives that trained everyone to ignore it
• Building scheduled refresh-only checks in GitHub Actions, GitLab CI or a systemd timer, with correct exit code handling and a deadman alert
• Sorting recurring drift into families and deciding, per attribute, what to fix, adopt or suppress
• Building severity routing from the plan’s structured output so exposure changes page and tag changes do not
• Enriching alerts with CloudTrail, Activity Log or Cloud Audit Log lookups so findings arrive with an actor attached
• Splitting oversized state so refresh finishes inside its schedule, and judging whether a managed platform is worth it at your size

If you want a second opinion, send me a week of your drift output or the workflow file that produces it, and I will tell you what I would cut first.

The post Terraform Drift Detection That Nobody Mutes appeared first on John Nessime.

That is the shape of most EKS upgrade incidents. The part everyone books a maintenance window for, the control plane bump, is the part AWS handles and almost never goes wrong. The damage happens in the data plane rollout, because of decisions made months earlier that nobody connected to upgrades.

This post covers the three failure families that actually break EKS version upgrades: removed APIs you did not know you were calling, add-on updates that silently discard your configuration, and pod disruption budgets that deadlock the drain. For each one: what it looks like, how to catch it beforehand, and what the escape hatch costs.

Why the control plane is the easy half

An EKS cluster has two versions that must stay compatible. AWS manages the control plane; you manage the data plane, which in practice means the kubelet version on your nodes.

Two constraints shape everything else. In-place upgrades move one minor version at a time, so being three versions behind means three sequential upgrades and three data plane rollouts. And the kubelet cannot run ahead of the API server, so once the control plane moves, the clock starts on getting nodes to follow.

Failure one: the API calls you did not know you were making

Deprecation and removal are different events, and the gap between them is where people get comfortable. An API version gets deprecated, everything keeps working, the warning scrolls past in CI, and several releases later the version is gone and whatever depended on it starts getting 404s.

Your own manifests are usually fine; someone fixed those when the warning appeared. What breaks is the third-party agent nobody owns: a metrics collector, an ingress controller, a Datadog or New Relic agent installed by a Helm chart pinned two years ago. These call the API on a watch loop, so they hit removed APIs immediately, and they tend to fail quietly rather than crash-loop.

Cluster insights, and what it is actually reading

aws eks list-insights --cluster-name my-cluster

aws eks describe-insight 
    --cluster-name my-cluster 
    --id a1b2c3d4-5678-90ab-cdef-EXAMPLE11111

EKS runs upgrade readiness checks and exposes them through the insights API. The deprecated API check reads your control plane audit logs on a thirty-day rolling window, scanned about once a day. That has a consequence people trip over constantly: after you fix a deprecated call, the insight stays in ERROR until the last matching log entry ages out. The work is done, the check still fails, and if you gate automation on insight status your pipeline is blocked for weeks on a solved problem. Force a fresh scan rather than waiting for the daily one:

aws eks start-insights-refresh --cluster-name my-cluster
aws eks describe-insights-refresh --cluster-name my-cluster

The refresh re-reads the window; it does not shorten it. Audit-log detection also only sees APIs that were actually called, so a monthly CronJob or a chart sitting unapplied in a repo never appears. Insights tells you what your cluster did, not what it could do.

Ask the API server directly

The API server keeps its own counter, and it names the release that removes each API. Fastest check available:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Read the removed_release label first: it tells you whether this is urgent for the next upgrade or a warning for the one after. Scrape the metric into Prometheus, Grafana Cloud or Amazon Managed Grafana and you get a standing signal instead of a scramble.

For the callers, go to the audit logs. With control plane audit logging on, CloudWatch Logs Insights gives you the identity behind each deprecated call:

fields @message
| filter `annotations.k8s.io/deprecated` = "true"

The user.username field is what you need. It usually points at a service account in a namespace you did not install, which is your cue to find the chart’s owner. Static scanners like kubent and pluto cover the other half, the manifests that have not run recently. Use both.

Failure two: add-ons and the conflict resolution you picked once

Managed add-ons cover what a cluster cannot run without: the VPC CNI, CoreDNS, kube-proxy, the EBS CSI driver. They are version-pinned separately from the cluster, and their compatible ranges move with the Kubernetes version. The failure here is rarely the version, though. It is the conflict resolution mode.

When you update an add-on, EKS wants to apply its own field values. If you edited those fields in-cluster, that is a conflict, resolved by what you pass:

• NONE leaves your value alone and the update may simply fail. Safe, loud, and it stops your upgrade.
• OVERWRITE reverts your customization to the EKS default. The update succeeds. Your custom Corefile, your CNI tuning, your tolerations, gone, with no error.
• PRESERVE keeps your value through the update.

OVERWRITE is the one that hurts, because it is also the one people reach for when an update fails and they want it to stop failing. It is the standard fix when migrating a self-managed add-on to a managed one, so it gets learned there and carried into the update path, where it means something different. A CoreDNS update with OVERWRITE will replace a Corefile carrying your stub zones and forwarders, and resolution for those zones stops cluster-wide while every health check stays green.

aws eks update-addon 
    --cluster-name my-cluster 
    --addon-name coredns 
    --addon-version <compatible-version> 
    --resolve-conflicts PRESERVE

One more trap: the same value means different things on create and on update. On create, PRESERVE behaves close to NONE, leaving an existing self-managed add-on alone rather than adopting your settings. Check the operation, not the word.

The durable fix is to stop editing add-on resources with kubectl at all. Put customizations in the add-on’s configuration values so EKS owns them and there is nothing left to conflict over. An afternoon of work, and this failure family goes away permanently.

Failure three: PDB deadlocks and the fifteen-minute drain clock

This is the one that stalls upgrades halfway, and the mechanics matter because the fix depends on where you are stuck. A managed node group update runs in four phases: setup, scale up, upgrade, scale down. On the default strategy, scale up launches replacements before anything is terminated, which is what preserves capacity. Then the upgrade phase does this, per node:

1. Pick a node to replace, up to the configured maxUnavailable.
2. Cordon it once a new node reaches Ready.
3. Drain the pods off it.
4. Wait sixty seconds after eviction, then ask the Auto Scaling group to terminate it.
5. Repeat until nothing is left on the old launch template version.

Step three has a fifteen minute budget. If pods have not left in that time and you did not pass the force flag, the whole update fails with PodEvictionFailure and you are left with a half-migrated node group.

The deadlock nobody sees coming

A PDB says how many pods must stay available during voluntary disruption, and eviction is voluntary disruption. So the arithmetic is simple and, once you see it, obvious: a deployment with two replicas and a budget requiring two available pods can never be evicted. Not slowly. Never. The drain retries until the clock runs out.

One replica with a one-pod minimum is the same deadlock in friendlier clothing, as is a percentage budget that rounds up to the full replica count. And if two PDBs select the same pod, the more restrictive one wins, producing deadlocks invisible in either manifest read alone. The pre-flight check is cheap, and I run it before every upgrade:

kubectl get pdb --all-namespaces

Read the ALLOWED DISRUPTIONS column, not the configured minimum. Any budget sitting at zero is a drain that will hang. Fix those before touching the node group, by raising replicas or loosening the budget so at least one disruption is permitted.

There is a second cause of PodEvictionFailure with nothing to do with PDBs. The upgrade taints nodes as unschedulable, so after eviction the node should be empty. Anything tolerating every taint keeps it non-empty, and the drain never satisfies its exit condition. If your budgets look fine and eviction still stalls, look at what tolerates NoSchedule unconditionally.

What the force flag actually costs

aws eks update-nodegroup-version 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup 
    --force

Force does not negotiate with budgets. It deletes the pods and terminates the node regardless. For a stateless service behind a load balancer with several replicas, that is fine and the upgrade completes. For a quorum-based system it is a bad idea: if the budget in front of your etcd, Consul, Kafka or ZooKeeper pods is doing its job, forcing past it can take the cluster below quorum. The budget was the protection; force is the decision to ignore it.

My rule is that force is reasonable on a node group I know holds only stateless workloads, and never a first response on a mixed one. Reaching for it because you do not know what is on the node is the actual problem.

If you run Karpenter, the levers are different

Karpenter has no update phases. It detects drift when a NodePool or EC2NodeClass stops matching a node’s configuration, then replaces the drifted nodes. Disruption budgets control the pace and can be scheduled, which is how you build a maintenance window without booking one:

spec:
  disruption:
    budgets:
      - nodes: "0"
        reasons: ["Drifted"]
        schedule: "0 6 * * mon-fri"
        duration: 12h

Two things bite people. Schedules are evaluated in UTC, so a window that looks like business hours to you may not be. And any period not covered by a budget is unrestricted, so gaps are not “no disruption”, they are “unlimited disruption”.

The other lever, terminationGracePeriod on the NodePool, overrides everything. Once it elapses, remaining pods are force-deleted regardless of PDBs or the karpenter.sh/do-not-disrupt annotation. It is a deadlock breaker, not a safety setting, and if it is shorter than your budget schedule you can get forced terminations inside a window you thought was protected.

The rollback window changes the calculation

Upstream Kubernetes has no control plane rollback. That single fact is why so many teams built heavyweight approval processes around upgrades. EKS now supports reverting an in-place control plane upgrade to the previous minor version within a seven day window, preserving etcd data and running workloads. It covers in-place upgrades only, which makes sense: with blue/green the old cluster is still there and rollback is a DNS change.

What changes is the shape of the risk. An upgrade used to be a one-way door, so the rational response was exhaustive testing beforehand. Now the first week is the test, under real traffic, with a way back. That argues for upgrading more often on smaller jumps, and treating the week after as an active watch period rather than a closed ticket.

It is not a general undo. Once the window closes, or you upgrade again, the previous version is gone, and it does not roll back add-ons or application state. Blue/green still has a place when you are several versions behind, since sequential in-place hops multiply exposure. The cost is running two clusters during migration, worth modelling first in Cost Explorer with both tagged, or in a tool like Vantage or CloudZero if you already run one.

Troubleshooting a stalled or failed upgrade

When a node group update fails, the reason comes back as an error code. Start there rather than guessing:

aws eks describe-nodegroup 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup

• PodEvictionFailure. A budget blocked the drain, or something tolerates all taints. Check ALLOWED DISRUPTIONS first, then tolerations.
• NodeCreationFailure in scale up. New nodes never reached Ready. Usually no capacity for your instance type in an Availability Zone, an EC2 instance limit in the account, or custom user data breaking the bootstrap.
• Slow bootstrap. A node has fifteen minutes to join, measured from when the Auto Scaling group size increases. Heavy user data, large images pulled at boot or a slow init path push past it.
• Pods stuck terminating with no budget in sight. Look for blocked I/O. A pod with an NFS mount whose server is unreachable will not terminate, and eviction pressure changes nothing.
• Insights still in ERROR after the fix. The thirty day audit window. Refresh, confirm another way, move on.

One structural note: if the Cluster Autoscaler scales the node group during the scale down phase, the upgrade workflow exits without restoring the group to its original size. Check your desired and max counts afterwards.

Common mistakes

• Treating insights as the only pre-flight check. It reads audit logs, so it cannot see code paths that have not run.
• Reaching for OVERWRITE to make a failing add-on update succeed. It succeeds by discarding your configuration.
• Upgrading add-ons and the control plane in one unbroken pipeline run, so you cannot tell which change broke things.
• Leaving maxUnavailable at its default of one on a large node group, then wondering why the rollout takes hours.
• Forgetting self-managed components. Cluster Autoscaler, the AWS Load Balancer Controller and metrics-server are versioned against Kubernetes too, and nothing reminds you.

How I sequence EKS version upgrades

1. Read the release notes for the target version. Not the summary post, the removals section.
2. Run all three deprecation checks: insights for what the cluster did, the API server metric for what it does now, a static scanner for what sits in the repo.
3. Audit PDBs and fix every zero. Highest-value ten minutes in the process.
4. Update add-ons with PRESERVE to versions compatible with both the current and target Kubernetes version, so components talking to the API server are ready before it moves.
5. Upgrade the control plane and stop. Watch error rates and API server latency long enough to mean something.
6. Roll the least critical node group. Confirm pods reschedule cleanly and nothing depended on a node label or an AMI-specific detail.
7. Roll the rest, confirm every kubelet matches the control plane, then treat the following week as the real test. That is what the rollback window is for.

FAQ

Can you do EKS version upgrades with zero downtime?

For stateless workloads with sensible replica counts, spread across zones, with budgets allowing at least one disruption, yes. The default node group strategy launches replacements before terminating anything, so capacity holds throughout. Stateful workloads can too, but only if the application handles rolling leadership changes and the budget permits exactly one disruption at a time.

Why does the update fail with PodEvictionFailure when my PDBs look fine?

Two likely causes. Multiple budgets may select the same pod, in which case the most restrictive applies and neither manifest reveals it alone. Or something on the node tolerates every taint, so the node never becomes empty and the drain never completes. Check ALLOWED DISRUPTIONS across all namespaces, then check tolerations.

Is the force update option safe?

Safe for stateless workloads with multiple replicas. Not safe for quorum-based systems, because it deletes pods regardless of the budget protecting quorum. Use it when you know what is on the node group and the answer is nothing that minds being killed.

Does Karpenter make upgrades easier than managed node groups?

More automatic, which is not the same thing. Drift detection replaces outdated nodes without you running a command, but you inherit a different set of controls to get wrong: budgets evaluated in UTC, uncovered schedule gaps that mean unlimited disruption, and a grace period that overrides PDBs entirely. Neither is easier. They fail differently.

The one thing worth remembering

EKS version upgrades do not fail at the moment you run the command. They fail on decisions already sitting in the cluster: a budget written for a two-replica deployment, an add-on customized with kubectl rather than through its configuration values, a chart pinned before an API was removed.

Which means the pre-flight checks are the upgrade. Ten minutes reading ALLOWED DISRUPTIONS and the deprecated API metric tells you more about how tomorrow goes than any amount of care during the rollout. The rollback window gives you a second chance on the control plane. It will not un-delete a pod that force-eviction pulled out of a quorum.

Need a second pair of eyes on your upgrade path?

Most of this work is unglamorous and specific. What I would actually do on a cluster facing a version bump:

• Audit every PDB against its replica count and flag the ones that will deadlock a drain.
• Trace deprecated API calls back to the chart, controller or service account making them, from audit logs rather than guesswork.
• Move add-on customizations into managed configuration values, so conflict resolution stops being a decision made under pressure.
• Write a repeatable upgrade runbook: pre-flight checks, sequencing, and the rollback path for your specific workloads.
• Alert on the deprecated API metric and node group update events, so the next version is a scheduled task rather than an incident.
• Review Karpenter disruption budgets and grace periods for the gaps that turn into unplanned rollouts.

If you are staring at a failed node group update right now, send me the output of kubectl get pdb --all-namespaces and the error code from describe-nodegroup. That is usually enough to say what is holding the drain.

The post EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks appeared first on John Nessime.

The expensive stuff is not the running instances. It’s the access keys nobody can attribute, the CloudTrail trail that quietly stopped delivering months ago, the Terraform state that no longer matches reality, and the region somebody spun a proof of concept up in two years ago. None of that comes up on a kickoff call. All of it comes up in week three, after you’ve quoted.

So I run an AWS account health check before I give anyone a number. It’s read-only, it takes a couple of hours, and it’s the cheapest insurance in consulting. This post is the actual check: the six passes, the commands, what each finding does to the estimate, and what to do when the client won’t give you the access to run it.

Why the AWS account health check comes before the quote

A quote is a bet on how much unknown there is. You’re pricing the work in the brief plus everything you’ll find while doing it, and the second half is invisible until you look. Two accounts can look identical and differ by a factor of three in effort. The difference is nearly always archaeology: how much has to be reverse engineered before you can safely change anything.

There’s a second reason that matters more than pricing. If you touch an account that already had a compromised key sitting in it, the incident becomes yours by proximity. Establishing what the account looked like before you arrived isn’t paranoia. It’s the only version of events you’ll be able to evidence later.

The access I ask for, and what a refusal tells me

I ask for a cross-account IAM role that the client creates and my account assumes. Not an IAM user. Not keys in an email. A role, with an external ID the client generates.

The external ID solves one specific problem. Without it, a trust policy that names my account can be used by anyone who talks me into assuming a role on their behalf, and I end up acting inside a third party’s account without knowing it. AWS calls this the confused deputy problem. It’s the piece of cross-account setup people skip most often.

On that role I ask for two AWS managed policies: SecurityAudit and ViewOnlyAccess. The pairing is the point. ViewOnlyAccess lets you enumerate resources across services, so you can see what exists. SecurityAudit lets you read configuration and policy documents, so you can see how it’s wired together. Neither alone gives you the picture, and neither lets you change anything. Billing is separate again: Cost Explorer read access has to be added explicitly and the feature has to be enabled in the account.

The reply to that request tells you as much as the check does.

• “Here’s the role ARN and external ID” means somebody there understands AWS identity. Good sign.
• “I’ll just make you an admin user, easier” means nobody is managing access. Expect that looseness everywhere else.
• “Our previous developer set it up and we can’t reach him” means you’re inheriting credentials nobody can account for. Price the rotation work as its own line.
• “We’d rather not grant access before signing” is fair. Offer a paid discovery engagement. If they won’t pay for two hours of assessment, they won’t pay for the fix.

The six passes

Each pass answers one question. Everything here is read-only, and the order matters, because each pass changes how you read the next.

Pass 1: Who can do what

Confirm where you actually are first. That sounds trivial until you’re in an organization with a dozen similarly named accounts.

aws sts get-caller-identity
aws iam list-account-aliases
aws organizations describe-organization

If the third command fails because the account isn’t in an organization, that’s a finding by itself. Standalone means no service control policies, no organization trail, and no central place to revoke anything.

Then the credential surface. The summary gives counts; the credential report gives per-principal detail, including when each password and key was last used.

aws iam get-account-summary

aws iam generate-credential-report
aws iam get-credential-report --query Content --output text | base64 --decode > credential-report.csv

The report generates asynchronously, so the second call may need a retry the first time. What I read from it: how many IAM users exist at all, how many have keys that have never been used, and how many have keys last used a long time ago that are still active.

An unused key is worse than a busy one. A key in daily use is attributable to something. A key last used eighteen months ago and still enabled belongs to a person or script nobody remembers, which means nobody will notice when someone else starts using it.

Then check whether anyone has ever asked this question before you:

aws accessanalyzer list-analyzers

IAM Access Analyzer offers external access analyzers, which flag resources shared outside a zone of trust you define, and unused access analyzers, which surface unused roles, keys and passwords. Unused access analysis is billed per IAM role and user analyzed per month, so it isn’t free, but it answers the least privilege question far faster than reading policy documents by hand. An empty list means nobody has ever asked this account who can reach what.

Root closes the pass. In an AWS Organizations setup, centralized root access management lets the management account or a delegated IAM administrator remove root credentials from member accounts entirely, so there’s no root password, key or MFA device to manage per account. Live root credentials on every member account is a real finding, and one of the few security fixes that’s genuinely quick.

Pass test: you can name every principal that can reach production, and say when each last did.

Pass 2: Would anyone notice

This is the pass that most often changes my number, because it’s the failure that stays invisible until it isn’t. Logging that exists but has silently stopped working is more dangerous than no logging, since everybody believes they’re covered.

aws cloudtrail describe-trails 
  --query 'trailList[].{Name:Name,MultiRegion:IsMultiRegionTrail,Org:IsOrganizationTrail}'

aws cloudtrail get-trail-status --name TRAIL_NAME_HERE

aws configservice describe-configuration-recorder-status

The first command tells you a trail is configured. The second tells you whether it’s delivering, which is the part that matters. A trail whose last delivery failed because someone edited the destination bucket policy still shows up in the console as a configured trail. The status call is where the truth is.

Multi-region matters for the same reason region sprawl does: a single-region trail records nothing about the region where somebody is quietly running compute. Check GuardDuty too, and how long logs are actually retained. Ninety days of retention against a one-year obligation is a project, not a task, and it belongs in the quote rather than in week four.

One naming trap worth knowing: AWS renamed the original Security Hub to Security Hub CSPM and launched a separate, newer service under the Security Hub name. When a client says “we have Security Hub,” ask which one, because the findings format and the configuration are not the same thing.

Pass test: a trail is delivering right now, covers every region, and its logs are kept for at least as long as the business needs them.

Pass 3: How wide the blast radius is

People forget regions. Not occasionally, constantly. A demo in Singapore, a test bucket in Ireland, a Lambda deployed from a tutorial that defaulted to us-east-1.

for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  count=$(aws ec2 describe-instances --region "$r" 
    --query 'length(Reservations[].Instances[])' --output text)
  echo "$r $count"
done

Left without --all-regions, describe-regions returns the regions enabled for the account, which is exactly what you want. Adding the flag pulls in disabled regions too, and the per-region calls against those just error out.

Run the same loop shape for anything carrying state or cost: RDS instances, load balancers, EBS volumes, NAT gateways. NAT gateways are the classic. They bill hourly whether or not anything routes through them, and a forgotten one in a region nobody visits is pure waste. Then the two exposure checks I never skip:

aws ec2 describe-security-groups 
  --filters Name=ip-permission.cidr,Values=0.0.0.0/0 
  --query 'SecurityGroups[].{Id:GroupId,Name:GroupName,VpcId:VpcId}'

aws s3api list-buckets --query 'Buckets[].Name' --output text

For each bucket that matters, check its public access block configuration and its policy. An open security group on 443 in front of a load balancer is fine. An open security group on a database port is a different conversation, and you want that conversation before you own the account.

Pass test: you have a list of every region with resources in it, and nothing on that list surprises the client.

Pass 4: Whether any of this is described in code

The question isn’t “do they use Terraform.” It’s whether the code still describes the account. Those diverge quietly. Somebody makes an urgent console change during an incident, never brings it back into code, and from then on the repo is fiction that plans cleanly right up until the day it destroys something.

terraform plan -refresh-only

terraform plan -detailed-exitcode

The refresh-only plan shows where real infrastructure has drifted from state without proposing to change anything. The detailed exit code version is the one for scripts: it exits 0 for no changes, 1 for an error and 2 when there’s a diff, so you can gate on drift instead of reading output by eye.

Three things I want to know here:

• What fraction of live resources are managed by code? Half a codebase is often worse than none, because it creates false confidence about which half.
• Where does state live and who can write to it? Remote state in a versioned, locked S3 bucket is a different risk profile from a state file on a laptop.
• Does the pipeline that applies changes still run? Credentials that expired six months ago mean everything since went in by hand.

Tagging belongs here too. With nothing tagged you can’t attribute cost, can’t scope a change safely, and can’t tell a production resource from an abandoned experiment. Untagged environments add hours to everything that follows.

Pass test: a refresh-only plan comes back clean, or the drift is a short list you can read out loud.

Pass 5: What shape the spend is in

I don’t care much about the total. I care about the shape: which services dominate, whether the curve is flat or climbing, and whether anything jumped without an explanation.

aws ce get-cost-and-usage 
  --time-period Start=$(date -d '6 months ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=SERVICE

The date -d syntax is GNU coreutils; on macOS use the BSD equivalent or a fixed start date. Worth knowing before you loop this: the Cost Explorer API bills per paginated request, unlike the console view, so a wide query across several dimensions is a small but real charge on the client’s bill. Keep it narrow and mention it in your notes.

• Data transfer in the top three usually means an architecture problem rather than a rate problem. Cross-AZ chatter and NAT gateway egress are the usual suspects.
• A step change with no matching launch deserves a direct question. Sometimes it’s a feature. Sometimes it’s a runaway job. Occasionally it’s mining on a stolen key.
• No budgets and no anomaly monitors means nobody finds out about a spike until the invoice arrives.
• Heavy commitment purchases against workloads you’re about to re-architect are a constraint on your design, and need to be on the table before you propose anything.

If the client wants ongoing cost visibility past what Cost Explorer offers, platforms like Vantage or CloudZero sit on the Cost and Usage Report and give per-team allocation and unit economics that native tooling makes awkward. Raise it as a separate line, not part of the engineering work.

Pass test: you can explain the top three cost lines in a sentence each, and account for every month-on-month jump.

Pass 6: What happens at two in the morning

Short pass, about escalation. What support plan is the account on, and would anything wake a human up?

Support plan matters more than people expect. On Basic you get the service limits checks and a small subset of Trusted Advisor security and fault tolerance checks, manual refresh only, and no technical support cases at all. The full check library and the Trusted Advisor API need a paid plan. If your engagement depends on opening a case during a cutover, that’s a prerequisite, not a nice to have.

There’s one date worth carrying in your head, because it’s a real deprecation rather than a version number. AWS has announced that Developer Support, Business Support and Enterprise On-Ramp are discontinued on 1 January 2027, with Business Support+ as the replacement tier. If you’re scoping anything that runs past that boundary, check which plan the client is on and whether their support cost is about to move.

Then monitoring. CloudWatch alarms wired to an SNS topic with no confirmed subscribers are the same failure as the dead trail: configured, visible, useless. If they run anything beyond alarms, whether that’s Grafana Cloud, Datadog or a self-hosted Prometheus stack, find out who holds the credentials and whether the routing has been tested this year.

Pass test: you can name the human who gets woken up, and the path that reaches them.

The findings that actually move the number

• Unattributable credentials. Each one has to be rotated or removed, and each needs a conversation with whoever might break. That’s coordination time, which is the hardest kind to compress.
• No working audit trail. No baseline means you can’t prove what you did or didn’t change. Add the cost of standing logging up before the real work starts.
• Significant drift between code and reality. Reconciliation is slow, fiddly and carries destruction risk. The single largest multiplier I see.
• No tags. A tax on every subsequent task rather than a task of its own, since every scoping decision becomes an investigation.
• Resources in unmentioned regions. Each is a small parallel environment with its own security groups, its own logging gap and its own bill.
• One shared admin identity. Nothing in CloudTrail is attributable to a person, so your first task is identity work the client didn’t ask for.

I don’t present these as criticism. I present them as line items with an hour estimate against each, and let the client decide what’s in scope. That conversation is much easier before a contract exists than after.

If you only run five checks

Sometimes you get twenty minutes and a screen share.

1. Pull the credential report and count active keys that haven’t been used recently.
2. Confirm a CloudTrail trail exists, is multi-region, and is currently delivering.
3. Loop the enabled regions for running instances and NAT gateways.
4. Run a refresh-only plan against whatever infrastructure code exists.
5. Pull six months of cost grouped by service and look at the shape.

Those cover identity, evidence, blast radius, delivery speed and money. The rest of the check refines a picture these five already gave you.

Where this check goes wrong

• Treating it as an audit. You’re scoping work, not issuing a compliance report. A twenty page findings document makes a prospect defensive and costs you a day.
• Checking one region. The default region in your profile is not where the problems are.
• Trusting the console over the API. Configured and working are different states, and the console is better at showing the first.
• Taking long-lived keys because it’s faster. Accept an admin access key by email to save fifteen minutes and you’ve become part of the problem you were hired to assess.
• Running the loops unannounced. A burst of API calls from an unfamiliar principal is exactly what GuardDuty exists to flag. Tell them you’re doing it and roughly when.

Frequently asked questions

How long should an AWS account health check take?

Two to three hours for a single account once you have access, including the write-up. Multi-account organizations take longer but not linearly, because the interesting variation concentrates in one or two accounts and the rest are copies of a landing zone pattern.

Should I charge for the assessment?

For a short engagement, folding it into the proposal is fine. For anything substantial, or where the client is shopping several consultants, quote it as a small standalone piece with a written deliverable. It qualifies the client, protects your time, and the findings have value to them even if they hire someone else.

What if the client won’t give me read-only access before signing?

Quote time and materials, or quote fixed price with an explicit assumptions list naming every unknown. Do not quote a fixed price on an account you haven’t seen. The pattern that works is a small paid discovery phase, then a fixed price set once you know what you’re dealing with.

Is ReadOnlyAccess enough on its own?

It’s broader than you probably want. ReadOnlyAccess extends to reading data in some services, not just configuration, which is more than you need for scoping and more than many clients will be comfortable granting. The SecurityAudit and ViewOnlyAccess pairing is narrower and easier to justify in writing.

Can I automate the whole thing?

The enumeration, yes, and it’s worth scripting since you’ll run it repeatedly. The interpretation, no. Whether five unused IAM users are a problem depends entirely on what the client is trying to do, and that judgement is most of what you’re selling.

What if I find something actively compromised?

Stop and tell them immediately, in writing, with timestamps. Don’t remediate unilaterally: you’re read-only, and containment steps can destroy evidence. Your job is a clear description of what you saw and when, so they can involve whoever they need to.

Does this apply to small accounts too?

Especially to small accounts. Larger organizations usually have some governance, even imperfect. A three year old account at a small business that four different contractors have touched is where the surprises live, and it’s exactly the account that gets described as simple.

The one thing worth remembering

An AWS account health check isn’t a security exercise dressed up as sales. It’s the only reliable way to price uncertainty, and uncertainty is what you’re actually quoting on.

If you take one habit from this, take the second pass. Everything else on the list is something you’d eventually trip over while doing the work. A logging gap is different, because it stays invisible while it costs you, and by the time it matters you’re the person who was in the account when it happened. Check the trail is delivering. Then check the rest.

Need a second pair of eyes on an AWS account?

I do this for consultancies scoping engagements and for businesses who inherited an account they don’t fully understand. Usually that looks like:

• A read-only AWS account health check with a written findings list, hour estimates per item, and a clear split between urgent and later.
• Cross-account audit role setup, with a trust policy and external ID you can hand to any consultant without granting more than you meant to.
• Credential archaeology: working out what every active access key belongs to, then rotating or removing the ones that belong to nothing.
• Getting CloudTrail, AWS Config and log retention into a state where they’re delivering rather than just configured.
• Terraform drift reconciliation, including importing unmanaged resources back under code without a destructive plan.
• Cost shape reviews: the forgotten NAT gateways, orphaned volumes and idle regions, plus budgets and anomaly alerts that reach a real inbox.

If you’ve got a credential report, a terraform plan output or a Cost Explorer screenshot you can’t make sense of, send it over and I’ll tell you what I’d look at next.

The post The AWS Account Health Check I Run Before Quoting a Project appeared first on John Nessime.