That gap is the whole problem with permission cleanup. IAM Access Analyzer unused access findings are genuinely useful, and they are also built on evidence with known holes in it. AWS says so in its own documentation. Treat a finding as proof rather than as a lead and sooner or later you take something down.

This post covers what an unused access finding actually asserts, the four blind spots that turn a clean-looking finding into an outage, and a removal procedure that fails safe.

What an unused access finding actually claims

An unused access analyzer is a separate analyzer type from the external access one. You create it per account or across an organization, and it produces four kinds of finding: unused roles, unused IAM user access keys, unused IAM user passwords, and unused permissions at the service and action level for principals that are active.

aws accessanalyzer create-analyzer 
  --analyzer-name unused-access-prod 
  --type ACCOUNT_UNUSED_ACCESS 
  --configuration '{"unusedAccess": {"unusedAccessAge": 90}}'

unusedAccessAge is the threshold in days beyond which access counts as unused, and it accepts 1 to 365. Use ORGANIZATION_UNUSED_ACCESS from a delegated administrator account. Unused access findings don’t vary by Region, so you don’t need one analyzer per Region.

Here is the part that matters. The analyzer does not replay CloudTrail and reconstruct what your workload did. It reads IAM last accessed information, the same data behind the Access Advisor tab in the console. Everything the analyzer knows, and everything it doesn’t, comes from there. The blind spots below are last accessed blind spots, inherited wholesale.

The four blind spots that break production

1. Action-level tracking doesn’t cover everything

Action last accessed information only exists for the services and actions AWS has explicitly added to the tracked list, and it covers management actions. The IAM console spells out one case in plain text: action last accessed information is not available for Amazon S3 data events.

Think about what that means for a data pipeline. A role whose entire purpose is s3:GetObject and s3:PutObject against one bucket is doing object-level work all day. Bucket-level calls like ListBucket are management events and get tracked. The object-level calls that constitute the actual job are data events, and they sit outside action last accessed.

AWS puts the warning in the documentation itself: do not make permissions decisions based solely on the absence of tracking information. Almost nobody reads that sentence.

2. Denied attempts count as access

Service last accessed data includes all attempts to call an AWS API, not just successful ones. A role that has been hammering a service it has no permission for, failing every time, shows up as having used that service.

This runs opposite to the first blind spot, and it is why a findings list can be simultaneously too aggressive and too timid. Dead permissions look alive because a misconfigured client keeps retrying against them, so you clean up around them and leave the real cruft in place.

3. The calendar is longer than your window

A ninety-day tracking period is the common default and it is wrong for a predictable class of role:

• Disaster recovery and failover roles, exercised once or twice a year
• Quarterly close, audit, and regulatory reporting roles
• Annual key rotation or certificate renewal automation
• Break-glass roles, which should be unused by design
• Seasonal workloads that only wake up for a peak period

There is also a ceiling worth knowing. unusedAccessAge tops out at 365 days, while IAM reports last accessed activity for at least the trailing 400 days. The analyzer cannot be configured to span the full window the underlying data actually holds, so for anything on an annual cycle it will not settle the question on its own.

4. Service-level use says nothing about resource-level need

A finding operates on services and actions. Your policy probably operates on services, actions, resources, and conditions. Knowing a role used secretsmanager:GetSecretValue last week does not tell you which secret, and narrowing the policy to a specific resource ARN based on a finding is a guess dressed up as evidence.

This is where policy generation earns its place. It reads CloudTrail activity for a principal over a window and drafts a policy from what it observed, including resource-level detail where the log has it. Same caveat: policy generation does not identify action-level activity for data events either. Better first draft, not an oracle.

A removal procedure that fails safe

The core idea: deny before you delete. An explicit deny is one API call to add and one to remove, effective immediately in both directions. Deleting a policy statement or a role is not symmetrical; restoring it means someone reconstructing intent under pressure. Use deny as a reversible dry run for deletion.

1. Sort findings by blast radius, not by count. An unused permission on a role assumed by one Lambda is a different risk from one on a role fifty engineers assume daily.
2. Confirm the role’s activity independently before touching permissions.
3. Attach a scoped deny as an inline policy on the role, leaving the existing allow untouched. An explicit deny wins over any allow, so this simulates removal exactly with none of the loss.
4. Watch CloudTrail for denials attributed to that role. This is the observation window and it is the step people skip.
5. Wait through at least one full business cycle, longer if the workload has monthly or quarterly rhythm.
6. Only then edit the allow and remove the deny.

For step two, the cheapest independent check is the role’s own last-used timestamp:

aws iam get-role --role-name legacy-reporting-role 
  --query 'Role.RoleLastUsed'

That returns LastUsedDate and the Region where the role was last used. LastUsedDate is null if the role has not been used within the tracking period, which is not the same as never used. The Region field is quietly valuable: a supposedly dead role showing activity in a Region nobody works in is worth a conversation before a deletion.

For step three, the deny is deliberately narrow. Deny the flagged actions on the flagged service and nothing else:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CanaryDenyUnusedRedshift",
      "Effect": "Deny",
      "Action": "redshift:*",
      "Resource": "*"
    }
  ]
}

Give it a Sid you can grep for, because in six months somebody will find this policy and have no idea whether it is a canary or a control.

For step four, you want CloudTrail records where the request failed on authorization. The field is errorCode, and the values that matter are AccessDenied plus the EC2-specific Client.UnauthorizedOperation. Filter on the assumed-role session issuer so you catch every session. CloudTrail Lake or Athena over your trail bucket both work.

To make this an ongoing signal rather than a one-off query, wire a CloudWatch Logs metric filter on AccessDenied to a dashboard. Amazon Managed Grafana, self-hosted Grafana, or Grafana Cloud all render it fine. What matters is that it is broken down by principal, so a spike points at a role rather than at the account.

Deleting the role itself has a separate trap

When you name a principal by ARN in a policy, IAM stores the underlying unique principal ID rather than the string. Delete that role, recreate it with the identical name, and you get a new unique ID. Every policy that referenced the old one now points at something that no longer exists. Nothing errors at deletion time. The break surfaces later, in another account, when a cross-account assume-role call starts failing and the trust policy on screen looks exactly correct.

Before deleting a role, check the things a findings list will never tell you:

• Is it named as a principal in any resource policy, S3 bucket policy, KMS key policy, or another account’s trust policy?
• Is it attached to an instance profile?
• Is it managed by CloudFormation, Terraform, or OpenTofu? Deleting out of band leaves drift that surfaces on the next apply.
• Is it a service-linked role? Those are owned by a service and have their own deletion procedure.
• Is it the execution role for a Lambda function, ECS task, or Step Functions state machine that runs rarely?

Safe order: detach the permissions, leave the role and its trust policy in place, wait another cycle, then delete. An empty role is harmless. A missing one is not.

Keeping the findings list small enough to read

A findings list nobody triages is worse than none, because it produces the appearance of coverage. Two levers keep it honest. The first is exclusions, which can skip whole accounts or skip principals by tag, and can be updated at any time:

aws accessanalyzer update-analyzer 
  --analyzer-name unused-access-prod 
  --configuration '{
    "unusedAccess": {
      "analysisRule": {
        "exclusions": [
          {"accountIds": ["111122223333"]},
          {"resourceTags": [{"lifecycle": "break-glass"}]}
        ]
      }
    }
  }'

Tag-based exclusion is the one to reach for. Sandbox accounts churn roles and generate noise with no security value; break-glass roles are supposed to look unused. Excluding by tag survives new accounts joining the organization. An account ID list does not.

The second lever is archiving findings you have consciously accepted:

aws accessanalyzer update-findings 
  --analyzer-arn <your-analyzer-arn> 
  --ids <finding-id> 
  --status ARCHIVED

Archiving is a decision record, not a dismissal. Write down why somewhere durable, because the same finding will come back to a different person.

Scope has a cost dimension too. Unused access analysis is billed on the number of IAM roles and users analyzed per month, per analyzer, so a second analyzer with a different tracking period doubles the principals counted. If you watch spend through Cost Explorer or a layer like Vantage or CloudZero, confirm where this line lands before widening scope organization-wide.

Troubleshooting

No findings yet, or a finding that won’t clear after you fixed it. After creating or updating an analyzer it takes time for findings to appear, recent activity takes around four hours to reach last accessed data, and scans are periodic rather than instant. Give it a full cycle. If you use the CLI or API for last accessed reports, generate a fresh one; the old report will not update itself.

The finding says unused but the team insists otherwise. Assume the team is right and the tracking has a hole. Check whether the work is data-plane activity, then whether the actions involved are on the tracked list at all. Absence of tracking data and absence of use look identical from the findings page.

Your canary deny broke something unexpected. That is the canary working. Detach the inline policy, confirm the failures stop, then narrow the deny to specific actions rather than the whole service and run it again.

Something broke but no AccessDenied appears in CloudTrail. Check whether the failing call is a data event. A trail logs management events by default; data event logging is opt-in per resource type and carries its own charges. No configured data events, no record of the denial.

Common mistakes

• Treating a finding as evidence of non-use rather than absence of recorded use. Different claims.
• Deleting the role in the same change as removing its permissions, collapsing two reversible steps into one that isn’t.
• Setting the tracking period once and never revisiting it against the workload’s actual rhythm.
• Cleaning up in the console while Terraform or CloudFormation remains the source of truth, so the next apply puts it all back.
• Batching removals across many roles at once. When something breaks you have a bisect problem instead of a rollback.

Best practices for IAM Access Analyzer unused access cleanup

• Tag principals with their expected usage cadence at creation time. It costs nothing then and answers the hardest question later.
• Start with unused access keys and passwords on IAM users. Highest value, lowest risk removals in the whole list.
• Route findings to EventBridge and into your ticket queue so cleanup is tracked work rather than a periodic heroic sweep.
• Make the permissions change in code, apply it through the normal pipeline, and let the diff be the record.
• Keep an AccessDenied dashboard permanently, not just during a cleanup. Cheapest early warning you can build for this class of change.

Frequently asked questions

Does IAM Access Analyzer read my CloudTrail logs?

Not for unused access findings. Those come from IAM last accessed information. Policy generation is the feature that reads CloudTrail, and it is a separate operation run against a specific principal and time range.

What tracking period should I set?

Ninety days is reasonable for interactive human roles and continuously running workloads, and too short for anything on a quarterly or annual cycle. The value is configurable from 1 to 365 days. The right answer is a property of your workload, not a security best practice number.

Can a finding be wrong?

It can mislead in both directions. Permissions used only through data events can appear unused, and permissions that only ever produced denied attempts can appear used, because last accessed data counts all attempts rather than only successful ones.

Do I need an analyzer in every Region?

No. Unused access findings do not change based on Region, so one per account, or one at the organization level, is enough. External access analyzers are the ones with a Region dimension.

How is unused access analysis billed?

On the number of IAM roles and users analyzed per month, per analyzer. Scope is the cost lever, which is why exclusions matter for the bill and not just the noise. Check current published rates before enabling it organization-wide.

Should I delete unused roles or just strip their permissions?

Strip first, delete later, in two separate changes. A role with no permissions has almost no attack surface, and keeping it alive preserves any principal references pointing at it while you find out if anything still cares.

Conclusion

IAM Access Analyzer unused access findings answer a narrower question than the one you care about. They tell you what AWS did not record activity for. They do not tell you what your workload does not need, and they cannot tell you whether a permission that is used should be. The distance between those statements is where the outages live.

The one thing worth remembering: deny before you delete. An explicit deny gives you the exact behaviour of a removal with a one-call undo, and that single habit turns permission cleanup from a change you have to be brave about into one you can be systematic about.

Need help trimming IAM permissions without the outage

The risk in permission cleanup sits almost entirely in what the tooling cannot see. Things I can help with:

• Setting up account or organization-level unused access analyzers with tracking periods and tag-based exclusions that fit how your workloads run
• Building the AccessDenied observability layer, from CloudTrail queries through metric filters to a dashboard the team will actually look at
• Running staged deny-then-delete cleanups with the rollback path defined before anything changes
• Auditing roles for cross-account principal references, instance profiles, and IaC ownership before deletion
• Moving console-era IAM into Terraform or OpenTofu so the next cleanup is a reviewable diff

If you have a findings list you are not sure how to act on, send me the role’s policy JSON and the finding detail and I will tell you what I would check before touching it.

The post IAM Access Analyzer Unused Access: Removing Permissions Without Breaking Production appeared first on John Nessime.

That is the failure worth writing about. Drift itself is normal. Somebody fixes production at 2am, a managed service rotates a value, a policy engine stamps a tag. The part that breaks is the alert, and it breaks quietly, because a muted channel looks identical to a quiet one.

Switching on Terraform drift detection takes one flag. Building Terraform drift detection that engineers still read six months later means deciding what not to send: which plan mode you run, what you classify as noise, how you route by blast radius, and whether the message names who made the change. A detector firing daily with the same content is worse than none, because no detector leaves you knowingly blind while a muted one leaves you blind with a green dashboard. Every section below asks the same question differently: does this message change what a human does in the next ten minutes?

Pick the right plan mode or you will alert on your own backlog

This is the biggest source of false drift, and it catches almost everyone building their first check. A normal terraform plan compares three things: your configuration, your state, and the real infrastructure. It returns a non-zero exit code when any of them disagree, so it fires for real drift and just as loudly for a pull request that merged this morning and has not been applied. Your drift channel becomes a notification feed for your own deployment backlog, and people learn that most messages are work in progress.

Refresh-only mode narrows the question. HashiCorp documents it as a plan whose goal is only to update state and root module outputs to match changes made to remote objects outside of Terraform. That is the actual definition of drift.

terraform init -input=false

terraform plan -refresh-only -detailed-exitcode -lock=false -input=false -no-color

# exit 0  state matches the real infrastructure
# exit 1  the check itself failed
# exit 2  something changed outside Terraform

Two flags there matter more than they look. -detailed-exitcode is what makes the result machine-readable at all: without it a successful plan returns zero whether or not anything changed, so your job passes forever. And -lock=false is safe here specifically because a plan never writes state. Only terraform apply -refresh-only updates the state file. A read-only detector that grabs the state lock will collide with real applies, and the loser of that race is the engineer trying to ship.

The trade-off, stated plainly: refresh-only will not tell you that merged code has not shipped, and it will not find resources created outside Terraform, because it can only refresh what is already in state. I run both modes and route them separately. Refresh-only feeds the alert. Normal plan feeds a weekly digest tied to the repository, where unapplied changes are a delivery problem rather than a security one.

Classify the noise before you route anything

Most recurring drift falls into a few families. Sorting yours takes an hour and decides most of your routing rules.

Provider normalization

A JSON policy document comes back with keys reordered. An identifier returns in different case. An empty block reads back as null instead of an empty map. Terraform sees a difference because it compares a stored representation against an API response, and that round trip is not always lossless. Fix these in the configuration where you can, because a phantom diff that follows you into every plan also hides real diffs on the same resource.

Attributes another system legitimately owns

The classic is autoscaling capacity. Your config says two instances, a scaling policy moved it to five during a spike, and the next plan proposes taking it back to two. Applying that kills three instances mid-spike. The drift is real; reverting it is the wrong move.

resource "aws_autoscaling_group" "web" {
  min_size         = 2
  max_size         = 10
  desired_capacity = 2

  lifecycle {
    # Owned by the scaling policy after creation.
    # Reverting this during a spike removes live capacity.
    ignore_changes = [desired_capacity]
  }
}

Understand what you bought. ignore_changes stops Terraform proposing an update for that attribute, and it applies only after creation, not on the first apply. It also delegates the attribute permanently. If someone sets desired capacity to zero at 3am, your check will not raise a hand. Fair trade for capacity. Terrible trade for an ingress rule.

Tags stamped by something else

Organization tag policies, cost allocation tooling, backup schedulers and compliance agents all add tags. The AWS provider exposes both the tags you set and the effective set including defaults, and ignoring the effective set silences all tag drift at once. Convenient, and slightly dangerous if you use tags for access control, because tag-based policy conditions stop being watched. The same logic covers values the platform rotates or defaults on its own, like a managed secret version: they drift on a schedule that has nothing to do with your team, and belong in a digest.

The goal here is not a clean plan. A clean plan is a side effect. The goal is a decision per family: see it immediately, see it weekly, or never see it again.

Route by blast radius, not by count

“Drift detected: 12 resources changed” makes a human open a terminal before they know whether to care, and that friction is where attention goes to die. Split the output by what each resource can do if it is wrong:

• Wake someone. Anything governing exposure or access: security groups and firewall rules, IAM policies and roles, key policies, public access blocks, bucket policies, DNS records pointing at production.
• Review today. Instance sizes, scaling bounds, database parameters, retention settings, anything that costs money or changes capacity.
• Weekly digest. Tags, descriptions, names, and every family you classified as platform noise.

To split them you need structure, not text. Terraform’s machine-readable output emits a message for each resource that changed outside Terraform, carrying the resource address and the action it would take.

terraform plan -refresh-only -no-color -json 
  | jq -c 'select(.type == "resource_drift") | .change'

Check the exact field names inside that object against the machine-readable output reference for your Terraform version before building matching logic on them. Keep the severity mapping in the repository next to the code it describes. A resource type moving from digest to page should be a pull request, not something changed in a monitoring UI at midnight.

Make the alert answer “who did this”

Terraform tells you what changed. It has no idea who changed it, and that gap is where the investigation time goes. Half the drift you find is a legitimate incident fix nobody wrote down. If the alert names the actor, the response is a two-word reply in the thread instead of twenty minutes of archaeology. On AWS the audit trail can be queried by resource:

aws cloudtrail lookup-events 
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0123456789abcdef0

CloudTrail’s console event history only reaches back a limited window, so a weekly check may be hunting an event that has aged out; a trail delivering to S3, or CloudTrail Lake, covers the gap. Azure exposes the equivalent through Activity Logs, Google Cloud through Cloud Audit Logs. Wiring the lookup into the alert body does more for adoption than any amount of formatting.

Decide the remediation path before you automate it

There are three answers to a drift finding, and picking the wrong one can be worse than the drift.

1. Revert. Apply the configuration and push reality back. Right when the change was unauthorized or wrong.
2. Adopt. Change the code to match what is running. Right when the change was correct and the code was stale, which happens more than people admit.
3. Tolerate. Suppress the attribute deliberately, with a comment and an owner. Right when another system owns it.

The trap is automating option one. Auto-approving a remediation apply sounds like the natural endpoint of a drift pipeline, and it is fine right up until the drifted attribute is one that forces replacement. Then “restore the desired state” means destroying and recreating a live resource, unattended, at whatever hour the schedule runs. If you automate anyway, gate it hard: an allowlist of resource types, a check that the planned action is an in-place update rather than a replace, and a notification either way.

Troubleshooting a drift job that quietly stopped working

The nastiest failure mode is not a noisy check. It is a check reporting “no drift” because it never ran the plan successfully.

Exit code 2 kills your script

With set -e, a non-zero exit terminates the script. Exit code 2 means drift, which is the case you care about, so a naive script dies exactly when it has something to report. Capture the code without letting the shell act on it:

set -euo pipefail

terraform plan -refresh-only -detailed-exitcode -lock=false 
  -input=false -no-color > plan.txt 2>&1 && code=0 || code=$?

case "$code" in
  0) echo "no drift" ;;
  2) notify_drift plan.txt ;;
  *) notify_broken plan.txt; exit 1 ;;
esac

Note that the failure branch notifies too. Exit code 1 deserves the same urgency as drift, because a broken detector and a clean estate produce identical silence.

Credentials that rot without complaining

Drift checks run unattended, so nobody notices when a role trust policy changes, an OIDC subject claim stops matching after a repository rename, or a long-lived key gets rotated out. Add a deadman check: if the job has not reported within its expected interval, that absence is the alert. Same reasoning as heartbeat monitoring for backups.

A provider upgrade that drifts everything

Bump a provider and a new attribute default can appear across every resource of a type. Overnight the check reports hundreds of changed resources and whoever is on call assumes the tool is broken, which it sort of is. Commit the dependency lock file, pin the provider version, run the drift job on the same Terraform version as your apply pipeline, and treat provider upgrades as a reviewed change rather than something a bot merges.

Refresh times that outgrow the schedule

Refresh queries the API for every resource in state. Large workspaces get slow, then overlap their own schedule, then time out. The durable fix is splitting state along boundaries that already exist in your team structure. Terraform does offer targeting to narrow a plan, but it is documented as an exceptional-use option and prints a warning for good reason, so treat it as a stopgap rather than the design.

When to stop building this and buy it

A scheduled job in GitHub Actions or GitLab CI, or a systemd timer on a small always-on box from a provider like Contabo or InterServer, covers a handful of workspaces perfectly well and costs almost nothing. I would not reach for a platform before that stops working. What eventually stops working is the surrounding machinery: credentials for a dozen accounts across providers, concurrency against real applies, and retaining history so you can see that the same resource has drifted repeatedly this quarter, which is a process problem no single alert reveals.

That is where HCP Terraform’s health assessments, or a platform like Spacelift, Scalr or env0, starts earning its fee, since drift detection becomes a workspace setting with history and notifications attached rather than code you maintain. Two honest caveats: HCP Terraform gates health assessments behind its paid editions and requires remote or agent execution, so it is not a drop-in for local-state workflows, and any platform will happily send you the same undifferentiated wall of drift the homegrown script did. Severity tiering is a decision about your infrastructure that no vendor can make for you. Buying the runner does not buy the judgement.

Common mistakes

• Using normal plan mode for the alert. You notify yourself about unapplied work and teach people the channel is noise.
• Alerting on a count instead of a resource. “Nine resources changed” makes someone open a terminal to find out if they should care.
• Treating exit code 1 as a non-event. A detector that errors is indistinguishable from an estate with no drift.
• Taking the state lock for a read-only check. Monitoring should not be able to block a deployment.
• Auto-approving remediation across all resource types. One force-replacement attribute and the fix becomes the outage.

Best practices worth the effort

• Run refresh-only for drift and normal plan for unapplied changes, routed to different destinations.
• Keep the severity mapping in the repository, version controlled and reviewed like code.
• Give every suppression a comment naming the system that owns the attribute and a date to revisit.
• Alert on the detector failing and on the detector going quiet, not only on findings.
• Review recurring drift monthly. The same resource drifting repeatedly is a process gap, not an infrastructure one.

Terraform drift detection FAQ

What is the difference between a normal plan and refresh-only?

A normal plan compares configuration, state and reality, so it flags genuine drift and any committed change that has not been applied. Refresh-only compares state against reality only, which matches what most people mean by drift. Use refresh-only for the alert and normal plan for a delivery-focused report.

Does a refresh-only plan modify my state file?

No. Plan operations do not write state. Only applying a refresh-only plan updates the state file to match the drifted remote objects, and even then it makes no changes to infrastructure. That is why a detection job can safely skip the state lock.

How often should Terraform drift detection run?

Tie the interval to how long you would tolerate not knowing. Daily for production and anything holding customer data, weekly for lower environments, less often for slow-moving workspaces where refresh is expensive. Frequency matters less than whether the message gets read.

How do I find resources created outside Terraform?

Not with a plan. Terraform refreshes only what is already in state, so a resource it has never seen is invisible. You need a cloud-side inventory such as AWS Config, Azure Resource Graph or Cloud Asset Inventory, compared against your state, and import blocks to bring the results under management. Worth knowing that driftctl, the open-source tool most often recommended here, has moved into maintenance mode.

Does ignore_changes hide drift I actually care about?

Yes, and that is the point of using it. It stops Terraform proposing updates for that attribute regardless of what changed it, so a deliberate suppression and a malicious change look identical afterwards. Scope it to specific attributes, never whole resources you still care about, and document the owner.

Why did drift suddenly appear across hundreds of resources?

Almost always a provider or Terraform version change rather than something happening in your cloud account. A new provider release can add or reshape an attribute and every resource of that type reports a difference at once. Compare the versions used by the drift runner and the apply pipeline before investigating the infrastructure.

The one thing worth keeping

Terraform drift detection is not a monitoring problem, it is an attention problem. The flag that finds drift took an afternoon. Everything determining whether anyone still reads the output next quarter is judgement: refresh-only rather than normal plan, noise families sorted and suppressed deliberately, severity by blast radius rather than resource count, an actor name in the message, and an alert when the detector itself goes dark.

Judge it by one test: when a message lands in that channel, does anyone open it? If the honest answer is no, your detection is working and your alert is not.

Need help making your drift alerts worth reading?

This is common work once a Terraform estate grows past one team. Things I can help with:

• Auditing an existing drift job and finding the false positives that trained everyone to ignore it
• Building scheduled refresh-only checks in GitHub Actions, GitLab CI or a systemd timer, with correct exit code handling and a deadman alert
• Sorting recurring drift into families and deciding, per attribute, what to fix, adopt or suppress
• Building severity routing from the plan’s structured output so exposure changes page and tag changes do not
• Enriching alerts with CloudTrail, Activity Log or Cloud Audit Log lookups so findings arrive with an actor attached
• Splitting oversized state so refresh finishes inside its schedule, and judging whether a managed platform is worth it at your size

If you want a second opinion, send me a week of your drift output or the workflow file that produces it, and I will tell you what I would cut first.

The post Terraform Drift Detection That Nobody Mutes appeared first on John Nessime.

That failure should shape how you think about EKS Pod Identity vs IRSA. Both deliver the same end result to your application: short-lived, rotated credentials with nothing hardcoded. The differences are in how trust is expressed, who has to touch IAM to change it, and what happens when the wiring is subtly wrong.

This post compares the two on what actually decides the choice: how each scales across clusters, where each refuses to work at all, the failures each hides, and a decision procedure you can apply to a real cluster.

Same outcome, two very different trust shapes

Most of the trade-offs fall out of the mechanism, so it is worth being precise about both.

IRSA: federation through an OIDC provider

The cluster has an OIDC issuer, which you register as an identity provider in IAM. A projected service account token is mounted into the pod, and the SDK exchanges it for credentials through the web identity flow against STS. The role’s trust policy is what makes that work, and it names one specific cluster:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub":
          "system:serviceaccount:production:my-app",
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud":
          "sts.amazonaws.com"
      }
    }
  }]
}

The important detail is that the issuer URL is unique per cluster. The relationship between a workload and a role lives inside IAM, and it is cluster-specific by construction.

Pod Identity: an EKS API and an on-node agent

Pod Identity moves that mapping out of IAM. You install the EKS Pod Identity Agent add-on, which runs as a DaemonSet, then create an association through the EKS API binding a namespace plus service account name to a role. The agent calls the EKS Auth API for temporary credentials, and the pod picks them up through the container credential provider that AWS SDKs already support.

The trust policy contains nothing cluster-specific:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
    "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": [ "sts:AssumeRole", "sts:TagSession" ]
  }]
}

sts:TagSession is not decoration. Pod Identity attaches session tags on every assume, so a policy allowing only sts:AssumeRole fails. It is the most common first-attempt error.

Creating the association is a single call:

aws eks create-pod-identity-association 
  --cluster-name prod-eu 
  --namespace production 
  --service-account my-app 
  --role-arn arn:aws:iam::111122223333:role/my-app

Where IRSA still wins

Treating IRSA as legacy is the most common mistake here. For some workloads it is not the older option, it is the only option.

• Fargate pods. The Pod Identity Agent is a DaemonSet on EC2 nodes. Fargate has no node for it to land on, and AWS documents Fargate pods as unsupported. If part of your workload runs on Fargate, IRSA is the mechanism there, permanently, not as a stopgap.
• Windows nodes. Also documented as unsupported for Pod Identity.
• Older SDKs you cannot rebuild. The web identity flow has been in the SDKs far longer, so a vendor image you cannot patch is more likely to work with IRSA.
• Direct federation into workload accounts. If your model wants the pod’s identity federating straight into a role in the resource-owning account, IRSA does that natively. Pod Identity reaches other accounts by role chaining, a different trust shape even when the outcome looks the same.
• You already have it working at a scale you control. A few clusters with a Terraform module that provisions the provider and the roles is not a problem that needs solving.

Where IRSA hurts

The pain is administrative, and it grows with cluster count rather than workload count. Every new cluster brings a new OIDC issuer, which means editing the trust policy of every role those workloads use. Trust policies have a size limit, so a role trusting many issuers eventually runs out of room and you end up duplicating roles to work around it.

The same asymmetry shows up in blue/green cluster upgrades. Standing a new cluster beside the old one means every role used by every migrating workload needs a trust policy edit before the new cluster can do anything, and another edit afterwards to clean up. That is IAM work on the critical path of an upgrade, usually owned by a different team than the one doing it.

Where Pod Identity wins

• Roles become portable. One role, one generic trust policy, any number of clusters. Adding a cluster is an EKS API call per workload, not an IAM edit per role.
• The permission split matches how teams are organised. Creating an association needs EKS permissions, not iam:UpdateAssumeRolePolicy, so cluster admins stop filing IAM tickets for every new service account.
• Session tags arrive for free. Every assume carries eks-cluster-arn, eks-cluster-name, kubernetes-namespace, kubernetes-service-account, kubernetes-pod-name and kubernetes-pod-uid. That is usable attribute-based access control and the CloudTrail audit trail you always wanted.
• Cross-account access without touching application code. An association can carry a target role ARN, and EKS performs the role chaining for you.

The ABAC angle is underrated. One policy can serve dozens of workloads, because the session tag identifies the caller:

"Condition": {
  "StringEquals": {
    "s3:ExistingObjectTag/team": "${aws:PrincipalTag/kubernetes-namespace}"
  }
}

Where Pod Identity hurts

The generic trust policy is a real trade-off. Written exactly as AWS shows it, that role can be assumed on behalf of any service account in any namespace in any cluster in the account, as long as someone with EKS permissions creates an association pointing at it. Blast radius control has moved from IAM into the EKS API. If those permissions are loosely held, you widened it.

The fix is to put the constraint back in the trust policy using the request tags Pod Identity sends:

"Condition": {
  "StringEquals": {
    "aws:RequestTag/kubernetes-namespace": "production",
    "aws:RequestTag/kubernetes-service-account": "my-app"
  }
}

You can also pin to a cluster with aws:SourceArn. That costs some of the portability you switched for, which is the point: decide per role. A role reading one team’s bucket should be pinned. A role used by an identical workload in six clusters should not.

The other cost is a new moving part. The agent is a DaemonSet, so it needs to tolerate your taints, and it must be running before pods on that node can get credentials.

The failures that actually bite

Silent fallback to the node role

This is the one from the opening, and it is invisible by design. Pod Identity works through the container credential provider, which the SDK finds by reading two environment variables EKS injects into the pod:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

An SDK that predates that provider does not error on these. It does not know what they mean, so it keeps walking the credential chain until it reaches instance metadata and picks up the node role. The application works. The permissions are wrong. Nothing complains.

Two defences, and you want both. Check the minimum SDK version list in the EKS documentation and pin base images above it, and block pod access to IMDS so falling back has nowhere to fall. The first prevents the problem; the second turns a silent success into a loud failure.

Verify from inside the pod. The ARN in the response tells you which identity is really in use:

kubectl exec -n production deploy/my-app -- aws sts get-caller-identity

The association that matches nothing

An association is an exact match on cluster, namespace and service account name. A typo in any of the three produces an association that exists, looks fine in the console, and never applies to a pod. List them and read them against your manifests:

aws eks list-pod-identity-associations --cluster-name prod-eu

Associations also only take effect for pods started after they exist. If you create one and nothing changes, restart the workload before you start debugging IAM.

Both mechanisms configured at once

During a migration you will often have a service account carrying an IRSA annotation and a Pod Identity association at the same time. Both paths can work, and which one the SDK uses depends on credential chain ordering. AWS documents that credentials found earlier in the chain keep being used even when an association exists.

Do not reason about it. Check the caller identity and check CloudTrail, then remove the one you are not keeping so nobody has to work it out again.

PackedPolicyTooLarge

Session tags, managed policy ARNs and inline session policies are compressed into a packed format with its own size limit, and six session tags on every assume is not free. The association can disable the automatic tags, at the cost of the ABAC and audit benefits that were half the reason to use Pod Identity.

A decision procedure for EKS Pod Identity vs IRSA

Work through these in order and stop at the first one that answers.

1. Does this workload run on Fargate or Windows nodes? IRSA. There is no decision to make.
2. Can you rebuild the image with a current SDK? If not, IRSA, or fix the image first.
3. Does your model require direct OIDC federation into the resource account? IRSA. Role chaining is not the same shape, even if it reaches the same bucket.
4. Will this role be used from more than one cluster, now or after the next upgrade? Pod Identity. The saving compounds every time you add or replace a cluster.
5. Do you want cluster and namespace in the CloudTrail record without extra work? Pod Identity, because of the session tags.
6. None of the above? Pod Identity for anything new, IRSA left alone where it already works.

AWS’s own guidance lands in the same place: Pod Identity for new applications on supported node types, IRSA where you already have it working or where Pod Identity cannot run.

Arguments that don’t survive contact

• “IRSA is deprecated.” It is not. AWS has said it continues to invest in it, and Fargate support alone guarantees it stays.
• “Pod Identity is less secure because the trust policy is generic.” Only if you leave it generic. Request tag conditions scope it as tightly as an IRSA trust policy, and you keep the session tags IRSA does not give you.
• “Run one mechanism per cluster, cleanly.” A mixed cluster is documented, supported, and for clusters with Fargate pods it is the permanent steady state, not a transitional mess.
• “EKS add-ons can only use IRSA.” True early on, and still repeated in older write-ups. Add-on creation and update now accept Pod Identity associations directly. Check your specific add-on rather than trusting a blog post, including this one.
• “Migrate everything, then delete the OIDC provider.” Deleting it breaks every role still trusting it, including ones you forgot. Migrate workload by workload, and delete the provider only after CloudTrail shows zero AssumeRoleWithWebIdentity calls for that issuer across a full batch cycle.

How I’d decide, and what I’d do first

• Default new workloads on EC2 or Karpenter-managed nodes to Pod Identity, and leave working IRSA alone until you have a reason to touch it.
• Add request tag conditions for anything sensitive. Portability helps a shared platform role and hurts a role that reads customer data.
• Restrict pod access to IMDS before you migrate anything, not after. It converts the silent failure into a visible one.
• Migrate with the role trusting both principals at once, so rollback is a manifest change rather than an IAM change.
• Put credential identity in your dashboards. A CloudTrail query counting calls made by node roles from workload namespaces catches the exact failure this post opened with. Amazon Managed Grafana or Grafana Cloud will render it if you already ship CloudTrail somewhere; a scheduled CLI check from a CI runner or a small VPS from Contabo or InterServer is a fine low-tech alternative.
• Codify associations in Terraform or OpenTofu from the start. They are cheap to create by hand and easy to lose track of.

Frequently asked questions

Is EKS Pod Identity replacing IRSA?

No. AWS recommends Pod Identity for new applications on supported node types and continues to support IRSA. Since Pod Identity cannot run on Fargate or Windows nodes, IRSA stays for the workloads that live there.

Can I use both in the same cluster?

Yes, and it is a documented, supported pattern. Older workloads on IRSA and new ones on Pod Identity is normal. Avoid configuring both for the same service account beyond a migration window, because which one wins depends on SDK credential chain ordering.

Why does my pod get AccessDenied after switching to Pod Identity?

Check the trust policy for sts:TagSession first. Pod Identity always sends session tags, so a trust policy allowing only sts:AssumeRole is rejected. After that, check that the namespace and service account in the association match the pod exactly.

Does Pod Identity work for cross-account access?

Yes, through role chaining. The association takes a role in the cluster account plus a target role in the resource account, and EKS chains them. It also exposes an external ID you can require in the target role’s trust policy to prevent a confused deputy problem.

Do I have to change my application code?

No, provided the SDK is recent enough. Both work through the default credential chain, so the application calls AWS as usual. The only change that bites is an SDK too old to understand the container credential provider.

How do I confirm which identity a pod is actually using?

Run aws sts get-caller-identity inside the container and read the assumed role ARN. If it names the node role rather than the one you assigned, the chain fell through. Cross-check in CloudTrail, where session tags also give you the namespace and service account for Pod Identity sessions.

The one thing worth remembering

The real question in EKS Pod Identity vs IRSA is not which is more modern. It is where you want the cluster-to-role relationship to live. IRSA puts it in IAM trust policies: precise, cluster-specific, and increasing administrative drag as clusters multiply. Pod Identity puts it in the EKS API: portable, fast to change, and it hands you the job of deciding per role how much of that portability you actually want.

Pick per workload, not per cluster. And whichever you pick, verify the identity from inside the pod, because the worst outcome here is not a permission error. It is a workload that has been running with the wrong credentials since the day you deployed it.

Need a second pair of eyes on your EKS workload identity?

Most of this work is unglamorous: reading trust policies, matching associations against manifests, proving what a pod is really authenticating as. Things I help with:

• Auditing every service account in a cluster and reporting which identity its pods actually use, not the one the manifest claims
• Planning and running an IRSA to Pod Identity migration workload by workload, with a rollback that needs no IAM change
• Writing scoped trust policies with request tag conditions, so portability is a deliberate choice per role
• Restricting pod access to IMDS and confirming nothing was silently depending on the node role
• Setting up cross-account access with target roles and external IDs, plus the CloudTrail view that proves it is used as intended
• Codifying associations and roles in Terraform or OpenTofu, so the next cluster is a variable change rather than a project

Send me a trust policy, a service account manifest, or the output of aws sts get-caller-identity from inside one of your pods, and I will tell you what it is really doing.

The post EKS Pod Identity vs IRSA: What Actually Decides the Choice appeared first on John Nessime.