That is the shape of most EKS upgrade incidents. The part everyone books a maintenance window for, the control plane bump, is the part AWS handles and almost never goes wrong. The damage happens in the data plane rollout, because of decisions made months earlier that nobody connected to upgrades.

This post covers the three failure families that actually break EKS version upgrades: removed APIs you did not know you were calling, add-on updates that silently discard your configuration, and pod disruption budgets that deadlock the drain. For each one: what it looks like, how to catch it beforehand, and what the escape hatch costs.

Why the control plane is the easy half

An EKS cluster has two versions that must stay compatible. AWS manages the control plane; you manage the data plane, which in practice means the kubelet version on your nodes.

Two constraints shape everything else. In-place upgrades move one minor version at a time, so being three versions behind means three sequential upgrades and three data plane rollouts. And the kubelet cannot run ahead of the API server, so once the control plane moves, the clock starts on getting nodes to follow.

Failure one: the API calls you did not know you were making

Deprecation and removal are different events, and the gap between them is where people get comfortable. An API version gets deprecated, everything keeps working, the warning scrolls past in CI, and several releases later the version is gone and whatever depended on it starts getting 404s.

Your own manifests are usually fine; someone fixed those when the warning appeared. What breaks is the third-party agent nobody owns: a metrics collector, an ingress controller, a Datadog or New Relic agent installed by a Helm chart pinned two years ago. These call the API on a watch loop, so they hit removed APIs immediately, and they tend to fail quietly rather than crash-loop.

Cluster insights, and what it is actually reading

aws eks list-insights --cluster-name my-cluster

aws eks describe-insight 
    --cluster-name my-cluster 
    --id a1b2c3d4-5678-90ab-cdef-EXAMPLE11111

EKS runs upgrade readiness checks and exposes them through the insights API. The deprecated API check reads your control plane audit logs on a thirty-day rolling window, scanned about once a day. That has a consequence people trip over constantly: after you fix a deprecated call, the insight stays in ERROR until the last matching log entry ages out. The work is done, the check still fails, and if you gate automation on insight status your pipeline is blocked for weeks on a solved problem. Force a fresh scan rather than waiting for the daily one:

aws eks start-insights-refresh --cluster-name my-cluster
aws eks describe-insights-refresh --cluster-name my-cluster

The refresh re-reads the window; it does not shorten it. Audit-log detection also only sees APIs that were actually called, so a monthly CronJob or a chart sitting unapplied in a repo never appears. Insights tells you what your cluster did, not what it could do.

Ask the API server directly

The API server keeps its own counter, and it names the release that removes each API. Fastest check available:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Read the removed_release label first: it tells you whether this is urgent for the next upgrade or a warning for the one after. Scrape the metric into Prometheus, Grafana Cloud or Amazon Managed Grafana and you get a standing signal instead of a scramble.

For the callers, go to the audit logs. With control plane audit logging on, CloudWatch Logs Insights gives you the identity behind each deprecated call:

fields @message
| filter `annotations.k8s.io/deprecated` = "true"

The user.username field is what you need. It usually points at a service account in a namespace you did not install, which is your cue to find the chart’s owner. Static scanners like kubent and pluto cover the other half, the manifests that have not run recently. Use both.

Failure two: add-ons and the conflict resolution you picked once

Managed add-ons cover what a cluster cannot run without: the VPC CNI, CoreDNS, kube-proxy, the EBS CSI driver. They are version-pinned separately from the cluster, and their compatible ranges move with the Kubernetes version. The failure here is rarely the version, though. It is the conflict resolution mode.

When you update an add-on, EKS wants to apply its own field values. If you edited those fields in-cluster, that is a conflict, resolved by what you pass:

• NONE leaves your value alone and the update may simply fail. Safe, loud, and it stops your upgrade.
• OVERWRITE reverts your customization to the EKS default. The update succeeds. Your custom Corefile, your CNI tuning, your tolerations, gone, with no error.
• PRESERVE keeps your value through the update.

OVERWRITE is the one that hurts, because it is also the one people reach for when an update fails and they want it to stop failing. It is the standard fix when migrating a self-managed add-on to a managed one, so it gets learned there and carried into the update path, where it means something different. A CoreDNS update with OVERWRITE will replace a Corefile carrying your stub zones and forwarders, and resolution for those zones stops cluster-wide while every health check stays green.

aws eks update-addon 
    --cluster-name my-cluster 
    --addon-name coredns 
    --addon-version <compatible-version> 
    --resolve-conflicts PRESERVE

One more trap: the same value means different things on create and on update. On create, PRESERVE behaves close to NONE, leaving an existing self-managed add-on alone rather than adopting your settings. Check the operation, not the word.

The durable fix is to stop editing add-on resources with kubectl at all. Put customizations in the add-on’s configuration values so EKS owns them and there is nothing left to conflict over. An afternoon of work, and this failure family goes away permanently.

Failure three: PDB deadlocks and the fifteen-minute drain clock

This is the one that stalls upgrades halfway, and the mechanics matter because the fix depends on where you are stuck. A managed node group update runs in four phases: setup, scale up, upgrade, scale down. On the default strategy, scale up launches replacements before anything is terminated, which is what preserves capacity. Then the upgrade phase does this, per node:

1. Pick a node to replace, up to the configured maxUnavailable.
2. Cordon it once a new node reaches Ready.
3. Drain the pods off it.
4. Wait sixty seconds after eviction, then ask the Auto Scaling group to terminate it.
5. Repeat until nothing is left on the old launch template version.

Step three has a fifteen minute budget. If pods have not left in that time and you did not pass the force flag, the whole update fails with PodEvictionFailure and you are left with a half-migrated node group.

The deadlock nobody sees coming

A PDB says how many pods must stay available during voluntary disruption, and eviction is voluntary disruption. So the arithmetic is simple and, once you see it, obvious: a deployment with two replicas and a budget requiring two available pods can never be evicted. Not slowly. Never. The drain retries until the clock runs out.

One replica with a one-pod minimum is the same deadlock in friendlier clothing, as is a percentage budget that rounds up to the full replica count. And if two PDBs select the same pod, the more restrictive one wins, producing deadlocks invisible in either manifest read alone. The pre-flight check is cheap, and I run it before every upgrade:

kubectl get pdb --all-namespaces

Read the ALLOWED DISRUPTIONS column, not the configured minimum. Any budget sitting at zero is a drain that will hang. Fix those before touching the node group, by raising replicas or loosening the budget so at least one disruption is permitted.

There is a second cause of PodEvictionFailure with nothing to do with PDBs. The upgrade taints nodes as unschedulable, so after eviction the node should be empty. Anything tolerating every taint keeps it non-empty, and the drain never satisfies its exit condition. If your budgets look fine and eviction still stalls, look at what tolerates NoSchedule unconditionally.

What the force flag actually costs

aws eks update-nodegroup-version 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup 
    --force

Force does not negotiate with budgets. It deletes the pods and terminates the node regardless. For a stateless service behind a load balancer with several replicas, that is fine and the upgrade completes. For a quorum-based system it is a bad idea: if the budget in front of your etcd, Consul, Kafka or ZooKeeper pods is doing its job, forcing past it can take the cluster below quorum. The budget was the protection; force is the decision to ignore it.

My rule is that force is reasonable on a node group I know holds only stateless workloads, and never a first response on a mixed one. Reaching for it because you do not know what is on the node is the actual problem.

If you run Karpenter, the levers are different

Karpenter has no update phases. It detects drift when a NodePool or EC2NodeClass stops matching a node’s configuration, then replaces the drifted nodes. Disruption budgets control the pace and can be scheduled, which is how you build a maintenance window without booking one:

spec:
  disruption:
    budgets:
      - nodes: "0"
        reasons: ["Drifted"]
        schedule: "0 6 * * mon-fri"
        duration: 12h

Two things bite people. Schedules are evaluated in UTC, so a window that looks like business hours to you may not be. And any period not covered by a budget is unrestricted, so gaps are not “no disruption”, they are “unlimited disruption”.

The other lever, terminationGracePeriod on the NodePool, overrides everything. Once it elapses, remaining pods are force-deleted regardless of PDBs or the karpenter.sh/do-not-disrupt annotation. It is a deadlock breaker, not a safety setting, and if it is shorter than your budget schedule you can get forced terminations inside a window you thought was protected.

The rollback window changes the calculation

Upstream Kubernetes has no control plane rollback. That single fact is why so many teams built heavyweight approval processes around upgrades. EKS now supports reverting an in-place control plane upgrade to the previous minor version within a seven day window, preserving etcd data and running workloads. It covers in-place upgrades only, which makes sense: with blue/green the old cluster is still there and rollback is a DNS change.

What changes is the shape of the risk. An upgrade used to be a one-way door, so the rational response was exhaustive testing beforehand. Now the first week is the test, under real traffic, with a way back. That argues for upgrading more often on smaller jumps, and treating the week after as an active watch period rather than a closed ticket.

It is not a general undo. Once the window closes, or you upgrade again, the previous version is gone, and it does not roll back add-ons or application state. Blue/green still has a place when you are several versions behind, since sequential in-place hops multiply exposure. The cost is running two clusters during migration, worth modelling first in Cost Explorer with both tagged, or in a tool like Vantage or CloudZero if you already run one.

Troubleshooting a stalled or failed upgrade

When a node group update fails, the reason comes back as an error code. Start there rather than guessing:

aws eks describe-nodegroup 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup

• PodEvictionFailure. A budget blocked the drain, or something tolerates all taints. Check ALLOWED DISRUPTIONS first, then tolerations.
• NodeCreationFailure in scale up. New nodes never reached Ready. Usually no capacity for your instance type in an Availability Zone, an EC2 instance limit in the account, or custom user data breaking the bootstrap.
• Slow bootstrap. A node has fifteen minutes to join, measured from when the Auto Scaling group size increases. Heavy user data, large images pulled at boot or a slow init path push past it.
• Pods stuck terminating with no budget in sight. Look for blocked I/O. A pod with an NFS mount whose server is unreachable will not terminate, and eviction pressure changes nothing.
• Insights still in ERROR after the fix. The thirty day audit window. Refresh, confirm another way, move on.

One structural note: if the Cluster Autoscaler scales the node group during the scale down phase, the upgrade workflow exits without restoring the group to its original size. Check your desired and max counts afterwards.

Common mistakes

• Treating insights as the only pre-flight check. It reads audit logs, so it cannot see code paths that have not run.
• Reaching for OVERWRITE to make a failing add-on update succeed. It succeeds by discarding your configuration.
• Upgrading add-ons and the control plane in one unbroken pipeline run, so you cannot tell which change broke things.
• Leaving maxUnavailable at its default of one on a large node group, then wondering why the rollout takes hours.
• Forgetting self-managed components. Cluster Autoscaler, the AWS Load Balancer Controller and metrics-server are versioned against Kubernetes too, and nothing reminds you.

How I sequence EKS version upgrades

1. Read the release notes for the target version. Not the summary post, the removals section.
2. Run all three deprecation checks: insights for what the cluster did, the API server metric for what it does now, a static scanner for what sits in the repo.
3. Audit PDBs and fix every zero. Highest-value ten minutes in the process.
4. Update add-ons with PRESERVE to versions compatible with both the current and target Kubernetes version, so components talking to the API server are ready before it moves.
5. Upgrade the control plane and stop. Watch error rates and API server latency long enough to mean something.
6. Roll the least critical node group. Confirm pods reschedule cleanly and nothing depended on a node label or an AMI-specific detail.
7. Roll the rest, confirm every kubelet matches the control plane, then treat the following week as the real test. That is what the rollback window is for.

FAQ

Can you do EKS version upgrades with zero downtime?

For stateless workloads with sensible replica counts, spread across zones, with budgets allowing at least one disruption, yes. The default node group strategy launches replacements before terminating anything, so capacity holds throughout. Stateful workloads can too, but only if the application handles rolling leadership changes and the budget permits exactly one disruption at a time.

Why does the update fail with PodEvictionFailure when my PDBs look fine?

Two likely causes. Multiple budgets may select the same pod, in which case the most restrictive applies and neither manifest reveals it alone. Or something on the node tolerates every taint, so the node never becomes empty and the drain never completes. Check ALLOWED DISRUPTIONS across all namespaces, then check tolerations.

Is the force update option safe?

Safe for stateless workloads with multiple replicas. Not safe for quorum-based systems, because it deletes pods regardless of the budget protecting quorum. Use it when you know what is on the node group and the answer is nothing that minds being killed.

Does Karpenter make upgrades easier than managed node groups?

More automatic, which is not the same thing. Drift detection replaces outdated nodes without you running a command, but you inherit a different set of controls to get wrong: budgets evaluated in UTC, uncovered schedule gaps that mean unlimited disruption, and a grace period that overrides PDBs entirely. Neither is easier. They fail differently.

The one thing worth remembering

EKS version upgrades do not fail at the moment you run the command. They fail on decisions already sitting in the cluster: a budget written for a two-replica deployment, an add-on customized with kubectl rather than through its configuration values, a chart pinned before an API was removed.

Which means the pre-flight checks are the upgrade. Ten minutes reading ALLOWED DISRUPTIONS and the deprecated API metric tells you more about how tomorrow goes than any amount of care during the rollout. The rollback window gives you a second chance on the control plane. It will not un-delete a pod that force-eviction pulled out of a quorum.

Need a second pair of eyes on your upgrade path?

Most of this work is unglamorous and specific. What I would actually do on a cluster facing a version bump:

• Audit every PDB against its replica count and flag the ones that will deadlock a drain.
• Trace deprecated API calls back to the chart, controller or service account making them, from audit logs rather than guesswork.
• Move add-on customizations into managed configuration values, so conflict resolution stops being a decision made under pressure.
• Write a repeatable upgrade runbook: pre-flight checks, sequencing, and the rollback path for your specific workloads.
• Alert on the deprecated API metric and node group update events, so the next version is a scheduled task rather than an incident.
• Review Karpenter disruption budgets and grace periods for the gaps that turn into unplanned rollouts.

If you are staring at a failed node group update right now, send me the output of kubectl get pdb --all-namespaces and the error code from describe-nodegroup. That is usually enough to say what is holding the drain.

The post EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks appeared first on John Nessime.

That failure should shape how you think about EKS Pod Identity vs IRSA. Both deliver the same end result to your application: short-lived, rotated credentials with nothing hardcoded. The differences are in how trust is expressed, who has to touch IAM to change it, and what happens when the wiring is subtly wrong.

This post compares the two on what actually decides the choice: how each scales across clusters, where each refuses to work at all, the failures each hides, and a decision procedure you can apply to a real cluster.

Same outcome, two very different trust shapes

Most of the trade-offs fall out of the mechanism, so it is worth being precise about both.

IRSA: federation through an OIDC provider

The cluster has an OIDC issuer, which you register as an identity provider in IAM. A projected service account token is mounted into the pod, and the SDK exchanges it for credentials through the web identity flow against STS. The role’s trust policy is what makes that work, and it names one specific cluster:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub":
          "system:serviceaccount:production:my-app",
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud":
          "sts.amazonaws.com"
      }
    }
  }]
}

The important detail is that the issuer URL is unique per cluster. The relationship between a workload and a role lives inside IAM, and it is cluster-specific by construction.

Pod Identity: an EKS API and an on-node agent

Pod Identity moves that mapping out of IAM. You install the EKS Pod Identity Agent add-on, which runs as a DaemonSet, then create an association through the EKS API binding a namespace plus service account name to a role. The agent calls the EKS Auth API for temporary credentials, and the pod picks them up through the container credential provider that AWS SDKs already support.

The trust policy contains nothing cluster-specific:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
    "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": [ "sts:AssumeRole", "sts:TagSession" ]
  }]
}

sts:TagSession is not decoration. Pod Identity attaches session tags on every assume, so a policy allowing only sts:AssumeRole fails. It is the most common first-attempt error.

Creating the association is a single call:

aws eks create-pod-identity-association 
  --cluster-name prod-eu 
  --namespace production 
  --service-account my-app 
  --role-arn arn:aws:iam::111122223333:role/my-app

Where IRSA still wins

Treating IRSA as legacy is the most common mistake here. For some workloads it is not the older option, it is the only option.

• Fargate pods. The Pod Identity Agent is a DaemonSet on EC2 nodes. Fargate has no node for it to land on, and AWS documents Fargate pods as unsupported. If part of your workload runs on Fargate, IRSA is the mechanism there, permanently, not as a stopgap.
• Windows nodes. Also documented as unsupported for Pod Identity.
• Older SDKs you cannot rebuild. The web identity flow has been in the SDKs far longer, so a vendor image you cannot patch is more likely to work with IRSA.
• Direct federation into workload accounts. If your model wants the pod’s identity federating straight into a role in the resource-owning account, IRSA does that natively. Pod Identity reaches other accounts by role chaining, a different trust shape even when the outcome looks the same.
• You already have it working at a scale you control. A few clusters with a Terraform module that provisions the provider and the roles is not a problem that needs solving.

Where IRSA hurts

The pain is administrative, and it grows with cluster count rather than workload count. Every new cluster brings a new OIDC issuer, which means editing the trust policy of every role those workloads use. Trust policies have a size limit, so a role trusting many issuers eventually runs out of room and you end up duplicating roles to work around it.

The same asymmetry shows up in blue/green cluster upgrades. Standing a new cluster beside the old one means every role used by every migrating workload needs a trust policy edit before the new cluster can do anything, and another edit afterwards to clean up. That is IAM work on the critical path of an upgrade, usually owned by a different team than the one doing it.

Where Pod Identity wins

• Roles become portable. One role, one generic trust policy, any number of clusters. Adding a cluster is an EKS API call per workload, not an IAM edit per role.
• The permission split matches how teams are organised. Creating an association needs EKS permissions, not iam:UpdateAssumeRolePolicy, so cluster admins stop filing IAM tickets for every new service account.
• Session tags arrive for free. Every assume carries eks-cluster-arn, eks-cluster-name, kubernetes-namespace, kubernetes-service-account, kubernetes-pod-name and kubernetes-pod-uid. That is usable attribute-based access control and the CloudTrail audit trail you always wanted.
• Cross-account access without touching application code. An association can carry a target role ARN, and EKS performs the role chaining for you.

The ABAC angle is underrated. One policy can serve dozens of workloads, because the session tag identifies the caller:

"Condition": {
  "StringEquals": {
    "s3:ExistingObjectTag/team": "${aws:PrincipalTag/kubernetes-namespace}"
  }
}

Where Pod Identity hurts

The generic trust policy is a real trade-off. Written exactly as AWS shows it, that role can be assumed on behalf of any service account in any namespace in any cluster in the account, as long as someone with EKS permissions creates an association pointing at it. Blast radius control has moved from IAM into the EKS API. If those permissions are loosely held, you widened it.

The fix is to put the constraint back in the trust policy using the request tags Pod Identity sends:

"Condition": {
  "StringEquals": {
    "aws:RequestTag/kubernetes-namespace": "production",
    "aws:RequestTag/kubernetes-service-account": "my-app"
  }
}

You can also pin to a cluster with aws:SourceArn. That costs some of the portability you switched for, which is the point: decide per role. A role reading one team’s bucket should be pinned. A role used by an identical workload in six clusters should not.

The other cost is a new moving part. The agent is a DaemonSet, so it needs to tolerate your taints, and it must be running before pods on that node can get credentials.

The failures that actually bite

Silent fallback to the node role

This is the one from the opening, and it is invisible by design. Pod Identity works through the container credential provider, which the SDK finds by reading two environment variables EKS injects into the pod:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

An SDK that predates that provider does not error on these. It does not know what they mean, so it keeps walking the credential chain until it reaches instance metadata and picks up the node role. The application works. The permissions are wrong. Nothing complains.

Two defences, and you want both. Check the minimum SDK version list in the EKS documentation and pin base images above it, and block pod access to IMDS so falling back has nowhere to fall. The first prevents the problem; the second turns a silent success into a loud failure.

Verify from inside the pod. The ARN in the response tells you which identity is really in use:

kubectl exec -n production deploy/my-app -- aws sts get-caller-identity

The association that matches nothing

An association is an exact match on cluster, namespace and service account name. A typo in any of the three produces an association that exists, looks fine in the console, and never applies to a pod. List them and read them against your manifests:

aws eks list-pod-identity-associations --cluster-name prod-eu

Associations also only take effect for pods started after they exist. If you create one and nothing changes, restart the workload before you start debugging IAM.

Both mechanisms configured at once

During a migration you will often have a service account carrying an IRSA annotation and a Pod Identity association at the same time. Both paths can work, and which one the SDK uses depends on credential chain ordering. AWS documents that credentials found earlier in the chain keep being used even when an association exists.

Do not reason about it. Check the caller identity and check CloudTrail, then remove the one you are not keeping so nobody has to work it out again.

PackedPolicyTooLarge

Session tags, managed policy ARNs and inline session policies are compressed into a packed format with its own size limit, and six session tags on every assume is not free. The association can disable the automatic tags, at the cost of the ABAC and audit benefits that were half the reason to use Pod Identity.

A decision procedure for EKS Pod Identity vs IRSA

Work through these in order and stop at the first one that answers.

1. Does this workload run on Fargate or Windows nodes? IRSA. There is no decision to make.
2. Can you rebuild the image with a current SDK? If not, IRSA, or fix the image first.
3. Does your model require direct OIDC federation into the resource account? IRSA. Role chaining is not the same shape, even if it reaches the same bucket.
4. Will this role be used from more than one cluster, now or after the next upgrade? Pod Identity. The saving compounds every time you add or replace a cluster.
5. Do you want cluster and namespace in the CloudTrail record without extra work? Pod Identity, because of the session tags.
6. None of the above? Pod Identity for anything new, IRSA left alone where it already works.

AWS’s own guidance lands in the same place: Pod Identity for new applications on supported node types, IRSA where you already have it working or where Pod Identity cannot run.

Arguments that don’t survive contact

• “IRSA is deprecated.” It is not. AWS has said it continues to invest in it, and Fargate support alone guarantees it stays.
• “Pod Identity is less secure because the trust policy is generic.” Only if you leave it generic. Request tag conditions scope it as tightly as an IRSA trust policy, and you keep the session tags IRSA does not give you.
• “Run one mechanism per cluster, cleanly.” A mixed cluster is documented, supported, and for clusters with Fargate pods it is the permanent steady state, not a transitional mess.
• “EKS add-ons can only use IRSA.” True early on, and still repeated in older write-ups. Add-on creation and update now accept Pod Identity associations directly. Check your specific add-on rather than trusting a blog post, including this one.
• “Migrate everything, then delete the OIDC provider.” Deleting it breaks every role still trusting it, including ones you forgot. Migrate workload by workload, and delete the provider only after CloudTrail shows zero AssumeRoleWithWebIdentity calls for that issuer across a full batch cycle.

How I’d decide, and what I’d do first

• Default new workloads on EC2 or Karpenter-managed nodes to Pod Identity, and leave working IRSA alone until you have a reason to touch it.
• Add request tag conditions for anything sensitive. Portability helps a shared platform role and hurts a role that reads customer data.
• Restrict pod access to IMDS before you migrate anything, not after. It converts the silent failure into a visible one.
• Migrate with the role trusting both principals at once, so rollback is a manifest change rather than an IAM change.
• Put credential identity in your dashboards. A CloudTrail query counting calls made by node roles from workload namespaces catches the exact failure this post opened with. Amazon Managed Grafana or Grafana Cloud will render it if you already ship CloudTrail somewhere; a scheduled CLI check from a CI runner or a small VPS from Contabo or InterServer is a fine low-tech alternative.
• Codify associations in Terraform or OpenTofu from the start. They are cheap to create by hand and easy to lose track of.

Frequently asked questions

Is EKS Pod Identity replacing IRSA?

No. AWS recommends Pod Identity for new applications on supported node types and continues to support IRSA. Since Pod Identity cannot run on Fargate or Windows nodes, IRSA stays for the workloads that live there.

Can I use both in the same cluster?

Yes, and it is a documented, supported pattern. Older workloads on IRSA and new ones on Pod Identity is normal. Avoid configuring both for the same service account beyond a migration window, because which one wins depends on SDK credential chain ordering.

Why does my pod get AccessDenied after switching to Pod Identity?

Check the trust policy for sts:TagSession first. Pod Identity always sends session tags, so a trust policy allowing only sts:AssumeRole is rejected. After that, check that the namespace and service account in the association match the pod exactly.

Does Pod Identity work for cross-account access?

Yes, through role chaining. The association takes a role in the cluster account plus a target role in the resource account, and EKS chains them. It also exposes an external ID you can require in the target role’s trust policy to prevent a confused deputy problem.

Do I have to change my application code?

No, provided the SDK is recent enough. Both work through the default credential chain, so the application calls AWS as usual. The only change that bites is an SDK too old to understand the container credential provider.

How do I confirm which identity a pod is actually using?

Run aws sts get-caller-identity inside the container and read the assumed role ARN. If it names the node role rather than the one you assigned, the chain fell through. Cross-check in CloudTrail, where session tags also give you the namespace and service account for Pod Identity sessions.

The one thing worth remembering

The real question in EKS Pod Identity vs IRSA is not which is more modern. It is where you want the cluster-to-role relationship to live. IRSA puts it in IAM trust policies: precise, cluster-specific, and increasing administrative drag as clusters multiply. Pod Identity puts it in the EKS API: portable, fast to change, and it hands you the job of deciding per role how much of that portability you actually want.

Pick per workload, not per cluster. And whichever you pick, verify the identity from inside the pod, because the worst outcome here is not a permission error. It is a workload that has been running with the wrong credentials since the day you deployed it.

Need a second pair of eyes on your EKS workload identity?

Most of this work is unglamorous: reading trust policies, matching associations against manifests, proving what a pod is really authenticating as. Things I help with:

• Auditing every service account in a cluster and reporting which identity its pods actually use, not the one the manifest claims
• Planning and running an IRSA to Pod Identity migration workload by workload, with a rollback that needs no IAM change
• Writing scoped trust policies with request tag conditions, so portability is a deliberate choice per role
• Restricting pod access to IMDS and confirming nothing was silently depending on the node role
• Setting up cross-account access with target roles and external IDs, plus the CloudTrail view that proves it is used as intended
• Codifying associations and roles in Terraform or OpenTofu, so the next cluster is a variable change rather than a project

Send me a trust policy, a service account manifest, or the output of aws sts get-caller-identity from inside one of your pods, and I will tell you what it is really doing.

The post EKS Pod Identity vs IRSA: What Actually Decides the Choice appeared first on John Nessime.