That is the shape of most EKS upgrade incidents. The part everyone books a maintenance window for, the control plane bump, is the part AWS handles and almost never goes wrong. The damage happens in the data plane rollout, because of decisions made months earlier that nobody connected to upgrades.

This post covers the three failure families that actually break EKS version upgrades: removed APIs you did not know you were calling, add-on updates that silently discard your configuration, and pod disruption budgets that deadlock the drain. For each one: what it looks like, how to catch it beforehand, and what the escape hatch costs.

Why the control plane is the easy half

An EKS cluster has two versions that must stay compatible. AWS manages the control plane; you manage the data plane, which in practice means the kubelet version on your nodes.

Two constraints shape everything else. In-place upgrades move one minor version at a time, so being three versions behind means three sequential upgrades and three data plane rollouts. And the kubelet cannot run ahead of the API server, so once the control plane moves, the clock starts on getting nodes to follow.

Failure one: the API calls you did not know you were making

Deprecation and removal are different events, and the gap between them is where people get comfortable. An API version gets deprecated, everything keeps working, the warning scrolls past in CI, and several releases later the version is gone and whatever depended on it starts getting 404s.

Your own manifests are usually fine; someone fixed those when the warning appeared. What breaks is the third-party agent nobody owns: a metrics collector, an ingress controller, a Datadog or New Relic agent installed by a Helm chart pinned two years ago. These call the API on a watch loop, so they hit removed APIs immediately, and they tend to fail quietly rather than crash-loop.

Cluster insights, and what it is actually reading

aws eks list-insights --cluster-name my-cluster

aws eks describe-insight 
    --cluster-name my-cluster 
    --id a1b2c3d4-5678-90ab-cdef-EXAMPLE11111

EKS runs upgrade readiness checks and exposes them through the insights API. The deprecated API check reads your control plane audit logs on a thirty-day rolling window, scanned about once a day. That has a consequence people trip over constantly: after you fix a deprecated call, the insight stays in ERROR until the last matching log entry ages out. The work is done, the check still fails, and if you gate automation on insight status your pipeline is blocked for weeks on a solved problem. Force a fresh scan rather than waiting for the daily one:

aws eks start-insights-refresh --cluster-name my-cluster
aws eks describe-insights-refresh --cluster-name my-cluster

The refresh re-reads the window; it does not shorten it. Audit-log detection also only sees APIs that were actually called, so a monthly CronJob or a chart sitting unapplied in a repo never appears. Insights tells you what your cluster did, not what it could do.

Ask the API server directly

The API server keeps its own counter, and it names the release that removes each API. Fastest check available:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Read the removed_release label first: it tells you whether this is urgent for the next upgrade or a warning for the one after. Scrape the metric into Prometheus, Grafana Cloud or Amazon Managed Grafana and you get a standing signal instead of a scramble.

For the callers, go to the audit logs. With control plane audit logging on, CloudWatch Logs Insights gives you the identity behind each deprecated call:

fields @message
| filter `annotations.k8s.io/deprecated` = "true"

The user.username field is what you need. It usually points at a service account in a namespace you did not install, which is your cue to find the chart’s owner. Static scanners like kubent and pluto cover the other half, the manifests that have not run recently. Use both.

Failure two: add-ons and the conflict resolution you picked once

Managed add-ons cover what a cluster cannot run without: the VPC CNI, CoreDNS, kube-proxy, the EBS CSI driver. They are version-pinned separately from the cluster, and their compatible ranges move with the Kubernetes version. The failure here is rarely the version, though. It is the conflict resolution mode.

When you update an add-on, EKS wants to apply its own field values. If you edited those fields in-cluster, that is a conflict, resolved by what you pass:

• NONE leaves your value alone and the update may simply fail. Safe, loud, and it stops your upgrade.
• OVERWRITE reverts your customization to the EKS default. The update succeeds. Your custom Corefile, your CNI tuning, your tolerations, gone, with no error.
• PRESERVE keeps your value through the update.

OVERWRITE is the one that hurts, because it is also the one people reach for when an update fails and they want it to stop failing. It is the standard fix when migrating a self-managed add-on to a managed one, so it gets learned there and carried into the update path, where it means something different. A CoreDNS update with OVERWRITE will replace a Corefile carrying your stub zones and forwarders, and resolution for those zones stops cluster-wide while every health check stays green.

aws eks update-addon 
    --cluster-name my-cluster 
    --addon-name coredns 
    --addon-version <compatible-version> 
    --resolve-conflicts PRESERVE

One more trap: the same value means different things on create and on update. On create, PRESERVE behaves close to NONE, leaving an existing self-managed add-on alone rather than adopting your settings. Check the operation, not the word.

The durable fix is to stop editing add-on resources with kubectl at all. Put customizations in the add-on’s configuration values so EKS owns them and there is nothing left to conflict over. An afternoon of work, and this failure family goes away permanently.

Failure three: PDB deadlocks and the fifteen-minute drain clock

This is the one that stalls upgrades halfway, and the mechanics matter because the fix depends on where you are stuck. A managed node group update runs in four phases: setup, scale up, upgrade, scale down. On the default strategy, scale up launches replacements before anything is terminated, which is what preserves capacity. Then the upgrade phase does this, per node:

1. Pick a node to replace, up to the configured maxUnavailable.
2. Cordon it once a new node reaches Ready.
3. Drain the pods off it.
4. Wait sixty seconds after eviction, then ask the Auto Scaling group to terminate it.
5. Repeat until nothing is left on the old launch template version.

Step three has a fifteen minute budget. If pods have not left in that time and you did not pass the force flag, the whole update fails with PodEvictionFailure and you are left with a half-migrated node group.

The deadlock nobody sees coming

A PDB says how many pods must stay available during voluntary disruption, and eviction is voluntary disruption. So the arithmetic is simple and, once you see it, obvious: a deployment with two replicas and a budget requiring two available pods can never be evicted. Not slowly. Never. The drain retries until the clock runs out.

One replica with a one-pod minimum is the same deadlock in friendlier clothing, as is a percentage budget that rounds up to the full replica count. And if two PDBs select the same pod, the more restrictive one wins, producing deadlocks invisible in either manifest read alone. The pre-flight check is cheap, and I run it before every upgrade:

kubectl get pdb --all-namespaces

Read the ALLOWED DISRUPTIONS column, not the configured minimum. Any budget sitting at zero is a drain that will hang. Fix those before touching the node group, by raising replicas or loosening the budget so at least one disruption is permitted.

There is a second cause of PodEvictionFailure with nothing to do with PDBs. The upgrade taints nodes as unschedulable, so after eviction the node should be empty. Anything tolerating every taint keeps it non-empty, and the drain never satisfies its exit condition. If your budgets look fine and eviction still stalls, look at what tolerates NoSchedule unconditionally.

What the force flag actually costs

aws eks update-nodegroup-version 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup 
    --force

Force does not negotiate with budgets. It deletes the pods and terminates the node regardless. For a stateless service behind a load balancer with several replicas, that is fine and the upgrade completes. For a quorum-based system it is a bad idea: if the budget in front of your etcd, Consul, Kafka or ZooKeeper pods is doing its job, forcing past it can take the cluster below quorum. The budget was the protection; force is the decision to ignore it.

My rule is that force is reasonable on a node group I know holds only stateless workloads, and never a first response on a mixed one. Reaching for it because you do not know what is on the node is the actual problem.

If you run Karpenter, the levers are different

Karpenter has no update phases. It detects drift when a NodePool or EC2NodeClass stops matching a node’s configuration, then replaces the drifted nodes. Disruption budgets control the pace and can be scheduled, which is how you build a maintenance window without booking one:

spec:
  disruption:
    budgets:
      - nodes: "0"
        reasons: ["Drifted"]
        schedule: "0 6 * * mon-fri"
        duration: 12h

Two things bite people. Schedules are evaluated in UTC, so a window that looks like business hours to you may not be. And any period not covered by a budget is unrestricted, so gaps are not “no disruption”, they are “unlimited disruption”.

The other lever, terminationGracePeriod on the NodePool, overrides everything. Once it elapses, remaining pods are force-deleted regardless of PDBs or the karpenter.sh/do-not-disrupt annotation. It is a deadlock breaker, not a safety setting, and if it is shorter than your budget schedule you can get forced terminations inside a window you thought was protected.

The rollback window changes the calculation

Upstream Kubernetes has no control plane rollback. That single fact is why so many teams built heavyweight approval processes around upgrades. EKS now supports reverting an in-place control plane upgrade to the previous minor version within a seven day window, preserving etcd data and running workloads. It covers in-place upgrades only, which makes sense: with blue/green the old cluster is still there and rollback is a DNS change.

What changes is the shape of the risk. An upgrade used to be a one-way door, so the rational response was exhaustive testing beforehand. Now the first week is the test, under real traffic, with a way back. That argues for upgrading more often on smaller jumps, and treating the week after as an active watch period rather than a closed ticket.

It is not a general undo. Once the window closes, or you upgrade again, the previous version is gone, and it does not roll back add-ons or application state. Blue/green still has a place when you are several versions behind, since sequential in-place hops multiply exposure. The cost is running two clusters during migration, worth modelling first in Cost Explorer with both tagged, or in a tool like Vantage or CloudZero if you already run one.

Troubleshooting a stalled or failed upgrade

When a node group update fails, the reason comes back as an error code. Start there rather than guessing:

aws eks describe-nodegroup 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup

• PodEvictionFailure. A budget blocked the drain, or something tolerates all taints. Check ALLOWED DISRUPTIONS first, then tolerations.
• NodeCreationFailure in scale up. New nodes never reached Ready. Usually no capacity for your instance type in an Availability Zone, an EC2 instance limit in the account, or custom user data breaking the bootstrap.
• Slow bootstrap. A node has fifteen minutes to join, measured from when the Auto Scaling group size increases. Heavy user data, large images pulled at boot or a slow init path push past it.
• Pods stuck terminating with no budget in sight. Look for blocked I/O. A pod with an NFS mount whose server is unreachable will not terminate, and eviction pressure changes nothing.
• Insights still in ERROR after the fix. The thirty day audit window. Refresh, confirm another way, move on.

One structural note: if the Cluster Autoscaler scales the node group during the scale down phase, the upgrade workflow exits without restoring the group to its original size. Check your desired and max counts afterwards.

Common mistakes

• Treating insights as the only pre-flight check. It reads audit logs, so it cannot see code paths that have not run.
• Reaching for OVERWRITE to make a failing add-on update succeed. It succeeds by discarding your configuration.
• Upgrading add-ons and the control plane in one unbroken pipeline run, so you cannot tell which change broke things.
• Leaving maxUnavailable at its default of one on a large node group, then wondering why the rollout takes hours.
• Forgetting self-managed components. Cluster Autoscaler, the AWS Load Balancer Controller and metrics-server are versioned against Kubernetes too, and nothing reminds you.

How I sequence EKS version upgrades

1. Read the release notes for the target version. Not the summary post, the removals section.
2. Run all three deprecation checks: insights for what the cluster did, the API server metric for what it does now, a static scanner for what sits in the repo.
3. Audit PDBs and fix every zero. Highest-value ten minutes in the process.
4. Update add-ons with PRESERVE to versions compatible with both the current and target Kubernetes version, so components talking to the API server are ready before it moves.
5. Upgrade the control plane and stop. Watch error rates and API server latency long enough to mean something.
6. Roll the least critical node group. Confirm pods reschedule cleanly and nothing depended on a node label or an AMI-specific detail.
7. Roll the rest, confirm every kubelet matches the control plane, then treat the following week as the real test. That is what the rollback window is for.

FAQ

Can you do EKS version upgrades with zero downtime?

For stateless workloads with sensible replica counts, spread across zones, with budgets allowing at least one disruption, yes. The default node group strategy launches replacements before terminating anything, so capacity holds throughout. Stateful workloads can too, but only if the application handles rolling leadership changes and the budget permits exactly one disruption at a time.

Why does the update fail with PodEvictionFailure when my PDBs look fine?

Two likely causes. Multiple budgets may select the same pod, in which case the most restrictive applies and neither manifest reveals it alone. Or something on the node tolerates every taint, so the node never becomes empty and the drain never completes. Check ALLOWED DISRUPTIONS across all namespaces, then check tolerations.

Is the force update option safe?

Safe for stateless workloads with multiple replicas. Not safe for quorum-based systems, because it deletes pods regardless of the budget protecting quorum. Use it when you know what is on the node group and the answer is nothing that minds being killed.

Does Karpenter make upgrades easier than managed node groups?

More automatic, which is not the same thing. Drift detection replaces outdated nodes without you running a command, but you inherit a different set of controls to get wrong: budgets evaluated in UTC, uncovered schedule gaps that mean unlimited disruption, and a grace period that overrides PDBs entirely. Neither is easier. They fail differently.

The one thing worth remembering

EKS version upgrades do not fail at the moment you run the command. They fail on decisions already sitting in the cluster: a budget written for a two-replica deployment, an add-on customized with kubectl rather than through its configuration values, a chart pinned before an API was removed.

Which means the pre-flight checks are the upgrade. Ten minutes reading ALLOWED DISRUPTIONS and the deprecated API metric tells you more about how tomorrow goes than any amount of care during the rollout. The rollback window gives you a second chance on the control plane. It will not un-delete a pod that force-eviction pulled out of a quorum.

Need a second pair of eyes on your upgrade path?

Most of this work is unglamorous and specific. What I would actually do on a cluster facing a version bump:

• Audit every PDB against its replica count and flag the ones that will deadlock a drain.
• Trace deprecated API calls back to the chart, controller or service account making them, from audit logs rather than guesswork.
• Move add-on customizations into managed configuration values, so conflict resolution stops being a decision made under pressure.
• Write a repeatable upgrade runbook: pre-flight checks, sequencing, and the rollback path for your specific workloads.
• Alert on the deprecated API metric and node group update events, so the next version is a scheduled task rather than an incident.
• Review Karpenter disruption budgets and grace periods for the gaps that turn into unplanned rollouts.

If you are staring at a failed node group update right now, send me the output of kubectl get pdb --all-namespaces and the error code from describe-nodegroup. That is usually enough to say what is holding the drain.

The post EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks appeared first on John Nessime.

That failure should shape how you think about EKS Pod Identity vs IRSA. Both deliver the same end result to your application: short-lived, rotated credentials with nothing hardcoded. The differences are in how trust is expressed, who has to touch IAM to change it, and what happens when the wiring is subtly wrong.

This post compares the two on what actually decides the choice: how each scales across clusters, where each refuses to work at all, the failures each hides, and a decision procedure you can apply to a real cluster.

Same outcome, two very different trust shapes

Most of the trade-offs fall out of the mechanism, so it is worth being precise about both.

IRSA: federation through an OIDC provider

The cluster has an OIDC issuer, which you register as an identity provider in IAM. A projected service account token is mounted into the pod, and the SDK exchanges it for credentials through the web identity flow against STS. The role’s trust policy is what makes that work, and it names one specific cluster:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub":
          "system:serviceaccount:production:my-app",
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud":
          "sts.amazonaws.com"
      }
    }
  }]
}

The important detail is that the issuer URL is unique per cluster. The relationship between a workload and a role lives inside IAM, and it is cluster-specific by construction.

Pod Identity: an EKS API and an on-node agent

Pod Identity moves that mapping out of IAM. You install the EKS Pod Identity Agent add-on, which runs as a DaemonSet, then create an association through the EKS API binding a namespace plus service account name to a role. The agent calls the EKS Auth API for temporary credentials, and the pod picks them up through the container credential provider that AWS SDKs already support.

The trust policy contains nothing cluster-specific:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
    "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": [ "sts:AssumeRole", "sts:TagSession" ]
  }]
}

sts:TagSession is not decoration. Pod Identity attaches session tags on every assume, so a policy allowing only sts:AssumeRole fails. It is the most common first-attempt error.

Creating the association is a single call:

aws eks create-pod-identity-association 
  --cluster-name prod-eu 
  --namespace production 
  --service-account my-app 
  --role-arn arn:aws:iam::111122223333:role/my-app

Where IRSA still wins

Treating IRSA as legacy is the most common mistake here. For some workloads it is not the older option, it is the only option.

• Fargate pods. The Pod Identity Agent is a DaemonSet on EC2 nodes. Fargate has no node for it to land on, and AWS documents Fargate pods as unsupported. If part of your workload runs on Fargate, IRSA is the mechanism there, permanently, not as a stopgap.
• Windows nodes. Also documented as unsupported for Pod Identity.
• Older SDKs you cannot rebuild. The web identity flow has been in the SDKs far longer, so a vendor image you cannot patch is more likely to work with IRSA.
• Direct federation into workload accounts. If your model wants the pod’s identity federating straight into a role in the resource-owning account, IRSA does that natively. Pod Identity reaches other accounts by role chaining, a different trust shape even when the outcome looks the same.
• You already have it working at a scale you control. A few clusters with a Terraform module that provisions the provider and the roles is not a problem that needs solving.

Where IRSA hurts

The pain is administrative, and it grows with cluster count rather than workload count. Every new cluster brings a new OIDC issuer, which means editing the trust policy of every role those workloads use. Trust policies have a size limit, so a role trusting many issuers eventually runs out of room and you end up duplicating roles to work around it.

The same asymmetry shows up in blue/green cluster upgrades. Standing a new cluster beside the old one means every role used by every migrating workload needs a trust policy edit before the new cluster can do anything, and another edit afterwards to clean up. That is IAM work on the critical path of an upgrade, usually owned by a different team than the one doing it.

Where Pod Identity wins

• Roles become portable. One role, one generic trust policy, any number of clusters. Adding a cluster is an EKS API call per workload, not an IAM edit per role.
• The permission split matches how teams are organised. Creating an association needs EKS permissions, not iam:UpdateAssumeRolePolicy, so cluster admins stop filing IAM tickets for every new service account.
• Session tags arrive for free. Every assume carries eks-cluster-arn, eks-cluster-name, kubernetes-namespace, kubernetes-service-account, kubernetes-pod-name and kubernetes-pod-uid. That is usable attribute-based access control and the CloudTrail audit trail you always wanted.
• Cross-account access without touching application code. An association can carry a target role ARN, and EKS performs the role chaining for you.

The ABAC angle is underrated. One policy can serve dozens of workloads, because the session tag identifies the caller:

"Condition": {
  "StringEquals": {
    "s3:ExistingObjectTag/team": "${aws:PrincipalTag/kubernetes-namespace}"
  }
}

Where Pod Identity hurts

The generic trust policy is a real trade-off. Written exactly as AWS shows it, that role can be assumed on behalf of any service account in any namespace in any cluster in the account, as long as someone with EKS permissions creates an association pointing at it. Blast radius control has moved from IAM into the EKS API. If those permissions are loosely held, you widened it.

The fix is to put the constraint back in the trust policy using the request tags Pod Identity sends:

"Condition": {
  "StringEquals": {
    "aws:RequestTag/kubernetes-namespace": "production",
    "aws:RequestTag/kubernetes-service-account": "my-app"
  }
}

You can also pin to a cluster with aws:SourceArn. That costs some of the portability you switched for, which is the point: decide per role. A role reading one team’s bucket should be pinned. A role used by an identical workload in six clusters should not.

The other cost is a new moving part. The agent is a DaemonSet, so it needs to tolerate your taints, and it must be running before pods on that node can get credentials.

The failures that actually bite

Silent fallback to the node role

This is the one from the opening, and it is invisible by design. Pod Identity works through the container credential provider, which the SDK finds by reading two environment variables EKS injects into the pod:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

An SDK that predates that provider does not error on these. It does not know what they mean, so it keeps walking the credential chain until it reaches instance metadata and picks up the node role. The application works. The permissions are wrong. Nothing complains.

Two defences, and you want both. Check the minimum SDK version list in the EKS documentation and pin base images above it, and block pod access to IMDS so falling back has nowhere to fall. The first prevents the problem; the second turns a silent success into a loud failure.

Verify from inside the pod. The ARN in the response tells you which identity is really in use:

kubectl exec -n production deploy/my-app -- aws sts get-caller-identity

The association that matches nothing

An association is an exact match on cluster, namespace and service account name. A typo in any of the three produces an association that exists, looks fine in the console, and never applies to a pod. List them and read them against your manifests:

aws eks list-pod-identity-associations --cluster-name prod-eu

Associations also only take effect for pods started after they exist. If you create one and nothing changes, restart the workload before you start debugging IAM.

Both mechanisms configured at once

During a migration you will often have a service account carrying an IRSA annotation and a Pod Identity association at the same time. Both paths can work, and which one the SDK uses depends on credential chain ordering. AWS documents that credentials found earlier in the chain keep being used even when an association exists.

Do not reason about it. Check the caller identity and check CloudTrail, then remove the one you are not keeping so nobody has to work it out again.

PackedPolicyTooLarge

Session tags, managed policy ARNs and inline session policies are compressed into a packed format with its own size limit, and six session tags on every assume is not free. The association can disable the automatic tags, at the cost of the ABAC and audit benefits that were half the reason to use Pod Identity.

A decision procedure for EKS Pod Identity vs IRSA

Work through these in order and stop at the first one that answers.

1. Does this workload run on Fargate or Windows nodes? IRSA. There is no decision to make.
2. Can you rebuild the image with a current SDK? If not, IRSA, or fix the image first.
3. Does your model require direct OIDC federation into the resource account? IRSA. Role chaining is not the same shape, even if it reaches the same bucket.
4. Will this role be used from more than one cluster, now or after the next upgrade? Pod Identity. The saving compounds every time you add or replace a cluster.
5. Do you want cluster and namespace in the CloudTrail record without extra work? Pod Identity, because of the session tags.
6. None of the above? Pod Identity for anything new, IRSA left alone where it already works.

AWS’s own guidance lands in the same place: Pod Identity for new applications on supported node types, IRSA where you already have it working or where Pod Identity cannot run.

Arguments that don’t survive contact

• “IRSA is deprecated.” It is not. AWS has said it continues to invest in it, and Fargate support alone guarantees it stays.
• “Pod Identity is less secure because the trust policy is generic.” Only if you leave it generic. Request tag conditions scope it as tightly as an IRSA trust policy, and you keep the session tags IRSA does not give you.
• “Run one mechanism per cluster, cleanly.” A mixed cluster is documented, supported, and for clusters with Fargate pods it is the permanent steady state, not a transitional mess.
• “EKS add-ons can only use IRSA.” True early on, and still repeated in older write-ups. Add-on creation and update now accept Pod Identity associations directly. Check your specific add-on rather than trusting a blog post, including this one.
• “Migrate everything, then delete the OIDC provider.” Deleting it breaks every role still trusting it, including ones you forgot. Migrate workload by workload, and delete the provider only after CloudTrail shows zero AssumeRoleWithWebIdentity calls for that issuer across a full batch cycle.

How I’d decide, and what I’d do first

• Default new workloads on EC2 or Karpenter-managed nodes to Pod Identity, and leave working IRSA alone until you have a reason to touch it.
• Add request tag conditions for anything sensitive. Portability helps a shared platform role and hurts a role that reads customer data.
• Restrict pod access to IMDS before you migrate anything, not after. It converts the silent failure into a visible one.
• Migrate with the role trusting both principals at once, so rollback is a manifest change rather than an IAM change.
• Put credential identity in your dashboards. A CloudTrail query counting calls made by node roles from workload namespaces catches the exact failure this post opened with. Amazon Managed Grafana or Grafana Cloud will render it if you already ship CloudTrail somewhere; a scheduled CLI check from a CI runner or a small VPS from Contabo or InterServer is a fine low-tech alternative.
• Codify associations in Terraform or OpenTofu from the start. They are cheap to create by hand and easy to lose track of.

Frequently asked questions

Is EKS Pod Identity replacing IRSA?

No. AWS recommends Pod Identity for new applications on supported node types and continues to support IRSA. Since Pod Identity cannot run on Fargate or Windows nodes, IRSA stays for the workloads that live there.

Can I use both in the same cluster?

Yes, and it is a documented, supported pattern. Older workloads on IRSA and new ones on Pod Identity is normal. Avoid configuring both for the same service account beyond a migration window, because which one wins depends on SDK credential chain ordering.

Why does my pod get AccessDenied after switching to Pod Identity?

Check the trust policy for sts:TagSession first. Pod Identity always sends session tags, so a trust policy allowing only sts:AssumeRole is rejected. After that, check that the namespace and service account in the association match the pod exactly.

Does Pod Identity work for cross-account access?

Yes, through role chaining. The association takes a role in the cluster account plus a target role in the resource account, and EKS chains them. It also exposes an external ID you can require in the target role’s trust policy to prevent a confused deputy problem.

Do I have to change my application code?

No, provided the SDK is recent enough. Both work through the default credential chain, so the application calls AWS as usual. The only change that bites is an SDK too old to understand the container credential provider.

How do I confirm which identity a pod is actually using?

Run aws sts get-caller-identity inside the container and read the assumed role ARN. If it names the node role rather than the one you assigned, the chain fell through. Cross-check in CloudTrail, where session tags also give you the namespace and service account for Pod Identity sessions.

The one thing worth remembering

The real question in EKS Pod Identity vs IRSA is not which is more modern. It is where you want the cluster-to-role relationship to live. IRSA puts it in IAM trust policies: precise, cluster-specific, and increasing administrative drag as clusters multiply. Pod Identity puts it in the EKS API: portable, fast to change, and it hands you the job of deciding per role how much of that portability you actually want.

Pick per workload, not per cluster. And whichever you pick, verify the identity from inside the pod, because the worst outcome here is not a permission error. It is a workload that has been running with the wrong credentials since the day you deployed it.

Need a second pair of eyes on your EKS workload identity?

Most of this work is unglamorous: reading trust policies, matching associations against manifests, proving what a pod is really authenticating as. Things I help with:

• Auditing every service account in a cluster and reporting which identity its pods actually use, not the one the manifest claims
• Planning and running an IRSA to Pod Identity migration workload by workload, with a rollback that needs no IAM change
• Writing scoped trust policies with request tag conditions, so portability is a deliberate choice per role
• Restricting pod access to IMDS and confirming nothing was silently depending on the node role
• Setting up cross-account access with target roles and external IDs, plus the CloudTrail view that proves it is used as intended
• Codifying associations and roles in Terraform or OpenTofu, so the next cluster is a variable change rather than a project

Send me a trust policy, a service account manifest, or the output of aws sts get-caller-identity from inside one of your pods, and I will tell you what it is really doing.

The post EKS Pod Identity vs IRSA: What Actually Decides the Choice appeared first on John Nessime.

That’s the signature failure of telemetry pipelines on AWS. Transport worked, authentication worked, the endpoint answered 200. It kept part of the payload and dropped the rest, because something in the batch broke a limit your collector has no idea exists.

Getting started with OpenTelemetry on AWS is not the hard part. AWS Distro for OpenTelemetry (ADOT) will have you shipping traces in an afternoon. The hard part is four decisions underneath it: where the collector sits, which door telemetry uses to get into AWS, what happens to your metrics on the way through, and whether a second backend later costs you a config line or a migration. This post covers those four decisions and how each one fails without announcing itself.

What ADOT actually buys you

ADOT is not a separate protocol or agent. It’s a downstream build of the upstream OpenTelemetry Collector plus AWS-flavoured SDKs, tested by AWS and covered by AWS Support. The config syntax is the upstream syntax, so a config written for vanilla OTel runs on ADOT and vice versa. The difference is which components are compiled in: ADOT ships the X-Ray exporter, the CloudWatch EMF exporter, the SigV4 extension, and ECS metric receivers already there.

So the choice is narrower than it looks. Take ADOT for a build someone else validates and supports. Take upstream Contrib if you need a component ADOT hasn’t bundled, or you run one collector image across AWS, another cloud, and bare metal. Neither choice touches your instrumentation, which is the entire point.

Decision one: where the collector runs

Three shapes, trading the same three things: blast radius, cost, and how much processing happens before data leaves your network.

Sidecar

One collector container per ECS task or pod. The application talks to localhost, so no service discovery and no network hop to get wrong, and failure stays contained to one workload. The cost: you pay for that container everywhere, batches stay small because batching is per-instance, and any config change is a redeploy of every task.

Agent plus gateway

A light collector per node (a DaemonSet on EKS) forwarding to a small pool of gateways. This is the shape I reach for first on anything past a handful of services. The agent does host-level enrichment, the gateway does the expensive work: large batches, tail sampling, redaction, fan-out. Config changes hit the gateway only.

The catch: a gateway pool is a thing you now operate. It needs autoscaling and its own alerting, and undersized it starts refusing data at exactly the moment you have an incident and volume spikes.

No collector at all

The ADOT SDKs can export straight to the CloudWatch OTLP endpoints, signing requests with credentials already available to the process. For a handful of Lambda functions this is the right answer: nothing to run, nothing to scale, no extra container in the cold start path. What you give up is control. Batching, retries, sampling policy, redaction, and backend routing all move into application processes and their environment variables, so changing any of them is a fleet redeploy rather than a config push. Good starting point, poor steady state.

Decision two: which door into AWS

Two ways to hand telemetry to AWS, and they behave very differently.

The older path uses AWS-specific exporters. awsxray converts OTLP spans into X-Ray segment documents and calls the X-Ray API. awsemf converts OTLP metrics into CloudWatch Embedded Metric Format and writes them as log events. Both work, both are well trodden, and both reshape your data into an AWS-native format on the way out.

The newer path is native OTLP. CloudWatch exposes OTLP endpoints per signal, reached with the plain otlphttp exporter and a SigV4 signer. Same exporter you’d point at Grafana Cloud or Honeycomb, different URL and authenticator. That’s what “export anywhere” means in practice, and it’s why I default to it on new work.

The endpoints follow a per-service pattern:

• Traces: https://xray.<region>.amazonaws.com/v1/traces
• Metrics: https://monitoring.<region>.amazonaws.com/v1/metrics
• Logs: https://logs.<region>.amazonaws.com/v1/logs

Three constraints there will each cost you an afternoon if you don’t know them going in.

HTTP only. No gRPC. If your collector exports over otlp on 4317 you can’t just change the URL, you have to switch exporters. Receiving gRPC from applications is fine, it’s the outbound leg that must be HTTP.

SigV4 required. That means the sigv4auth extension, configured per signal because the signing service name differs: xray, monitoring, and logs respectively. Bearer tokens are an alternative for metrics and logs, but not for traces.

Logs need headers, not just a URL. Target log group and stream travel in x-aws-log-group and x-aws-log-stream headers. Omit them and the request has nowhere to land.

A working shape for traces and logs looks like this:

extensions:
  sigv4auth/traces:
    region: "us-east-1"
    service: "xray"
  sigv4auth/logs:
    region: "us-east-1"
    service: "logs"

exporters:
  otlphttp/traces:
    compression: gzip
    traces_endpoint: https://xray.us-east-1.amazonaws.com/v1/traces
    auth:
      authenticator: sigv4auth/traces

  otlphttp/logs:
    compression: gzip
    logs_endpoint: https://logs.us-east-1.amazonaws.com/v1/logs
    headers:
      x-aws-log-group: MyApplicationLogs
      x-aws-log-stream: default
    auth:
      authenticator: sigv4auth/logs

service:
  extensions: [sigv4auth/traces, sigv4auth/logs]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/traces]
    logs:
      receivers: [otlp]
      exporters: [otlphttp/logs]

Note traces_endpoint and logs_endpoint rather than endpoint. This one bites people. The endpoint setting is a base URL and the exporter appends the signal path itself, so endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics actually requests /v1/metrics/v1/metrics. The per-signal settings take a full path. If you’re getting 404s from a URL you’re certain is right, this is why.

Also worth a comment in your config: upstream has renamed the component to otlp_http and otlphttp is now a deprecated alias scheduled for removal. Both work today.

Traces need Transaction Search turned on first

This is the single most common reason a correctly configured trace pipeline produces nothing. The X-Ray OTLP endpoint requires Transaction Search to be enabled on the account, which redirects span ingestion into CloudWatch Logs:

aws xray update-trace-segment-destination --destination CloudWatchLogs

Spans then land in a log group named aws/spans, with a percentage indexed in X-Ray as trace summaries for search. The default index rate is one percent, enough to find traces while all spans stay queryable as structured logs. The caller needs xray:UpdateTraceSegmentDestination and xray:UpdateIndexingRule plus log group creation rights.

Two consequences to plan for. Span ingestion is billed separately from log ingestion, so it’s a line item rather than a rounding error. And AWS recommends always_on sampling in the SDK on this path, because the indexing rule already handles reduction. Sample in both places and your service map goes patchy.

The X-Ray SDK clock is running

The dates matter here, so plainly: the X-Ray SDKs and daemon entered maintenance mode on 25 February 2026 (security fixes only, no new instrumentation support), with end of support on 25 February 2027. The X-Ray service is fine and still gaining features. It’s the client libraries and the UDP daemon winding down. If you run aws-xray-sdk and a daemon sidecar, that is technical debt with a published expiry, and either the collector or the CloudWatch agent replaces the daemon.

Decision three: metrics, temporality, and the cardinality bill

Metrics are where the silent drops live, because collector defaults and endpoint limits disagree.

The metrics endpoint caps a single request at 1 MB uncompressed and 1,000 datapoints, counted across resource, scope, and metric levels combined. The upstream batch processor’s default is far larger. Leave it alone and you build oversized requests, and the response to an oversized or partially invalid request is not always a clean failure: it can come back 200 with some metrics accepted and others rejected or throttled. So set the batch size deliberately. AWS’s own examples use a conservative value:

processors:
  batch:
    send_batch_size: 200
    timeout: 10s

The other limits shape attribute design more than config:

• 150 labels maximum across resource, scope, and datapoint attributes per datapoint
• 40 KB combined label and value size per series per datapoint
• One million new series creatable per ten-minute window, per account
• Timestamps no more than ten minutes in the future or fourteen days in the past

That new-series ceiling catches teams out. Attach a request ID, a session ID, or a raw URL path to a metric attribute and every request mints a fresh series. You’ll hit a million faster than you expect, and it presents as metrics randomly going missing rather than a quota error. Unbounded identifiers belong on spans and logs. Metric attributes should be values you could enumerate on a whiteboard.

Temporality is the other conscious decision. The OpenTelemetry SDK spec defaults to cumulative, while CloudWatch’s metric model is delta-shaped: it wants what happened this period, not the total since your process booted. Counters that look like ever-climbing staircases instead of rates are that mismatch. The cumulativetodelta processor converts in the pipeline, keeping the decision in the collector rather than scattered across SDK environment variables in every service.

Decide once and centrally either way. Mixed temporality across one account produces dashboards that are subtly wrong for months before anyone notices. And once volume climbs, pointing a cost tool such as Vantage or CloudZero at the CloudWatch line items earns its setup time, because telemetry spend grows in steps nobody approved.

Decision four: a second backend without a second instrumentation

Here’s the payoff. A collector pipeline holds multiple exporters, and adding one is a config change:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/traces, otlphttp/vendor]

The same spans now reach CloudWatch and Grafana Cloud, Honeycomb, Datadog, or a self-hosted Tempo, with no application redeploy and no second agent. That’s the cashable value of instrumenting once against an open protocol: the switching cost of a backend drops from a migration project to a pull request.

Be honest about the price. You pay ingest twice, and egress from the gateway is real money at volume. Collector memory scales with exporter queue count, so a fan-out gateway needs headroom. And if one backend slows, backpressure can reach the pipeline feeding the other, which argues for separate pipelines per destination when their reliability differs.

It works in reverse too. Because the metrics and logs endpoints accept bearer tokens, a machine with no AWS credentials at all can ship into CloudWatch: a CI runner, another cloud, or a VPS at a provider like Contabo or InterServer running part of your stack. One collector config, one destination, regardless of who owns the hardware. Traces are the exception and still require SigV4.

Why OpenTelemetry on AWS fails quietly

Most breakages here don’t throw. How to recognise the common ones:

Data is missing but nothing errors

Almost always a limit breach inside an accepted request. Check batch sizes against the per-signal caps first, then attribute counts. Turn on the collector’s internal telemetry and compare the exporter’s sent counters against its failed counters. If sent looks healthy and data is still missing, the loss is happening server-side after acceptance, which narrows it to limits.

403 with a signature mismatch

A SigV4 problem, not an IAM problem. The usual cause is the wrong signing service name in the extension, since xray, monitoring, and logs are not interchangeable. If you sign requests yourself rather than using the extension, note that the traces endpoint is stricter about which headers land in the signed set, so sign a minimal stable set rather than everything the HTTP client added.

Backfilled data vanishes

Every endpoint enforces a timestamp window, and fourteen days in the past is the outer edge for all three signals. Replaying an old queue past that boundary gets rejected. Not a bug, but it looks like one when the replay job appears to succeed and produces nothing.

Traces arrive but the service map is empty

Usually a missing or inconsistent service.name resource attribute, or double sampling between the SDK and the indexing rule. Confirm what’s actually leaving the collector before hunting in the console: add the debug exporter to a copy of the pipeline, set verbosity: detailed, and read the resource attributes on real spans.

Common mistakes

• Pointing an OTLP gRPC exporter at a CloudWatch endpoint. They’re HTTP only, and the error won’t say so clearly.
• Using endpoint with a full signal path instead of the per-signal traces_endpoint, metrics_endpoint, or logs_endpoint.
• Leaving the batch processor at its default size on the metrics pipeline.
• Sampling in the SDK and relying on Transaction Search indexing rules, halving visibility twice over.
• Putting request IDs, user IDs, or raw paths into metric attributes.
• Running a gateway pool with no alerting on the collector’s own health, so saturation is invisible until you need it.

Best practices

• Keep instrumentation vendor-neutral. Plain OpenTelemetry SDK APIs, with the collector owning every AWS-specific decision.
• Set service.name, service.version, and deployment.environment as resource attributes everywhere. Almost every correlation feature depends on them.
• Alert on exporter failure counters and queue depth, and route those alerts somewhere that doesn’t depend on the pipeline being healthy.
• Version the collector config in Git and deploy it like application code, with a staging pipeline you can break safely.
• Redact in a processor before export. Once telemetry reaches a backend, removing it is a support ticket.
• Test a second exporter early, even to a throwaway account. A vendor migration is the wrong time to learn the fan-out path doesn’t work.

Frequently asked questions

Do I still need a collector if CloudWatch accepts OTLP directly?

Not to get data in, no. You need one to control what happens before it leaves: batching to fit endpoint limits, tail sampling, redaction, and fan-out. Small serverless estates can reasonably skip it. Anything with a dozen services will want one.

Can I send OTLP to CloudWatch over gRPC?

No. The CloudWatch OTLP endpoints are HTTP 1.1 only, accept binary or JSON payloads, and support gzip or no compression. Your collector can still receive gRPC from applications, it just can’t forward over it.

What’s the difference between ADOT and the upstream collector?

Same codebase, different build. ADOT is AWS’s tested distribution with AWS components bundled and AWS Support behind it; upstream Contrib carries a wider component set and moves faster. Configuration is compatible either way, so it’s a support and packaging choice, not an architectural one.

Do I have to enable Transaction Search to send traces?

For the X-Ray OTLP endpoint, yes. It’s a prerequisite, and enabling it routes span ingestion through CloudWatch Logs. Using the awsxray exporter against the classic X-Ray API instead doesn’t require it, but you give up the span-level analytics Transaction Search provides.

Can I ship telemetry to CloudWatch from outside AWS?

For metrics and logs, yes, using bearer token authentication instead of SigV4, which removes the need for AWS credentials on the host. Traces still require SigV4. Never hardcode the token in the config; read it from a mounted secret file or an injected environment variable.

Is it urgent to migrate off the X-Ray SDK?

Not an emergency, but it’s on a clock: maintenance mode now, end of support 25 February 2027. Existing applications keep working, they just won’t get new library instrumentation. Plan the migration on your schedule rather than someone else’s.

The one thing worth remembering

A pipeline that returns 200 is not a pipeline that works. Judge every design decision in OpenTelemetry on AWS against one question: when this drops data, do I find out from a dashboard or from a colleague asking why a service vanished off the map?

Size batches against the published limits, keep unbounded identifiers off metric attributes, decide temporality in one place, and instrument with plain OpenTelemetry so the export target stays a config line. Do that and “instrument once, export anywhere” stops being a slogan and becomes a property you can test.

Need help with your OpenTelemetry pipeline on AWS?

Most of this work is unglamorous and specific, which is why it gets deferred. Things I can help with:

• Auditing an ADOT or upstream collector config for silent drops, oversized batches, and limit breaches
• Designing collector topology for ECS, EKS, or Lambda, including gateway sizing and autoscaling
• Migrating X-Ray SDK and daemon workloads to OpenTelemetry ahead of end of support
• Cutting telemetry spend through sampling policy, cardinality control, and attribute pruning
• Dual export to CloudWatch and a third-party backend, so a future switch is a config change
• Meta-monitoring that tells you the pipeline is broken before your users do

Send me a collector config, an exporter log, or a screenshot of the gap in your dashboard and I’ll tell you what I’d look at first.

The post Instrument Once, Export Anywhere: OpenTelemetry on AWS With ADOT appeared first on John Nessime.