So you sketch the obvious thing. Amazon Cognito for sign-in, Amazon S3 for the documents, a small front end in between. It’s three services and a weekend, and it works on the first try, which is exactly the problem. A secure patient document portal built this way passes every test you think to run, because the tests you think to run are all “can Alice download Alice’s file?” The one that matters is “what can Alice see about Bob?” and that one has an answer you won’t like.

This post covers the parts that go wrong after the demo. Where to put the authorization boundary, how to scope the S3 policy so one patient can’t enumerate another, why your download links expire earlier than you configured, what encryption actually gates, and what your audit trail looks like on the day someone asks for it. Everything here is mechanism, not a compliance checklist. Getting a Business Associate Agreement in place is a separate conversation with your lawyer and your AWS account team.

The failure that never pages you

Here’s the shape of it. You write an IAM policy that lets a signed-in user read objects under their own prefix. Something like arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*. You test it. Alice gets her file. Alice cannot get Bob’s file. Ship it.

Then, because the front end needs to render a file list, someone adds s3:ListBucket. And s3:ListBucket is not an object-level action. It applies to the bucket ARN, not the object ARN, so the ${cognito-identity.amazonaws.com:sub} substitution in your resource string does nothing for it. Unless you add a separate condition on the request, a signed-in patient can list every key in the bucket.

They still can’t download anything. Every GetObject outside their prefix returns a denial. But they can read the key names. And key names in a clinic bucket are rarely neutral. patients/8f2a.../oncology-referral-2024.pdf tells you something about a person even if the bytes stay locked.

Nothing alerts on this. No error appears in any log you’re watching, because nothing errored. The listing succeeded. That’s the failure mode worth building around: in a document portal, the metadata leak is the one that doesn’t announce itself.

The fix is a condition key on the list action, not a resource string:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOwnPrefixOnly",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::clinic-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "patients/${cognito-identity.amazonaws.com:sub}/*"
          ]
        }
      }
    },
    {
      "Sid": "ReadOwnObjectsOnly",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*"
    }
  ]
}

Two statements, two different jobs. The first constrains what the caller may ask the bucket to enumerate. The second constrains what bytes the caller may pull. Write only the second one and you’ve secured the download path while leaving the discovery path wide open.

One trap in that variable. ${cognito-identity.amazonaws.com:sub} resolves to the identity ID from the Cognito identity pool, not the sub claim from the user pool. They are different values for the same human. If you name your prefixes with the user pool sub because that’s what your application code has handy, the policy will match nothing and every request will be denied, which at least fails loudly.

Decide where the authorization boundary lives

There are two honest architectures here and the choice shapes everything downstream. Pick deliberately.

Pattern A: browser holds AWS credentials

The user signs in to a Cognito user pool. The application exchanges that token with a Cognito identity pool for temporary AWS credentials, and the browser calls S3 directly using the AWS SDK.

It’s less code and there’s no server to run. The authorization boundary is entirely the IAM role policy attached to the identity pool. That policy is the whole security model, so it had better be right, and it’s harder to reason about than application code because the enforcement happens in a service you can’t step through with a debugger.

The real limitation is expressiveness. IAM policy conditions do not know that Bob’s referral was superseded, that Alice revoked consent last Tuesday, or that a document should only be visible after a clinician releases it. Anything with business logic in it has to be encoded as prefix structure or object tags, and that gets brittle fast.

Pattern B: backend brokers every object

The browser never touches AWS credentials. It sends the Cognito access token to your own API. The API validates the token, checks its own records for who may see what, and returns a presigned URL for that one object. S3 is reachable only by the API’s execution role.

You now have a place to put logic, a place to log intent alongside access, and a single choke point to rate limit. The cost is that you own a service, its uptime, and its dependencies.

For a small clinic build, this is the one I reach for first. Not because Pattern A is insecure, but because the questions clients ask in month three (“can we hide results until the doctor reviews them?”) are answerable in Pattern B and require a re-architecture in Pattern A. The identity-pool approach earns its keep when access really is a pure function of who you are, which in healthcare it rarely stays.

If you do want attribute-driven access without a backend, Cognito identity pools can map claims from the token onto principal tags on the temporary session, which you then reference in policy conditions. It works, and it is the right tool for coarse distinctions like patient versus staff. It is not the right tool for per-document consent state.

Your download links expire before you think they do

This one generates support tickets on week two. You set a presigned URL to last a few hours. Patients report links dying in under one. Nothing in your code changed.

A presigned URL carries a signature made with whatever credentials signed it, and it cannot outlive those credentials. AWS documents the ceilings clearly: with long-lived IAM user keys and Signature Version 4 the maximum is seven days; from the S3 console the range is one minute to twelve hours; with temporary credentials from an assumed role, the URL dies when the role session does, regardless of the expiry you asked for. A default AssumeRole session is one hour.

Your Lambda or container signing those URLs is using temporary credentials. So the effective lifetime is the shorter of your requested expiry and the remaining time on that role session, and the remaining time depends on when the runtime last refreshed. That’s the source of the flapping. You can watch it yourself:

# Sign a URL and inspect what was actually encoded into it.
aws s3 presign s3://clinic-docs/patients/example/scan.pdf --expires-in 3600

# The returned URL contains X-Amz-Date, X-Amz-Expires and X-Amz-Credential.
# X-Amz-Expires is the window you asked for. If X-Amz-Credential shows an
# ASIA... key id, you signed with temporary credentials and the real ceiling
# is the session, not the window.

The honest resolution is to stop fighting it. Sign short, on demand, when the user clicks. Fifteen minutes is generous for a download the patient is standing in front of. Long links are not a convenience feature in this context anyway, because a presigned URL is a bearer credential: whoever holds the string can fetch the object, signed in or not. Forwarding the email forwards the access. If your portal mails links, you have moved the authorization boundary into someone’s inbox.

Mail a notification that says a document is ready and link to the portal login. Never mail the signed URL itself.

Encryption that actually gates something

New S3 objects are encrypted at rest by default with S3-managed keys. That satisfies the sentence “data is encrypted at rest” and gates precisely nothing, because anyone with s3:GetObject gets plaintext back. It is a storage property, not an access control.

Server-side encryption with an AWS KMS customer managed key is different in kind. Reading an object now requires two independent grants: the S3 permission and kms:Decrypt on the key. Those live in different policies and can be owned by different people. A bucket policy mistake no longer hands over plaintext on its own, and revoking the key grant cuts off reads without touching a single S3 permission.

You also get a second audit stream. Every decrypt is a KMS API call, recorded separately from the S3 call, which is useful precisely because an attacker who tampers with one trail probably didn’t think about the other.

The costs are real and worth stating. KMS charges per request, so a portal that fetches many small objects will notice. S3 Bucket Keys reduce that request volume substantially by caching a bucket-level key, at the price of coarser detail in the KMS audit records. For a document portal with modest traffic and high sensitivity, I’d take the per-request cost and keep the granularity. For a thumbnail-heavy application, the opposite.

The audit trail you don’t have yet

Assume someone will eventually ask: who downloaded this specific file, and when? If you have not explicitly turned something on, the answer is that you don’t know.

CloudTrail is on for management events by default, and management events are control-plane operations: creating the bucket, changing its policy, attaching a role. Object-level reads and writes are data events, and data events are off by default because of volume and cost. GetObject on a patient scan is a data event. Unless you configured a trail with data events for that bucket, no record of the download exists.

# Log S3 object-level activity for one bucket on an existing trail.
# Advanced event selectors let you scope to a single bucket prefix rather
# than every object in the account.
aws cloudtrail put-event-selectors 
  --trail-name clinic-audit-trail 
  --advanced-event-selectors '[
    {
      "Name": "PHI bucket object access",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Data"]},
        {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
        {"Field": "resources.ARN", "StartsWith":
          ["arn:aws:s3:::clinic-docs/patients/"]}
      ]
    }
  ]'

S3 server access logs are the other option and they are not equivalent. They record HTTP-level request detail and are delivered on a best-effort basis with no delivery guarantee, which makes them useful for traffic analysis and weak as evidence. CloudTrail data events carry caller identity, which is the field you actually need when the question is “who.”

Two things to sort out while you’re in there. Send the trail to a bucket in a different account, or at minimum a bucket the portal’s roles cannot write to, so that compromising the application doesn’t compromise its own log. And put a lifecycle rule on the log bucket, because data event volume grows with usage and nobody budgets for it. If you already run centralized dashboards on something like Grafana Cloud or CloudWatch, wire an alert to unusual read volume per identity now rather than after the incident.

Cutting off access when someone leaves

Disabling a Cognito user does not immediately end their session, and this surprises people. Tokens are self-contained. An access token already in a browser stays cryptographically valid until it expires, typically about an hour.

Refresh tokens are the longer exposure. The default is thirty days, and app clients can be configured anywhere from sixty minutes to ten years. Ten-year refresh tokens on a portal holding medical records is not a hypothetical misconfiguration; it’s a checkbox someone ticked to stop users complaining about re-login.

Cognito gives you revocation. Token revocation is enabled by default on newly created app clients. AdminUserGlobalSignOut revokes all of a user’s refresh, ID, and access tokens from the server side, and the /oauth2/revoke endpoint kills one refresh token and everything descended from it.

# Server-side, authorized with IAM credentials. Revokes the user's refresh,
# ID and access tokens.
aws cognito-idp admin-user-global-sign-out 
  --user-pool-id us-east-1_EXAMPLE 
  --username patient-example

# Confirm revocation is switched on for the app client before you rely on it.
aws cognito-idp describe-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --query 'UserPoolClient.EnableTokenRevocation'

Now the part that catches teams out. AWS documents that a revoked token still passes verification in a generic JWT library that only checks signature and expiry. Revocation is enforced by Cognito, not baked into the token. If your API validates tokens by fetching the JWKS and calling a verify function, a revoked token sails straight through until it expires on its own.

Which means short access token lifetimes are load-bearing. Revocation shortens the tail; it doesn’t close the window. Refresh token rotation, available on the Essentials and Plus feature tiers, replaces the refresh token on each exchange and narrows the window a stolen one is useful for. If you’re paying for Plus for threat protection features, you already have it.

Troubleshooting the usual suspects

Every request returns access denied after login

Nine times out of ten the prefix in your policy variable doesn’t match the prefix in your bucket. Print the identity ID your session actually resolved to and compare it character by character against a real key. Remember the identity pool ID and the user pool sub are different values.

Signature does not match on a presigned URL

Usually something modified the request after signing. A client library adding a header, a proxy rewriting the path, or an email gateway URL-encoding the query string. Compare the X-Amz-SignedHeaders list in the URL against what the client actually sent.

Uploads succeed but downloads fail

Check the KMS key policy separately from the IAM policy. A role can hold kms:GenerateDataKey (enough to write) without kms:Decrypt (needed to read). The S3 error will not say the word KMS prominently, which is why this one burns an afternoon.

Browser uploads fail before any request reaches S3

That’s CORS. The bucket needs a CORS configuration allowing your portal’s origin, the methods you use, and any headers the SDK sends. Browser preflight failures never appear in S3 logs, so check the browser console first.

Nothing appears in CloudTrail

Data events are not shown in CloudTrail Event History. Query the destination bucket directly, with Athena or CloudTrail Lake, and allow a few minutes for delivery.

Common mistakes

• Granting s3:ListBucket without an s3:prefix condition, so patients can enumerate every key even though downloads are blocked.
• Putting identifying information in object keys or file names. Assume the key will leak eventually and use opaque identifiers.
• Emailing presigned URLs. The link is the credential, and inboxes get forwarded, backed up, and breached.
• Extending refresh token lifetime to silence complaints about re-login, instead of implementing rotation.
• Assuming default encryption means access control. It means the disk is encrypted, nothing more.
• Enabling versioning without a lifecycle rule, so deleted documents linger indefinitely as noncurrent versions you forgot exist.
• Skipping CloudTrail data events to save money, then discovering the gap during an access review.

Best practices for a secure patient document portal

• Keep one bucket for patient documents and nothing else. Mixed-purpose buckets make policies ambiguous and audits slow.
• Turn on S3 Block Public Access at the account level, not just the bucket. Account level survives someone creating a new bucket in a hurry.
• Enable versioning and set a noncurrent version expiration rule, so an overwritten document is recoverable but not permanent.
• Use a KMS customer managed key so decrypt authority is separable from S3 authority.
• Require MFA for staff accounts. TOTP costs nothing to enable and staff accounts are the ones with breadth.
• Sign download URLs on demand with short expiry, and always from a backend that logged why it signed.
• Put the portal behind a WAF and rate limiting. Cloudflare in front of the application, or AWS WAF on the distribution, both work; the point is that credential stuffing against a login page is the most likely thing to actually happen to you.
• Restrict administrative access to the AWS console and to any bastion by network as well as identity, whether that’s IAM Identity Center with condition keys or a VPN such as NordVPN or Surfshark for a small team without central identity.
• Write down the retention period before launch, then encode it as a lifecycle rule. Retention decided later is retention never decided.

Frequently asked questions

Are S3 and Cognito enough for HIPAA compliance?

Both are covered under AWS’s HIPAA eligibility program, and Cognito’s documentation states it is HIPAA-BAA eligible. That makes them usable building blocks, not compliance. You need a signed Business Associate Agreement with AWS, and the configuration, access reviews, training, and documented policies remain yours. Service eligibility is a prerequisite, not an outcome.

Should I use a Cognito identity pool or a backend API?

Use an identity pool when access is a pure function of identity and the rules will not change. Use a backend when documents have states, when a clinician gates release, or when you need to log intent alongside access. Most clinic portals become the second within a year.

How long should a presigned URL last?

Minutes, not hours. Sign at click time. The maximum with long-lived credentials is seven days, but that ceiling is irrelevant here because the link is a bearer credential and a longer window is a longer leak.

Can patients upload documents back to the portal?

Yes, using presigned PUT or presigned POST. Presigned POST is worth knowing about because its policy document can constrain content type and size before the object lands, which presigned PUT cannot. Either way, write uploads into a quarantine prefix, scan them, then move them.

Does enabling CloudTrail data events cost much?

Data events are billed per event recorded with no free tier, unlike management events. For a portal serving a few thousand downloads a month the amount is small; for a bucket also serving application assets it is not. Scope your event selector to the patient prefix rather than the whole account and the question mostly goes away.

What happens to sessions when I disable a user?

Nothing immediate. Existing access tokens remain valid until they expire. Call AdminUserGlobalSignOut to revoke tokens, and keep access token lifetime short so the residual window is measured in minutes.

Do I need CloudFront in front of the bucket?

Not for correctness. It helps when patients are geographically spread, when you want signed cookies for a set of objects rather than one URL per object, or when you want a single edge to attach a WAF to. For a single-clinic portal, direct presigned URLs are simpler and one fewer thing to misconfigure.

The one thing to remember

A signed-in token proves who someone is. It does not decide what they may reach. In a secure patient document portal the boundary is the policy condition, the KMS grant, and the code that decides whether to sign a URL at all, and every one of those is something you have to write on purpose.

The demo passes either way. The difference only shows up when someone reads a key name they shouldn’t have seen, and nothing in your system considers that an error.

If you build one of these, spend an hour before launch on exactly two things: sign in as a test patient and try to list the bucket, then ask your logs who downloaded a specific file last week. If either answer is uncomfortable, you’ve found your next task.

Need a second pair of eyes on your portal?

This is the kind of build where the gap between working and safe is invisible from the outside. I help small teams close it.

• Reviewing S3 bucket policies and Cognito identity pool roles for prefix isolation gaps, including the list-versus-get split most policies miss.
• Designing the boundary: identity pool credentials in the browser versus a backend that brokers presigned URLs, with the trade-offs written down for your stakeholders.
• Setting up KMS customer managed keys, key policies, and grant separation so decrypt authority is not implied by storage authority.
• Turning on scoped CloudTrail data events, building the Athena tables to query them, and alerting on abnormal per-identity read volume.
• Hardening the Cognito side: MFA, token lifetimes, refresh token rotation, and a revocation path that actually ends sessions.
• Writing the whole thing as Terraform or OpenTofu so the next change is a reviewed pull request rather than a console click.

Send me your bucket policy and your identity pool role, with the account IDs redacted, and I’ll tell you what a signed-in patient can actually see.

The post Building a Secure Patient Document Portal on S3 and Cognito appeared first on John Nessime.

That gap is the whole problem with permission cleanup. IAM Access Analyzer unused access findings are genuinely useful, and they are also built on evidence with known holes in it. AWS says so in its own documentation. Treat a finding as proof rather than as a lead and sooner or later you take something down.

This post covers what an unused access finding actually asserts, the four blind spots that turn a clean-looking finding into an outage, and a removal procedure that fails safe.

What an unused access finding actually claims

An unused access analyzer is a separate analyzer type from the external access one. You create it per account or across an organization, and it produces four kinds of finding: unused roles, unused IAM user access keys, unused IAM user passwords, and unused permissions at the service and action level for principals that are active.

aws accessanalyzer create-analyzer 
  --analyzer-name unused-access-prod 
  --type ACCOUNT_UNUSED_ACCESS 
  --configuration '{"unusedAccess": {"unusedAccessAge": 90}}'

unusedAccessAge is the threshold in days beyond which access counts as unused, and it accepts 1 to 365. Use ORGANIZATION_UNUSED_ACCESS from a delegated administrator account. Unused access findings don’t vary by Region, so you don’t need one analyzer per Region.

Here is the part that matters. The analyzer does not replay CloudTrail and reconstruct what your workload did. It reads IAM last accessed information, the same data behind the Access Advisor tab in the console. Everything the analyzer knows, and everything it doesn’t, comes from there. The blind spots below are last accessed blind spots, inherited wholesale.

The four blind spots that break production

1. Action-level tracking doesn’t cover everything

Action last accessed information only exists for the services and actions AWS has explicitly added to the tracked list, and it covers management actions. The IAM console spells out one case in plain text: action last accessed information is not available for Amazon S3 data events.

Think about what that means for a data pipeline. A role whose entire purpose is s3:GetObject and s3:PutObject against one bucket is doing object-level work all day. Bucket-level calls like ListBucket are management events and get tracked. The object-level calls that constitute the actual job are data events, and they sit outside action last accessed.

AWS puts the warning in the documentation itself: do not make permissions decisions based solely on the absence of tracking information. Almost nobody reads that sentence.

2. Denied attempts count as access

Service last accessed data includes all attempts to call an AWS API, not just successful ones. A role that has been hammering a service it has no permission for, failing every time, shows up as having used that service.

This runs opposite to the first blind spot, and it is why a findings list can be simultaneously too aggressive and too timid. Dead permissions look alive because a misconfigured client keeps retrying against them, so you clean up around them and leave the real cruft in place.

3. The calendar is longer than your window

A ninety-day tracking period is the common default and it is wrong for a predictable class of role:

• Disaster recovery and failover roles, exercised once or twice a year
• Quarterly close, audit, and regulatory reporting roles
• Annual key rotation or certificate renewal automation
• Break-glass roles, which should be unused by design
• Seasonal workloads that only wake up for a peak period

There is also a ceiling worth knowing. unusedAccessAge tops out at 365 days, while IAM reports last accessed activity for at least the trailing 400 days. The analyzer cannot be configured to span the full window the underlying data actually holds, so for anything on an annual cycle it will not settle the question on its own.

4. Service-level use says nothing about resource-level need

A finding operates on services and actions. Your policy probably operates on services, actions, resources, and conditions. Knowing a role used secretsmanager:GetSecretValue last week does not tell you which secret, and narrowing the policy to a specific resource ARN based on a finding is a guess dressed up as evidence.

This is where policy generation earns its place. It reads CloudTrail activity for a principal over a window and drafts a policy from what it observed, including resource-level detail where the log has it. Same caveat: policy generation does not identify action-level activity for data events either. Better first draft, not an oracle.

A removal procedure that fails safe

The core idea: deny before you delete. An explicit deny is one API call to add and one to remove, effective immediately in both directions. Deleting a policy statement or a role is not symmetrical; restoring it means someone reconstructing intent under pressure. Use deny as a reversible dry run for deletion.

1. Sort findings by blast radius, not by count. An unused permission on a role assumed by one Lambda is a different risk from one on a role fifty engineers assume daily.
2. Confirm the role’s activity independently before touching permissions.
3. Attach a scoped deny as an inline policy on the role, leaving the existing allow untouched. An explicit deny wins over any allow, so this simulates removal exactly with none of the loss.
4. Watch CloudTrail for denials attributed to that role. This is the observation window and it is the step people skip.
5. Wait through at least one full business cycle, longer if the workload has monthly or quarterly rhythm.
6. Only then edit the allow and remove the deny.

For step two, the cheapest independent check is the role’s own last-used timestamp:

aws iam get-role --role-name legacy-reporting-role 
  --query 'Role.RoleLastUsed'

That returns LastUsedDate and the Region where the role was last used. LastUsedDate is null if the role has not been used within the tracking period, which is not the same as never used. The Region field is quietly valuable: a supposedly dead role showing activity in a Region nobody works in is worth a conversation before a deletion.

For step three, the deny is deliberately narrow. Deny the flagged actions on the flagged service and nothing else:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CanaryDenyUnusedRedshift",
      "Effect": "Deny",
      "Action": "redshift:*",
      "Resource": "*"
    }
  ]
}

Give it a Sid you can grep for, because in six months somebody will find this policy and have no idea whether it is a canary or a control.

For step four, you want CloudTrail records where the request failed on authorization. The field is errorCode, and the values that matter are AccessDenied plus the EC2-specific Client.UnauthorizedOperation. Filter on the assumed-role session issuer so you catch every session. CloudTrail Lake or Athena over your trail bucket both work.

To make this an ongoing signal rather than a one-off query, wire a CloudWatch Logs metric filter on AccessDenied to a dashboard. Amazon Managed Grafana, self-hosted Grafana, or Grafana Cloud all render it fine. What matters is that it is broken down by principal, so a spike points at a role rather than at the account.

Deleting the role itself has a separate trap

When you name a principal by ARN in a policy, IAM stores the underlying unique principal ID rather than the string. Delete that role, recreate it with the identical name, and you get a new unique ID. Every policy that referenced the old one now points at something that no longer exists. Nothing errors at deletion time. The break surfaces later, in another account, when a cross-account assume-role call starts failing and the trust policy on screen looks exactly correct.

Before deleting a role, check the things a findings list will never tell you:

• Is it named as a principal in any resource policy, S3 bucket policy, KMS key policy, or another account’s trust policy?
• Is it attached to an instance profile?
• Is it managed by CloudFormation, Terraform, or OpenTofu? Deleting out of band leaves drift that surfaces on the next apply.
• Is it a service-linked role? Those are owned by a service and have their own deletion procedure.
• Is it the execution role for a Lambda function, ECS task, or Step Functions state machine that runs rarely?

Safe order: detach the permissions, leave the role and its trust policy in place, wait another cycle, then delete. An empty role is harmless. A missing one is not.

Keeping the findings list small enough to read

A findings list nobody triages is worse than none, because it produces the appearance of coverage. Two levers keep it honest. The first is exclusions, which can skip whole accounts or skip principals by tag, and can be updated at any time:

aws accessanalyzer update-analyzer 
  --analyzer-name unused-access-prod 
  --configuration '{
    "unusedAccess": {
      "analysisRule": {
        "exclusions": [
          {"accountIds": ["111122223333"]},
          {"resourceTags": [{"lifecycle": "break-glass"}]}
        ]
      }
    }
  }'

Tag-based exclusion is the one to reach for. Sandbox accounts churn roles and generate noise with no security value; break-glass roles are supposed to look unused. Excluding by tag survives new accounts joining the organization. An account ID list does not.

The second lever is archiving findings you have consciously accepted:

aws accessanalyzer update-findings 
  --analyzer-arn <your-analyzer-arn> 
  --ids <finding-id> 
  --status ARCHIVED

Archiving is a decision record, not a dismissal. Write down why somewhere durable, because the same finding will come back to a different person.

Scope has a cost dimension too. Unused access analysis is billed on the number of IAM roles and users analyzed per month, per analyzer, so a second analyzer with a different tracking period doubles the principals counted. If you watch spend through Cost Explorer or a layer like Vantage or CloudZero, confirm where this line lands before widening scope organization-wide.

Troubleshooting

No findings yet, or a finding that won’t clear after you fixed it. After creating or updating an analyzer it takes time for findings to appear, recent activity takes around four hours to reach last accessed data, and scans are periodic rather than instant. Give it a full cycle. If you use the CLI or API for last accessed reports, generate a fresh one; the old report will not update itself.

The finding says unused but the team insists otherwise. Assume the team is right and the tracking has a hole. Check whether the work is data-plane activity, then whether the actions involved are on the tracked list at all. Absence of tracking data and absence of use look identical from the findings page.

Your canary deny broke something unexpected. That is the canary working. Detach the inline policy, confirm the failures stop, then narrow the deny to specific actions rather than the whole service and run it again.

Something broke but no AccessDenied appears in CloudTrail. Check whether the failing call is a data event. A trail logs management events by default; data event logging is opt-in per resource type and carries its own charges. No configured data events, no record of the denial.

Common mistakes

• Treating a finding as evidence of non-use rather than absence of recorded use. Different claims.
• Deleting the role in the same change as removing its permissions, collapsing two reversible steps into one that isn’t.
• Setting the tracking period once and never revisiting it against the workload’s actual rhythm.
• Cleaning up in the console while Terraform or CloudFormation remains the source of truth, so the next apply puts it all back.
• Batching removals across many roles at once. When something breaks you have a bisect problem instead of a rollback.

Best practices for IAM Access Analyzer unused access cleanup

• Tag principals with their expected usage cadence at creation time. It costs nothing then and answers the hardest question later.
• Start with unused access keys and passwords on IAM users. Highest value, lowest risk removals in the whole list.
• Route findings to EventBridge and into your ticket queue so cleanup is tracked work rather than a periodic heroic sweep.
• Make the permissions change in code, apply it through the normal pipeline, and let the diff be the record.
• Keep an AccessDenied dashboard permanently, not just during a cleanup. Cheapest early warning you can build for this class of change.

Frequently asked questions

Does IAM Access Analyzer read my CloudTrail logs?

Not for unused access findings. Those come from IAM last accessed information. Policy generation is the feature that reads CloudTrail, and it is a separate operation run against a specific principal and time range.

What tracking period should I set?

Ninety days is reasonable for interactive human roles and continuously running workloads, and too short for anything on a quarterly or annual cycle. The value is configurable from 1 to 365 days. The right answer is a property of your workload, not a security best practice number.

Can a finding be wrong?

It can mislead in both directions. Permissions used only through data events can appear unused, and permissions that only ever produced denied attempts can appear used, because last accessed data counts all attempts rather than only successful ones.

Do I need an analyzer in every Region?

No. Unused access findings do not change based on Region, so one per account, or one at the organization level, is enough. External access analyzers are the ones with a Region dimension.

How is unused access analysis billed?

On the number of IAM roles and users analyzed per month, per analyzer. Scope is the cost lever, which is why exclusions matter for the bill and not just the noise. Check current published rates before enabling it organization-wide.

Should I delete unused roles or just strip their permissions?

Strip first, delete later, in two separate changes. A role with no permissions has almost no attack surface, and keeping it alive preserves any principal references pointing at it while you find out if anything still cares.

Conclusion

IAM Access Analyzer unused access findings answer a narrower question than the one you care about. They tell you what AWS did not record activity for. They do not tell you what your workload does not need, and they cannot tell you whether a permission that is used should be. The distance between those statements is where the outages live.

The one thing worth remembering: deny before you delete. An explicit deny gives you the exact behaviour of a removal with a one-call undo, and that single habit turns permission cleanup from a change you have to be brave about into one you can be systematic about.

Need help trimming IAM permissions without the outage

The risk in permission cleanup sits almost entirely in what the tooling cannot see. Things I can help with:

• Setting up account or organization-level unused access analyzers with tracking periods and tag-based exclusions that fit how your workloads run
• Building the AccessDenied observability layer, from CloudTrail queries through metric filters to a dashboard the team will actually look at
• Running staged deny-then-delete cleanups with the rollback path defined before anything changes
• Auditing roles for cross-account principal references, instance profiles, and IaC ownership before deletion
• Moving console-era IAM into Terraform or OpenTofu so the next cleanup is a reviewable diff

If you have a findings list you are not sure how to act on, send me the role’s policy JSON and the finding detail and I will tell you what I would check before touching it.

The post IAM Access Analyzer Unused Access: Removing Permissions Without Breaking Production appeared first on John Nessime.

It granted iam:PassRole on one role ARN and lambda:CreateFunction in the same document. That is not a wildcard and it does not read as administrative access. It is administrative access anyway, to whatever that role can do, for anyone who can write a Lambda function.

This is the shape of the problem with AI-generated IAM policies. The failure is almost never the obvious one. Models have absorbed enough style guidance to avoid "Action": "*", so what comes back passes the eyeball test and the linter and still hands over more than you meant to give. The dangerous output is the one that looks careful.

This post covers what policy generation genuinely does well, the failure shapes that survive human review, and where the hard boundary belongs: not in the review step, which is fallible, but in a mechanism that cannot be talked out of its answer.

What AI-generated IAM policies get right

Start with the honest case, because it is a strong one. IAM has thousands of actions across hundreds of services and the naming is inconsistent in ways nobody holds in their head. Some services use Describe, some use Get, some use both for different things. ARN formats vary per service and per resource type inside a service. Condition keys apply to some actions and not others.

A model is good at that recall problem, and at shape work: splitting a monolithic statement into per-resource statements, adding conditions, converting an inline policy into Terraform. Paste an AccessDenied message and you get a plausible read of which layer refused. So the argument is not that the tooling is useless. It is that recall and reasoning about consequences are different capabilities, and IAM is a system where the second one protects you.

The failure modes that survive review

Permissions that compose into something bigger

Every action is individually defensible. The escalation lives in the combination.

The canonical case is iam:PassRole beside any service that runs code: Lambda, EC2, ECS, Glue. Passing a role to a compute service means your code executes with that role’s permissions, so your ceiling is not your own policy, it is whatever the most powerful passable role can do. Rhino Security Labs catalogued a long list of these chains and every one of them is built from permissions that look unremarkable alone.

Another that catches people: iam:CreatePolicyVersion on a policy you are attached to. A new version normally needs to be set as default to take effect, but the create call accepts a flag that makes it default immediately, and that flag does not require iam:SetDefaultPolicyVersion. A permission that reads as “can update policy documents” reads to an attacker as “can write myself an admin policy.”

Ask a model whether a policy grants admin and it will check the statements and say no, correctly, one at a time. Composition is the part it does not reliably do, and it is the part that matters.

Actions that do not exist

Plausible action names that no service publishes. s3:ListBucketContents instead of s3:ListBucket. IAM accepts a policy containing an unrecognized action string because the document is syntactically valid, so nothing fails at attach time.

What happens instead is worse. The application throws AccessDenied in production, somebody debugging under pressure decides the granular list must be wrong, and the fix is a service wildcard. The invented action does not grant too much. It causes a human to grant too much a week later, and by then nobody connects the two events.

Resource ARNs that look constrained and are not

An ARN with real characters in it feels safer than a star. Often it is not. The common versions are a trailing wildcard placed one segment too high, and an action whose resource element the service ignores. s3:ListAllMyBuckets is account-scoped no matter what you put in Resource. Several ec2:Describe actions behave the same way. A policy can read as resource-scoped throughout and be account-wide for the actions that count.

The related trap is the missing condition: cross-account trust policies without an external ID, bucket policies without aws:PrincipalOrgID, integration roles without aws:SourceIp where the caller has a stable egress address. Generated policies include conditions when you ask and omit them when you do not, because an absent constraint never produces an error.

Four things the model cannot know

Some of this is not a capability gap you can prompt past. It is missing input:

• Which resources exist. A bucket name is a guess unless you supplied it. Guessed names either fail closed or, if the wildcard is loose enough, match something you never intended.
• Which resources are sensitive. Your backup bucket and your public assets bucket are indistinguishable strings. Sensitivity lives nowhere in the document.
• What else is attached. Effective permissions come from identity policies, resource policies, boundaries, SCPs and session policies together. Reviewing one document tells you very little.
• What the blast radius is. “Can delete objects” means one thing for a scratch bucket and another for the only copy of a client’s data.

The training material is also skewed. Public IAM examples, including plenty of vendor documentation and countless forum answers, are permissive because permissive examples work on the first try. A model that has read the internet has read a lot of over-broad policies presented as correct.

Where the hard boundary belongs

Review is not a boundary. Review is a filter with a pass rate that drops the more policies you look at in a week, and generated policies are cheap to produce, so volume rises exactly when attention falls.

The boundary has to be something that says no without reading the document. AWS gives you two mechanisms and they are not interchangeable.

A permissions boundary is a managed policy attached to a user or role that caps what identity-based policies can grant it. It grants nothing itself. Effective permissions become the intersection of the identity policy and the boundary, so an identity policy allowing iam:* against a boundary that omits IAM produces no IAM access at all. If no boundary is attached, none is evaluated, so this is opt-in per identity.

A service control policy sets the maximum available permissions for every principal in an account or organizational unit, and also grants nothing. It is the right home for things nobody in that account should ever do, and it applies whether or not somebody remembered to attach a boundary.

The division I use: SCPs carry account-wide invariants, such as denying escalation-relevant IAM write actions to everyone outside the break-glass role and denying regions you do not operate in. Boundaries carry the per-workload ceiling, and they are what makes it safe to let a pipeline role create other roles. You can permit that role to create roles while requiring, through a condition on the boundary key, that every role it creates carries your boundary. The pipeline can then generate whatever policy it likes, because the ceiling is enforced at evaluation time rather than at review time.

If the only thing between a generated policy and production is a person reading JSON, you do not have a boundary. You have a habit.

The gate I would actually run

Mechanical checks in a pipeline, failing the build, not a checklist somebody is supposed to remember.

1. Lint for nonsense. Parliament catches unknown action names, resource formats that cannot match the action, and type mismatches. Cheapest way to catch an invented action before it becomes a wildcard.
2. Run AWS policy validation. Access Analyzer knows service-specific rules a generic linter does not.
3. Assert the escalation actions are absent. Name the actions you never want granted and fail if any appear. This is the check that maps to the composition failure.
4. Compare against the current policy. For edits, check whether the proposed document grants access the existing one does not. This one is billed per call, so run it on changed policies rather than the whole repository.
5. Scan for known risky patterns. Cloudsplaining scores policies against categories including privilege escalation, data exfiltration and resource exposure, and produces something you can hand to a client.

# 1. Lint. Exit status is non-zero when there are findings.
parliament --file policy.json

# 2. AWS-side validation
aws accessanalyzer validate-policy 
  --policy-document file://policy.json 
  --policy-type IDENTITY_POLICY

# 3. Fail if any escalation-relevant action is granted
aws accessanalyzer check-access-not-granted 
  --policy-document file://policy.json 
  --access actions="iam:PassRole","iam:CreatePolicyVersion","iam:AttachRolePolicy" 
  --policy-type IDENTITY_POLICY

# 4. Does the proposed policy grant more than the current one?
aws accessanalyzer check-no-new-access 
  --existing-policy-document file://current.json 
  --new-policy-document file://proposed.json 
  --policy-type IDENTITY_POLICY

# 5. Risk-categorised report
cloudsplaining scan-policy-file --input-file policy.json

Step three is the one to add first if you only add one. It turns “we reviewed it” into “the build fails.”

Attaching the boundary is a single call, and it is the part people skip because the role already works without it:

aws iam put-role-permissions-boundary 
  --role-name app-deploy-role 
  --permissions-boundary arn:aws:iam::123456789012:policy/WorkloadCeiling

Generate from observed behavior instead

There is a better source of truth than a model’s guess about what an application needs, and it is the application. Access Analyzer policy generation reads CloudTrail events for a role over a chosen window and writes a policy from what was actually called. Know its limits before relying on it:

• It analyzes up to ninety days of history and needs a trail already logging for the account, so a role that only exercises certain paths quarterly produces an incomplete policy.
• Coverage is per-service. For some services it identifies individual actions; for others it can only tell you the service was used and prompts you to fill in the actions yourself.
• It does not produce action-level detail for data events such as S3 object-level operations.
• iam:PassRole is not included in generated policies. If your workload needs it, a human adds it back by hand, so the most escalation-relevant line in the document is the one the generator did not write.

Behavior-derived policies also fail in a useful direction. A missing permission produces a denial you can see in CloudTrail, which beats a permission you did not know you granted.

Arguments that do not survive contact

• “We review every policy before merge.” Review quality is a function of volume and attention, and generation raises volume. Composition failures are also the hardest to spot by reading, because spotting them means knowing what other roles exist and what they can do.
• “Access Analyzer came back clean.” Validation checks the document. It does not know that the role ARN you may pass happens to have AdministratorAccess attached. Clean findings on a policy that grants a dangerous chain is the normal case, not an anomaly.
• “A permissions boundary is overkill for a small project.” Small projects are where this bites hardest. One account, one deploy role, no organization, nothing between a bad policy and everything you own. The boundary is one managed policy and one CLI call.
• “We will tighten it after launch.” Nobody does. If you genuinely intend to, drive it from Access Analyzer’s unused access findings so you have a work queue instead of an intention.
• “Just prompt the model to follow least privilege.” That changes the output surface, not the reasoning. Fewer wildcards, better statements, no evaluation of what the combination permits.

How I would decide

My rule is about which actions are in the document, not about who or what wrote it.

Ship generated policies freely when the document touches only data-plane actions on named resources, the identity has a boundary, and the pipeline checks pass. Read a bucket prefix, publish to a topic, write to a table. That is most policies, and hand-writing them wastes your afternoon.

Treat generated output as a first draft when the policy includes any IAM write action, any PassRole, any trust policy edit, any resource policy readable from outside the account, or any KMS key policy. Use it to save typing, then verify every statement against the service authorization reference yourself.

Do not generate at all for the boundary policies and SCPs. Those are the mechanism. Something that constrains everything else should be short, hand-written, understood by whoever owns the account, and changed rarely.

Two notes for small engagements. If you run application infrastructure on a VPS from a provider like Contabo or InterServer and pull AWS in for storage or mail, that server holds long-lived credentials somewhere your AWS controls do not reach, so scope its policy to a bucket prefix and attach a boundary before worrying about anything else. And if the credentials go to a SaaS connector, a marketing platform such as GoHighLevel or a payments integration, you are handing keys to a system whose behavior you cannot audit. Those are the policies to write by hand.

Frequently asked questions

Are AI-generated IAM policies safe to use in production?

For data-plane permissions on named resources, generally yes, provided the identity carries a permissions boundary and the policy passes automated validation. For anything involving IAM write actions, PassRole, trust policies or key policies, treat the output as a draft and verify each statement yourself. What matters is the blast radius of the actions, not the authorship.

Why does a generated policy pass validation and still grant too much?

Validation examines one document in isolation. Privilege escalation usually comes from a combination of individually reasonable permissions, or from the interaction between the policy and something outside it, such as which roles are passable and what those roles can do. A document can be internally correct and still open a path.

What is the difference between a permissions boundary and an SCP?

Both cap permissions and neither grants any. A boundary attaches to a single IAM user or role and limits what that identity’s policies can grant it. An SCP applies to every principal in an account or organizational unit and needs AWS Organizations. With one account and no organization, boundaries are what you have. With an organization, use both.

Can IAM Access Analyzer replace writing policies by hand?

Partly. Generation from CloudTrail activity beats a guess because it reflects what the workload actually called. It is limited by the analysis window, by per-service coverage differences, by the absence of action-level detail for data events, and by iam:PassRole being excluded. Treat it as a strong draft that needs a human pass over anything the generator could not see.

Which IAM actions should I block outright in a small account?

The ones that let a principal rewrite its own permissions or borrow another identity’s: creating and setting default policy versions, attaching and putting inline policies on users, roles and groups, creating access keys for other users, and passing roles to compute services. Deny them for everything except a break-glass identity, then grant them back deliberately where a workload genuinely needs one.

How do I catch an invented action name before production?

Run a linter that carries the IAM action catalogue. Parliament flags unknown actions and exits non-zero when it finds anything, so it drops straight into a pipeline step. Catching it early matters less because the fake action is dangerous and more because the eventual fix for the denial tends to be a service-level wildcard.

The one thing worth remembering

AI-generated IAM policies are not more dangerous because a model wrote them. They are more dangerous because they arrive faster than you can think about them, and because they arrive looking finished.

So stop trying to make the review step better. Put the boundary somewhere that does not depend on anyone paying attention: a permissions boundary on every identity a pipeline can create or modify, an SCP denying the escalation actions to everyone who does not need them, and a build step that fails when a forbidden action shows up. Then let generation happen at whatever speed it wants.

The policy approved in ninety seconds is not the failure. The absence of anything underneath it was.

Need help drawing that boundary in your account?

Most of the IAM work I take on is exactly this: an account that grew organically, permissions nobody wants to touch in case something breaks, and no ceiling underneath any of it. Things I can help with:

• Auditing existing roles and policies for privilege escalation chains, including the passable-role paths that never show up in a single-document review
• Designing and attaching permissions boundaries for pipeline, workload and human identities, including the delegation pattern that lets a deploy role create roles safely
• Writing the SCP set for a small organization: escalation-relevant IAM actions, region restrictions, and protection for logging and billing resources
• Adding Access Analyzer validation and custom policy checks to a CI pipeline so forbidden actions fail the build instead of relying on review
• Replacing over-broad roles with policies generated from CloudTrail activity, then closing the gaps the generator cannot see
• Scoping down credentials handed to third-party SaaS integrations and to servers outside AWS, where your account controls do not apply

If you want a second opinion on something specific, send me the policy JSON, the output of a failing check-access-not-granted run, or the role list from your account, and I will tell you what I would change and why.

The post AI-Generated IAM Policies: Where the Hard Boundary Belongs appeared first on John Nessime.

That failure should shape how you think about EKS Pod Identity vs IRSA. Both deliver the same end result to your application: short-lived, rotated credentials with nothing hardcoded. The differences are in how trust is expressed, who has to touch IAM to change it, and what happens when the wiring is subtly wrong.

This post compares the two on what actually decides the choice: how each scales across clusters, where each refuses to work at all, the failures each hides, and a decision procedure you can apply to a real cluster.

Same outcome, two very different trust shapes

Most of the trade-offs fall out of the mechanism, so it is worth being precise about both.

IRSA: federation through an OIDC provider

The cluster has an OIDC issuer, which you register as an identity provider in IAM. A projected service account token is mounted into the pod, and the SDK exchanges it for credentials through the web identity flow against STS. The role’s trust policy is what makes that work, and it names one specific cluster:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub":
          "system:serviceaccount:production:my-app",
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud":
          "sts.amazonaws.com"
      }
    }
  }]
}

The important detail is that the issuer URL is unique per cluster. The relationship between a workload and a role lives inside IAM, and it is cluster-specific by construction.

Pod Identity: an EKS API and an on-node agent

Pod Identity moves that mapping out of IAM. You install the EKS Pod Identity Agent add-on, which runs as a DaemonSet, then create an association through the EKS API binding a namespace plus service account name to a role. The agent calls the EKS Auth API for temporary credentials, and the pod picks them up through the container credential provider that AWS SDKs already support.

The trust policy contains nothing cluster-specific:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
    "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": [ "sts:AssumeRole", "sts:TagSession" ]
  }]
}

sts:TagSession is not decoration. Pod Identity attaches session tags on every assume, so a policy allowing only sts:AssumeRole fails. It is the most common first-attempt error.

Creating the association is a single call:

aws eks create-pod-identity-association 
  --cluster-name prod-eu 
  --namespace production 
  --service-account my-app 
  --role-arn arn:aws:iam::111122223333:role/my-app

Where IRSA still wins

Treating IRSA as legacy is the most common mistake here. For some workloads it is not the older option, it is the only option.

• Fargate pods. The Pod Identity Agent is a DaemonSet on EC2 nodes. Fargate has no node for it to land on, and AWS documents Fargate pods as unsupported. If part of your workload runs on Fargate, IRSA is the mechanism there, permanently, not as a stopgap.
• Windows nodes. Also documented as unsupported for Pod Identity.
• Older SDKs you cannot rebuild. The web identity flow has been in the SDKs far longer, so a vendor image you cannot patch is more likely to work with IRSA.
• Direct federation into workload accounts. If your model wants the pod’s identity federating straight into a role in the resource-owning account, IRSA does that natively. Pod Identity reaches other accounts by role chaining, a different trust shape even when the outcome looks the same.
• You already have it working at a scale you control. A few clusters with a Terraform module that provisions the provider and the roles is not a problem that needs solving.

Where IRSA hurts

The pain is administrative, and it grows with cluster count rather than workload count. Every new cluster brings a new OIDC issuer, which means editing the trust policy of every role those workloads use. Trust policies have a size limit, so a role trusting many issuers eventually runs out of room and you end up duplicating roles to work around it.

The same asymmetry shows up in blue/green cluster upgrades. Standing a new cluster beside the old one means every role used by every migrating workload needs a trust policy edit before the new cluster can do anything, and another edit afterwards to clean up. That is IAM work on the critical path of an upgrade, usually owned by a different team than the one doing it.

Where Pod Identity wins

• Roles become portable. One role, one generic trust policy, any number of clusters. Adding a cluster is an EKS API call per workload, not an IAM edit per role.
• The permission split matches how teams are organised. Creating an association needs EKS permissions, not iam:UpdateAssumeRolePolicy, so cluster admins stop filing IAM tickets for every new service account.
• Session tags arrive for free. Every assume carries eks-cluster-arn, eks-cluster-name, kubernetes-namespace, kubernetes-service-account, kubernetes-pod-name and kubernetes-pod-uid. That is usable attribute-based access control and the CloudTrail audit trail you always wanted.
• Cross-account access without touching application code. An association can carry a target role ARN, and EKS performs the role chaining for you.

The ABAC angle is underrated. One policy can serve dozens of workloads, because the session tag identifies the caller:

"Condition": {
  "StringEquals": {
    "s3:ExistingObjectTag/team": "${aws:PrincipalTag/kubernetes-namespace}"
  }
}

Where Pod Identity hurts

The generic trust policy is a real trade-off. Written exactly as AWS shows it, that role can be assumed on behalf of any service account in any namespace in any cluster in the account, as long as someone with EKS permissions creates an association pointing at it. Blast radius control has moved from IAM into the EKS API. If those permissions are loosely held, you widened it.

The fix is to put the constraint back in the trust policy using the request tags Pod Identity sends:

"Condition": {
  "StringEquals": {
    "aws:RequestTag/kubernetes-namespace": "production",
    "aws:RequestTag/kubernetes-service-account": "my-app"
  }
}

You can also pin to a cluster with aws:SourceArn. That costs some of the portability you switched for, which is the point: decide per role. A role reading one team’s bucket should be pinned. A role used by an identical workload in six clusters should not.

The other cost is a new moving part. The agent is a DaemonSet, so it needs to tolerate your taints, and it must be running before pods on that node can get credentials.

The failures that actually bite

Silent fallback to the node role

This is the one from the opening, and it is invisible by design. Pod Identity works through the container credential provider, which the SDK finds by reading two environment variables EKS injects into the pod:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

An SDK that predates that provider does not error on these. It does not know what they mean, so it keeps walking the credential chain until it reaches instance metadata and picks up the node role. The application works. The permissions are wrong. Nothing complains.

Two defences, and you want both. Check the minimum SDK version list in the EKS documentation and pin base images above it, and block pod access to IMDS so falling back has nowhere to fall. The first prevents the problem; the second turns a silent success into a loud failure.

Verify from inside the pod. The ARN in the response tells you which identity is really in use:

kubectl exec -n production deploy/my-app -- aws sts get-caller-identity

The association that matches nothing

An association is an exact match on cluster, namespace and service account name. A typo in any of the three produces an association that exists, looks fine in the console, and never applies to a pod. List them and read them against your manifests:

aws eks list-pod-identity-associations --cluster-name prod-eu

Associations also only take effect for pods started after they exist. If you create one and nothing changes, restart the workload before you start debugging IAM.

Both mechanisms configured at once

During a migration you will often have a service account carrying an IRSA annotation and a Pod Identity association at the same time. Both paths can work, and which one the SDK uses depends on credential chain ordering. AWS documents that credentials found earlier in the chain keep being used even when an association exists.

Do not reason about it. Check the caller identity and check CloudTrail, then remove the one you are not keeping so nobody has to work it out again.

PackedPolicyTooLarge

Session tags, managed policy ARNs and inline session policies are compressed into a packed format with its own size limit, and six session tags on every assume is not free. The association can disable the automatic tags, at the cost of the ABAC and audit benefits that were half the reason to use Pod Identity.

A decision procedure for EKS Pod Identity vs IRSA

Work through these in order and stop at the first one that answers.

1. Does this workload run on Fargate or Windows nodes? IRSA. There is no decision to make.
2. Can you rebuild the image with a current SDK? If not, IRSA, or fix the image first.
3. Does your model require direct OIDC federation into the resource account? IRSA. Role chaining is not the same shape, even if it reaches the same bucket.
4. Will this role be used from more than one cluster, now or after the next upgrade? Pod Identity. The saving compounds every time you add or replace a cluster.
5. Do you want cluster and namespace in the CloudTrail record without extra work? Pod Identity, because of the session tags.
6. None of the above? Pod Identity for anything new, IRSA left alone where it already works.

AWS’s own guidance lands in the same place: Pod Identity for new applications on supported node types, IRSA where you already have it working or where Pod Identity cannot run.

Arguments that don’t survive contact

• “IRSA is deprecated.” It is not. AWS has said it continues to invest in it, and Fargate support alone guarantees it stays.
• “Pod Identity is less secure because the trust policy is generic.” Only if you leave it generic. Request tag conditions scope it as tightly as an IRSA trust policy, and you keep the session tags IRSA does not give you.
• “Run one mechanism per cluster, cleanly.” A mixed cluster is documented, supported, and for clusters with Fargate pods it is the permanent steady state, not a transitional mess.
• “EKS add-ons can only use IRSA.” True early on, and still repeated in older write-ups. Add-on creation and update now accept Pod Identity associations directly. Check your specific add-on rather than trusting a blog post, including this one.
• “Migrate everything, then delete the OIDC provider.” Deleting it breaks every role still trusting it, including ones you forgot. Migrate workload by workload, and delete the provider only after CloudTrail shows zero AssumeRoleWithWebIdentity calls for that issuer across a full batch cycle.

How I’d decide, and what I’d do first

• Default new workloads on EC2 or Karpenter-managed nodes to Pod Identity, and leave working IRSA alone until you have a reason to touch it.
• Add request tag conditions for anything sensitive. Portability helps a shared platform role and hurts a role that reads customer data.
• Restrict pod access to IMDS before you migrate anything, not after. It converts the silent failure into a visible one.
• Migrate with the role trusting both principals at once, so rollback is a manifest change rather than an IAM change.
• Put credential identity in your dashboards. A CloudTrail query counting calls made by node roles from workload namespaces catches the exact failure this post opened with. Amazon Managed Grafana or Grafana Cloud will render it if you already ship CloudTrail somewhere; a scheduled CLI check from a CI runner or a small VPS from Contabo or InterServer is a fine low-tech alternative.
• Codify associations in Terraform or OpenTofu from the start. They are cheap to create by hand and easy to lose track of.

Frequently asked questions

Is EKS Pod Identity replacing IRSA?

No. AWS recommends Pod Identity for new applications on supported node types and continues to support IRSA. Since Pod Identity cannot run on Fargate or Windows nodes, IRSA stays for the workloads that live there.

Can I use both in the same cluster?

Yes, and it is a documented, supported pattern. Older workloads on IRSA and new ones on Pod Identity is normal. Avoid configuring both for the same service account beyond a migration window, because which one wins depends on SDK credential chain ordering.

Why does my pod get AccessDenied after switching to Pod Identity?

Check the trust policy for sts:TagSession first. Pod Identity always sends session tags, so a trust policy allowing only sts:AssumeRole is rejected. After that, check that the namespace and service account in the association match the pod exactly.

Does Pod Identity work for cross-account access?

Yes, through role chaining. The association takes a role in the cluster account plus a target role in the resource account, and EKS chains them. It also exposes an external ID you can require in the target role’s trust policy to prevent a confused deputy problem.

Do I have to change my application code?

No, provided the SDK is recent enough. Both work through the default credential chain, so the application calls AWS as usual. The only change that bites is an SDK too old to understand the container credential provider.

How do I confirm which identity a pod is actually using?

Run aws sts get-caller-identity inside the container and read the assumed role ARN. If it names the node role rather than the one you assigned, the chain fell through. Cross-check in CloudTrail, where session tags also give you the namespace and service account for Pod Identity sessions.

The one thing worth remembering

The real question in EKS Pod Identity vs IRSA is not which is more modern. It is where you want the cluster-to-role relationship to live. IRSA puts it in IAM trust policies: precise, cluster-specific, and increasing administrative drag as clusters multiply. Pod Identity puts it in the EKS API: portable, fast to change, and it hands you the job of deciding per role how much of that portability you actually want.

Pick per workload, not per cluster. And whichever you pick, verify the identity from inside the pod, because the worst outcome here is not a permission error. It is a workload that has been running with the wrong credentials since the day you deployed it.

Need a second pair of eyes on your EKS workload identity?

Most of this work is unglamorous: reading trust policies, matching associations against manifests, proving what a pod is really authenticating as. Things I help with:

• Auditing every service account in a cluster and reporting which identity its pods actually use, not the one the manifest claims
• Planning and running an IRSA to Pod Identity migration workload by workload, with a rollback that needs no IAM change
• Writing scoped trust policies with request tag conditions, so portability is a deliberate choice per role
• Restricting pod access to IMDS and confirming nothing was silently depending on the node role
• Setting up cross-account access with target roles and external IDs, plus the CloudTrail view that proves it is used as intended
• Codifying associations and roles in Terraform or OpenTofu, so the next cluster is a variable change rather than a project

Send me a trust policy, a service account manifest, or the output of aws sts get-caller-identity from inside one of your pods, and I will tell you what it is really doing.

The post EKS Pod Identity vs IRSA: What Actually Decides the Choice appeared first on John Nessime.

By the time that ticket lands, the leak already happened. You can patch the query in an hour. What you cannot do is tell the affected garage how many other reports were wrong, how long it had been wrong, or whether anyone downloaded a CSV. There’s no log that answers those questions, because nothing ever denied anything. The database happily returned the rows. The API happily serialised them.

That’s the shape of the problem. Tenant isolation on AWS is not really about writing careful code. It’s about arranging things so that careless code fails loudly instead of quietly returning somebody else’s data.

This post walks through building a multi-tenant workshop management platform on AWS with isolation baked in from the first commit: where tenant context comes from, how the boundary changes shape per storage service, what the control plane has to own, and how you prove any of it works. The example is a shop management system, with job cards, vehicle histories, parts inventory and technician timesheets, but the patterns apply to any vertical SaaS product where one customer’s records must never touch another’s.

The failure mode that stays invisible

Most multi-tenant systems start with a tenant_id column and a convention: every query filters on it. That works right up until it doesn’t.

The convention breaks in ordinary ways. Someone adds a reporting endpoint and copies a query from a script that ran as an admin. A new join pulls in a table nobody remembered to filter. An ORM lazy-loads a relationship and the filter lives on the parent, not the child. A background job that recalculates parts margins runs without any tenant in scope at all, because it processes everything.

None of these throw. That’s the whole issue. A missing authorisation check produces a 403 you’ll notice in staging. A missing tenant filter produces a 200 with too many rows, and 200s don’t page anyone.

The fix is not more discipline. It’s moving the filter somewhere the application cannot forget it: into IAM, into the database engine, or both. AWS makes this point directly in its own SaaS guidance, and it’s the right one. If your only defence against cross-tenant reads is that developers remember, you don’t have a defence, you have a habit.

Pick the isolation model before you write code

There are three shapes, and they’re usually described as pool, silo and bridge.

• Pool. Every tenant shares the same tables, buckets and compute. Cheapest to run, cheapest to deploy, and the model where isolation has to be enforced explicitly because nothing physical separates anyone.
• Silo. Each tenant gets dedicated resources: its own database, its own bucket, sometimes its own account. Isolation is close to free, operations are not. Migrations, deploys and monitoring all multiply by tenant count.
• Bridge. Mixed. Shared compute, dedicated storage, or pooled for the standard tier and siloed for the enterprise tier that asked hard questions in procurement.

For a workshop platform, most independent garages will be small, and pooling is the only sane starting point. The trap is treating that as permanent. Sooner or later a dealer group with forty sites will ask for a dedicated database, and if you have not left room for a per-tenant routing decision, retrofitting it means rewriting your data access layer.

What I’d actually do: build pooled, but put the storage target behind a resolver from day one. A function that takes a tenant ID and returns a connection, a table name or a bucket prefix. When the first silo tenant arrives, you change the resolver, not four hundred call sites.

Where tenant context comes from

This is the part people get subtly wrong, and it undermines everything downstream. The tenant identifier must come from the authenticated identity, never from the request. Not a header, not a query parameter, not a field in the JSON body. If a client can influence the tenant ID, your isolation model is decoration.

In practice that means a custom claim in the token your identity provider issues. Amazon Cognito can carry a custom attribute for this, and so can any external IdP you federate with. The API layer reads the claim, and from that point the tenant is a fact about the caller rather than an input to the call.

Then you push that fact down into AWS itself using a session tag. When your service assumes a role, it attaches the tenant as a tag on the session. Every subsequent AWS API call made with those credentials carries the tag in the request context, where IAM policies can reference it as aws:PrincipalTag.

import boto3

def scoped_session(tenant_id, role_arn):
    sts = boto3.client("sts")
    resp = sts.assume_role(
        RoleArn=role_arn,
        RoleSessionName=f"workshop-{tenant_id}",
        Tags=[{"Key": "TenantID", "Value": tenant_id}],
    )
    c = resp["Credentials"]
    return boto3.Session(
        aws_access_key_id=c["AccessKeyId"],
        aws_secret_access_key=c["SecretAccessKey"],
        aws_session_token=c["SessionToken"],
    )

Two things make this work, and both are easy to miss. The role’s trust policy has to allow sts:TagSession alongside the assume-role action, or the call fails. And the calling principal needs permission to pass that tag. Get either wrong and you’ll spend an afternoon reading an error that sounds like it’s about the role rather than the tag.

One session tag, one role, one policy. That’s attribute-based access control, and it’s the reason ABAC scales where a role per tenant does not. Roles are a finite resource in an AWS account. Tags are not.

Tenant isolation on AWS changes shape per service

There is no single isolation control. Each service exposes a different lever, and you have to learn every one your architecture touches.

DynamoDB: the partition key is the boundary

If job cards live in DynamoDB with the tenant ID as the partition key, IAM can pin every read and write to that key. The condition key is dynamodb:LeadingKeys.

{
  "Effect": "Allow",
  "Action": [
    "dynamodb:GetItem",
    "dynamodb:PutItem",
    "dynamodb:UpdateItem",
    "dynamodb:Query"
  ],
  "Resource": "arn:aws:dynamodb:REGION:ACCOUNT:table/JobCards",
  "Condition": {
    "ForAllValues:StringEquals": {
      "dynamodb:LeadingKeys": ["${aws:PrincipalTag/TenantID}"]
    }
  }
}

Now a Query that omits the tenant partition key doesn’t return other tenants’ job cards. It gets denied. That’s the behaviour you want: loud, logged, and impossible to miss in CloudTrail.

Two caveats worth knowing before you commit. Scan operations don’t have a leading key to constrain, so granting dynamodb:Scan alongside this condition undermines the whole arrangement. And global secondary indexes have their own key structure, so an index whose partition key isn’t the tenant needs separate thought. Design the access patterns so that no query ever needs to look across tenants, and this stops being a problem.

S3: two surfaces, not one

Vehicle photos, inspection PDFs and signed job sheets go to S3 under a per-tenant prefix. The mistake is scoping only the object actions and leaving ListBucket open, which lets a tenant enumerate every other garage’s filenames even without reading them. Filenames leak plenty: customer names, registration plates, invoice numbers.

Object actions are scoped through the resource ARN. Listing is scoped through the s3:prefix request condition. You need both statements, because they protect different operations.

[
  {
    "Effect": "Allow",
    "Action": "s3:ListBucket",
    "Resource": "arn:aws:s3:::workshop-tenant-files",
    "Condition": {
      "StringLike": {
        "s3:prefix": ["${aws:PrincipalTag/TenantID}/*"]
      }
    }
  },
  {
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:PutObject"],
    "Resource": "arn:aws:s3:::workshop-tenant-files/${aws:PrincipalTag/TenantID}/*"
  }
]

The policy variable in the resource ARN is doing real work there. One policy, every tenant, no template rendering at request time.

Aurora PostgreSQL: row-level security, and the trap in it

Relational data is where most workshop platforms actually live, because job cards, parts lines and labour rates are relational. IAM cannot see inside a table, so the boundary moves into PostgreSQL itself via row-level security.

ALTER TABLE job_cards ENABLE ROW LEVEL SECURITY;
ALTER TABLE job_cards FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON job_cards
  USING (tenant_id = current_setting('app.tenant_id', true))
  WITH CHECK (tenant_id = current_setting('app.tenant_id', true));

USING controls which rows are visible to reads, updates and deletes. WITH CHECK controls what can be written. Without the second clause, a tenant can read only its own rows but insert a row stamped with someone else’s tenant ID. The first protects the read path, the second protects the write path, and you want both.

Now the trap, and it’s the one that gives teams false confidence. PostgreSQL superusers and roles carrying BYPASSRLS ignore row security entirely, and by default so does the table owner. If your application connects as the same role that ran the migrations, your policies are not in effect and your tests pass anyway. That’s why FORCE ROW LEVEL SECURITY is in the snippet above, and why the application should connect as a dedicated non-owner, non-superuser role.

The second trap is connection reuse. Set the tenant with SET LOCAL inside a transaction, or with set_config using the transaction-local flag. Plain SET persists for the life of the connection, and a pooled connection outlives the request. That’s how one garage’s context ends up serving the next garage’s query.

Test row-level security as the application role, against a pooled connection, with at least two tenants in the table. Testing as the owner tells you nothing.

Compute isolation is a different question

Data isolation stops one tenant reading another’s records. It does nothing about one tenant consuming everyone’s capacity.

Workshop platforms have a specific version of this. End of month, every garage runs its invoicing and MOT reminder batch at roughly the same time. A dealer group importing three years of service history will happily saturate a shared worker pool while forty independents wait for their job cards to load.

Levers worth knowing, roughly in order of how much they cost you:

• API Gateway usage plans, keyed per tenant, to cap request rates at the edge before anything expensive runs.
• Separate queues, or at minimum separate consumer concurrency, for bulk imports versus interactive requests. Bulk work should never share a lane with a screen someone is waiting on.
• Reserved or provisioned concurrency on the Lambda functions serving interactive paths, so a batch surge cannot starve them.
• Dedicated compute for premium tenants. This is silo by another name, and it’s the honest answer when a customer’s load profile genuinely doesn’t fit the pool.

Be honest with yourself about which problem you’re solving. Throttling is not isolation. It limits blast radius, it doesn’t create a boundary.

What the control plane has to own

Separate the control plane from the application plane early. The control plane manages tenants; the application plane serves them. Mixing the two is how a bug in the onboarding flow ends up with production credentials.

Onboarding a new workshop is a sequence, and it should be a single idempotent workflow rather than a checklist someone follows:

1. Generate a non-guessable tenant identifier. Lowercase alphanumeric, no customer name in it, because it will end up inside resource ARNs and key prefixes.
2. Write the tenant record: tier, isolation model, storage target, status.
3. Provision identity. User pool group or IdP mapping, with the tenant claim wired in.
4. Provision storage. For a pooled tenant that’s a prefix and a seeded row. For a siloed one it’s real infrastructure, which is why this step must be asynchronous.
5. Apply tags used for cost allocation and reporting.
6. Run a verification step that proves the new tenant can reach its own data and cannot reach a canary tenant’s data.

Step six is the one teams skip. It’s also the only step that tells you the previous five worked.

On per-tenant cost: activated cost allocation tags attribute anything that is a distinct tagged resource, which covers siloed tenants nicely. Pooled resources will not split by themselves, because a shared table doesn’t know which garage caused which read. If you need per-tenant margin, emit consumption as a metric dimension from the application: request counts, storage bytes, document pages processed. Tools like Vantage or CloudZero can allocate shared spend afterwards, but only from the signal you produce. Nothing recovers attribution you never recorded.

The AWS SaaS Builder Toolkit is worth a look here. It codifies control plane concepts as CDK constructs and will save you real time on onboarding plumbing. Read its own guidance first: the project describes itself as sample code, and expects you to review security fit before production. That’s a fair description rather than a warning label, but treat it as a starting point, not a finished platform.

Proving isolation actually holds

An isolation model you haven’t tried to break is a design document, not a control.

The tests that earn their keep are negative ones. Seed two tenants. Authenticate as the first. Then deliberately do the wrong thing: request the second tenant’s job card by ID, list the second tenant’s S3 prefix, run a query with the session variable set to the wrong value. Every one of those should fail, and the test should assert on the failure.

• Run the cross-tenant suite on every pull request, not nightly. It’s the regression that matters most and the one most likely to be introduced by an innocent refactor.
• Use the IAM policy simulator to check a policy change before it ships, particularly when someone widens a resource ARN.
• Turn on IAM Access Analyzer so external access grants surface without anyone having to notice them.
• Audit the database on a schedule: tables with RLS enabled but no policy attached, tables missing FORCE, application roles that have quietly acquired ownership or BYPASSRLS during an incident.
• Alert on AccessDenied volume per tenant. A spike is either a bug you introduced or someone probing, and both are worth knowing about.

Keep a permanent canary tenant in every environment, including production, holding nothing but synthetic data. Every negative test targets it. It costs almost nothing and it means your isolation tests never need real customer records.

Troubleshooting the failures you’ll actually hit

AccessDenied on a policy that looks correct

Almost always the session tag isn’t present. If the tag is missing from the request context, the condition can’t match and the statement doesn’t apply. Call sts:GetCallerIdentity with the scoped credentials and confirm you’re on the assumed role you think you are, then check CloudTrail for the AssumeRole event and look at whether the tag was actually passed. Tag keys are case sensitive, and TenantId is not TenantID.

Queries return zero rows instead of the right rows

Classic RLS symptom. The session variable is unset, so the policy predicate compares against null and nothing matches. Check current_setting inside the same transaction as the query, not in a separate connection from your SQL client. Empty results are the safe failure here, which is exactly why they’re easy to misread as a data problem.

One tenant intermittently sees another’s data

Intermittent means state reuse. Look at connection pooling first, then at any per-request context stored in thread-local or async-local storage that isn’t reset when the request finishes. If you’re using a proxy in front of the database, understand how it handles session state, because some proxies pin a connection to a client once session-level settings are detected, which changes the behaviour you tested against.

Isolation works in the API but not in background jobs

Because the job has no request, so it has no token, so it has no tenant. Whatever runs asynchronously needs the tenant carried on the message and a session assumed per tenant when the work is processed. A worker that loops over all tenants with admin credentials is the single most common place isolation quietly stops applying.

Session tags don’t survive a second AssumeRole

Session tags are not automatically carried forward when you chain roles unless they were marked transitive. If your architecture hops through more than one role, this is where the tenant context evaporates. Flatten the chain if you can; if you can’t, mark the tag transitive deliberately and document why.

Common mistakes

• Taking the tenant ID from a request header or body instead of the authenticated token.
• Using the customer’s name or a sequential integer as the tenant identifier, then putting it in bucket prefixes where it becomes both guessable and enumerable.
• Scoping S3 object actions but leaving bucket listing wide open.
• Running the application as the PostgreSQL table owner, so RLS is enabled and silently inert.
• Writing a USING clause with no WITH CHECK, leaving the write path open.
• Creating one IAM role per tenant and discovering the account ceiling somewhere around the point the business gets interesting.
• Assuming cost allocation tags will attribute pooled spend. They won’t, and by the time you need the numbers the history is gone.
• Testing isolation only through the UI, where the frontend is already sending the right tenant every time.

Best practices worth the effort

• Derive tenant context from the token, propagate it as a session tag, and never let application code choose it.
• Enforce the boundary at the layer below your code: IAM conditions for AWS resources, RLS for relational rows.
• Put storage targets behind a resolver so moving a tenant from pool to silo is a configuration change.
• Make onboarding one idempotent workflow that ends in a verification step.
• Emit tenant as a dimension on logs and metrics from the start, so cost and performance questions stay answerable.
• Keep a canary tenant and run cross-tenant negative tests in CI on every change.
• Put a WAF or edge layer such as Cloudflare in front of tenant subdomains, and keep tenant routing decisions out of the origin application where you can.

Frequently asked questions

Is a shared database with a tenant_id column ever acceptable?

Yes, provided the column is enforced by the engine rather than by convention. A shared table with row-level security, forced, queried by a non-owner role, is a real boundary. A shared table where every query is expected to include the filter is not. The schema is the same; the guarantee is completely different.

ABAC or dynamically generated IAM policies?

ABAC for most cases. One role, one policy, tenant supplied per session, and nothing grows as you add customers. Dynamic policy generation, sometimes called a token vending machine, earns its place when a single policy genuinely cannot express the rule, such as when per-tenant resource names have to be injected rather than a key prefix. The cost is that you now own the correctness of a policy generator, plus per-request latency, and session policies have a size ceiling you can hit.

Should each tenant get its own AWS account?

It’s the strongest boundary available and the most expensive to operate. For a workshop platform serving independent garages it’s overkill. It becomes reasonable when a customer’s contract, regulator or data residency requirement makes shared infrastructure a non-starter, and at that point you’re pricing it as a premium tier rather than absorbing it.

How do I handle a user who works at two workshops?

Model it as one identity with multiple tenant memberships and an explicit active tenant per session, rather than a token carrying a list. The active tenant becomes the session tag. Switching workshops means a new session, which is exactly the behaviour you want because it makes the switch visible in your audit trail.

Does row-level security hurt query performance?

It adds a predicate the planner has to satisfy, so the answer depends on your indexes. Index the tenant column, and index it as the leading column of composite indexes that support your common filters. Compare plans as the application role before and after enabling policies, because a plan captured as the owner may not reflect what the application actually runs.

Where does application-level authorisation fit?

Alongside, not instead. Tenant isolation answers “which organisation’s data is this”. Authorisation answers “may this technician void an invoice”. Different questions, different layers. A policy engine such as Amazon Verified Permissions handles the second cleanly, and keeping them separate stops role logic creeping into your isolation boundary.

Can I retrofit isolation onto a platform that already has tenants?

You can, and it’s tedious rather than impossible. Enforce at the database first, because that’s where the leak actually happens: table by table, FORCE enabled, application moved to a non-owner role. Then move the tenant ID out of request payloads and into the token. Add IAM conditions last, since they’re the least likely source of a live leak. Expect to find at least one background job with no tenant scope at all.

The one thing to remember

Tenant isolation on AWS works when the boundary sits below your application code, in a layer that denies rather than trusts. IAM conditions on session tags for AWS resources, forced row-level security for relational data, and a control plane that verifies a new tenant is properly fenced before anyone logs in.

Retrofitting that onto a running multi-tenant platform is possible but grim, because you’re doing it while real workshops have real data in the system. Doing it on day one costs a week. That’s the entire trade, and it’s not a close call.

Need a second pair of eyes on your multi-tenant architecture?

Most of the isolation problems I see are not exotic. They’re a missing WITH CHECK clause, an application connecting as the table owner, or a background job nobody scoped. Things I can help with:

• Reviewing an existing multi-tenant design and finding where the boundary is enforced by convention rather than by the platform.
• Implementing ABAC with STS session tags across DynamoDB, S3 and Aurora, including the trust policy wiring that trips people up.
• Setting up PostgreSQL row-level security correctly, with forced policies, a dedicated application role and pooling that doesn’t leak session state.
• Building a tenant onboarding workflow that provisions identity, storage and tagging idempotently and verifies itself.
• Writing the cross-tenant negative test suite and wiring it into CI so isolation regressions fail the build.
• Adding per-tenant usage metering so cost, performance and tier decisions rest on data instead of guesses.

Send me a policy document, an RLS definition or a CloudTrail AccessDenied event and I’ll tell you what it’s actually enforcing.

The post Tenant Isolation on AWS: Building a Multi-Tenant Workshop Platform That Doesn’t Leak appeared first on John Nessime.

That question is what actually bites when you run infrastructure for twenty clients. Not a breach. An audit, or a resignation. Someone gives notice on a Friday and you spend Monday working out which accounts they still hold keys to, because the credential everybody shared was never attached to a person. You can’t revoke a person from a password. You can only rotate it and hope you got all twenty.

This post covers how to design client AWS account access so that humans authenticate once against something you control, every client account holds a role that trusts exactly one thing, and every API call in a client’s CloudTrail can be traced back to a named engineer. It’s organised by failure family, because the mechanics are well documented and the failure modes are not.

The shape: one identity plane, twenty separate trust decisions

The instinct that causes all the trouble is putting credentials in the middle. Client gives you a login, you store it somewhere shared, engineers take it out when they need it. Everything downstream of that decision is damage control.

The shape that works is boring. One AWS account whose only job is identity. Your engineers exist there, once, in IAM Identity Center or federated from whatever directory you already pay for. Each client account holds an IAM role whose trust policy names one principal in that account and requires an external ID. Nobody stores a client credential, because there is no client credential. There’s a role your engineers can ask STS for a short-lived session against, and a client who can delete that role whenever they like.

There is a second design worth knowing about: inviting client accounts into your own AWS Organization. It’s genuinely better in some ways. You get service control policies, resource control policies, and centralised root access management across the estate, all of which are organisation-scoped features you simply cannot use on an account you don’t own. It’s also a much bigger commitment. You take on consolidated billing, the client’s spend flows through you, and pulling an account back out of an organisation at the end of a contract is a real project rather than a policy change.

For most consultancies and managed service providers, the client keeps their own organisation and you get a role. That’s the model the rest of this assumes.

Failure family 1: credentials you can’t revoke

The shared IAM user with an access key is the default because it works immediately. It keeps working right up until one of three moments: an engineer leaves, a client asks who did something, or a laptop gets stolen.

Access keys have no expiry. They’re bearer tokens with an indefinite lifetime, and the only way to invalidate one is to go into each account and delete it. At twenty accounts that’s twenty deliberate actions performed under time pressure, and the one you forget is the one that matters. A role session, by contrast, expires on its own. That’s the entire argument.

A shared password manager still earns its place, just not for admin keys. Bitwarden or 1Password is the right home for per-client external IDs, account numbers, role ARNs and the client-side contacts you need when something breaks at two in the morning. Metadata, not credentials.

Failure family 2: a trust policy that trusts more than you meant

This is the invisible one. The role gets created, the access works, everyone moves on, and the trust policy quietly grants far more than anyone intended.

Two mistakes travel together. The first is naming your account root in the principal. Writing arn:aws:iam::111122223333:root does not mean “the root user of my account.” It means any principal in that account that has permission to call AssumeRole. Every engineer, every Lambda function, every CI runner, every future role you haven’t created yet. It delegates the decision to your own account’s IAM, which is fine when your IAM is tidy and quietly catastrophic when it isn’t.

The second is skipping the external ID. AWS calls the problem it solves the confused deputy: you are a third party that holds the ability to assume roles in many customer accounts, and without a shared secret in the condition, one customer who learns another customer’s role ARN could persuade your systems to act against it. The external ID is the value that makes each trust relationship specific rather than generic.

Generate it yourself, one per client, as a UUID or similar unguessable value. The client must not be the one choosing it, because if they can choose it they can choose someone else’s, and the control does nothing.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/MspOperationsAccess"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
        }
      }
    }
  ]
}

The principal names one role, not the account. If that role is compromised you have one thing to fix. If the account is named, you have an inventory problem you didn’t know you had.

Failure family 3: CloudTrail says the role, not the human

You’ve done everything above and a client emails asking who deleted the staging load balancer. You open their CloudTrail and every event attributes to the same assumed role. The role is shared by your whole team by design, so the log tells you your company did it. That’s not what they asked.

There are two knobs and they are not equivalent. The role session name is chosen freely by whoever calls AssumeRole, appears in the assumed-role ARN, and is trivially set to anything. It’s useful, but it’s a label, not evidence. Source identity is the one that carries weight: once set on a session it cannot be changed, it persists across chained role sessions, and it lands in CloudTrail in the session context of every subsequent call.

The catch is that the role’s trust policy has to permit sts:SetSourceIdentity alongside sts:AssumeRole, and you can require it with a condition so a session without one is refused outright.

"Action": [
  "sts:AssumeRole",
  "sts:SetSourceIdentity"
],
"Condition": {
  "StringEquals": {
    "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
  },
  "StringLike": {
    "sts:SourceIdentity": "*@yourcompany.com"
  }
}

If your engineers arrive through SAML federation, the identity provider can populate source identity from a directory attribute such as email, which means nobody has to remember to pass it. For direct CLI use it’s an explicit flag:

aws sts assume-role 
  --role-arn arn:aws:iam::444455556666:role/MspEngineerAccess 
  --role-session-name acme-investigation 
  --external-id 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02 
  --source-identity jane.doe@yourcompany.com

Getting this right is what turns “we take security seriously” into a sentence you can back with a log query. It is also, in my experience of what clients actually ask for, the single most persuasive thing on the list.

Failure family 4: one identity, twenty blast radii

A hub account is a concentration of risk. That’s the trade you’re making, and it’s worth making, but only if the hub is held to a higher standard than the accounts it reaches into.

Practically: not every engineer gets permission to assume every client’s role. Group permissions in the hub so the people on the Acme engagement reach Acme and nothing else. Use per-client role names rather than one shared name across all twenty, so a policy in the hub can’t wildcard its way into a client you stopped working with a year ago.

Tier the roles by what they’re for, not by who’s asking. A read-only role for investigation covers most of the work most days. A change role carries the write permissions the engagement actually needs. Break-glass sits separate, with a short maximum session duration and an alarm on every use.

Two extras are worth knowing about even if you don’t reach for them immediately. Session policies let the caller narrow a session further at assume time, which is useful for automation that only needs one bucket. Permissions boundaries cap what any role or user created inside the client account can do, which matters when part of your job is creating IAM there.

If your team already routes through a business VPN or zero trust gateway with stable egress addresses, a source IP condition on the trust policy is a cheap extra layer. NordLayer, Tailscale and Cloudflare’s zero trust products all give you predictable egress. Be honest about the cost, though: the moment something runs from CI, a client’s network or a laptop on hotel wifi, an IP condition becomes a support ticket. I’d add it once everything else works, not before.

Failure family 5: standing administrator access nobody remembers granting

Somewhere in the twenty accounts there is a role with AdministratorAccess attached, created during a migration, described in the ticket as temporary. It’s been eighteen months.

Here’s the honest part: AWS does not give you a native just-in-time approval workflow for cross-account role assumption. There’s no built-in “request four hours of write access, get approved in Slack, access expires automatically” for this pattern. You either build that on top of STS and your ticketing system, or you accept standing access and compensate with tight session durations, aggressive monitoring and a genuine periodic review. Both are defensible. Pretending you have JIT when you have a wiki page is not.

The cheap version that actually helps: set the maximum session duration on client roles to something short, alert on assumption of the break-glass role, and run a quarterly pass over service last accessed data to strip permissions nobody has used. IAM can generate that report per role.

# Kick off the report, then fetch it with the returned job ID
aws iam generate-service-last-accessed-details 
  --arn arn:aws:iam::444455556666:role/MspEngineerAccess

aws iam get-service-last-accessed-details --job-id JOB_ID_FROM_ABOVE

Run that in each client account and the conversation about narrowing permissions stops being theoretical. You’re not guessing what the engagement needs; you’re reading it.

Failure family 6: offboarding that only half happens

The whole point of the hub is that offboarding is one action. Disable the person in your directory and every path into every client account closes at once, because there was never a credential in a client account with their name on it.

That’s true only if nothing crept in sideways. Things that creep in sideways:

• IAM users created directly in a client account during an incident, because it was faster
• Access keys minted under a role by automation and never inventoried
• A client’s own console user shared with your team “just for the migration”
• Third party tools connected with their own cross-account roles that outlived the trial
• A departing engineer’s laptop with cached SSO tokens and stale profiles still on disk

IAM Access Analyzer is the tool that finds most of the first four. Its external access findings show every principal outside the account that can reach into it, which is exactly the list a client wants to see, and it’s also the list that reveals the monitoring vendor nobody remembers signing up for. For the laptop, whatever your hardware disposal process is, make sure it involves an actual wipe rather than a factory reset; tools like O&O SafeErase exist for that reason and are cheap next to the alternative.

Rolling client AWS account access out across twenty accounts

Doing this by hand twenty times guarantees twenty subtly different configurations. Make it a template and a checklist.

1. Write one CloudFormation template that creates the role, its trust policy and its permissions, parameterised by external ID and your operations account ID.
2. Generate a unique external ID per client and store it with the client’s account metadata, not in the template.
3. Send the client a launch stack link rather than asking for credentials. They deploy it in their own account, in their own console, and can read exactly what they’re granting.
4. Record the account ID, role ARN, external ID location, engagement scope and review date in a single inventory file in version control.
5. Generate everyone’s local AWS profiles from that inventory so nobody hand-edits a config file.
6. Verify each one with a call that proves identity and nothing else before you touch anything real.

The generated profile chains from your federated session into the client role. The AWS CLI reads these from the shared config file and handles the assume-role call for you:

# ~/.aws/config
[profile msp-hub]
sso_session = company
sso_account_id = 111122223333
sso_role_name = MspOperationsAccess
region = eu-west-1

[profile acme-readonly]
role_arn = arn:aws:iam::444455556666:role/MspEngineerReadOnly
source_profile = msp-hub
external_id = 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02
role_session_name = jane-doe
duration_seconds = 3600
region = eu-west-1

Then the two-second sanity check that has saved me from more mistakes than any other command:

aws sts get-caller-identity --profile acme-readonly

It returns the account ID and the assumed-role ARN. Read the account number before you run anything destructive. Twenty profiles in one config file is exactly the situation where muscle memory picks the wrong one.

For scheduled work that runs without a human, the same trust relationship serves a Lambda function or an EC2 instance role in your hub. A low-cost VPS from Contabo or InterServer will also do the job, but note the trade: an instance profile inside AWS gets rotated credentials automatically, while a box outside AWS needs a long-lived credential to bootstrap from, which reintroduces exactly what this design removes. Inside AWS is the cleaner answer.

Troubleshooting the failures you will actually hit

AccessDenied on AssumeRole

Three separate causes produce nearly the same message, and they need different fixes. Either the client’s trust policy doesn’t name your principal, or the external ID you sent doesn’t match what the condition expects, or your own identity in the hub lacks permission to call AssumeRole on that role ARN. Check them in that order, because the first two require the client and the third doesn’t.

Switch Role in the console silently fails

This one wastes an afternoon if you don’t know it. The console’s switch role feature does not support roles that require an external ID. It’s a documented limitation, not a bug in your policy. Engineers who need console access to those accounts should reach them through federation rather than the switch role box.

Sessions expiring after an hour when you asked for longer

When one role session assumes another role, that’s role chaining, and a chained session is capped at one hour regardless of the role’s maximum session duration setting. Since your engineers assume the client role from an already-assumed session in the hub, you’re chaining by definition. Plan around the hour rather than fighting it.

An authorization message you can’t read

Some denials come back with a long encoded blob attached. STS will decode it if your identity has permission to, and the decoded version usually names the exact statement that denied you.

aws sts decode-authorization-message --encoded-message <encoded-blob>

Common mistakes

• Naming the account root as the trust principal instead of the specific operations role
• Letting the client pick the external ID, or reusing one value across several clients
• Treating the role session name as an audit trail when it’s a caller-supplied string
• Asking a client for console credentials because a template felt like too much effort
• Granting AdministratorAccess for a migration and never revisiting it
• Building the hub properly and then leaving one legacy IAM user in one account as a fallback
• Assuming that removing someone from your directory also removed the keys they created by hand

Best practices worth the effort

• One identity plane, federated, with MFA enforced at the point the human authenticates
• A unique unguessable external ID per client, generated by you and stored with client metadata
• Source identity required in the trust policy so CloudTrail names a person, not a role
• Roles tiered by purpose: read-only for investigation, change for the engagement, break-glass alarmed
• Client-owned policies, deployed by the client from your template, revocable by the client in one action
• A version-controlled inventory of every account, role, scope and review date
• Access Analyzer external access findings reviewed on a schedule, in every client account

FAQ

Can’t I just ask each client to create an IAM user for me?

You can, and it’ll work on day one. The problem is day four hundred. That user’s access key never expires, it isn’t attached to any individual on your team, and revoking it means someone in the client’s organisation remembering to act. A role costs the client the same five minutes to create and hands them a control they can exercise unilaterally.

Do I need a different external ID for every client?

Yes. A shared external ID across your client base defeats the purpose, because the value’s entire job is to be specific to one trust relationship. Generate a UUID per client, treat it as a secret, and never let the client choose it.

Should I bring client accounts into my own AWS Organization?

Only if you’re also taking on their billing and the commercial relationship supports it. Inside your organisation you get service control policies, resource control policies and centralised root access management, which are real advantages. Outside it, those stay the client’s tools and your job is to recommend them well. Most consulting engagements sit outside.

How do I prove a departed engineer no longer has access?

Show the directory record disabling them, then show that no IAM user or access key in the client account carries their name, then show Access Analyzer external access findings listing only your operations role as an external principal. Three artefacts, all exportable. That’s the evidence pack.

Does MFA carry across the cross-account role assumption?

Don’t assume it does. The reliable place to enforce MFA is where the human authenticates in your identity plane, and the reliable control at the client boundary is the combination of a narrowly scoped principal and the external ID condition. If you want an MFA condition in the client’s trust policy as well, test it end to end against your actual federation path before you rely on it, because the request context differs between direct assumption and chained sessions.

What’s the minimum viable version if I only have three clients?

A hub account with federated engineers, one role per client scoped to a named principal with a unique external ID, and source identity required. Skip the tiering, skip the IP conditions, skip the JIT ambitions. Those three things give you revocation, attribution and a clean answer for the auditor, which is most of the value.

The one thing to take away

Client AWS account access is not really a permissions problem. It’s a revocation and attribution problem wearing a permissions costume. Any design where a credential exists that isn’t tied to a specific human will fail you at exactly the wrong moment, and no amount of policy tuning fixes that, because the gap isn’t in the policy.

Put the humans in one place you control. Put a narrowly scoped, externally identified role in each client account. Require source identity so the log names a person. Everything else on this page is refinement; those three are the structure. And the pleasant side effect is that “here’s how you revoke us, unilaterally, in one action” is a genuinely good thing to be able to say in a sales conversation.

Need a second pair of eyes on your client access model?

This is the kind of work I do for consultancies and managed service providers who’ve grown past the point where a shared vault is survivable. Specifically:

• Reviewing existing cross-account trust policies for over-broad principals and missing external IDs
• Designing a hub account and federation model that scales past a handful of clients without per-client special cases
• Building the CloudFormation or Terraform template clients deploy themselves, so onboarding stops involving credentials
• Turning on source identity end to end, including the identity provider side, so CloudTrail attributes to named engineers
• Producing the offboarding evidence pack clients and auditors ask for, and automating it so it isn’t a scramble
• Narrowing long-standing administrator roles down using service last accessed data, without breaking the engagement

If you want a concrete starting point, send me one trust policy and the AccessDenied message it produces. That’s usually enough to tell where the model is leaking.

The post Twenty Client AWS Accounts and Not One Shared Admin Credential appeared first on John Nessime.

The expensive stuff is not the running instances. It’s the access keys nobody can attribute, the CloudTrail trail that quietly stopped delivering months ago, the Terraform state that no longer matches reality, and the region somebody spun a proof of concept up in two years ago. None of that comes up on a kickoff call. All of it comes up in week three, after you’ve quoted.

So I run an AWS account health check before I give anyone a number. It’s read-only, it takes a couple of hours, and it’s the cheapest insurance in consulting. This post is the actual check: the six passes, the commands, what each finding does to the estimate, and what to do when the client won’t give you the access to run it.

Why the AWS account health check comes before the quote

A quote is a bet on how much unknown there is. You’re pricing the work in the brief plus everything you’ll find while doing it, and the second half is invisible until you look. Two accounts can look identical and differ by a factor of three in effort. The difference is nearly always archaeology: how much has to be reverse engineered before you can safely change anything.

There’s a second reason that matters more than pricing. If you touch an account that already had a compromised key sitting in it, the incident becomes yours by proximity. Establishing what the account looked like before you arrived isn’t paranoia. It’s the only version of events you’ll be able to evidence later.

The access I ask for, and what a refusal tells me

I ask for a cross-account IAM role that the client creates and my account assumes. Not an IAM user. Not keys in an email. A role, with an external ID the client generates.

The external ID solves one specific problem. Without it, a trust policy that names my account can be used by anyone who talks me into assuming a role on their behalf, and I end up acting inside a third party’s account without knowing it. AWS calls this the confused deputy problem. It’s the piece of cross-account setup people skip most often.

On that role I ask for two AWS managed policies: SecurityAudit and ViewOnlyAccess. The pairing is the point. ViewOnlyAccess lets you enumerate resources across services, so you can see what exists. SecurityAudit lets you read configuration and policy documents, so you can see how it’s wired together. Neither alone gives you the picture, and neither lets you change anything. Billing is separate again: Cost Explorer read access has to be added explicitly and the feature has to be enabled in the account.

The reply to that request tells you as much as the check does.

• “Here’s the role ARN and external ID” means somebody there understands AWS identity. Good sign.
• “I’ll just make you an admin user, easier” means nobody is managing access. Expect that looseness everywhere else.
• “Our previous developer set it up and we can’t reach him” means you’re inheriting credentials nobody can account for. Price the rotation work as its own line.
• “We’d rather not grant access before signing” is fair. Offer a paid discovery engagement. If they won’t pay for two hours of assessment, they won’t pay for the fix.

The six passes

Each pass answers one question. Everything here is read-only, and the order matters, because each pass changes how you read the next.

Pass 1: Who can do what

Confirm where you actually are first. That sounds trivial until you’re in an organization with a dozen similarly named accounts.

aws sts get-caller-identity
aws iam list-account-aliases
aws organizations describe-organization

If the third command fails because the account isn’t in an organization, that’s a finding by itself. Standalone means no service control policies, no organization trail, and no central place to revoke anything.

Then the credential surface. The summary gives counts; the credential report gives per-principal detail, including when each password and key was last used.

aws iam get-account-summary

aws iam generate-credential-report
aws iam get-credential-report --query Content --output text | base64 --decode > credential-report.csv

The report generates asynchronously, so the second call may need a retry the first time. What I read from it: how many IAM users exist at all, how many have keys that have never been used, and how many have keys last used a long time ago that are still active.

An unused key is worse than a busy one. A key in daily use is attributable to something. A key last used eighteen months ago and still enabled belongs to a person or script nobody remembers, which means nobody will notice when someone else starts using it.

Then check whether anyone has ever asked this question before you:

aws accessanalyzer list-analyzers

IAM Access Analyzer offers external access analyzers, which flag resources shared outside a zone of trust you define, and unused access analyzers, which surface unused roles, keys and passwords. Unused access analysis is billed per IAM role and user analyzed per month, so it isn’t free, but it answers the least privilege question far faster than reading policy documents by hand. An empty list means nobody has ever asked this account who can reach what.

Root closes the pass. In an AWS Organizations setup, centralized root access management lets the management account or a delegated IAM administrator remove root credentials from member accounts entirely, so there’s no root password, key or MFA device to manage per account. Live root credentials on every member account is a real finding, and one of the few security fixes that’s genuinely quick.

Pass test: you can name every principal that can reach production, and say when each last did.

Pass 2: Would anyone notice

This is the pass that most often changes my number, because it’s the failure that stays invisible until it isn’t. Logging that exists but has silently stopped working is more dangerous than no logging, since everybody believes they’re covered.

aws cloudtrail describe-trails 
  --query 'trailList[].{Name:Name,MultiRegion:IsMultiRegionTrail,Org:IsOrganizationTrail}'

aws cloudtrail get-trail-status --name TRAIL_NAME_HERE

aws configservice describe-configuration-recorder-status

The first command tells you a trail is configured. The second tells you whether it’s delivering, which is the part that matters. A trail whose last delivery failed because someone edited the destination bucket policy still shows up in the console as a configured trail. The status call is where the truth is.

Multi-region matters for the same reason region sprawl does: a single-region trail records nothing about the region where somebody is quietly running compute. Check GuardDuty too, and how long logs are actually retained. Ninety days of retention against a one-year obligation is a project, not a task, and it belongs in the quote rather than in week four.

One naming trap worth knowing: AWS renamed the original Security Hub to Security Hub CSPM and launched a separate, newer service under the Security Hub name. When a client says “we have Security Hub,” ask which one, because the findings format and the configuration are not the same thing.

Pass test: a trail is delivering right now, covers every region, and its logs are kept for at least as long as the business needs them.

Pass 3: How wide the blast radius is

People forget regions. Not occasionally, constantly. A demo in Singapore, a test bucket in Ireland, a Lambda deployed from a tutorial that defaulted to us-east-1.

for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  count=$(aws ec2 describe-instances --region "$r" 
    --query 'length(Reservations[].Instances[])' --output text)
  echo "$r $count"
done

Left without --all-regions, describe-regions returns the regions enabled for the account, which is exactly what you want. Adding the flag pulls in disabled regions too, and the per-region calls against those just error out.

Run the same loop shape for anything carrying state or cost: RDS instances, load balancers, EBS volumes, NAT gateways. NAT gateways are the classic. They bill hourly whether or not anything routes through them, and a forgotten one in a region nobody visits is pure waste. Then the two exposure checks I never skip:

aws ec2 describe-security-groups 
  --filters Name=ip-permission.cidr,Values=0.0.0.0/0 
  --query 'SecurityGroups[].{Id:GroupId,Name:GroupName,VpcId:VpcId}'

aws s3api list-buckets --query 'Buckets[].Name' --output text

For each bucket that matters, check its public access block configuration and its policy. An open security group on 443 in front of a load balancer is fine. An open security group on a database port is a different conversation, and you want that conversation before you own the account.

Pass test: you have a list of every region with resources in it, and nothing on that list surprises the client.

Pass 4: Whether any of this is described in code

The question isn’t “do they use Terraform.” It’s whether the code still describes the account. Those diverge quietly. Somebody makes an urgent console change during an incident, never brings it back into code, and from then on the repo is fiction that plans cleanly right up until the day it destroys something.

terraform plan -refresh-only

terraform plan -detailed-exitcode

The refresh-only plan shows where real infrastructure has drifted from state without proposing to change anything. The detailed exit code version is the one for scripts: it exits 0 for no changes, 1 for an error and 2 when there’s a diff, so you can gate on drift instead of reading output by eye.

Three things I want to know here:

• What fraction of live resources are managed by code? Half a codebase is often worse than none, because it creates false confidence about which half.
• Where does state live and who can write to it? Remote state in a versioned, locked S3 bucket is a different risk profile from a state file on a laptop.
• Does the pipeline that applies changes still run? Credentials that expired six months ago mean everything since went in by hand.

Tagging belongs here too. With nothing tagged you can’t attribute cost, can’t scope a change safely, and can’t tell a production resource from an abandoned experiment. Untagged environments add hours to everything that follows.

Pass test: a refresh-only plan comes back clean, or the drift is a short list you can read out loud.

Pass 5: What shape the spend is in

I don’t care much about the total. I care about the shape: which services dominate, whether the curve is flat or climbing, and whether anything jumped without an explanation.

aws ce get-cost-and-usage 
  --time-period Start=$(date -d '6 months ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=SERVICE

The date -d syntax is GNU coreutils; on macOS use the BSD equivalent or a fixed start date. Worth knowing before you loop this: the Cost Explorer API bills per paginated request, unlike the console view, so a wide query across several dimensions is a small but real charge on the client’s bill. Keep it narrow and mention it in your notes.

• Data transfer in the top three usually means an architecture problem rather than a rate problem. Cross-AZ chatter and NAT gateway egress are the usual suspects.
• A step change with no matching launch deserves a direct question. Sometimes it’s a feature. Sometimes it’s a runaway job. Occasionally it’s mining on a stolen key.
• No budgets and no anomaly monitors means nobody finds out about a spike until the invoice arrives.
• Heavy commitment purchases against workloads you’re about to re-architect are a constraint on your design, and need to be on the table before you propose anything.

If the client wants ongoing cost visibility past what Cost Explorer offers, platforms like Vantage or CloudZero sit on the Cost and Usage Report and give per-team allocation and unit economics that native tooling makes awkward. Raise it as a separate line, not part of the engineering work.

Pass test: you can explain the top three cost lines in a sentence each, and account for every month-on-month jump.

Pass 6: What happens at two in the morning

Short pass, about escalation. What support plan is the account on, and would anything wake a human up?

Support plan matters more than people expect. On Basic you get the service limits checks and a small subset of Trusted Advisor security and fault tolerance checks, manual refresh only, and no technical support cases at all. The full check library and the Trusted Advisor API need a paid plan. If your engagement depends on opening a case during a cutover, that’s a prerequisite, not a nice to have.

There’s one date worth carrying in your head, because it’s a real deprecation rather than a version number. AWS has announced that Developer Support, Business Support and Enterprise On-Ramp are discontinued on 1 January 2027, with Business Support+ as the replacement tier. If you’re scoping anything that runs past that boundary, check which plan the client is on and whether their support cost is about to move.

Then monitoring. CloudWatch alarms wired to an SNS topic with no confirmed subscribers are the same failure as the dead trail: configured, visible, useless. If they run anything beyond alarms, whether that’s Grafana Cloud, Datadog or a self-hosted Prometheus stack, find out who holds the credentials and whether the routing has been tested this year.

Pass test: you can name the human who gets woken up, and the path that reaches them.

The findings that actually move the number

• Unattributable credentials. Each one has to be rotated or removed, and each needs a conversation with whoever might break. That’s coordination time, which is the hardest kind to compress.
• No working audit trail. No baseline means you can’t prove what you did or didn’t change. Add the cost of standing logging up before the real work starts.
• Significant drift between code and reality. Reconciliation is slow, fiddly and carries destruction risk. The single largest multiplier I see.
• No tags. A tax on every subsequent task rather than a task of its own, since every scoping decision becomes an investigation.
• Resources in unmentioned regions. Each is a small parallel environment with its own security groups, its own logging gap and its own bill.
• One shared admin identity. Nothing in CloudTrail is attributable to a person, so your first task is identity work the client didn’t ask for.

I don’t present these as criticism. I present them as line items with an hour estimate against each, and let the client decide what’s in scope. That conversation is much easier before a contract exists than after.

If you only run five checks

Sometimes you get twenty minutes and a screen share.

1. Pull the credential report and count active keys that haven’t been used recently.
2. Confirm a CloudTrail trail exists, is multi-region, and is currently delivering.
3. Loop the enabled regions for running instances and NAT gateways.
4. Run a refresh-only plan against whatever infrastructure code exists.
5. Pull six months of cost grouped by service and look at the shape.

Those cover identity, evidence, blast radius, delivery speed and money. The rest of the check refines a picture these five already gave you.

Where this check goes wrong

• Treating it as an audit. You’re scoping work, not issuing a compliance report. A twenty page findings document makes a prospect defensive and costs you a day.
• Checking one region. The default region in your profile is not where the problems are.
• Trusting the console over the API. Configured and working are different states, and the console is better at showing the first.
• Taking long-lived keys because it’s faster. Accept an admin access key by email to save fifteen minutes and you’ve become part of the problem you were hired to assess.
• Running the loops unannounced. A burst of API calls from an unfamiliar principal is exactly what GuardDuty exists to flag. Tell them you’re doing it and roughly when.

Frequently asked questions

How long should an AWS account health check take?

Two to three hours for a single account once you have access, including the write-up. Multi-account organizations take longer but not linearly, because the interesting variation concentrates in one or two accounts and the rest are copies of a landing zone pattern.

Should I charge for the assessment?

For a short engagement, folding it into the proposal is fine. For anything substantial, or where the client is shopping several consultants, quote it as a small standalone piece with a written deliverable. It qualifies the client, protects your time, and the findings have value to them even if they hire someone else.

What if the client won’t give me read-only access before signing?

Quote time and materials, or quote fixed price with an explicit assumptions list naming every unknown. Do not quote a fixed price on an account you haven’t seen. The pattern that works is a small paid discovery phase, then a fixed price set once you know what you’re dealing with.

Is ReadOnlyAccess enough on its own?

It’s broader than you probably want. ReadOnlyAccess extends to reading data in some services, not just configuration, which is more than you need for scoping and more than many clients will be comfortable granting. The SecurityAudit and ViewOnlyAccess pairing is narrower and easier to justify in writing.

Can I automate the whole thing?

The enumeration, yes, and it’s worth scripting since you’ll run it repeatedly. The interpretation, no. Whether five unused IAM users are a problem depends entirely on what the client is trying to do, and that judgement is most of what you’re selling.

What if I find something actively compromised?

Stop and tell them immediately, in writing, with timestamps. Don’t remediate unilaterally: you’re read-only, and containment steps can destroy evidence. Your job is a clear description of what you saw and when, so they can involve whoever they need to.

Does this apply to small accounts too?

Especially to small accounts. Larger organizations usually have some governance, even imperfect. A three year old account at a small business that four different contractors have touched is where the surprises live, and it’s exactly the account that gets described as simple.

The one thing worth remembering

An AWS account health check isn’t a security exercise dressed up as sales. It’s the only reliable way to price uncertainty, and uncertainty is what you’re actually quoting on.

If you take one habit from this, take the second pass. Everything else on the list is something you’d eventually trip over while doing the work. A logging gap is different, because it stays invisible while it costs you, and by the time it matters you’re the person who was in the account when it happened. Check the trail is delivering. Then check the rest.

Need a second pair of eyes on an AWS account?

I do this for consultancies scoping engagements and for businesses who inherited an account they don’t fully understand. Usually that looks like:

• A read-only AWS account health check with a written findings list, hour estimates per item, and a clear split between urgent and later.
• Cross-account audit role setup, with a trust policy and external ID you can hand to any consultant without granting more than you meant to.
• Credential archaeology: working out what every active access key belongs to, then rotating or removing the ones that belong to nothing.
• Getting CloudTrail, AWS Config and log retention into a state where they’re delivering rather than just configured.
• Terraform drift reconciliation, including importing unmanaged resources back under code without a destructive plan.
• Cost shape reviews: the forgotten NAT gateways, orphaned volumes and idle regions, plus budgets and anomaly alerts that reach a real inbox.

If you’ve got a credential report, a terraform plan output or a Cost Explorer screenshot you can’t make sense of, send it over and I’ll tell you what I’d look at next.

The post The AWS Account Health Check I Run Before Quoting a Project appeared first on John Nessime.

Then someone points out that this particular support lead has never had access to Opportunity records. Not through her profile, not through the role hierarchy, not through a sharing rule. And the assistant just read one out to her.

That is the failure that kills most CRM assistant projects, and it almost never shows up in testing, because you test as an admin. Everything you ask works. Everything you ask is supposed to work. The gap only opens when a real user with a real, narrow profile starts asking questions.

This post walks through building a CRM knowledge assistant on Salesforce and Amazon Bedrock Knowledge Bases: which of the two knowledge base paths you are actually allowed to use, how the Salesforce connector authenticates, what it does and does not carry across from your CRM, and the two architectures that survive a real permission model. The wiring is the easy part. The scoping is the work.

The three moving parts

Strip away the marketing and a CRM knowledge assistant is three things bolted together:

• Ingestion — something pulls Cases, Knowledge articles, Opportunities and Accounts out of Salesforce on a schedule, chunks them, embeds them, and writes the vectors somewhere.
• Retrieval — a query comes in, gets embedded, and the closest chunks come back. This is where filtering lives, and it is the security surface.
• Generation — a foundation model reads those chunks and writes an answer with citations.

Generation is the part everyone stares at and the part you should worry about least. It is a model choice and a prompt. Retrieval is where the assistant becomes either trustworthy or a liability.

Decision one: which knowledge base you are allowed to use

Bedrock now has two shapes of knowledge base, and this decision gets made for you the moment you say “Salesforce”.

Bedrock Managed Knowledge Base is the newer, opinionated one. AWS manages the vector store, picks the embedding and reranking models by default, handles parsing, and exposes agentic multi-hop retrieval. It ships with a small set of first-party connectors: Amazon S3, SharePoint, Confluence, Google Drive, OneDrive, Web Crawler, and a custom ingestion path. Salesforce is not on that list.

Customer-managed knowledge bases are the older shape. You bring the vector store, you pick the embedding model, you own more of the pipeline. This is where the Salesforce connector lives, and AWS still flags that connector as a preview release subject to change. It also has a hard constraint worth reading twice: with a Salesforce data source, Amazon OpenSearch Serverless is currently the only supported vector store. If your organisation has standardised on Aurora pgvector or something else, that standard does not apply here.

So the fork is real. Either you use the preview Salesforce connector on the customer-managed path, or you land Salesforce data in S3 yourself and use the managed path. I will come back to why the second option is often the right call, because it is not just a workaround.

One piece of context that matters if you are inheriting an older design: Amazon Kendra entered maintenance mode on 30 June 2026 and stopped accepting new customers on 30 July 2026. Kendra was the standard answer to “search my Salesforce org” for years, and AWS now points people at Bedrock Knowledge Bases instead. Existing indexes keep running and no end-of-service date has been set, but Kendra will not gain features. If a proposal on your desk starts with a Kendra index, that is worth raising before anyone writes code.

The permission model is the whole problem

Salesforce spends enormous effort on who can see what. Profiles, permission sets, the role hierarchy, org-wide defaults, criteria-based sharing rules, field-level security, restriction rules. A given Opportunity is visible to a specific, computed set of users, and that computation happens on every query.

None of that survives ingestion.

The AWS documentation is blunt about it, and this is the single most important sentence in the whole feature: everything you sync from your data source becomes available to anyone holding bedrock:Retrieve on that knowledge base, including content that had controlled permissions at the source. Three layers of Salesforce scoping collapse into one IAM action.

Worse, the connector authenticates as a single identity. You configure a Connected App using the OAuth 2.0 client credentials flow, and that flow runs as one named user. Whatever that user can see, the crawler ingests. Pick a System Administrator to “make sure the sync works” and you have just copied the entire org into a flat index that your whole application tier can read.

Why this is invisible until it isn’t

Nothing errors. There is no warning in the console, no failed sync, no denied API call. Retrieval returns chunks with good similarity scores and the model writes a fluent, cited answer. The only signal that something is wrong is a human recognising a record they should not be looking at, and that human is usually a customer-facing employee, not you.

Treat the knowledge base as a published dataset, not as a view over Salesforce. Ask the question you would ask about an S3 bucket: if every row of this landed in front of every user of the app, who gets hurt?

The two architectures that actually hold up

Option A: ingest only what everyone is allowed to see. Run the Connected App as a deliberately low-privilege integration user whose profile grants read access to Knowledge articles, published solutions, product data and price books, and nothing else. Exclude Contact, Lead, Case and Opportunity entirely. You get a genuinely useful support and enablement assistant, the permission question disappears, and you can ship it without a security review turning into a project.

This is the one I reach for first. Most of the value people want from a CRM knowledge assistant is “what do we know about this product, this policy, this recurring problem”, and none of that needs record-level data.

Option B: export to S3 with your own access metadata. If you genuinely need per-record answers, stop using the connector. Pull the objects you need through the Salesforce API or a scheduled extract, write each record to S3 as a document, and write a sidecar metadata file next to it carrying the access attribute you will filter on. Then point a knowledge base at the bucket and apply an explicit metadata filter on every retrieval, derived from the caller’s identity rather than from the query text.

The cost is honest: you now own an extraction pipeline, a metadata schema, and the job of keeping the access attribute correct when a record changes owner. That is real work. The benefit is that you can point at exactly where the boundary is enforced and test it.

What you cannot do is enforce the boundary in the prompt. “Only answer about accounts the user owns” is a suggestion to a language model, not an access control. Retrieval already happened by then.

Wiring the Salesforce connector

Assuming you have chosen Option A and want the connector, here is the shape of it.

On the Salesforce side

1. Create a Connected App in your org and enable the OAuth 2.0 client credentials flow.
2. Under the client credentials flow settings, set the Run As field to your integration user. This is the step people skip, and the sync fails without it. It is also the step that decides your entire blast radius, so pick that user before you pick anything else.
3. Copy the consumer key and consumer secret from the app’s OAuth settings.
4. Note your instance URL.

On the AWS side

Store the credentials in AWS Secrets Manager. The connector looks for three specific keys, and it will not fall back to alternatives if you name them differently:

{
  "consumerKey": "your Connected App client ID",
  "consumerSecret": "your Connected App client secret",
  "authenticationUrl": "https://yourcompany.my.salesforce.com/"
}

The secret has to live in the same AWS region as the knowledge base. Cross-region reads are not supported here, and the resulting error is generic enough to waste an hour. Do not reuse this secret for any other data source; rotate it on the same schedule as your other integration credentials.

Creating the data source

The data source configuration is a JSON document. The interesting part is patternObjectFilter, which is how you keep the crawler away from objects you never wanted:

{
  "salesforceConfiguration": {
    "sourceConfiguration": {
      "hostUrl": "https://yourcompany.my.salesforce.com/",
      "authType": "OAUTH2_CLIENT_CREDENTIALS",
      "credentialsSecretArn": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:NAME"
    },
    "crawlerConfiguration": {
      "filterConfiguration": {
        "type": "PATTERN",
        "patternObjectFilter": {
          "filters": [
            {
              "objectType": "Knowledge__kav",
              "inclusionFilters": [".*"]
            },
            {
              "objectType": "Case",
              "exclusionFilters": [".*"]
            }
          ]
        }
      }
    }
  },
  "type": "SALESFORCE"
}

Two things to internalise about these filters. First, they are regular expressions matched against a per-object identifier, not against record contents: for Knowledge__kav that is the article title, for Case the case number, for Attachment the file name. Second, when an inclusion and an exclusion filter both match the same document, exclusion wins. That precedence is what makes an allow-list pattern safe to write.

Then create the data source and attach it to an existing knowledge base:

aws bedrock create-data-source 
  --name "salesforce-knowledge" 
  --knowledge-base-id "YOUR_KB_ID" 
  --data-source-configuration file://salesforce-connector-config.json 
  --data-deletion-policy "DELETE" 
  --vector-ingestion-configuration '{"chunkingConfiguration":[{"chunkingStrategy":"FIXED_SIZE","fixedSizeChunkingConfiguration":[{"maxTokens":"300","overlapPercentage":"10"}]}]}'

A note on the command namespace: the agent-plane operations have historically lived under a separate CLI command group from the model-inference ones, and this has shifted between CLI versions. Run aws bedrock help against your installed version before assuming. The JSON payload is the stable part.

--data-deletion-policy DELETE means the vectors get removed from the store when you delete the data source. The alternative retains them. Choose DELETE unless you have a specific reason not to, because orphaned vectors in an OpenSearch Serverless collection are both a cost line and a surprise waiting for whoever inherits this.

After that, kick off a sync with StartIngestionJob, or hit Sync in the console. The connector does incremental syncs after the first run: added, updated and deleted content only. The first run crawls everything.

Chunking is a one-way door

You cannot change the chunking strategy after connecting the data source. Not tune it, not swap it. You delete the data source and start over, which means re-crawling and re-embedding everything.

So spend twenty minutes here rather than five. CRM content is not uniform. Knowledge articles are structured prose and behave like documentation. Case comment threads are short, conversational, and lose their meaning when split mid-exchange. Opportunity descriptions are often two sentences. A fixed-size chunk that suits one of these will mangle another.

The practical move is to stop trying to find one strategy that fits everything. Create separate data sources for materially different content shapes, each with its own chunking configuration, all pointing at the same knowledge base. It costs you a bit of configuration and saves you a full re-index later.

One more constraint: the Salesforce connector does not handle multimodal content. Tables, charts and diagrams embedded in your Knowledge articles are not going to be understood. If your best troubleshooting articles are mostly screenshots, this connector will quietly ingest the surrounding text and nothing else, and your retrieval quality will be worse than the article count suggests.

Retrieval, filtering, and one trap

Bedrock gives you two kinds of metadata filter at query time, and confusing them is a security bug.

Explicit filters are the ones you construct in code and pass in the retrieval configuration. Your application decides them from the authenticated caller. They are deterministic.

{
  "retrievalQuery": { "text": "renewal value on the Northwind account" },
  "retrievalConfiguration": {
    "vectorSearchConfiguration": {
      "numberOfResults": 8,
      "filter": {
        "andAll": [
          { "equals": { "key": "visibility_group", "value": "emea-enterprise" } },
          { "equals": { "key": "record_class", "value": "internal" } }
        ]
      }
    }
  }
}

Implicit filters are the other kind. You declare which metadata attributes exist and Bedrock uses a model to infer filters from the natural-language query. Someone asks about last quarter, it infers a date filter. This is a genuinely nice relevance feature and it is not an access control. A model inferring a filter from user-supplied text is a model that can be talked out of it. Use implicit filters to improve answers; use explicit filters to enforce boundaries.

The trap: filters on the Salesforce connector operate on auto-detected fields from your CRM records. You cannot inject your own visibility_group attribute at ingestion. That capability comes from sidecar metadata files, which is an S3 data source feature. This is the concrete reason Option B routes through a bucket. If per-user scoping is a requirement, the connector cannot give it to you no matter how carefully you configure it.

Where the money goes

Rates change, so learn the meters instead of memorising numbers.

• Embedding at ingestion — billed per token processed. A full re-crawl re-embeds everything, which is exactly why the chunking decision is expensive to reverse.
• Vector storage — on the customer-managed path this is your OpenSearch Serverless collection, billed on provisioned compute units and storage, with a floor whether or not anyone queries it. This is the line item that surprises people on a proof of concept that nobody uses. Managed Knowledge Base bills on indexed data size and retrieval count instead, with no idle collection sitting underneath.
• Generation — input and output tokens on whichever foundation model you chose. Retrieved chunks are input tokens, so numberOfResults is a direct cost lever and worth tuning down until quality actually degrades.
• Everything around it — the API layer, the chat frontend, and the observability stack. A small VPS from a provider like InterServer or a Lambda behind API Gateway both work; the choice barely matters next to the Bedrock line. Push retrieval latency and citation counts into Grafana Cloud or Datadog early, because “the assistant feels worse this week” is otherwise unfalsifiable.

Troubleshooting

• Sync fails immediately with an authentication error. Check the Run As user on the Connected App’s client credentials flow first. An empty Run As field is the most common cause. Then check that the secret keys are spelled exactly consumerKey, consumerSecret, authenticationUrl, and that the secret is in the knowledge base’s region.
• Sync succeeds but the document count is far lower than expected. Usually the integration user’s profile, not the filters. Query the object as that user in Salesforce and count. Also check the per-knowledge-base file count and file size quotas, which are separate limits and are silently enforced.
• Retrieval returns nothing for queries you know are covered. Test with no filter at all. If results appear, your filter keys do not match the auto-detected field names. If they do not, the content probably never made it past chunking; check whether those articles are mostly images.
• Deleted Salesforce records still turn up in answers. Incremental sync handles deletions, but only on the next sync. If you deleted something sensitive, run a sync immediately rather than waiting for the schedule.
• Answers are confidently wrong about numbers. Fixed-size chunking split a record away from its label. Retrieve the raw chunks for that query and read them. Nine times out of ten the problem is visible in the chunk, not in the model.

Common mistakes

• Running the Connected App as a System Administrator “just to get the sync working”. That decision is almost never revisited.
• Assuming sharing rules and field-level security follow the data into the index. They do not.
• Enforcing access in the system prompt. Retrieval has already happened by the time the model reads your instructions.
• Ingesting Contact and Lead because they were there. Both carry personal data and both are usually irrelevant to the questions people actually ask.
• Accepting default chunking, then discovering it is immutable.
• Building on a preview connector without telling stakeholders it is preview. The configuration surface can change under you.
• Testing only as an admin. Every acceptance test should run as the narrowest real profile you have.

Best practices

• Decide the integration user’s profile before you touch AWS. It defines the security boundary more than any IAM policy will.
• Start with Knowledge articles only. Ship that, get real queries, then argue about whether Opportunity data is worth the architecture it requires.
• Write filters as allow-lists. Exclusion beats inclusion, so an allow-list fails closed.
• Split content shapes across separate data sources so chunking suits each one.
• Keep a fixed evaluation set of about fifty real questions with known-good answers, and re-run it after every configuration change.
• Log the retrieved chunk IDs alongside every generated answer. When someone reports a bad answer months from now, that log is the only thing that makes it debuggable.
• Treat bedrock:Retrieve as a privileged permission and scope it to the specific knowledge base ARN, not to a wildcard.

Frequently asked questions

Does the Bedrock Salesforce connector respect sharing rules?

No. The crawler authenticates as one user and everything it can read becomes retrievable by anyone with bedrock:Retrieve on the knowledge base. Per-user scoping has to be built at the retrieval layer with explicit metadata filters, which in practice means routing through S3 rather than using the connector.

Can I use Aurora or pgvector instead of OpenSearch Serverless?

Not with the Salesforce data source. AWS currently restricts that connector to Amazon OpenSearch Serverless. If your vector store is non-negotiable, extract to S3 and use a data source that supports your chosen store.

Should I use Agentforce instead of building this on Bedrock?

If your questions are answerable entirely from Salesforce data and your users already live in Salesforce, Agentforce is the shorter path and it operates inside the platform’s existing permission model, which removes the hardest problem in this post. Bedrock earns its complexity when the assistant has to reason across CRM data plus sources that are not in Salesforce, or when you need a specific foundation model, or when the data must stay inside your AWS account. Cost profiles differ substantially and are worth modelling on your real query volume before committing.

How current will the assistant’s answers be?

As current as your last sync. Ingestion is scheduled, not live. For rapidly changing fields such as case status or pipeline stage, do not answer from the index at all: have the assistant call the Salesforce API for the live value and use the knowledge base only for the surrounding context.

What happens to the connector when it leaves preview?

Unknown, which is the point of the label. Preview means the configuration surface, supported objects and behaviour can change without the compatibility guarantees you would expect from a GA service. Keep your data source configuration in version control so a re-create is a rerun rather than an archaeology exercise.

Is a CRM knowledge assistant worth building if we already have Salesforce search?

It depends on the question shape. Search is better when people know what record they want. A CRM knowledge assistant wins on synthesis: “what usually causes this error for enterprise customers” pulls from twenty articles and cases at once, which no search result page does well. If your users mostly navigate to known records, you will not get much return.

Do I need Bedrock AgentCore for this?

Not for retrieval and generation. AgentCore matters when you want the assistant to take actions, chain tools, or be exposed to agent frameworks over MCP. A read-only question-answering assistant works fine against the knowledge base APIs directly, and skipping AgentCore removes a layer you would otherwise have to operate.

The one thing to remember

Connecting Salesforce to Amazon Bedrock takes an afternoon. The connector works, the syncs run, the answers are good. That speed is exactly what makes this dangerous, because nothing in the setup path forces you to think about who the answers are for.

A CRM knowledge assistant is a publishing decision wearing an engineering costume. Every record you ingest, you are publishing to everyone who can call retrieve. Decide what that audience is allowed to see, encode it in the integration user’s profile and in your inclusion filters, and test every question as your narrowest real user. Do that first and the rest of the build is routine.

Need help getting this into production?

I work with teams building retrieval systems on AWS, usually at the point where a working prototype has to survive a security review. Things I can help with on a CRM knowledge assistant:

• Reviewing an existing Bedrock knowledge base for records that should never have been ingested, and scoping the integration user’s Salesforce profile to match.
• Designing the S3 extraction path with sidecar access metadata when per-user filtering is a hard requirement.
• Choosing between the managed and customer-managed knowledge base paths, with the OpenSearch Serverless cost floor modelled against your actual query volume.
• Building the chunking and data source split before it becomes an immutable mistake, plus a repeatable evaluation set to measure changes against.
• Terraform or OpenTofu for the whole stack: knowledge base, data sources, Secrets Manager, IAM, and the vector store.
• Retrieval observability so you can tell whether answer quality moved, and why.
• Migration planning for teams currently on a Kendra index.

If you have a data source configuration, a failing sync, or a retrieval result that looks wrong, send it over and we can work out what it is doing.

The post Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records appeared first on John Nessime.

It is a one-line change. It works on the first try. And it is the grant most likely to still be sitting there long after that analytics project shipped, covering a wider set of identities than the person who wrote it ever had in mind.

That asymmetry is the whole problem. A denied cross-account request is a nuisance: someone pings you, you read the policy, you fix it before lunch. A cross-account grant that succeeds, keeps succeeding, and quietly reaches further than intended produces no error, no alert, and no ticket. It surfaces months later during an audit, when someone asks who has been reading that prefix and the honest answer is “anyone the other account decided to let in.”

This post covers AWS cross-account data sharing as a set of failure families rather than a service tour. How the delegation model actually behaves, where encryption keys break or widen access without warning, who ends up owning the bytes after a handoff, why a copy is worse than a reference, and which guardrails catch the mistake you are eventually going to make. Real policy snippets and commands throughout, plus the denials that are hardest to read.

AWS cross-account data sharing is a delegation, not a permission

Start with the mechanic that everything else hangs off. For a request that crosses an account boundary, two separate accounts have to say yes. The resource-based policy in the account that owns the data has to allow the caller, and an identity-based policy in the calling account has to allow the action. Neither side alone is enough.

That is why “I added them to the bucket policy and it still fails” is such a routine ticket. The producer did their half. Nobody attached the matching IAM policy on the consumer side.

Now the part that bites. Look at how most cross-account bucket policies get written:

{
  "Sid": "AllowAnalyticsAccount",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ]
}

Despite how it reads, that :root suffix does not mean the root user. It means the account. You have not granted access to a principal at all. You have delegated the guest list to the administrators of account 111122223333, and they can hand that access to any principal they like, including a role somebody creates next quarter for a purpose you will never hear about.

Sometimes that delegation is exactly what you want, because the consumer team manages its own roles and you do not want to be in the loop for every change. Usually you want the narrower version:

{
  "Sid": "AllowAnalyticsRoleOnly",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ],
  "Condition": {
    "StringEquals": { "aws:PrincipalOrgID": "o-abc123def4" }
  }
}

Both halves earn their place. The named ARN stops the wrong role in the right account. The organization condition stops the right role name in the wrong account, which matters if the statement is ever copied, templated, or widened by someone in a hurry. One protects against internal drift, the other against the policy escaping your organization entirely.

Be clear-eyed about the trade-off, though. Naming ARNs is precise and brittle: every new consumer role is a policy change, a pull request, and a review. Organization-wide conditions scale beautifully and narrow nothing inside a member account. If your organization contains twelve accounts and forty teams, aws:PrincipalOrgID on its own is a fairly wide door.

The grant that stops working after a rebuild

When you name a role ARN as a principal, IAM stores the role’s internal unique ID behind the scenes. Delete that role and the policy will start displaying an opaque AROA... string where the ARN used to be, because the ARN no longer resolves to anything. Recreate a role with the identical name and the access does not come back, because the new role has a new unique ID.

A terraform destroy followed by terraform apply on the consumer side reproduces this perfectly. If you are staring at a bucket policy that contains a principal starting with AROA, that is your answer. Rewrite the statement with the current ARN.

The encryption boundary nobody tests until it is live

Second failure family: the key. Encrypting shared data is the easy decision. What people miss is that a KMS key is a second, fully independent authorization surface, with its own resource policy, evaluated separately from the bucket.

• Downloading an object encrypted with SSE-KMS requires kms:Decrypt on the key.
• Uploading one requires kms:GenerateDataKey.
• A multipart upload requires both.

The AWS managed key for S3 cannot carry you here, because you cannot edit its key policy. Sharing KMS-encrypted objects across accounts means a customer managed key, with a statement naming the consumer:

{
  "Sid": "AllowAnalyticsRoleToDecrypt",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*",
  "Condition": {
    "StringEquals": { "kms:ViaService": "s3.us-east-1.amazonaws.com" }
  }
}

The kms:ViaService condition is cheap insurance. It restricts use of the key to requests that arrive through S3 in that region, so a leaked set of consumer credentials cannot call KMS directly and start decrypting ciphertext it obtained some other way. And the two-key rule applies to the key exactly as it does to the bucket: the consumer’s own IAM policy has to allow the KMS actions too.

Now the sharp edge, which is worth reading twice. If a KMS key is specified by alias rather than by fully qualified ARN, the alias is resolved in the requester’s account. In a cross-account write that means data landing in your bucket can end up encrypted under a key belonging to the writer. You own the object and cannot read it, and no error is raised at write time. Use full key ARNs in anything that crosses an account boundary.

Who owns the bytes after the handoff

Third family: ownership, and it depends entirely on which direction the data moves.

New buckets are created with Object Ownership set to Bucket owner enforced. ACLs are off, and the bucket owner owns every object written into the bucket regardless of who wrote it. That default removed the single most confusing trap in S3 sharing.

Buckets created before that default are the ones to check. If a bucket still sits on the Object writer setting, an object uploaded by another account is owned by that account, and you, holding the bucket, may not be able to read it. Nothing fails at write time. It fails later, when a job in your own account tries to read your own bucket and gets a 403.

Check and fix it with the ownership controls API:

aws s3api get-bucket-ownership-controls --bucket shared-events-bucket

aws s3api put-bucket-ownership-controls 
  --bucket shared-events-bucket 
  --ownership-controls '{"Rules":[{"ObjectOwnership":"BucketOwnerEnforced"}]}'

Order matters when you make that change on a live bucket. Once ACLs are disabled, any PUT that still carries an ACL is rejected with AccessControlListNotSupported. Requests that send no ACL, or send bucket-owner-full-control, keep working. So the sequence is:

1. Find every writer that sets an ACL, including old scripts using --acl and any SDK call passing an ACL argument.
2. Update those clients so they stop sending one.
3. Confirm from CloudTrail that ACL-bearing PUTs have stopped.
4. Flip the bucket to Bucket owner enforced.

Given a free choice of design, I would rather the consumer pull from a bucket I own than push into a bucket I do not. Pull keeps ownership, lifecycle rules, encryption configuration, and access logging in one account, under one team’s control. Push scatters them.

Copies you cannot recall

Fourth family, and the one with the longest tail. There is a large difference between granting a reference and shipping a copy. A reference can be revoked on a Tuesday afternoon. A copy is simply gone.

Three reference-shaped patterns are worth knowing before you reach for a bulk export.

S3 Access Points for object-level sharing

Instead of a bucket policy that grows a new statement every time a team asks for something, you write the bucket policy once to delegate to access points, then give each consumer their own access point with its own focused policy. The delegation is expressed with the s3:DataAccessPointAccount condition. Access points can also be owned by the consumer’s account, so they manage their own detail without touching your bucket policy.

The gotcha: when a request goes through an access point, both the access point policy and the underlying bucket policy have to allow it. A 403 here is easy to misdiagnose because the bucket policy looks correct and is correct. The access point is the layer that said no.

Lake Formation for table-level sharing

When the thing being shared is a table rather than a prefix, Lake Formation is the better fit, because it lets you grant on databases, tables, columns, and row filters instead of on object paths. Cross-account grants ride on AWS Resource Access Manager, so the recipient gets an invitation that a data lake administrator has to accept.

aws ram get-resource-share-invitations

aws ram accept-resource-share-invitation 
  --resource-share-invitation-arn arn:aws:ram:us-east-1:111122223333:resource-share-invitation/EXAMPLE

After the invitation is accepted, the shared database and tables are visible but still not queryable from Athena or Redshift Spectrum until someone creates a resource link, which behaves roughly like a symlink in the recipient’s Data Catalog. Missing resource links account for most of the “the share went through but I cannot see the table” messages.

One setting to check on the producer side is the cross-account data sharing version. Newer versions collapse many grants into far fewer RAM resource shares, which matters once you are sharing at any scale, and they allow granting directly to IAM principals in another account rather than only at account level. Read the current version with:

aws lakeformation get-data-lake-settings

When a copy really is the answer

Sometimes the partner genuinely needs the data on their own infrastructure: a warehouse you have no reach into, a VPS at a provider like Contabo or InterServer, or in the worst case an analyst’s laptop. That is a legitimate outcome, but be honest with yourself and with them that your controls stop at the handoff.

Practical hygiene for that case: scope the extract to the columns actually needed rather than the whole table, put an expiry date in the agreement rather than in your head, and write down what happens to the copy when the engagement ends. If the copy lives on machines you or the client control, verified erasure with something like O&O SafeErase reads a great deal better in an audit than “we deleted the folder.” And if the recipient has no AWS account at all, a time-limited presigned URL is usually a better answer than minting an IAM user with long-lived keys.

The guardrail that catches the mistake you will make

Everything above describes what a careful engineer does per grant. Guardrails are what happens when somebody is not careful, which over a long enough window includes all of us.

Resource control policies are the strongest lever if you run AWS Organizations. An RCP sets a ceiling on what any resource-based policy inside the organization can grant. The canonical statement is “no principal outside my organization may access S3 buckets in my accounts, regardless of what an individual bucket policy says.” That inverts the risk model: a careless bucket policy is no longer an exposure, it is just an ineffective statement.

Two limits are worth stating plainly. RCPs cover a specific set of services rather than everything in AWS, so check the current supported list rather than assuming coverage. And they constrain your resources, not a partner’s behavior once data has legitimately left. An RCP is a wall around your side of the fence.

Service control policies do the mirror-image job on the consumer side, limiting which external accounts your own principals are allowed to assume roles into. Account-level S3 Block Public Access remains a sensible floor underneath both.

For sharing with a third party rather than an internal account, the external ID is not optional in my book:

{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::444455556666:role/vendor-connector" },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringEquals": { "sts:ExternalId": "a-value-the-vendor-generated-for-you" }
  }
}

The external ID exists because of the confused deputy problem. A vendor holds credentials that can assume roles across many customers, and without a per-customer secret in the condition, customer A can trick the vendor into pointing that access at customer B. If a vendor asks you to create a role for them with no external ID and no conditions at all, that is worth a conversation before it is worth a role.

Verifying that the grant is what you think it is

Reading policies is not the same as knowing what they permit. IAM Access Analyzer answers the question “what is currently reachable from outside my zone of trust” using automated reasoning rather than pattern matching, which is why it catches combinations a human reviewer skims past.

aws accessanalyzer create-analyzer 
  --analyzer-name external-access 
  --type ACCOUNT

aws accessanalyzer list-findings 
  --analyzer-arn arn:aws:access-analyzer:us-east-1:999988887777:analyzer/external-access

Scope the zone of trust to your organization where you have one, then archive the findings that represent intentional shares, so that a new entry in the list actually means something.

The same service exposes custom policy checks you can run in a pipeline before a policy change merges, including checks for public access, for specific actions being granted, and for whether an updated policy grants more than the one it replaces. One thing to understand about them: they evaluate the document you hand them and nothing else. They do not read your account state, so a public-access check can fail on a policy in an account that has Block Public Access enabled. For a merge gate that is correct behavior, because the policy should not be relying on a distant safety net.

Finally, CloudTrail tells you who actually used a grant, which is the input to the only cleanup that ever really works: delete the statements nobody has exercised. Routing that history into CloudWatch, or into a Grafana Cloud dashboard alongside the rest of your platform telemetry, turns “is this still needed” from an argument into a query.

Troubleshooting the denials that are hard to read

• You can list the bucket but every GET returns 403. That pattern points at KMS, not S3. Listing does not touch the key; reading does. Check kms:Decrypt on both the key policy and the consumer’s identity policy.
• 403 through an access point while the bucket policy is clearly correct. The access point has its own policy and it also has to allow the calling principal. Read that one next.
• AssumeRole is denied although the trust policy names the caller. Cross-account needs both halves: the caller’s identity policy must permit sts:AssumeRole against that role ARN. Then check the external ID value and any SCP or permissions boundary in the calling account.
• A principal in the policy displays as AROA followed by random characters. The referenced role was deleted. Recreating it under the same name will not restore access.
• A Lake Formation share appears in the catalog but queries fail. Usually a missing resource link in the recipient account, or a RAM invitation that was never accepted.
• Access worked yesterday and stopped today with nothing changed locally. Look at the organization layer before re-reading the bucket policy. A newly attached RCP or SCP denies in a way that no amount of resource-policy editing will fix.

Common mistakes

• Granting to an account principal when a role ARN would have done, then never revisiting it.
• Treating aws:PrincipalOrgID as least privilege. It narrows the organization, not the account.
• Sharing a bucket and forgetting the key, then debugging S3 for an hour over a KMS denial.
• Referring to KMS keys by alias in cross-account configuration instead of by full ARN.
• Letting a partner push into your bucket without confirming the ownership setting first.
• Exporting a full table when the consumer needed four columns, because the export was easier to write.
• Creating a vendor role with no external ID and no conditions because the vendor’s setup guide said to.
• Granting access with no expiry, no owner, and no note in the policy Sid explaining why it exists.

Best practices worth the effort

• Give every cross-account grant a descriptive Sid. It is the only free documentation slot IAM offers, and future-you will read it.
• Prefer one dedicated consumer role per relationship over a shared role used by several integrations. Revocation becomes a single delete.
• Pair a specific principal with a broad condition, rather than relying on either alone.
• Put policy checks in CI so a widened grant fails a pull request instead of a quarterly review.
• Keep an inventory of active shares with an owner and a review date, even if it is a table in a repository README.
• Enable an external access analyzer scoped to your organization and treat new findings as work, not noise.
• Watch cross-region and internet data transfer when a consumer sits somewhere unexpected. Cost visibility tooling such as Vantage or CloudZero will attribute it faster than reading a bill.

Frequently asked questions

What is the safest way to share an S3 bucket with another AWS account?

Name a specific role ARN in the bucket policy, add an organization or account condition alongside it, scope the resource to a prefix rather than the whole bucket, and make sure the consumer’s identity policy is equally narrow. If the bucket is encrypted with a customer managed key, grant on the key as well. If the list of consumers is going to grow, move to access points rather than growing the bucket policy.

Do I need both a bucket policy and an IAM policy?

For cross-account access, yes. The resource-based policy in the owning account and the identity-based policy in the calling account both have to allow the action. That is the single most common reason a correct-looking bucket policy still returns Access Denied.

Why do I get Access Denied when the bucket policy clearly allows the other account?

Work through it in this order: the consumer’s identity policy, then the KMS key policy if the objects are encrypted with a customer managed key, then any access point policy in the path, then organization-level SCPs and RCPs. Denials from the organization layer are the ones people waste the most time on, because nothing in either account’s policies looks wrong.

Can I share data with an account outside my AWS Organization?

You can, and organization conditions will not help you there. Use a named role ARN, an external ID on any assumed role, a tightly scoped resource, and a defined end date. Also confirm whether an RCP in your organization already blocks external principals for that service, because that is a common reason a partner grant refuses to work no matter how carefully it is written.

Should I use S3 Access Points or Lake Formation?

It depends on the unit of sharing. If the consumer wants objects under a prefix, access points are simpler and cheaper to operate. If the consumer wants a table, and especially if you need column or row filtering, Lake Formation is the right tool and access points would force you to fake fine-grained control with prefix layout. Plenty of environments run both.

How do I revoke cross-account access cleanly?

Remove the statement from the resource policy, remove the KMS key grant, and check for any access point, RAM share, or resource link that also carries permission. Then confirm with CloudTrail that the principal has stopped appearing. Revocation feels done long before it is done, which is exactly why an inventory of active shares pays for itself.

Does cross-account access change anything about cost?

The request charges land on the bucket owner by default, and data transfer depends on whether the consumer is in the same region. A consumer that quietly starts scanning a large prefix from another region is a billing event as much as a security one, which is a good argument for reviewing access patterns and spend together rather than in separate meetings.

The one thing to remember

Getting AWS cross-account data sharing right is less about knowing which service to reach for and more about noticing which grants are quietly larger than they read. An account principal delegates your guest list. An organization condition narrows the wrong dimension. A copy leaves your perimeter permanently and no policy edit brings it back.

Grant references rather than copies, name principals rather than accounts, remember that the key is a second door, and put something in place that watches the perimeter for you. The cross-account grant worth worrying about is never the one that failed. It is the one that worked, and kept working, long after anybody was still looking at it.

Need a second pair of eyes on your cross-account setup?

Most of the cross-account problems I see are not exotic. They are a policy written in a hurry two years ago that nobody has read since. If that sounds familiar, here is where I can help:

• Auditing existing bucket, key, and role trust policies to find grants that are wider than intended, and rewriting them to named principals with conditions.
• Designing a producer and consumer pattern for a specific data handoff, including whether access points, Lake Formation, or a plain scoped role is the right shape.
• Setting up IAM Access Analyzer with a sensible zone of trust and archive rules, so the findings list stays worth reading.
• Adding policy checks to your pipeline so a widened grant blocks a merge rather than surviving to production.
• Building an organization-level perimeter with resource control policies and service control policies without breaking the integrations you already depend on.
• Untangling stubborn cross-account denials, including the KMS, access point, and organization-layer cases that hide well.

Send me a bucket policy, a trust policy, or the CloudTrail entry for the denial you cannot explain, and I will tell you what it actually permits.

The post AWS Cross-Account Data Sharing Done Safely: Share Access, Not Copies appeared first on John Nessime.

That is the shape of most trouble with SAP data integration with AWS Glue. The connector does not usually crash. It returns an empty or partial result set, the job exits zero, and your data lake quietly diverges from the source system. By the time anyone notices, you have no idea which run was the last correct one.

This post covers the parts that bite after the proof of concept: how delta tokens actually behave, why deleted records keep showing up in your reports, where partitioning stops working, and the licensing distinction between two things that both get called “ODP” and are not the same thing at all. There is a troubleshooting section at the end for when a job is already broken.

First, get the ODP terminology right, because licensing depends on it

Operational Data Provisioning is SAP’s framework for exposing extraction interfaces in a delta-aware way. A provider (a BW extractor, an ABAP CDS view, SLT, a HANA information view) writes changes into an Operational Delta Queue, and subscribers pull from that queue. This part is uncontroversial.

What matters is that there are two ways to talk to ODP, and SAP treats them very differently. SAP Note 3255746 restricts third-party and customer applications from using the RFC modules of the ODP Data Replication API against ABAP sources. That note has been sharpened over several revisions, from “unsupported” to “not permitted”, and SAP has stated it reserves the right to introduce technical measures and audit unpermitted use. A number of established extraction tools were built directly on ODP-RFC and have had to move.

The AWS Glue SAP OData connector takes the other path. It talks OData over HTTPS to an activated SAP Gateway service, and uses the ODP framework through that OData layer rather than through RFC. That is the route SAP points customers at. If you are being asked in an architecture review whether your pipeline is on the wrong side of Note 3255746, this distinction is the whole answer, and it is worth having the note number written down before the meeting.

Two honest caveats. First, notes get revised, so read the current version rather than a vendor blog summarising it, including this one. Second, “permitted” is not the same as “free”: you still need the OData service activated, a service user with the right authorisations, and a Basis team willing to own the queue growth on their side.

Failure family one: the delta token is shared state you do not own

This is the one that produces silent drift, and it is the reason the finance report was short.

When you read an ODP-enabled entity with change data capture switched on, the connector asks SAP to initialise a delta queue subscription. SAP hands back a delta token. That token is your bookmark. On the next run you pass it back, and you get only what changed since.

sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "ENABLE_CDC": "true"
    },
    transformation_ctx="initial_load"
)

ENABLE_CDC defaults to false. Leave it off and you get a full load every single run, which is expensive but at least obvious. Turn it on and the token appears in the DELTA_TOKEN column, populated only on the last row of the frame. That last row is not a business record. It is a carrier for the token, and if you write your frame straight to S3 without stripping it you have just added a junk row to your table.

The subsequent run passes the token back:

sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "DELTA_TOKEN": last_known_token
    },
    transformation_ctx="incremental_load"
)

Why this goes wrong

A delta token is bound to a service collection and a user. If a new initialisation runs with ENABLE_CDC set true for that same collection and user, SAP invalidates every token previously issued from the earlier initialisation. Not “expires eventually”. Invalidates, immediately.

So picture two Glue jobs sharing one service account, both reading the same extractor. Someone redeploys job B with the CDC flag still set from testing. Job B re-initialises. Job A’s stored token is now dead, and the next time job A runs you get:

Could not open data access via extraction API RODPS_REPL_ODP_OPEN

An error is the good outcome. The bad outcome is code that catches the exception, falls back to a full load, and writes it over the top of an incremental table with no dedupe. Or code that re-initialises on failure, which fixes job A and breaks job B, and now the two jobs take turns invalidating each other forever while both keep exiting zero.

Practical rules that come out of this:

• One SAP service user per consuming pipeline. Sharing a user is what couples independent jobs together.
• Persist the token somewhere durable and versioned. DynamoDB, Parameter Store, S3 with versioning enabled, all fine. Job bookmarks are not a substitute, because the token is source state, not read position.
• Never auto re-initialise on error. Fail loudly, page someone, and make the reinitialisation a deliberate human action with a documented backfill.
• Alert on zero-change runs. A pipeline that returns no changes for several consecutive runs against a busy table is telling you something, and it is rarely “business is quiet”.

Retention on the SAP side is a hard deadline

The delta queue is not infinite. In transaction ODQMON the Basis team configures retention, and a reorganisation job cleans up data past that window. If your pipeline is down longer than the retention period, the changes you missed are gone from the queue and no token will bring them back. You are into a full reload and a reconciliation.

Two things to find out before you go live: what the retention setting is, and whether the cleanup job is actually scheduled. Then make sure your pipeline’s alerting escalates well inside that window. A weekend outage on a queue with a short retention is a very expensive Monday.

Related: if anyone has been testing extraction with RODPS_REPL_TEST, that program registers as a real additional subscriber rather than simulating one. Because the queue is shared and data is only cleaned once every subscriber has collected it, forgotten test subscriptions hold delta data indefinitely and the underlying queue tables grow. Ask whoever ran the tests to reset those subscriptions in ODQMON.

Failure family two: deletes arrive as a column, not as a deletion

CDC extraction from ODP entities returns a couple of extra fields alongside your business columns. DML_STATUS carries UPDATED for inserts and updates, and DELETED for records removed at source. GLUE_FETCH_SQ is a per-record sequence derived from an epoch timestamp, in receipt order, and it is your tie-breaker when the same key changes twice inside one delta.

Nothing acts on those fields for you. If your job appends the frame to S3 and calls it done, a deleted sales order is now a row in your lake with DML_STATUS = DELETED and every other column populated exactly as before. Any query that does not filter on that column counts it. Cancelled orders keep appearing in revenue. Closed accounts stay open. The pipeline is green, the numbers are wrong, and the discrepancy grows slowly enough that nobody catches it early.

You have to apply the changes rather than accumulate them. Broadly two options:

1. Merge into a table format that supports it. Apache Iceberg, which Glue supports natively, gives you a real merge with delete handling. Land the raw delta, then merge on the business key with GLUE_FETCH_SQ deciding the winner within a batch, deleting where the status says deleted.
2. Keep an append-only history and build a current-state view. Cheaper to write, and it preserves an audit trail. It also means every downstream consumer has to use the view, and someone eventually will not.

I reach for Iceberg first on anything that feeds reporting, because the correctness lives in the table rather than in the discipline of whoever writes the next query. Append-only is the right call when you genuinely need history, or when the consumer is a downstream warehouse like Amazon Redshift that will do its own merge anyway.

Worth knowing: not every ODP provider emits deletes reliably. Some older extractors express them as reversal records rather than a delete flag. Confirm the behaviour for your specific entity with the functional owner before you build merge logic on an assumption.

Failure family three: pagination and partitioning do not compose

The connector’s default page size is 50,000 records, and you can raise it up to 500,000. Anything above that is ignored and the maximum is used instead. Raising the page size cuts the number of round trips, which is usually the dominant cost on a wide table across a slow link.

The catch is what your SAP Gateway will tolerate. A large page means a large response the ABAP stack has to assemble and hold, and Gateway or an intervening load balancer may cut it off before it completes. Symptoms are timeouts, connection resets, or a job that appears to paginate without ever finishing. Tune the page size with your Basis team, not in isolation, and treat it as a negotiation between round trips and response size rather than a value you set once.

On partitioning, there is a specific documented limit that catches people out: ODP entities are not compatible with record-based partitioning, because pagination is already handled through skip and delta tokens. For record-based partitioning, the default concurrency is forced to null regardless of what you asked for. So the obvious lever for making a slow ODP extraction faster is not available.

What you can still do on non-ODP OData entities is field-based partitioning, splitting a read across a bounded range:

• PARTITION_FIELD is the field to split on, typically a date.
• LOWER_BOUND is inclusive, UPPER_BOUND is exclusive. Getting that backwards is how you duplicate or drop a boundary day.
• NUM_PARTITIONS defaults to 1, so leaving it out means you did not partition at all.
• Where a limit and a partition are both applied, the limit wins. That is the usual cause of “I partitioned it and got fewer rows”.

For non-ODP entities you can also do timestamp-based incremental reads with filteringExpression, pointing at whatever last-changed field the entity exposes. Note the casing: that option is camelCase while most of the others are upper snake case, which trips people up when a filter appears to be ignored.

Failure family four: connection, auth and network

These fail early and loudly, which makes them the easy ones. The trap is that they fail during setup, when you have three people on a call and someone suggests loosening something to move on.

The connector supports two authentication types: CUSTOM, which is SAP basic authentication, and OAUTH2. The custom secret in AWS Secrets Manager takes this shape:

{
  "basicAuthUsername": "<your SAP username>",
  "basicAuthPassword": "<your SAP password>",
  "basicAuthDisableSSO": "<True/False>",
  "customAuthenticationType": "CustomBasicAuth"
}

For OAuth 2.0 the client secret goes into Secrets Manager under USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET, and the client ID, authorisation code URL, token URL and scopes are supplied on the connection itself.

Basic auth is genuinely easier for a scheduled pipeline: no interactive redirect, no refresh token to keep alive, and rotation is a Secrets Manager operation. OAuth gets you short-lived tokens and revocation you can actually reason about, which is what a security review will want. Both are defensible. What is not defensible is one shared service user across every pipeline, for the delta token reasons above as much as the audit ones.

A few things that reliably waste a morning:

• The service path on the connection is the catalog service path, not the path to your object. The entity path goes in ENTITY_NAME on the job.
• The OData service has to be activated in the SAP system. Until it is, nothing you do on the AWS side matters.
• For a non-VPC connection, the application host URL must be reachable from the public internet. If your SAP system sits in a private network, configure VPC connectivity properly rather than opening it up.
• The Glue role needs Secrets Manager read plus the EC2 network interface permissions when a VPC is involved. A missing ec2:CreateNetworkInterface shows up as a connection failure that looks like a network problem.
• Private certificate authorities need the chain in a form Glue accepts. If you are wiring this up yourself, expect it to take a pass or two.

Choosing your path: Glue jobs, zero-ETL, or something else

The same connector is reachable through more than one product surface, and the choice is really about how much state you want to own.

• Glue Studio or script jobs. You control page size, filters, output format, transformation and scheduling, and you own delta token persistence. Pick this when you need real transformation in flight, or when the merge logic has to live next to the extraction.
• Glue zero-ETL integrations. AWS manages token state internally, runs the initial full load and then the incremental cycle, and preserves the token on failure so retries do not lose data. Less code, less control. Worth noting that once an incremental update field is chosen for a connection and entity, a later integration on the same pair cannot pick a different one.
• Amazon AppFlow. Simpler, flow-shaped, well suited to modest volumes and quick prototypes. Less appropriate once you need Spark-scale processing.
• Commercial replication tools. Fivetran, Airbyte and SAP-specialist vendors all play here. They cost money and add a vendor, and in exchange they absorb exactly the state management this post is about. If your team has no SAP depth, that trade is often correct.

On cost, Glue bills for the resources a job consumes for as long as it runs, so the levers are the same as any Spark workload: fewer round trips, less data pulled, fewer idle executors, and not re-reading history you already have. Rates change and vary by region, so model it against current pricing rather than a figure from a blog post. If your incremental run is pulling the same volume as your initial load, the delta mechanism is not working and the bill is the symptom.

Also worth writing back. The connector supports write operations to SAP OData, using INSERT and UPDATE, with ID_FIELD_NAMES required on update to identify the external ID field. Useful for enrichment loops, and something to treat with a good deal more caution than a read.

Troubleshooting

“Could not open data access via extraction API RODPS_REPL_ODP_OPEN”

Your token is invalid or expired. Find out why before you re-initialise, because re-initialising destroys the evidence. Check ODQMON for a recent initialisation on that collection and user, check whether another job or a test program shares the credentials, and check whether the request you wanted has aged out of retention. Then re-initialise deliberately and reconcile the gap.

The job succeeds but returns no rows

Most often a stale or wrong token, a filter predicate that matches nothing, or CDC never having been initialised so the connector has nothing to hand back. Run once without ENABLE_CDC against a small filtered slice and confirm you get data at all. That separates “no changes” from “not reading”.

Pagination never terminates

Usually the SAP side enforcing its own page limit under your requested size, so the connector keeps requesting more. Reduce the page size, verify the filter is actually being applied server side rather than after retrieval, and have Basis confirm what Gateway is capping.

Row counts drift over time

Almost always unhandled DML_STATUS = DELETED records, or a token invalidation that silently turned an incremental into a partial. Count distinct business keys in your target against a direct count in SAP for a bounded date range. Make that comparison a scheduled check, not something you run when someone complains.

Duplicates after a rerun

A delta was consumed, the write failed, and the token advanced anyway. Store the token only after the write commits, never before. If you are appending rather than merging, add a dedupe on business key ordered by GLUE_FETCH_SQ.

Common mistakes

• Sharing one SAP service user across multiple pipelines, which couples their delta tokens together.
• Automatic re-initialisation in an exception handler. It converts a loud failure into permanent silent drift.
• Writing the token-carrier row into the target table as if it were data.
• Appending CDC output and ignoring DML_STATUS.
• Persisting the token before the write succeeds.
• Assuming an entity is ODP-enabled because it looks analytical. Confirm it, because the incremental mechanism differs.
• Leaving test subscriptions from RODPS_REPL_TEST registered against the queue.
• Building the pipeline without ever asking Basis what the ODQ retention window is.

Best practices for SAP data integration with AWS Glue

• Treat the delta token as production state. Versioned storage, a documented recovery procedure, and a written-down answer to “how do we backfill”.
• One service user per pipeline, one connection per environment. Isolation is the cheapest fix for the worst failure.
• Land raw, merge second. Keep the untouched connector output so you can replay a merge without going back to SAP.
• Alert on absence, not just errors. Zero rows for several consecutive runs against a busy entity should page someone. Push Glue metrics into CloudWatch and forward them wherever you already look, whether that is Grafana Cloud, Datadog or your own Prometheus.
• Reconcile on a schedule against SAP for a bounded window, and record the result. Drift you can see is a bug; drift you cannot see is an incident.
• Start with a filtered slice and a small page size while you are building. Get it correct, then tune throughput.
• Write down what each field means while the functional owner is still on the call. SAP field names do not explain themselves later.

Frequently asked questions

Does the AWS Glue SAP OData connector fall under SAP Note 3255746?

The note restricts the RFC modules of the ODP Data Replication API. The Glue connector uses OData over HTTPS against an activated Gateway service, which is the interface SAP directs customers toward. Read the current revision of the note yourself and get it confirmed with your SAP account team before you rely on any summary, including this one.

Where should I store the delta token?

Anywhere durable and versioned that your job can write to transactionally: DynamoDB, Parameter Store, or S3 with versioning. What matters more than the store is the ordering. Write it only after the target write commits, and keep the previous value so you can roll back after a bad run.

Can I run two Glue jobs against the same SAP entity?

Yes, but give them separate SAP service users. A delta token is scoped to the service collection and the user, so two jobs on one user will invalidate each other’s tokens whenever either re-initialises.

Why do deleted SAP records still appear in my data lake?

Because the connector reports them rather than applying them. Deletes come back as rows with DML_STATUS set to DELETED. If your pipeline appends without a merge, those rows sit in the target looking like live data. Merge into Iceberg, or filter through a current-state view that every consumer actually uses.

What page size should I use?

The default is 50,000 and the ceiling is 500,000. Larger pages mean fewer round trips but bigger responses for SAP to assemble, and Gateway or a load balancer may cut them short. Start at the default, raise it while watching for timeouts, and agree the ceiling with your Basis team.

Can I partition an ODP extraction for speed?

Not with record-based partitioning. ODP entities handle pagination through skip and delta tokens, so concurrency is forced to null there regardless of your input. Field-based partitioning on a bounded range is available for non-ODP OData entities. For ODP, tune page size and narrow the entity instead.

Can Glue write data back into SAP?

Yes. The SAP OData connector supports insert and update operations, with the external ID field named through ID_FIELD_NAMES on updates. Test that against a sandbox client with a functional owner watching, because a bad write into an ERP system is a very different class of incident to a bad read.

The one thing worth remembering

SAP data integration with AWS Glue is not hard because the connector is difficult to configure. Configuration takes an afternoon. It is hard because the correctness of your pipeline depends on a token held in a queue on a system your team does not administer, subject to a retention policy you may not have been told about, and shared with every other subscriber using the same credentials.

Design for that. Isolate the users, persist the token deliberately, apply deletes rather than collecting them, and alert on silence as loudly as you alert on errors. Do that and the connector is a good piece of engineering that will run for a long time without your attention. Skip it and you get a job that stays green while the numbers quietly stop being true.

Need a hand with an SAP to AWS pipeline?

This is the kind of work I take on regularly. If you are standing one up, or you have one that has started drifting, I can help with:

• Designing delta token state management that survives failed runs, redeploys and concurrent pipelines.
• Building merge logic that applies DML_STATUS deletes correctly into Iceberg or Redshift instead of accumulating ghost rows.
• Diagnosing RODPS_REPL_ODP_OPEN failures, silent zero-row runs and row count drift, and working out how far back the damage goes.
• Setting up SAP OData connections properly: Secrets Manager, least-privilege IAM, VPC connectivity and private certificate authorities.
• Tuning page size, filters and job sizing so incremental runs cost what an incremental run should cost.
• Adding reconciliation and absence alerting so drift shows up in a dashboard rather than in a finance report.

Send me a job log, a connection config, or the row counts that stopped matching, and I will tell you what I think is going on.

The post SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos appeared first on John Nessime.