So you open the audit table. And you find it records who requested a link to the document. Not who opened it. The actual downloads never touched your application at all, because you handed out a signed storage URL and the browser went straight to the bucket. Your log is a record of intentions.

That gap is typical of how property SaaS document storage goes wrong. The failures are rarely dramatic. Nobody dumps a bucket publicly. The guarantees you thought you had just turn out to be enforced in application code that ran once, long before the file was actually read. What follows is five failure families worth designing against when a platform stores tenant ID scans, signed leases, deposit certificates, right-to-rent evidence and referencing packs.

What’s actually in the pile

Before architecture, be honest about the contents. A property platform’s document store is at least four risk classes wearing the same file extension:

• Identity evidence. Passport and driving licence scans, visa documents, sometimes a selfie held next to the document. Highest blast radius, lowest genuine need for long retention.
• Contracts. Tenancy agreements, addenda, guarantor deeds, signed inventories. Legally driven retention, plus an integrity requirement: proving the file hasn’t changed matters more than hiding it.
• Financial and referencing material. Bank statements, payslips, credit reference reports. Sensitive, third-party sourced, and often covered by contractual restrictions from the referencing provider.
• Operational clutter. Inspection photos, safety certificates, maintenance invoices. Low sensitivity, high volume, and the thing that dominates your storage bill.

These want different retention clocks, different access rules and arguably different buckets. Treating them as one “documents” table is where most of the later pain originates.

Failure one: the access check and the access happen at different times

Presigned URLs are the right pattern. They keep the bucket private, take the transfer off your servers, and mean you never hand storage credentials to a browser. I reach for them first.

The problem is what a presigned URL is. It’s a request you already authorised, signed, and then gave away. Anyone holding it can replay it until it expires. Your permission logic ran once and never runs again. Which means all of these keep working after the fact:

• A letting agent who left the company yesterday and whose account you disabled this morning.
• A link forwarded into a WhatsApp group, pasted into a support ticket, or sitting in browser history on a shared office machine.
• A document the tenant asked you to delete, if the object is still there behind a delete marker.

Make the URL die when the session does

Amazon’s own guidance is clearer on this than most people realise. A presigned URL created with temporary credentials expires when those credentials expire, even if you asked for a longer expiry. It also stops working if the signing credential is revoked, deleted or deactivated.

That’s a lever, not a footnote. Sign with a long-lived IAM user key and the seven day maximum the SDK allows is a real seven days. Sign inside a short-lived assumed-role session and the URL cannot outlive that session, whatever number you passed in. So: never sign document URLs with static credentials. Assume a role per request, tag the session with the tenant, sign inside it.

import boto3

sts = boto3.client("sts")

# The tag key here must match the principal tag key used in the role's policy.
session = sts.assume_role(
    RoleArn="arn:aws:iam::111122223333:role/TenantDocumentAccess",
    RoleSessionName="doc-download",
    Tags=[{"Key": "TenantID", "Value": tenant_id}],
    DurationSeconds=900,
)["Credentials"]

s3 = boto3.client(
    "s3",
    aws_access_key_id=session["AccessKeyId"],
    aws_secret_access_key=session["SecretAccessKey"],
    aws_session_token=session["SessionToken"],
)

url = s3.generate_presigned_url(
    "get_object",
    Params={
        "Bucket": "prop-docs",
        "Key": f"tenants/{tenant_id}/identity/{doc_id}.pdf",
        "ResponseContentDisposition": 'attachment; filename="id-document.pdf"',
        "ResponseContentType": "application/octet-stream",
    },
    ExpiresIn=120,
)

Two things are doing work. DurationSeconds caps the real lifetime of the link regardless of ExpiresIn. And ResponseContentDisposition overrides how the object is served on this request, which matters for the serving problem further down.

The trade-off is real: an STS call per download adds latency, adds a dependency, and STS has request limits you can hit at volume. AWS’s prescriptive guidance says as much. For identity documents I’d pay that cost. For inspection photos I wouldn’t.

Log where the read actually happens

Your application log records issuance. To record retrieval you need logging at the storage layer: CloudTrail data events for object-level operations, or S3 server access logging. Data events are billed per event, so scope them to the identity and contract prefixes rather than the bucket holding thousands of inspection photos.

Failure two: isolation that lives in your ORM instead of your IAM policy

Ask most teams how tenant isolation works and you get a description of a WHERE clause. The key is built as tenants/{tenant_id}/... and the tenant ID comes from the session. Fine until one code path takes it from a request parameter, or a background job iterates without scoping, or someone writes an admin endpoint in a hurry.

Move the boundary below your code. With attribute-based access control the tenant becomes a session tag and the IAM policy interpolates it into the resource ARN. One role serves every tenant, and the policy count doesn’t grow as you onboard.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadWriteOwnTenantPrefix",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::prop-docs/tenants/${aws:PrincipalTag/TenantID}/*"
    },
    {
      "Sid": "ListOnlyOwnTenantPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::prop-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": "tenants/${aws:PrincipalTag/TenantID}/*"
        }
      }
    }
  ]
}

Two statements because S3 splits the problem. The first governs the object, where the tenant prefix is part of the resource ARN. The second governs listing, a bucket-level action, so the prefix has to be constrained through a condition instead. Skip it and a tenant can enumerate every key in the bucket even though they can’t read any of them, and key names in this domain leak plenty on their own.

The failure mode worth internalising: if a code path forgets the session tag, ${aws:PrincipalTag/TenantID} resolves to nothing and the statement doesn’t match. That’s a deny, which is what you want. But it only holds if the role has no broader fallback statement. One well-meaning s3:GetObject on the whole bucket, added to unblock a migration script, quietly turns the scheme into decoration.

When to use a bucket per tenant instead

A bucket or an account per tenant is a genuinely harder boundary and I wouldn’t dismiss it. With a few dozen large agency customers it’s defensible and makes offboarding trivially provable. It stops being defensible once you onboard self-serve landlords, because bucket and account counts run into service quotas and the operational surface grows with every signup. Pick on tenant count, not on which sounds more secure. And note that ABAC is only as trustworthy as the tenant claim: if the caller can influence the value reaching the tagging call, you’ve built an elegant no-op.

Failure three: the upload and serve path

Uploads here come from people you have no relationship with yet. Prospective tenants, guarantors, contractors. That’s an untrusted channel by definition, and it produces two separate problems.

Constrain the upload itself

A presigned PUT URL lets the holder write whatever they like to that key, at whatever size. Presigned POST is the better tool for public-facing uploads: the signature covers a policy document, and the policy carries conditions the browser cannot alter.

post = s3.generate_presigned_post(
    Bucket="prop-docs",
    Key=f"tenants/{tenant_id}/inbox/{upload_id}",
    Fields={"Content-Type": "application/pdf"},
    Conditions=[
        {"Content-Type": "application/pdf"},
        ["content-length-range", 1, 15728640],
    ],
    ExpiresIn=600,
)

The content-length-range condition is the one people skip, and it’s the one that stops a stranger using your referencing portal as free storage for a 40 GB file. A declared Content-Type is a claim, not a fact, so land uploads in a quarantine prefix, verify the actual bytes, and only then move them to the tenant prefix.

Scan on that path too. ClamAV in a container is the zero-cost option and covers known malware; GuardDuty Malware Protection for S3 and hosted APIs such as Cloudmersive trade money for not maintaining it. What you’re protecting is mostly the letting agent who opens the attachment on a Windows machine, so this is a control for your customers rather than your servers.

Never serve tenant documents from your application origin

This is the one that turns a document store into an account takeover. SVG and HTML are formats browsers execute. If a file uploaded by a prospective tenant is served inline from the same origin as your app, any script inside it runs with your application’s cookies and session context. A long stream of real advisories follows exactly this shape, particularly in self-hosted deployments where the object store sits on the app’s own domain.

Three controls, and they stack rather than substitute:

1. Force download. Serve with Content-Disposition: attachment and X-Content-Type-Options: nosniff. Set these as object metadata at upload time and override them per request. The override protects links you mint; the stored metadata protects anything that bypasses your minting path.
2. Use a separate origin. Put user content on a distinct domain, not a subdomain of the app. GitHub’s separate apex for user content is the reference implementation. Same-origin policy then contains the blast radius if control one is missed somewhere.
3. Normalise where you can. Rendering an uploaded ID to a flattened image server side removes the executable content entirely. It costs CPU and fidelity. For identity documents, where you rarely need the original bytes back, it’s the right trade.

Failure four: deleting the document doesn’t delete the document

A tenant asks you to erase their passport scan. Your app sets a flag, the file disappears from the UI, and you reply confirming deletion. Then count the copies that still exist.

• The previous object versions, if versioning is on. A delete without a version ID just writes a delete marker.
• Cross-region replicas with their own versioning and lifecycle rules, plus any backup or snapshot taken before the deletion.
• Derivatives you generated and forgot: thumbnails, page previews, OCR text in a search index. Run a passport through Amazon Textract and the extracted fields are a copy of the sensitive data in a different shape.
• Copies that left the platform. A document attached to a CRM record in something like GoHighLevel, forwarded by email, or downloaded to a letting agent’s laptop.

The ICO’s position is that a valid erasure request extends to backups as well as live systems, and that where backup data cannot be immediately overwritten it must be put beyond use: isolated, unused for anything else, deleted when that backup cycles. Workable, but only if you know which backup sets contain documents and how long they cycle. If you can’t answer that, you can’t answer the request.

Crypto-shredding, and what it actually costs

The one technique that makes deletion provable across copies you can’t reach is a per-tenant KMS key. Encrypt that tenant’s objects under their own customer managed key; when they offboard, schedule the key for deletion. Every ciphertext copy anywhere, backups included, becomes undecryptable. You delete the data by deleting the ability to read it. The honest costs:

• KMS bills per key per month plus per request, so per-tenant keys are a real line item. Check current rates and quotas first: there’s an account limit on customer managed keys that a self-serve signup flow will eventually meet.
• Key deletion has a mandatory waiting period. Crypto-shredding is not instant, and your erasure response has to say so.
• It’s all-or-nothing per key. If a tenant asks you to erase one document rather than offboard, the key doesn’t help.

Object Lock fights erasure, on purpose

S3 Object Lock is tempting for signed tenancy agreements, because write-once storage is a strong integrity story. But compliance mode means nobody, root included, can delete the object until retention lapses. That is the feature. It is also a direct collision with an erasure request, and you cannot engineer your way out of it afterwards.

My line: Object Lock on executed contracts, where a statutory retention basis overrides erasure, and never on identity documents or referencing material. Different buckets, different rules, decided before anything is written. For per-document erasure elsewhere, lifecycle rules that expire noncurrent versions on the sensitive prefixes are worth setting up before you need them.

Failure five: storing what you never needed to store

The cheapest control on this list is not storing the file. The ICO is clear that UK GDPR sets no fixed retention periods and puts the burden on you to justify how long you keep personal data in identifiable form. That’s a design instruction, not a legal footnote.

For identity documents, ask what your platform needs once the check is done. Usually it’s the result, not the evidence: document type, issuing country, expiry date, a check reference, who performed it and when. If your customer’s regulator requires the image itself, that’s a retention rule with a stated basis, which is a different conversation from keeping it because deleting felt risky.

Practical version: give every document class a retention clock wired to a business event rather than upload date, and let lifecycle rules do the deleting. Tenancy end plus a defined period. Check completion plus a defined period. Not “whenever someone remembers.”

Troubleshooting the breakages you’ll actually hit

Everything broke after someone enabled S3 Bucket Keys

This one catches teams who did the sophisticated thing. If your IAM or KMS key policies use the object ARN as the encryption context to scope access, enabling an S3 Bucket Key changes that context to the bucket ARN and those policies stop matching. AWS documents it explicitly as something to fix before flipping the setting.

The quieter side effect: KMS CloudTrail events then log the bucket ARN instead of the object ARN, and there are far fewer of them because S3 caches the bucket-level key. If you were treating KMS CloudTrail volume as a proxy for document access, that signal degrades without anything appearing to break. Bucket Keys are still worth enabling for the request cost reduction. Move the conditions first, then re-check what your dashboards count.

The presigned URL expires far sooner than requested

Almost always the signing credential is a role session shorter than the expiry you asked for. This is documented behaviour, not a bug. Either raise the session duration within the role’s maximum, or accept the shorter window as the security property it is.

Listing works, reading returns Access Denied

Classic split between the bucket-level and object-level statements. Check both exist and reference the same tag key spelling. Session tag keys are case sensitive, and a tenantId in the code against a TenantID in the policy fails silently as a non-match.

Access denied only for encrypted objects

An SSE-KMS object needs two grants, not one. The S3 action on the object, and kms:Decrypt on the key it was encrypted under. In a per-tenant key setup that second grant is usually also tag-conditioned, so a missing or mismatched tag on the key itself produces a denial that looks like an S3 problem and isn’t.

Common mistakes

• Signing document URLs with a long-lived IAM user access key, which makes every link independently long-lived.
• Granting the tenant role a broad bucket-wide read statement alongside the tag-scoped one, usually to unblock a one-off script that then stayed.
• Serving user uploads from a subdomain of the application and assuming that counts as origin separation. It doesn’t for cookie scope.
• Enabling versioning for safety and never adding a noncurrent version expiry rule, so every deletion is reversible forever.
• Building deletion that handles the object but not the OCR output, the thumbnail, or the search index entry.
• Putting Object Lock on the bucket that also holds identity documents.

Design rules worth holding to

1. Separate buckets by risk class, not by feature. Identity, contracts, financial, operational. Different encryption, retention, logging and lock settings follow naturally from that split.
2. Enforce isolation in IAM, with application checks as defence in depth rather than the boundary itself.
3. Sign short, sign from a session. Minutes, not days, and always from assumed-role credentials so revocation actually revokes.
4. Deny unencrypted writes at the bucket policy, using the server-side encryption request headers as conditions, so a misconfigured client fails loudly instead of writing plaintext.
5. Give every document class an expiry clock tied to a business event, implemented in lifecycle rules rather than a cron job someone maintains.
6. Write the deletion runbook before launch, listing every place a copy lands. If you can’t enumerate them, that’s the finding.
7. Test cross-tenant access in CI. A negative test that asserts tenant A’s session cannot read tenant B’s key is worth more than any amount of policy review.

Frequently asked questions

How long should a presigned URL for a tenant ID document last?

Short enough that the user clicks it immediately. A couple of minutes covers a click-to-download flow. If the UI needs longer, re-mint on demand rather than extend. And remember the ceiling is set by the signing credential, so a short assumed-role session enforces the limit even if a later code change passes a bigger number.

Do I need AWS, or will Backblaze B2 or Cloudflare R2 do?

They all speak the S3 API and presigned URLs work on all of them, so the core patterns transfer. Wasabi, Backblaze B2, Cloudflare R2 and DigitalOcean Spaces are reasonable for the operational tier, and R2’s egress model is attractive if agents download a lot. They differ in the depth of the IAM story: session-tag ABAC and per-tenant KMS keys are AWS-specific. If isolation and crypto-shredding are hard requirements, keep the sensitive prefixes on S3 even when the bulk photos live somewhere cheaper.

Can I store scanned passports on a normal VPS instead?

You can, and plenty of platforms run the app tier on a VPS from Contabo or InterServer while keeping documents in object storage. What you lose on local disk is the deletion story and the access log. Proving an image is gone from a filesystem, its backups and its snapshots is far harder than scheduling a KMS key for deletion. If documents stay local, budget real effort for secure disposal, including on staff machines where tools like O&O SafeErase have a place.

What about e-signed tenancy agreements?

With a provider like DocuSign, the executed document and its audit trail live there as well as in your platform. That’s a second copy under a second retention regime, and it belongs in your deletion runbook and data map. It also strengthens your integrity story, which is a fair reason to skip Object Lock on contracts you retrieve from the provider.

Should the CDN cache tenant documents?

Not the sensitive classes. A CDN in front of private documents adds a cache that can serve content after your access decision changed, and header handling at the edge is a common source of the inline-rendering bug. CloudFront with origin access control and signed URLs is legitimate if you need the performance, but for identity documents I’d go straight to the bucket. For inspection photos, cache away.

How do I prove any of this to a prospect’s security team?

Artefacts, not assertions. The bucket policy, the ABAC policy, CI output showing cross-tenant denial, a sample of storage-layer access logs, and a retention schedule with a clock per document class. That set answers most questionnaires without a call.

The one thing worth remembering

Good property SaaS document storage is mostly about closing the gap between when you check access and when access happens. Every failure family above is a version of that gap: the presigned link that outlives the permission, the isolation enforced once in code and never at the storage layer, the deletion that updates a flag while the bytes sit behind a delete marker.

Close it by pushing the decision down: into IAM policies that evaluate on every request, into credentials short enough that revocation is real, into lifecycle rules that delete without anyone remembering to. Then the answer to “show us who read this tenant’s ID document” is a query, not a project.

Need a second pair of eyes on your document storage?

Most of what I do on property and lettings platforms is this exact work. Things I’d help with here:

• Reviewing bucket and IAM policies for cross-tenant read paths, including the ones added to unblock a migration and never removed.
• Moving tenant isolation from application code into session-tag ABAC, with a CI test that fails the build on cross-tenant access.
• Rebuilding the upload path with presigned POST conditions, quarantine, content verification and scanning before anything lands in a tenant prefix.
• Fixing the serving path so uploaded files can’t execute against your origin, including header handling through a CDN.
• Designing retention clocks and lifecycle rules per document class, plus a deletion runbook that names every copy including derivatives and third-party systems.
• Setting up storage-layer access logging scoped to the sensitive prefixes so you can actually answer an access request.

If it’s useful, send me your bucket policy and the IAM policy attached to your document role, with the account IDs redacted, and I’ll tell you what a determined tenant could reach.

The post Property SaaS Document Storage: The Five Failures That Show Up in an Audit appeared first on John Nessime.

That failure should shape how you think about EKS Pod Identity vs IRSA. Both deliver the same end result to your application: short-lived, rotated credentials with nothing hardcoded. The differences are in how trust is expressed, who has to touch IAM to change it, and what happens when the wiring is subtly wrong.

This post compares the two on what actually decides the choice: how each scales across clusters, where each refuses to work at all, the failures each hides, and a decision procedure you can apply to a real cluster.

Same outcome, two very different trust shapes

Most of the trade-offs fall out of the mechanism, so it is worth being precise about both.

IRSA: federation through an OIDC provider

The cluster has an OIDC issuer, which you register as an identity provider in IAM. A projected service account token is mounted into the pod, and the SDK exchanges it for credentials through the web identity flow against STS. The role’s trust policy is what makes that work, and it names one specific cluster:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub":
          "system:serviceaccount:production:my-app",
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud":
          "sts.amazonaws.com"
      }
    }
  }]
}

The important detail is that the issuer URL is unique per cluster. The relationship between a workload and a role lives inside IAM, and it is cluster-specific by construction.

Pod Identity: an EKS API and an on-node agent

Pod Identity moves that mapping out of IAM. You install the EKS Pod Identity Agent add-on, which runs as a DaemonSet, then create an association through the EKS API binding a namespace plus service account name to a role. The agent calls the EKS Auth API for temporary credentials, and the pod picks them up through the container credential provider that AWS SDKs already support.

The trust policy contains nothing cluster-specific:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
    "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": [ "sts:AssumeRole", "sts:TagSession" ]
  }]
}

sts:TagSession is not decoration. Pod Identity attaches session tags on every assume, so a policy allowing only sts:AssumeRole fails. It is the most common first-attempt error.

Creating the association is a single call:

aws eks create-pod-identity-association 
  --cluster-name prod-eu 
  --namespace production 
  --service-account my-app 
  --role-arn arn:aws:iam::111122223333:role/my-app

Where IRSA still wins

Treating IRSA as legacy is the most common mistake here. For some workloads it is not the older option, it is the only option.

• Fargate pods. The Pod Identity Agent is a DaemonSet on EC2 nodes. Fargate has no node for it to land on, and AWS documents Fargate pods as unsupported. If part of your workload runs on Fargate, IRSA is the mechanism there, permanently, not as a stopgap.
• Windows nodes. Also documented as unsupported for Pod Identity.
• Older SDKs you cannot rebuild. The web identity flow has been in the SDKs far longer, so a vendor image you cannot patch is more likely to work with IRSA.
• Direct federation into workload accounts. If your model wants the pod’s identity federating straight into a role in the resource-owning account, IRSA does that natively. Pod Identity reaches other accounts by role chaining, a different trust shape even when the outcome looks the same.
• You already have it working at a scale you control. A few clusters with a Terraform module that provisions the provider and the roles is not a problem that needs solving.

Where IRSA hurts

The pain is administrative, and it grows with cluster count rather than workload count. Every new cluster brings a new OIDC issuer, which means editing the trust policy of every role those workloads use. Trust policies have a size limit, so a role trusting many issuers eventually runs out of room and you end up duplicating roles to work around it.

The same asymmetry shows up in blue/green cluster upgrades. Standing a new cluster beside the old one means every role used by every migrating workload needs a trust policy edit before the new cluster can do anything, and another edit afterwards to clean up. That is IAM work on the critical path of an upgrade, usually owned by a different team than the one doing it.

Where Pod Identity wins

• Roles become portable. One role, one generic trust policy, any number of clusters. Adding a cluster is an EKS API call per workload, not an IAM edit per role.
• The permission split matches how teams are organised. Creating an association needs EKS permissions, not iam:UpdateAssumeRolePolicy, so cluster admins stop filing IAM tickets for every new service account.
• Session tags arrive for free. Every assume carries eks-cluster-arn, eks-cluster-name, kubernetes-namespace, kubernetes-service-account, kubernetes-pod-name and kubernetes-pod-uid. That is usable attribute-based access control and the CloudTrail audit trail you always wanted.
• Cross-account access without touching application code. An association can carry a target role ARN, and EKS performs the role chaining for you.

The ABAC angle is underrated. One policy can serve dozens of workloads, because the session tag identifies the caller:

"Condition": {
  "StringEquals": {
    "s3:ExistingObjectTag/team": "${aws:PrincipalTag/kubernetes-namespace}"
  }
}

Where Pod Identity hurts

The generic trust policy is a real trade-off. Written exactly as AWS shows it, that role can be assumed on behalf of any service account in any namespace in any cluster in the account, as long as someone with EKS permissions creates an association pointing at it. Blast radius control has moved from IAM into the EKS API. If those permissions are loosely held, you widened it.

The fix is to put the constraint back in the trust policy using the request tags Pod Identity sends:

"Condition": {
  "StringEquals": {
    "aws:RequestTag/kubernetes-namespace": "production",
    "aws:RequestTag/kubernetes-service-account": "my-app"
  }
}

You can also pin to a cluster with aws:SourceArn. That costs some of the portability you switched for, which is the point: decide per role. A role reading one team’s bucket should be pinned. A role used by an identical workload in six clusters should not.

The other cost is a new moving part. The agent is a DaemonSet, so it needs to tolerate your taints, and it must be running before pods on that node can get credentials.

The failures that actually bite

Silent fallback to the node role

This is the one from the opening, and it is invisible by design. Pod Identity works through the container credential provider, which the SDK finds by reading two environment variables EKS injects into the pod:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

An SDK that predates that provider does not error on these. It does not know what they mean, so it keeps walking the credential chain until it reaches instance metadata and picks up the node role. The application works. The permissions are wrong. Nothing complains.

Two defences, and you want both. Check the minimum SDK version list in the EKS documentation and pin base images above it, and block pod access to IMDS so falling back has nowhere to fall. The first prevents the problem; the second turns a silent success into a loud failure.

Verify from inside the pod. The ARN in the response tells you which identity is really in use:

kubectl exec -n production deploy/my-app -- aws sts get-caller-identity

The association that matches nothing

An association is an exact match on cluster, namespace and service account name. A typo in any of the three produces an association that exists, looks fine in the console, and never applies to a pod. List them and read them against your manifests:

aws eks list-pod-identity-associations --cluster-name prod-eu

Associations also only take effect for pods started after they exist. If you create one and nothing changes, restart the workload before you start debugging IAM.

Both mechanisms configured at once

During a migration you will often have a service account carrying an IRSA annotation and a Pod Identity association at the same time. Both paths can work, and which one the SDK uses depends on credential chain ordering. AWS documents that credentials found earlier in the chain keep being used even when an association exists.

Do not reason about it. Check the caller identity and check CloudTrail, then remove the one you are not keeping so nobody has to work it out again.

PackedPolicyTooLarge

Session tags, managed policy ARNs and inline session policies are compressed into a packed format with its own size limit, and six session tags on every assume is not free. The association can disable the automatic tags, at the cost of the ABAC and audit benefits that were half the reason to use Pod Identity.

A decision procedure for EKS Pod Identity vs IRSA

Work through these in order and stop at the first one that answers.

1. Does this workload run on Fargate or Windows nodes? IRSA. There is no decision to make.
2. Can you rebuild the image with a current SDK? If not, IRSA, or fix the image first.
3. Does your model require direct OIDC federation into the resource account? IRSA. Role chaining is not the same shape, even if it reaches the same bucket.
4. Will this role be used from more than one cluster, now or after the next upgrade? Pod Identity. The saving compounds every time you add or replace a cluster.
5. Do you want cluster and namespace in the CloudTrail record without extra work? Pod Identity, because of the session tags.
6. None of the above? Pod Identity for anything new, IRSA left alone where it already works.

AWS’s own guidance lands in the same place: Pod Identity for new applications on supported node types, IRSA where you already have it working or where Pod Identity cannot run.

Arguments that don’t survive contact

• “IRSA is deprecated.” It is not. AWS has said it continues to invest in it, and Fargate support alone guarantees it stays.
• “Pod Identity is less secure because the trust policy is generic.” Only if you leave it generic. Request tag conditions scope it as tightly as an IRSA trust policy, and you keep the session tags IRSA does not give you.
• “Run one mechanism per cluster, cleanly.” A mixed cluster is documented, supported, and for clusters with Fargate pods it is the permanent steady state, not a transitional mess.
• “EKS add-ons can only use IRSA.” True early on, and still repeated in older write-ups. Add-on creation and update now accept Pod Identity associations directly. Check your specific add-on rather than trusting a blog post, including this one.
• “Migrate everything, then delete the OIDC provider.” Deleting it breaks every role still trusting it, including ones you forgot. Migrate workload by workload, and delete the provider only after CloudTrail shows zero AssumeRoleWithWebIdentity calls for that issuer across a full batch cycle.

How I’d decide, and what I’d do first

• Default new workloads on EC2 or Karpenter-managed nodes to Pod Identity, and leave working IRSA alone until you have a reason to touch it.
• Add request tag conditions for anything sensitive. Portability helps a shared platform role and hurts a role that reads customer data.
• Restrict pod access to IMDS before you migrate anything, not after. It converts the silent failure into a visible one.
• Migrate with the role trusting both principals at once, so rollback is a manifest change rather than an IAM change.
• Put credential identity in your dashboards. A CloudTrail query counting calls made by node roles from workload namespaces catches the exact failure this post opened with. Amazon Managed Grafana or Grafana Cloud will render it if you already ship CloudTrail somewhere; a scheduled CLI check from a CI runner or a small VPS from Contabo or InterServer is a fine low-tech alternative.
• Codify associations in Terraform or OpenTofu from the start. They are cheap to create by hand and easy to lose track of.

Frequently asked questions

Is EKS Pod Identity replacing IRSA?

No. AWS recommends Pod Identity for new applications on supported node types and continues to support IRSA. Since Pod Identity cannot run on Fargate or Windows nodes, IRSA stays for the workloads that live there.

Can I use both in the same cluster?

Yes, and it is a documented, supported pattern. Older workloads on IRSA and new ones on Pod Identity is normal. Avoid configuring both for the same service account beyond a migration window, because which one wins depends on SDK credential chain ordering.

Why does my pod get AccessDenied after switching to Pod Identity?

Check the trust policy for sts:TagSession first. Pod Identity always sends session tags, so a trust policy allowing only sts:AssumeRole is rejected. After that, check that the namespace and service account in the association match the pod exactly.

Does Pod Identity work for cross-account access?

Yes, through role chaining. The association takes a role in the cluster account plus a target role in the resource account, and EKS chains them. It also exposes an external ID you can require in the target role’s trust policy to prevent a confused deputy problem.

Do I have to change my application code?

No, provided the SDK is recent enough. Both work through the default credential chain, so the application calls AWS as usual. The only change that bites is an SDK too old to understand the container credential provider.

How do I confirm which identity a pod is actually using?

Run aws sts get-caller-identity inside the container and read the assumed role ARN. If it names the node role rather than the one you assigned, the chain fell through. Cross-check in CloudTrail, where session tags also give you the namespace and service account for Pod Identity sessions.

The one thing worth remembering

The real question in EKS Pod Identity vs IRSA is not which is more modern. It is where you want the cluster-to-role relationship to live. IRSA puts it in IAM trust policies: precise, cluster-specific, and increasing administrative drag as clusters multiply. Pod Identity puts it in the EKS API: portable, fast to change, and it hands you the job of deciding per role how much of that portability you actually want.

Pick per workload, not per cluster. And whichever you pick, verify the identity from inside the pod, because the worst outcome here is not a permission error. It is a workload that has been running with the wrong credentials since the day you deployed it.

Need a second pair of eyes on your EKS workload identity?

Most of this work is unglamorous: reading trust policies, matching associations against manifests, proving what a pod is really authenticating as. Things I help with:

• Auditing every service account in a cluster and reporting which identity its pods actually use, not the one the manifest claims
• Planning and running an IRSA to Pod Identity migration workload by workload, with a rollback that needs no IAM change
• Writing scoped trust policies with request tag conditions, so portability is a deliberate choice per role
• Restricting pod access to IMDS and confirming nothing was silently depending on the node role
• Setting up cross-account access with target roles and external IDs, plus the CloudTrail view that proves it is used as intended
• Codifying associations and roles in Terraform or OpenTofu, so the next cluster is a variable change rather than a project

Send me a trust policy, a service account manifest, or the output of aws sts get-caller-identity from inside one of your pods, and I will tell you what it is really doing.

The post EKS Pod Identity vs IRSA: What Actually Decides the Choice appeared first on John Nessime.

By the time that ticket lands, the leak already happened. You can patch the query in an hour. What you cannot do is tell the affected garage how many other reports were wrong, how long it had been wrong, or whether anyone downloaded a CSV. There’s no log that answers those questions, because nothing ever denied anything. The database happily returned the rows. The API happily serialised them.

That’s the shape of the problem. Tenant isolation on AWS is not really about writing careful code. It’s about arranging things so that careless code fails loudly instead of quietly returning somebody else’s data.

This post walks through building a multi-tenant workshop management platform on AWS with isolation baked in from the first commit: where tenant context comes from, how the boundary changes shape per storage service, what the control plane has to own, and how you prove any of it works. The example is a shop management system, with job cards, vehicle histories, parts inventory and technician timesheets, but the patterns apply to any vertical SaaS product where one customer’s records must never touch another’s.

The failure mode that stays invisible

Most multi-tenant systems start with a tenant_id column and a convention: every query filters on it. That works right up until it doesn’t.

The convention breaks in ordinary ways. Someone adds a reporting endpoint and copies a query from a script that ran as an admin. A new join pulls in a table nobody remembered to filter. An ORM lazy-loads a relationship and the filter lives on the parent, not the child. A background job that recalculates parts margins runs without any tenant in scope at all, because it processes everything.

None of these throw. That’s the whole issue. A missing authorisation check produces a 403 you’ll notice in staging. A missing tenant filter produces a 200 with too many rows, and 200s don’t page anyone.

The fix is not more discipline. It’s moving the filter somewhere the application cannot forget it: into IAM, into the database engine, or both. AWS makes this point directly in its own SaaS guidance, and it’s the right one. If your only defence against cross-tenant reads is that developers remember, you don’t have a defence, you have a habit.

Pick the isolation model before you write code

There are three shapes, and they’re usually described as pool, silo and bridge.

• Pool. Every tenant shares the same tables, buckets and compute. Cheapest to run, cheapest to deploy, and the model where isolation has to be enforced explicitly because nothing physical separates anyone.
• Silo. Each tenant gets dedicated resources: its own database, its own bucket, sometimes its own account. Isolation is close to free, operations are not. Migrations, deploys and monitoring all multiply by tenant count.
• Bridge. Mixed. Shared compute, dedicated storage, or pooled for the standard tier and siloed for the enterprise tier that asked hard questions in procurement.

For a workshop platform, most independent garages will be small, and pooling is the only sane starting point. The trap is treating that as permanent. Sooner or later a dealer group with forty sites will ask for a dedicated database, and if you have not left room for a per-tenant routing decision, retrofitting it means rewriting your data access layer.

What I’d actually do: build pooled, but put the storage target behind a resolver from day one. A function that takes a tenant ID and returns a connection, a table name or a bucket prefix. When the first silo tenant arrives, you change the resolver, not four hundred call sites.

Where tenant context comes from

This is the part people get subtly wrong, and it undermines everything downstream. The tenant identifier must come from the authenticated identity, never from the request. Not a header, not a query parameter, not a field in the JSON body. If a client can influence the tenant ID, your isolation model is decoration.

In practice that means a custom claim in the token your identity provider issues. Amazon Cognito can carry a custom attribute for this, and so can any external IdP you federate with. The API layer reads the claim, and from that point the tenant is a fact about the caller rather than an input to the call.

Then you push that fact down into AWS itself using a session tag. When your service assumes a role, it attaches the tenant as a tag on the session. Every subsequent AWS API call made with those credentials carries the tag in the request context, where IAM policies can reference it as aws:PrincipalTag.

import boto3

def scoped_session(tenant_id, role_arn):
    sts = boto3.client("sts")
    resp = sts.assume_role(
        RoleArn=role_arn,
        RoleSessionName=f"workshop-{tenant_id}",
        Tags=[{"Key": "TenantID", "Value": tenant_id}],
    )
    c = resp["Credentials"]
    return boto3.Session(
        aws_access_key_id=c["AccessKeyId"],
        aws_secret_access_key=c["SecretAccessKey"],
        aws_session_token=c["SessionToken"],
    )

Two things make this work, and both are easy to miss. The role’s trust policy has to allow sts:TagSession alongside the assume-role action, or the call fails. And the calling principal needs permission to pass that tag. Get either wrong and you’ll spend an afternoon reading an error that sounds like it’s about the role rather than the tag.

One session tag, one role, one policy. That’s attribute-based access control, and it’s the reason ABAC scales where a role per tenant does not. Roles are a finite resource in an AWS account. Tags are not.

Tenant isolation on AWS changes shape per service

There is no single isolation control. Each service exposes a different lever, and you have to learn every one your architecture touches.

DynamoDB: the partition key is the boundary

If job cards live in DynamoDB with the tenant ID as the partition key, IAM can pin every read and write to that key. The condition key is dynamodb:LeadingKeys.

{
  "Effect": "Allow",
  "Action": [
    "dynamodb:GetItem",
    "dynamodb:PutItem",
    "dynamodb:UpdateItem",
    "dynamodb:Query"
  ],
  "Resource": "arn:aws:dynamodb:REGION:ACCOUNT:table/JobCards",
  "Condition": {
    "ForAllValues:StringEquals": {
      "dynamodb:LeadingKeys": ["${aws:PrincipalTag/TenantID}"]
    }
  }
}

Now a Query that omits the tenant partition key doesn’t return other tenants’ job cards. It gets denied. That’s the behaviour you want: loud, logged, and impossible to miss in CloudTrail.

Two caveats worth knowing before you commit. Scan operations don’t have a leading key to constrain, so granting dynamodb:Scan alongside this condition undermines the whole arrangement. And global secondary indexes have their own key structure, so an index whose partition key isn’t the tenant needs separate thought. Design the access patterns so that no query ever needs to look across tenants, and this stops being a problem.

S3: two surfaces, not one

Vehicle photos, inspection PDFs and signed job sheets go to S3 under a per-tenant prefix. The mistake is scoping only the object actions and leaving ListBucket open, which lets a tenant enumerate every other garage’s filenames even without reading them. Filenames leak plenty: customer names, registration plates, invoice numbers.

Object actions are scoped through the resource ARN. Listing is scoped through the s3:prefix request condition. You need both statements, because they protect different operations.

[
  {
    "Effect": "Allow",
    "Action": "s3:ListBucket",
    "Resource": "arn:aws:s3:::workshop-tenant-files",
    "Condition": {
      "StringLike": {
        "s3:prefix": ["${aws:PrincipalTag/TenantID}/*"]
      }
    }
  },
  {
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:PutObject"],
    "Resource": "arn:aws:s3:::workshop-tenant-files/${aws:PrincipalTag/TenantID}/*"
  }
]

The policy variable in the resource ARN is doing real work there. One policy, every tenant, no template rendering at request time.

Aurora PostgreSQL: row-level security, and the trap in it

Relational data is where most workshop platforms actually live, because job cards, parts lines and labour rates are relational. IAM cannot see inside a table, so the boundary moves into PostgreSQL itself via row-level security.

ALTER TABLE job_cards ENABLE ROW LEVEL SECURITY;
ALTER TABLE job_cards FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON job_cards
  USING (tenant_id = current_setting('app.tenant_id', true))
  WITH CHECK (tenant_id = current_setting('app.tenant_id', true));

USING controls which rows are visible to reads, updates and deletes. WITH CHECK controls what can be written. Without the second clause, a tenant can read only its own rows but insert a row stamped with someone else’s tenant ID. The first protects the read path, the second protects the write path, and you want both.

Now the trap, and it’s the one that gives teams false confidence. PostgreSQL superusers and roles carrying BYPASSRLS ignore row security entirely, and by default so does the table owner. If your application connects as the same role that ran the migrations, your policies are not in effect and your tests pass anyway. That’s why FORCE ROW LEVEL SECURITY is in the snippet above, and why the application should connect as a dedicated non-owner, non-superuser role.

The second trap is connection reuse. Set the tenant with SET LOCAL inside a transaction, or with set_config using the transaction-local flag. Plain SET persists for the life of the connection, and a pooled connection outlives the request. That’s how one garage’s context ends up serving the next garage’s query.

Test row-level security as the application role, against a pooled connection, with at least two tenants in the table. Testing as the owner tells you nothing.

Compute isolation is a different question

Data isolation stops one tenant reading another’s records. It does nothing about one tenant consuming everyone’s capacity.

Workshop platforms have a specific version of this. End of month, every garage runs its invoicing and MOT reminder batch at roughly the same time. A dealer group importing three years of service history will happily saturate a shared worker pool while forty independents wait for their job cards to load.

Levers worth knowing, roughly in order of how much they cost you:

• API Gateway usage plans, keyed per tenant, to cap request rates at the edge before anything expensive runs.
• Separate queues, or at minimum separate consumer concurrency, for bulk imports versus interactive requests. Bulk work should never share a lane with a screen someone is waiting on.
• Reserved or provisioned concurrency on the Lambda functions serving interactive paths, so a batch surge cannot starve them.
• Dedicated compute for premium tenants. This is silo by another name, and it’s the honest answer when a customer’s load profile genuinely doesn’t fit the pool.

Be honest with yourself about which problem you’re solving. Throttling is not isolation. It limits blast radius, it doesn’t create a boundary.

What the control plane has to own

Separate the control plane from the application plane early. The control plane manages tenants; the application plane serves them. Mixing the two is how a bug in the onboarding flow ends up with production credentials.

Onboarding a new workshop is a sequence, and it should be a single idempotent workflow rather than a checklist someone follows:

1. Generate a non-guessable tenant identifier. Lowercase alphanumeric, no customer name in it, because it will end up inside resource ARNs and key prefixes.
2. Write the tenant record: tier, isolation model, storage target, status.
3. Provision identity. User pool group or IdP mapping, with the tenant claim wired in.
4. Provision storage. For a pooled tenant that’s a prefix and a seeded row. For a siloed one it’s real infrastructure, which is why this step must be asynchronous.
5. Apply tags used for cost allocation and reporting.
6. Run a verification step that proves the new tenant can reach its own data and cannot reach a canary tenant’s data.

Step six is the one teams skip. It’s also the only step that tells you the previous five worked.

On per-tenant cost: activated cost allocation tags attribute anything that is a distinct tagged resource, which covers siloed tenants nicely. Pooled resources will not split by themselves, because a shared table doesn’t know which garage caused which read. If you need per-tenant margin, emit consumption as a metric dimension from the application: request counts, storage bytes, document pages processed. Tools like Vantage or CloudZero can allocate shared spend afterwards, but only from the signal you produce. Nothing recovers attribution you never recorded.

The AWS SaaS Builder Toolkit is worth a look here. It codifies control plane concepts as CDK constructs and will save you real time on onboarding plumbing. Read its own guidance first: the project describes itself as sample code, and expects you to review security fit before production. That’s a fair description rather than a warning label, but treat it as a starting point, not a finished platform.

Proving isolation actually holds

An isolation model you haven’t tried to break is a design document, not a control.

The tests that earn their keep are negative ones. Seed two tenants. Authenticate as the first. Then deliberately do the wrong thing: request the second tenant’s job card by ID, list the second tenant’s S3 prefix, run a query with the session variable set to the wrong value. Every one of those should fail, and the test should assert on the failure.

• Run the cross-tenant suite on every pull request, not nightly. It’s the regression that matters most and the one most likely to be introduced by an innocent refactor.
• Use the IAM policy simulator to check a policy change before it ships, particularly when someone widens a resource ARN.
• Turn on IAM Access Analyzer so external access grants surface without anyone having to notice them.
• Audit the database on a schedule: tables with RLS enabled but no policy attached, tables missing FORCE, application roles that have quietly acquired ownership or BYPASSRLS during an incident.
• Alert on AccessDenied volume per tenant. A spike is either a bug you introduced or someone probing, and both are worth knowing about.

Keep a permanent canary tenant in every environment, including production, holding nothing but synthetic data. Every negative test targets it. It costs almost nothing and it means your isolation tests never need real customer records.

Troubleshooting the failures you’ll actually hit

AccessDenied on a policy that looks correct

Almost always the session tag isn’t present. If the tag is missing from the request context, the condition can’t match and the statement doesn’t apply. Call sts:GetCallerIdentity with the scoped credentials and confirm you’re on the assumed role you think you are, then check CloudTrail for the AssumeRole event and look at whether the tag was actually passed. Tag keys are case sensitive, and TenantId is not TenantID.

Queries return zero rows instead of the right rows

Classic RLS symptom. The session variable is unset, so the policy predicate compares against null and nothing matches. Check current_setting inside the same transaction as the query, not in a separate connection from your SQL client. Empty results are the safe failure here, which is exactly why they’re easy to misread as a data problem.

One tenant intermittently sees another’s data

Intermittent means state reuse. Look at connection pooling first, then at any per-request context stored in thread-local or async-local storage that isn’t reset when the request finishes. If you’re using a proxy in front of the database, understand how it handles session state, because some proxies pin a connection to a client once session-level settings are detected, which changes the behaviour you tested against.

Isolation works in the API but not in background jobs

Because the job has no request, so it has no token, so it has no tenant. Whatever runs asynchronously needs the tenant carried on the message and a session assumed per tenant when the work is processed. A worker that loops over all tenants with admin credentials is the single most common place isolation quietly stops applying.

Session tags don’t survive a second AssumeRole

Session tags are not automatically carried forward when you chain roles unless they were marked transitive. If your architecture hops through more than one role, this is where the tenant context evaporates. Flatten the chain if you can; if you can’t, mark the tag transitive deliberately and document why.

Common mistakes

• Taking the tenant ID from a request header or body instead of the authenticated token.
• Using the customer’s name or a sequential integer as the tenant identifier, then putting it in bucket prefixes where it becomes both guessable and enumerable.
• Scoping S3 object actions but leaving bucket listing wide open.
• Running the application as the PostgreSQL table owner, so RLS is enabled and silently inert.
• Writing a USING clause with no WITH CHECK, leaving the write path open.
• Creating one IAM role per tenant and discovering the account ceiling somewhere around the point the business gets interesting.
• Assuming cost allocation tags will attribute pooled spend. They won’t, and by the time you need the numbers the history is gone.
• Testing isolation only through the UI, where the frontend is already sending the right tenant every time.

Best practices worth the effort

• Derive tenant context from the token, propagate it as a session tag, and never let application code choose it.
• Enforce the boundary at the layer below your code: IAM conditions for AWS resources, RLS for relational rows.
• Put storage targets behind a resolver so moving a tenant from pool to silo is a configuration change.
• Make onboarding one idempotent workflow that ends in a verification step.
• Emit tenant as a dimension on logs and metrics from the start, so cost and performance questions stay answerable.
• Keep a canary tenant and run cross-tenant negative tests in CI on every change.
• Put a WAF or edge layer such as Cloudflare in front of tenant subdomains, and keep tenant routing decisions out of the origin application where you can.

Frequently asked questions

Is a shared database with a tenant_id column ever acceptable?

Yes, provided the column is enforced by the engine rather than by convention. A shared table with row-level security, forced, queried by a non-owner role, is a real boundary. A shared table where every query is expected to include the filter is not. The schema is the same; the guarantee is completely different.

ABAC or dynamically generated IAM policies?

ABAC for most cases. One role, one policy, tenant supplied per session, and nothing grows as you add customers. Dynamic policy generation, sometimes called a token vending machine, earns its place when a single policy genuinely cannot express the rule, such as when per-tenant resource names have to be injected rather than a key prefix. The cost is that you now own the correctness of a policy generator, plus per-request latency, and session policies have a size ceiling you can hit.

Should each tenant get its own AWS account?

It’s the strongest boundary available and the most expensive to operate. For a workshop platform serving independent garages it’s overkill. It becomes reasonable when a customer’s contract, regulator or data residency requirement makes shared infrastructure a non-starter, and at that point you’re pricing it as a premium tier rather than absorbing it.

How do I handle a user who works at two workshops?

Model it as one identity with multiple tenant memberships and an explicit active tenant per session, rather than a token carrying a list. The active tenant becomes the session tag. Switching workshops means a new session, which is exactly the behaviour you want because it makes the switch visible in your audit trail.

Does row-level security hurt query performance?

It adds a predicate the planner has to satisfy, so the answer depends on your indexes. Index the tenant column, and index it as the leading column of composite indexes that support your common filters. Compare plans as the application role before and after enabling policies, because a plan captured as the owner may not reflect what the application actually runs.

Where does application-level authorisation fit?

Alongside, not instead. Tenant isolation answers “which organisation’s data is this”. Authorisation answers “may this technician void an invoice”. Different questions, different layers. A policy engine such as Amazon Verified Permissions handles the second cleanly, and keeping them separate stops role logic creeping into your isolation boundary.

Can I retrofit isolation onto a platform that already has tenants?

You can, and it’s tedious rather than impossible. Enforce at the database first, because that’s where the leak actually happens: table by table, FORCE enabled, application moved to a non-owner role. Then move the tenant ID out of request payloads and into the token. Add IAM conditions last, since they’re the least likely source of a live leak. Expect to find at least one background job with no tenant scope at all.

The one thing to remember

Tenant isolation on AWS works when the boundary sits below your application code, in a layer that denies rather than trusts. IAM conditions on session tags for AWS resources, forced row-level security for relational data, and a control plane that verifies a new tenant is properly fenced before anyone logs in.

Retrofitting that onto a running multi-tenant platform is possible but grim, because you’re doing it while real workshops have real data in the system. Doing it on day one costs a week. That’s the entire trade, and it’s not a close call.

Need a second pair of eyes on your multi-tenant architecture?

Most of the isolation problems I see are not exotic. They’re a missing WITH CHECK clause, an application connecting as the table owner, or a background job nobody scoped. Things I can help with:

• Reviewing an existing multi-tenant design and finding where the boundary is enforced by convention rather than by the platform.
• Implementing ABAC with STS session tags across DynamoDB, S3 and Aurora, including the trust policy wiring that trips people up.
• Setting up PostgreSQL row-level security correctly, with forced policies, a dedicated application role and pooling that doesn’t leak session state.
• Building a tenant onboarding workflow that provisions identity, storage and tagging idempotently and verifies itself.
• Writing the cross-tenant negative test suite and wiring it into CI so isolation regressions fail the build.
• Adding per-tenant usage metering so cost, performance and tier decisions rest on data instead of guesses.

Send me a policy document, an RLS definition or a CloudTrail AccessDenied event and I’ll tell you what it’s actually enforcing.

The post Tenant Isolation on AWS: Building a Multi-Tenant Workshop Platform That Doesn’t Leak appeared first on John Nessime.

That question is what actually bites when you run infrastructure for twenty clients. Not a breach. An audit, or a resignation. Someone gives notice on a Friday and you spend Monday working out which accounts they still hold keys to, because the credential everybody shared was never attached to a person. You can’t revoke a person from a password. You can only rotate it and hope you got all twenty.

This post covers how to design client AWS account access so that humans authenticate once against something you control, every client account holds a role that trusts exactly one thing, and every API call in a client’s CloudTrail can be traced back to a named engineer. It’s organised by failure family, because the mechanics are well documented and the failure modes are not.

The shape: one identity plane, twenty separate trust decisions

The instinct that causes all the trouble is putting credentials in the middle. Client gives you a login, you store it somewhere shared, engineers take it out when they need it. Everything downstream of that decision is damage control.

The shape that works is boring. One AWS account whose only job is identity. Your engineers exist there, once, in IAM Identity Center or federated from whatever directory you already pay for. Each client account holds an IAM role whose trust policy names one principal in that account and requires an external ID. Nobody stores a client credential, because there is no client credential. There’s a role your engineers can ask STS for a short-lived session against, and a client who can delete that role whenever they like.

There is a second design worth knowing about: inviting client accounts into your own AWS Organization. It’s genuinely better in some ways. You get service control policies, resource control policies, and centralised root access management across the estate, all of which are organisation-scoped features you simply cannot use on an account you don’t own. It’s also a much bigger commitment. You take on consolidated billing, the client’s spend flows through you, and pulling an account back out of an organisation at the end of a contract is a real project rather than a policy change.

For most consultancies and managed service providers, the client keeps their own organisation and you get a role. That’s the model the rest of this assumes.

Failure family 1: credentials you can’t revoke

The shared IAM user with an access key is the default because it works immediately. It keeps working right up until one of three moments: an engineer leaves, a client asks who did something, or a laptop gets stolen.

Access keys have no expiry. They’re bearer tokens with an indefinite lifetime, and the only way to invalidate one is to go into each account and delete it. At twenty accounts that’s twenty deliberate actions performed under time pressure, and the one you forget is the one that matters. A role session, by contrast, expires on its own. That’s the entire argument.

A shared password manager still earns its place, just not for admin keys. Bitwarden or 1Password is the right home for per-client external IDs, account numbers, role ARNs and the client-side contacts you need when something breaks at two in the morning. Metadata, not credentials.

Failure family 2: a trust policy that trusts more than you meant

This is the invisible one. The role gets created, the access works, everyone moves on, and the trust policy quietly grants far more than anyone intended.

Two mistakes travel together. The first is naming your account root in the principal. Writing arn:aws:iam::111122223333:root does not mean “the root user of my account.” It means any principal in that account that has permission to call AssumeRole. Every engineer, every Lambda function, every CI runner, every future role you haven’t created yet. It delegates the decision to your own account’s IAM, which is fine when your IAM is tidy and quietly catastrophic when it isn’t.

The second is skipping the external ID. AWS calls the problem it solves the confused deputy: you are a third party that holds the ability to assume roles in many customer accounts, and without a shared secret in the condition, one customer who learns another customer’s role ARN could persuade your systems to act against it. The external ID is the value that makes each trust relationship specific rather than generic.

Generate it yourself, one per client, as a UUID or similar unguessable value. The client must not be the one choosing it, because if they can choose it they can choose someone else’s, and the control does nothing.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/MspOperationsAccess"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
        }
      }
    }
  ]
}

The principal names one role, not the account. If that role is compromised you have one thing to fix. If the account is named, you have an inventory problem you didn’t know you had.

Failure family 3: CloudTrail says the role, not the human

You’ve done everything above and a client emails asking who deleted the staging load balancer. You open their CloudTrail and every event attributes to the same assumed role. The role is shared by your whole team by design, so the log tells you your company did it. That’s not what they asked.

There are two knobs and they are not equivalent. The role session name is chosen freely by whoever calls AssumeRole, appears in the assumed-role ARN, and is trivially set to anything. It’s useful, but it’s a label, not evidence. Source identity is the one that carries weight: once set on a session it cannot be changed, it persists across chained role sessions, and it lands in CloudTrail in the session context of every subsequent call.

The catch is that the role’s trust policy has to permit sts:SetSourceIdentity alongside sts:AssumeRole, and you can require it with a condition so a session without one is refused outright.

"Action": [
  "sts:AssumeRole",
  "sts:SetSourceIdentity"
],
"Condition": {
  "StringEquals": {
    "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
  },
  "StringLike": {
    "sts:SourceIdentity": "*@yourcompany.com"
  }
}

If your engineers arrive through SAML federation, the identity provider can populate source identity from a directory attribute such as email, which means nobody has to remember to pass it. For direct CLI use it’s an explicit flag:

aws sts assume-role 
  --role-arn arn:aws:iam::444455556666:role/MspEngineerAccess 
  --role-session-name acme-investigation 
  --external-id 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02 
  --source-identity jane.doe@yourcompany.com

Getting this right is what turns “we take security seriously” into a sentence you can back with a log query. It is also, in my experience of what clients actually ask for, the single most persuasive thing on the list.

Failure family 4: one identity, twenty blast radii

A hub account is a concentration of risk. That’s the trade you’re making, and it’s worth making, but only if the hub is held to a higher standard than the accounts it reaches into.

Practically: not every engineer gets permission to assume every client’s role. Group permissions in the hub so the people on the Acme engagement reach Acme and nothing else. Use per-client role names rather than one shared name across all twenty, so a policy in the hub can’t wildcard its way into a client you stopped working with a year ago.

Tier the roles by what they’re for, not by who’s asking. A read-only role for investigation covers most of the work most days. A change role carries the write permissions the engagement actually needs. Break-glass sits separate, with a short maximum session duration and an alarm on every use.

Two extras are worth knowing about even if you don’t reach for them immediately. Session policies let the caller narrow a session further at assume time, which is useful for automation that only needs one bucket. Permissions boundaries cap what any role or user created inside the client account can do, which matters when part of your job is creating IAM there.

If your team already routes through a business VPN or zero trust gateway with stable egress addresses, a source IP condition on the trust policy is a cheap extra layer. NordLayer, Tailscale and Cloudflare’s zero trust products all give you predictable egress. Be honest about the cost, though: the moment something runs from CI, a client’s network or a laptop on hotel wifi, an IP condition becomes a support ticket. I’d add it once everything else works, not before.

Failure family 5: standing administrator access nobody remembers granting

Somewhere in the twenty accounts there is a role with AdministratorAccess attached, created during a migration, described in the ticket as temporary. It’s been eighteen months.

Here’s the honest part: AWS does not give you a native just-in-time approval workflow for cross-account role assumption. There’s no built-in “request four hours of write access, get approved in Slack, access expires automatically” for this pattern. You either build that on top of STS and your ticketing system, or you accept standing access and compensate with tight session durations, aggressive monitoring and a genuine periodic review. Both are defensible. Pretending you have JIT when you have a wiki page is not.

The cheap version that actually helps: set the maximum session duration on client roles to something short, alert on assumption of the break-glass role, and run a quarterly pass over service last accessed data to strip permissions nobody has used. IAM can generate that report per role.

# Kick off the report, then fetch it with the returned job ID
aws iam generate-service-last-accessed-details 
  --arn arn:aws:iam::444455556666:role/MspEngineerAccess

aws iam get-service-last-accessed-details --job-id JOB_ID_FROM_ABOVE

Run that in each client account and the conversation about narrowing permissions stops being theoretical. You’re not guessing what the engagement needs; you’re reading it.

Failure family 6: offboarding that only half happens

The whole point of the hub is that offboarding is one action. Disable the person in your directory and every path into every client account closes at once, because there was never a credential in a client account with their name on it.

That’s true only if nothing crept in sideways. Things that creep in sideways:

• IAM users created directly in a client account during an incident, because it was faster
• Access keys minted under a role by automation and never inventoried
• A client’s own console user shared with your team “just for the migration”
• Third party tools connected with their own cross-account roles that outlived the trial
• A departing engineer’s laptop with cached SSO tokens and stale profiles still on disk

IAM Access Analyzer is the tool that finds most of the first four. Its external access findings show every principal outside the account that can reach into it, which is exactly the list a client wants to see, and it’s also the list that reveals the monitoring vendor nobody remembers signing up for. For the laptop, whatever your hardware disposal process is, make sure it involves an actual wipe rather than a factory reset; tools like O&O SafeErase exist for that reason and are cheap next to the alternative.

Rolling client AWS account access out across twenty accounts

Doing this by hand twenty times guarantees twenty subtly different configurations. Make it a template and a checklist.

1. Write one CloudFormation template that creates the role, its trust policy and its permissions, parameterised by external ID and your operations account ID.
2. Generate a unique external ID per client and store it with the client’s account metadata, not in the template.
3. Send the client a launch stack link rather than asking for credentials. They deploy it in their own account, in their own console, and can read exactly what they’re granting.
4. Record the account ID, role ARN, external ID location, engagement scope and review date in a single inventory file in version control.
5. Generate everyone’s local AWS profiles from that inventory so nobody hand-edits a config file.
6. Verify each one with a call that proves identity and nothing else before you touch anything real.

The generated profile chains from your federated session into the client role. The AWS CLI reads these from the shared config file and handles the assume-role call for you:

# ~/.aws/config
[profile msp-hub]
sso_session = company
sso_account_id = 111122223333
sso_role_name = MspOperationsAccess
region = eu-west-1

[profile acme-readonly]
role_arn = arn:aws:iam::444455556666:role/MspEngineerReadOnly
source_profile = msp-hub
external_id = 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02
role_session_name = jane-doe
duration_seconds = 3600
region = eu-west-1

Then the two-second sanity check that has saved me from more mistakes than any other command:

aws sts get-caller-identity --profile acme-readonly

It returns the account ID and the assumed-role ARN. Read the account number before you run anything destructive. Twenty profiles in one config file is exactly the situation where muscle memory picks the wrong one.

For scheduled work that runs without a human, the same trust relationship serves a Lambda function or an EC2 instance role in your hub. A low-cost VPS from Contabo or InterServer will also do the job, but note the trade: an instance profile inside AWS gets rotated credentials automatically, while a box outside AWS needs a long-lived credential to bootstrap from, which reintroduces exactly what this design removes. Inside AWS is the cleaner answer.

Troubleshooting the failures you will actually hit

AccessDenied on AssumeRole

Three separate causes produce nearly the same message, and they need different fixes. Either the client’s trust policy doesn’t name your principal, or the external ID you sent doesn’t match what the condition expects, or your own identity in the hub lacks permission to call AssumeRole on that role ARN. Check them in that order, because the first two require the client and the third doesn’t.

Switch Role in the console silently fails

This one wastes an afternoon if you don’t know it. The console’s switch role feature does not support roles that require an external ID. It’s a documented limitation, not a bug in your policy. Engineers who need console access to those accounts should reach them through federation rather than the switch role box.

Sessions expiring after an hour when you asked for longer

When one role session assumes another role, that’s role chaining, and a chained session is capped at one hour regardless of the role’s maximum session duration setting. Since your engineers assume the client role from an already-assumed session in the hub, you’re chaining by definition. Plan around the hour rather than fighting it.

An authorization message you can’t read

Some denials come back with a long encoded blob attached. STS will decode it if your identity has permission to, and the decoded version usually names the exact statement that denied you.

aws sts decode-authorization-message --encoded-message <encoded-blob>

Common mistakes

• Naming the account root as the trust principal instead of the specific operations role
• Letting the client pick the external ID, or reusing one value across several clients
• Treating the role session name as an audit trail when it’s a caller-supplied string
• Asking a client for console credentials because a template felt like too much effort
• Granting AdministratorAccess for a migration and never revisiting it
• Building the hub properly and then leaving one legacy IAM user in one account as a fallback
• Assuming that removing someone from your directory also removed the keys they created by hand

Best practices worth the effort

• One identity plane, federated, with MFA enforced at the point the human authenticates
• A unique unguessable external ID per client, generated by you and stored with client metadata
• Source identity required in the trust policy so CloudTrail names a person, not a role
• Roles tiered by purpose: read-only for investigation, change for the engagement, break-glass alarmed
• Client-owned policies, deployed by the client from your template, revocable by the client in one action
• A version-controlled inventory of every account, role, scope and review date
• Access Analyzer external access findings reviewed on a schedule, in every client account

FAQ

Can’t I just ask each client to create an IAM user for me?

You can, and it’ll work on day one. The problem is day four hundred. That user’s access key never expires, it isn’t attached to any individual on your team, and revoking it means someone in the client’s organisation remembering to act. A role costs the client the same five minutes to create and hands them a control they can exercise unilaterally.

Do I need a different external ID for every client?

Yes. A shared external ID across your client base defeats the purpose, because the value’s entire job is to be specific to one trust relationship. Generate a UUID per client, treat it as a secret, and never let the client choose it.

Should I bring client accounts into my own AWS Organization?

Only if you’re also taking on their billing and the commercial relationship supports it. Inside your organisation you get service control policies, resource control policies and centralised root access management, which are real advantages. Outside it, those stay the client’s tools and your job is to recommend them well. Most consulting engagements sit outside.

How do I prove a departed engineer no longer has access?

Show the directory record disabling them, then show that no IAM user or access key in the client account carries their name, then show Access Analyzer external access findings listing only your operations role as an external principal. Three artefacts, all exportable. That’s the evidence pack.

Does MFA carry across the cross-account role assumption?

Don’t assume it does. The reliable place to enforce MFA is where the human authenticates in your identity plane, and the reliable control at the client boundary is the combination of a narrowly scoped principal and the external ID condition. If you want an MFA condition in the client’s trust policy as well, test it end to end against your actual federation path before you rely on it, because the request context differs between direct assumption and chained sessions.

What’s the minimum viable version if I only have three clients?

A hub account with federated engineers, one role per client scoped to a named principal with a unique external ID, and source identity required. Skip the tiering, skip the IP conditions, skip the JIT ambitions. Those three things give you revocation, attribution and a clean answer for the auditor, which is most of the value.

The one thing to take away

Client AWS account access is not really a permissions problem. It’s a revocation and attribution problem wearing a permissions costume. Any design where a credential exists that isn’t tied to a specific human will fail you at exactly the wrong moment, and no amount of policy tuning fixes that, because the gap isn’t in the policy.

Put the humans in one place you control. Put a narrowly scoped, externally identified role in each client account. Require source identity so the log names a person. Everything else on this page is refinement; those three are the structure. And the pleasant side effect is that “here’s how you revoke us, unilaterally, in one action” is a genuinely good thing to be able to say in a sales conversation.

Need a second pair of eyes on your client access model?

This is the kind of work I do for consultancies and managed service providers who’ve grown past the point where a shared vault is survivable. Specifically:

• Reviewing existing cross-account trust policies for over-broad principals and missing external IDs
• Designing a hub account and federation model that scales past a handful of clients without per-client special cases
• Building the CloudFormation or Terraform template clients deploy themselves, so onboarding stops involving credentials
• Turning on source identity end to end, including the identity provider side, so CloudTrail attributes to named engineers
• Producing the offboarding evidence pack clients and auditors ask for, and automating it so it isn’t a scramble
• Narrowing long-standing administrator roles down using service last accessed data, without breaking the engagement

If you want a concrete starting point, send me one trust policy and the AccessDenied message it produces. That’s usually enough to tell where the model is leaking.

The post Twenty Client AWS Accounts and Not One Shared Admin Credential appeared first on John Nessime.