That is a different question, and most evidence buckets fail it quietly. Versioning is on, the bucket is encrypted, access logging is enabled, and there is still an admin role in the account that can call DeleteObjectVersion on every object in it. Nothing is wrong until someone asks you to prove a negative.

An audit evidence repository on AWS is not a bucket with a good name. It is a write path, a retention decision you cannot take back, and a set of guardrails that survive the person who built them leaving the company. This post covers how to build one with S3 Object Lock and AWS Backup Vault Lock, and more usefully, the five ways I have seen the design go wrong in ways nobody notices until the evidence is needed or the bill arrives.

What “immutable” actually buys you on AWS

Three separate mechanisms get called immutability, and they protect different things.

• S3 Object Lock stops an object version from being overwritten or deleted for a defined period. It works only on versioned buckets, and once you enable it on a bucket you cannot disable it or suspend versioning again.
• AWS Backup Vault Lock does the same job for recovery points in a backup vault, with its own retention floor and ceiling enforced at the vault level.
• Integrity validation, such as CloudTrail’s digest files, proves a file has not changed since delivery. That is a separate property from being undeletable, and auditors increasingly ask for both.

Object Lock is the core of an audit evidence repository on AWS because evidence usually arrives as files: exported reports, signed approvals, config snapshots, log archives. Vault Lock matters when the evidence is a backup of a database or an EBS volume. Integrity validation is what turns “we kept it” into “we kept it and here is the hash chain”.

Failure one: picking the retention mode you cannot walk back

Object Lock has two retention modes and the difference is not cosmetic. In governance mode, a caller who holds the s3:BypassGovernanceRetention permission can shorten or remove retention and delete the version. In compliance mode, nobody can, including the account root user, and the retention period cannot be shortened or downgraded to governance.

Read that second sentence again with a typo in mind. Set seven years where you meant seven days, in compliance mode, and the only exit AWS documents is closing the account. That is not a bug. It is the entire point of the mode. But it means the mode is a business decision wearing a technical costume.

How I decide: compliance mode only where an external rule names a retention period and an external party will inspect it. Everything else starts in governance mode, with the bypass permission granted to nobody by default and denied outright in a service control policy. Governance mode with no principal holding the bypass right is functionally immutable for day-to-day purposes, and it leaves you a documented, auditable escape hatch that requires an explicit policy change to use.

The bypass is not silent, either. Overriding governance retention requires the x-amz-bypass-governance-retention:true header on the request, which means the action shows up in CloudTrail as a distinctly shaped call. That is a good alarm to build.

# Create the bucket with Object Lock on. Versioning is enabled automatically
# and cannot be suspended afterwards.
aws s3api create-bucket 
  --bucket my-evidence-archive 
  --object-lock-enabled-for-bucket 
  --region eu-west-1 
  --create-bucket-configuration LocationConstraint=eu-west-1

# Default retention for NEW objects written to the bucket.
aws s3api put-object-lock-configuration 
  --bucket my-evidence-archive 
  --object-lock-configuration 
  '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}}'

Start there. Run it for a quarter. Watch what actually lands in the bucket and how big it gets. Then decide whether compliance mode is worth its permanence.

Failure two: retention that was never applied to the objects that matter

This is the invisible one, and it is the reason to read the rest of this post.

Object Lock is enabled at the bucket level, but retention lives on the individual object version. A default retention rule applies to objects written after the rule exists. It does not reach backwards. Enable Object Lock on an existing bucket holding three years of evidence and you get a bucket that reports Object Lock as enabled while every object already in it carries no retention at all.

The bucket console looks correct. The compliance checklist gets ticked. The old evidence is still deletable by anyone with delete permissions.

Two things to do about it. First, backfill retention on existing versions, either object by object with put-object-retention or, for anything above a few thousand keys, with S3 Batch Operations, which exists precisely for applying retention at scale. Second, verify per object rather than per bucket.

# Check what protection a specific version actually carries.
# Look for ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus.
aws s3api head-object 
  --bucket my-evidence-archive 
  --key 2024/q3/access-review-export.csv 
  --version-id "SOME_VERSION_ID"

The stronger fix is to stop trusting the writer to set retention correctly and enforce a floor in the bucket policy. S3 exposes the s3:object-lock-remaining-retention-days condition key for exactly this, so you can reject writes that ask for too short a lock.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RejectShortRetention",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-evidence-archive/*",
      "Condition": {
        "NumericLessThan": {
          "s3:object-lock-remaining-retention-days": "365"
        }
      }
    }
  ]
}

The write path now fails loudly when someone gets it wrong, which is what you want. A rejected upload gets noticed. A silently unprotected object does not.

Failure three: the delete that looks like a delete

Object Lock protects object versions. It does not stop new versions being created, and it does not stop delete markers being placed on top of a protected object.

So a user with ordinary delete permissions can issue a delete against a locked object, get a success response, and make it vanish from every normal listing. The protected version is still there. Nothing was destroyed. But an auditor browsing the bucket sees nothing, and the person who ran the command believes the file is gone.

Three practical consequences:

• Deny s3:DeleteObject on the evidence prefix for everyone except a break-glass role. Immutability is not a substitute for permissions, it is a backstop under them.
• Any retrieval tooling you hand to an auditor must list object versions, not objects. list-object-versions, not list-objects-v2.
• Lifecycle rules interact with this in a way that surprises people. An expiration rule on a versioned bucket adds a delete marker rather than removing data, and a rule that tries to permanently expire a noncurrent version will not remove a version still under retention. Your cleanup policy and your retention policy are two different clocks, and the lock always wins.

Failure four: evidence that arrives already unprovable

Immutability starts when the object lands. Everything before that is chain of custody, and it is where most repositories are weakest.

If the evidence is CloudTrail, use the integrity validation CloudTrail already ships. With log file validation enabled, CloudTrail writes an hourly digest file containing hashes of the log files delivered in that hour, signs each digest with a per-region private key, and puts the digests in a separate folder in the same bucket. You can then verify the chain later.

# Turn on digest files for an existing trail.
aws cloudtrail update-trail 
  --name org-audit-trail 
  --enable-log-file-validation

# Verify a time range later. Files must still be in the location
# CloudTrail delivered them to, or validation cannot run.
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-audit-trail 
  --start-time 2020-01-01T00:00:00Z

Two details worth knowing before you design around it. Validation only covers log files referenced by digest files, so a gap in digest coverage is a gap in your proof. And the files must stay where CloudTrail put them, which means copying logs into a tidier prefix breaks the built-in verification and hands you the job of writing your own checker.

For evidence that is not CloudTrail, the equivalent is doing the work yourself: hash the artifact at the point of generation, record the hash somewhere separate from the artifact, and write both. A manifest per collection run, checksummed and locked alongside the files, is unglamorous and holds up.

One constraint that catches people mid-build: a bucket with Object Lock enabled cannot be used as the destination for S3 server access logs. If your design had one bucket receiving both the evidence and its own access logs, it needs splitting.

Failure five: the bill you cannot switch off

Compliance mode plus a long retention period plus archival storage classes is where cost surprises live, because every lever you would normally pull to reduce spend has been deliberately disabled.

The mechanics that drive it, none of which are secret but all of which get discovered late:

• Minimum storage durations. S3 Glacier Flexible Retrieval carries a 90 day minimum and S3 Glacier Deep Archive carries 180 days. Transition earlier than that and you are billed for the remainder of the minimum anyway.
• Per-object overhead in Glacier classes. AWS documents 40 KB of metadata overhead per object for Glacier Flexible Retrieval and Deep Archive, of which 8 KB is billed at S3 Standard rates and 32 KB at the destination rate. For a repository full of 2 KB JSON approvals, the overhead is the bill.
• Billing starts when the rule is satisfied. Lifecycle transitions to Glacier classes happen asynchronously, but you are charged at the destination rate, and the minimum duration clock starts, from the date the rule matched rather than the date the bytes physically moved.
• You cannot delete your way out. Under compliance mode, oversized or mistakenly retained objects stay until their retain-until date passes, whatever the finance conversation says.

The design response is to aggregate before you lock. Batch small artifacts into per-day or per-collection archives, checksum the archive, and lock the archive rather than thousands of tiny files. It costs you a little retrieval convenience and saves you a per-object overhead multiplier you cannot renegotiate later.

Also pick the retrieval tier against the real deadline. Deep Archive is the cheapest place to keep evidence and one of the slowest to get it back. If a regulator or a legal team may need files at short notice, that trade is a bad one regardless of the storage line item.

Putting the repository together

With the failure families in view, the build order is fairly short.

1. Separate the account. Evidence lives in its own AWS account, not a prefix in the production log bucket. The blast radius of a compromised production admin should not include the record of what they did.
2. Write the service control policies first. At the organisation level, deny s3:BypassGovernanceRetention, deny s3:PutBucketPolicy and s3:DeleteBucketPolicy on the evidence bucket, and deny disabling the trail. SCPs apply to the account root too, which is the point.
3. Create the bucket with Object Lock and a default governance retention. Block public access, enforce TLS with an aws:SecureTransport condition, and encrypt with a customer managed KMS key whose policy is written for this bucket alone.
4. Enforce the retention floor in the bucket policy using the remaining-retention-days condition, so a misconfigured collector fails rather than writes something unprotected.
5. Replicate to a second account or region. Object Lock works with S3 Replication, and the destination bucket must also have Object Lock enabled. This is your answer to region-level loss and to an account-level mistake.
6. Add Vault Lock for backup-shaped evidence. Snapshots and database backups belong in a locked backup vault rather than being exported to S3 by hand.
7. Alarm on the write path. A day with no new evidence objects is the failure that never pages anyone. Alert on absence, not just on errors.

Vault Lock deserves its own note because its compliance mode has a grace period that Object Lock does not. You choose a cooling-off window, which AWS requires to be at least three days, during which the lock can still be removed. After that the vault and its lock are immutable and cannot be changed or deleted by anyone, including AWS.

# Compliance mode: the presence of --changeable-for-days is what makes it
# compliance rather than governance. Minimum value is 3 (72 hours).
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name evidence-vault 
  --min-retention-days 365 
  --max-retention-days 2555 
  --changeable-for-days 3

Use that window. Run a backup job through the vault, confirm the retention arithmetic against what the policy document actually requires, and only then let the clock run out.

One more design question worth asking out loud: does every copy need to be inside AWS? A second, offline-ish copy on independent infrastructure answers a different threat than cross-region replication does, because it survives an account-level compromise. Providers like InterServer, Contabo and Hetzner sell cheap storage that can hold an encrypted archive, and several S3-compatible services implement Object Lock semantics of their own. Confirm the provider actually supports Object Lock rather than assuming an S3-compatible API implies it, because that varies.

Troubleshooting

AccessDenied when deleting a version you are allowed to delete

Usually governance retention plus a missing bypass. The IAM permission s3:BypassGovernanceRetention and the explicit bypass on the request are both required. If the object is in compliance mode, there is no combination of permissions that will work, and the error is telling you the truth.

# Only valid for governance mode, and only with the matching IAM permission.
aws s3api delete-object 
  --bucket my-evidence-archive 
  --key path/to/object 
  --version-id "SOME_VERSION_ID" 
  --bypass-governance-retention

Replication fails on an Object Lock bucket

The destination needs Object Lock enabled too. Replicated objects take the lock configuration of the source, so a destination without Object Lock cannot accept them. Check the destination bucket first, then the replication role’s permissions.

Lifecycle rule appears to do nothing

Two likely causes. The versions it targets are still under retention, so expiration cannot remove them. Or the rule targets current versions on a versioned bucket, in which case it is adding delete markers and the data is still there as noncurrent versions, still being billed.

validate-logs reports invalid or missing files

Confirm nothing moved the objects. Validation requires the log and digest files to be in the location CloudTrail delivered them to, and the command runs per region. A copy job that reorganised the prefix layout is the usual culprit.

Object Lock cannot be enabled on the bucket

Versioning has to be enabled first, and Object Lock cannot be turned off later. If the bucket is currently a server access log destination, that has to move before Object Lock will work for you at all.

Common mistakes

• Enabling Object Lock on an existing bucket and assuming the objects already in it are now protected. They are not, until retention is applied to each version.
• Choosing compliance mode on day one to look thorough, then discovering the retention period was wrong.
• Treating Object Lock as an access control. It is a floor under your permissions, not a replacement for them.
• Leaving the bypass permission attached to a general administrator role because it was needed once during testing.
• Locking millions of tiny objects into Glacier classes and paying per-object overhead on all of them.
• Storing evidence in the same account as the systems it describes, which makes the evidence only as trustworthy as that account’s admins.
• Never testing retrieval, so the first restore attempt happens under audit pressure with an unfamiliar Glacier restore workflow.

Best practices for an audit evidence repository on AWS

• Governance mode by default, compliance mode only where an external requirement names the period. Write down which rule drove the choice, in the same repository.
• Enforce the retention floor in the bucket policy so the write path fails loudly instead of writing unprotected objects.
• Use legal hold, not extended retention, for investigations. Legal holds have no expiry and are removed explicitly, which is the right shape for something that ends on an unknown date.
• Aggregate small artifacts into archives with a checksummed manifest before locking them.
• Replicate to a second account with Object Lock enabled, and check replication metrics rather than assuming.
• Rehearse retrieval quarterly, including the Glacier restore step, and keep the output as evidence of the rehearsal.
• Monitor for absence of writes. Tools like Amazon CloudWatch, Grafana or Datadog are all fine here. The alert that matters is “no evidence arrived today”.

Frequently asked questions

Can I enable S3 Object Lock on a bucket that already exists?

Yes. S3 supports enabling Object Lock on existing buckets, and versioning must be enabled. The important caveat is that enabling it does not retroactively protect the objects already stored. You apply retention to those separately, per object or with S3 Batch Operations.

What is the difference between governance mode and compliance mode?

Governance mode can be overridden by a principal holding s3:BypassGovernanceRetention who sends the bypass header. Compliance mode cannot be overridden by anyone, including the root user, and the retention period cannot be shortened or the mode downgraded.

Does S3 Object Lock stop someone deleting the file?

It stops the protected version being destroyed. It does not stop a delete marker being placed on top, which hides the object from ordinary listings while the version remains. Keep delete permissions tight and list versions when you audit.

Do I need Object Lock if I already have versioning and MFA delete?

They solve different problems. Versioning preserves history but a sufficiently privileged caller can still remove versions. Object Lock removes that possibility for the duration of the retention period, which is the property an auditor is asking about.

How do I prove the evidence has not been modified?

For CloudTrail, enable log file integrity validation and keep the signed digest files with the logs, then run aws cloudtrail validate-logs for the period in question. For other artifacts, hash at generation time, store the manifest separately, and lock both.

Can a lifecycle rule delete objects that are still locked?

No. Retention wins over lifecycle expiration. Plan the two together, because a cleanup rule that quietly does nothing for years is a cost problem you will not see until you look at storage growth.

Is Vault Lock compliance mode reversible?

Only during the cooling-off window you set at creation, which must be at least three days. After the lock date, the vault and its lock cannot be changed or deleted by anyone, including AWS.

The one thing worth remembering

An audit evidence repository on AWS fails in a specific, quiet way: the bucket says Object Lock is enabled, and the objects inside carry no retention. Everything else in this post is downstream of that gap. Check protection at the object version, not at the bucket, and enforce the floor in the write path so the system tells you when it is wrong.

Then pick the mode like the one-way door it is. Governance mode with a locked-down bypass gets you almost all of the protection and keeps a documented way back. Compliance mode is worth its permanence when a real external rule demands it, and expensive theatre when it does not.

Need help building or reviewing one of these?

Immutable retention is easy to enable and easy to get subtly wrong, and the mistakes are the kind you find out about at the worst moment. This is work I take on directly:

• Designing an evidence account with Object Lock, KMS, service control policies and a locked-down write path
• Auditing an existing bucket to find object versions that carry no retention despite Object Lock being enabled
• Backfilling retention across existing objects with S3 Batch Operations without locking in the wrong period
• Modelling the long-run cost of a retention decision before compliance mode makes it permanent
• Setting up cross-account replication, AWS Backup Vault Lock and CloudTrail integrity validation as one coherent design
• Writing the retrieval runbook and rehearsing it, so the first restore is not during an audit

If you want a second opinion on what you already have, send me the bucket policy, the Object Lock configuration and the output of a head-object call on one old evidence file. That is usually enough to tell you where the gap is.

The post Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret appeared first on John Nessime.

So you sketch the obvious thing. Amazon Cognito for sign-in, Amazon S3 for the documents, a small front end in between. It’s three services and a weekend, and it works on the first try, which is exactly the problem. A secure patient document portal built this way passes every test you think to run, because the tests you think to run are all “can Alice download Alice’s file?” The one that matters is “what can Alice see about Bob?” and that one has an answer you won’t like.

This post covers the parts that go wrong after the demo. Where to put the authorization boundary, how to scope the S3 policy so one patient can’t enumerate another, why your download links expire earlier than you configured, what encryption actually gates, and what your audit trail looks like on the day someone asks for it. Everything here is mechanism, not a compliance checklist. Getting a Business Associate Agreement in place is a separate conversation with your lawyer and your AWS account team.

The failure that never pages you

Here’s the shape of it. You write an IAM policy that lets a signed-in user read objects under their own prefix. Something like arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*. You test it. Alice gets her file. Alice cannot get Bob’s file. Ship it.

Then, because the front end needs to render a file list, someone adds s3:ListBucket. And s3:ListBucket is not an object-level action. It applies to the bucket ARN, not the object ARN, so the ${cognito-identity.amazonaws.com:sub} substitution in your resource string does nothing for it. Unless you add a separate condition on the request, a signed-in patient can list every key in the bucket.

They still can’t download anything. Every GetObject outside their prefix returns a denial. But they can read the key names. And key names in a clinic bucket are rarely neutral. patients/8f2a.../oncology-referral-2024.pdf tells you something about a person even if the bytes stay locked.

Nothing alerts on this. No error appears in any log you’re watching, because nothing errored. The listing succeeded. That’s the failure mode worth building around: in a document portal, the metadata leak is the one that doesn’t announce itself.

The fix is a condition key on the list action, not a resource string:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOwnPrefixOnly",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::clinic-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "patients/${cognito-identity.amazonaws.com:sub}/*"
          ]
        }
      }
    },
    {
      "Sid": "ReadOwnObjectsOnly",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*"
    }
  ]
}

Two statements, two different jobs. The first constrains what the caller may ask the bucket to enumerate. The second constrains what bytes the caller may pull. Write only the second one and you’ve secured the download path while leaving the discovery path wide open.

One trap in that variable. ${cognito-identity.amazonaws.com:sub} resolves to the identity ID from the Cognito identity pool, not the sub claim from the user pool. They are different values for the same human. If you name your prefixes with the user pool sub because that’s what your application code has handy, the policy will match nothing and every request will be denied, which at least fails loudly.

Decide where the authorization boundary lives

There are two honest architectures here and the choice shapes everything downstream. Pick deliberately.

Pattern A: browser holds AWS credentials

The user signs in to a Cognito user pool. The application exchanges that token with a Cognito identity pool for temporary AWS credentials, and the browser calls S3 directly using the AWS SDK.

It’s less code and there’s no server to run. The authorization boundary is entirely the IAM role policy attached to the identity pool. That policy is the whole security model, so it had better be right, and it’s harder to reason about than application code because the enforcement happens in a service you can’t step through with a debugger.

The real limitation is expressiveness. IAM policy conditions do not know that Bob’s referral was superseded, that Alice revoked consent last Tuesday, or that a document should only be visible after a clinician releases it. Anything with business logic in it has to be encoded as prefix structure or object tags, and that gets brittle fast.

Pattern B: backend brokers every object

The browser never touches AWS credentials. It sends the Cognito access token to your own API. The API validates the token, checks its own records for who may see what, and returns a presigned URL for that one object. S3 is reachable only by the API’s execution role.

You now have a place to put logic, a place to log intent alongside access, and a single choke point to rate limit. The cost is that you own a service, its uptime, and its dependencies.

For a small clinic build, this is the one I reach for first. Not because Pattern A is insecure, but because the questions clients ask in month three (“can we hide results until the doctor reviews them?”) are answerable in Pattern B and require a re-architecture in Pattern A. The identity-pool approach earns its keep when access really is a pure function of who you are, which in healthcare it rarely stays.

If you do want attribute-driven access without a backend, Cognito identity pools can map claims from the token onto principal tags on the temporary session, which you then reference in policy conditions. It works, and it is the right tool for coarse distinctions like patient versus staff. It is not the right tool for per-document consent state.

Your download links expire before you think they do

This one generates support tickets on week two. You set a presigned URL to last a few hours. Patients report links dying in under one. Nothing in your code changed.

A presigned URL carries a signature made with whatever credentials signed it, and it cannot outlive those credentials. AWS documents the ceilings clearly: with long-lived IAM user keys and Signature Version 4 the maximum is seven days; from the S3 console the range is one minute to twelve hours; with temporary credentials from an assumed role, the URL dies when the role session does, regardless of the expiry you asked for. A default AssumeRole session is one hour.

Your Lambda or container signing those URLs is using temporary credentials. So the effective lifetime is the shorter of your requested expiry and the remaining time on that role session, and the remaining time depends on when the runtime last refreshed. That’s the source of the flapping. You can watch it yourself:

# Sign a URL and inspect what was actually encoded into it.
aws s3 presign s3://clinic-docs/patients/example/scan.pdf --expires-in 3600

# The returned URL contains X-Amz-Date, X-Amz-Expires and X-Amz-Credential.
# X-Amz-Expires is the window you asked for. If X-Amz-Credential shows an
# ASIA... key id, you signed with temporary credentials and the real ceiling
# is the session, not the window.

The honest resolution is to stop fighting it. Sign short, on demand, when the user clicks. Fifteen minutes is generous for a download the patient is standing in front of. Long links are not a convenience feature in this context anyway, because a presigned URL is a bearer credential: whoever holds the string can fetch the object, signed in or not. Forwarding the email forwards the access. If your portal mails links, you have moved the authorization boundary into someone’s inbox.

Mail a notification that says a document is ready and link to the portal login. Never mail the signed URL itself.

Encryption that actually gates something

New S3 objects are encrypted at rest by default with S3-managed keys. That satisfies the sentence “data is encrypted at rest” and gates precisely nothing, because anyone with s3:GetObject gets plaintext back. It is a storage property, not an access control.

Server-side encryption with an AWS KMS customer managed key is different in kind. Reading an object now requires two independent grants: the S3 permission and kms:Decrypt on the key. Those live in different policies and can be owned by different people. A bucket policy mistake no longer hands over plaintext on its own, and revoking the key grant cuts off reads without touching a single S3 permission.

You also get a second audit stream. Every decrypt is a KMS API call, recorded separately from the S3 call, which is useful precisely because an attacker who tampers with one trail probably didn’t think about the other.

The costs are real and worth stating. KMS charges per request, so a portal that fetches many small objects will notice. S3 Bucket Keys reduce that request volume substantially by caching a bucket-level key, at the price of coarser detail in the KMS audit records. For a document portal with modest traffic and high sensitivity, I’d take the per-request cost and keep the granularity. For a thumbnail-heavy application, the opposite.

The audit trail you don’t have yet

Assume someone will eventually ask: who downloaded this specific file, and when? If you have not explicitly turned something on, the answer is that you don’t know.

CloudTrail is on for management events by default, and management events are control-plane operations: creating the bucket, changing its policy, attaching a role. Object-level reads and writes are data events, and data events are off by default because of volume and cost. GetObject on a patient scan is a data event. Unless you configured a trail with data events for that bucket, no record of the download exists.

# Log S3 object-level activity for one bucket on an existing trail.
# Advanced event selectors let you scope to a single bucket prefix rather
# than every object in the account.
aws cloudtrail put-event-selectors 
  --trail-name clinic-audit-trail 
  --advanced-event-selectors '[
    {
      "Name": "PHI bucket object access",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Data"]},
        {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
        {"Field": "resources.ARN", "StartsWith":
          ["arn:aws:s3:::clinic-docs/patients/"]}
      ]
    }
  ]'

S3 server access logs are the other option and they are not equivalent. They record HTTP-level request detail and are delivered on a best-effort basis with no delivery guarantee, which makes them useful for traffic analysis and weak as evidence. CloudTrail data events carry caller identity, which is the field you actually need when the question is “who.”

Two things to sort out while you’re in there. Send the trail to a bucket in a different account, or at minimum a bucket the portal’s roles cannot write to, so that compromising the application doesn’t compromise its own log. And put a lifecycle rule on the log bucket, because data event volume grows with usage and nobody budgets for it. If you already run centralized dashboards on something like Grafana Cloud or CloudWatch, wire an alert to unusual read volume per identity now rather than after the incident.

Cutting off access when someone leaves

Disabling a Cognito user does not immediately end their session, and this surprises people. Tokens are self-contained. An access token already in a browser stays cryptographically valid until it expires, typically about an hour.

Refresh tokens are the longer exposure. The default is thirty days, and app clients can be configured anywhere from sixty minutes to ten years. Ten-year refresh tokens on a portal holding medical records is not a hypothetical misconfiguration; it’s a checkbox someone ticked to stop users complaining about re-login.

Cognito gives you revocation. Token revocation is enabled by default on newly created app clients. AdminUserGlobalSignOut revokes all of a user’s refresh, ID, and access tokens from the server side, and the /oauth2/revoke endpoint kills one refresh token and everything descended from it.

# Server-side, authorized with IAM credentials. Revokes the user's refresh,
# ID and access tokens.
aws cognito-idp admin-user-global-sign-out 
  --user-pool-id us-east-1_EXAMPLE 
  --username patient-example

# Confirm revocation is switched on for the app client before you rely on it.
aws cognito-idp describe-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --query 'UserPoolClient.EnableTokenRevocation'

Now the part that catches teams out. AWS documents that a revoked token still passes verification in a generic JWT library that only checks signature and expiry. Revocation is enforced by Cognito, not baked into the token. If your API validates tokens by fetching the JWKS and calling a verify function, a revoked token sails straight through until it expires on its own.

Which means short access token lifetimes are load-bearing. Revocation shortens the tail; it doesn’t close the window. Refresh token rotation, available on the Essentials and Plus feature tiers, replaces the refresh token on each exchange and narrows the window a stolen one is useful for. If you’re paying for Plus for threat protection features, you already have it.

Troubleshooting the usual suspects

Every request returns access denied after login

Nine times out of ten the prefix in your policy variable doesn’t match the prefix in your bucket. Print the identity ID your session actually resolved to and compare it character by character against a real key. Remember the identity pool ID and the user pool sub are different values.

Signature does not match on a presigned URL

Usually something modified the request after signing. A client library adding a header, a proxy rewriting the path, or an email gateway URL-encoding the query string. Compare the X-Amz-SignedHeaders list in the URL against what the client actually sent.

Uploads succeed but downloads fail

Check the KMS key policy separately from the IAM policy. A role can hold kms:GenerateDataKey (enough to write) without kms:Decrypt (needed to read). The S3 error will not say the word KMS prominently, which is why this one burns an afternoon.

Browser uploads fail before any request reaches S3

That’s CORS. The bucket needs a CORS configuration allowing your portal’s origin, the methods you use, and any headers the SDK sends. Browser preflight failures never appear in S3 logs, so check the browser console first.

Nothing appears in CloudTrail

Data events are not shown in CloudTrail Event History. Query the destination bucket directly, with Athena or CloudTrail Lake, and allow a few minutes for delivery.

Common mistakes

• Granting s3:ListBucket without an s3:prefix condition, so patients can enumerate every key even though downloads are blocked.
• Putting identifying information in object keys or file names. Assume the key will leak eventually and use opaque identifiers.
• Emailing presigned URLs. The link is the credential, and inboxes get forwarded, backed up, and breached.
• Extending refresh token lifetime to silence complaints about re-login, instead of implementing rotation.
• Assuming default encryption means access control. It means the disk is encrypted, nothing more.
• Enabling versioning without a lifecycle rule, so deleted documents linger indefinitely as noncurrent versions you forgot exist.
• Skipping CloudTrail data events to save money, then discovering the gap during an access review.

Best practices for a secure patient document portal

• Keep one bucket for patient documents and nothing else. Mixed-purpose buckets make policies ambiguous and audits slow.
• Turn on S3 Block Public Access at the account level, not just the bucket. Account level survives someone creating a new bucket in a hurry.
• Enable versioning and set a noncurrent version expiration rule, so an overwritten document is recoverable but not permanent.
• Use a KMS customer managed key so decrypt authority is separable from S3 authority.
• Require MFA for staff accounts. TOTP costs nothing to enable and staff accounts are the ones with breadth.
• Sign download URLs on demand with short expiry, and always from a backend that logged why it signed.
• Put the portal behind a WAF and rate limiting. Cloudflare in front of the application, or AWS WAF on the distribution, both work; the point is that credential stuffing against a login page is the most likely thing to actually happen to you.
• Restrict administrative access to the AWS console and to any bastion by network as well as identity, whether that’s IAM Identity Center with condition keys or a VPN such as NordVPN or Surfshark for a small team without central identity.
• Write down the retention period before launch, then encode it as a lifecycle rule. Retention decided later is retention never decided.

Frequently asked questions

Are S3 and Cognito enough for HIPAA compliance?

Both are covered under AWS’s HIPAA eligibility program, and Cognito’s documentation states it is HIPAA-BAA eligible. That makes them usable building blocks, not compliance. You need a signed Business Associate Agreement with AWS, and the configuration, access reviews, training, and documented policies remain yours. Service eligibility is a prerequisite, not an outcome.

Should I use a Cognito identity pool or a backend API?

Use an identity pool when access is a pure function of identity and the rules will not change. Use a backend when documents have states, when a clinician gates release, or when you need to log intent alongside access. Most clinic portals become the second within a year.

How long should a presigned URL last?

Minutes, not hours. Sign at click time. The maximum with long-lived credentials is seven days, but that ceiling is irrelevant here because the link is a bearer credential and a longer window is a longer leak.

Can patients upload documents back to the portal?

Yes, using presigned PUT or presigned POST. Presigned POST is worth knowing about because its policy document can constrain content type and size before the object lands, which presigned PUT cannot. Either way, write uploads into a quarantine prefix, scan them, then move them.

Does enabling CloudTrail data events cost much?

Data events are billed per event recorded with no free tier, unlike management events. For a portal serving a few thousand downloads a month the amount is small; for a bucket also serving application assets it is not. Scope your event selector to the patient prefix rather than the whole account and the question mostly goes away.

What happens to sessions when I disable a user?

Nothing immediate. Existing access tokens remain valid until they expire. Call AdminUserGlobalSignOut to revoke tokens, and keep access token lifetime short so the residual window is measured in minutes.

Do I need CloudFront in front of the bucket?

Not for correctness. It helps when patients are geographically spread, when you want signed cookies for a set of objects rather than one URL per object, or when you want a single edge to attach a WAF to. For a single-clinic portal, direct presigned URLs are simpler and one fewer thing to misconfigure.

The one thing to remember

A signed-in token proves who someone is. It does not decide what they may reach. In a secure patient document portal the boundary is the policy condition, the KMS grant, and the code that decides whether to sign a URL at all, and every one of those is something you have to write on purpose.

The demo passes either way. The difference only shows up when someone reads a key name they shouldn’t have seen, and nothing in your system considers that an error.

If you build one of these, spend an hour before launch on exactly two things: sign in as a test patient and try to list the bucket, then ask your logs who downloaded a specific file last week. If either answer is uncomfortable, you’ve found your next task.

Need a second pair of eyes on your portal?

This is the kind of build where the gap between working and safe is invisible from the outside. I help small teams close it.

• Reviewing S3 bucket policies and Cognito identity pool roles for prefix isolation gaps, including the list-versus-get split most policies miss.
• Designing the boundary: identity pool credentials in the browser versus a backend that brokers presigned URLs, with the trade-offs written down for your stakeholders.
• Setting up KMS customer managed keys, key policies, and grant separation so decrypt authority is not implied by storage authority.
• Turning on scoped CloudTrail data events, building the Athena tables to query them, and alerting on abnormal per-identity read volume.
• Hardening the Cognito side: MFA, token lifetimes, refresh token rotation, and a revocation path that actually ends sessions.
• Writing the whole thing as Terraform or OpenTofu so the next change is a reviewed pull request rather than a console click.

Send me your bucket policy and your identity pool role, with the account IDs redacted, and I’ll tell you what a signed-in patient can actually see.

The post Building a Secure Patient Document Portal on S3 and Cognito appeared first on John Nessime.

That gap is the whole problem with permission cleanup. IAM Access Analyzer unused access findings are genuinely useful, and they are also built on evidence with known holes in it. AWS says so in its own documentation. Treat a finding as proof rather than as a lead and sooner or later you take something down.

This post covers what an unused access finding actually asserts, the four blind spots that turn a clean-looking finding into an outage, and a removal procedure that fails safe.

What an unused access finding actually claims

An unused access analyzer is a separate analyzer type from the external access one. You create it per account or across an organization, and it produces four kinds of finding: unused roles, unused IAM user access keys, unused IAM user passwords, and unused permissions at the service and action level for principals that are active.

aws accessanalyzer create-analyzer 
  --analyzer-name unused-access-prod 
  --type ACCOUNT_UNUSED_ACCESS 
  --configuration '{"unusedAccess": {"unusedAccessAge": 90}}'

unusedAccessAge is the threshold in days beyond which access counts as unused, and it accepts 1 to 365. Use ORGANIZATION_UNUSED_ACCESS from a delegated administrator account. Unused access findings don’t vary by Region, so you don’t need one analyzer per Region.

Here is the part that matters. The analyzer does not replay CloudTrail and reconstruct what your workload did. It reads IAM last accessed information, the same data behind the Access Advisor tab in the console. Everything the analyzer knows, and everything it doesn’t, comes from there. The blind spots below are last accessed blind spots, inherited wholesale.

The four blind spots that break production

1. Action-level tracking doesn’t cover everything

Action last accessed information only exists for the services and actions AWS has explicitly added to the tracked list, and it covers management actions. The IAM console spells out one case in plain text: action last accessed information is not available for Amazon S3 data events.

Think about what that means for a data pipeline. A role whose entire purpose is s3:GetObject and s3:PutObject against one bucket is doing object-level work all day. Bucket-level calls like ListBucket are management events and get tracked. The object-level calls that constitute the actual job are data events, and they sit outside action last accessed.

AWS puts the warning in the documentation itself: do not make permissions decisions based solely on the absence of tracking information. Almost nobody reads that sentence.

2. Denied attempts count as access

Service last accessed data includes all attempts to call an AWS API, not just successful ones. A role that has been hammering a service it has no permission for, failing every time, shows up as having used that service.

This runs opposite to the first blind spot, and it is why a findings list can be simultaneously too aggressive and too timid. Dead permissions look alive because a misconfigured client keeps retrying against them, so you clean up around them and leave the real cruft in place.

3. The calendar is longer than your window

A ninety-day tracking period is the common default and it is wrong for a predictable class of role:

• Disaster recovery and failover roles, exercised once or twice a year
• Quarterly close, audit, and regulatory reporting roles
• Annual key rotation or certificate renewal automation
• Break-glass roles, which should be unused by design
• Seasonal workloads that only wake up for a peak period

There is also a ceiling worth knowing. unusedAccessAge tops out at 365 days, while IAM reports last accessed activity for at least the trailing 400 days. The analyzer cannot be configured to span the full window the underlying data actually holds, so for anything on an annual cycle it will not settle the question on its own.

4. Service-level use says nothing about resource-level need

A finding operates on services and actions. Your policy probably operates on services, actions, resources, and conditions. Knowing a role used secretsmanager:GetSecretValue last week does not tell you which secret, and narrowing the policy to a specific resource ARN based on a finding is a guess dressed up as evidence.

This is where policy generation earns its place. It reads CloudTrail activity for a principal over a window and drafts a policy from what it observed, including resource-level detail where the log has it. Same caveat: policy generation does not identify action-level activity for data events either. Better first draft, not an oracle.

A removal procedure that fails safe

The core idea: deny before you delete. An explicit deny is one API call to add and one to remove, effective immediately in both directions. Deleting a policy statement or a role is not symmetrical; restoring it means someone reconstructing intent under pressure. Use deny as a reversible dry run for deletion.

1. Sort findings by blast radius, not by count. An unused permission on a role assumed by one Lambda is a different risk from one on a role fifty engineers assume daily.
2. Confirm the role’s activity independently before touching permissions.
3. Attach a scoped deny as an inline policy on the role, leaving the existing allow untouched. An explicit deny wins over any allow, so this simulates removal exactly with none of the loss.
4. Watch CloudTrail for denials attributed to that role. This is the observation window and it is the step people skip.
5. Wait through at least one full business cycle, longer if the workload has monthly or quarterly rhythm.
6. Only then edit the allow and remove the deny.

For step two, the cheapest independent check is the role’s own last-used timestamp:

aws iam get-role --role-name legacy-reporting-role 
  --query 'Role.RoleLastUsed'

That returns LastUsedDate and the Region where the role was last used. LastUsedDate is null if the role has not been used within the tracking period, which is not the same as never used. The Region field is quietly valuable: a supposedly dead role showing activity in a Region nobody works in is worth a conversation before a deletion.

For step three, the deny is deliberately narrow. Deny the flagged actions on the flagged service and nothing else:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CanaryDenyUnusedRedshift",
      "Effect": "Deny",
      "Action": "redshift:*",
      "Resource": "*"
    }
  ]
}

Give it a Sid you can grep for, because in six months somebody will find this policy and have no idea whether it is a canary or a control.

For step four, you want CloudTrail records where the request failed on authorization. The field is errorCode, and the values that matter are AccessDenied plus the EC2-specific Client.UnauthorizedOperation. Filter on the assumed-role session issuer so you catch every session. CloudTrail Lake or Athena over your trail bucket both work.

To make this an ongoing signal rather than a one-off query, wire a CloudWatch Logs metric filter on AccessDenied to a dashboard. Amazon Managed Grafana, self-hosted Grafana, or Grafana Cloud all render it fine. What matters is that it is broken down by principal, so a spike points at a role rather than at the account.

Deleting the role itself has a separate trap

When you name a principal by ARN in a policy, IAM stores the underlying unique principal ID rather than the string. Delete that role, recreate it with the identical name, and you get a new unique ID. Every policy that referenced the old one now points at something that no longer exists. Nothing errors at deletion time. The break surfaces later, in another account, when a cross-account assume-role call starts failing and the trust policy on screen looks exactly correct.

Before deleting a role, check the things a findings list will never tell you:

• Is it named as a principal in any resource policy, S3 bucket policy, KMS key policy, or another account’s trust policy?
• Is it attached to an instance profile?
• Is it managed by CloudFormation, Terraform, or OpenTofu? Deleting out of band leaves drift that surfaces on the next apply.
• Is it a service-linked role? Those are owned by a service and have their own deletion procedure.
• Is it the execution role for a Lambda function, ECS task, or Step Functions state machine that runs rarely?

Safe order: detach the permissions, leave the role and its trust policy in place, wait another cycle, then delete. An empty role is harmless. A missing one is not.

Keeping the findings list small enough to read

A findings list nobody triages is worse than none, because it produces the appearance of coverage. Two levers keep it honest. The first is exclusions, which can skip whole accounts or skip principals by tag, and can be updated at any time:

aws accessanalyzer update-analyzer 
  --analyzer-name unused-access-prod 
  --configuration '{
    "unusedAccess": {
      "analysisRule": {
        "exclusions": [
          {"accountIds": ["111122223333"]},
          {"resourceTags": [{"lifecycle": "break-glass"}]}
        ]
      }
    }
  }'

Tag-based exclusion is the one to reach for. Sandbox accounts churn roles and generate noise with no security value; break-glass roles are supposed to look unused. Excluding by tag survives new accounts joining the organization. An account ID list does not.

The second lever is archiving findings you have consciously accepted:

aws accessanalyzer update-findings 
  --analyzer-arn <your-analyzer-arn> 
  --ids <finding-id> 
  --status ARCHIVED

Archiving is a decision record, not a dismissal. Write down why somewhere durable, because the same finding will come back to a different person.

Scope has a cost dimension too. Unused access analysis is billed on the number of IAM roles and users analyzed per month, per analyzer, so a second analyzer with a different tracking period doubles the principals counted. If you watch spend through Cost Explorer or a layer like Vantage or CloudZero, confirm where this line lands before widening scope organization-wide.

Troubleshooting

No findings yet, or a finding that won’t clear after you fixed it. After creating or updating an analyzer it takes time for findings to appear, recent activity takes around four hours to reach last accessed data, and scans are periodic rather than instant. Give it a full cycle. If you use the CLI or API for last accessed reports, generate a fresh one; the old report will not update itself.

The finding says unused but the team insists otherwise. Assume the team is right and the tracking has a hole. Check whether the work is data-plane activity, then whether the actions involved are on the tracked list at all. Absence of tracking data and absence of use look identical from the findings page.

Your canary deny broke something unexpected. That is the canary working. Detach the inline policy, confirm the failures stop, then narrow the deny to specific actions rather than the whole service and run it again.

Something broke but no AccessDenied appears in CloudTrail. Check whether the failing call is a data event. A trail logs management events by default; data event logging is opt-in per resource type and carries its own charges. No configured data events, no record of the denial.

Common mistakes

• Treating a finding as evidence of non-use rather than absence of recorded use. Different claims.
• Deleting the role in the same change as removing its permissions, collapsing two reversible steps into one that isn’t.
• Setting the tracking period once and never revisiting it against the workload’s actual rhythm.
• Cleaning up in the console while Terraform or CloudFormation remains the source of truth, so the next apply puts it all back.
• Batching removals across many roles at once. When something breaks you have a bisect problem instead of a rollback.

Best practices for IAM Access Analyzer unused access cleanup

• Tag principals with their expected usage cadence at creation time. It costs nothing then and answers the hardest question later.
• Start with unused access keys and passwords on IAM users. Highest value, lowest risk removals in the whole list.
• Route findings to EventBridge and into your ticket queue so cleanup is tracked work rather than a periodic heroic sweep.
• Make the permissions change in code, apply it through the normal pipeline, and let the diff be the record.
• Keep an AccessDenied dashboard permanently, not just during a cleanup. Cheapest early warning you can build for this class of change.

Frequently asked questions

Does IAM Access Analyzer read my CloudTrail logs?

Not for unused access findings. Those come from IAM last accessed information. Policy generation is the feature that reads CloudTrail, and it is a separate operation run against a specific principal and time range.

What tracking period should I set?

Ninety days is reasonable for interactive human roles and continuously running workloads, and too short for anything on a quarterly or annual cycle. The value is configurable from 1 to 365 days. The right answer is a property of your workload, not a security best practice number.

Can a finding be wrong?

It can mislead in both directions. Permissions used only through data events can appear unused, and permissions that only ever produced denied attempts can appear used, because last accessed data counts all attempts rather than only successful ones.

Do I need an analyzer in every Region?

No. Unused access findings do not change based on Region, so one per account, or one at the organization level, is enough. External access analyzers are the ones with a Region dimension.

How is unused access analysis billed?

On the number of IAM roles and users analyzed per month, per analyzer. Scope is the cost lever, which is why exclusions matter for the bill and not just the noise. Check current published rates before enabling it organization-wide.

Should I delete unused roles or just strip their permissions?

Strip first, delete later, in two separate changes. A role with no permissions has almost no attack surface, and keeping it alive preserves any principal references pointing at it while you find out if anything still cares.

Conclusion

IAM Access Analyzer unused access findings answer a narrower question than the one you care about. They tell you what AWS did not record activity for. They do not tell you what your workload does not need, and they cannot tell you whether a permission that is used should be. The distance between those statements is where the outages live.

The one thing worth remembering: deny before you delete. An explicit deny gives you the exact behaviour of a removal with a one-call undo, and that single habit turns permission cleanup from a change you have to be brave about into one you can be systematic about.

Need help trimming IAM permissions without the outage

The risk in permission cleanup sits almost entirely in what the tooling cannot see. Things I can help with:

• Setting up account or organization-level unused access analyzers with tracking periods and tag-based exclusions that fit how your workloads run
• Building the AccessDenied observability layer, from CloudTrail queries through metric filters to a dashboard the team will actually look at
• Running staged deny-then-delete cleanups with the rollback path defined before anything changes
• Auditing roles for cross-account principal references, instance profiles, and IaC ownership before deletion
• Moving console-era IAM into Terraform or OpenTofu so the next cleanup is a reviewable diff

If you have a findings list you are not sure how to act on, send me the role’s policy JSON and the finding detail and I will tell you what I would check before touching it.

The post IAM Access Analyzer Unused Access: Removing Permissions Without Breaking Production appeared first on John Nessime.

That is the failure worth writing about. Drift itself is normal. Somebody fixes production at 2am, a managed service rotates a value, a policy engine stamps a tag. The part that breaks is the alert, and it breaks quietly, because a muted channel looks identical to a quiet one.

Switching on Terraform drift detection takes one flag. Building Terraform drift detection that engineers still read six months later means deciding what not to send: which plan mode you run, what you classify as noise, how you route by blast radius, and whether the message names who made the change. A detector firing daily with the same content is worse than none, because no detector leaves you knowingly blind while a muted one leaves you blind with a green dashboard. Every section below asks the same question differently: does this message change what a human does in the next ten minutes?

Pick the right plan mode or you will alert on your own backlog

This is the biggest source of false drift, and it catches almost everyone building their first check. A normal terraform plan compares three things: your configuration, your state, and the real infrastructure. It returns a non-zero exit code when any of them disagree, so it fires for real drift and just as loudly for a pull request that merged this morning and has not been applied. Your drift channel becomes a notification feed for your own deployment backlog, and people learn that most messages are work in progress.

Refresh-only mode narrows the question. HashiCorp documents it as a plan whose goal is only to update state and root module outputs to match changes made to remote objects outside of Terraform. That is the actual definition of drift.

terraform init -input=false

terraform plan -refresh-only -detailed-exitcode -lock=false -input=false -no-color

# exit 0  state matches the real infrastructure
# exit 1  the check itself failed
# exit 2  something changed outside Terraform

Two flags there matter more than they look. -detailed-exitcode is what makes the result machine-readable at all: without it a successful plan returns zero whether or not anything changed, so your job passes forever. And -lock=false is safe here specifically because a plan never writes state. Only terraform apply -refresh-only updates the state file. A read-only detector that grabs the state lock will collide with real applies, and the loser of that race is the engineer trying to ship.

The trade-off, stated plainly: refresh-only will not tell you that merged code has not shipped, and it will not find resources created outside Terraform, because it can only refresh what is already in state. I run both modes and route them separately. Refresh-only feeds the alert. Normal plan feeds a weekly digest tied to the repository, where unapplied changes are a delivery problem rather than a security one.

Classify the noise before you route anything

Most recurring drift falls into a few families. Sorting yours takes an hour and decides most of your routing rules.

Provider normalization

A JSON policy document comes back with keys reordered. An identifier returns in different case. An empty block reads back as null instead of an empty map. Terraform sees a difference because it compares a stored representation against an API response, and that round trip is not always lossless. Fix these in the configuration where you can, because a phantom diff that follows you into every plan also hides real diffs on the same resource.

Attributes another system legitimately owns

The classic is autoscaling capacity. Your config says two instances, a scaling policy moved it to five during a spike, and the next plan proposes taking it back to two. Applying that kills three instances mid-spike. The drift is real; reverting it is the wrong move.

resource "aws_autoscaling_group" "web" {
  min_size         = 2
  max_size         = 10
  desired_capacity = 2

  lifecycle {
    # Owned by the scaling policy after creation.
    # Reverting this during a spike removes live capacity.
    ignore_changes = [desired_capacity]
  }
}

Understand what you bought. ignore_changes stops Terraform proposing an update for that attribute, and it applies only after creation, not on the first apply. It also delegates the attribute permanently. If someone sets desired capacity to zero at 3am, your check will not raise a hand. Fair trade for capacity. Terrible trade for an ingress rule.

Tags stamped by something else

Organization tag policies, cost allocation tooling, backup schedulers and compliance agents all add tags. The AWS provider exposes both the tags you set and the effective set including defaults, and ignoring the effective set silences all tag drift at once. Convenient, and slightly dangerous if you use tags for access control, because tag-based policy conditions stop being watched. The same logic covers values the platform rotates or defaults on its own, like a managed secret version: they drift on a schedule that has nothing to do with your team, and belong in a digest.

The goal here is not a clean plan. A clean plan is a side effect. The goal is a decision per family: see it immediately, see it weekly, or never see it again.

Route by blast radius, not by count

“Drift detected: 12 resources changed” makes a human open a terminal before they know whether to care, and that friction is where attention goes to die. Split the output by what each resource can do if it is wrong:

• Wake someone. Anything governing exposure or access: security groups and firewall rules, IAM policies and roles, key policies, public access blocks, bucket policies, DNS records pointing at production.
• Review today. Instance sizes, scaling bounds, database parameters, retention settings, anything that costs money or changes capacity.
• Weekly digest. Tags, descriptions, names, and every family you classified as platform noise.

To split them you need structure, not text. Terraform’s machine-readable output emits a message for each resource that changed outside Terraform, carrying the resource address and the action it would take.

terraform plan -refresh-only -no-color -json 
  | jq -c 'select(.type == "resource_drift") | .change'

Check the exact field names inside that object against the machine-readable output reference for your Terraform version before building matching logic on them. Keep the severity mapping in the repository next to the code it describes. A resource type moving from digest to page should be a pull request, not something changed in a monitoring UI at midnight.

Make the alert answer “who did this”

Terraform tells you what changed. It has no idea who changed it, and that gap is where the investigation time goes. Half the drift you find is a legitimate incident fix nobody wrote down. If the alert names the actor, the response is a two-word reply in the thread instead of twenty minutes of archaeology. On AWS the audit trail can be queried by resource:

aws cloudtrail lookup-events 
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0123456789abcdef0

CloudTrail’s console event history only reaches back a limited window, so a weekly check may be hunting an event that has aged out; a trail delivering to S3, or CloudTrail Lake, covers the gap. Azure exposes the equivalent through Activity Logs, Google Cloud through Cloud Audit Logs. Wiring the lookup into the alert body does more for adoption than any amount of formatting.

Decide the remediation path before you automate it

There are three answers to a drift finding, and picking the wrong one can be worse than the drift.

1. Revert. Apply the configuration and push reality back. Right when the change was unauthorized or wrong.
2. Adopt. Change the code to match what is running. Right when the change was correct and the code was stale, which happens more than people admit.
3. Tolerate. Suppress the attribute deliberately, with a comment and an owner. Right when another system owns it.

The trap is automating option one. Auto-approving a remediation apply sounds like the natural endpoint of a drift pipeline, and it is fine right up until the drifted attribute is one that forces replacement. Then “restore the desired state” means destroying and recreating a live resource, unattended, at whatever hour the schedule runs. If you automate anyway, gate it hard: an allowlist of resource types, a check that the planned action is an in-place update rather than a replace, and a notification either way.

Troubleshooting a drift job that quietly stopped working

The nastiest failure mode is not a noisy check. It is a check reporting “no drift” because it never ran the plan successfully.

Exit code 2 kills your script

With set -e, a non-zero exit terminates the script. Exit code 2 means drift, which is the case you care about, so a naive script dies exactly when it has something to report. Capture the code without letting the shell act on it:

set -euo pipefail

terraform plan -refresh-only -detailed-exitcode -lock=false 
  -input=false -no-color > plan.txt 2>&1 && code=0 || code=$?

case "$code" in
  0) echo "no drift" ;;
  2) notify_drift plan.txt ;;
  *) notify_broken plan.txt; exit 1 ;;
esac

Note that the failure branch notifies too. Exit code 1 deserves the same urgency as drift, because a broken detector and a clean estate produce identical silence.

Credentials that rot without complaining

Drift checks run unattended, so nobody notices when a role trust policy changes, an OIDC subject claim stops matching after a repository rename, or a long-lived key gets rotated out. Add a deadman check: if the job has not reported within its expected interval, that absence is the alert. Same reasoning as heartbeat monitoring for backups.

A provider upgrade that drifts everything

Bump a provider and a new attribute default can appear across every resource of a type. Overnight the check reports hundreds of changed resources and whoever is on call assumes the tool is broken, which it sort of is. Commit the dependency lock file, pin the provider version, run the drift job on the same Terraform version as your apply pipeline, and treat provider upgrades as a reviewed change rather than something a bot merges.

Refresh times that outgrow the schedule

Refresh queries the API for every resource in state. Large workspaces get slow, then overlap their own schedule, then time out. The durable fix is splitting state along boundaries that already exist in your team structure. Terraform does offer targeting to narrow a plan, but it is documented as an exceptional-use option and prints a warning for good reason, so treat it as a stopgap rather than the design.

When to stop building this and buy it

A scheduled job in GitHub Actions or GitLab CI, or a systemd timer on a small always-on box from a provider like Contabo or InterServer, covers a handful of workspaces perfectly well and costs almost nothing. I would not reach for a platform before that stops working. What eventually stops working is the surrounding machinery: credentials for a dozen accounts across providers, concurrency against real applies, and retaining history so you can see that the same resource has drifted repeatedly this quarter, which is a process problem no single alert reveals.

That is where HCP Terraform’s health assessments, or a platform like Spacelift, Scalr or env0, starts earning its fee, since drift detection becomes a workspace setting with history and notifications attached rather than code you maintain. Two honest caveats: HCP Terraform gates health assessments behind its paid editions and requires remote or agent execution, so it is not a drop-in for local-state workflows, and any platform will happily send you the same undifferentiated wall of drift the homegrown script did. Severity tiering is a decision about your infrastructure that no vendor can make for you. Buying the runner does not buy the judgement.

Common mistakes

• Using normal plan mode for the alert. You notify yourself about unapplied work and teach people the channel is noise.
• Alerting on a count instead of a resource. “Nine resources changed” makes someone open a terminal to find out if they should care.
• Treating exit code 1 as a non-event. A detector that errors is indistinguishable from an estate with no drift.
• Taking the state lock for a read-only check. Monitoring should not be able to block a deployment.
• Auto-approving remediation across all resource types. One force-replacement attribute and the fix becomes the outage.

Best practices worth the effort

• Run refresh-only for drift and normal plan for unapplied changes, routed to different destinations.
• Keep the severity mapping in the repository, version controlled and reviewed like code.
• Give every suppression a comment naming the system that owns the attribute and a date to revisit.
• Alert on the detector failing and on the detector going quiet, not only on findings.
• Review recurring drift monthly. The same resource drifting repeatedly is a process gap, not an infrastructure one.

Terraform drift detection FAQ

What is the difference between a normal plan and refresh-only?

A normal plan compares configuration, state and reality, so it flags genuine drift and any committed change that has not been applied. Refresh-only compares state against reality only, which matches what most people mean by drift. Use refresh-only for the alert and normal plan for a delivery-focused report.

Does a refresh-only plan modify my state file?

No. Plan operations do not write state. Only applying a refresh-only plan updates the state file to match the drifted remote objects, and even then it makes no changes to infrastructure. That is why a detection job can safely skip the state lock.

How often should Terraform drift detection run?

Tie the interval to how long you would tolerate not knowing. Daily for production and anything holding customer data, weekly for lower environments, less often for slow-moving workspaces where refresh is expensive. Frequency matters less than whether the message gets read.

How do I find resources created outside Terraform?

Not with a plan. Terraform refreshes only what is already in state, so a resource it has never seen is invisible. You need a cloud-side inventory such as AWS Config, Azure Resource Graph or Cloud Asset Inventory, compared against your state, and import blocks to bring the results under management. Worth knowing that driftctl, the open-source tool most often recommended here, has moved into maintenance mode.

Does ignore_changes hide drift I actually care about?

Yes, and that is the point of using it. It stops Terraform proposing updates for that attribute regardless of what changed it, so a deliberate suppression and a malicious change look identical afterwards. Scope it to specific attributes, never whole resources you still care about, and document the owner.

Why did drift suddenly appear across hundreds of resources?

Almost always a provider or Terraform version change rather than something happening in your cloud account. A new provider release can add or reshape an attribute and every resource of that type reports a difference at once. Compare the versions used by the drift runner and the apply pipeline before investigating the infrastructure.

The one thing worth keeping

Terraform drift detection is not a monitoring problem, it is an attention problem. The flag that finds drift took an afternoon. Everything determining whether anyone still reads the output next quarter is judgement: refresh-only rather than normal plan, noise families sorted and suppressed deliberately, severity by blast radius rather than resource count, an actor name in the message, and an alert when the detector itself goes dark.

Judge it by one test: when a message lands in that channel, does anyone open it? If the honest answer is no, your detection is working and your alert is not.

Need help making your drift alerts worth reading?

This is common work once a Terraform estate grows past one team. Things I can help with:

• Auditing an existing drift job and finding the false positives that trained everyone to ignore it
• Building scheduled refresh-only checks in GitHub Actions, GitLab CI or a systemd timer, with correct exit code handling and a deadman alert
• Sorting recurring drift into families and deciding, per attribute, what to fix, adopt or suppress
• Building severity routing from the plan’s structured output so exposure changes page and tag changes do not
• Enriching alerts with CloudTrail, Activity Log or Cloud Audit Log lookups so findings arrive with an actor attached
• Splitting oversized state so refresh finishes inside its schedule, and judging whether a managed platform is worth it at your size

If you want a second opinion, send me a week of your drift output or the workflow file that produces it, and I will tell you what I would cut first.

The post Terraform Drift Detection That Nobody Mutes appeared first on John Nessime.

It granted iam:PassRole on one role ARN and lambda:CreateFunction in the same document. That is not a wildcard and it does not read as administrative access. It is administrative access anyway, to whatever that role can do, for anyone who can write a Lambda function.

This is the shape of the problem with AI-generated IAM policies. The failure is almost never the obvious one. Models have absorbed enough style guidance to avoid "Action": "*", so what comes back passes the eyeball test and the linter and still hands over more than you meant to give. The dangerous output is the one that looks careful.

This post covers what policy generation genuinely does well, the failure shapes that survive human review, and where the hard boundary belongs: not in the review step, which is fallible, but in a mechanism that cannot be talked out of its answer.

What AI-generated IAM policies get right

Start with the honest case, because it is a strong one. IAM has thousands of actions across hundreds of services and the naming is inconsistent in ways nobody holds in their head. Some services use Describe, some use Get, some use both for different things. ARN formats vary per service and per resource type inside a service. Condition keys apply to some actions and not others.

A model is good at that recall problem, and at shape work: splitting a monolithic statement into per-resource statements, adding conditions, converting an inline policy into Terraform. Paste an AccessDenied message and you get a plausible read of which layer refused. So the argument is not that the tooling is useless. It is that recall and reasoning about consequences are different capabilities, and IAM is a system where the second one protects you.

The failure modes that survive review

Permissions that compose into something bigger

Every action is individually defensible. The escalation lives in the combination.

The canonical case is iam:PassRole beside any service that runs code: Lambda, EC2, ECS, Glue. Passing a role to a compute service means your code executes with that role’s permissions, so your ceiling is not your own policy, it is whatever the most powerful passable role can do. Rhino Security Labs catalogued a long list of these chains and every one of them is built from permissions that look unremarkable alone.

Another that catches people: iam:CreatePolicyVersion on a policy you are attached to. A new version normally needs to be set as default to take effect, but the create call accepts a flag that makes it default immediately, and that flag does not require iam:SetDefaultPolicyVersion. A permission that reads as “can update policy documents” reads to an attacker as “can write myself an admin policy.”

Ask a model whether a policy grants admin and it will check the statements and say no, correctly, one at a time. Composition is the part it does not reliably do, and it is the part that matters.

Actions that do not exist

Plausible action names that no service publishes. s3:ListBucketContents instead of s3:ListBucket. IAM accepts a policy containing an unrecognized action string because the document is syntactically valid, so nothing fails at attach time.

What happens instead is worse. The application throws AccessDenied in production, somebody debugging under pressure decides the granular list must be wrong, and the fix is a service wildcard. The invented action does not grant too much. It causes a human to grant too much a week later, and by then nobody connects the two events.

Resource ARNs that look constrained and are not

An ARN with real characters in it feels safer than a star. Often it is not. The common versions are a trailing wildcard placed one segment too high, and an action whose resource element the service ignores. s3:ListAllMyBuckets is account-scoped no matter what you put in Resource. Several ec2:Describe actions behave the same way. A policy can read as resource-scoped throughout and be account-wide for the actions that count.

The related trap is the missing condition: cross-account trust policies without an external ID, bucket policies without aws:PrincipalOrgID, integration roles without aws:SourceIp where the caller has a stable egress address. Generated policies include conditions when you ask and omit them when you do not, because an absent constraint never produces an error.

Four things the model cannot know

Some of this is not a capability gap you can prompt past. It is missing input:

• Which resources exist. A bucket name is a guess unless you supplied it. Guessed names either fail closed or, if the wildcard is loose enough, match something you never intended.
• Which resources are sensitive. Your backup bucket and your public assets bucket are indistinguishable strings. Sensitivity lives nowhere in the document.
• What else is attached. Effective permissions come from identity policies, resource policies, boundaries, SCPs and session policies together. Reviewing one document tells you very little.
• What the blast radius is. “Can delete objects” means one thing for a scratch bucket and another for the only copy of a client’s data.

The training material is also skewed. Public IAM examples, including plenty of vendor documentation and countless forum answers, are permissive because permissive examples work on the first try. A model that has read the internet has read a lot of over-broad policies presented as correct.

Where the hard boundary belongs

Review is not a boundary. Review is a filter with a pass rate that drops the more policies you look at in a week, and generated policies are cheap to produce, so volume rises exactly when attention falls.

The boundary has to be something that says no without reading the document. AWS gives you two mechanisms and they are not interchangeable.

A permissions boundary is a managed policy attached to a user or role that caps what identity-based policies can grant it. It grants nothing itself. Effective permissions become the intersection of the identity policy and the boundary, so an identity policy allowing iam:* against a boundary that omits IAM produces no IAM access at all. If no boundary is attached, none is evaluated, so this is opt-in per identity.

A service control policy sets the maximum available permissions for every principal in an account or organizational unit, and also grants nothing. It is the right home for things nobody in that account should ever do, and it applies whether or not somebody remembered to attach a boundary.

The division I use: SCPs carry account-wide invariants, such as denying escalation-relevant IAM write actions to everyone outside the break-glass role and denying regions you do not operate in. Boundaries carry the per-workload ceiling, and they are what makes it safe to let a pipeline role create other roles. You can permit that role to create roles while requiring, through a condition on the boundary key, that every role it creates carries your boundary. The pipeline can then generate whatever policy it likes, because the ceiling is enforced at evaluation time rather than at review time.

If the only thing between a generated policy and production is a person reading JSON, you do not have a boundary. You have a habit.

The gate I would actually run

Mechanical checks in a pipeline, failing the build, not a checklist somebody is supposed to remember.

1. Lint for nonsense. Parliament catches unknown action names, resource formats that cannot match the action, and type mismatches. Cheapest way to catch an invented action before it becomes a wildcard.
2. Run AWS policy validation. Access Analyzer knows service-specific rules a generic linter does not.
3. Assert the escalation actions are absent. Name the actions you never want granted and fail if any appear. This is the check that maps to the composition failure.
4. Compare against the current policy. For edits, check whether the proposed document grants access the existing one does not. This one is billed per call, so run it on changed policies rather than the whole repository.
5. Scan for known risky patterns. Cloudsplaining scores policies against categories including privilege escalation, data exfiltration and resource exposure, and produces something you can hand to a client.

# 1. Lint. Exit status is non-zero when there are findings.
parliament --file policy.json

# 2. AWS-side validation
aws accessanalyzer validate-policy 
  --policy-document file://policy.json 
  --policy-type IDENTITY_POLICY

# 3. Fail if any escalation-relevant action is granted
aws accessanalyzer check-access-not-granted 
  --policy-document file://policy.json 
  --access actions="iam:PassRole","iam:CreatePolicyVersion","iam:AttachRolePolicy" 
  --policy-type IDENTITY_POLICY

# 4. Does the proposed policy grant more than the current one?
aws accessanalyzer check-no-new-access 
  --existing-policy-document file://current.json 
  --new-policy-document file://proposed.json 
  --policy-type IDENTITY_POLICY

# 5. Risk-categorised report
cloudsplaining scan-policy-file --input-file policy.json

Step three is the one to add first if you only add one. It turns “we reviewed it” into “the build fails.”

Attaching the boundary is a single call, and it is the part people skip because the role already works without it:

aws iam put-role-permissions-boundary 
  --role-name app-deploy-role 
  --permissions-boundary arn:aws:iam::123456789012:policy/WorkloadCeiling

Generate from observed behavior instead

There is a better source of truth than a model’s guess about what an application needs, and it is the application. Access Analyzer policy generation reads CloudTrail events for a role over a chosen window and writes a policy from what was actually called. Know its limits before relying on it:

• It analyzes up to ninety days of history and needs a trail already logging for the account, so a role that only exercises certain paths quarterly produces an incomplete policy.
• Coverage is per-service. For some services it identifies individual actions; for others it can only tell you the service was used and prompts you to fill in the actions yourself.
• It does not produce action-level detail for data events such as S3 object-level operations.
• iam:PassRole is not included in generated policies. If your workload needs it, a human adds it back by hand, so the most escalation-relevant line in the document is the one the generator did not write.

Behavior-derived policies also fail in a useful direction. A missing permission produces a denial you can see in CloudTrail, which beats a permission you did not know you granted.

Arguments that do not survive contact

• “We review every policy before merge.” Review quality is a function of volume and attention, and generation raises volume. Composition failures are also the hardest to spot by reading, because spotting them means knowing what other roles exist and what they can do.
• “Access Analyzer came back clean.” Validation checks the document. It does not know that the role ARN you may pass happens to have AdministratorAccess attached. Clean findings on a policy that grants a dangerous chain is the normal case, not an anomaly.
• “A permissions boundary is overkill for a small project.” Small projects are where this bites hardest. One account, one deploy role, no organization, nothing between a bad policy and everything you own. The boundary is one managed policy and one CLI call.
• “We will tighten it after launch.” Nobody does. If you genuinely intend to, drive it from Access Analyzer’s unused access findings so you have a work queue instead of an intention.
• “Just prompt the model to follow least privilege.” That changes the output surface, not the reasoning. Fewer wildcards, better statements, no evaluation of what the combination permits.

How I would decide

My rule is about which actions are in the document, not about who or what wrote it.

Ship generated policies freely when the document touches only data-plane actions on named resources, the identity has a boundary, and the pipeline checks pass. Read a bucket prefix, publish to a topic, write to a table. That is most policies, and hand-writing them wastes your afternoon.

Treat generated output as a first draft when the policy includes any IAM write action, any PassRole, any trust policy edit, any resource policy readable from outside the account, or any KMS key policy. Use it to save typing, then verify every statement against the service authorization reference yourself.

Do not generate at all for the boundary policies and SCPs. Those are the mechanism. Something that constrains everything else should be short, hand-written, understood by whoever owns the account, and changed rarely.

Two notes for small engagements. If you run application infrastructure on a VPS from a provider like Contabo or InterServer and pull AWS in for storage or mail, that server holds long-lived credentials somewhere your AWS controls do not reach, so scope its policy to a bucket prefix and attach a boundary before worrying about anything else. And if the credentials go to a SaaS connector, a marketing platform such as GoHighLevel or a payments integration, you are handing keys to a system whose behavior you cannot audit. Those are the policies to write by hand.

Frequently asked questions

Are AI-generated IAM policies safe to use in production?

For data-plane permissions on named resources, generally yes, provided the identity carries a permissions boundary and the policy passes automated validation. For anything involving IAM write actions, PassRole, trust policies or key policies, treat the output as a draft and verify each statement yourself. What matters is the blast radius of the actions, not the authorship.

Why does a generated policy pass validation and still grant too much?

Validation examines one document in isolation. Privilege escalation usually comes from a combination of individually reasonable permissions, or from the interaction between the policy and something outside it, such as which roles are passable and what those roles can do. A document can be internally correct and still open a path.

What is the difference between a permissions boundary and an SCP?

Both cap permissions and neither grants any. A boundary attaches to a single IAM user or role and limits what that identity’s policies can grant it. An SCP applies to every principal in an account or organizational unit and needs AWS Organizations. With one account and no organization, boundaries are what you have. With an organization, use both.

Can IAM Access Analyzer replace writing policies by hand?

Partly. Generation from CloudTrail activity beats a guess because it reflects what the workload actually called. It is limited by the analysis window, by per-service coverage differences, by the absence of action-level detail for data events, and by iam:PassRole being excluded. Treat it as a strong draft that needs a human pass over anything the generator could not see.

Which IAM actions should I block outright in a small account?

The ones that let a principal rewrite its own permissions or borrow another identity’s: creating and setting default policy versions, attaching and putting inline policies on users, roles and groups, creating access keys for other users, and passing roles to compute services. Deny them for everything except a break-glass identity, then grant them back deliberately where a workload genuinely needs one.

How do I catch an invented action name before production?

Run a linter that carries the IAM action catalogue. Parliament flags unknown actions and exits non-zero when it finds anything, so it drops straight into a pipeline step. Catching it early matters less because the fake action is dangerous and more because the eventual fix for the denial tends to be a service-level wildcard.

The one thing worth remembering

AI-generated IAM policies are not more dangerous because a model wrote them. They are more dangerous because they arrive faster than you can think about them, and because they arrive looking finished.

So stop trying to make the review step better. Put the boundary somewhere that does not depend on anyone paying attention: a permissions boundary on every identity a pipeline can create or modify, an SCP denying the escalation actions to everyone who does not need them, and a build step that fails when a forbidden action shows up. Then let generation happen at whatever speed it wants.

The policy approved in ninety seconds is not the failure. The absence of anything underneath it was.

Need help drawing that boundary in your account?

Most of the IAM work I take on is exactly this: an account that grew organically, permissions nobody wants to touch in case something breaks, and no ceiling underneath any of it. Things I can help with:

• Auditing existing roles and policies for privilege escalation chains, including the passable-role paths that never show up in a single-document review
• Designing and attaching permissions boundaries for pipeline, workload and human identities, including the delegation pattern that lets a deploy role create roles safely
• Writing the SCP set for a small organization: escalation-relevant IAM actions, region restrictions, and protection for logging and billing resources
• Adding Access Analyzer validation and custom policy checks to a CI pipeline so forbidden actions fail the build instead of relying on review
• Replacing over-broad roles with policies generated from CloudTrail activity, then closing the gaps the generator cannot see
• Scoping down credentials handed to third-party SaaS integrations and to servers outside AWS, where your account controls do not apply

If you want a second opinion on something specific, send me the policy JSON, the output of a failing check-access-not-granted run, or the role list from your account, and I will tell you what I would change and why.

The post AI-Generated IAM Policies: Where the Hard Boundary Belongs appeared first on John Nessime.

Nobody looks at the old link. If it was signed with a long expiry and mailed out, it is still live: in the client’s inbox, in the inbox of the bookkeeper on the CC line, and in whatever mail archive that firm runs. It keeps working for anyone holding the URL until the clock runs out, because a presigned S3 URL is a bearer token. AWS says exactly that in its own documentation. The URL does not know who you are. It knows that you have it.

That is the shape of most document vault failures at a small accounting firm. Not a wide-open bucket. Not a stolen partner password. An ordinary delivery mechanism that leaks quietly and writes no log line that looks wrong six months later.

This post covers how to build a secure client document vault for an accounting practice: what the regulation actually demands, the six failure families that decide whether it holds, and the control that closes each one. The examples are AWS because that is where most of these land, but the failure families are platform independent.

Let the regulation write your requirements

Most firms treat compliance as paperwork done after the build. That is backwards. For a US accounting or tax practice, the FTC Safeguards Rule at 16 CFR Part 314 is the closest thing you will get to a free requirements document.

It treats tax preparers, bookkeepers and accounting firms as financial institutions under the Gramm-Leach-Bliley Act. Size does not get you out of it. The parts that translate directly into architecture:

• Access controls limiting each user to the customer information they need, and limiting customers to their own information.
• Encryption in transit over external networks and at rest, with compensating controls only where encryption is infeasible and your Qualified Individual signs off in writing.
• Multi-factor authentication for any individual accessing any information system, subject to the same written-approval exception.
• Secure disposal, with an outer bound of two years after last use unless retention is required by law or necessary for legitimate business purposes.
• Logging of authorized user activity, plus a named Qualified Individual who owns the program.

One more clause should shape your design more than any other. The Rule requires notifying the FTC within thirty days of discovering a security event involving the unencrypted information of at least five hundred consumers, and it counts encrypted information as unencrypted if the key was accessed by an unauthorized person.

Read that twice. Encryption only helps at the breach-analysis stage if the key was genuinely out of reach. That is the entire argument for treating key custody as a control separate from storage.

Separately, IRS Publication 4557 expects paid preparers to keep a Written Information Security Plan, and PTIN renewal asks you to confirm you have one. Treat the WISP as the document describing the vault you built, not a substitute for building it.

Failure one: the link that outlives the engagement

Presigned URLs are the sensible answer to “let the client download this without an AWS account”. The problem is their expiry semantics, which almost nobody has read.

Two clocks apply and the shorter wins: the expiry you asked for, and the lifetime of the credential you signed with. Long-lived IAM user credentials under Signature Version 4 reach seven days. A role session from AssumeRole dies when the session ends, one hour by default, regardless of what you requested. An EC2 instance profile is bound by metadata credential rotation. The console caps at twelve hours.

So the two symptoms are opposites and both are common. Links dying early is a credential problem. Links living too long is a policy problem, and that is the one that hurts.

The fix is not a smaller number. It is changing what goes in the email.

1. Email a notification, not the file link. The client logs in with MFA and the portal mints a URL at click time.
2. Keep that expiry in minutes. The URL only has to survive a redirect and a download.
3. Scope every URL to one object and one verb.
4. If you truly need a long-lived share, use CloudFront signed URLs or signed cookies, where you can revoke by rotating the key group without touching the object.

# Sign a download URL valid for five minutes.
# --expires-in takes seconds; the default is 3600.
aws s3 presign s3://firm-client-vault/clients/acme-llc/workpapers.pdf 
    --expires-in 300

Run that under an assumed role, wait past the session expiry, and watch it fail well before five minutes. That is behaviour worth seeing once in a test rather than discovering during an incident review.

Failure two: everyone inside the firm sees everything

This shows up in nearly every small-firm build, and the reasoning is always the same. Nine people, everyone helps with everything during filing season, and compartmentalising access is friction exactly when friction is expensive.

The case for flat access is real, so say it out loud: a scoped model people fight gets worked around, and a workaround is worse than the flat model you started with. If your access control makes a senior accountant email a PDF to themselves at 11pm, you have made things less safe.

So scope by engagement rather than by person, and automate the grant so nobody files a request. Make the key prefix the boundary, then grant the prefix instead of the bucket. Two statements, two jobs: the first controls what someone can enumerate, the second what they can fetch.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOnlyThisClient",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::firm-client-vault",
      "Condition": {
        "StringLike": { "s3:prefix": "clients/acme-llc/*" }
      }
    },
    {
      "Sid": "ReadWriteThisClient",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::firm-client-vault/clients/acme-llc/*"
    }
  ]
}

Without the prefix condition on ListBucket, a scoped user still gets a full inventory of your client list. Client names are confidential in this business. A directory listing that reads like a book of business is a disclosure even when no file opens.

The other half of this family is standing administrative access. Somebody has to fix things, so somebody ends up with a permanent s3:* on the whole bucket. That principal is the highest-value target in the firm and usually has the weakest MFA, because it is a service account nobody wants to touch.

Replace it with an assumable break-glass role: MFA required, short session, and an alert on every assumption. The alert is the point. Emergency access used twice a year is fine. Used weekly, it means your normal path is broken.

Failure three: encrypted at rest, key sitting right next to it

Default bucket encryption gets ticked in week one and forgotten. It satisfies the letter of “encrypted at rest” and protects against exactly one threat: somebody walking out of a data centre with a disk. That is not your threat.

Go back to the notification clause. If an attacker reaches a principal that can both read the object and use the key, your data is unencrypted for reporting purposes. The question is not “was the bucket encrypted”. It is “could whoever got in also use the key”.

• Use a customer managed KMS key. Only that gives you a key policy you control.
• Name the principals allowed to call Decrypt, and keep that list shorter than the list allowed to call GetObject. Two different lists means an attacker needs two wins.
• Use encryption context so grants can be conditioned per client, and so CloudTrail entries are readable later.
• Enable S3 Bucket Keys to cut KMS request volume. KMS bills per API call, so a vault serving many small reads generates a surprising line item.

Encryption in transit needs its own control, because IAM policies say nothing about the connection. A bucket policy does:

{
  "Sid": "DenyPlaintextTransport",
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": [
    "arn:aws:s3:::firm-client-vault",
    "arn:aws:s3:::firm-client-vault/*"
  ],
  "Condition": {
    "Bool": { "aws:SecureTransport": "false" }
  }
}

An explicit deny beats every allow, including one you add by mistake later. Five lines that permanently remove a category of misconfiguration, which makes it the best value item on this page.

Failure four: retention rules that fight your disposal duty

Here is the trap, and it is why I would not let an accounting firm switch on Object Lock without a conversation first.

Object Lock has two retention modes. In governance mode, a principal holding s3:BypassGovernanceRetention can remove or shorten the lock. In compliance mode, nobody can. Not an administrator, not the account root user. AWS is explicit that the only way to delete a compliance-locked object before its retention date is to close the AWS account.

Set that against the Safeguards Rule’s two-year outer bound on disposal of information you no longer need. Apply a blanket seven-year compliance-mode retention to everything landing in the vault and you have written a promise you cannot revoke, on documents you may later be obliged to destroy.

1. Classify by document type before client. Filed returns, engagement letters and supporting workpapers have real statutory retention arguments. Raw intake uploads and scratch scans usually do not.
2. Default to governance mode. It stops accidental and casual deletion, which is what you need, while keeping a documented and audited escape hatch.
3. Give s3:BypassGovernanceRetention to almost nobody and alert on every use through CloudTrail.
4. Reserve compliance mode for records where an external mandate genuinely requires immutability and you chose the period deliberately.
5. Use legal hold, not a retention period, for litigation. A legal hold has no end date and stays on until explicitly removed, which is right when you cannot know how long a matter runs.

Two details catch people out. Object Lock requires versioning and normally has to be enabled at bucket creation, so it is a design decision rather than a retrofit. And lifecycle expiration will not delete a version still under retention, so lifecycle and Object Lock are two layers that must agree, not alternatives.

# What lock configuration is this bucket actually running?
aws s3api get-object-lock-configuration --bucket firm-client-vault

# Freeze one object version for a matter with no known end date.
aws s3api put-object-legal-hold 
    --bucket firm-client-vault 
    --key clients/acme-llc/2023/return.pdf 
    --legal-hold Status=ON

For deletion itself, crypto-shredding is worth knowing. If a client’s documents sit under a per-client KMS key, scheduling that key for deletion renders every object unreadable without touching the objects. Clean, fast, provable, and irreversible once the waiting period ends, so it belongs behind two-person approval rather than in a script.

Failure five: no evidence when somebody asks

Every control above is worth nothing if you cannot show it held. The Rule expects logging of authorized user activity and an annual written report from the Qualified Individual.

The trap is that CloudTrail does not record S3 object-level reads and writes by default. Management events cover things like changing a bucket policy. Data events cover the actual GetObject calls, and they are opt-in and billed per event. Firms enable CloudTrail, see a trail in the console, assume they have a record of file access, then find during an incident that they have nothing at object level.

Turn on data events for the vault bucket. At firm scale the volume is manageable, and it is the difference between “we believe two people opened that file” and being able to show it.

Then decide where logs live. Access records for client financial data are themselves sensitive, so keeping them in the same account under the same administrators defeats the purpose. A separate logging account, or at minimum a separate bucket with its own key and retention, is the shape you want. If you already run Grafana Cloud or similar for infrastructure monitoring, pointing vault access logs there gives you dashboards and alerting without standing anything new up.

The alerts worth having are boring and few: any use of the break-glass role, any change to the bucket or key policy, any bulk download outside normal working patterns, and any failed decrypt. Four alerts that fire almost never beat forty that everyone mutes.

Failure six: the two ends of the pipeline

A vault only protects documents that are in it. The two ends are where firms quietly undo everything.

Intake

If clients can still email a W-2 as an attachment, they will, because it is easier. Each attachment becomes customer information sitting in a mailbox, a sent-items folder and a mail archive, each with a different retention policy and access model than your vault.

A policy memo will not fix this. Make the vault path faster than the email path: an upload link that works on a phone, no account creation for a one-off drop, and a mail rule that returns certain attachment types with the upload link instead. Then sweep the mailboxes you already have, because that backlog is real customer information and it counts.

Offboarding

Seasonal staff are normal here, so offboarding happens in batches and gets rushed. Disabling the SSO login is the part everyone does. The parts that get missed:

• Active role sessions and refresh tokens, which survive the account being disabled until they expire on their own.
• Presigned URLs that person generated, which stay valid because they are bound to the signing credential, not the login session.
• Named principals still sitting in the KMS key policy.
• Local copies on laptops and external drives. When hardware is retired, a certified erase tool such as O&O SafeErase is the difference between disposal and hoping.

The same applies to service providers. The Rule makes you responsible for selecting providers capable of maintaining appropriate safeguards and holding them to it contractually. If an outsourced bookkeeping team touches your document workflow, that relationship is in scope, and direct network access means they need MFA too. Cloudflare Zero Trust or a business VPN such as NordLayer in front of the portal gives you somewhere to enforce that without rebuilding the application.

Troubleshooting a secure client document vault

• Links expire far earlier than the expiry you set. You signed with temporary credentials that expired first. Sign from a principal whose credential outlives the link, or shorten the link and mint on demand.
• Uploads fail with access denied despite an allow on PutObject. With a customer managed key, both the IAM policy and the key policy must permit the caller. Check kms:GenerateDataKey, not just S3 actions.
• A file will not delete and no policy seems to block it. Look for Object Lock retention or a legal hold on that version. They are independent and either one alone blocks the delete.
• CloudTrail shows the bucket being created but no file access. Data events are not enabled. Management events will never show GetObject.
• A scoped user can see every client folder name. The prefix condition is missing from ListBucket, or was applied to the object ARN instead of the bucket ARN.
• Lifecycle rules appear to do nothing. Versions under retention are not expired by lifecycle. Confirm lock state before assuming the rule is wrong.

Common mistakes, and what to do instead

• Emailing the download link. Email a notification that a document is waiting, and mint the link after the client authenticates.
• Treating default bucket encryption as finished. Use a customer managed key and keep the decrypt list shorter than the read list.
• Granting bucket-wide list permissions. Condition ListBucket on the client prefix, or the roster leaks even when no document opens.
• Applying compliance-mode Object Lock broadly because it sounds safer. Write the retention schedule per document class first, get partners to sign it, then configure to match.
• Keeping a standing full-access principal. Replace it with an MFA-gated break-glass role that alerts on every assumption.
• Building a portal and leaving email intake open. Both channels then carry client data and only one is governed. Close the mail path and sweep the backlog.
• Storing access logs beside the data they describe. Ship S3 data events somewhere with different administrators and different keys.
• Never testing any of it. Verify a restore and a scoped-access denial quarterly. An unverified control is an assumption.

Frequently asked questions

Does a small accounting firm really fall under the FTC Safeguards Rule?

Almost certainly, if it prepares returns or handles client financial data in the United States. The Rule’s definition of a financial institution covers activities financial in nature, and the FTC’s own examples include tax preparation firms. A narrow exception at 16 CFR 314.6 covers institutions holding information on fewer than five thousand consumers, but it exempts only four specific provisions. Encryption, MFA, access controls and disposal still apply. Confirm your own position with counsel, not a blog post.

Should I build this or buy a client portal product?

Buy, if the product genuinely does what you need. A firm with no engineering capacity is safer on a maintained product than a bespoke stack nobody patches. Build when you need controls the product will not give you: per-client key isolation, your own retention logic, or integration with a pipeline you already run. The honest test is whether someone will still own the system in a year.

Can I self-host a secure client document vault instead of using AWS?

Yes. Nextcloud or similar on a well-run VPS from a provider such as InterServer or Contabo can meet the same requirements, and it is often simpler to reason about. What you take on is patching, backups, TLS renewal and log retention. The regulation does not care where the data sits. It cares that the controls exist and that you can show they held.

How long should client documents be retained?

There is no single number, which is why a schedule beats a default. Tax and professional standards drive minimums for filed returns and engagement documentation, and those vary by jurisdiction and engagement type. The Safeguards Rule pushes the other way. Write both directions down per document class, then configure to match.

Is a presigned URL safe to send to a client?

It is safe to use, not safe to send. Treat it as a bearer token with a short life, minted after the client authenticates and consumed immediately. What travels over email should be a notification, not the credential.

Does encrypting the bucket mean I avoid breach notification?

Only if the key stayed out of the attacker’s hands. The Rule treats encrypted customer information as unencrypted where the key was accessed by an unauthorized person. That is the whole argument for keeping key permissions narrower than storage permissions.

What is the minimum viable version for a two-person practice?

One bucket with per-client prefixes, a customer managed key, an explicit deny on non-TLS transport, MFA on every human account, short mint-on-click links, S3 data events enabled, and a written retention schedule. That is a weekend of work and it closes five of the six failure families above.

The one thing worth remembering

A secure client document vault is not defined by the storage you picked. It is defined by how narrow the path is between an attacker and a readable file, and by your ability to prove afterwards how many people walked it.

Most firms get the storage right and lose at the edges: a link that outlives the engagement, a key policy as wide as the bucket policy, a retention setting that cannot be undone, an audit trail nobody turned on. Fix those four and the rest is ordinary engineering.

Where I can help

I work with small firms and the MSPs that support them on exactly this kind of build. Things I take on regularly:

• Designing and building the vault itself: bucket layout, per-client prefix isolation, KMS key policy, and the portal glue that mints short-lived links.
• Reviewing an existing setup against the Safeguards Rule’s technical elements and producing a gap list your Qualified Individual can work from.
• Untangling retention: turning a vague policy into a per-document-class schedule, implemented with lifecycle rules, Object Lock and legal holds that do not conflict.
• Standing up the evidence layer: S3 data events, a separate logging account, dashboards, and the small set of alerts worth waking up for.
• Closing the email intake path without making life harder for clients, including mobile upload flows and a sweep of the historic mailbox backlog.
• Offboarding and access reviews: break-glass roles, session revocation, key policy hygiene, and a repeatable quarterly check.

If you want a second opinion, send the thing itself: a bucket policy, an IAM policy, a CloudTrail event selector, or a description of how documents reach your clients today. It is much easier to say something useful about a real configuration than about a general question.

The post Building a Secure Client Document Vault Your Accounting Firm Can Defend appeared first on John Nessime.

That question is what actually bites when you run infrastructure for twenty clients. Not a breach. An audit, or a resignation. Someone gives notice on a Friday and you spend Monday working out which accounts they still hold keys to, because the credential everybody shared was never attached to a person. You can’t revoke a person from a password. You can only rotate it and hope you got all twenty.

This post covers how to design client AWS account access so that humans authenticate once against something you control, every client account holds a role that trusts exactly one thing, and every API call in a client’s CloudTrail can be traced back to a named engineer. It’s organised by failure family, because the mechanics are well documented and the failure modes are not.

The shape: one identity plane, twenty separate trust decisions

The instinct that causes all the trouble is putting credentials in the middle. Client gives you a login, you store it somewhere shared, engineers take it out when they need it. Everything downstream of that decision is damage control.

The shape that works is boring. One AWS account whose only job is identity. Your engineers exist there, once, in IAM Identity Center or federated from whatever directory you already pay for. Each client account holds an IAM role whose trust policy names one principal in that account and requires an external ID. Nobody stores a client credential, because there is no client credential. There’s a role your engineers can ask STS for a short-lived session against, and a client who can delete that role whenever they like.

There is a second design worth knowing about: inviting client accounts into your own AWS Organization. It’s genuinely better in some ways. You get service control policies, resource control policies, and centralised root access management across the estate, all of which are organisation-scoped features you simply cannot use on an account you don’t own. It’s also a much bigger commitment. You take on consolidated billing, the client’s spend flows through you, and pulling an account back out of an organisation at the end of a contract is a real project rather than a policy change.

For most consultancies and managed service providers, the client keeps their own organisation and you get a role. That’s the model the rest of this assumes.

Failure family 1: credentials you can’t revoke

The shared IAM user with an access key is the default because it works immediately. It keeps working right up until one of three moments: an engineer leaves, a client asks who did something, or a laptop gets stolen.

Access keys have no expiry. They’re bearer tokens with an indefinite lifetime, and the only way to invalidate one is to go into each account and delete it. At twenty accounts that’s twenty deliberate actions performed under time pressure, and the one you forget is the one that matters. A role session, by contrast, expires on its own. That’s the entire argument.

A shared password manager still earns its place, just not for admin keys. Bitwarden or 1Password is the right home for per-client external IDs, account numbers, role ARNs and the client-side contacts you need when something breaks at two in the morning. Metadata, not credentials.

Failure family 2: a trust policy that trusts more than you meant

This is the invisible one. The role gets created, the access works, everyone moves on, and the trust policy quietly grants far more than anyone intended.

Two mistakes travel together. The first is naming your account root in the principal. Writing arn:aws:iam::111122223333:root does not mean “the root user of my account.” It means any principal in that account that has permission to call AssumeRole. Every engineer, every Lambda function, every CI runner, every future role you haven’t created yet. It delegates the decision to your own account’s IAM, which is fine when your IAM is tidy and quietly catastrophic when it isn’t.

The second is skipping the external ID. AWS calls the problem it solves the confused deputy: you are a third party that holds the ability to assume roles in many customer accounts, and without a shared secret in the condition, one customer who learns another customer’s role ARN could persuade your systems to act against it. The external ID is the value that makes each trust relationship specific rather than generic.

Generate it yourself, one per client, as a UUID or similar unguessable value. The client must not be the one choosing it, because if they can choose it they can choose someone else’s, and the control does nothing.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/MspOperationsAccess"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
        }
      }
    }
  ]
}

The principal names one role, not the account. If that role is compromised you have one thing to fix. If the account is named, you have an inventory problem you didn’t know you had.

Failure family 3: CloudTrail says the role, not the human

You’ve done everything above and a client emails asking who deleted the staging load balancer. You open their CloudTrail and every event attributes to the same assumed role. The role is shared by your whole team by design, so the log tells you your company did it. That’s not what they asked.

There are two knobs and they are not equivalent. The role session name is chosen freely by whoever calls AssumeRole, appears in the assumed-role ARN, and is trivially set to anything. It’s useful, but it’s a label, not evidence. Source identity is the one that carries weight: once set on a session it cannot be changed, it persists across chained role sessions, and it lands in CloudTrail in the session context of every subsequent call.

The catch is that the role’s trust policy has to permit sts:SetSourceIdentity alongside sts:AssumeRole, and you can require it with a condition so a session without one is refused outright.

"Action": [
  "sts:AssumeRole",
  "sts:SetSourceIdentity"
],
"Condition": {
  "StringEquals": {
    "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
  },
  "StringLike": {
    "sts:SourceIdentity": "*@yourcompany.com"
  }
}

If your engineers arrive through SAML federation, the identity provider can populate source identity from a directory attribute such as email, which means nobody has to remember to pass it. For direct CLI use it’s an explicit flag:

aws sts assume-role 
  --role-arn arn:aws:iam::444455556666:role/MspEngineerAccess 
  --role-session-name acme-investigation 
  --external-id 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02 
  --source-identity jane.doe@yourcompany.com

Getting this right is what turns “we take security seriously” into a sentence you can back with a log query. It is also, in my experience of what clients actually ask for, the single most persuasive thing on the list.

Failure family 4: one identity, twenty blast radii

A hub account is a concentration of risk. That’s the trade you’re making, and it’s worth making, but only if the hub is held to a higher standard than the accounts it reaches into.

Practically: not every engineer gets permission to assume every client’s role. Group permissions in the hub so the people on the Acme engagement reach Acme and nothing else. Use per-client role names rather than one shared name across all twenty, so a policy in the hub can’t wildcard its way into a client you stopped working with a year ago.

Tier the roles by what they’re for, not by who’s asking. A read-only role for investigation covers most of the work most days. A change role carries the write permissions the engagement actually needs. Break-glass sits separate, with a short maximum session duration and an alarm on every use.

Two extras are worth knowing about even if you don’t reach for them immediately. Session policies let the caller narrow a session further at assume time, which is useful for automation that only needs one bucket. Permissions boundaries cap what any role or user created inside the client account can do, which matters when part of your job is creating IAM there.

If your team already routes through a business VPN or zero trust gateway with stable egress addresses, a source IP condition on the trust policy is a cheap extra layer. NordLayer, Tailscale and Cloudflare’s zero trust products all give you predictable egress. Be honest about the cost, though: the moment something runs from CI, a client’s network or a laptop on hotel wifi, an IP condition becomes a support ticket. I’d add it once everything else works, not before.

Failure family 5: standing administrator access nobody remembers granting

Somewhere in the twenty accounts there is a role with AdministratorAccess attached, created during a migration, described in the ticket as temporary. It’s been eighteen months.

Here’s the honest part: AWS does not give you a native just-in-time approval workflow for cross-account role assumption. There’s no built-in “request four hours of write access, get approved in Slack, access expires automatically” for this pattern. You either build that on top of STS and your ticketing system, or you accept standing access and compensate with tight session durations, aggressive monitoring and a genuine periodic review. Both are defensible. Pretending you have JIT when you have a wiki page is not.

The cheap version that actually helps: set the maximum session duration on client roles to something short, alert on assumption of the break-glass role, and run a quarterly pass over service last accessed data to strip permissions nobody has used. IAM can generate that report per role.

# Kick off the report, then fetch it with the returned job ID
aws iam generate-service-last-accessed-details 
  --arn arn:aws:iam::444455556666:role/MspEngineerAccess

aws iam get-service-last-accessed-details --job-id JOB_ID_FROM_ABOVE

Run that in each client account and the conversation about narrowing permissions stops being theoretical. You’re not guessing what the engagement needs; you’re reading it.

Failure family 6: offboarding that only half happens

The whole point of the hub is that offboarding is one action. Disable the person in your directory and every path into every client account closes at once, because there was never a credential in a client account with their name on it.

That’s true only if nothing crept in sideways. Things that creep in sideways:

• IAM users created directly in a client account during an incident, because it was faster
• Access keys minted under a role by automation and never inventoried
• A client’s own console user shared with your team “just for the migration”
• Third party tools connected with their own cross-account roles that outlived the trial
• A departing engineer’s laptop with cached SSO tokens and stale profiles still on disk

IAM Access Analyzer is the tool that finds most of the first four. Its external access findings show every principal outside the account that can reach into it, which is exactly the list a client wants to see, and it’s also the list that reveals the monitoring vendor nobody remembers signing up for. For the laptop, whatever your hardware disposal process is, make sure it involves an actual wipe rather than a factory reset; tools like O&O SafeErase exist for that reason and are cheap next to the alternative.

Rolling client AWS account access out across twenty accounts

Doing this by hand twenty times guarantees twenty subtly different configurations. Make it a template and a checklist.

1. Write one CloudFormation template that creates the role, its trust policy and its permissions, parameterised by external ID and your operations account ID.
2. Generate a unique external ID per client and store it with the client’s account metadata, not in the template.
3. Send the client a launch stack link rather than asking for credentials. They deploy it in their own account, in their own console, and can read exactly what they’re granting.
4. Record the account ID, role ARN, external ID location, engagement scope and review date in a single inventory file in version control.
5. Generate everyone’s local AWS profiles from that inventory so nobody hand-edits a config file.
6. Verify each one with a call that proves identity and nothing else before you touch anything real.

The generated profile chains from your federated session into the client role. The AWS CLI reads these from the shared config file and handles the assume-role call for you:

# ~/.aws/config
[profile msp-hub]
sso_session = company
sso_account_id = 111122223333
sso_role_name = MspOperationsAccess
region = eu-west-1

[profile acme-readonly]
role_arn = arn:aws:iam::444455556666:role/MspEngineerReadOnly
source_profile = msp-hub
external_id = 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02
role_session_name = jane-doe
duration_seconds = 3600
region = eu-west-1

Then the two-second sanity check that has saved me from more mistakes than any other command:

aws sts get-caller-identity --profile acme-readonly

It returns the account ID and the assumed-role ARN. Read the account number before you run anything destructive. Twenty profiles in one config file is exactly the situation where muscle memory picks the wrong one.

For scheduled work that runs without a human, the same trust relationship serves a Lambda function or an EC2 instance role in your hub. A low-cost VPS from Contabo or InterServer will also do the job, but note the trade: an instance profile inside AWS gets rotated credentials automatically, while a box outside AWS needs a long-lived credential to bootstrap from, which reintroduces exactly what this design removes. Inside AWS is the cleaner answer.

Troubleshooting the failures you will actually hit

AccessDenied on AssumeRole

Three separate causes produce nearly the same message, and they need different fixes. Either the client’s trust policy doesn’t name your principal, or the external ID you sent doesn’t match what the condition expects, or your own identity in the hub lacks permission to call AssumeRole on that role ARN. Check them in that order, because the first two require the client and the third doesn’t.

Switch Role in the console silently fails

This one wastes an afternoon if you don’t know it. The console’s switch role feature does not support roles that require an external ID. It’s a documented limitation, not a bug in your policy. Engineers who need console access to those accounts should reach them through federation rather than the switch role box.

Sessions expiring after an hour when you asked for longer

When one role session assumes another role, that’s role chaining, and a chained session is capped at one hour regardless of the role’s maximum session duration setting. Since your engineers assume the client role from an already-assumed session in the hub, you’re chaining by definition. Plan around the hour rather than fighting it.

An authorization message you can’t read

Some denials come back with a long encoded blob attached. STS will decode it if your identity has permission to, and the decoded version usually names the exact statement that denied you.

aws sts decode-authorization-message --encoded-message <encoded-blob>

Common mistakes

• Naming the account root as the trust principal instead of the specific operations role
• Letting the client pick the external ID, or reusing one value across several clients
• Treating the role session name as an audit trail when it’s a caller-supplied string
• Asking a client for console credentials because a template felt like too much effort
• Granting AdministratorAccess for a migration and never revisiting it
• Building the hub properly and then leaving one legacy IAM user in one account as a fallback
• Assuming that removing someone from your directory also removed the keys they created by hand

Best practices worth the effort

• One identity plane, federated, with MFA enforced at the point the human authenticates
• A unique unguessable external ID per client, generated by you and stored with client metadata
• Source identity required in the trust policy so CloudTrail names a person, not a role
• Roles tiered by purpose: read-only for investigation, change for the engagement, break-glass alarmed
• Client-owned policies, deployed by the client from your template, revocable by the client in one action
• A version-controlled inventory of every account, role, scope and review date
• Access Analyzer external access findings reviewed on a schedule, in every client account

FAQ

Can’t I just ask each client to create an IAM user for me?

You can, and it’ll work on day one. The problem is day four hundred. That user’s access key never expires, it isn’t attached to any individual on your team, and revoking it means someone in the client’s organisation remembering to act. A role costs the client the same five minutes to create and hands them a control they can exercise unilaterally.

Do I need a different external ID for every client?

Yes. A shared external ID across your client base defeats the purpose, because the value’s entire job is to be specific to one trust relationship. Generate a UUID per client, treat it as a secret, and never let the client choose it.

Should I bring client accounts into my own AWS Organization?

Only if you’re also taking on their billing and the commercial relationship supports it. Inside your organisation you get service control policies, resource control policies and centralised root access management, which are real advantages. Outside it, those stay the client’s tools and your job is to recommend them well. Most consulting engagements sit outside.

How do I prove a departed engineer no longer has access?

Show the directory record disabling them, then show that no IAM user or access key in the client account carries their name, then show Access Analyzer external access findings listing only your operations role as an external principal. Three artefacts, all exportable. That’s the evidence pack.

Does MFA carry across the cross-account role assumption?

Don’t assume it does. The reliable place to enforce MFA is where the human authenticates in your identity plane, and the reliable control at the client boundary is the combination of a narrowly scoped principal and the external ID condition. If you want an MFA condition in the client’s trust policy as well, test it end to end against your actual federation path before you rely on it, because the request context differs between direct assumption and chained sessions.

What’s the minimum viable version if I only have three clients?

A hub account with federated engineers, one role per client scoped to a named principal with a unique external ID, and source identity required. Skip the tiering, skip the IP conditions, skip the JIT ambitions. Those three things give you revocation, attribution and a clean answer for the auditor, which is most of the value.

The one thing to take away

Client AWS account access is not really a permissions problem. It’s a revocation and attribution problem wearing a permissions costume. Any design where a credential exists that isn’t tied to a specific human will fail you at exactly the wrong moment, and no amount of policy tuning fixes that, because the gap isn’t in the policy.

Put the humans in one place you control. Put a narrowly scoped, externally identified role in each client account. Require source identity so the log names a person. Everything else on this page is refinement; those three are the structure. And the pleasant side effect is that “here’s how you revoke us, unilaterally, in one action” is a genuinely good thing to be able to say in a sales conversation.

Need a second pair of eyes on your client access model?

This is the kind of work I do for consultancies and managed service providers who’ve grown past the point where a shared vault is survivable. Specifically:

• Reviewing existing cross-account trust policies for over-broad principals and missing external IDs
• Designing a hub account and federation model that scales past a handful of clients without per-client special cases
• Building the CloudFormation or Terraform template clients deploy themselves, so onboarding stops involving credentials
• Turning on source identity end to end, including the identity provider side, so CloudTrail attributes to named engineers
• Producing the offboarding evidence pack clients and auditors ask for, and automating it so it isn’t a scramble
• Narrowing long-standing administrator roles down using service last accessed data, without breaking the engagement

If you want a concrete starting point, send me one trust policy and the AccessDenied message it produces. That’s usually enough to tell where the model is leaking.

The post Twenty Client AWS Accounts and Not One Shared Admin Credential appeared first on John Nessime.

The expensive stuff is not the running instances. It’s the access keys nobody can attribute, the CloudTrail trail that quietly stopped delivering months ago, the Terraform state that no longer matches reality, and the region somebody spun a proof of concept up in two years ago. None of that comes up on a kickoff call. All of it comes up in week three, after you’ve quoted.

So I run an AWS account health check before I give anyone a number. It’s read-only, it takes a couple of hours, and it’s the cheapest insurance in consulting. This post is the actual check: the six passes, the commands, what each finding does to the estimate, and what to do when the client won’t give you the access to run it.

Why the AWS account health check comes before the quote

A quote is a bet on how much unknown there is. You’re pricing the work in the brief plus everything you’ll find while doing it, and the second half is invisible until you look. Two accounts can look identical and differ by a factor of three in effort. The difference is nearly always archaeology: how much has to be reverse engineered before you can safely change anything.

There’s a second reason that matters more than pricing. If you touch an account that already had a compromised key sitting in it, the incident becomes yours by proximity. Establishing what the account looked like before you arrived isn’t paranoia. It’s the only version of events you’ll be able to evidence later.

The access I ask for, and what a refusal tells me

I ask for a cross-account IAM role that the client creates and my account assumes. Not an IAM user. Not keys in an email. A role, with an external ID the client generates.

The external ID solves one specific problem. Without it, a trust policy that names my account can be used by anyone who talks me into assuming a role on their behalf, and I end up acting inside a third party’s account without knowing it. AWS calls this the confused deputy problem. It’s the piece of cross-account setup people skip most often.

On that role I ask for two AWS managed policies: SecurityAudit and ViewOnlyAccess. The pairing is the point. ViewOnlyAccess lets you enumerate resources across services, so you can see what exists. SecurityAudit lets you read configuration and policy documents, so you can see how it’s wired together. Neither alone gives you the picture, and neither lets you change anything. Billing is separate again: Cost Explorer read access has to be added explicitly and the feature has to be enabled in the account.

The reply to that request tells you as much as the check does.

• “Here’s the role ARN and external ID” means somebody there understands AWS identity. Good sign.
• “I’ll just make you an admin user, easier” means nobody is managing access. Expect that looseness everywhere else.
• “Our previous developer set it up and we can’t reach him” means you’re inheriting credentials nobody can account for. Price the rotation work as its own line.
• “We’d rather not grant access before signing” is fair. Offer a paid discovery engagement. If they won’t pay for two hours of assessment, they won’t pay for the fix.

The six passes

Each pass answers one question. Everything here is read-only, and the order matters, because each pass changes how you read the next.

Pass 1: Who can do what

Confirm where you actually are first. That sounds trivial until you’re in an organization with a dozen similarly named accounts.

aws sts get-caller-identity
aws iam list-account-aliases
aws organizations describe-organization

If the third command fails because the account isn’t in an organization, that’s a finding by itself. Standalone means no service control policies, no organization trail, and no central place to revoke anything.

Then the credential surface. The summary gives counts; the credential report gives per-principal detail, including when each password and key was last used.

aws iam get-account-summary

aws iam generate-credential-report
aws iam get-credential-report --query Content --output text | base64 --decode > credential-report.csv

The report generates asynchronously, so the second call may need a retry the first time. What I read from it: how many IAM users exist at all, how many have keys that have never been used, and how many have keys last used a long time ago that are still active.

An unused key is worse than a busy one. A key in daily use is attributable to something. A key last used eighteen months ago and still enabled belongs to a person or script nobody remembers, which means nobody will notice when someone else starts using it.

Then check whether anyone has ever asked this question before you:

aws accessanalyzer list-analyzers

IAM Access Analyzer offers external access analyzers, which flag resources shared outside a zone of trust you define, and unused access analyzers, which surface unused roles, keys and passwords. Unused access analysis is billed per IAM role and user analyzed per month, so it isn’t free, but it answers the least privilege question far faster than reading policy documents by hand. An empty list means nobody has ever asked this account who can reach what.

Root closes the pass. In an AWS Organizations setup, centralized root access management lets the management account or a delegated IAM administrator remove root credentials from member accounts entirely, so there’s no root password, key or MFA device to manage per account. Live root credentials on every member account is a real finding, and one of the few security fixes that’s genuinely quick.

Pass test: you can name every principal that can reach production, and say when each last did.

Pass 2: Would anyone notice

This is the pass that most often changes my number, because it’s the failure that stays invisible until it isn’t. Logging that exists but has silently stopped working is more dangerous than no logging, since everybody believes they’re covered.

aws cloudtrail describe-trails 
  --query 'trailList[].{Name:Name,MultiRegion:IsMultiRegionTrail,Org:IsOrganizationTrail}'

aws cloudtrail get-trail-status --name TRAIL_NAME_HERE

aws configservice describe-configuration-recorder-status

The first command tells you a trail is configured. The second tells you whether it’s delivering, which is the part that matters. A trail whose last delivery failed because someone edited the destination bucket policy still shows up in the console as a configured trail. The status call is where the truth is.

Multi-region matters for the same reason region sprawl does: a single-region trail records nothing about the region where somebody is quietly running compute. Check GuardDuty too, and how long logs are actually retained. Ninety days of retention against a one-year obligation is a project, not a task, and it belongs in the quote rather than in week four.

One naming trap worth knowing: AWS renamed the original Security Hub to Security Hub CSPM and launched a separate, newer service under the Security Hub name. When a client says “we have Security Hub,” ask which one, because the findings format and the configuration are not the same thing.

Pass test: a trail is delivering right now, covers every region, and its logs are kept for at least as long as the business needs them.

Pass 3: How wide the blast radius is

People forget regions. Not occasionally, constantly. A demo in Singapore, a test bucket in Ireland, a Lambda deployed from a tutorial that defaulted to us-east-1.

for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  count=$(aws ec2 describe-instances --region "$r" 
    --query 'length(Reservations[].Instances[])' --output text)
  echo "$r $count"
done

Left without --all-regions, describe-regions returns the regions enabled for the account, which is exactly what you want. Adding the flag pulls in disabled regions too, and the per-region calls against those just error out.

Run the same loop shape for anything carrying state or cost: RDS instances, load balancers, EBS volumes, NAT gateways. NAT gateways are the classic. They bill hourly whether or not anything routes through them, and a forgotten one in a region nobody visits is pure waste. Then the two exposure checks I never skip:

aws ec2 describe-security-groups 
  --filters Name=ip-permission.cidr,Values=0.0.0.0/0 
  --query 'SecurityGroups[].{Id:GroupId,Name:GroupName,VpcId:VpcId}'

aws s3api list-buckets --query 'Buckets[].Name' --output text

For each bucket that matters, check its public access block configuration and its policy. An open security group on 443 in front of a load balancer is fine. An open security group on a database port is a different conversation, and you want that conversation before you own the account.

Pass test: you have a list of every region with resources in it, and nothing on that list surprises the client.

Pass 4: Whether any of this is described in code

The question isn’t “do they use Terraform.” It’s whether the code still describes the account. Those diverge quietly. Somebody makes an urgent console change during an incident, never brings it back into code, and from then on the repo is fiction that plans cleanly right up until the day it destroys something.

terraform plan -refresh-only

terraform plan -detailed-exitcode

The refresh-only plan shows where real infrastructure has drifted from state without proposing to change anything. The detailed exit code version is the one for scripts: it exits 0 for no changes, 1 for an error and 2 when there’s a diff, so you can gate on drift instead of reading output by eye.

Three things I want to know here:

• What fraction of live resources are managed by code? Half a codebase is often worse than none, because it creates false confidence about which half.
• Where does state live and who can write to it? Remote state in a versioned, locked S3 bucket is a different risk profile from a state file on a laptop.
• Does the pipeline that applies changes still run? Credentials that expired six months ago mean everything since went in by hand.

Tagging belongs here too. With nothing tagged you can’t attribute cost, can’t scope a change safely, and can’t tell a production resource from an abandoned experiment. Untagged environments add hours to everything that follows.

Pass test: a refresh-only plan comes back clean, or the drift is a short list you can read out loud.

Pass 5: What shape the spend is in

I don’t care much about the total. I care about the shape: which services dominate, whether the curve is flat or climbing, and whether anything jumped without an explanation.

aws ce get-cost-and-usage 
  --time-period Start=$(date -d '6 months ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=SERVICE

The date -d syntax is GNU coreutils; on macOS use the BSD equivalent or a fixed start date. Worth knowing before you loop this: the Cost Explorer API bills per paginated request, unlike the console view, so a wide query across several dimensions is a small but real charge on the client’s bill. Keep it narrow and mention it in your notes.

• Data transfer in the top three usually means an architecture problem rather than a rate problem. Cross-AZ chatter and NAT gateway egress are the usual suspects.
• A step change with no matching launch deserves a direct question. Sometimes it’s a feature. Sometimes it’s a runaway job. Occasionally it’s mining on a stolen key.
• No budgets and no anomaly monitors means nobody finds out about a spike until the invoice arrives.
• Heavy commitment purchases against workloads you’re about to re-architect are a constraint on your design, and need to be on the table before you propose anything.

If the client wants ongoing cost visibility past what Cost Explorer offers, platforms like Vantage or CloudZero sit on the Cost and Usage Report and give per-team allocation and unit economics that native tooling makes awkward. Raise it as a separate line, not part of the engineering work.

Pass test: you can explain the top three cost lines in a sentence each, and account for every month-on-month jump.

Pass 6: What happens at two in the morning

Short pass, about escalation. What support plan is the account on, and would anything wake a human up?

Support plan matters more than people expect. On Basic you get the service limits checks and a small subset of Trusted Advisor security and fault tolerance checks, manual refresh only, and no technical support cases at all. The full check library and the Trusted Advisor API need a paid plan. If your engagement depends on opening a case during a cutover, that’s a prerequisite, not a nice to have.

There’s one date worth carrying in your head, because it’s a real deprecation rather than a version number. AWS has announced that Developer Support, Business Support and Enterprise On-Ramp are discontinued on 1 January 2027, with Business Support+ as the replacement tier. If you’re scoping anything that runs past that boundary, check which plan the client is on and whether their support cost is about to move.

Then monitoring. CloudWatch alarms wired to an SNS topic with no confirmed subscribers are the same failure as the dead trail: configured, visible, useless. If they run anything beyond alarms, whether that’s Grafana Cloud, Datadog or a self-hosted Prometheus stack, find out who holds the credentials and whether the routing has been tested this year.

Pass test: you can name the human who gets woken up, and the path that reaches them.

The findings that actually move the number

• Unattributable credentials. Each one has to be rotated or removed, and each needs a conversation with whoever might break. That’s coordination time, which is the hardest kind to compress.
• No working audit trail. No baseline means you can’t prove what you did or didn’t change. Add the cost of standing logging up before the real work starts.
• Significant drift between code and reality. Reconciliation is slow, fiddly and carries destruction risk. The single largest multiplier I see.
• No tags. A tax on every subsequent task rather than a task of its own, since every scoping decision becomes an investigation.
• Resources in unmentioned regions. Each is a small parallel environment with its own security groups, its own logging gap and its own bill.
• One shared admin identity. Nothing in CloudTrail is attributable to a person, so your first task is identity work the client didn’t ask for.

I don’t present these as criticism. I present them as line items with an hour estimate against each, and let the client decide what’s in scope. That conversation is much easier before a contract exists than after.

If you only run five checks

Sometimes you get twenty minutes and a screen share.

1. Pull the credential report and count active keys that haven’t been used recently.
2. Confirm a CloudTrail trail exists, is multi-region, and is currently delivering.
3. Loop the enabled regions for running instances and NAT gateways.
4. Run a refresh-only plan against whatever infrastructure code exists.
5. Pull six months of cost grouped by service and look at the shape.

Those cover identity, evidence, blast radius, delivery speed and money. The rest of the check refines a picture these five already gave you.

Where this check goes wrong

• Treating it as an audit. You’re scoping work, not issuing a compliance report. A twenty page findings document makes a prospect defensive and costs you a day.
• Checking one region. The default region in your profile is not where the problems are.
• Trusting the console over the API. Configured and working are different states, and the console is better at showing the first.
• Taking long-lived keys because it’s faster. Accept an admin access key by email to save fifteen minutes and you’ve become part of the problem you were hired to assess.
• Running the loops unannounced. A burst of API calls from an unfamiliar principal is exactly what GuardDuty exists to flag. Tell them you’re doing it and roughly when.

Frequently asked questions

How long should an AWS account health check take?

Two to three hours for a single account once you have access, including the write-up. Multi-account organizations take longer but not linearly, because the interesting variation concentrates in one or two accounts and the rest are copies of a landing zone pattern.

Should I charge for the assessment?

For a short engagement, folding it into the proposal is fine. For anything substantial, or where the client is shopping several consultants, quote it as a small standalone piece with a written deliverable. It qualifies the client, protects your time, and the findings have value to them even if they hire someone else.

What if the client won’t give me read-only access before signing?

Quote time and materials, or quote fixed price with an explicit assumptions list naming every unknown. Do not quote a fixed price on an account you haven’t seen. The pattern that works is a small paid discovery phase, then a fixed price set once you know what you’re dealing with.

Is ReadOnlyAccess enough on its own?

It’s broader than you probably want. ReadOnlyAccess extends to reading data in some services, not just configuration, which is more than you need for scoping and more than many clients will be comfortable granting. The SecurityAudit and ViewOnlyAccess pairing is narrower and easier to justify in writing.

Can I automate the whole thing?

The enumeration, yes, and it’s worth scripting since you’ll run it repeatedly. The interpretation, no. Whether five unused IAM users are a problem depends entirely on what the client is trying to do, and that judgement is most of what you’re selling.

What if I find something actively compromised?

Stop and tell them immediately, in writing, with timestamps. Don’t remediate unilaterally: you’re read-only, and containment steps can destroy evidence. Your job is a clear description of what you saw and when, so they can involve whoever they need to.

Does this apply to small accounts too?

Especially to small accounts. Larger organizations usually have some governance, even imperfect. A three year old account at a small business that four different contractors have touched is where the surprises live, and it’s exactly the account that gets described as simple.

The one thing worth remembering

An AWS account health check isn’t a security exercise dressed up as sales. It’s the only reliable way to price uncertainty, and uncertainty is what you’re actually quoting on.

If you take one habit from this, take the second pass. Everything else on the list is something you’d eventually trip over while doing the work. A logging gap is different, because it stays invisible while it costs you, and by the time it matters you’re the person who was in the account when it happened. Check the trail is delivering. Then check the rest.

Need a second pair of eyes on an AWS account?

I do this for consultancies scoping engagements and for businesses who inherited an account they don’t fully understand. Usually that looks like:

• A read-only AWS account health check with a written findings list, hour estimates per item, and a clear split between urgent and later.
• Cross-account audit role setup, with a trust policy and external ID you can hand to any consultant without granting more than you meant to.
• Credential archaeology: working out what every active access key belongs to, then rotating or removing the ones that belong to nothing.
• Getting CloudTrail, AWS Config and log retention into a state where they’re delivering rather than just configured.
• Terraform drift reconciliation, including importing unmanaged resources back under code without a destructive plan.
• Cost shape reviews: the forgotten NAT gateways, orphaned volumes and idle regions, plus budgets and anomaly alerts that reach a real inbox.

If you’ve got a credential report, a terraform plan output or a Cost Explorer screenshot you can’t make sense of, send it over and I’ll tell you what I’d look at next.

The post The AWS Account Health Check I Run Before Quoting a Project appeared first on John Nessime.