That is a different question, and most evidence buckets fail it quietly. Versioning is on, the bucket is encrypted, access logging is enabled, and there is still an admin role in the account that can call DeleteObjectVersion on every object in it. Nothing is wrong until someone asks you to prove a negative.

An audit evidence repository on AWS is not a bucket with a good name. It is a write path, a retention decision you cannot take back, and a set of guardrails that survive the person who built them leaving the company. This post covers how to build one with S3 Object Lock and AWS Backup Vault Lock, and more usefully, the five ways I have seen the design go wrong in ways nobody notices until the evidence is needed or the bill arrives.

What “immutable” actually buys you on AWS

Three separate mechanisms get called immutability, and they protect different things.

• S3 Object Lock stops an object version from being overwritten or deleted for a defined period. It works only on versioned buckets, and once you enable it on a bucket you cannot disable it or suspend versioning again.
• AWS Backup Vault Lock does the same job for recovery points in a backup vault, with its own retention floor and ceiling enforced at the vault level.
• Integrity validation, such as CloudTrail’s digest files, proves a file has not changed since delivery. That is a separate property from being undeletable, and auditors increasingly ask for both.

Object Lock is the core of an audit evidence repository on AWS because evidence usually arrives as files: exported reports, signed approvals, config snapshots, log archives. Vault Lock matters when the evidence is a backup of a database or an EBS volume. Integrity validation is what turns “we kept it” into “we kept it and here is the hash chain”.

Failure one: picking the retention mode you cannot walk back

Object Lock has two retention modes and the difference is not cosmetic. In governance mode, a caller who holds the s3:BypassGovernanceRetention permission can shorten or remove retention and delete the version. In compliance mode, nobody can, including the account root user, and the retention period cannot be shortened or downgraded to governance.

Read that second sentence again with a typo in mind. Set seven years where you meant seven days, in compliance mode, and the only exit AWS documents is closing the account. That is not a bug. It is the entire point of the mode. But it means the mode is a business decision wearing a technical costume.

How I decide: compliance mode only where an external rule names a retention period and an external party will inspect it. Everything else starts in governance mode, with the bypass permission granted to nobody by default and denied outright in a service control policy. Governance mode with no principal holding the bypass right is functionally immutable for day-to-day purposes, and it leaves you a documented, auditable escape hatch that requires an explicit policy change to use.

The bypass is not silent, either. Overriding governance retention requires the x-amz-bypass-governance-retention:true header on the request, which means the action shows up in CloudTrail as a distinctly shaped call. That is a good alarm to build.

# Create the bucket with Object Lock on. Versioning is enabled automatically
# and cannot be suspended afterwards.
aws s3api create-bucket 
  --bucket my-evidence-archive 
  --object-lock-enabled-for-bucket 
  --region eu-west-1 
  --create-bucket-configuration LocationConstraint=eu-west-1

# Default retention for NEW objects written to the bucket.
aws s3api put-object-lock-configuration 
  --bucket my-evidence-archive 
  --object-lock-configuration 
  '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}}'

Start there. Run it for a quarter. Watch what actually lands in the bucket and how big it gets. Then decide whether compliance mode is worth its permanence.

Failure two: retention that was never applied to the objects that matter

This is the invisible one, and it is the reason to read the rest of this post.

Object Lock is enabled at the bucket level, but retention lives on the individual object version. A default retention rule applies to objects written after the rule exists. It does not reach backwards. Enable Object Lock on an existing bucket holding three years of evidence and you get a bucket that reports Object Lock as enabled while every object already in it carries no retention at all.

The bucket console looks correct. The compliance checklist gets ticked. The old evidence is still deletable by anyone with delete permissions.

Two things to do about it. First, backfill retention on existing versions, either object by object with put-object-retention or, for anything above a few thousand keys, with S3 Batch Operations, which exists precisely for applying retention at scale. Second, verify per object rather than per bucket.

# Check what protection a specific version actually carries.
# Look for ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus.
aws s3api head-object 
  --bucket my-evidence-archive 
  --key 2024/q3/access-review-export.csv 
  --version-id "SOME_VERSION_ID"

The stronger fix is to stop trusting the writer to set retention correctly and enforce a floor in the bucket policy. S3 exposes the s3:object-lock-remaining-retention-days condition key for exactly this, so you can reject writes that ask for too short a lock.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RejectShortRetention",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-evidence-archive/*",
      "Condition": {
        "NumericLessThan": {
          "s3:object-lock-remaining-retention-days": "365"
        }
      }
    }
  ]
}

The write path now fails loudly when someone gets it wrong, which is what you want. A rejected upload gets noticed. A silently unprotected object does not.

Failure three: the delete that looks like a delete

Object Lock protects object versions. It does not stop new versions being created, and it does not stop delete markers being placed on top of a protected object.

So a user with ordinary delete permissions can issue a delete against a locked object, get a success response, and make it vanish from every normal listing. The protected version is still there. Nothing was destroyed. But an auditor browsing the bucket sees nothing, and the person who ran the command believes the file is gone.

Three practical consequences:

• Deny s3:DeleteObject on the evidence prefix for everyone except a break-glass role. Immutability is not a substitute for permissions, it is a backstop under them.
• Any retrieval tooling you hand to an auditor must list object versions, not objects. list-object-versions, not list-objects-v2.
• Lifecycle rules interact with this in a way that surprises people. An expiration rule on a versioned bucket adds a delete marker rather than removing data, and a rule that tries to permanently expire a noncurrent version will not remove a version still under retention. Your cleanup policy and your retention policy are two different clocks, and the lock always wins.

Failure four: evidence that arrives already unprovable

Immutability starts when the object lands. Everything before that is chain of custody, and it is where most repositories are weakest.

If the evidence is CloudTrail, use the integrity validation CloudTrail already ships. With log file validation enabled, CloudTrail writes an hourly digest file containing hashes of the log files delivered in that hour, signs each digest with a per-region private key, and puts the digests in a separate folder in the same bucket. You can then verify the chain later.

# Turn on digest files for an existing trail.
aws cloudtrail update-trail 
  --name org-audit-trail 
  --enable-log-file-validation

# Verify a time range later. Files must still be in the location
# CloudTrail delivered them to, or validation cannot run.
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-audit-trail 
  --start-time 2020-01-01T00:00:00Z

Two details worth knowing before you design around it. Validation only covers log files referenced by digest files, so a gap in digest coverage is a gap in your proof. And the files must stay where CloudTrail put them, which means copying logs into a tidier prefix breaks the built-in verification and hands you the job of writing your own checker.

For evidence that is not CloudTrail, the equivalent is doing the work yourself: hash the artifact at the point of generation, record the hash somewhere separate from the artifact, and write both. A manifest per collection run, checksummed and locked alongside the files, is unglamorous and holds up.

One constraint that catches people mid-build: a bucket with Object Lock enabled cannot be used as the destination for S3 server access logs. If your design had one bucket receiving both the evidence and its own access logs, it needs splitting.

Failure five: the bill you cannot switch off

Compliance mode plus a long retention period plus archival storage classes is where cost surprises live, because every lever you would normally pull to reduce spend has been deliberately disabled.

The mechanics that drive it, none of which are secret but all of which get discovered late:

• Minimum storage durations. S3 Glacier Flexible Retrieval carries a 90 day minimum and S3 Glacier Deep Archive carries 180 days. Transition earlier than that and you are billed for the remainder of the minimum anyway.
• Per-object overhead in Glacier classes. AWS documents 40 KB of metadata overhead per object for Glacier Flexible Retrieval and Deep Archive, of which 8 KB is billed at S3 Standard rates and 32 KB at the destination rate. For a repository full of 2 KB JSON approvals, the overhead is the bill.
• Billing starts when the rule is satisfied. Lifecycle transitions to Glacier classes happen asynchronously, but you are charged at the destination rate, and the minimum duration clock starts, from the date the rule matched rather than the date the bytes physically moved.
• You cannot delete your way out. Under compliance mode, oversized or mistakenly retained objects stay until their retain-until date passes, whatever the finance conversation says.

The design response is to aggregate before you lock. Batch small artifacts into per-day or per-collection archives, checksum the archive, and lock the archive rather than thousands of tiny files. It costs you a little retrieval convenience and saves you a per-object overhead multiplier you cannot renegotiate later.

Also pick the retrieval tier against the real deadline. Deep Archive is the cheapest place to keep evidence and one of the slowest to get it back. If a regulator or a legal team may need files at short notice, that trade is a bad one regardless of the storage line item.

Putting the repository together

With the failure families in view, the build order is fairly short.

1. Separate the account. Evidence lives in its own AWS account, not a prefix in the production log bucket. The blast radius of a compromised production admin should not include the record of what they did.
2. Write the service control policies first. At the organisation level, deny s3:BypassGovernanceRetention, deny s3:PutBucketPolicy and s3:DeleteBucketPolicy on the evidence bucket, and deny disabling the trail. SCPs apply to the account root too, which is the point.
3. Create the bucket with Object Lock and a default governance retention. Block public access, enforce TLS with an aws:SecureTransport condition, and encrypt with a customer managed KMS key whose policy is written for this bucket alone.
4. Enforce the retention floor in the bucket policy using the remaining-retention-days condition, so a misconfigured collector fails rather than writes something unprotected.
5. Replicate to a second account or region. Object Lock works with S3 Replication, and the destination bucket must also have Object Lock enabled. This is your answer to region-level loss and to an account-level mistake.
6. Add Vault Lock for backup-shaped evidence. Snapshots and database backups belong in a locked backup vault rather than being exported to S3 by hand.
7. Alarm on the write path. A day with no new evidence objects is the failure that never pages anyone. Alert on absence, not just on errors.

Vault Lock deserves its own note because its compliance mode has a grace period that Object Lock does not. You choose a cooling-off window, which AWS requires to be at least three days, during which the lock can still be removed. After that the vault and its lock are immutable and cannot be changed or deleted by anyone, including AWS.

# Compliance mode: the presence of --changeable-for-days is what makes it
# compliance rather than governance. Minimum value is 3 (72 hours).
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name evidence-vault 
  --min-retention-days 365 
  --max-retention-days 2555 
  --changeable-for-days 3

Use that window. Run a backup job through the vault, confirm the retention arithmetic against what the policy document actually requires, and only then let the clock run out.

One more design question worth asking out loud: does every copy need to be inside AWS? A second, offline-ish copy on independent infrastructure answers a different threat than cross-region replication does, because it survives an account-level compromise. Providers like InterServer, Contabo and Hetzner sell cheap storage that can hold an encrypted archive, and several S3-compatible services implement Object Lock semantics of their own. Confirm the provider actually supports Object Lock rather than assuming an S3-compatible API implies it, because that varies.

Troubleshooting

AccessDenied when deleting a version you are allowed to delete

Usually governance retention plus a missing bypass. The IAM permission s3:BypassGovernanceRetention and the explicit bypass on the request are both required. If the object is in compliance mode, there is no combination of permissions that will work, and the error is telling you the truth.

# Only valid for governance mode, and only with the matching IAM permission.
aws s3api delete-object 
  --bucket my-evidence-archive 
  --key path/to/object 
  --version-id "SOME_VERSION_ID" 
  --bypass-governance-retention

Replication fails on an Object Lock bucket

The destination needs Object Lock enabled too. Replicated objects take the lock configuration of the source, so a destination without Object Lock cannot accept them. Check the destination bucket first, then the replication role’s permissions.

Lifecycle rule appears to do nothing

Two likely causes. The versions it targets are still under retention, so expiration cannot remove them. Or the rule targets current versions on a versioned bucket, in which case it is adding delete markers and the data is still there as noncurrent versions, still being billed.

validate-logs reports invalid or missing files

Confirm nothing moved the objects. Validation requires the log and digest files to be in the location CloudTrail delivered them to, and the command runs per region. A copy job that reorganised the prefix layout is the usual culprit.

Object Lock cannot be enabled on the bucket

Versioning has to be enabled first, and Object Lock cannot be turned off later. If the bucket is currently a server access log destination, that has to move before Object Lock will work for you at all.

Common mistakes

• Enabling Object Lock on an existing bucket and assuming the objects already in it are now protected. They are not, until retention is applied to each version.
• Choosing compliance mode on day one to look thorough, then discovering the retention period was wrong.
• Treating Object Lock as an access control. It is a floor under your permissions, not a replacement for them.
• Leaving the bypass permission attached to a general administrator role because it was needed once during testing.
• Locking millions of tiny objects into Glacier classes and paying per-object overhead on all of them.
• Storing evidence in the same account as the systems it describes, which makes the evidence only as trustworthy as that account’s admins.
• Never testing retrieval, so the first restore attempt happens under audit pressure with an unfamiliar Glacier restore workflow.

Best practices for an audit evidence repository on AWS

• Governance mode by default, compliance mode only where an external requirement names the period. Write down which rule drove the choice, in the same repository.
• Enforce the retention floor in the bucket policy so the write path fails loudly instead of writing unprotected objects.
• Use legal hold, not extended retention, for investigations. Legal holds have no expiry and are removed explicitly, which is the right shape for something that ends on an unknown date.
• Aggregate small artifacts into archives with a checksummed manifest before locking them.
• Replicate to a second account with Object Lock enabled, and check replication metrics rather than assuming.
• Rehearse retrieval quarterly, including the Glacier restore step, and keep the output as evidence of the rehearsal.
• Monitor for absence of writes. Tools like Amazon CloudWatch, Grafana or Datadog are all fine here. The alert that matters is “no evidence arrived today”.

Frequently asked questions

Can I enable S3 Object Lock on a bucket that already exists?

Yes. S3 supports enabling Object Lock on existing buckets, and versioning must be enabled. The important caveat is that enabling it does not retroactively protect the objects already stored. You apply retention to those separately, per object or with S3 Batch Operations.

What is the difference between governance mode and compliance mode?

Governance mode can be overridden by a principal holding s3:BypassGovernanceRetention who sends the bypass header. Compliance mode cannot be overridden by anyone, including the root user, and the retention period cannot be shortened or the mode downgraded.

Does S3 Object Lock stop someone deleting the file?

It stops the protected version being destroyed. It does not stop a delete marker being placed on top, which hides the object from ordinary listings while the version remains. Keep delete permissions tight and list versions when you audit.

Do I need Object Lock if I already have versioning and MFA delete?

They solve different problems. Versioning preserves history but a sufficiently privileged caller can still remove versions. Object Lock removes that possibility for the duration of the retention period, which is the property an auditor is asking about.

How do I prove the evidence has not been modified?

For CloudTrail, enable log file integrity validation and keep the signed digest files with the logs, then run aws cloudtrail validate-logs for the period in question. For other artifacts, hash at generation time, store the manifest separately, and lock both.

Can a lifecycle rule delete objects that are still locked?

No. Retention wins over lifecycle expiration. Plan the two together, because a cleanup rule that quietly does nothing for years is a cost problem you will not see until you look at storage growth.

Is Vault Lock compliance mode reversible?

Only during the cooling-off window you set at creation, which must be at least three days. After the lock date, the vault and its lock cannot be changed or deleted by anyone, including AWS.

The one thing worth remembering

An audit evidence repository on AWS fails in a specific, quiet way: the bucket says Object Lock is enabled, and the objects inside carry no retention. Everything else in this post is downstream of that gap. Check protection at the object version, not at the bucket, and enforce the floor in the write path so the system tells you when it is wrong.

Then pick the mode like the one-way door it is. Governance mode with a locked-down bypass gets you almost all of the protection and keeps a documented way back. Compliance mode is worth its permanence when a real external rule demands it, and expensive theatre when it does not.

Need help building or reviewing one of these?

Immutable retention is easy to enable and easy to get subtly wrong, and the mistakes are the kind you find out about at the worst moment. This is work I take on directly:

• Designing an evidence account with Object Lock, KMS, service control policies and a locked-down write path
• Auditing an existing bucket to find object versions that carry no retention despite Object Lock being enabled
• Backfilling retention across existing objects with S3 Batch Operations without locking in the wrong period
• Modelling the long-run cost of a retention decision before compliance mode makes it permanent
• Setting up cross-account replication, AWS Backup Vault Lock and CloudTrail integrity validation as one coherent design
• Writing the retrieval runbook and rehearsing it, so the first restore is not during an audit

If you want a second opinion on what you already have, send me the bucket policy, the Object Lock configuration and the output of a head-object call on one old evidence file. That is usually enough to tell you where the gap is.

The post Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret appeared first on John Nessime.