That is a different question, and most evidence buckets fail it quietly. Versioning is on, the bucket is encrypted, access logging is enabled, and there is still an admin role in the account that can call DeleteObjectVersion on every object in it. Nothing is wrong until someone asks you to prove a negative.

An audit evidence repository on AWS is not a bucket with a good name. It is a write path, a retention decision you cannot take back, and a set of guardrails that survive the person who built them leaving the company. This post covers how to build one with S3 Object Lock and AWS Backup Vault Lock, and more usefully, the five ways I have seen the design go wrong in ways nobody notices until the evidence is needed or the bill arrives.

What “immutable” actually buys you on AWS

Three separate mechanisms get called immutability, and they protect different things.

• S3 Object Lock stops an object version from being overwritten or deleted for a defined period. It works only on versioned buckets, and once you enable it on a bucket you cannot disable it or suspend versioning again.
• AWS Backup Vault Lock does the same job for recovery points in a backup vault, with its own retention floor and ceiling enforced at the vault level.
• Integrity validation, such as CloudTrail’s digest files, proves a file has not changed since delivery. That is a separate property from being undeletable, and auditors increasingly ask for both.

Object Lock is the core of an audit evidence repository on AWS because evidence usually arrives as files: exported reports, signed approvals, config snapshots, log archives. Vault Lock matters when the evidence is a backup of a database or an EBS volume. Integrity validation is what turns “we kept it” into “we kept it and here is the hash chain”.

Failure one: picking the retention mode you cannot walk back

Object Lock has two retention modes and the difference is not cosmetic. In governance mode, a caller who holds the s3:BypassGovernanceRetention permission can shorten or remove retention and delete the version. In compliance mode, nobody can, including the account root user, and the retention period cannot be shortened or downgraded to governance.

Read that second sentence again with a typo in mind. Set seven years where you meant seven days, in compliance mode, and the only exit AWS documents is closing the account. That is not a bug. It is the entire point of the mode. But it means the mode is a business decision wearing a technical costume.

How I decide: compliance mode only where an external rule names a retention period and an external party will inspect it. Everything else starts in governance mode, with the bypass permission granted to nobody by default and denied outright in a service control policy. Governance mode with no principal holding the bypass right is functionally immutable for day-to-day purposes, and it leaves you a documented, auditable escape hatch that requires an explicit policy change to use.

The bypass is not silent, either. Overriding governance retention requires the x-amz-bypass-governance-retention:true header on the request, which means the action shows up in CloudTrail as a distinctly shaped call. That is a good alarm to build.

# Create the bucket with Object Lock on. Versioning is enabled automatically
# and cannot be suspended afterwards.
aws s3api create-bucket 
  --bucket my-evidence-archive 
  --object-lock-enabled-for-bucket 
  --region eu-west-1 
  --create-bucket-configuration LocationConstraint=eu-west-1

# Default retention for NEW objects written to the bucket.
aws s3api put-object-lock-configuration 
  --bucket my-evidence-archive 
  --object-lock-configuration 
  '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}}'

Start there. Run it for a quarter. Watch what actually lands in the bucket and how big it gets. Then decide whether compliance mode is worth its permanence.

Failure two: retention that was never applied to the objects that matter

This is the invisible one, and it is the reason to read the rest of this post.

Object Lock is enabled at the bucket level, but retention lives on the individual object version. A default retention rule applies to objects written after the rule exists. It does not reach backwards. Enable Object Lock on an existing bucket holding three years of evidence and you get a bucket that reports Object Lock as enabled while every object already in it carries no retention at all.

The bucket console looks correct. The compliance checklist gets ticked. The old evidence is still deletable by anyone with delete permissions.

Two things to do about it. First, backfill retention on existing versions, either object by object with put-object-retention or, for anything above a few thousand keys, with S3 Batch Operations, which exists precisely for applying retention at scale. Second, verify per object rather than per bucket.

# Check what protection a specific version actually carries.
# Look for ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus.
aws s3api head-object 
  --bucket my-evidence-archive 
  --key 2024/q3/access-review-export.csv 
  --version-id "SOME_VERSION_ID"

The stronger fix is to stop trusting the writer to set retention correctly and enforce a floor in the bucket policy. S3 exposes the s3:object-lock-remaining-retention-days condition key for exactly this, so you can reject writes that ask for too short a lock.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RejectShortRetention",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-evidence-archive/*",
      "Condition": {
        "NumericLessThan": {
          "s3:object-lock-remaining-retention-days": "365"
        }
      }
    }
  ]
}

The write path now fails loudly when someone gets it wrong, which is what you want. A rejected upload gets noticed. A silently unprotected object does not.

Failure three: the delete that looks like a delete

Object Lock protects object versions. It does not stop new versions being created, and it does not stop delete markers being placed on top of a protected object.

So a user with ordinary delete permissions can issue a delete against a locked object, get a success response, and make it vanish from every normal listing. The protected version is still there. Nothing was destroyed. But an auditor browsing the bucket sees nothing, and the person who ran the command believes the file is gone.

Three practical consequences:

• Deny s3:DeleteObject on the evidence prefix for everyone except a break-glass role. Immutability is not a substitute for permissions, it is a backstop under them.
• Any retrieval tooling you hand to an auditor must list object versions, not objects. list-object-versions, not list-objects-v2.
• Lifecycle rules interact with this in a way that surprises people. An expiration rule on a versioned bucket adds a delete marker rather than removing data, and a rule that tries to permanently expire a noncurrent version will not remove a version still under retention. Your cleanup policy and your retention policy are two different clocks, and the lock always wins.

Failure four: evidence that arrives already unprovable

Immutability starts when the object lands. Everything before that is chain of custody, and it is where most repositories are weakest.

If the evidence is CloudTrail, use the integrity validation CloudTrail already ships. With log file validation enabled, CloudTrail writes an hourly digest file containing hashes of the log files delivered in that hour, signs each digest with a per-region private key, and puts the digests in a separate folder in the same bucket. You can then verify the chain later.

# Turn on digest files for an existing trail.
aws cloudtrail update-trail 
  --name org-audit-trail 
  --enable-log-file-validation

# Verify a time range later. Files must still be in the location
# CloudTrail delivered them to, or validation cannot run.
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-audit-trail 
  --start-time 2020-01-01T00:00:00Z

Two details worth knowing before you design around it. Validation only covers log files referenced by digest files, so a gap in digest coverage is a gap in your proof. And the files must stay where CloudTrail put them, which means copying logs into a tidier prefix breaks the built-in verification and hands you the job of writing your own checker.

For evidence that is not CloudTrail, the equivalent is doing the work yourself: hash the artifact at the point of generation, record the hash somewhere separate from the artifact, and write both. A manifest per collection run, checksummed and locked alongside the files, is unglamorous and holds up.

One constraint that catches people mid-build: a bucket with Object Lock enabled cannot be used as the destination for S3 server access logs. If your design had one bucket receiving both the evidence and its own access logs, it needs splitting.

Failure five: the bill you cannot switch off

Compliance mode plus a long retention period plus archival storage classes is where cost surprises live, because every lever you would normally pull to reduce spend has been deliberately disabled.

The mechanics that drive it, none of which are secret but all of which get discovered late:

• Minimum storage durations. S3 Glacier Flexible Retrieval carries a 90 day minimum and S3 Glacier Deep Archive carries 180 days. Transition earlier than that and you are billed for the remainder of the minimum anyway.
• Per-object overhead in Glacier classes. AWS documents 40 KB of metadata overhead per object for Glacier Flexible Retrieval and Deep Archive, of which 8 KB is billed at S3 Standard rates and 32 KB at the destination rate. For a repository full of 2 KB JSON approvals, the overhead is the bill.
• Billing starts when the rule is satisfied. Lifecycle transitions to Glacier classes happen asynchronously, but you are charged at the destination rate, and the minimum duration clock starts, from the date the rule matched rather than the date the bytes physically moved.
• You cannot delete your way out. Under compliance mode, oversized or mistakenly retained objects stay until their retain-until date passes, whatever the finance conversation says.

The design response is to aggregate before you lock. Batch small artifacts into per-day or per-collection archives, checksum the archive, and lock the archive rather than thousands of tiny files. It costs you a little retrieval convenience and saves you a per-object overhead multiplier you cannot renegotiate later.

Also pick the retrieval tier against the real deadline. Deep Archive is the cheapest place to keep evidence and one of the slowest to get it back. If a regulator or a legal team may need files at short notice, that trade is a bad one regardless of the storage line item.

Putting the repository together

With the failure families in view, the build order is fairly short.

1. Separate the account. Evidence lives in its own AWS account, not a prefix in the production log bucket. The blast radius of a compromised production admin should not include the record of what they did.
2. Write the service control policies first. At the organisation level, deny s3:BypassGovernanceRetention, deny s3:PutBucketPolicy and s3:DeleteBucketPolicy on the evidence bucket, and deny disabling the trail. SCPs apply to the account root too, which is the point.
3. Create the bucket with Object Lock and a default governance retention. Block public access, enforce TLS with an aws:SecureTransport condition, and encrypt with a customer managed KMS key whose policy is written for this bucket alone.
4. Enforce the retention floor in the bucket policy using the remaining-retention-days condition, so a misconfigured collector fails rather than writes something unprotected.
5. Replicate to a second account or region. Object Lock works with S3 Replication, and the destination bucket must also have Object Lock enabled. This is your answer to region-level loss and to an account-level mistake.
6. Add Vault Lock for backup-shaped evidence. Snapshots and database backups belong in a locked backup vault rather than being exported to S3 by hand.
7. Alarm on the write path. A day with no new evidence objects is the failure that never pages anyone. Alert on absence, not just on errors.

Vault Lock deserves its own note because its compliance mode has a grace period that Object Lock does not. You choose a cooling-off window, which AWS requires to be at least three days, during which the lock can still be removed. After that the vault and its lock are immutable and cannot be changed or deleted by anyone, including AWS.

# Compliance mode: the presence of --changeable-for-days is what makes it
# compliance rather than governance. Minimum value is 3 (72 hours).
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name evidence-vault 
  --min-retention-days 365 
  --max-retention-days 2555 
  --changeable-for-days 3

Use that window. Run a backup job through the vault, confirm the retention arithmetic against what the policy document actually requires, and only then let the clock run out.

One more design question worth asking out loud: does every copy need to be inside AWS? A second, offline-ish copy on independent infrastructure answers a different threat than cross-region replication does, because it survives an account-level compromise. Providers like InterServer, Contabo and Hetzner sell cheap storage that can hold an encrypted archive, and several S3-compatible services implement Object Lock semantics of their own. Confirm the provider actually supports Object Lock rather than assuming an S3-compatible API implies it, because that varies.

Troubleshooting

AccessDenied when deleting a version you are allowed to delete

Usually governance retention plus a missing bypass. The IAM permission s3:BypassGovernanceRetention and the explicit bypass on the request are both required. If the object is in compliance mode, there is no combination of permissions that will work, and the error is telling you the truth.

# Only valid for governance mode, and only with the matching IAM permission.
aws s3api delete-object 
  --bucket my-evidence-archive 
  --key path/to/object 
  --version-id "SOME_VERSION_ID" 
  --bypass-governance-retention

Replication fails on an Object Lock bucket

The destination needs Object Lock enabled too. Replicated objects take the lock configuration of the source, so a destination without Object Lock cannot accept them. Check the destination bucket first, then the replication role’s permissions.

Lifecycle rule appears to do nothing

Two likely causes. The versions it targets are still under retention, so expiration cannot remove them. Or the rule targets current versions on a versioned bucket, in which case it is adding delete markers and the data is still there as noncurrent versions, still being billed.

validate-logs reports invalid or missing files

Confirm nothing moved the objects. Validation requires the log and digest files to be in the location CloudTrail delivered them to, and the command runs per region. A copy job that reorganised the prefix layout is the usual culprit.

Object Lock cannot be enabled on the bucket

Versioning has to be enabled first, and Object Lock cannot be turned off later. If the bucket is currently a server access log destination, that has to move before Object Lock will work for you at all.

Common mistakes

• Enabling Object Lock on an existing bucket and assuming the objects already in it are now protected. They are not, until retention is applied to each version.
• Choosing compliance mode on day one to look thorough, then discovering the retention period was wrong.
• Treating Object Lock as an access control. It is a floor under your permissions, not a replacement for them.
• Leaving the bypass permission attached to a general administrator role because it was needed once during testing.
• Locking millions of tiny objects into Glacier classes and paying per-object overhead on all of them.
• Storing evidence in the same account as the systems it describes, which makes the evidence only as trustworthy as that account’s admins.
• Never testing retrieval, so the first restore attempt happens under audit pressure with an unfamiliar Glacier restore workflow.

Best practices for an audit evidence repository on AWS

• Governance mode by default, compliance mode only where an external requirement names the period. Write down which rule drove the choice, in the same repository.
• Enforce the retention floor in the bucket policy so the write path fails loudly instead of writing unprotected objects.
• Use legal hold, not extended retention, for investigations. Legal holds have no expiry and are removed explicitly, which is the right shape for something that ends on an unknown date.
• Aggregate small artifacts into archives with a checksummed manifest before locking them.
• Replicate to a second account with Object Lock enabled, and check replication metrics rather than assuming.
• Rehearse retrieval quarterly, including the Glacier restore step, and keep the output as evidence of the rehearsal.
• Monitor for absence of writes. Tools like Amazon CloudWatch, Grafana or Datadog are all fine here. The alert that matters is “no evidence arrived today”.

Frequently asked questions

Can I enable S3 Object Lock on a bucket that already exists?

Yes. S3 supports enabling Object Lock on existing buckets, and versioning must be enabled. The important caveat is that enabling it does not retroactively protect the objects already stored. You apply retention to those separately, per object or with S3 Batch Operations.

What is the difference between governance mode and compliance mode?

Governance mode can be overridden by a principal holding s3:BypassGovernanceRetention who sends the bypass header. Compliance mode cannot be overridden by anyone, including the root user, and the retention period cannot be shortened or the mode downgraded.

Does S3 Object Lock stop someone deleting the file?

It stops the protected version being destroyed. It does not stop a delete marker being placed on top, which hides the object from ordinary listings while the version remains. Keep delete permissions tight and list versions when you audit.

Do I need Object Lock if I already have versioning and MFA delete?

They solve different problems. Versioning preserves history but a sufficiently privileged caller can still remove versions. Object Lock removes that possibility for the duration of the retention period, which is the property an auditor is asking about.

How do I prove the evidence has not been modified?

For CloudTrail, enable log file integrity validation and keep the signed digest files with the logs, then run aws cloudtrail validate-logs for the period in question. For other artifacts, hash at generation time, store the manifest separately, and lock both.

Can a lifecycle rule delete objects that are still locked?

No. Retention wins over lifecycle expiration. Plan the two together, because a cleanup rule that quietly does nothing for years is a cost problem you will not see until you look at storage growth.

Is Vault Lock compliance mode reversible?

Only during the cooling-off window you set at creation, which must be at least three days. After the lock date, the vault and its lock cannot be changed or deleted by anyone, including AWS.

The one thing worth remembering

An audit evidence repository on AWS fails in a specific, quiet way: the bucket says Object Lock is enabled, and the objects inside carry no retention. Everything else in this post is downstream of that gap. Check protection at the object version, not at the bucket, and enforce the floor in the write path so the system tells you when it is wrong.

Then pick the mode like the one-way door it is. Governance mode with a locked-down bypass gets you almost all of the protection and keeps a documented way back. Compliance mode is worth its permanence when a real external rule demands it, and expensive theatre when it does not.

Need help building or reviewing one of these?

Immutable retention is easy to enable and easy to get subtly wrong, and the mistakes are the kind you find out about at the worst moment. This is work I take on directly:

• Designing an evidence account with Object Lock, KMS, service control policies and a locked-down write path
• Auditing an existing bucket to find object versions that carry no retention despite Object Lock being enabled
• Backfilling retention across existing objects with S3 Batch Operations without locking in the wrong period
• Modelling the long-run cost of a retention decision before compliance mode makes it permanent
• Setting up cross-account replication, AWS Backup Vault Lock and CloudTrail integrity validation as one coherent design
• Writing the retrieval runbook and rehearsing it, so the first restore is not during an audit

If you want a second opinion on what you already have, send me the bucket policy, the Object Lock configuration and the output of a head-object call on one old evidence file. That is usually enough to tell you where the gap is.

The post Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret appeared first on John Nessime.

Then someone in finance asks what the reporting pipeline costs. You open Cost Explorer, group by Project, and over half the bill lands in a bucket called “No tag key”.

Nothing is broken. The tags are there. They just aren’t on the bill, because tagging a resource and activating a cost allocation tag are two separate operations, and AWS never warns you when you’ve done only the first.

This post covers getting AWS cost allocation tags to cover a messy account: the activation gap, what backfill will and won’t repair, where coverage leaks even with decent discipline, and a retrofit order that converges.

A resource tag and a cost allocation tag are not the same thing

A resource tag is metadata on the resource. A cost allocation tag is a billing dimension. Getting the second means taking a key that already exists on your resources and explicitly switching it on in the Billing and Cost Management console.

Two details make this bite. First, activation lives in the management account. From a member account, the page tells you tags are managed at the payer level and that’s the end of it. Plenty of teams have a good tagging standard and no access to the one screen that makes it count. Related trap: an account moving between organizations has its activated keys reset to inactive.

Second, the lag. AWS documents up to 24 hours for a newly applied key to appear on the activation page, and up to another 24 hours after you activate before it takes effect. The loop between “we agreed a standard” and “I can see it in Cost Explorer” is two days at best.

Past a handful of keys, drive it from the API. Listing the inactive ones is the fastest way to find what you forgot:

# Every key AWS knows about that is NOT currently billing-active
aws ce list-cost-allocation-tags --status Inactive

# Switch a set of them on
aws ce update-cost-allocation-tags-status --cli-input-json '{
  "CostAllocationTagsStatus": [
    {"TagKey": "Project",     "Status": "Active"},
    {"TagKey": "Environment", "Status": "Active"},
    {"TagKey": "Owner",       "Status": "Active"}
  ]
}'

User-defined keys appear in the report with a user: prefix and AWS-generated ones carry the reserved aws: prefix, so they never collide. There’s a cap on active keys, raisable through Service Quotas, which you almost certainly don’t need to raise.

What the backfill repairs, and what it quietly doesn’t

Activation used to be strictly forward-looking, so activating late permanently orphaned everything before it. That’s no longer true. The management account can ask AWS to reapply current activation status across up to twelve months of historical cost data.

# Must be the first day of a month, no earlier than twelve months back
aws ce start-cost-allocation-tag-backfill 
  --backfill-from 2025-01-01T00:00:00Z

• Start date must be the first day of a month, and can’t reach back more than twelve months.
• One request per 24 hours, and you can’t submit another while one is processing.
• It backfills activation status, not tag values.

That last point catches people. If the resource wasn’t carrying the tag when the cost was incurred, backfill gives you nothing for those months. It fixes “I forgot to activate.” It does not fix “I forgot to tag.” A tag applied in June and activated in November backfills to June and no further, whatever start date you hand it.

It also cuts both ways. Backfill applies the current status of every key, so a key sitting inactive right now gets stripped out of historical data too. If you’re pruning an overgrown tag list, backfill before you deactivate the old keys, not after, or you’ll erase the history you were trying to keep.

Where AWS cost allocation tags actually leak coverage

With activation sorted and every Terraform-managed resource tagged, a stubborn slice of the bill still refuses to attribute. It matters which slice, because the fixes have nothing in common.

Charges no resource can carry

Support charges, refunds, credits, tax, some Marketplace subscriptions. There’s no resource to hang a tag on, so tagging discipline never touches them. On accounts with a large support plan or heavy credit activity, that’s a visible fraction of the invoice.

The answer is account tags. AWS added support for using AWS Organizations account tags as a cost allocation dimension, applying to all metered usage inside the tagged account and flowing into Cost Explorer, Budgets, Cost Anomaly Detection and Cost Categories. Because attribution happens at the account level, it picks up exactly what resource tags can’t reach.

This is recent enough that most tagging guides predate it, and it’s the highest-value item here if your accounts already map roughly onto teams.

Resources created by other resources

This is the quiet one, and usually the biggest surprise gap in an otherwise well-run account. Terraform’s provider-level default_tags applies to resources Terraform creates. It does not reach the EC2 instances an Auto Scaling group launches, because Terraform didn’t create those, the ASG did. Your ASG is tagged. Your instances aren’t. Same for volumes attached at launch, ENIs, snapshots and AMIs.

data "aws_default_tags" "current" {}

resource "aws_autoscaling_group" "app" {
  # ... other config ...

  # Push provider default tags down to launched instances
  dynamic "tag" {
    for_each = data.aws_default_tags.current.tags
    content {
      key                 = tag.key
      value               = tag.value
      propagate_at_launch = true
    }
  }
}

resource "aws_launch_template" "app" {
  # ... other config ...

  tag_specifications {
    resource_type = "instance"
    tags          = data.aws_default_tags.current.tags
  }

  tag_specifications {
    resource_type = "volume"
    tags          = data.aws_default_tags.current.tags
  }
}

For a standalone instance, volume_tags on aws_instance covers its attached volumes. General rule: whenever one AWS resource provisions another on your behalf, assume tags don’t propagate until you’ve checked the created resource in the console, not the plan output.

Shared infrastructure that belongs to everyone

NAT gateway data processing, VPC interface endpoints, a Transit Gateway, a shared load balancer, the central logging bucket. You can tag all of these perfectly and the tag says Project=platform, which is accurate and useless for chargeback.

Transit Gateway is worth calling out because it supports tagging the gateway resource itself, so data processing and transfer charges pick up the tag rather than only attachment hours. That gap used to swallow a lot of network spend.

Beyond that, stop chasing tags and do allocation math instead. Cost Categories support split charge rules that redistribute a shared bucket across your other categories, using proportional splits, fixed percentages or an even division. Proportional is the sane default; fixed percentages are for when finance already agreed a number and the report has to match it.

Container spend hiding behind one instance tag

A shared EKS or ECS cluster is one tagged compute footprint running fifteen teams’ workloads. The instance tag gives you the total and nothing about who caused it. Split cost allocation data is the native answer: opt in and the cost and usage export gains rows breaking cluster cost down to individual pods or tasks. It won’t account for idle headroom you’re deliberately carrying, but it beats one line item labelled “the cluster”.

Case drift and near-duplicate keys

Keys and values are case-sensitive. Environment, environment and ENV are three distinct keys producing three columns, each holding a partial and misleading picture. Tag Policies in AWS Organizations exist for this: define the key including its capitalization, define permitted values, and get an org-wide compliance report. Cheapest governance on this list, because it costs nobody a deploy.

A retrofit order that converges

Retrofits stall because people start at the bottom, sweeping untagged resources by hand while the creation path keeps producing more. This order avoids that.

1. Measure first. Group Cost Explorer by your intended key and record what percentage of spend falls into “No tag key”. That’s your baseline and your only honest progress metric.
2. Cut the key set down. Three to five keys: owner, environment, and a project or cost centre. Coverage on five beats partial coverage on thirty.
3. Activate immediately, even with terrible coverage. Both clocks start now: the activation delay and the rolling twelve-month backfill window. There’s no downside to activating a key that’s barely applied.
4. Rank the gap by spend, not resource count. Sort untagged line items by cost descending. It’s usually a few NAT gateways, a couple of oversized databases and one forgotten bucket. Ten thousand untagged security groups cost nothing.
5. Fix the creation path before the backlog. IaC defaults, launch template tag specifications, ASG propagation. Sweeping first means sweeping again next month.
6. Sweep the stragglers in bulk. The Resource Groups Tagging API works across services and regions, which beats going service by service.
7. Backfill, then re-measure. Compare against step one. If the number barely moved, you fixed the wrong resources.

# Find what already carries a Project tag, to diff against inventory
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project

# Apply tags to a specific set of ARNs
aws resourcegroupstaggingapi tag-resources 
  --resource-arn-list arn:aws:rds:eu-west-1:111122223333:db:reporting-01 
  --tags Project=reporting,Environment=prod

Run it against a couple of ARNs and confirm in the console before looping over a list. Bulk tagging is trivially easy to point at the wrong account.

Enforcing tags without becoming the tag police

Four mechanisms, in increasing order of blast radius. Most teams need the first three and should be careful with the fourth.

• IaC defaults. Provider-level default tags plus required module variables. Cheapest, catches the majority, nobody thinks about it.
• CI validation. A linter rule that fails the pull request on missing required tags. The failure lands on the person who can fix it, when they can fix it.
• Detective controls. The AWS Config managed rule for required tags flags non-compliant resources without blocking anything. Tells the truth, breaks nothing.
• Preventive controls. A service control policy denying creation when a required key is absent, keyed on a null check against the request tag.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyRDSCreateWithoutProjectTag",
      "Effect": "Deny",
      "Action": "rds:CreateDBInstance",
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/Project": "true"
        }
      }
    }
  ]
}

The honest trade-off: service control policies are where tagging programmes go to die. Not every API supports tag-on-create, and not every legitimate workflow passes tags through the way you expect. Roll a broad deny across the org on day one and you break somebody’s deployment on day two, the ticket lands on you, and the policy is gone by day three along with everyone’s appetite for the project.

Scope it to one service at a time, starting in sandbox. I’d rather run Config plus Tag Policies for a quarter and fix the top three offenders in the IaC. The deny is a finishing move, not an opening one.

Measuring coverage so it doesn’t rot

Cost Explorer gives the headline number. For a per-service breakdown, query the cost and usage export in Athena. One caveat: tag column naming differs between export versions, older ones giving a flattened column per activated key and newer formats a map column. Run a DESCRIBE first rather than guessing.

-- Confirm the actual column names first
DESCRIBE your_cur_database.your_cur_table;

-- Then: untagged spend, ranked by service
SELECT   line_item_product_code,
         SUM(line_item_unblended_cost) AS untagged_cost
FROM     your_cur_database.your_cur_table
WHERE    your_project_tag_column = ''
  AND    line_item_unblended_cost > 0
GROUP BY 1
ORDER BY 2 DESC
LIMIT    20;

That ranked list is your work queue; everything below the top twenty is noise. The list call also returns a last-used timestamp per key, the cleanest way to find keys nobody has applied to anything in months.

If you’d rather not maintain the dashboard yourself, this is a fair place to buy rather than build. Grafana Cloud sits on Athena comfortably if you already run Grafana, and platforms like CloudZero, Vantage and nOps handle allocation modelling out of the box. Their real value is the shared-cost and commitment maths. Their real cost is another vendor between you and your billing data, and several price against a percentage of the spend they watch, so run that number first.

Troubleshooting

• The key isn’t in the activation list. It must exist on at least one resource first, and propagation takes up to a day. Past that, check you’re in the management account and the key matches exactly, including case.
• Activated, but everything still shows untagged. Give it a full billing cycle. A key switched on mid-month leaves the first half of that month untagged until you backfill.
• Tags stopped working after an account move. Activation doesn’t survive a move between organizations. Reactivate from the new management account, then backfill.
• Cluster costs still land on one team. Instance tags can’t split a shared cluster. Opt into split cost allocation data, or treat the cluster as a shared cost in a split charge rule.
• You’ve hit the active key limit. Almost always too many keys rather than a real need for a quota increase. Use last-used timestamps to find dead ones, and backfill before deactivating.

Common mistakes

• Assuming a tagged resource is a tagged cost. It isn’t until you activate the key.
• Waiting for good coverage before activating. Activate first, improve coverage second, backfill third.
• Deactivating stale keys and then backfilling, which wipes their history out of the data.
• Measuring coverage by resource count instead of by spend.
• Rolling out a broad tag-enforcing SCP in production before anyone has read a compliance report.
• Trusting default_tags to reach resources Terraform didn’t directly create.
• Putting anything sensitive in a tag value. Tags surface in billing exports that finance and third-party tools read.

Best practices

• Keep the mandatory set to three to five keys, with defined allowed values rather than free text.
• Pin casing in the standard and enforce it with Tag Policies, because AWS won’t do it for you.
• Tag accounts as well as resources. Account tags cover charges resource tags structurally cannot.
• Treat coverage as a tracked percentage of spend with an owner and a monthly trend, not a project that finishes.
• Accept an irreducible remainder. Chasing the last few percent by hand costs more than it returns.
• Put allocation rules in Cost Categories rather than inventing tag values that encode a split.

Frequently asked questions

Are AWS cost allocation tags retroactive?

Partly. Activation applies going forward, but the management account can request a backfill reapplying current activation status across up to twelve months. The catch is that the resource must have carried the tag at the time, so backfill recovers late activation, never late tagging.

Why don’t my tags appear in Cost Explorer?

In order of likelihood: the key was never activated in the Billing console; you’re looking from a member account; the activation delay hasn’t elapsed; or the key differs in case from the one you activated.

How many cost allocation tags can I activate?

There’s a documented per-account cap on active keys, raisable through Service Quotas, plus a separate per-resource cap on total tags. Check current numbers in the AWS billing quotas docs. In practice the binding limit is human: past roughly a dozen keys, consistency collapses.

Should I use account tags or resource tags?

Both, for different jobs. Account tags handle top-level attribution and cover untaggable charges like support, credits and refunds. Resource tags handle granularity inside an account. If your accounts already map cleanly to teams, account tags are the cheaper win and worth doing first.

Is this worth doing on a small account?

Scale the effort to the shape of the bill. If you run a handful of services under one account and can name every resource from memory, account tags plus two resource keys is plenty. Worth noticing too that a fixed-price VPS from a provider like Contabo or InterServer has trivially perfect cost allocation, because the bill is one predictable line. Elastic infrastructure buys flexibility and pays for it in attribution work.

How do I allocate shared costs like NAT gateways?

Not with tags. Put them in a Cost Categories split charge rule and redistribute proportionally across consuming categories. Tag the shared resource to identify the bucket, then let the rule do the allocation.

The one thing worth remembering

AWS cost allocation tags fail silently, and that’s what makes them frustrating. Nothing errors. No alarm fires. Your resources look correctly tagged in every console you check, the bill quietly disagrees, and you find out when someone asks a question you can’t answer.

So check one thing now: open the cost allocation tags page in your management account and see whether the keys you’ve been diligently applying are actually switched on. If they aren’t, activate them today, then backfill. That twelve-month window rolls, and every month you wait is history you can’t recover.

Everything else here is incremental. Coverage is a percentage you improve, not a state you reach.

Need help getting your AWS bill to explain itself?

Cost attribution is unglamorous and easy to postpone until the quarter someone actually needs the numbers. If your account is already messy, it goes faster with someone who has untangled one before. I help with:

• Auditing tag coverage as a percentage of spend, broken down by service, so you know what you’re actually fixing
• Designing a minimal tag schema and the Tag Policies to keep it consistent across an organization
• Closing propagation gaps in Terraform and CloudFormation, around Auto Scaling groups, launch templates and attached storage
• Setting up Cost Categories with split charge rules so shared infrastructure stops distorting every team’s number
• Building cost and usage export pipelines in Athena, partitioned so the dashboard isn’t expensive to load
• Planning a staged tag enforcement rollout that doesn’t break anyone’s deployment pipeline

Send me a Cost Explorer screenshot grouped by your main tag key, or the output of a list call against your active tags, and I’ll tell you where the gap actually is.

The post AWS Cost Allocation Tags: Getting Real Coverage on a Messy Bill appeared first on John Nessime.

It granted iam:PassRole on one role ARN and lambda:CreateFunction in the same document. That is not a wildcard and it does not read as administrative access. It is administrative access anyway, to whatever that role can do, for anyone who can write a Lambda function.

This is the shape of the problem with AI-generated IAM policies. The failure is almost never the obvious one. Models have absorbed enough style guidance to avoid "Action": "*", so what comes back passes the eyeball test and the linter and still hands over more than you meant to give. The dangerous output is the one that looks careful.

This post covers what policy generation genuinely does well, the failure shapes that survive human review, and where the hard boundary belongs: not in the review step, which is fallible, but in a mechanism that cannot be talked out of its answer.

What AI-generated IAM policies get right

Start with the honest case, because it is a strong one. IAM has thousands of actions across hundreds of services and the naming is inconsistent in ways nobody holds in their head. Some services use Describe, some use Get, some use both for different things. ARN formats vary per service and per resource type inside a service. Condition keys apply to some actions and not others.

A model is good at that recall problem, and at shape work: splitting a monolithic statement into per-resource statements, adding conditions, converting an inline policy into Terraform. Paste an AccessDenied message and you get a plausible read of which layer refused. So the argument is not that the tooling is useless. It is that recall and reasoning about consequences are different capabilities, and IAM is a system where the second one protects you.

The failure modes that survive review

Permissions that compose into something bigger

Every action is individually defensible. The escalation lives in the combination.

The canonical case is iam:PassRole beside any service that runs code: Lambda, EC2, ECS, Glue. Passing a role to a compute service means your code executes with that role’s permissions, so your ceiling is not your own policy, it is whatever the most powerful passable role can do. Rhino Security Labs catalogued a long list of these chains and every one of them is built from permissions that look unremarkable alone.

Another that catches people: iam:CreatePolicyVersion on a policy you are attached to. A new version normally needs to be set as default to take effect, but the create call accepts a flag that makes it default immediately, and that flag does not require iam:SetDefaultPolicyVersion. A permission that reads as “can update policy documents” reads to an attacker as “can write myself an admin policy.”

Ask a model whether a policy grants admin and it will check the statements and say no, correctly, one at a time. Composition is the part it does not reliably do, and it is the part that matters.

Actions that do not exist

Plausible action names that no service publishes. s3:ListBucketContents instead of s3:ListBucket. IAM accepts a policy containing an unrecognized action string because the document is syntactically valid, so nothing fails at attach time.

What happens instead is worse. The application throws AccessDenied in production, somebody debugging under pressure decides the granular list must be wrong, and the fix is a service wildcard. The invented action does not grant too much. It causes a human to grant too much a week later, and by then nobody connects the two events.

Resource ARNs that look constrained and are not

An ARN with real characters in it feels safer than a star. Often it is not. The common versions are a trailing wildcard placed one segment too high, and an action whose resource element the service ignores. s3:ListAllMyBuckets is account-scoped no matter what you put in Resource. Several ec2:Describe actions behave the same way. A policy can read as resource-scoped throughout and be account-wide for the actions that count.

The related trap is the missing condition: cross-account trust policies without an external ID, bucket policies without aws:PrincipalOrgID, integration roles without aws:SourceIp where the caller has a stable egress address. Generated policies include conditions when you ask and omit them when you do not, because an absent constraint never produces an error.

Four things the model cannot know

Some of this is not a capability gap you can prompt past. It is missing input:

• Which resources exist. A bucket name is a guess unless you supplied it. Guessed names either fail closed or, if the wildcard is loose enough, match something you never intended.
• Which resources are sensitive. Your backup bucket and your public assets bucket are indistinguishable strings. Sensitivity lives nowhere in the document.
• What else is attached. Effective permissions come from identity policies, resource policies, boundaries, SCPs and session policies together. Reviewing one document tells you very little.
• What the blast radius is. “Can delete objects” means one thing for a scratch bucket and another for the only copy of a client’s data.

The training material is also skewed. Public IAM examples, including plenty of vendor documentation and countless forum answers, are permissive because permissive examples work on the first try. A model that has read the internet has read a lot of over-broad policies presented as correct.

Where the hard boundary belongs

Review is not a boundary. Review is a filter with a pass rate that drops the more policies you look at in a week, and generated policies are cheap to produce, so volume rises exactly when attention falls.

The boundary has to be something that says no without reading the document. AWS gives you two mechanisms and they are not interchangeable.

A permissions boundary is a managed policy attached to a user or role that caps what identity-based policies can grant it. It grants nothing itself. Effective permissions become the intersection of the identity policy and the boundary, so an identity policy allowing iam:* against a boundary that omits IAM produces no IAM access at all. If no boundary is attached, none is evaluated, so this is opt-in per identity.

A service control policy sets the maximum available permissions for every principal in an account or organizational unit, and also grants nothing. It is the right home for things nobody in that account should ever do, and it applies whether or not somebody remembered to attach a boundary.

The division I use: SCPs carry account-wide invariants, such as denying escalation-relevant IAM write actions to everyone outside the break-glass role and denying regions you do not operate in. Boundaries carry the per-workload ceiling, and they are what makes it safe to let a pipeline role create other roles. You can permit that role to create roles while requiring, through a condition on the boundary key, that every role it creates carries your boundary. The pipeline can then generate whatever policy it likes, because the ceiling is enforced at evaluation time rather than at review time.

If the only thing between a generated policy and production is a person reading JSON, you do not have a boundary. You have a habit.

The gate I would actually run

Mechanical checks in a pipeline, failing the build, not a checklist somebody is supposed to remember.

1. Lint for nonsense. Parliament catches unknown action names, resource formats that cannot match the action, and type mismatches. Cheapest way to catch an invented action before it becomes a wildcard.
2. Run AWS policy validation. Access Analyzer knows service-specific rules a generic linter does not.
3. Assert the escalation actions are absent. Name the actions you never want granted and fail if any appear. This is the check that maps to the composition failure.
4. Compare against the current policy. For edits, check whether the proposed document grants access the existing one does not. This one is billed per call, so run it on changed policies rather than the whole repository.
5. Scan for known risky patterns. Cloudsplaining scores policies against categories including privilege escalation, data exfiltration and resource exposure, and produces something you can hand to a client.

# 1. Lint. Exit status is non-zero when there are findings.
parliament --file policy.json

# 2. AWS-side validation
aws accessanalyzer validate-policy 
  --policy-document file://policy.json 
  --policy-type IDENTITY_POLICY

# 3. Fail if any escalation-relevant action is granted
aws accessanalyzer check-access-not-granted 
  --policy-document file://policy.json 
  --access actions="iam:PassRole","iam:CreatePolicyVersion","iam:AttachRolePolicy" 
  --policy-type IDENTITY_POLICY

# 4. Does the proposed policy grant more than the current one?
aws accessanalyzer check-no-new-access 
  --existing-policy-document file://current.json 
  --new-policy-document file://proposed.json 
  --policy-type IDENTITY_POLICY

# 5. Risk-categorised report
cloudsplaining scan-policy-file --input-file policy.json

Step three is the one to add first if you only add one. It turns “we reviewed it” into “the build fails.”

Attaching the boundary is a single call, and it is the part people skip because the role already works without it:

aws iam put-role-permissions-boundary 
  --role-name app-deploy-role 
  --permissions-boundary arn:aws:iam::123456789012:policy/WorkloadCeiling

Generate from observed behavior instead

There is a better source of truth than a model’s guess about what an application needs, and it is the application. Access Analyzer policy generation reads CloudTrail events for a role over a chosen window and writes a policy from what was actually called. Know its limits before relying on it:

• It analyzes up to ninety days of history and needs a trail already logging for the account, so a role that only exercises certain paths quarterly produces an incomplete policy.
• Coverage is per-service. For some services it identifies individual actions; for others it can only tell you the service was used and prompts you to fill in the actions yourself.
• It does not produce action-level detail for data events such as S3 object-level operations.
• iam:PassRole is not included in generated policies. If your workload needs it, a human adds it back by hand, so the most escalation-relevant line in the document is the one the generator did not write.

Behavior-derived policies also fail in a useful direction. A missing permission produces a denial you can see in CloudTrail, which beats a permission you did not know you granted.

Arguments that do not survive contact

• “We review every policy before merge.” Review quality is a function of volume and attention, and generation raises volume. Composition failures are also the hardest to spot by reading, because spotting them means knowing what other roles exist and what they can do.
• “Access Analyzer came back clean.” Validation checks the document. It does not know that the role ARN you may pass happens to have AdministratorAccess attached. Clean findings on a policy that grants a dangerous chain is the normal case, not an anomaly.
• “A permissions boundary is overkill for a small project.” Small projects are where this bites hardest. One account, one deploy role, no organization, nothing between a bad policy and everything you own. The boundary is one managed policy and one CLI call.
• “We will tighten it after launch.” Nobody does. If you genuinely intend to, drive it from Access Analyzer’s unused access findings so you have a work queue instead of an intention.
• “Just prompt the model to follow least privilege.” That changes the output surface, not the reasoning. Fewer wildcards, better statements, no evaluation of what the combination permits.

How I would decide

My rule is about which actions are in the document, not about who or what wrote it.

Ship generated policies freely when the document touches only data-plane actions on named resources, the identity has a boundary, and the pipeline checks pass. Read a bucket prefix, publish to a topic, write to a table. That is most policies, and hand-writing them wastes your afternoon.

Treat generated output as a first draft when the policy includes any IAM write action, any PassRole, any trust policy edit, any resource policy readable from outside the account, or any KMS key policy. Use it to save typing, then verify every statement against the service authorization reference yourself.

Do not generate at all for the boundary policies and SCPs. Those are the mechanism. Something that constrains everything else should be short, hand-written, understood by whoever owns the account, and changed rarely.

Two notes for small engagements. If you run application infrastructure on a VPS from a provider like Contabo or InterServer and pull AWS in for storage or mail, that server holds long-lived credentials somewhere your AWS controls do not reach, so scope its policy to a bucket prefix and attach a boundary before worrying about anything else. And if the credentials go to a SaaS connector, a marketing platform such as GoHighLevel or a payments integration, you are handing keys to a system whose behavior you cannot audit. Those are the policies to write by hand.

Frequently asked questions

Are AI-generated IAM policies safe to use in production?

For data-plane permissions on named resources, generally yes, provided the identity carries a permissions boundary and the policy passes automated validation. For anything involving IAM write actions, PassRole, trust policies or key policies, treat the output as a draft and verify each statement yourself. What matters is the blast radius of the actions, not the authorship.

Why does a generated policy pass validation and still grant too much?

Validation examines one document in isolation. Privilege escalation usually comes from a combination of individually reasonable permissions, or from the interaction between the policy and something outside it, such as which roles are passable and what those roles can do. A document can be internally correct and still open a path.

What is the difference between a permissions boundary and an SCP?

Both cap permissions and neither grants any. A boundary attaches to a single IAM user or role and limits what that identity’s policies can grant it. An SCP applies to every principal in an account or organizational unit and needs AWS Organizations. With one account and no organization, boundaries are what you have. With an organization, use both.

Can IAM Access Analyzer replace writing policies by hand?

Partly. Generation from CloudTrail activity beats a guess because it reflects what the workload actually called. It is limited by the analysis window, by per-service coverage differences, by the absence of action-level detail for data events, and by iam:PassRole being excluded. Treat it as a strong draft that needs a human pass over anything the generator could not see.

Which IAM actions should I block outright in a small account?

The ones that let a principal rewrite its own permissions or borrow another identity’s: creating and setting default policy versions, attaching and putting inline policies on users, roles and groups, creating access keys for other users, and passing roles to compute services. Deny them for everything except a break-glass identity, then grant them back deliberately where a workload genuinely needs one.

How do I catch an invented action name before production?

Run a linter that carries the IAM action catalogue. Parliament flags unknown actions and exits non-zero when it finds anything, so it drops straight into a pipeline step. Catching it early matters less because the fake action is dangerous and more because the eventual fix for the denial tends to be a service-level wildcard.

The one thing worth remembering

AI-generated IAM policies are not more dangerous because a model wrote them. They are more dangerous because they arrive faster than you can think about them, and because they arrive looking finished.

So stop trying to make the review step better. Put the boundary somewhere that does not depend on anyone paying attention: a permissions boundary on every identity a pipeline can create or modify, an SCP denying the escalation actions to everyone who does not need them, and a build step that fails when a forbidden action shows up. Then let generation happen at whatever speed it wants.

The policy approved in ninety seconds is not the failure. The absence of anything underneath it was.

Need help drawing that boundary in your account?

Most of the IAM work I take on is exactly this: an account that grew organically, permissions nobody wants to touch in case something breaks, and no ceiling underneath any of it. Things I can help with:

• Auditing existing roles and policies for privilege escalation chains, including the passable-role paths that never show up in a single-document review
• Designing and attaching permissions boundaries for pipeline, workload and human identities, including the delegation pattern that lets a deploy role create roles safely
• Writing the SCP set for a small organization: escalation-relevant IAM actions, region restrictions, and protection for logging and billing resources
• Adding Access Analyzer validation and custom policy checks to a CI pipeline so forbidden actions fail the build instead of relying on review
• Replacing over-broad roles with policies generated from CloudTrail activity, then closing the gaps the generator cannot see
• Scoping down credentials handed to third-party SaaS integrations and to servers outside AWS, where your account controls do not apply

If you want a second opinion on something specific, send me the policy JSON, the output of a failing check-access-not-granted run, or the role list from your account, and I will tell you what I would change and why.

The post AI-Generated IAM Policies: Where the Hard Boundary Belongs appeared first on John Nessime.

That is the moment the region choice stops being a performance question and turns into an expensive one. Choosing an AWS region for MENA workloads looks like a latency problem and almost never is. Latency is the constraint you can measure in an afternoon and fix with a CDN. Residency is the constraint that can force you to rebuild.

This post walks through the order I work in when picking an AWS region for MENA clients: which constraints bind hard, which ones bend, how to check service availability programmatically instead of trusting a table someone screenshotted, and what tends to break after the decision is made.

Work the constraints in the order they bind

Region selection is a filter, not a scoring exercise. You are not hunting for the best average across four categories. You are eliminating regions that cannot work, in the order that matters, then picking from whatever survives.

The order I use:

1. Residency and regulatory obligations. Not engineerable. Either the law permits the data to sit there or it does not.
2. Service availability. Not engineerable cheaply. If the region lacks something your design needs, you change the design or change the region.
3. Resilience and blast radius. Partly engineerable, bounded by which regions are near enough to pair with.
4. Latency. Very engineerable. Edge caching, connection reuse and smaller payloads beat a few hundred kilometres of fibre most of the time.

Most guides run this list backwards, because latency is the fun part and the one with a nice map. Running it backwards is how you end up with a beautifully fast architecture that legal makes you move.

Constraint one: residency, and the region that does not exist

Here is the thing that catches people who have only worked with EU or US clients. In Europe you can usually satisfy a residency requirement by picking a region in the right country. Across MENA, that option frequently is not on the table, because AWS does not have an infrastructure region in most MENA countries.

Check the current region table before you promise anything. The list changes, and the geography column is what a regulator cares about, not the region’s marketing name. The programmatic version:

# List every region your account can see, with opt-in status
aws account list-regions 
  --query 'Regions[*].[RegionName,RegionOptStatus]' 
  --output table

# Confirm the physical country AWS records for a region
aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/regions/me-central-1 
  --region us-east-1 
  --output table

That second command returns the metadata AWS publishes for the region, including a geolocation country code. Better source than a blog post, this one included, because it comes from the same data the console uses.

One caution on the path itself: the global-infrastructure parameter tree is only queryable from a subset of regions, so pin --region us-east-1 on those calls rather than assuming your default profile region will work.

Announced is not the same as available

AWS announces regions well ahead of general availability, and those announcements get quoted back at you by sales teams and clients as though the capacity already exists. Treat an announcement as a roadmap item. If the region code does not appear in the table your account can query, you cannot deploy to it, and you certainly cannot write a residency clause around it. That gap matters in the Gulf specifically, where announced-but-not-yet-open regions have been part of the conversation for a while.

Residency and localisation are not the same requirement

The word “residency” gets used for at least three different obligations, and clients rarely distinguish between them. Before you look at a single region, get the client’s counsel to tell you which one applies:

• Hard localisation. The data must sit in-country, full stop. With no region in that country, no AWS region satisfies this and you are into a local provider, on-premises kit, or a hybrid split.
• Conditional transfer. Data may leave if a condition is met: regulator authorisation, an adequacy finding, explicit consent, a specific contractual basis. This is the common case, and it is more paperwork problem than architecture problem.
• Sector rules on top. Finance, health and government workloads usually add a supervisor with its own cloud rules, which may name approved jurisdictions or require notice before you move anything.

Egypt is the clearest illustration of why this matters to an engineer. Its Personal Data Protection Law and the executive regulations under it set up a regime where moving personal data out of the country generally needs a licence or permit from the supervisory authority, rather than a self-assessment you document internally and move on from. There is no AWS region in Egypt. So for an Egyptian client with personal data in scope, picking a region does not answer the residency question. It restates it: which foreign region will we be authorised to transfer into, and who is filing that application?

I am not a lawyer and neither are you. What you can do is refuse to let the region be chosen before that answer exists, because picking first and asking later is how the rebuild happens.

Enforce the decision in the account, not in the runbook

Once the approved regions are settled, stop treating them as a convention. A convention survives until someone opens the console in the wrong region at eleven at night.

The mechanism is the aws:RequestedRegion condition key, applied either as an IAM policy or, better, as a Service Control Policy across the organisation so it cannot be edited away by an account admin. AWS Control Tower ships a managed version of the same idea as its Region deny control if you are running a landing zone.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyOutsideApprovedRegions",
      "Effect": "Deny",
      "NotAction": [
        "cloudfront:*",
        "iam:*",
        "organizations:*",
        "route53:*",
        "support:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": [
            "me-central-1"
          ]
        }
      }
    }
  ]
}

The NotAction block is the part people get wrong, and it is worth understanding rather than copying. Several AWS services are global with a single control-plane endpoint that lives in us-east-1. A blanket deny on everything outside your approved region catches those calls too, and locks you out of IAM. Exempting them by action prefix is what keeps the policy usable. Extend the list to match what the client actually runs, and test it in a sandbox account before it goes anywhere near production.

One wrinkle worth checking: managed inference services can route requests to other regions within a geography to absorb load. Sensible availability feature, simultaneous residency problem. If generative AI is in scope, confirm the routing behaviour explicitly rather than assuming a request to a regional endpoint stays there.

Constraint two: service availability, where the nasty surprises live

Not every AWS service exists in every region, and the gap is widest in the newest and smallest ones. This is the failure mode that bites hardest, because it is invisible during design and obvious during implementation. The diagram gets approved, the Terraform plan runs, and one resource type throws an error that reads like a permissions problem but is really a “this does not exist here” problem.

Do not check this against the services page by eye. Check it in a script, at design time, against the same source AWS uses:

# Is a given service available in a given region?
# An empty result means no.
aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/services/athena/regions/me-central-1 
  --region us-east-1 
  --output text

# Every service AWS records for a region
aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/regions/me-central-1/services 
  --region us-east-1 
  --query 'Parameters[].Name' 
  --output text

# The canonical list of service identifiers to test against
aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/services 
  --region us-east-1 
  --query 'Parameters[].Name | sort(@)' 
  --output text

The identifiers in that tree are short API names, not display names, which trips people up. Pull the full list and grep it rather than guessing how a service name is spelled.

Two refinements make this useful rather than a party trick:

• Write the check as a matrix, not a lookup. Put the services your design depends on in a file, loop over the candidate regions, print a grid. You want the gaps visible before any infrastructure code exists.
• Presence is not feature parity. The parameter tree says a service exists in a region. It says nothing about which instance families, engine versions, storage classes or model options are offered there, and newer regions routinely carry a narrower menu. Where the specific variant matters, check that service’s own regional documentation too.

Opt-in regions and the quiet failures they cause

Every AWS region introduced in recent years, including all of the Middle East regions, is disabled by default and must be enabled on the account before you can use it. Small piece of admin, disproportionate number of confusing failures.

The failure shape is always the same: it works in your account and fails in the client’s, or works in the management account and fails in a member account created last week. Enablement is per-account and not instantaneous, so any automated account vending process has to handle it.

aws account get-region-opt-status --region-name me-central-1

Run that against every account in scope as a pre-flight check, not as a debugging step after a pipeline fails.

Constraint three: resilience and concentration risk

Every AWS region carries at least three Availability Zones, and the Middle East regions are three-AZ regions. That is enough for a properly designed multi-AZ deployment, and multi-AZ should be your default for anything the client would notice going down.

The subtlety is what you pair the region with. Cross-region DR inside MENA pairs two regions that sit closer together than a European or North American pair would, and that share more exposure to the same events: a cut on a common submarine route, a regulatory action, a physical incident. That is not an argument against a MENA pair. It is an argument for saying plainly what the DR plan protects against, because a nearby second region covers a regional AWS failure and not much wider than that.

Two practical points on AZ handling that catch people:

• AZ names are per-account. The letter suffix on a zone name does not map to the same physical zone across two different accounts. If you are coordinating placement between accounts, use the zone ID rather than the name. aws ec2 describe-availability-zones --region me-central-1 returns both.
• Capacity is not uniform. Smaller regions have less headroom in specific instance families, and you can hit an insufficient-capacity error on a launch that would have succeeded in a large region. Design for instance-family flexibility, and if the workload is steady state, talk to the client about a capacity commitment rather than discovering the ceiling during a launch window.

Constraint four: latency, finally

By the time residency and service availability have done their work you often have one candidate left, and the latency question answers itself. When you do still have a choice, measure properly.

Measuring from your own machine tells you about your own connection, which is not what you care about. Regional routing is often counter-intuitive, and traffic between two Gulf cities does not necessarily take the short path. You want a measurement from the network the users are actually on. Options, in rough order of how much I trust them:

• Real user monitoring on an existing property. If the client already has a site or app with users in that market, instrument it and read the field data. Nothing synthetic beats this.
• A small instance in each candidate region, probed from the target country. Cheap, quick, measures the actual path rather than a claim about it.
• A commercial VPN endpoint in the target country when you have no presence there. NordVPN or Surfshark will sanity-check the relative ordering of two regions, but treat the absolute numbers with suspicion: you are measuring a path through the provider’s network, not a real user’s path.
• Published latency matrices. Fine for a first pass. Not evidence.

Split the problem before you optimise the region

Most of what users experience as slowness is not round-trip time to the origin. It is uncached static assets, chatty APIs, TLS handshakes on cold connections, and payloads bigger than they need to be. A CDN fixes a large share of that whatever region the origin sits in, and the edge network is far denser across MENA than the region footprint is. Amazon CloudFront is the obvious in-stack choice; Cloudflare is worth a look when the client wants DNS, WAF and edge in one place, or when the origin is not on AWS at all.

The caveat matters though. Caching at the edge puts copies of responses in edge locations worldwide. For static assets that is fine. For authenticated responses carrying personal data it is a residency question of its own, and the answer is usually to scope caching tightly and leave dynamic personal responses uncached.

What breaks after you have chosen

A short troubleshooting list for the first few weeks, drawn from the things that reliably go wrong.

• Access denied on a service you definitely have permissions for. Check the region deny SCP first, then check whether the call is hitting a global endpoint you forgot to exempt. The error will not tell you an SCP is responsible.
• A certificate that will not attach to your distribution. Certificates used with CloudFront must be requested in us-east-1 regardless of where the origin lives. This looks like a residency violation to a nervous auditor and is not one, since a certificate holds no personal data. Have the explanation ready.
• Terraform plans that differ between engineers. Someone has a different default region in their profile. Pin it in the provider block and stop relying on environment configuration.
• An unexpected data transfer line on the bill. Cross-region traffic is charged, and replication, backups and cross-region reads all generate it quietly. Get cost allocation tags working before the first full month closes so you can attribute it rather than argue about it.
• A service that exists but behaves differently. Quotas start lower in newer regions and some features lag. Check the quota console before assuming your code is wrong.

Common mistakes

• Choosing the region on the kickoff call, before anyone has asked counsel what the transfer basis is.
• Treating proximity as the residency answer. A region in a neighbouring country is still a cross-border transfer.
• Copying a region deny policy from a blog without adapting the global-service exemptions to the client’s real service inventory.
• Checking service availability once at the start and never again as the design grows new components.
• Forgetting the observability stack. Logs and metrics carry personal data more often than people expect, and a monitoring platform hosted in another jurisdiction is a transfer. Grafana Cloud and the AWS-native options both let you choose where data lands, so choose deliberately.
• Assuming this has to be AWS at all. For a low-sensitivity workload with no compliance surface, a VPS from Contabo or InterServer may beat a thin footprint in an expensive region. Say so if it is true.

How I would pick an AWS region for MENA

1. Get the in-scope data classes in writing. Personal data, health data, financial records and anonymised telemetry carry different rules.
2. Ask counsel which residency shape applies and whether a sector supervisor sits on top. Eliminate the regions that fail it. If nothing survives, have the hybrid conversation now rather than in month four.
3. Run the availability matrix against the survivors and eliminate again. If more than one region is left, measure latency from the target country and pick.
4. Write down the choice and the reasoning, enable the region on every account in scope, and apply the guardrail that same week.

That last step is the one people skip. A region decision without a written rationale gets relitigated every time a new engineer joins, and a region decision without a guardrail quietly stops being true.

Frequently asked questions

Which AWS region is best for a client in the UAE?

If in-country storage is required, the UAE region is the only AWS option that puts data inside the country. If the requirement is softer, Bahrain is a fair alternative and has been open longer, which usually means broader service coverage. Run the availability matrix against both before committing.

Is there an AWS region in Egypt or Saudi Arabia?

Check the region table rather than trusting any article, this one included. AWS has publicly announced intent to build in the area and those announcements circulate widely, but only regions appearing in the table your account can query are deployable. Where there is no region, the real question is which nearby region you are permitted to transfer into.

Does choosing a nearby AWS region satisfy MENA data residency law?

Not by itself. Several MENA data protection regimes treat any movement of personal data across national borders as a transfer needing a legal basis, and some require prior authorisation from a regulator. Geographic closeness carries no legal weight. Get the transfer basis confirmed before the region is fixed.

How do I check whether a specific AWS service is available in a region?

Query the global infrastructure tree in Systems Manager Parameter Store. An empty result for a service and region pair means it is not offered there. It is the most reliable programmatic source, but it only tells you the service exists, not which features or instance types are available.

Why do I get access denied in a Middle East region even as an administrator?

Two likely causes: the region is opt-in and has not been enabled on that account, or an SCP is denying requests outside an approved region list. Administrator permissions do not override an SCP, which is rather the point of one.

How hard is it to change region later?

Harder than a move between accounts and much harder than people estimate. Every region-scoped identifier changes, stateful services need a data move with a cutover, and any regional endpoint baked into client configuration has to be coordinated. Budget it as a project, not a task. Which is exactly why the upfront decision deserves the care.

The one thing worth remembering

Choosing an AWS region for MENA clients is a legal decision with a technical implementation, not the reverse. Residency eliminates candidates, service availability eliminates more, latency picks between whatever survives. Run the filter in that order and you will sometimes land on a region that is not the closest one, which is fine. Run it backwards and you get a fast architecture in the wrong jurisdiction, which no amount of edge caching fixes.

Get the transfer basis in writing, enforce the decision with a guardrail rather than a convention, and record why you chose what you chose.

Need help choosing or defending a region decision?

This is the kind of work where an outside opinion early saves a rebuild later. Things I can help with:

• Running a service availability matrix across candidate regions for your actual architecture, so the gaps surface before any infrastructure code is written
• Writing and testing a region deny SCP with the global-service exemptions tuned to your service inventory, without locking anyone out
• Building a repeatable latency measurement from your target markets rather than from a vendor’s map
• Designing the CDN and caching split so static content goes to the edge and personal data does not
• Reviewing an existing deployment for accidental cross-region data flows in backups, replication, logging and monitoring
• Producing the written region rationale your client’s auditor or counsel is going to ask for

Send me the architecture diagram, a Terraform provider block, or the output of your region opt-in check, and I will tell you what I would look at first.

The post Choosing an AWS Region for a MENA Client: Latency Is the Last Question appeared first on John Nessime.