<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>AWS Organizations | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/aws-organizations/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/aws-organizations/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 07 Nov 2026 07:00:02 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>AWS Organizations | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/aws-organizations/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Your Admin Got Phished. Can They Still Delete Your AWS Backups?</title>
		<link>https://john-nessime.com/blog/cloud-security/aws-backup-vault-lock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 07 Nov 2026 07:00:02 +0000</pubDate>
				<category><![CDATA[Backup and Recovery]]></category>
		<category><![CDATA[Cloud Governance]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Backup]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Backup Retention]]></category>
		<category><![CDATA[Break-Glass Access]]></category>
		<category><![CDATA[Cross-Account Backup]]></category>
		<category><![CDATA[Disaster Recovery]]></category>
		<category><![CDATA[EBS Snapshots]]></category>
		<category><![CDATA[Immutable Storage]]></category>
		<category><![CDATA[Incident Response]]></category>
		<category><![CDATA[Logically Air-Gapped Vault]]></category>
		<category><![CDATA[Multi-Party Approval]]></category>
		<category><![CDATA[Ransomware Protection]]></category>
		<category><![CDATA[Restore Testing]]></category>
		<category><![CDATA[Service Control Policies]]></category>
		<category><![CDATA[WORM Storage]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=1134</guid>

					<description><![CDATA[<p>AWS Backup Vault Lock stops anyone deleting recovery points, root included. It doesn't stop a compromised admin deleting the KMS key, halting new backups or closing the account. Here's how to close each gap.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/aws-backup-vault-lock/">Your Admin Got Phished. Can They Still Delete Your AWS Backups?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">It usually comes up in a tabletop exercise, or right after a phishing report lands in someone&#8217;s inbox. A person asks the plain question: <em>if an attacker had our admin role for an afternoon, could they delete the backups?</em> The room goes quiet, because in most AWS accounts the answer is yes. The backups live in the same account as the workloads, and the role that can manage one can delete the other.</p>



<p class="wp-block-paragraph"><strong>AWS Backup Vault Lock</strong> is the feature built to change that answer. It puts a backup vault into a write-once, read-many state: recovery points can&#8217;t be deleted or have retention shortened, not even by root. It works. The catch is that the lock protects exactly one thing, the recovery points inside the vault, and a patient attacker with admin rights has several other ways to leave you without a usable restore.</p>



<p class="wp-block-paragraph">Below: how the two lock modes behave, the gaps a compromised admin can still use, a setup that closes them, and the problems you&#8217;ll hit once the lock is on.</p>



<h2 class="wp-block-heading">What AWS Backup Vault Lock actually protects</h2>



<p class="wp-block-paragraph">A vault lock does three things. It blocks deletion of recovery points in the vault. It blocks changes to their lifecycle, so nobody can shorten retention to make a backup expire early. And, if you set them, it enforces a minimum and maximum retention on every new backup or copy job that targets the vault. Jobs outside that range fail rather than land.</p>



<p class="wp-block-paragraph">Two details matter more than the documentation makes them look:</p>



<ul class="wp-block-list">
<li>The retention limits only apply to <strong>new</strong> jobs. Recovery points already in the vault when you lock it keep whatever lifecycle they had.</li>

<li>Each recovery point still expires on its own schedule. The lock guarantees a backup survives until its retention ends. It does not keep backups forever, and it does not keep new ones coming.</li>
</ul>



<h3 class="wp-block-heading">Governance mode vs compliance mode</h3>



<p class="wp-block-paragraph">There are two modes, and the difference is who can take the lock off.</p>



<ul class="wp-block-list">
<li><strong>Governance mode</strong> can be removed by any principal with the <code>backup:DeleteBackupVaultLockConfiguration</code> permission. It stops accidents and casual deletion. It does not stop an attacker holding a broad admin role, because that role almost certainly includes the permission.</li>

<li><strong>Compliance mode</strong> starts with a grace period (AWS calls it the cooling-off period) of at least three days. During that window you can still change or delete the lock. Once it ends, the lock is immutable. No user, no root account and not AWS can remove it or change it while the vault holds recovery points.</li>
</ul>



<p class="wp-block-paragraph">The API makes the choice in an unusual way. There&#8217;s no mode parameter. If you include <code>ChangeableForDays</code>, you get compliance mode. Leave it out and you get governance mode. That asymmetry is easy to get wrong in Terraform or CloudFormation, so read your plan output before applying.</p>



<pre class="wp-block-code"><code># Governance mode: no --changeable-for-days
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name prod-backups-locked 
  --min-retention-days 14 
  --max-retention-days 400

# Compliance mode: --changeable-for-days sets the grace period (minimum 3)
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name prod-backups-locked 
  --min-retention-days 14 
  --max-retention-days 400 
  --changeable-for-days 7</code></pre>



<p class="wp-block-paragraph">The numbers are placeholders. Minimum retention should come from how long an intrusion could go unnoticed; maximum from any rule that says data must be destroyed.</p>



<h2 class="wp-block-heading">Seven ways a compromised admin still beats a locked vault</h2>



<p class="wp-block-paragraph">A lock on the vault is necessary. It isn&#8217;t sufficient. Each of these is a separate path with its own control.</p>



<h3 class="wp-block-heading">1. The lock is in governance mode</h3>



<p class="wp-block-paragraph">If the attacker&#8217;s role can call <code>DeleteBackupVaultLockConfiguration</code>, a governance lock is one API call away from gone. Governance mode is still useful as a staging step and as protection against a tired engineer, but if your threat is a stolen admin session, it isn&#8217;t the control you think it is.</p>



<h3 class="wp-block-heading">2. The lock is still inside its grace period</h3>



<p class="wp-block-paragraph">Here&#8217;s a trap in the console and the API. <code>DescribeBackupVault</code> reports <code>"Locked": true</code> as soon as you apply a compliance lock, including during the grace period. Deletes of recovery points do fail during that time. But the lock itself can still be removed until the <code>LockDate</code> passes. A dashboard showing &#8220;Locked: true&#8221; is not proof of immutability. Check the date.</p>



<pre class="wp-block-code"><code>aws backup describe-backup-vault 
  --backup-vault-name prod-backups-locked 
  --query '{Locked:Locked,LockDate:LockDate,Min:MinRetentionDays,Max:MaxRetentionDays}'</code></pre>



<p class="wp-block-paragraph">If <code>LockDate</code> is in the future, you&#8217;re still in the window where an attacker (or you) can undo it.</p>



<h3 class="wp-block-heading">3. The attacker goes after the key, not the vault</h3>



<p class="wp-block-paragraph">This is the one that stays invisible until a restore fails. Vault Lock stops deletion of recovery points. It says nothing about the KMS keys that encrypt them. An encrypted backup whose key has been deleted is a very well-protected pile of noise.</p>



<p class="wp-block-paragraph">Two encryption behaviors matter here:</p>



<ul class="wp-block-list">
<li>Resources that support full AWS Backup management (S3, DynamoDB with advanced backup, EFS and others) are encrypted with the <strong>vault&#8217;s</strong> KMS key. Protect that key and you protect those backups.</li>

<li>Resources that don&#8217;t (EBS snapshots and RDS snapshots are the common ones) inherit the <strong>source resource&#8217;s</strong> key in the backup they create in the same account. The backup is only as safe as the key on the production volume or database.</li>
</ul>



<p class="wp-block-paragraph">A compromised admin with <code>kms:ScheduleKeyDeletion</code> can start the clock on that key. KMS enforces a waiting period before deletion, which gives you time to cancel, but only if something alerts you. <code>kms:DisableKey</code> is quieter and takes effect immediately, though it&#8217;s reversible. Put both behind explicit denies and alert on both.</p>



<h3 class="wp-block-heading">4. The attacker stops new backups and waits</h3>



<p class="wp-block-paragraph">The lock guarantees old backups survive until their retention ends. It does nothing to guarantee new backups get made. An attacker can edit the backup plan, delete a resource selection, or remove tags that a plan selects on, then simply wait. If your retention is seven days and nobody notices for eight, every clean copy has aged out on schedule, fully compliant with the lock.</p>



<p class="wp-block-paragraph">This is why minimum retention should be set against detection time, not against how much storage you&#8217;d like to pay for. It&#8217;s also why changes to backup plans need an alert, even if you allow them.</p>



<h3 class="wp-block-heading">5. The account is closed</h3>



<p class="wp-block-paragraph">If an account holding a backup vault is closed, AWS suspends it for 90 days with backups intact. If it isn&#8217;t reopened, AWS deletes the vault contents, <strong>even with a vault lock in place</strong>. The account itself is part of the attack surface.</p>



<p class="wp-block-paragraph">The practical answer is to keep the locked copies in an account whose root user and closure path are controlled by different people than your workload admins.</p>



<h3 class="wp-block-heading">6. Restores are blocked without deleting anything</h3>



<p class="wp-block-paragraph">EC2 backups are AMIs, and <code>ec2:DisableImage</code> can disable one. It stays in the locked vault, but you can&#8217;t restore from it until it&#8217;s re-enabled. Recoverable, but better found in a drill than mid-incident. Deny it alongside the backup actions.</p>



<h3 class="wp-block-heading">7. The backups were never restorable in the first place</h3>



<p class="wp-block-paragraph">Not an attack, but the same outcome. Snapshots of a database that ransomware had already encrypted are immutable garbage. Immutability protects the copy, not its usefulness. Only a timed restore tells you that.</p>



<h2 class="wp-block-heading">A setup that holds up, in the order I&#8217;d build it</h2>



<p class="wp-block-paragraph">For a small team this doesn&#8217;t need to be elaborate. Put the locked copy where the compromised role can&#8217;t reach, and make every remaining gap noisy.</p>



<ol class="wp-block-list">
<li><strong>Create a separate backup account in AWS Organizations.</strong> Workloads back up locally as they do now, then a copy rule sends recovery points to a vault in the backup account. The workload admin role has no access there.</li>

<li><strong>Give the backup vault a customer managed KMS key owned by the backup account.</strong> Cross-account copies are re-encrypted with the destination vault&#8217;s key, which breaks the dependency on the production key from gap 3. Check the source side too: backups encrypted with AWS managed keys such as <code>aws/ebs</code> or <code>aws/rds</code> can&#8217;t be copied across accounts, so those resources need a customer managed key.</li>

<li><strong>Lock the destination vault in governance mode first.</strong> Run at least one full backup and copy cycle, confirm jobs succeed with your retention settings, and do a test restore from the copy.</li>

<li><strong>Switch to compliance mode with a generous grace period.</strong> The minimum is three days. There&#8217;s no reason to use the minimum on your first lock. Put the <code>LockDate</code> in someone&#8217;s calendar, and review the configuration one more time before it passes.</li>

<li><strong>Apply service control policies</strong> that deny destructive backup and KMS actions to everyone except a named break-glass role (example below).</li>

<li><strong>Alert on the actions you allow.</strong> Plan changes, selection deletes and key state changes should page someone, because they&#8217;re exactly what gap 4 looks like.</li>

<li><strong>Schedule restore tests.</strong> AWS Backup has a restore testing feature that restores recovery points on a schedule and records how long it took. Use it, or run the restore by hand, but make it recurring.</li>
</ol>



<h3 class="wp-block-heading">The logically air-gapped vault option</h3>



<p class="wp-block-paragraph">AWS also offers a logically air-gapped vault, which bundles much of the above into one resource. It comes with a compliance-mode lock built in, uses an AWS owned key by default (a customer managed key is optional), and can be shared to other accounts through AWS RAM so you can restore from a clean account.</p>



<p class="wp-block-paragraph">Its strongest feature for this threat model is Multi-party approval. You set up an approval team in a separate recovery organization, backed by IAM Identity Center, and that team can authorize access to the vault even if the owning account is compromised or unreachable. That addresses gap 5 far better than anything you&#8217;d build by hand.</p>



<p class="wp-block-paragraph">The trade-off is setup effort and a second organization to govern. For one or two production accounts, a compliance-locked standard vault in a separate backup account is often enough. If auditors or your insurer ask about recovery from a fully compromised organization, the air-gapped vault is the cleaner answer.</p>



<h3 class="wp-block-heading">Service control policy for the backup guardrails</h3>



<p class="wp-block-paragraph">This SCP denies the actions that remove protection or quietly make backups unusable. It exempts one break-glass role. Adjust the role name to your own, and test it in a non-production OU first, because a bad SCP can lock out your own automation.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ProtectBackupsAndKeys",
      "Effect": "Deny",
      "Action": [
        "backup:DeleteBackupVault",
        "backup:DeleteBackupVaultLockConfiguration",
        "backup:DeleteRecoveryPoint",
        "backup:UpdateRecoveryPointLifecycle",
        "backup:PutBackupVaultAccessPolicy",
        "backup:DeleteBackupVaultAccessPolicy",
        "kms:ScheduleKeyDeletion",
        "kms:DisableKey",
        "ec2:DisableImage"
      ],
      "Resource": "*",
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalArn": "arn:aws:iam::*:role/BackupBreakGlass"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Two limits to know. SCPs don&#8217;t apply to the Organizations management account, which is one more reason the backups shouldn&#8217;t live there. And SCPs don&#8217;t restrict service-linked roles. Before you attach anything, paste the policy into my <a href="https://john-nessime.com/tools/aws-iam-policy-checker/">free AWS IAM Policy Checker</a> to catch wildcards and typos in action names.</p>



<p class="wp-block-paragraph">I&#8217;ve left <code>UpdateBackupPlan</code> and <code>DeleteBackupSelection</code> out of the deny on purpose. If your plans are managed through Terraform or CloudFormation, blocking them breaks your own pipeline. Restrict them to the pipeline&#8217;s role instead, and alert on every call.</p>



<h3 class="wp-block-heading">Alerting on the quiet changes</h3>



<p class="wp-block-paragraph">An EventBridge rule can match CloudTrail management events for the calls that matter. You need a CloudTrail trail recording management events for these to arrive. A pattern like this covers the backup side:</p>



<pre class="wp-block-code"><code>{
  "source": ["aws.backup"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventName": [
      "UpdateBackupPlan",
      "DeleteBackupPlan",
      "DeleteBackupSelection",
      "DeleteBackupVaultLockConfiguration",
      "PutBackupVaultLockConfiguration"
    ]
  }
}</code></pre>



<p class="wp-block-paragraph">Add a second rule with <code>"source": ["aws.kms"]</code> for <code>ScheduleKeyDeletion</code> and <code>DisableKey</code>. Route both to SNS or your paging tool, such as PagerDuty or Opsgenie. AWS Backup Audit Manager also has a control checking that resources sit in a backup plan with a Vault Lock, which catches resources nobody enrolled.</p>



<h2 class="wp-block-heading">Troubleshooting a locked vault</h2>



<h3 class="wp-block-heading">Backup jobs started failing right after the lock</h3>



<p class="wp-block-paragraph">Almost always a retention mismatch. Any job whose lifecycle is shorter than <code>MinRetentionDays</code> or longer than <code>MaxRetentionDays</code> fails. Check every backup rule and every copy rule that targets the vault, not just the obvious one.</p>



<p class="wp-block-paragraph">Continuous backups are the usual culprit. They can&#8217;t retain longer than 35 days, so a vault with a minimum retention above that will reject them. Send continuous backups to a separate vault with its own lock settings.</p>



<h3 class="wp-block-heading">Cross-account copy jobs fail</h3>



<p class="wp-block-paragraph">Check how the source is encrypted. Resources that aren&#8217;t fully managed by AWS Backup and use an AWS managed key can&#8217;t be copied to another account. You&#8217;ll need to move them to a customer managed key, which for EBS and RDS usually means copying a snapshot with a new key and restoring from it. Plan that as a change, not a quick fix.</p>



<h3 class="wp-block-heading">You can&#8217;t remove the lock</h3>



<p class="wp-block-paragraph">If <code>DeleteBackupVaultLockConfiguration</code> returns <code>InvalidRequestException</code>, the grace period has ended. There&#8217;s no support ticket that reverses it. The vault can be deleted only once it&#8217;s empty, which means waiting for every recovery point to expire.</p>



<h3 class="wp-block-heading">Storage keeps growing on a vault you want to retire</h3>



<p class="wp-block-paragraph">Look for recovery points with a retention of &#8220;Always&#8221;. AWS warns these become permanent once the grace period ends. Audit lifecycles before the lock date, not after.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating governance mode as ransomware protection when the admin role can remove it.</li>

<li>Trusting <code>"Locked": true</code> without checking whether <code>LockDate</code> has passed.</li>

<li>Locking the vault but leaving the KMS key deletable by the same role that was compromised.</li>

<li>Setting a short minimum retention to save storage, shorter than your realistic detection time.</li>

<li>Keeping the only locked copy in the same account, or in the Organizations management account.</li>

<li>Going straight to compliance mode with the three-day minimum grace period and an untested backup plan.</li>

<li>Never restoring from the locked copy, so the first real test happens during the incident.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth keeping</h2>



<ul class="wp-block-list">
<li>Keep locked copies in a dedicated backup account with its own KMS key and very few people who can reach it.</li>

<li>Keep the break-glass role&#8217;s credentials in a password manager such as 1Password or Bitwarden, behind hardware MFA keys like YubiKey.</li>

<li>Record restore times from every test, so you know your real recovery time rather than a guess.</li>

<li>Consider a third copy outside AWS entirely if a single-provider failure is in your risk register. Backblaze B2 and Wasabi both support object lock on their storage, and backup products such as Veeam and N2WS can manage that copy for you. That&#8217;s more to operate, so weigh it against what you&#8217;re protecting.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Does AWS Backup Vault Lock cost extra?</h3>



<p class="wp-block-paragraph">The lock itself has no separate charge. You pay normal AWS Backup storage, and because locked backups can&#8217;t be deleted early, you pay for their full retention.</p>



<h3 class="wp-block-heading">Can the root user delete backups in a compliance-mode locked vault?</h3>



<p class="wp-block-paragraph">No. Once the grace period ends, neither the root user nor AWS can delete recovery points or remove the lock while the vault contains recovery points. Closing the account is the exception: AWS deletes vault contents 90 days after closure if the account isn&#8217;t reopened.</p>



<h3 class="wp-block-heading">What&#8217;s the difference between governance and compliance mode?</h3>



<p class="wp-block-paragraph">Governance mode can be removed by anyone with the right IAM permission. Compliance mode becomes permanent after a grace period of at least three days. Use governance to test, compliance to protect.</p>



<h3 class="wp-block-heading">Is Vault Lock the same as S3 Object Lock?</h3>



<p class="wp-block-paragraph">No. Vault Lock covers AWS Backup vaults and their recovery points. S3 Object Lock covers objects in an S3 bucket. Neither is the older Glacier vault lock.</p>



<h3 class="wp-block-heading">Can I shorten retention after I lock the vault?</h3>



<p class="wp-block-paragraph">Not for recovery points already in a compliance-locked vault. You can change the retention in your backup plan for future backups, as long as the new value still falls within the vault&#8217;s minimum and maximum.</p>



<h3 class="wp-block-heading">Do I need a logically air-gapped vault, or is a locked standard vault enough?</h3>



<p class="wp-block-paragraph">A locked standard vault in a separate account covers most small and mid-size setups. The air-gapped vault adds Multi-party approval for recovery when the owning account or organization is compromised. If that scenario is in scope for you, it&#8217;s the better choice.</p>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">AWS Backup Vault Lock stops a compromised admin from deleting your recovery points. It doesn&#8217;t stop them from deleting the key, stopping new backups, or closing the account. Lock the vault in compliance mode, but put it in a separate account, deny key deletion, alert on plan changes, and prove the copy restores. The lock is the last line. Make sure it isn&#8217;t the only one.</p>



<p class="wp-block-paragraph">To put a number on an unrecoverable afternoon before the storage budget conversation, try the <a href="https://john-nessime.com/tools/downtime-cost-calculator/">Downtime Cost Calculator</a>.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your AWS backups?</h2>



<p class="wp-block-paragraph">This is the work I do on <a href="https://john-nessime.com/aws/backup-recovery/">AWS backup and recovery</a> engagements. I can help with:</p>



<ul class="wp-block-list">
<li>Checking if your current backups survive a compromised admin role, gap by gap as above.</li>

<li>Designing a separate backup account with cross-account copies and the KMS keys to match.</li>

<li>Moving EBS and RDS resources off AWS managed keys so their backups can be copied.</li>

<li>Writing and testing the SCPs and EventBridge alerts in a non-production OU before rollout.</li>

<li>Choosing between a locked standard vault and a logically air-gapped vault with Multi-party approval.</li>

<li>Running a timed restore from the locked copy and writing down what came back and how long it took.</li>
</ul>



<p class="wp-block-paragraph">Send me the output of <code>aws backup describe-backup-vault</code> for your main vault and a copy of one backup plan, and I&#8217;ll tell you which of the seven gaps are open.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/cloud-resilience-review/">See the Cloud Resilience Review</a></div>



<div class="wp-block-button is-style-outline is-style-outline--1"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/aws-backup-vault-lock/">Your Admin Got Phished. Can They Still Delete Your AWS Backups?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Configuration Drift Across 50 Client AWS Accounts: The Checks That Actually Fire</title>
		<link>https://john-nessime.com/blog/devops/aws-configuration-drift-detection-multi-account/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 31 Oct 2026 07:06:41 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Alert Fatigue]]></category>
		<category><![CDATA[AWS Config]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[AWS STS]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[CloudFormation]]></category>
		<category><![CDATA[Conformance Packs]]></category>
		<category><![CDATA[Confused Deputy]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[Delegated Administrator]]></category>
		<category><![CDATA[Drift Detection]]></category>
		<category><![CDATA[External ID]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Managed Service Providers]]></category>
		<category><![CDATA[Multi-Account Strategy]]></category>
		<category><![CDATA[Refresh-Only Mode]]></category>
		<category><![CDATA[Silent Failures]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[Terraform State]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=913</guid>

					<description><![CDATA[<p>Detecting configuration drift across dozens of client AWS accounts is not one tool, it is three detectors with three different blind spots. How to set up cross-account read-only access with per-client external IDs, what AWS Config, CloudFormation drift detection and refresh-only Terraform plans each fail to see, how to keep Config costs sane, and why automatic remediation is more dangerous than the drift itself.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-configuration-drift-detection-multi-account/">Configuration Drift Across 50 Client AWS Accounts: The Checks That Actually Fire</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A client emails on a Tuesday afternoon: &#8220;Our auditor is asking whether anything in the AWS account has changed since you handed it over. Can you confirm?&#8221; You built that account eleven months ago. You have the Terraform repo. You have no idea what has happened inside it since.</p>



<p class="wp-block-paragraph">Multiply that by forty-nine other clients and you have the actual problem. It is not &#8220;is there drift&#8221;, because there is always drift. It is that you cannot answer the question per account in under a minute, and you cannot tell a change someone made on purpose from a change nobody remembers making.</p>



<p class="wp-block-paragraph">This post covers how to build AWS configuration drift detection across a fleet of client accounts you do not own and cannot put into your own organization. The access model, the three detectors and what each one is blind to, how to stop the cost from running away, and why the automated fix is more dangerous than the drift.</p>



<h2 class="wp-block-heading">The failure mode: a green dashboard over an unwatched resource</h2>



<p class="wp-block-paragraph">Everything else in this post is downstream of one thing. Every drift detector on AWS reports on a subset of reality, and none of them tell you loudly where the subset ends. You get a clean report, you forward it to the client, and the thing that actually changed was never in scope.</p>



<p class="wp-block-paragraph">There are three distinct shapes of this blind spot, and they map exactly onto the three detectors:</p>



<ul class="wp-block-list">
<li><strong>The resource type is not being recorded.</strong> AWS Config only knows about resource types the recorder is configured to capture. Turn off a noisy type to save money and you have also turned off drift detection for it.</li>



<li><strong>The property is not in the template.</strong> CloudFormation drift detection compares only the resource properties explicitly written in your stack template. Rely on an AWS default and never declare it, and a change to it is invisible.</li>



<li><strong>The resource is not in state.</strong> A refresh-only Terraform plan compares state against reality. Anything a human created by hand in the console is not in state, so the plan is perfectly clean while the account fills up with unmanaged resources.</li>
</ul>



<p class="wp-block-paragraph">Once you internalise that, the design falls out of it. You do not pick a detector. You run more than one, on purpose, because each covers the other&#8217;s hole.</p>



<h2 class="wp-block-heading">Access comes first, because you do not own the organization</h2>



<p class="wp-block-paragraph">Most writing on multi-account drift assumes one AWS Organization with a delegated administrator and an organization-wide Config aggregator. That is the right answer inside a company. It is the wrong answer for consulting, because fifty clients are fifty separate organizations and none of them are joining yours.</p>



<p class="wp-block-paragraph">What you get instead is a read-only role in each client account, assumed from one account you control. The trust policy is where the security lives.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/drift-scanner" },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "unique-value-for-this-client-only"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The external ID is not decoration. AWS documents it specifically as the fix for the confused deputy problem: you hold credentials that can reach fifty client accounts, and without a per-client secret, one client who learns another client&#8217;s account ID could persuade your tooling to act on an account it should not touch. Generate a different external ID per client, store it with the client&#8217;s config, and never reuse one.</p>



<p class="wp-block-paragraph">For permissions, start with the two AWS managed policies built for exactly this. <code>arn:aws:iam::aws:policy/SecurityAudit</code> grants read access to security configuration across services. <code>arn:aws:iam::aws:policy/job-function/ViewOnlyAccess</code> adds the ability to list and describe resources generally. Between them you can run every detector below without holding a single write permission in a client account. That is worth insisting on in the contract, because it changes the conversation from &#8220;the consultant has admin&#8221; to &#8220;the consultant can look and nothing else&#8221;.</p>



<p class="wp-block-paragraph">If you want a further lock, run the scanner from a machine with a stable outbound address and add an <code>aws:SourceIp</code> condition to the trust policy. A small always-on box at a provider like Contabo or InterServer, or a fixed-IP egress from a business VPN such as NordLayer, is enough. The trade-off is real: you now have infrastructure to patch and a single point of failure. For fewer than ten accounts I would skip it. Past twenty, clients start asking for it during security review and it is easier to already have it.</p>



<h2 class="wp-block-heading">Detector one: AWS Config, and the cost trap inside it</h2>



<p class="wp-block-paragraph">AWS Config is the only one of the three that sees resources nobody wrote code for. It records a configuration item every time a supported resource changes, and it evaluates rules against those items. That makes it the right tool for the &#8220;somebody clicked something in the console&#8221; case.</p>



<p class="wp-block-paragraph">If a client does happen to run an Organization, you can enable Config as a trusted service and query everything from one place:</p>



<pre class="wp-block-code"><code># Run from the client's management account, once
aws organizations enable-aws-service-access 
  --service-principal=config.amazonaws.com

aws organizations list-aws-service-access-for-organization</code></pre>



<p class="wp-block-paragraph">Then the aggregator becomes queryable with a SQL-like expression. This is the single most useful command in the whole toolkit, because it turns &#8220;check compliance&#8221; into one call per client instead of one call per account per region:</p>



<pre class="wp-block-code"><code>aws configservice select-aggregate-resource-config 
  --configuration-aggregator-name client-aggregator 
  --expression "SELECT configuration.complianceType, COUNT(*) 
    WHERE resourceType = 'AWS::Config::ResourceCompliance' 
    GROUP BY configuration.complianceType"

# What is actually in the account, by type
aws configservice get-aggregate-discovered-resource-counts 
  --configuration-aggregator-name client-aggregator 
  --group-by-key RESOURCE_TYPE</code></pre>



<h3 class="wp-block-heading">The bill is the reason people turn it off</h3>



<p class="wp-block-paragraph">I am not going to quote rates, because AWS changes them and they vary by region. What matters is the shape of the billing, which has not changed: you pay per configuration item recorded, per rule evaluation, and per conformance pack evaluation. Three separate meters, all of which multiply by account and region.</p>



<p class="wp-block-paragraph">Two things drive the surprise bills. First, recording mode. Continuous recording captures every change as it happens; periodic recording captures a resource&#8217;s latest state once every twenty-four hours, and only when it differs from the last one recorded. Periodic costs more per item but produces far fewer items for high-churn types, so the total usually drops hard. You can set the default for the account and then override it per resource type, which is the setting most people never find.</p>



<p class="wp-block-paragraph">Second, and this is the one that catches people: compliance evaluations themselves generate configuration items. Every time a rule re-evaluates a resource and its compliance state is recorded, that is another item on the meter. In an account with conformance packs enabled across many resources, compliance records can outnumber the records of actual infrastructure changes. If a client&#8217;s Config bill looks wrong, that is the first place to look, not the resource count.</p>



<p class="wp-block-paragraph">The practical setting for consulting work: record the resource types that carry security meaning, and leave the churny ones off. Network interfaces, auto scaling lifecycle hooks and Lambda versions generate enormous item volume and almost never represent drift anyone cares about.</p>



<h2 class="wp-block-heading">Detector two: CloudFormation drift, and the property that was never declared</h2>



<p class="wp-block-paragraph">CloudFormation has native drift detection, and it is genuinely good at what it does. The call is asynchronous, so you fire it and poll:</p>



<pre class="wp-block-code"><code>ID=$(aws cloudformation detect-stack-drift 
  --stack-name client-network 
  --query StackDriftDetectionId --output text)

aws cloudformation describe-stack-drift-detection-status 
  --stack-drift-detection-id "$ID"

# Only the resources that actually moved
aws cloudformation describe-stack-resource-drifts 
  --stack-name client-network 
  --stack-resource-drift-status-filters MODIFIED DELETED</code></pre>



<p class="wp-block-paragraph">The output is the best of the three detectors. For each drifted resource you get <code>ExpectedProperties</code>, <code>ActualProperties</code>, and a <code>PropertyDifferences</code> list naming the exact path that changed and whether the difference is <code>ADD</code>, <code>REMOVE</code> or <code>NOT_EQUAL</code>. You can paste that into a client email without editing it.</p>



<p class="wp-block-paragraph">Now the blind spot, stated the way AWS states it: only resource properties explicitly defined in the stack template are checked for drift. Read that twice. If your template creates a security group and does not declare the ingress rules inline, drift detection on that stack will not tell you someone opened port 22 to the world. The stack is <code>IN_SYNC</code>. The account is not.</p>



<p class="wp-block-paragraph">The second blind spot is resource type coverage. Not every resource type supports drift detection. Types that do not are simply excluded from the results rather than flagged, and <code>describe-stack-resource-drifts</code> omits anything that was not checked. So a stack with twelve resources can return three drift records and you have no signal about the other nine unless you go looking. Compare the count of drift records against the count of stack resources on every run and alert when the gap grows.</p>



<p class="wp-block-paragraph">The other thing to know: drift detection tells you. It does not fix. Reconciling means either reverting the change by hand or bringing the current state into the template through resource import.</p>



<h2 class="wp-block-heading">Detector three: refresh-only Terraform plans, and their false positives</h2>



<p class="wp-block-paragraph">For accounts you built with Terraform or OpenTofu, the detector is already in the tool. A refresh-only plan compares state against the live provider APIs without proposing any configuration changes, and the detailed exit code turns the result into something a scheduler can branch on.</p>



<pre class="wp-block-code"><code>set +e
terraform plan -refresh-only -detailed-exitcode 
  -input=false -lock=false -no-color -out=drift.tfplan
CODE=$?
set -e

# 0 = no drift, 1 = the plan itself failed, 2 = drift detected
if [ "$CODE" -eq 2 ]; then
  terraform show -json drift.tfplan &gt; drift.json
fi</code></pre>



<p class="wp-block-paragraph"><code>-lock=false</code> matters more than it looks. This is a read-only operation running on a schedule, and you do not want a nightly drift job holding the state lock when someone needs to apply a fix. <code>-input=false</code> stops the job hanging forever on a prompt nobody is there to answer. And <code>terraform show -json</code> is what makes the result machine-readable; parsing the human-readable plan text is a trap that breaks on every minor version.</p>



<p class="wp-block-paragraph">Two honest caveats. Refresh-only plans have a history of returning exit code 2 while reporting no changes in the console output, and there are open issues against Terraform describing exactly that. If your drift alerts fire every night on an account nobody touched, check the JSON before you go hunting; you may be looking at a provider that rewrites a computed attribute on every read rather than at real drift. The second caveat is scale: this needs one plan per state file, and a client with fifteen root modules is fifteen invocations, each doing a full provider refresh. That is where a fifty-account nightly run stops being fast.</p>



<p class="wp-block-paragraph">On dedicated open-source drift tools: driftctl was the obvious answer for years, and it is worth knowing about because its model was good, comparing live resources against state and specifically surfacing unmanaged resources. It has been in maintenance mode since Snyk stepped back from active development, so treat it as a tool you can still run rather than one that will keep pace with new resource types. The commercial platforms in this space will do the fan-out for you and are a legitimate choice if you would rather buy than build. What you are paying for is the scheduling and the reporting, not the detection itself, which is the part you already have.</p>



<h2 class="wp-block-heading">Making the fan-out survivable</h2>



<p class="wp-block-paragraph">Fifty accounts is where a working script becomes an operational problem. The things that matter, in the order they bite:</p>



<ol class="wp-block-list">
<li><strong>Isolate every account.</strong> One client&#8217;s expired external ID must not stop the other forty-nine from being scanned. Catch per-account, record the failure as a result, keep going.</li>



<li><strong>Treat a failed scan as a finding.</strong> An account that has not been successfully scanned in a week is a worse state than an account with three drifted properties, because you know nothing about it. Report scan coverage before you report drift.</li>



<li><strong>Stagger, do not blast.</strong> Deep scans hit provider APIs hard and throttling is the normal outcome of running fifty in parallel. Spread them across the window and back off on retry.</li>



<li><strong>Store every run, not just the alerts.</strong> The value of drift history is the trend. &#8220;This account has drifted four times this quarter, always in the same security group&#8221; is a conversation with the client. A single alert is noise.</li>



<li><strong>Alert on new drift, not on drift.</strong> If a client accepted a manual change last month and you never reconciled the code, your job will report it every night forever until everyone stops reading the emails. Diff against the last run and only surface what is new.</li>
</ol>



<p class="wp-block-paragraph">For output, a per-client dashboard beats an inbox. Grafana Cloud or a self-hosted Grafana over a small results table is plenty; you are plotting a handful of numbers per account per day, not high-cardinality metrics. Whatever you pick, the client should be able to see their own account and nobody else&#8217;s.</p>



<h2 class="wp-block-heading">Do not let it fix things by itself</h2>



<p class="wp-block-paragraph">This is the part I would push back on hardest if a client asked for it. Automatic remediation on a drift detector means re-applying code over reality on a schedule, and the code does not know why reality changed.</p>



<p class="wp-block-paragraph">Somebody widened a timeout at two in the morning to get an incident under control. Your nightly job reverts it and the incident comes back, this time without the person who fixed it awake. Worse, some attribute changes force replacement rather than update in place, so an unattended apply can destroy a live resource in order to make it match the file.</p>



<p class="wp-block-paragraph">There are three legitimate responses to detected drift and a machine cannot choose between them: revert infrastructure to match the code, change the code to match reality, or record the difference as deliberate and exclude it from future comparisons. Detection is automatable. That decision is not. Keep the scanner on read-only credentials and let it open a pull request rather than run an apply.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>AccessDenied on assume-role for one client only.</strong> Almost always a mismatched external ID or a role ARN in the trust policy that changed when the client recreated your role. Check the exact principal ARN, not the account ID.</li>



<li><strong>Config returns no data for a region you know has resources.</strong> The recorder is per account and per region. Enabling Config in one region tells you nothing about the others, and an aggregator only aggregates regions you listed.</li>



<li><strong>Organization Config API calls fail from a member account.</strong> The organization rule and conformance pack APIs must be called from the management account or a registered delegated administrator. Calls from a member account are rejected outright.</li>



<li><strong>Drift detection returns fewer resources than the stack has.</strong> Expected. Unsupported types are not checked and not returned. Compare counts rather than assuming an empty result means clean.</li>



<li><strong>Exit code 2 every night with nothing in the plan.</strong> Read the JSON output. Either a provider is normalising an attribute on read, or a computed field is being refreshed. Both look like drift and neither is.</li>



<li><strong>The scan works manually but not on schedule.</strong> Almost always credentials. An interactive run picks up a profile from your local config that a scheduled run does not have.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Using one external ID across all clients, which removes the entire protection the external ID exists to provide.</li>



<li>Reading an empty drift report as proof of no drift, when it usually means nothing was in scope.</li>



<li>Enabling every managed rule and every conformance pack on day one, then being surprised by the evaluation charges.</li>



<li>Running the scanner with write permissions &#8220;just in case&#8221;, which turns a read-only monitoring credential into a blast radius.</li>



<li>Parsing plan text instead of JSON, which works until the day the output format changes.</li>



<li>Alerting on total drift rather than new drift, which trains everyone to ignore the alert inside a month.</li>
</ul>



<h2 class="wp-block-heading">Best practices for AWS configuration drift detection at fleet scale</h2>



<ul class="wp-block-list">
<li>Run at least two detectors per account so each covers the other&#8217;s blind spot. Config for unmanaged resources, template or state comparison for managed ones.</li>



<li>Declare security-relevant properties explicitly in templates even when the AWS default is what you want, because undeclared properties are not checked.</li>



<li>Give every client account its own external ID and rotate it when the engagement scope changes.</li>



<li>Report scan coverage alongside drift findings, so an account that silently stopped being scanned shows up as a red number.</li>



<li>Tune recording frequency per resource type rather than globally, and revisit it whenever a client&#8217;s Config bill moves.</li>



<li>Keep remediation manual and route it through code review, so the decision about why something changed is made by a person.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I detect drift across client accounts without AWS Organizations?</h3>



<p class="wp-block-paragraph">Yes, and for consulting it is the only workable model. You assume a read-only role in each client account from one account you control, guarded by a per-client external ID. You lose the organization aggregator, so you fan out and aggregate the results yourself, but every individual API call works the same way.</p>



<h3 class="wp-block-heading">Is AWS Config or CloudFormation drift detection better?</h3>



<p class="wp-block-paragraph">They answer different questions. CloudFormation tells you which declared properties of a managed stack no longer match the template, with exact before and after values. Config tells you what exists and whether it satisfies a rule, including resources no template ever created. If you only run one, run Config, because unmanaged resources are the bigger risk in an account you did not watch for a year.</p>



<h3 class="wp-block-heading">How often should drift detection run?</h3>



<p class="wp-block-paragraph">Nightly for the full fleet is the sensible default, because the cost is time rather than money and the alternative is finding out at audit. Tighten to hourly only for the narrow set of resources where a change is an incident, typically identity and network exposure. Everything else can wait until morning.</p>



<h3 class="wp-block-heading">Does terraform plan detect drift on resources created outside Terraform?</h3>



<p class="wp-block-paragraph">No. A plan compares state against reality, and a console-created resource is not in state, so it does not appear. This is the single most common misunderstanding about Terraform drift detection and the reason a clean plan is not a clean account.</p>



<h3 class="wp-block-heading">How do I keep AWS Config costs under control across many accounts?</h3>



<p class="wp-block-paragraph">Restrict the recorded resource types, set high-churn types to periodic recording, and be selective about rules and conformance packs rather than enabling everything. Remember that compliance evaluations create billable configuration items of their own, so adding rules increases two meters, not one.</p>



<h3 class="wp-block-heading">What permissions does a drift scanner actually need?</h3>



<p class="wp-block-paragraph">Read only. The <code>SecurityAudit</code> and <code>ViewOnlyAccess</code> managed policies cover the detectors described here. Anything that wants write access is doing remediation, which should be a separate role with a separate approval path.</p>



<h3 class="wp-block-heading">Should I buy a drift detection platform instead?</h3>



<p class="wp-block-paragraph">If the fan-out and the reporting are what you are short on, buying is defensible, and the good platforms connect drift to security context rather than just reporting divergence. What you should not buy is detection alone. The commands above are the detection, they are free, and a platform that wraps them still has the same three blind spots.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The hard part of AWS configuration drift detection across a client fleet is not running the commands. It is knowing what each command cannot see, and building the fleet view so a silent gap looks different from a clean result.</p>



<p class="wp-block-paragraph">If you remember one thing: an empty drift report and a healthy account are not the same claim, and until you can tell them apart per account, you cannot answer the client&#8217;s Tuesday email.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need this built across your client accounts?</h2>



<p class="wp-block-paragraph">Multi-account drift detection is most of a week of fiddly work the first time and an afternoon every time after. If you would rather have it running than build it, this is the kind of thing I do:</p>



<ul class="wp-block-list">
<li>Designing the cross-account read-only role and per-client external ID scheme, including the CloudFormation or Terraform your clients can review before they deploy it</li>



<li>Building the scheduled fan-out scanner with per-account isolation, throttling and scan-coverage reporting</li>



<li>Tuning AWS Config recording scope and frequency so the bill matches what the client is actually getting from it</li>



<li>Auditing existing CloudFormation stacks for undeclared properties that make drift detection report clean when it should not</li>



<li>Wiring drift results into a Grafana dashboard or a per-client report your account managers can send without editing</li>



<li>Sorting out the noisy false positives so the alerts stay worth reading after month two</li>
</ul>



<p class="wp-block-paragraph">If you already have something running, send me a scan output or the trust policy you are using and I will tell you what it is missing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-configuration-drift-detection-multi-account/">Configuration Drift Across 50 Client AWS Accounts: The Checks That Actually Fire</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AWS Cost Allocation Tags: Getting Real Coverage on a Messy Bill</title>
		<link>https://john-nessime.com/blog/technical-guides/aws-cost-allocation-tags-coverage/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 15 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Config]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Backfill]]></category>
		<category><![CDATA[Cost Allocation Tags]]></category>
		<category><![CDATA[Cost and Usage Report]]></category>
		<category><![CDATA[Cost Explorer]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Multi-Account Strategy]]></category>
		<category><![CDATA[Service Control Policies]]></category>
		<category><![CDATA[Showback]]></category>
		<category><![CDATA[Tag Policies]]></category>
		<category><![CDATA[Terraform]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=519</guid>

					<description><![CDATA[<p>Tagging a resource and activating a cost allocation tag are two different operations, and AWS never tells you when you have only done the first. A practical guide to the activation gap, what backfill can and cannot repair, the five places coverage leaks, and a retrofit order that actually converges.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/aws-cost-allocation-tags-coverage/">AWS Cost Allocation Tags: Getting Real Coverage on a Messy Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">You tagged everything. The Terraform provider block carries <code>default_tags</code> with <code>Project</code> and <code>Environment</code>, every module inherits them, and you can open the console right now and see those tags sitting on the instances.</p>



<p class="wp-block-paragraph">Then someone in finance asks what the reporting pipeline costs. You open Cost Explorer, group by <code>Project</code>, and over half the bill lands in a bucket called &#8220;No tag key&#8221;.</p>



<p class="wp-block-paragraph">Nothing is broken. The tags are there. They just aren&#8217;t on the bill, because tagging a resource and activating a cost allocation tag are two separate operations, and AWS never warns you when you&#8217;ve done only the first.</p>



<p class="wp-block-paragraph">This post covers getting AWS cost allocation tags to cover a messy account: the activation gap, what backfill will and won&#8217;t repair, where coverage leaks even with decent discipline, and a retrofit order that converges.</p>



<h2 class="wp-block-heading">A resource tag and a cost allocation tag are not the same thing</h2>



<p class="wp-block-paragraph">A resource tag is metadata on the resource. A cost allocation tag is a billing dimension. Getting the second means taking a key that already exists on your resources and explicitly switching it on in the Billing and Cost Management console.</p>



<p class="wp-block-paragraph">Two details make this bite. First, activation lives in the management account. From a member account, the page tells you tags are managed at the payer level and that&#8217;s the end of it. Plenty of teams have a good tagging standard and no access to the one screen that makes it count. Related trap: an account moving between organizations has its activated keys reset to inactive.</p>



<p class="wp-block-paragraph">Second, the lag. AWS documents up to 24 hours for a newly applied key to appear on the activation page, and up to another 24 hours after you activate before it takes effect. The loop between &#8220;we agreed a standard&#8221; and &#8220;I can see it in Cost Explorer&#8221; is two days at best.</p>



<p class="wp-block-paragraph">Past a handful of keys, drive it from the API. Listing the inactive ones is the fastest way to find what you forgot:</p>



<pre class="wp-block-code"><code># Every key AWS knows about that is NOT currently billing-active
aws ce list-cost-allocation-tags --status Inactive

# Switch a set of them on
aws ce update-cost-allocation-tags-status --cli-input-json '{
  "CostAllocationTagsStatus": [
    {"TagKey": "Project",     "Status": "Active"},
    {"TagKey": "Environment", "Status": "Active"},
    {"TagKey": "Owner",       "Status": "Active"}
  ]
}'</code></pre>



<p class="wp-block-paragraph">User-defined keys appear in the report with a <code>user:</code> prefix and AWS-generated ones carry the reserved <code>aws:</code> prefix, so they never collide. There&#8217;s a cap on active keys, raisable through Service Quotas, which you almost certainly don&#8217;t need to raise.</p>



<h2 class="wp-block-heading">What the backfill repairs, and what it quietly doesn&#8217;t</h2>



<p class="wp-block-paragraph">Activation used to be strictly forward-looking, so activating late permanently orphaned everything before it. That&#8217;s no longer true. The management account can ask AWS to reapply current activation status across up to twelve months of historical cost data.</p>



<pre class="wp-block-code"><code># Must be the first day of a month, no earlier than twelve months back
aws ce start-cost-allocation-tag-backfill 
  --backfill-from 2025-01-01T00:00:00Z</code></pre>



<ul class="wp-block-list">
<li>Start date must be the first day of a month, and can&#8217;t reach back more than twelve months.</li>

<li>One request per 24 hours, and you can&#8217;t submit another while one is processing.</li>

<li>It backfills <em>activation status</em>, not tag values.</li>
</ul>



<p class="wp-block-paragraph">That last point catches people. If the resource wasn&#8217;t carrying the tag when the cost was incurred, backfill gives you nothing for those months. It fixes &#8220;I forgot to activate.&#8221; It does not fix &#8220;I forgot to tag.&#8221; A tag applied in June and activated in November backfills to June and no further, whatever start date you hand it.</p>



<p class="wp-block-paragraph">It also cuts both ways. Backfill applies the <em>current</em> status of every key, so a key sitting inactive right now gets stripped out of historical data too. If you&#8217;re pruning an overgrown tag list, backfill before you deactivate the old keys, not after, or you&#8217;ll erase the history you were trying to keep.</p>



<h2 class="wp-block-heading">Where AWS cost allocation tags actually leak coverage</h2>



<p class="wp-block-paragraph">With activation sorted and every Terraform-managed resource tagged, a stubborn slice of the bill still refuses to attribute. It matters which slice, because the fixes have nothing in common.</p>



<h3 class="wp-block-heading">Charges no resource can carry</h3>



<p class="wp-block-paragraph">Support charges, refunds, credits, tax, some Marketplace subscriptions. There&#8217;s no resource to hang a tag on, so tagging discipline never touches them. On accounts with a large support plan or heavy credit activity, that&#8217;s a visible fraction of the invoice.</p>



<p class="wp-block-paragraph">The answer is account tags. AWS added support for using AWS Organizations account tags as a cost allocation dimension, applying to all metered usage inside the tagged account and flowing into Cost Explorer, Budgets, Cost Anomaly Detection and Cost Categories. Because attribution happens at the account level, it picks up exactly what resource tags can&#8217;t reach.</p>



<p class="wp-block-paragraph">This is recent enough that most tagging guides predate it, and it&#8217;s the highest-value item here if your accounts already map roughly onto teams.</p>



<h3 class="wp-block-heading">Resources created by other resources</h3>



<p class="wp-block-paragraph">This is the quiet one, and usually the biggest surprise gap in an otherwise well-run account. Terraform&#8217;s provider-level <code>default_tags</code> applies to resources Terraform creates. It does not reach the EC2 instances an Auto Scaling group launches, because Terraform didn&#8217;t create those, the ASG did. Your ASG is tagged. Your instances aren&#8217;t. Same for volumes attached at launch, ENIs, snapshots and AMIs.</p>



<pre class="wp-block-code"><code>data "aws_default_tags" "current" {}

resource "aws_autoscaling_group" "app" {
  # ... other config ...

  # Push provider default tags down to launched instances
  dynamic "tag" {
    for_each = data.aws_default_tags.current.tags
    content {
      key                 = tag.key
      value               = tag.value
      propagate_at_launch = true
    }
  }
}

resource "aws_launch_template" "app" {
  # ... other config ...

  tag_specifications {
    resource_type = "instance"
    tags          = data.aws_default_tags.current.tags
  }

  tag_specifications {
    resource_type = "volume"
    tags          = data.aws_default_tags.current.tags
  }
}</code></pre>



<p class="wp-block-paragraph">For a standalone instance, <code>volume_tags</code> on <code>aws_instance</code> covers its attached volumes. General rule: whenever one AWS resource provisions another on your behalf, assume tags don&#8217;t propagate until you&#8217;ve checked the created resource in the console, not the plan output.</p>



<h3 class="wp-block-heading">Shared infrastructure that belongs to everyone</h3>



<p class="wp-block-paragraph">NAT gateway data processing, VPC interface endpoints, a Transit Gateway, a shared load balancer, the central logging bucket. You can tag all of these perfectly and the tag says <code>Project=platform</code>, which is accurate and useless for chargeback.</p>



<p class="wp-block-paragraph">Transit Gateway is worth calling out because it supports tagging the gateway resource itself, so data processing and transfer charges pick up the tag rather than only attachment hours. That gap used to swallow a lot of network spend.</p>



<p class="wp-block-paragraph">Beyond that, stop chasing tags and do allocation math instead. Cost Categories support split charge rules that redistribute a shared bucket across your other categories, using proportional splits, fixed percentages or an even division. Proportional is the sane default; fixed percentages are for when finance already agreed a number and the report has to match it.</p>



<h3 class="wp-block-heading">Container spend hiding behind one instance tag</h3>



<p class="wp-block-paragraph">A shared EKS or ECS cluster is one tagged compute footprint running fifteen teams&#8217; workloads. The instance tag gives you the total and nothing about who caused it. Split cost allocation data is the native answer: opt in and the cost and usage export gains rows breaking cluster cost down to individual pods or tasks. It won&#8217;t account for idle headroom you&#8217;re deliberately carrying, but it beats one line item labelled &#8220;the cluster&#8221;.</p>



<h3 class="wp-block-heading">Case drift and near-duplicate keys</h3>



<p class="wp-block-paragraph">Keys and values are case-sensitive. <code>Environment</code>, <code>environment</code> and <code>ENV</code> are three distinct keys producing three columns, each holding a partial and misleading picture. Tag Policies in AWS Organizations exist for this: define the key including its capitalization, define permitted values, and get an org-wide compliance report. Cheapest governance on this list, because it costs nobody a deploy.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A retrofit order that converges</h2>



<p class="wp-block-paragraph">Retrofits stall because people start at the bottom, sweeping untagged resources by hand while the creation path keeps producing more. This order avoids that.</p>



<ol class="wp-block-list">
<li><strong>Measure first.</strong> Group Cost Explorer by your intended key and record what percentage of spend falls into &#8220;No tag key&#8221;. That&#8217;s your baseline and your only honest progress metric.</li>

<li><strong>Cut the key set down.</strong> Three to five keys: owner, environment, and a project or cost centre. Coverage on five beats partial coverage on thirty.</li>

<li><strong>Activate immediately, even with terrible coverage.</strong> Both clocks start now: the activation delay and the rolling twelve-month backfill window. There&#8217;s no downside to activating a key that&#8217;s barely applied.</li>

<li><strong>Rank the gap by spend, not resource count.</strong> Sort untagged line items by cost descending. It&#8217;s usually a few NAT gateways, a couple of oversized databases and one forgotten bucket. Ten thousand untagged security groups cost nothing.</li>

<li><strong>Fix the creation path before the backlog.</strong> IaC defaults, launch template tag specifications, ASG propagation. Sweeping first means sweeping again next month.</li>

<li><strong>Sweep the stragglers in bulk.</strong> The Resource Groups Tagging API works across services and regions, which beats going service by service.</li>

<li><strong>Backfill, then re-measure.</strong> Compare against step one. If the number barely moved, you fixed the wrong resources.</li>
</ol>



<pre class="wp-block-code"><code># Find what already carries a Project tag, to diff against inventory
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project

# Apply tags to a specific set of ARNs
aws resourcegroupstaggingapi tag-resources 
  --resource-arn-list arn:aws:rds:eu-west-1:111122223333:db:reporting-01 
  --tags Project=reporting,Environment=prod</code></pre>



<p class="wp-block-paragraph">Run it against a couple of ARNs and confirm in the console before looping over a list. Bulk tagging is trivially easy to point at the wrong account.</p>



<h2 class="wp-block-heading">Enforcing tags without becoming the tag police</h2>



<p class="wp-block-paragraph">Four mechanisms, in increasing order of blast radius. Most teams need the first three and should be careful with the fourth.</p>



<ul class="wp-block-list">
<li><strong>IaC defaults.</strong> Provider-level default tags plus required module variables. Cheapest, catches the majority, nobody thinks about it.</li>

<li><strong>CI validation.</strong> A linter rule that fails the pull request on missing required tags. The failure lands on the person who can fix it, when they can fix it.</li>

<li><strong>Detective controls.</strong> The AWS Config managed rule for required tags flags non-compliant resources without blocking anything. Tells the truth, breaks nothing.</li>

<li><strong>Preventive controls.</strong> A service control policy denying creation when a required key is absent, keyed on a null check against the request tag.</li>
</ul>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyRDSCreateWithoutProjectTag",
      "Effect": "Deny",
      "Action": "rds:CreateDBInstance",
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/Project": "true"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The honest trade-off: service control policies are where tagging programmes go to die. Not every API supports tag-on-create, and not every legitimate workflow passes tags through the way you expect. Roll a broad deny across the org on day one and you break somebody&#8217;s deployment on day two, the ticket lands on you, and the policy is gone by day three along with everyone&#8217;s appetite for the project.</p>



<p class="wp-block-paragraph">Scope it to one service at a time, starting in sandbox. I&#8217;d rather run Config plus Tag Policies for a quarter and fix the top three offenders in the IaC. The deny is a finishing move, not an opening one.</p>



<h2 class="wp-block-heading">Measuring coverage so it doesn&#8217;t rot</h2>



<p class="wp-block-paragraph">Cost Explorer gives the headline number. For a per-service breakdown, query the cost and usage export in Athena. One caveat: tag column naming differs between export versions, older ones giving a flattened column per activated key and newer formats a map column. Run a <code>DESCRIBE</code> first rather than guessing.</p>



<pre class="wp-block-code"><code>-- Confirm the actual column names first
DESCRIBE your_cur_database.your_cur_table;

-- Then: untagged spend, ranked by service
SELECT   line_item_product_code,
         SUM(line_item_unblended_cost) AS untagged_cost
FROM     your_cur_database.your_cur_table
WHERE    your_project_tag_column = ''
  AND    line_item_unblended_cost &gt; 0
GROUP BY 1
ORDER BY 2 DESC
LIMIT    20;</code></pre>



<p class="wp-block-paragraph">That ranked list is your work queue; everything below the top twenty is noise. The list call also returns a last-used timestamp per key, the cleanest way to find keys nobody has applied to anything in months.</p>



<p class="wp-block-paragraph">If you&#8217;d rather not maintain the dashboard yourself, this is a fair place to buy rather than build. Grafana Cloud sits on Athena comfortably if you already run Grafana, and platforms like CloudZero, Vantage and nOps handle allocation modelling out of the box. Their real value is the shared-cost and commitment maths. Their real cost is another vendor between you and your billing data, and several price against a percentage of the spend they watch, so run that number first.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>The key isn&#8217;t in the activation list.</strong> It must exist on at least one resource first, and propagation takes up to a day. Past that, check you&#8217;re in the management account and the key matches exactly, including case.</li>

<li><strong>Activated, but everything still shows untagged.</strong> Give it a full billing cycle. A key switched on mid-month leaves the first half of that month untagged until you backfill.</li>

<li><strong>Tags stopped working after an account move.</strong> Activation doesn&#8217;t survive a move between organizations. Reactivate from the new management account, then backfill.</li>

<li><strong>Cluster costs still land on one team.</strong> Instance tags can&#8217;t split a shared cluster. Opt into split cost allocation data, or treat the cluster as a shared cost in a split charge rule.</li>

<li><strong>You&#8217;ve hit the active key limit.</strong> Almost always too many keys rather than a real need for a quota increase. Use last-used timestamps to find dead ones, and backfill before deactivating.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Assuming a tagged resource is a tagged cost. It isn&#8217;t until you activate the key.</li>

<li>Waiting for good coverage before activating. Activate first, improve coverage second, backfill third.</li>

<li>Deactivating stale keys and then backfilling, which wipes their history out of the data.</li>

<li>Measuring coverage by resource count instead of by spend.</li>

<li>Rolling out a broad tag-enforcing SCP in production before anyone has read a compliance report.</li>

<li>Trusting <code>default_tags</code> to reach resources Terraform didn&#8217;t directly create.</li>

<li>Putting anything sensitive in a tag value. Tags surface in billing exports that finance and third-party tools read.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep the mandatory set to three to five keys, with defined allowed values rather than free text.</li>

<li>Pin casing in the standard and enforce it with Tag Policies, because AWS won&#8217;t do it for you.</li>

<li>Tag accounts as well as resources. Account tags cover charges resource tags structurally cannot.</li>

<li>Treat coverage as a tracked percentage of spend with an owner and a monthly trend, not a project that finishes.</li>

<li>Accept an irreducible remainder. Chasing the last few percent by hand costs more than it returns.</li>

<li>Put allocation rules in Cost Categories rather than inventing tag values that encode a split.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Are AWS cost allocation tags retroactive?</h3>



<p class="wp-block-paragraph">Partly. Activation applies going forward, but the management account can request a backfill reapplying current activation status across up to twelve months. The catch is that the resource must have carried the tag at the time, so backfill recovers late activation, never late tagging.</p>



<h3 class="wp-block-heading">Why don&#8217;t my tags appear in Cost Explorer?</h3>



<p class="wp-block-paragraph">In order of likelihood: the key was never activated in the Billing console; you&#8217;re looking from a member account; the activation delay hasn&#8217;t elapsed; or the key differs in case from the one you activated.</p>



<h3 class="wp-block-heading">How many cost allocation tags can I activate?</h3>



<p class="wp-block-paragraph">There&#8217;s a documented per-account cap on active keys, raisable through Service Quotas, plus a separate per-resource cap on total tags. Check current numbers in the AWS billing quotas docs. In practice the binding limit is human: past roughly a dozen keys, consistency collapses.</p>



<h3 class="wp-block-heading">Should I use account tags or resource tags?</h3>



<p class="wp-block-paragraph">Both, for different jobs. Account tags handle top-level attribution and cover untaggable charges like support, credits and refunds. Resource tags handle granularity inside an account. If your accounts already map cleanly to teams, account tags are the cheaper win and worth doing first.</p>



<h3 class="wp-block-heading">Is this worth doing on a small account?</h3>



<p class="wp-block-paragraph">Scale the effort to the shape of the bill. If you run a handful of services under one account and can name every resource from memory, account tags plus two resource keys is plenty. Worth noticing too that a fixed-price VPS from a provider like Contabo or InterServer has trivially perfect cost allocation, because the bill is one predictable line. Elastic infrastructure buys flexibility and pays for it in attribution work.</p>



<h3 class="wp-block-heading">How do I allocate shared costs like NAT gateways?</h3>



<p class="wp-block-paragraph">Not with tags. Put them in a Cost Categories split charge rule and redistribute proportionally across consuming categories. Tag the shared resource to identify the bucket, then let the rule do the allocation.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">AWS cost allocation tags fail silently, and that&#8217;s what makes them frustrating. Nothing errors. No alarm fires. Your resources look correctly tagged in every console you check, the bill quietly disagrees, and you find out when someone asks a question you can&#8217;t answer.</p>



<p class="wp-block-paragraph">So check one thing now: open the cost allocation tags page in your management account and see whether the keys you&#8217;ve been diligently applying are actually switched on. If they aren&#8217;t, activate them today, then backfill. That twelve-month window rolls, and every month you wait is history you can&#8217;t recover.</p>



<p class="wp-block-paragraph">Everything else here is incremental. Coverage is a percentage you improve, not a state you reach.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help getting your AWS bill to explain itself?</h2>



<p class="wp-block-paragraph">Cost attribution is unglamorous and easy to postpone until the quarter someone actually needs the numbers. If your account is already messy, it goes faster with someone who has untangled one before. I help with:</p>



<ul class="wp-block-list">
<li>Auditing tag coverage as a percentage of spend, broken down by service, so you know what you&#8217;re actually fixing</li>

<li>Designing a minimal tag schema and the Tag Policies to keep it consistent across an organization</li>

<li>Closing propagation gaps in Terraform and CloudFormation, around Auto Scaling groups, launch templates and attached storage</li>

<li>Setting up Cost Categories with split charge rules so shared infrastructure stops distorting every team&#8217;s number</li>

<li>Building cost and usage export pipelines in Athena, partitioned so the dashboard isn&#8217;t expensive to load</li>

<li>Planning a staged tag enforcement rollout that doesn&#8217;t break anyone&#8217;s deployment pipeline</li>
</ul>



<p class="wp-block-paragraph">Send me a Cost Explorer screenshot grouped by your main tag key, or the output of a list call against your active tags, and I&#8217;ll tell you where the gap actually is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/aws-cost-allocation-tags-coverage/">AWS Cost Allocation Tags: Getting Real Coverage on a Messy Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Amazon Macie PII Detection: The Buckets It Never Opened</title>
		<link>https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 06 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Data Firehose]]></category>
		<category><![CDATA[Amazon Macie]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Classification]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[Security Hub]]></category>
		<category><![CDATA[Sensitive Data Discovery]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=405</guid>

					<description><![CDATA[<p>A Macie bucket labeled "Not sensitive" often just means Macie never read it. Extensionless objects, unsupported storage classes, unreachable KMS keys and quota truncation all produce silence that looks identical to a clean result. Here's how to measure coverage, fix the four gaps, tune identifiers, and keep the bill honest.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email came from legal, not from security. A customer had exercised a data access request, someone pulled the export by hand, and it contained email addresses and phone numbers sitting in a bucket Macie had labeled <em>Not sensitive</em>. Nobody had done anything wrong. The console was telling the truth as it understood it. It just didn&#8217;t understand very much about that bucket.</p>



<p class="wp-block-paragraph">That is the failure mode worth internalizing before anything else: Amazon Macie PII detection reports on what it managed to read, not on what is there. Objects it never opened produce no findings, and no findings looks identical to a clean result. The gap between &#8220;we scanned this&#8221; and &#8220;we found nothing&#8221; is where most Macie deployments quietly fail their first audit.</p>



<p class="wp-block-paragraph">This post covers where that gap comes from and how to close it: the eligibility chain every object goes through, the four distinct reasons an object gets skipped, how to tune identifiers so the findings are worth reading, what actually drives the bill, and how to route results somewhere a human will act on them.</p>



<h2 class="wp-block-heading">The eligibility chain behind Amazon Macie PII detection</h2>



<p class="wp-block-paragraph">Every object passes the same gates before a byte gets inspected. The order matters, because each gate fails differently and needs a different fix.</p>



<ol class="wp-block-list">
<li>Is it in a general purpose S3 bucket? Directory buckets are out of scope entirely.</li>



<li>Is the storage class supported? Standard, Standard-IA, One Zone-IA, Intelligent-Tiering, Glacier Instant Retrieval and Reduced Redundancy are in. Glacier Deep Archive and S3 Express One Zone are not.</li>



<li>Does the object key carry a recognized file extension? This is a string check on the name, not a content sniff.</li>



<li>Can Macie retrieve and decrypt it, given the bucket policy, object ACL and encryption key?</li>



<li>Does the content parse, and stay inside the per-file quotas?</li>



<li>Do the active data identifiers match anything in it?</li>
</ol>



<p class="wp-block-paragraph">Only the last gate produces a finding. Everything before it produces silence. Macie does record why each object was skipped, but that lives in coverage data and object samples, not on the findings page most people look at.</p>



<p class="wp-block-paragraph">One detail explains a lot of confusion: when you first enable automated sensitive data discovery, every bucket gets a sensitivity score of 50 and the label <em>Not yet analyzed</em>. A bucket whose permissions block Macie stays there permanently. It never turns red. It sits mid-list, looking unremarkable next to buckets that genuinely were analyzed.</p>



<h2 class="wp-block-heading">Gap one: objects Macie never opened</h2>



<p class="wp-block-paragraph">Macie calls these <em>unclassifiable</em>. They fail the storage class or extension check, so no retrieval is attempted. This is the largest source of false confidence I&#8217;ve seen, and the cheapest to fix.</p>



<h3 class="wp-block-heading">Missing file extensions</h3>



<p class="wp-block-paragraph">Classifiability comes from the file name extension. A file full of customer records named <code>part-00003-a4f9</code> is invisible. The same bytes named <code>part-00003-a4f9.json</code> get inspected and produce findings.</p>



<p class="wp-block-paragraph">This bites hardest on streaming ingestion. Amazon Data Firehose writes to S3 without appending an extension unless compression, format conversion, or the explicit file extension setting adds one. An uncompressed JSON stream lands as a tree of extensionless objects. Athena and Glue read them fine, because they infer format from the table definition. Macie skips every one.</p>



<p class="wp-block-paragraph">Fix it upstream: set the S3 file extension on the delivery configuration, or enable GZIP compression or Parquet conversion, both of which append a recognized extension. If you can&#8217;t change the producer, the fallback is a copy-and-rename step into a scan prefix, which costs you storage plus a pipeline to maintain. Fixing the producer is almost always cheaper.</p>



<h3 class="wp-block-heading">Unsupported storage classes</h3>



<p class="wp-block-paragraph">A lifecycle rule transitioning old exports to Glacier Deep Archive quietly removes them from scope. That may be the right storage decision, but it needs to be a conscious one, because archived customer data is still customer data when a regulator asks.</p>



<p class="wp-block-paragraph">Macie exposes these counts per bucket, split by cause. This is the first query I run against a new account:</p>



<pre class="wp-block-code"><code># Buckets where Macie can't classify objects, broken down by cause
aws macie2 describe-buckets 
  --query 'buckets[?unclassifiableObjectCount.total &gt; `0`].{
      bucket: bucketName,
      classifiable: classifiableObjectCount,
      skipped_total: unclassifiableObjectCount.total,
      skipped_extension: unclassifiableObjectCount.fileType,
      skipped_storage_class: unclassifiableObjectCount.storageClass
    }' 
  --output table</code></pre>



<p class="wp-block-paragraph">Read the ratio, not the raw number. Forty thousand classifiable objects and twelve skipped is fine. Three classifiable and four hundred thousand skipped is a reporting artifact pretending to be a scan result, and that bucket&#8217;s sensitivity score means nothing.</p>



<h2 class="wp-block-heading">Gap two: objects it opened and couldn&#8217;t read</h2>



<p class="wp-block-paragraph">These are classification errors rather than eligibility failures. Macie selected the object, tried to fetch it, and was refused. Three causes, and they need three different people to fix them.</p>



<ul class="wp-block-list">
<li><strong>Customer-provided keys (SSE-C).</strong> Macie cannot supply the key material, so it cannot retrieve the object. No permissions fix exists. Re-encrypt with S3 managed or KMS keys if you want coverage.</li>



<li><strong>KMS key policy.</strong> For customer managed keys, the key policy must allow the Macie service-linked role, <code>AWSServiceRoleForAmazonMacie</code>, to decrypt. Cross-account buckets need the key owner to grant it, not the bucket owner. AWS publishes a permission analyzer script in the <code>aws-samples/amazon-macie-scripts</code> repository that enumerates every key Macie needs and generates the CLI commands to fix them.</li>



<li><strong>Restrictive bucket policies.</strong> An explicit <code>Deny</code> conditioned on source IP or VPC endpoint blocks Macie along with everything else. The working pattern excludes the service-linked role ARN from the deny using the <code>aws:PrincipalArn</code> condition key.</li>
</ul>



<p class="wp-block-paragraph">To see which objects Macie actually touched in a bucket, pull the object samples. It&#8217;s the closest thing to a scan log you get:</p>



<pre class="wp-block-code"><code># Objects automated discovery selected, with per-object status
aws macie2 list-resource-profile-artifacts 
  --resource-arn arn:aws:s3:::your-bucket-name</code></pre>



<p class="wp-block-paragraph">Anything with a status of <code>SKIPPED</code> is a lead. Take the key, check its encryption settings in S3, and you usually have your answer inside a minute.</p>



<h2 class="wp-block-heading">Gap three: it read the file and nothing matched</h2>



<p class="wp-block-paragraph">Macie ships managed data identifiers covering common PII, financial data and credentials across many countries. Automated discovery uses a recommended subset by default rather than all of them. Sensible, but it is a default, and it will miss things that matter to you.</p>



<p class="wp-block-paragraph"><strong>Add the managed identifiers for your actual jurisdictions.</strong> If you hold records for customers in a country whose national ID identifier isn&#8217;t in the recommended set, you&#8217;re scanning for the wrong things with perfect efficiency. Pull the current list with <code>aws macie2 list-managed-data-identifiers</code> and compare it against where your customers live.</p>



<p class="wp-block-paragraph"><strong>Write custom identifiers for what only you know is sensitive.</strong> Internal account numbers, case references, employee IDs. A custom identifier is a regex plus optional keywords and a proximity rule, and the keywords are what save you. A bare <code>[0-9]{8}</code> matches timestamps, order totals and row counts, and you drown. The same regex with keywords and a match distance only fires when the number sits near a word that gives it meaning.</p>



<p class="wp-block-paragraph">Test the pattern before it goes near a job. This runs the criteria against sample text and returns match counts without creating anything:</p>



<pre class="wp-block-code"><code># Dry-run a custom identifier before creating it
aws macie2 test-custom-data-identifier 
  --regex 'ACC-[0-9]{4}-[0-9]{4}' 
  --keywords account customer acct 
  --maximum-match-distance 30 
  --sample-text 'customer account ACC-4821-9930 updated; total 1234-5678'</code></pre>



<p class="wp-block-paragraph">Run it against real samples, including the awkward ones, then against a file you know is clean and confirm zero. A pattern you only tested on positives is a pattern you haven&#8217;t tested.</p>



<p class="wp-block-paragraph">The mirror image is allow lists, which tell Macie to ignore specific text or patterns: your support inbox address, your published press contact, the seeded test records in everyone&#8217;s staging fixtures. Without them the same non-issues appear in every scan until the team stops reading findings. Allow lists accept predefined text or regex, and the per-account and per-job quotas are tight enough that you should curate rather than dump.</p>



<h2 class="wp-block-heading">Gap four: quota truncation inside large files</h2>



<p class="wp-block-paragraph">Subtle, because the object <em>is</em> analyzed. Just not completely, and a partial result reports like a full one.</p>



<ul class="wp-block-list">
<li>Per-file size quotas differ by format. Exceed the quota for a type and Macie analyzes none of that file, not part of it.</li>



<li>Archives have separate ceilings for nesting depth, extracted bytes and file count. If the metadata shows a breach up front, nothing is extracted. Cross a limit partway through and Macie stops, reporting only what it processed.</li>



<li>JSON and JSON Lines have a nested depth limit. Past it, the file is not analyzed at all.</li>



<li>Full names and mailing addresses cap out per file. After the cap Macie stops counting and stops reporting locations for that type, so the occurrence count is a floor, never a total.</li>
</ul>



<p class="wp-block-paragraph">Findings carry a status reason naming the limit that was hit, with values like <code>FILE_EXCEEDS_SIZE_LIMIT</code>, <code>ARCHIVE_NESTING_LEVEL_OVER_LIMIT</code> and <code>JSON_NESTING_LEVEL_OVER_LIMIT</code>. Filter on those before declaring a data lake of large compressed exports clean. Check current quota values in the Macie documentation rather than memorizing them, since they differ per format.</p>



<h2 class="wp-block-heading">Automated discovery or a discovery job?</h2>



<p class="wp-block-paragraph"><strong>Automated sensitive data discovery</strong> evaluates your inventory daily and samples representative objects across as many buckets as it can, grouping by metadata like prefix, extension and last-modified date. It is breadth-first and it does not re-analyze unchanged objects. Its job is to tell you which buckets deserve attention. It is not designed to prove a bucket is clean, and reading it that way is the root of most Macie misunderstandings.</p>



<p class="wp-block-paragraph"><strong>Sensitive data discovery jobs</strong> are the depth tool. You choose the buckets, scope by prefix, extension, size or object tag, choose the identifiers, and run once or on a schedule. This is what you point at the bucket automated discovery just flagged.</p>



<pre class="wp-block-code"><code># One-time job over a single prefix, recommended identifiers, 30% sampling
aws macie2 create-classification-job 
  --job-type ONE_TIME 
  --name "exports-prefix-review" 
  --managed-data-identifier-selector RECOMMENDED 
  --sampling-percentage 30 
  --s3-job-definition '{
    "bucketDefinitions": [
      {"accountId": "111122223333", "buckets": ["your-bucket-name"]}
    ],
    "scoping": {
      "includes": {
        "and": [
          {"simpleScopeTerm": {
             "comparator": "STARTS_WITH",
             "key": "OBJECT_KEY",
             "values": ["exports/"]
          }}
        ]
      }
    }
  }'</code></pre>



<p class="wp-block-paragraph">One detail about <code>--sampling-percentage</code> that trips people up: it selects a random percentage of eligible <em>objects</em> and then analyzes each selected object completely. It does not read a fraction of each file. For a bucket where every object comes from one pipeline, 20 or 30 percent tells you what you need. For heterogeneous uploads, sampling is a coin flip and you want the full pass.</p>



<h2 class="wp-block-heading">What actually drives the bill</h2>



<p class="wp-block-paragraph">Macie bills on three dimensions, and the one that surprises people is not the one they budget for.</p>



<ul class="wp-block-list">
<li><strong>Bucket evaluation.</strong> Per general purpose bucket monitored, prorated daily, up to the account ceiling. Predictable, driven by bucket count rather than data volume.</li>



<li><strong>Object monitoring.</strong> Per object tracked while automated discovery is enabled. This is inventory bookkeeping, not content reading, and it runs whether or not anything gets scanned that day. On an account with a log archive holding hundreds of millions of tiny objects, it can dwarf the other two.</li>



<li><strong>Data inspected.</strong> Per GB actually analyzed, covering both automated discovery and jobs.</li>
</ul>



<p class="wp-block-paragraph">So excluding buckets from automated discovery is a genuine cost lever, not just noise reduction. Log archives, CloudTrail destinations, build artifact stores and backup targets are usually safe exclusions. Exclude for the right reason though: a backup bucket holding database dumps of your customer table is exactly what you want scanned, however boring its name sounds.</p>



<p class="wp-block-paragraph">The three dimensions appear as distinct usage types in the Cost and Usage Report, so they split cleanly. A FinOps platform like Vantage or CloudZero, or just a Cost Explorer view grouped by usage type, tells you within a day whether object monitoring or inspection is driving the number. That changes the fix entirely: object monitoring is solved by scoping, inspection by sampling and scheduling.</p>



<h2 class="wp-block-heading">Getting findings somewhere a human sees them</h2>



<p class="wp-block-paragraph">Findings that only exist in the Macie console get read during onboarding week and never again. Two things are worth wiring up on day one.</p>



<p class="wp-block-paragraph"><strong>Publish findings outward.</strong> Macie emits findings to EventBridge and integrates with Security Hub. EventBridge earns its keep because you can filter on severity and finding type and route only what matters, whether that&#8217;s a Slack channel, a ticket, or a Lambda that tags the bucket. If you already centralize alerts in Grafana Cloud or similar, that path keeps sensitive data findings next to everything else on call sees.</p>



<p class="wp-block-paragraph"><strong>Configure the discovery results repository.</strong> Findings tell you what was found and where. The detailed discovery results, including analysis logs for objects where nothing was found, need an S3 bucket and a KMS key configured to persist. Nothing is retained long term until you set this up, and &#8220;we scanned it and found nothing&#8221; is exactly the record an auditor asks for.</p>



<p class="wp-block-paragraph">One operational note on the reveal feature, which retrieves sample occurrences so a human can confirm a finding is real. It needs a customer managed KMS key and it&#8217;s genuinely useful for triage. It also means someone is now looking at live customer PII, which belongs in your access model and audit trail. Same goes if an engineer downloads an object to investigate locally: that copy is unmanaged PII on a laptop and needs a secure deletion step, whether that&#8217;s a wipe tool from a vendor like O&amp;O Software or a documented and enforced process. Findings triage has a habit of creating the exposure it was meant to prevent.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting a bucket that reports nothing</h2>



<p class="wp-block-paragraph">When Amazon Macie PII detection reports nothing for a bucket you have doubts about, work down this in order. Each step rules out one gate, and stopping at the first hit saves you the rest.</p>



<ol class="wp-block-list">
<li><strong>Check the sensitivity label.</strong> <em>Not yet analyzed</em> at score 50 means nothing was analyzed. Do not read that as clean.</li>



<li><strong>Check the coverage page.</strong> Macie names the issue directly: access denied, invalid encryption, invalid KMS key, permission denied, or unclassifiable.</li>



<li><strong>Compare classifiable and unclassifiable counts</strong> with the <code>describe-buckets</code> query above.</li>



<li><strong>List a few object keys.</strong> Run <code>aws s3 ls</code> against a prefix and look for extensions. Extensionless keys explain the whole thing.</li>



<li><strong>Check the storage class.</strong> A lifecycle rule may have moved everything somewhere Macie doesn&#8217;t read.</li>



<li><strong>Check encryption.</strong> SSE-C is unfixable without re-encryption. A customer managed KMS key is fixable with a key policy change.</li>



<li><strong>Check object samples</strong> with <code>list-resource-profile-artifacts</code> to see exactly what was selected and skipped.</li>



<li><strong>Only now question the identifiers.</strong> Run a targeted job with a custom identifier for the data you expect and see whether it fires.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a low sensitivity score as evidence of no PII. It reflects what was found <em>and</em> how much was analyzed, so low coverage produces a low score.</li>



<li>Enabling Macie in one region and assuming account-wide coverage. Settings and results are per region, every time.</li>



<li>Running default identifiers in a business operating outside the jurisdictions the recommended set covers.</li>



<li>Building custom identifiers from regex alone, then abandoning Macie over the false positive volume.</li>



<li>Skipping the discovery results repository, then having no evidence trail when someone asks what was scanned and when.</li>



<li>Excluding buckets by name pattern without checking contents. Backup and export buckets are frequently the highest-risk ones you own.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Measure coverage before findings. A coverage report is the first deliverable of a Macie rollout, not an afterthought.</li>



<li>Fix extensions at the producer rather than building a rename pipeline downstream.</li>



<li>Run the KMS permission analyzer script once per account, then again whenever a new customer managed key appears.</li>



<li>Use automated discovery for breadth and targeted jobs for depth, letting the first choose targets for the second.</li>



<li>Test every custom identifier against both a positive and a negative sample before it reaches a job.</li>



<li>Curate allow lists early. The cost of a noisy first month is a team that stops reading findings.</li>



<li>Re-check coverage after any change to bucket policies, KMS keys, lifecycle rules or ingestion pipelines. All four silently remove data from scope.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Macie scan anything other than S3?</h3>



<p class="wp-block-paragraph">No. Macie analyzes objects in S3 general purpose buckets only. The documented workaround is bringing data to it: export RDS or Aurora snapshots to S3 in Parquet, or export a DynamoDB table to S3, then run a discovery job against the export. That works, but the export is now a second copy of your sensitive data with its own encryption and deletion requirements.</p>



<h3 class="wp-block-heading">Does Macie scan new objects as they land?</h3>



<p class="wp-block-paragraph">Not per object. Automated discovery evaluates inventory on a daily cycle and prioritizes objects that are new or recently changed, so fresh data moves up the queue rather than triggering an immediate scan. If you need scanning tied to arrival, drive a discovery job from S3 event notifications through EventBridge and accept that you own that orchestration.</p>



<h3 class="wp-block-heading">Why does a bucket I know has PII show a low sensitivity score?</h3>



<p class="wp-block-paragraph">Almost always coverage rather than detection. Check whether the objects are classifiable, whether Macie can decrypt them, and whether the label reads <em>Not yet analyzed</em>. If Macie genuinely read them and found nothing, the next suspect is the identifier set, and a targeted job with a custom identifier will tell you in one run.</p>



<h3 class="wp-block-heading">Do I need both automated discovery and discovery jobs?</h3>



<p class="wp-block-paragraph">For most teams, yes. Automated discovery gives you the map at predictable cost. Jobs give you proof for a specific bucket at a specific time, which is what compliance evidence actually requires. Running only jobs means you never discover the bucket nobody told you about.</p>



<h3 class="wp-block-heading">How do I cut false positives without missing real PII?</h3>



<p class="wp-block-paragraph">In order of preference: add keywords and a proximity rule so matches need context; use allow lists for specific known-benign values like your published contact details and test fixtures; and only then remove managed identifiers that are consistently wrong for your data. Removing identifiers is the bluntest option and the easiest to regret, so leave it last.</p>



<h3 class="wp-block-heading">Does Macie work across accounts in an organization?</h3>



<p class="wp-block-paragraph">Yes, through AWS Organizations with a delegated administrator. Design around this: the administrator&#8217;s automated discovery settings apply to member accounts, members can&#8217;t change them, and members see coverage and sensitivity data for their own buckets but not the sensitive data findings themselves. That shapes who can actually remediate what.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Amazon Macie PII detection is only as good as its coverage, and coverage fails silently by design. An object with the wrong extension, the wrong storage class, or an unreachable encryption key produces no finding, and no finding renders identically to a clean result. Before you hand a Macie report to anyone who will make a decision from it, put the coverage numbers next to it: how many objects were classifiable, how many were analyzed, how many were skipped and why.</p>



<p class="wp-block-paragraph">Do that once and the rest of the service becomes straightforward. Skip it and you&#8217;re maintaining an expensive dashboard reporting on whichever subset of your data happened to be readable.</p>



<h2 class="wp-block-heading">Need help getting Macie to actually see your data?</h2>



<p class="wp-block-paragraph">Most of the work in a Macie rollout isn&#8217;t turning it on. It&#8217;s the unglamorous part: proving what got scanned, fixing the reasons things didn&#8217;t, and making the output land somewhere a human acts on it. That&#8217;s the part I help with.</p>



<ul class="wp-block-list">
<li>Coverage audits across your S3 estate, with per-bucket classifiable, analyzed and skipped counts and the specific cause of each gap</li>



<li>Fixing those causes: KMS key policies, bucket policies blocking the service-linked role, lifecycle rules archiving data out of scope, pipelines writing extensionless objects</li>



<li>Custom data identifiers and allow lists tuned against your real data and tested on positive and negative samples first</li>



<li>Cost shaping: working out whether object monitoring or data inspection drives your bill, and scoping discovery so the number is defensible</li>



<li>Findings pipelines through EventBridge or Security Hub into the alerting and ticketing you already run, with severity filtering that keeps signal above noise</li>



<li>Multi-account setup under AWS Organizations, including the delegated administrator model and who can see and remediate what</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me the output of the <code>describe-buckets</code> query above, or a screenshot of your coverage page, and I&#8217;ll tell you what&#8217;s actually being scanned.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Twenty Client AWS Accounts and Not One Shared Admin Credential</title>
		<link>https://john-nessime.com/blog/devops/client-aws-account-access/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 27 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[AWS STS]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Confused Deputy]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[External ID]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[IAM Access Analyzer]]></category>
		<category><![CDATA[IAM Identity Center]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Managed Service Providers]]></category>
		<category><![CDATA[MFA]]></category>
		<category><![CDATA[Multi-Account Strategy]]></category>
		<category><![CDATA[Offboarding]]></category>
		<category><![CDATA[Permissions Boundaries]]></category>
		<category><![CDATA[Role Chaining]]></category>
		<category><![CDATA[Session Policies]]></category>
		<category><![CDATA[Source Identity]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=546</guid>

					<description><![CDATA[<p>Shared admin credentials work right up until someone resigns or an auditor asks who did what. Here's how to design client AWS account access around cross-account roles, unique external IDs and per-engineer attribution, so twenty accounts need zero shared keys and revocation is one action.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/client-aws-account-access/">Twenty Client AWS Accounts and Not One Shared Admin Credential</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A client&#8217;s auditor asks one question: list every person at your company who can sign in to our AWS account, and show us how you know. If the honest answer starts with &#8220;whoever has the shared vault,&#8221; that isn&#8217;t an answer. It&#8217;s a hope.</p>



<p class="wp-block-paragraph">That question is what actually bites when you run infrastructure for twenty clients. Not a breach. An audit, or a resignation. Someone gives notice on a Friday and you spend Monday working out which accounts they still hold keys to, because the credential everybody shared was never attached to a person. You can&#8217;t revoke a person from a password. You can only rotate it and hope you got all twenty.</p>



<p class="wp-block-paragraph">This post covers how to design client AWS account access so that humans authenticate once against something you control, every client account holds a role that trusts exactly one thing, and every API call in a client&#8217;s CloudTrail can be traced back to a named engineer. It&#8217;s organised by failure family, because the mechanics are well documented and the failure modes are not.</p>



<h2 class="wp-block-heading">The shape: one identity plane, twenty separate trust decisions</h2>



<p class="wp-block-paragraph">The instinct that causes all the trouble is putting credentials in the middle. Client gives you a login, you store it somewhere shared, engineers take it out when they need it. Everything downstream of that decision is damage control.</p>



<p class="wp-block-paragraph">The shape that works is boring. One AWS account whose only job is identity. Your engineers exist there, once, in IAM Identity Center or federated from whatever directory you already pay for. Each client account holds an IAM role whose trust policy names one principal in that account and requires an external ID. Nobody stores a client credential, because there is no client credential. There&#8217;s a role your engineers can ask STS for a short-lived session against, and a client who can delete that role whenever they like.</p>



<p class="wp-block-paragraph">There is a second design worth knowing about: inviting client accounts into your own AWS Organization. It&#8217;s genuinely better in some ways. You get service control policies, resource control policies, and centralised root access management across the estate, all of which are organisation-scoped features you simply cannot use on an account you don&#8217;t own. It&#8217;s also a much bigger commitment. You take on consolidated billing, the client&#8217;s spend flows through you, and pulling an account back out of an organisation at the end of a contract is a real project rather than a policy change.</p>



<p class="wp-block-paragraph">For most consultancies and managed service providers, the client keeps their own organisation and you get a role. That&#8217;s the model the rest of this assumes.</p>



<h2 class="wp-block-heading">Failure family 1: credentials you can&#8217;t revoke</h2>



<p class="wp-block-paragraph">The shared IAM user with an access key is the default because it works immediately. It keeps working right up until one of three moments: an engineer leaves, a client asks who did something, or a laptop gets stolen.</p>



<p class="wp-block-paragraph">Access keys have no expiry. They&#8217;re bearer tokens with an indefinite lifetime, and the only way to invalidate one is to go into each account and delete it. At twenty accounts that&#8217;s twenty deliberate actions performed under time pressure, and the one you forget is the one that matters. A role session, by contrast, expires on its own. That&#8217;s the entire argument.</p>



<p class="wp-block-paragraph">A shared password manager still earns its place, just not for admin keys. Bitwarden or 1Password is the right home for per-client external IDs, account numbers, role ARNs and the client-side contacts you need when something breaks at two in the morning. Metadata, not credentials.</p>



<h2 class="wp-block-heading">Failure family 2: a trust policy that trusts more than you meant</h2>



<p class="wp-block-paragraph">This is the invisible one. The role gets created, the access works, everyone moves on, and the trust policy quietly grants far more than anyone intended.</p>



<p class="wp-block-paragraph">Two mistakes travel together. The first is naming your account root in the principal. Writing <code>arn:aws:iam::111122223333:root</code> does not mean &#8220;the root user of my account.&#8221; It means any principal in that account that has permission to call AssumeRole. Every engineer, every Lambda function, every CI runner, every future role you haven&#8217;t created yet. It delegates the decision to your own account&#8217;s IAM, which is fine when your IAM is tidy and quietly catastrophic when it isn&#8217;t.</p>



<p class="wp-block-paragraph">The second is skipping the external ID. AWS calls the problem it solves the confused deputy: you are a third party that holds the ability to assume roles in many customer accounts, and without a shared secret in the condition, one customer who learns another customer&#8217;s role ARN could persuade your systems to act against it. The external ID is the value that makes each trust relationship specific rather than generic.</p>



<p class="wp-block-paragraph">Generate it yourself, one per client, as a UUID or similar unguessable value. The client must not be the one choosing it, because if they can choose it they can choose someone else&#8217;s, and the control does nothing.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/MspOperationsAccess"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The principal names one role, not the account. If that role is compromised you have one thing to fix. If the account is named, you have an inventory problem you didn&#8217;t know you had.</p>



<h2 class="wp-block-heading">Failure family 3: CloudTrail says the role, not the human</h2>



<p class="wp-block-paragraph">You&#8217;ve done everything above and a client emails asking who deleted the staging load balancer. You open their CloudTrail and every event attributes to the same assumed role. The role is shared by your whole team by design, so the log tells you your company did it. That&#8217;s not what they asked.</p>



<p class="wp-block-paragraph">There are two knobs and they are not equivalent. The role session name is chosen freely by whoever calls AssumeRole, appears in the assumed-role ARN, and is trivially set to anything. It&#8217;s useful, but it&#8217;s a label, not evidence. Source identity is the one that carries weight: once set on a session it cannot be changed, it persists across chained role sessions, and it lands in CloudTrail in the session context of every subsequent call.</p>



<p class="wp-block-paragraph">The catch is that the role&#8217;s trust policy has to permit <code>sts:SetSourceIdentity</code> alongside <code>sts:AssumeRole</code>, and you can require it with a condition so a session without one is refused outright.</p>



<pre class="wp-block-code"><code>"Action": [
  "sts:AssumeRole",
  "sts:SetSourceIdentity"
],
"Condition": {
  "StringEquals": {
    "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
  },
  "StringLike": {
    "sts:SourceIdentity": "*@yourcompany.com"
  }
}</code></pre>



<p class="wp-block-paragraph">If your engineers arrive through SAML federation, the identity provider can populate source identity from a directory attribute such as email, which means nobody has to remember to pass it. For direct CLI use it&#8217;s an explicit flag:</p>



<pre class="wp-block-code"><code>aws sts assume-role 
  --role-arn arn:aws:iam::444455556666:role/MspEngineerAccess 
  --role-session-name acme-investigation 
  --external-id 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02 
  --source-identity jane.doe@yourcompany.com</code></pre>



<p class="wp-block-paragraph">Getting this right is what turns &#8220;we take security seriously&#8221; into a sentence you can back with a log query. It is also, in my experience of what clients actually ask for, the single most persuasive thing on the list.</p>



<h2 class="wp-block-heading">Failure family 4: one identity, twenty blast radii</h2>



<p class="wp-block-paragraph">A hub account is a concentration of risk. That&#8217;s the trade you&#8217;re making, and it&#8217;s worth making, but only if the hub is held to a higher standard than the accounts it reaches into.</p>



<p class="wp-block-paragraph">Practically: not every engineer gets permission to assume every client&#8217;s role. Group permissions in the hub so the people on the Acme engagement reach Acme and nothing else. Use per-client role names rather than one shared name across all twenty, so a policy in the hub can&#8217;t wildcard its way into a client you stopped working with a year ago.</p>



<p class="wp-block-paragraph">Tier the roles by what they&#8217;re for, not by who&#8217;s asking. A read-only role for investigation covers most of the work most days. A change role carries the write permissions the engagement actually needs. Break-glass sits separate, with a short maximum session duration and an alarm on every use.</p>



<p class="wp-block-paragraph">Two extras are worth knowing about even if you don&#8217;t reach for them immediately. Session policies let the caller narrow a session further at assume time, which is useful for automation that only needs one bucket. Permissions boundaries cap what any role or user created inside the client account can do, which matters when part of your job is creating IAM there.</p>



<p class="wp-block-paragraph">If your team already routes through a business VPN or zero trust gateway with stable egress addresses, a source IP condition on the trust policy is a cheap extra layer. NordLayer, Tailscale and Cloudflare&#8217;s zero trust products all give you predictable egress. Be honest about the cost, though: the moment something runs from CI, a client&#8217;s network or a laptop on hotel wifi, an IP condition becomes a support ticket. I&#8217;d add it once everything else works, not before.</p>



<h2 class="wp-block-heading">Failure family 5: standing administrator access nobody remembers granting</h2>



<p class="wp-block-paragraph">Somewhere in the twenty accounts there is a role with AdministratorAccess attached, created during a migration, described in the ticket as temporary. It&#8217;s been eighteen months.</p>



<p class="wp-block-paragraph">Here&#8217;s the honest part: AWS does not give you a native just-in-time approval workflow for cross-account role assumption. There&#8217;s no built-in &#8220;request four hours of write access, get approved in Slack, access expires automatically&#8221; for this pattern. You either build that on top of STS and your ticketing system, or you accept standing access and compensate with tight session durations, aggressive monitoring and a genuine periodic review. Both are defensible. Pretending you have JIT when you have a wiki page is not.</p>



<p class="wp-block-paragraph">The cheap version that actually helps: set the maximum session duration on client roles to something short, alert on assumption of the break-glass role, and run a quarterly pass over service last accessed data to strip permissions nobody has used. IAM can generate that report per role.</p>



<pre class="wp-block-code"><code># Kick off the report, then fetch it with the returned job ID
aws iam generate-service-last-accessed-details 
  --arn arn:aws:iam::444455556666:role/MspEngineerAccess

aws iam get-service-last-accessed-details --job-id JOB_ID_FROM_ABOVE</code></pre>



<p class="wp-block-paragraph">Run that in each client account and the conversation about narrowing permissions stops being theoretical. You&#8217;re not guessing what the engagement needs; you&#8217;re reading it.</p>



<h2 class="wp-block-heading">Failure family 6: offboarding that only half happens</h2>



<p class="wp-block-paragraph">The whole point of the hub is that offboarding is one action. Disable the person in your directory and every path into every client account closes at once, because there was never a credential in a client account with their name on it.</p>



<p class="wp-block-paragraph">That&#8217;s true only if nothing crept in sideways. Things that creep in sideways:</p>



<ul class="wp-block-list">
<li>IAM users created directly in a client account during an incident, because it was faster</li>

<li>Access keys minted under a role by automation and never inventoried</li>

<li>A client&#8217;s own console user shared with your team &#8220;just for the migration&#8221;</li>

<li>Third party tools connected with their own cross-account roles that outlived the trial</li>

<li>A departing engineer&#8217;s laptop with cached SSO tokens and stale profiles still on disk</li>
</ul>



<p class="wp-block-paragraph">IAM Access Analyzer is the tool that finds most of the first four. Its external access findings show every principal outside the account that can reach into it, which is exactly the list a client wants to see, and it&#8217;s also the list that reveals the monitoring vendor nobody remembers signing up for. For the laptop, whatever your hardware disposal process is, make sure it involves an actual wipe rather than a factory reset; tools like O&amp;O SafeErase exist for that reason and are cheap next to the alternative.</p>



<h2 class="wp-block-heading">Rolling client AWS account access out across twenty accounts</h2>



<p class="wp-block-paragraph">Doing this by hand twenty times guarantees twenty subtly different configurations. Make it a template and a checklist.</p>



<ol class="wp-block-list">
<li>Write one CloudFormation template that creates the role, its trust policy and its permissions, parameterised by external ID and your operations account ID.</li>

<li>Generate a unique external ID per client and store it with the client&#8217;s account metadata, not in the template.</li>

<li>Send the client a launch stack link rather than asking for credentials. They deploy it in their own account, in their own console, and can read exactly what they&#8217;re granting.</li>

<li>Record the account ID, role ARN, external ID location, engagement scope and review date in a single inventory file in version control.</li>

<li>Generate everyone&#8217;s local AWS profiles from that inventory so nobody hand-edits a config file.</li>

<li>Verify each one with a call that proves identity and nothing else before you touch anything real.</li>
</ol>



<p class="wp-block-paragraph">The generated profile chains from your federated session into the client role. The AWS CLI reads these from the shared config file and handles the assume-role call for you:</p>



<pre class="wp-block-code"><code># ~/.aws/config
[profile msp-hub]
sso_session = company
sso_account_id = 111122223333
sso_role_name = MspOperationsAccess
region = eu-west-1

[profile acme-readonly]
role_arn = arn:aws:iam::444455556666:role/MspEngineerReadOnly
source_profile = msp-hub
external_id = 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02
role_session_name = jane-doe
duration_seconds = 3600
region = eu-west-1</code></pre>



<p class="wp-block-paragraph">Then the two-second sanity check that has saved me from more mistakes than any other command:</p>



<pre class="wp-block-code"><code>aws sts get-caller-identity --profile acme-readonly</code></pre>



<p class="wp-block-paragraph">It returns the account ID and the assumed-role ARN. Read the account number before you run anything destructive. Twenty profiles in one config file is exactly the situation where muscle memory picks the wrong one.</p>



<p class="wp-block-paragraph">For scheduled work that runs without a human, the same trust relationship serves a Lambda function or an EC2 instance role in your hub. A low-cost VPS from Contabo or InterServer will also do the job, but note the trade: an instance profile inside AWS gets rotated credentials automatically, while a box outside AWS needs a long-lived credential to bootstrap from, which reintroduces exactly what this design removes. Inside AWS is the cleaner answer.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the failures you will actually hit</h2>



<h3 class="wp-block-heading">AccessDenied on AssumeRole</h3>



<p class="wp-block-paragraph">Three separate causes produce nearly the same message, and they need different fixes. Either the client&#8217;s trust policy doesn&#8217;t name your principal, or the external ID you sent doesn&#8217;t match what the condition expects, or your own identity in the hub lacks permission to call AssumeRole on that role ARN. Check them in that order, because the first two require the client and the third doesn&#8217;t.</p>



<h3 class="wp-block-heading">Switch Role in the console silently fails</h3>



<p class="wp-block-paragraph">This one wastes an afternoon if you don&#8217;t know it. The console&#8217;s switch role feature does not support roles that require an external ID. It&#8217;s a documented limitation, not a bug in your policy. Engineers who need console access to those accounts should reach them through federation rather than the switch role box.</p>



<h3 class="wp-block-heading">Sessions expiring after an hour when you asked for longer</h3>



<p class="wp-block-paragraph">When one role session assumes another role, that&#8217;s role chaining, and a chained session is capped at one hour regardless of the role&#8217;s maximum session duration setting. Since your engineers assume the client role from an already-assumed session in the hub, you&#8217;re chaining by definition. Plan around the hour rather than fighting it.</p>



<h3 class="wp-block-heading">An authorization message you can&#8217;t read</h3>



<p class="wp-block-paragraph">Some denials come back with a long encoded blob attached. STS will decode it if your identity has permission to, and the decoded version usually names the exact statement that denied you.</p>



<pre class="wp-block-code"><code>aws sts decode-authorization-message --encoded-message &lt;encoded-blob&gt;</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Naming the account root as the trust principal instead of the specific operations role</li>

<li>Letting the client pick the external ID, or reusing one value across several clients</li>

<li>Treating the role session name as an audit trail when it&#8217;s a caller-supplied string</li>

<li>Asking a client for console credentials because a template felt like too much effort</li>

<li>Granting AdministratorAccess for a migration and never revisiting it</li>

<li>Building the hub properly and then leaving one legacy IAM user in one account as a fallback</li>

<li>Assuming that removing someone from your directory also removed the keys they created by hand</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>One identity plane, federated, with MFA enforced at the point the human authenticates</li>

<li>A unique unguessable external ID per client, generated by you and stored with client metadata</li>

<li>Source identity required in the trust policy so CloudTrail names a person, not a role</li>

<li>Roles tiered by purpose: read-only for investigation, change for the engagement, break-glass alarmed</li>

<li>Client-owned policies, deployed by the client from your template, revocable by the client in one action</li>

<li>A version-controlled inventory of every account, role, scope and review date</li>

<li>Access Analyzer external access findings reviewed on a schedule, in every client account</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Can&#8217;t I just ask each client to create an IAM user for me?</h3>



<p class="wp-block-paragraph">You can, and it&#8217;ll work on day one. The problem is day four hundred. That user&#8217;s access key never expires, it isn&#8217;t attached to any individual on your team, and revoking it means someone in the client&#8217;s organisation remembering to act. A role costs the client the same five minutes to create and hands them a control they can exercise unilaterally.</p>



<h3 class="wp-block-heading">Do I need a different external ID for every client?</h3>



<p class="wp-block-paragraph">Yes. A shared external ID across your client base defeats the purpose, because the value&#8217;s entire job is to be specific to one trust relationship. Generate a UUID per client, treat it as a secret, and never let the client choose it.</p>



<h3 class="wp-block-heading">Should I bring client accounts into my own AWS Organization?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re also taking on their billing and the commercial relationship supports it. Inside your organisation you get service control policies, resource control policies and centralised root access management, which are real advantages. Outside it, those stay the client&#8217;s tools and your job is to recommend them well. Most consulting engagements sit outside.</p>



<h3 class="wp-block-heading">How do I prove a departed engineer no longer has access?</h3>



<p class="wp-block-paragraph">Show the directory record disabling them, then show that no IAM user or access key in the client account carries their name, then show Access Analyzer external access findings listing only your operations role as an external principal. Three artefacts, all exportable. That&#8217;s the evidence pack.</p>



<h3 class="wp-block-heading">Does MFA carry across the cross-account role assumption?</h3>



<p class="wp-block-paragraph">Don&#8217;t assume it does. The reliable place to enforce MFA is where the human authenticates in your identity plane, and the reliable control at the client boundary is the combination of a narrowly scoped principal and the external ID condition. If you want an MFA condition in the client&#8217;s trust policy as well, test it end to end against your actual federation path before you rely on it, because the request context differs between direct assumption and chained sessions.</p>



<h3 class="wp-block-heading">What&#8217;s the minimum viable version if I only have three clients?</h3>



<p class="wp-block-paragraph">A hub account with federated engineers, one role per client scoped to a named principal with a unique external ID, and source identity required. Skip the tiering, skip the IP conditions, skip the JIT ambitions. Those three things give you revocation, attribution and a clean answer for the auditor, which is most of the value.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Client AWS account access is not really a permissions problem. It&#8217;s a revocation and attribution problem wearing a permissions costume. Any design where a credential exists that isn&#8217;t tied to a specific human will fail you at exactly the wrong moment, and no amount of policy tuning fixes that, because the gap isn&#8217;t in the policy.</p>



<p class="wp-block-paragraph">Put the humans in one place you control. Put a narrowly scoped, externally identified role in each client account. Require source identity so the log names a person. Everything else on this page is refinement; those three are the structure. And the pleasant side effect is that &#8220;here&#8217;s how you revoke us, unilaterally, in one action&#8221; is a genuinely good thing to be able to say in a sales conversation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your client access model?</h2>



<p class="wp-block-paragraph">This is the kind of work I do for consultancies and managed service providers who&#8217;ve grown past the point where a shared vault is survivable. Specifically:</p>



<ul class="wp-block-list">
<li>Reviewing existing cross-account trust policies for over-broad principals and missing external IDs</li>

<li>Designing a hub account and federation model that scales past a handful of clients without per-client special cases</li>

<li>Building the CloudFormation or Terraform template clients deploy themselves, so onboarding stops involving credentials</li>

<li>Turning on source identity end to end, including the identity provider side, so CloudTrail attributes to named engineers</li>

<li>Producing the offboarding evidence pack clients and auditors ask for, and automating it so it isn&#8217;t a scramble</li>

<li>Narrowing long-standing administrator roles down using service last accessed data, without breaking the engagement</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me one trust policy and the AccessDenied message it produces. That&#8217;s usually enough to tell where the model is leaking.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/client-aws-account-access/">Twenty Client AWS Accounts and Not One Shared Admin Credential</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</title>
		<link>https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Comprehend]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=272</guid>

					<description><![CDATA[<p>A confident answer with a citation from a matter the reader was walled off from. Nothing crashed, nothing alerted. Building legal document intelligence on AWS with Textract, Bedrock and OpenSearch is the easy half; the hard half is tenant isolation, verified deletion, S3 Object Lock holds, what leaves the account, and an audit trail that names the actual user.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo was going well until someone asked where the third citation came from.</p>



<p class="wp-block-paragraph">The answer on screen was good. Fluent, specific, correctly hedged, three sources listed underneath it. Then a partner in the room asked a simple question: which matter is source three from? Nobody could answer it in the room, and when we went and looked, it was from a matter that half the people watching the demo were formally walled off from.</p>



<p class="wp-block-paragraph">Nothing had crashed. No alarm fired. The retrieval layer had done exactly what a similarity search does, which is return the nearest vectors, and the nearest vectors did not care about the ethical wall. That is the thing about building a legal document intelligence platform on AWS: the failures that matter almost never look like failures. They look like a confident answer with a citation attached.</p>



<p class="wp-block-paragraph">This post is about the parts of that build that are hard, organised around the five boundaries that actually have to hold. The extraction pipeline is the easy half. I will cover it, but quickly, because the AWS documentation is good and the failure modes are visible. The other four boundaries fail quietly, and that is where the engineering goes.</p>



<h2 class="wp-block-heading">The shape of the thing</h2>



<p class="wp-block-paragraph">Before the boundaries, the skeleton. A legal document intelligence platform on AWS almost always ends up looking like this, whether you plan it or arrive at it:</p>



<ul class="wp-block-list">
<li>Documents land in Amazon S3, one prefix per matter, versioning on.</li>



<li>An S3 event triggers AWS Lambda, which starts an asynchronous Amazon Textract job.</li>



<li>Textract output lands back in S3 as JSON. Something normalises it into text plus page and bounding-box coordinates.</li>



<li>Amazon Comprehend or a foundation model classifies the document and pulls entities: parties, dates, governing law, clause types.</li>



<li>Chunks get embedded and written to a vector store, usually Amazon OpenSearch Serverless behind Amazon Bedrock Knowledge Bases.</li>



<li>An application layer, typically API Gateway plus Lambda, takes a question, retrieves, and calls a model on Amazon Bedrock.</li>
</ul>



<p class="wp-block-paragraph">AWS Step Functions is worth reaching for once you have more than three stages, because Textract jobs are long-running and Lambda timeouts are not a retry strategy. That is the whole architecture. You can stand it up in a fortnight. Then you spend six months on everything below.</p>



<h2 class="wp-block-heading">Boundary one: isolation, and why metadata filtering is not optional</h2>



<p class="wp-block-paragraph">This is the one from the demo, and it is the one that ends engagements.</p>



<p class="wp-block-paragraph">A vector index has no native concept of a matter, a client, or an ethical wall. If you pool every document into one index and rely on the prompt to keep things separate, you have built a system where a well-phrased question can pull privileged material across a wall. Prompt instructions are not an access control. They are a suggestion that usually works.</p>



<p class="wp-block-paragraph">There are two shapes that do work, and the choice between them is a real trade-off rather than a best practice.</p>



<h3 class="wp-block-heading">Pooled index with server-side filtering</h3>



<p class="wp-block-paragraph">Every chunk carries a <code>matter_id</code> and a <code>client_id</code> as metadata. Every retrieval call attaches a filter. With an S3 data source in Bedrock Knowledge Bases, the metadata comes from a sidecar JSON file that sits next to the document and carries a <code>metadataAttributes</code> object.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "matter_id": "M-4417",
    "client_id": "C-108",
    "doc_type": "engagement_letter",
    "privileged": true
  }
}</code></pre>



<p class="wp-block-paragraph">The retrieval call then narrows the search before the model ever sees a chunk:</p>



<pre class="wp-block-code"><code>"retrievalConfiguration": {
  "vectorSearchConfiguration": {
    "filter": {
      "equals": { "key": "matter_id", "value": "M-4417" }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two rules make this safe, and both are the kind of thing that gets skipped under deadline. First, the filter value is derived on the server from the authenticated caller&#8217;s claims, never accepted from the client. If the browser can send you a <code>matter_id</code>, the browser can send you a different one. Second, the filter is applied by code that no feature request can bypass. A helper that builds every retrieval request, and a code review rule that no other path may call the retrieve API directly.</p>



<p class="wp-block-paragraph">The trap here is ordering. You cannot filter on an attribute the index does not have. Add <code>matter_id</code> after ingestion and every chunk already in the index is invisible to that filter, which means it either returns for everyone or for no one, depending on how your filter is written. Design the metadata schema before the first ingestion run, not after the first demo.</p>



<h3 class="wp-block-heading">Separate collection per client</h3>



<p class="wp-block-paragraph">The heavier option. A separate OpenSearch Serverless collection per client, which buys you a separate AWS KMS key per client, separate index settings, and a failure mode where a bug in the filtering logic cannot reach across clients at all.</p>



<p class="wp-block-paragraph">It costs you sprawl. Every collection is a resource to provision, monitor, patch policy on, and eventually delete. Ingestion jobs run per data source per knowledge base, so freshness guarantees fragment. For a firm with twelve institutional clients this is fine. For a platform onboarding a hundred small clients it becomes the main operational burden of the product.</p>



<p class="wp-block-paragraph">My default is pooled with strict server-side filtering, and per-client collections only where the client&#8217;s own contract demands a dedicated encryption key. If you cannot say out loud which of those two you are running, you are running neither properly.</p>



<h2 class="wp-block-heading">Boundary two: extraction, and the confidence score everyone ignores</h2>



<p class="wp-block-paragraph">Textract returns a confidence score between 0 and 100 with each prediction. AWS is explicit in its own best-practice guidance that applications sensitive to detection errors should enforce a minimum threshold and route anything below it for human scrutiny, and that the right threshold depends entirely on how the output gets used.</p>



<p class="wp-block-paragraph">Legal documents sit at the harsh end of that. A misread date on an archival scan is a curiosity. A misread date on a limitation period is a problem with a name on it.</p>



<p class="wp-block-paragraph">Practical notes from this layer:</p>



<ul class="wp-block-list">
<li>Use the asynchronous operations for anything multipage. <code>StartDocumentAnalysis</code> and <code>GetDocumentAnalysis</code> handle long PDFs and TIFFs; the synchronous <code>AnalyzeDocument</code> path is for single pages and will fight you on a 400-page bundle.</li>



<li>Textract&#8217;s Queries feature earns its place on structured instruments. Instead of extracting everything and grepping, you ask the document a direct question and get a scoped answer with its own confidence.</li>



<li>Keep the page number and bounding box for every chunk. When a lawyer asks where a sentence came from, &#8220;page 14, second column&#8221; is an answer. &#8220;It&#8217;s in the corpus somewhere&#8221; is not, and the platform loses credibility the first time you say it.</li>



<li>Amazon Augmented AI wires low-confidence predictions into a human review queue with a private workforce, so the reviewers are your people rather than an anonymous pool. For privileged material that distinction is the entire point.</li>
</ul>



<p class="wp-block-paragraph">What I would skip early: building a custom entity recognition model. Comprehend supports custom entity recognition and it is genuinely useful, but you need labelled data you do not have yet, and a foundation model with a decent prompt gets you far enough to find out which entities the users actually care about. Train the custom model once the requirement has stopped moving.</p>



<h2 class="wp-block-heading">Boundary three: retention, where deletion is a distributed problem</h2>



<p class="wp-block-paragraph">Here is the failure that will not show up in any test suite you write.</p>



<p class="wp-block-paragraph">A client asks for a document to be removed. Someone deletes the S3 object. The ticket closes. The document is still fully searchable, because its chunks are sitting in the vector index and will stay there until the data source is re-synced. Meanwhile the Textract JSON output is in a second bucket, the extracted text may be in a database, and if model invocation logging is on, whole passages of the document are sitting in CloudWatch Logs or another S3 bucket entirely.</p>



<p class="wp-block-paragraph">One delete, at least four places holding a copy. Write the deletion path as a real workflow with an assertion at the end, not as a single API call:</p>



<ol class="wp-block-list">
<li>Delete the source object and, if versioning is on, its versions.</li>



<li>Delete the derived artefacts: Textract JSON, normalised text, any thumbnails or page images.</li>



<li>Trigger an ingestion job so the knowledge base drops the orphaned chunks.</li>



<li>Confirm removal by querying the index for the document identifier and expecting nothing back.</li>



<li>Deal with the logs, which means either a retention policy short enough to make the problem expire or a deliberate decision, written down, that logs are out of scope.</li>
</ol>



<p class="wp-block-paragraph">Step four is the one people leave out, and it is the only step that actually proves anything.</p>



<h3 class="wp-block-heading">The opposite problem: things that must not be deleted</h3>



<p class="wp-block-paragraph">Legal work has the reverse requirement too. When a matter goes into litigation hold, the documents need to survive an administrator with delete permissions and a bad afternoon.</p>



<p class="wp-block-paragraph">S3 Object Lock is the mechanism, and it has two independent controls that people routinely conflate. A retention period protects an object version until a fixed date, in either governance mode, which privileged users can override, or compliance mode, which nobody can override and where the period cannot be shortened. A legal hold has no date at all. It stays until someone with <code>s3:PutObjectLegalHold</code> explicitly removes it. The two can be active at once, and while either is active the object version cannot be deleted or overwritten.</p>



<pre class="wp-block-code"><code># Place an indefinite hold on a specific object version
aws s3api put-object-legal-hold 
  --bucket matters-archive 
  --key M-4417/exhibit-c.pdf 
  --version-id 3sL7f2Qz9pXvB1kR 
  --legal-hold Status=ON</code></pre>



<p class="wp-block-paragraph">Two things bite here. Object Lock requires versioning and turns it on automatically, and once enabled on a bucket you cannot turn it off or suspend versioning again. And compliance mode is genuinely permanent: if you set a seven-year retention when you meant seven days, that is the answer, for everyone, including the account root. Test in governance mode first. The bypass path exists and is deliberately awkward, requiring both the <code>s3:BypassGovernanceRetention</code> permission and an explicit <code>x-amz-bypass-governance-retention:true</code> header on the request, which is exactly the level of friction you want on that operation.</p>



<p class="wp-block-paragraph">Design the hold model before you design the deletion model, because holds win. A deletion request that collides with an active hold is a legal question, not an engineering one, and the platform&#8217;s job is to surface the collision clearly rather than resolve it silently.</p>



<h2 class="wp-block-heading">Boundary four: disclosure, or what leaves the account</h2>



<p class="wp-block-paragraph">This is the boundary the client&#8217;s general counsel will ask about, usually in writing, usually before signature. Three separate controls, and they are not interchangeable.</p>



<h3 class="wp-block-heading">The AI services opt-out policy</h3>



<p class="wp-block-paragraph">AWS Organizations has a governance policy type that opts your accounts out of having content processed by certain AI services stored and used for service improvement. The list includes Amazon Textract and Amazon Comprehend, which are precisely the two doing the reading in this architecture. The policy type has to be enabled at the organisation root before you can attach anything:</p>



<pre class="wp-block-code"><code>aws organizations enable-policy-type 
  --root-id r-example 
  --policy-type AISERVICES_OPT_OUT_POLICY</code></pre>



<p class="wp-block-paragraph">Read the AWS documentation on this one carefully rather than taking my summary as gospel, because there is a caveat in it that matters: the services may still need to store your data operationally even when you have opted out of it being used for improvement. Opting out is not the same as the data never existing outside your account. Say that plainly in the client conversation. It is a much better position than being asked about it later.</p>



<h3 class="wp-block-heading">Bedrock&#8217;s data position, and the log that undoes it</h3>



<p class="wp-block-paragraph">Amazon Bedrock&#8217;s published position is that inputs and outputs are not shared with third-party model providers and are not used to train the base models, and that fine-tuning operates on a private copy. That is a strong starting point for privileged content and it is the main reason Bedrock rather than a direct provider API shows up in these builds.</p>



<p class="wp-block-paragraph">Then there is model invocation logging. It captures full prompt and response payloads to CloudWatch Logs or S3, and you want it on, because without it you cannot reconstruct what the system told someone. But understand what you have just built: a log that contains privileged document text, at a per-region setting, in a destination that probably has looser access controls than the document store it came from. Encrypt it with the same KMS key discipline, restrict it harder than you think you need to, and set a retention period on purpose.</p>



<p class="wp-block-paragraph">Worth separating in your head: CloudTrail records that an API call happened and who made it. Model invocation logging records what was in it. You need both, for different questions, and only one of them contains client confidences.</p>



<h3 class="wp-block-heading">Network path</h3>



<p class="wp-block-paragraph">Interface VPC endpoints via AWS PrivateLink keep traffic to Bedrock, Textract and the rest on the AWS network rather than out through an internet gateway or NAT. Add a gateway endpoint for S3 while you are there.</p>



<p class="wp-block-paragraph">Endpoint policies are the part that gets forgotten. An endpoint without a policy is a private path to the whole service, including into accounts that are not yours. Scope it to the operations and resources you actually use, and pair it with an S3 bucket policy that rejects requests not arriving through your endpoint. The first protects what leaves; the second protects what can be reached.</p>



<h3 class="wp-block-heading">The parts that are not on the architecture diagram</h3>



<p class="wp-block-paragraph">Two of these have caught people out badly, and neither is an AWS control.</p>



<p class="wp-block-paragraph">The first is the development environment. You do not want real client documents on a laptop or in a scratch account, so build the parsing and chunking code against synthetic documents on a small separate box. A cheap VPS from a provider like Contabo or InterServer is fine for this, and the separation is worth more than the convenience of iterating in the production account.</p>



<p class="wp-block-paragraph">The second is the reviewers. If your human review loop involves contractors working remotely, their network path and their disks are part of your boundary whether or not you drew them. A managed VPN such as NordVPN or Surfshark handles the network side, and when a matter closes and local copies have to go, a dedicated erasure tool from something like O&amp;O Software does what dragging a folder to the bin does not. Unglamorous, and it is the layer that gets audited.</p>



<h2 class="wp-block-heading">Boundary five: evidence, because someone will ask</h2>



<p class="wp-block-paragraph">At some point the question stops being &#8220;does it work&#8221; and becomes &#8220;who saw what, and when&#8221;. If you cannot answer that from stored records, the platform is not defensible regardless of how good the answers are.</p>



<ul class="wp-block-list">
<li>Turn on CloudTrail data events for the document buckets. They are off by default, they are billed separately from management events, and they are the only way to see individual object-level reads.</li>



<li>Log the retrieval, not just the generation. Store which chunks came back and which filter was applied. When somebody asks whether the wall held on a specific query, this record is the answer and there is no reconstructing it later.</li>



<li>Carry the end user&#8217;s identity through to the audit record. A Lambda execution role in the logs tells you the platform did something. It does not tell you who asked.</li>



<li>Alarm on the filter, not just on errors. A retrieval that ran without a matter filter should page someone, even though it returned HTTP 200 and a perfectly good answer.</li>
</ul>



<p class="wp-block-paragraph">That last one is the closest thing to a single takeaway in this post. The dangerous events in this architecture are successful ones.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the things that will go wrong</h2>



<h3 class="wp-block-heading">Retrieval returns nothing after you add a filter</h3>



<p class="wp-block-paragraph">Nearly always a metadata problem rather than a query problem. Either the sidecar metadata file was missing at ingestion, so the chunks carry no attribute to match, or the attribute was added after the chunks were indexed. Check whether the attribute exists on a known chunk before you touch the query. If it does not, you need a re-sync, not a different filter expression.</p>



<h3 class="wp-block-heading">Textract returns text but the layout is scrambled</h3>



<p class="wp-block-paragraph">Common on two-column pleadings and on documents with headers and footers on every page. Raw text detection reads in a reading order that is not always yours. Use the layout and table features rather than plain detection, and hold onto the geometry so you can reassemble columns yourself if the default order is wrong for that document class.</p>



<h3 class="wp-block-heading">The model cites a document that does not exist</h3>



<p class="wp-block-paragraph">Usually the citation is being generated rather than passed through. If document names are being written by the model instead of read from the retrieval response, it will invent plausible ones. Build citations from the retrieval result metadata in your application code and never from the generated text. Bedrock Guardrails helps with contextual grounding, but the structural fix is not asking the model to produce the reference in the first place.</p>



<h3 class="wp-block-heading">Ingestion is slow and nobody knows where</h3>



<p class="wp-block-paragraph">Instrument per stage before you optimise anything. In most of these pipelines the time is in Textract for large scanned bundles, and the fix is parallelism at the document level rather than tuning the extraction itself. Step Functions with a distributed map over documents gets you there without a queue you have to babysit.</p>



<h3 class="wp-block-heading">Costs climb faster than volume</h3>



<p class="wp-block-paragraph">Look at re-processing first. A pipeline that re-extracts a document every time anything downstream changes will quietly multiply your Textract bill against a static corpus. Key the extraction cache on the object&#8217;s content hash, not its path, and make re-ingestion an explicit action. The billing mechanisms differ per service, so read the current pricing pages rather than trusting a number from a blog post.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating the system prompt as an access control.</strong> It is a formatting instruction that happens to look like a rule.</li>



<li><strong>Taking the tenant identifier from the request body.</strong> If the client can name the matter, the client can name someone else&#8217;s.</li>



<li><strong>Designing the metadata schema after the first ingestion run.</strong> Retrofitting a filterable attribute means re-indexing the whole corpus.</li>



<li><strong>Assuming a deleted S3 object is gone from the platform.</strong> The chunks, the extraction output and the invocation logs all outlive it.</li>



<li><strong>Setting compliance-mode retention without testing in governance mode.</strong> There is no support ticket that fixes a seven-year mistake.</li>



<li><strong>Enabling model invocation logging without treating the log as privileged.</strong> You have just made a second copy of the documents in a less protected place.</li>



<li><strong>Building VPC endpoints and leaving the default policy on them.</strong> A private path with no policy is still a path.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>One retrieval helper, server-side filter injection, and a review rule that no other code path calls the retrieve API directly.</li>



<li>Metadata schema agreed and frozen before the first production ingestion. Include the fields you might filter on later, even if unused today.</li>



<li>Page-level provenance on every chunk, surfaced in the interface. It builds trust faster than any accuracy improvement.</li>



<li>A confidence threshold chosen per document class, with a human review path for everything under it.</li>



<li>Deletion implemented as a verified workflow across every store, ending in an assertion that the content is actually unretrievable.</li>



<li>Object Lock legal holds for matters under litigation hold, tested in governance mode before anything runs in compliance mode.</li>



<li>Customer-managed KMS keys across S3, the vector store and the logs, so key access is a control you can actually revoke.</li>



<li>An alarm on unfiltered retrievals, because the failure mode is a success response.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a legal document intelligence platform on AWS safe for privileged material?</h3>



<p class="wp-block-paragraph">The services support it. Bedrock&#8217;s stated position is that inputs and outputs are not shared with model providers or used to train base models, PrivateLink keeps traffic off the public internet, and KMS covers encryption at rest. What determines safety is your own configuration: tenant isolation, log handling and deletion behaviour. The platform is exactly as confidential as its weakest copy of the text.</p>



<h3 class="wp-block-heading">Should I use Bedrock Knowledge Bases or build retrieval myself?</h3>



<p class="wp-block-paragraph">Start with Knowledge Bases. It handles chunking, embedding, sync and metadata filtering, and those are weeks of work with no differentiation in them. Build your own when you need retrieval behaviour it does not expose, such as unusual re-ranking or per-tenant chunking strategies. Going custom on day one usually means reimplementing the managed service badly.</p>



<h3 class="wp-block-heading">How do I stop one client&#8217;s documents surfacing in another client&#8217;s answers?</h3>



<p class="wp-block-paragraph">Tag every chunk with a tenant identifier at ingestion, and inject the matching filter server-side on every retrieval from the authenticated caller&#8217;s identity. Never accept the identifier from the client. If the contract requires per-client encryption keys, use a separate collection per client instead, accepting the extra operational load.</p>



<h3 class="wp-block-heading">Does deleting a document from S3 remove it from the search index?</h3>



<p class="wp-block-paragraph">No. The chunks stay in the vector store until an ingestion job runs and reconciles the data source. Until then the document is deleted and still fully searchable, which is the worst combination available. Always finish a deletion by querying the index and confirming nothing comes back.</p>



<h3 class="wp-block-heading">What is the difference between an S3 legal hold and a retention period?</h3>



<p class="wp-block-paragraph">A retention period runs until a fixed date and comes in governance mode, which privileged users can override, or compliance mode, which nobody can. A legal hold has no end date and stays until someone explicitly removes it. They are independent, they can both be active on the same object version, and while either is active the object cannot be deleted or overwritten.</p>



<h3 class="wp-block-heading">Do I need human review, or is the model accurate enough?</h3>



<p class="wp-block-paragraph">Accuracy is the wrong frame. The question is what happens to a wrong answer downstream. Where output feeds a decision with consequences, you want a confidence threshold and a review queue, and Amazon Augmented AI with a private workforce gives you that without building the review tooling yourself. Where output is a search aid a person will verify anyway, review is friction you do not need.</p>



<h3 class="wp-block-heading">Can I run this in a single AWS account?</h3>



<p class="wp-block-paragraph">Technically yes, and for a pilot it is reasonable. It gets uncomfortable once you have production documents and a development environment in the same place, because an IAM mistake has nowhere to stop. Separate accounts under Organizations also gives you the policy layer, including the AI services opt-out policy, which only exists at the organisation level.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A legal document intelligence platform on AWS is not hard to build. Textract reads, Bedrock reasons, OpenSearch retrieves, and the tutorial version works on the first afternoon.</p>



<p class="wp-block-paragraph">What is hard is that its failures are silent and well-formed. The cross-matter citation, the deleted document that still answers questions, the privileged passage sitting in a log bucket, the retrieval that ran without a filter and returned a beautiful paragraph. None of these throw an error. All of them are the kind of thing that ends a client relationship.</p>



<p class="wp-block-paragraph">So build the boundaries first and the features second, and make sure every one of them is something you can prove from a stored record rather than something you believe about the code. If you can only take one habit from this: alarm on the successful requests that should not have been possible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your document pipeline?</h2>



<p class="wp-block-paragraph">I work on AWS document processing and retrieval systems where the confidentiality requirements are real. Typical things I get called in for:</p>



<ul class="wp-block-list">
<li>Reviewing tenant isolation in an existing RAG setup and finding the paths where the filter can be bypassed.</li>



<li>Designing the metadata and chunking schema before ingestion, so filtering works without a re-index later.</li>



<li>Building Textract pipelines with confidence thresholds, human review routing and page-level provenance.</li>



<li>Implementing verified deletion across S3, derived artefacts, the vector index and logs, with a proof step at the end.</li>



<li>Setting up retention and legal hold with S3 Object Lock, including the governance-mode rehearsal before compliance mode.</li>



<li>Locking down the network and logging path: VPC endpoints with real policies, KMS key separation, and audit records that name the actual user.</li>
</ul>



<p class="wp-block-paragraph">If you have an architecture diagram, a Step Functions definition or a retrieval request you are unsure about, send it over and I will tell you what I would change and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AWS Cross-Account Data Sharing Done Safely: Share Access, Not Copies</title>
		<link>https://john-nessime.com/blog/cloud-computing/aws-cross-account-data-sharing/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 22 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[AWS RAM]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Sharing]]></category>
		<category><![CDATA[External ID]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[IAM Access Analyzer]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Multi-Account Strategy]]></category>
		<category><![CDATA[Resource Control Policies]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=445</guid>

					<description><![CDATA[<p>Adding another account to a bucket policy takes one line and delegates far more than most people realize. A practical look at AWS cross-account data sharing: the delegation model, the KMS gaps, object ownership, references versus copies, and the guardrails that catch the mistake you will eventually make.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-cross-account-data-sharing/">AWS Cross-Account Data Sharing Done Safely: Share Access, Not Copies</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The request usually arrives as one sentence in Slack. &#8220;Analytics moved to their own account, can you just add them to the bucket policy?&#8221;</p>



<p class="wp-block-paragraph">It is a one-line change. It works on the first try. And it is the grant most likely to still be sitting there long after that analytics project shipped, covering a wider set of identities than the person who wrote it ever had in mind.</p>



<p class="wp-block-paragraph">That asymmetry is the whole problem. A denied cross-account request is a nuisance: someone pings you, you read the policy, you fix it before lunch. A cross-account grant that succeeds, keeps succeeding, and quietly reaches further than intended produces no error, no alert, and no ticket. It surfaces months later during an audit, when someone asks who has been reading that prefix and the honest answer is &#8220;anyone the other account decided to let in.&#8221;</p>



<p class="wp-block-paragraph">This post covers AWS cross-account data sharing as a set of failure families rather than a service tour. How the delegation model actually behaves, where encryption keys break or widen access without warning, who ends up owning the bytes after a handoff, why a copy is worse than a reference, and which guardrails catch the mistake you are eventually going to make. Real policy snippets and commands throughout, plus the denials that are hardest to read.</p>



<h2 class="wp-block-heading">AWS cross-account data sharing is a delegation, not a permission</h2>



<p class="wp-block-paragraph">Start with the mechanic that everything else hangs off. For a request that crosses an account boundary, two separate accounts have to say yes. The resource-based policy in the account that owns the data has to allow the caller, and an identity-based policy in the calling account has to allow the action. Neither side alone is enough.</p>



<p class="wp-block-paragraph">That is why &#8220;I added them to the bucket policy and it still fails&#8221; is such a routine ticket. The producer did their half. Nobody attached the matching IAM policy on the consumer side.</p>



<p class="wp-block-paragraph">Now the part that bites. Look at how most cross-account bucket policies get written:</p>



<pre class="wp-block-code"><code>{
  "Sid": "AllowAnalyticsAccount",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ]
}</code></pre>



<p class="wp-block-paragraph">Despite how it reads, that <code>:root</code> suffix does not mean the root user. It means the account. You have not granted access to a principal at all. You have delegated the guest list to the administrators of account 111122223333, and they can hand that access to any principal they like, including a role somebody creates next quarter for a purpose you will never hear about.</p>



<p class="wp-block-paragraph">Sometimes that delegation is exactly what you want, because the consumer team manages its own roles and you do not want to be in the loop for every change. Usually you want the narrower version:</p>



<pre class="wp-block-code"><code>{
  "Sid": "AllowAnalyticsRoleOnly",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ],
  "Condition": {
    "StringEquals": { "aws:PrincipalOrgID": "o-abc123def4" }
  }
}</code></pre>



<p class="wp-block-paragraph">Both halves earn their place. The named ARN stops the wrong role in the right account. The organization condition stops the right role name in the wrong account, which matters if the statement is ever copied, templated, or widened by someone in a hurry. One protects against internal drift, the other against the policy escaping your organization entirely.</p>



<p class="wp-block-paragraph">Be clear-eyed about the trade-off, though. Naming ARNs is precise and brittle: every new consumer role is a policy change, a pull request, and a review. Organization-wide conditions scale beautifully and narrow nothing inside a member account. If your organization contains twelve accounts and forty teams, <code>aws:PrincipalOrgID</code> on its own is a fairly wide door.</p>



<h3 class="wp-block-heading">The grant that stops working after a rebuild</h3>



<p class="wp-block-paragraph">When you name a role ARN as a principal, IAM stores the role&#8217;s internal unique ID behind the scenes. Delete that role and the policy will start displaying an opaque <code>AROA...</code> string where the ARN used to be, because the ARN no longer resolves to anything. Recreate a role with the identical name and the access does not come back, because the new role has a new unique ID.</p>



<p class="wp-block-paragraph">A <code>terraform destroy</code> followed by <code>terraform apply</code> on the consumer side reproduces this perfectly. If you are staring at a bucket policy that contains a principal starting with AROA, that is your answer. Rewrite the statement with the current ARN.</p>



<h2 class="wp-block-heading">The encryption boundary nobody tests until it is live</h2>



<p class="wp-block-paragraph">Second failure family: the key. Encrypting shared data is the easy decision. What people miss is that a KMS key is a second, fully independent authorization surface, with its own resource policy, evaluated separately from the bucket.</p>



<ul class="wp-block-list">
<li>Downloading an object encrypted with SSE-KMS requires <code>kms:Decrypt</code> on the key.</li>

<li>Uploading one requires <code>kms:GenerateDataKey</code>.</li>

<li>A multipart upload requires both.</li>
</ul>



<p class="wp-block-paragraph">The AWS managed key for S3 cannot carry you here, because you cannot edit its key policy. Sharing KMS-encrypted objects across accounts means a customer managed key, with a statement naming the consumer:</p>



<pre class="wp-block-code"><code>{
  "Sid": "AllowAnalyticsRoleToDecrypt",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*",
  "Condition": {
    "StringEquals": { "kms:ViaService": "s3.us-east-1.amazonaws.com" }
  }
}</code></pre>



<p class="wp-block-paragraph">The <code>kms:ViaService</code> condition is cheap insurance. It restricts use of the key to requests that arrive through S3 in that region, so a leaked set of consumer credentials cannot call KMS directly and start decrypting ciphertext it obtained some other way. And the two-key rule applies to the key exactly as it does to the bucket: the consumer&#8217;s own IAM policy has to allow the KMS actions too.</p>



<p class="wp-block-paragraph">Now the sharp edge, which is worth reading twice. If a KMS key is specified by <em>alias</em> rather than by fully qualified ARN, the alias is resolved in the requester&#8217;s account. In a cross-account write that means data landing in your bucket can end up encrypted under a key belonging to the writer. You own the object and cannot read it, and no error is raised at write time. Use full key ARNs in anything that crosses an account boundary.</p>



<h2 class="wp-block-heading">Who owns the bytes after the handoff</h2>



<p class="wp-block-paragraph">Third family: ownership, and it depends entirely on which direction the data moves.</p>



<p class="wp-block-paragraph">New buckets are created with Object Ownership set to Bucket owner enforced. ACLs are off, and the bucket owner owns every object written into the bucket regardless of who wrote it. That default removed the single most confusing trap in S3 sharing.</p>



<p class="wp-block-paragraph">Buckets created before that default are the ones to check. If a bucket still sits on the Object writer setting, an object uploaded by another account is owned by that account, and you, holding the bucket, may not be able to read it. Nothing fails at write time. It fails later, when a job in your own account tries to read your own bucket and gets a 403.</p>



<p class="wp-block-paragraph">Check and fix it with the ownership controls API:</p>



<pre class="wp-block-code"><code>aws s3api get-bucket-ownership-controls --bucket shared-events-bucket

aws s3api put-bucket-ownership-controls 
  --bucket shared-events-bucket 
  --ownership-controls '{"Rules":[{"ObjectOwnership":"BucketOwnerEnforced"}]}'</code></pre>



<p class="wp-block-paragraph">Order matters when you make that change on a live bucket. Once ACLs are disabled, any PUT that still carries an ACL is rejected with <code>AccessControlListNotSupported</code>. Requests that send no ACL, or send <code>bucket-owner-full-control</code>, keep working. So the sequence is:</p>



<ol class="wp-block-list">
<li>Find every writer that sets an ACL, including old scripts using <code>--acl</code> and any SDK call passing an ACL argument.</li>

<li>Update those clients so they stop sending one.</li>

<li>Confirm from CloudTrail that ACL-bearing PUTs have stopped.</li>

<li>Flip the bucket to Bucket owner enforced.</li>
</ol>



<p class="wp-block-paragraph">Given a free choice of design, I would rather the consumer pull from a bucket I own than push into a bucket I do not. Pull keeps ownership, lifecycle rules, encryption configuration, and access logging in one account, under one team&#8217;s control. Push scatters them.</p>



<h2 class="wp-block-heading">Copies you cannot recall</h2>



<p class="wp-block-paragraph">Fourth family, and the one with the longest tail. There is a large difference between granting a reference and shipping a copy. A reference can be revoked on a Tuesday afternoon. A copy is simply gone.</p>



<p class="wp-block-paragraph">Three reference-shaped patterns are worth knowing before you reach for a bulk export.</p>



<h3 class="wp-block-heading">S3 Access Points for object-level sharing</h3>



<p class="wp-block-paragraph">Instead of a bucket policy that grows a new statement every time a team asks for something, you write the bucket policy once to delegate to access points, then give each consumer their own access point with its own focused policy. The delegation is expressed with the <code>s3:DataAccessPointAccount</code> condition. Access points can also be owned by the consumer&#8217;s account, so they manage their own detail without touching your bucket policy.</p>



<p class="wp-block-paragraph">The gotcha: when a request goes through an access point, both the access point policy and the underlying bucket policy have to allow it. A 403 here is easy to misdiagnose because the bucket policy looks correct and is correct. The access point is the layer that said no.</p>



<h3 class="wp-block-heading">Lake Formation for table-level sharing</h3>



<p class="wp-block-paragraph">When the thing being shared is a table rather than a prefix, Lake Formation is the better fit, because it lets you grant on databases, tables, columns, and row filters instead of on object paths. Cross-account grants ride on AWS Resource Access Manager, so the recipient gets an invitation that a data lake administrator has to accept.</p>



<pre class="wp-block-code"><code>aws ram get-resource-share-invitations

aws ram accept-resource-share-invitation 
  --resource-share-invitation-arn arn:aws:ram:us-east-1:111122223333:resource-share-invitation/EXAMPLE</code></pre>



<p class="wp-block-paragraph">After the invitation is accepted, the shared database and tables are visible but still not queryable from Athena or Redshift Spectrum until someone creates a <em>resource link</em>, which behaves roughly like a symlink in the recipient&#8217;s Data Catalog. Missing resource links account for most of the &#8220;the share went through but I cannot see the table&#8221; messages.</p>



<p class="wp-block-paragraph">One setting to check on the producer side is the cross-account data sharing version. Newer versions collapse many grants into far fewer RAM resource shares, which matters once you are sharing at any scale, and they allow granting directly to IAM principals in another account rather than only at account level. Read the current version with:</p>



<pre class="wp-block-code"><code>aws lakeformation get-data-lake-settings</code></pre>



<h3 class="wp-block-heading">When a copy really is the answer</h3>



<p class="wp-block-paragraph">Sometimes the partner genuinely needs the data on their own infrastructure: a warehouse you have no reach into, a VPS at a provider like Contabo or InterServer, or in the worst case an analyst&#8217;s laptop. That is a legitimate outcome, but be honest with yourself and with them that your controls stop at the handoff.</p>



<p class="wp-block-paragraph">Practical hygiene for that case: scope the extract to the columns actually needed rather than the whole table, put an expiry date in the agreement rather than in your head, and write down what happens to the copy when the engagement ends. If the copy lives on machines you or the client control, verified erasure with something like O&amp;O SafeErase reads a great deal better in an audit than &#8220;we deleted the folder.&#8221; And if the recipient has no AWS account at all, a time-limited presigned URL is usually a better answer than minting an IAM user with long-lived keys.</p>



<h2 class="wp-block-heading">The guardrail that catches the mistake you will make</h2>



<p class="wp-block-paragraph">Everything above describes what a careful engineer does per grant. Guardrails are what happens when somebody is not careful, which over a long enough window includes all of us.</p>



<p class="wp-block-paragraph">Resource control policies are the strongest lever if you run AWS Organizations. An RCP sets a ceiling on what any resource-based policy inside the organization can grant. The canonical statement is &#8220;no principal outside my organization may access S3 buckets in my accounts, regardless of what an individual bucket policy says.&#8221; That inverts the risk model: a careless bucket policy is no longer an exposure, it is just an ineffective statement.</p>



<p class="wp-block-paragraph">Two limits are worth stating plainly. RCPs cover a specific set of services rather than everything in AWS, so check the current supported list rather than assuming coverage. And they constrain your resources, not a partner&#8217;s behavior once data has legitimately left. An RCP is a wall around your side of the fence.</p>



<p class="wp-block-paragraph">Service control policies do the mirror-image job on the consumer side, limiting which external accounts your own principals are allowed to assume roles into. Account-level S3 Block Public Access remains a sensible floor underneath both.</p>



<p class="wp-block-paragraph">For sharing with a third party rather than an internal account, the external ID is not optional in my book:</p>



<pre class="wp-block-code"><code>{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::444455556666:role/vendor-connector" },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringEquals": { "sts:ExternalId": "a-value-the-vendor-generated-for-you" }
  }
}</code></pre>



<p class="wp-block-paragraph">The external ID exists because of the confused deputy problem. A vendor holds credentials that can assume roles across many customers, and without a per-customer secret in the condition, customer A can trick the vendor into pointing that access at customer B. If a vendor asks you to create a role for them with no external ID and no conditions at all, that is worth a conversation before it is worth a role.</p>



<h2 class="wp-block-heading">Verifying that the grant is what you think it is</h2>



<p class="wp-block-paragraph">Reading policies is not the same as knowing what they permit. IAM Access Analyzer answers the question &#8220;what is currently reachable from outside my zone of trust&#8221; using automated reasoning rather than pattern matching, which is why it catches combinations a human reviewer skims past.</p>



<pre class="wp-block-code"><code>aws accessanalyzer create-analyzer 
  --analyzer-name external-access 
  --type ACCOUNT

aws accessanalyzer list-findings 
  --analyzer-arn arn:aws:access-analyzer:us-east-1:999988887777:analyzer/external-access</code></pre>



<p class="wp-block-paragraph">Scope the zone of trust to your organization where you have one, then archive the findings that represent intentional shares, so that a new entry in the list actually means something.</p>



<p class="wp-block-paragraph">The same service exposes custom policy checks you can run in a pipeline before a policy change merges, including checks for public access, for specific actions being granted, and for whether an updated policy grants more than the one it replaces. One thing to understand about them: they evaluate the document you hand them and nothing else. They do not read your account state, so a public-access check can fail on a policy in an account that has Block Public Access enabled. For a merge gate that is correct behavior, because the policy should not be relying on a distant safety net.</p>



<p class="wp-block-paragraph">Finally, CloudTrail tells you who actually used a grant, which is the input to the only cleanup that ever really works: delete the statements nobody has exercised. Routing that history into CloudWatch, or into a Grafana Cloud dashboard alongside the rest of your platform telemetry, turns &#8220;is this still needed&#8221; from an argument into a query.</p>



<h2 class="wp-block-heading">Troubleshooting the denials that are hard to read</h2>



<ul class="wp-block-list">
<li><strong>You can list the bucket but every GET returns 403.</strong> That pattern points at KMS, not S3. Listing does not touch the key; reading does. Check <code>kms:Decrypt</code> on both the key policy and the consumer&#8217;s identity policy.</li>

<li><strong>403 through an access point while the bucket policy is clearly correct.</strong> The access point has its own policy and it also has to allow the calling principal. Read that one next.</li>

<li><strong>AssumeRole is denied although the trust policy names the caller.</strong> Cross-account needs both halves: the caller&#8217;s identity policy must permit <code>sts:AssumeRole</code> against that role ARN. Then check the external ID value and any SCP or permissions boundary in the calling account.</li>

<li><strong>A principal in the policy displays as AROA followed by random characters.</strong> The referenced role was deleted. Recreating it under the same name will not restore access.</li>

<li><strong>A Lake Formation share appears in the catalog but queries fail.</strong> Usually a missing resource link in the recipient account, or a RAM invitation that was never accepted.</li>

<li><strong>Access worked yesterday and stopped today with nothing changed locally.</strong> Look at the organization layer before re-reading the bucket policy. A newly attached RCP or SCP denies in a way that no amount of resource-policy editing will fix.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Granting to an account principal when a role ARN would have done, then never revisiting it.</li>

<li>Treating <code>aws:PrincipalOrgID</code> as least privilege. It narrows the organization, not the account.</li>

<li>Sharing a bucket and forgetting the key, then debugging S3 for an hour over a KMS denial.</li>

<li>Referring to KMS keys by alias in cross-account configuration instead of by full ARN.</li>

<li>Letting a partner push into your bucket without confirming the ownership setting first.</li>

<li>Exporting a full table when the consumer needed four columns, because the export was easier to write.</li>

<li>Creating a vendor role with no external ID and no conditions because the vendor&#8217;s setup guide said to.</li>

<li>Granting access with no expiry, no owner, and no note in the policy Sid explaining why it exists.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Give every cross-account grant a descriptive <code>Sid</code>. It is the only free documentation slot IAM offers, and future-you will read it.</li>

<li>Prefer one dedicated consumer role per relationship over a shared role used by several integrations. Revocation becomes a single delete.</li>

<li>Pair a specific principal with a broad condition, rather than relying on either alone.</li>

<li>Put policy checks in CI so a widened grant fails a pull request instead of a quarterly review.</li>

<li>Keep an inventory of active shares with an owner and a review date, even if it is a table in a repository README.</li>

<li>Enable an external access analyzer scoped to your organization and treat new findings as work, not noise.</li>

<li>Watch cross-region and internet data transfer when a consumer sits somewhere unexpected. Cost visibility tooling such as Vantage or CloudZero will attribute it faster than reading a bill.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the safest way to share an S3 bucket with another AWS account?</h3>



<p class="wp-block-paragraph">Name a specific role ARN in the bucket policy, add an organization or account condition alongside it, scope the resource to a prefix rather than the whole bucket, and make sure the consumer&#8217;s identity policy is equally narrow. If the bucket is encrypted with a customer managed key, grant on the key as well. If the list of consumers is going to grow, move to access points rather than growing the bucket policy.</p>



<h3 class="wp-block-heading">Do I need both a bucket policy and an IAM policy?</h3>



<p class="wp-block-paragraph">For cross-account access, yes. The resource-based policy in the owning account and the identity-based policy in the calling account both have to allow the action. That is the single most common reason a correct-looking bucket policy still returns Access Denied.</p>



<h3 class="wp-block-heading">Why do I get Access Denied when the bucket policy clearly allows the other account?</h3>



<p class="wp-block-paragraph">Work through it in this order: the consumer&#8217;s identity policy, then the KMS key policy if the objects are encrypted with a customer managed key, then any access point policy in the path, then organization-level SCPs and RCPs. Denials from the organization layer are the ones people waste the most time on, because nothing in either account&#8217;s policies looks wrong.</p>



<h3 class="wp-block-heading">Can I share data with an account outside my AWS Organization?</h3>



<p class="wp-block-paragraph">You can, and organization conditions will not help you there. Use a named role ARN, an external ID on any assumed role, a tightly scoped resource, and a defined end date. Also confirm whether an RCP in your organization already blocks external principals for that service, because that is a common reason a partner grant refuses to work no matter how carefully it is written.</p>



<h3 class="wp-block-heading">Should I use S3 Access Points or Lake Formation?</h3>



<p class="wp-block-paragraph">It depends on the unit of sharing. If the consumer wants objects under a prefix, access points are simpler and cheaper to operate. If the consumer wants a table, and especially if you need column or row filtering, Lake Formation is the right tool and access points would force you to fake fine-grained control with prefix layout. Plenty of environments run both.</p>



<h3 class="wp-block-heading">How do I revoke cross-account access cleanly?</h3>



<p class="wp-block-paragraph">Remove the statement from the resource policy, remove the KMS key grant, and check for any access point, RAM share, or resource link that also carries permission. Then confirm with CloudTrail that the principal has stopped appearing. Revocation feels done long before it is done, which is exactly why an inventory of active shares pays for itself.</p>



<h3 class="wp-block-heading">Does cross-account access change anything about cost?</h3>



<p class="wp-block-paragraph">The request charges land on the bucket owner by default, and data transfer depends on whether the consumer is in the same region. A consumer that quietly starts scanning a large prefix from another region is a billing event as much as a security one, which is a good argument for reviewing access patterns and spend together rather than in separate meetings.</p>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Getting AWS cross-account data sharing right is less about knowing which service to reach for and more about noticing which grants are quietly larger than they read. An account principal delegates your guest list. An organization condition narrows the wrong dimension. A copy leaves your perimeter permanently and no policy edit brings it back.</p>



<p class="wp-block-paragraph">Grant references rather than copies, name principals rather than accounts, remember that the key is a second door, and put something in place that watches the perimeter for you. The cross-account grant worth worrying about is never the one that failed. It is the one that worked, and kept working, long after anybody was still looking at it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your cross-account setup?</h2>



<p class="wp-block-paragraph">Most of the cross-account problems I see are not exotic. They are a policy written in a hurry two years ago that nobody has read since. If that sounds familiar, here is where I can help:</p>



<ul class="wp-block-list">
<li>Auditing existing bucket, key, and role trust policies to find grants that are wider than intended, and rewriting them to named principals with conditions.</li>

<li>Designing a producer and consumer pattern for a specific data handoff, including whether access points, Lake Formation, or a plain scoped role is the right shape.</li>

<li>Setting up IAM Access Analyzer with a sensible zone of trust and archive rules, so the findings list stays worth reading.</li>

<li>Adding policy checks to your pipeline so a widened grant blocks a merge rather than surviving to production.</li>

<li>Building an organization-level perimeter with resource control policies and service control policies without breaking the integrations you already depend on.</li>

<li>Untangling stubborn cross-account denials, including the KMS, access point, and organization-layer cases that hide well.</li>
</ul>



<p class="wp-block-paragraph">Send me a bucket policy, a trust policy, or the CloudTrail entry for the denial you cannot explain, and I will tell you what it actually permits.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-cross-account-data-sharing/">AWS Cross-Account Data Sharing Done Safely: Share Access, Not Copies</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</title>
		<link>https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 25 Jul 2026 12:04:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Config]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[Healthcare Cloud]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[Restore Testing]]></category>
		<category><![CDATA[VPC]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=184</guid>

					<description><![CDATA[<p>A working engineer's guide to HIPAA compliance on AWS, organised by the gap between the control you configured and the obligation you actually carry. Covers BAA account scope, the eligible services list as a contract boundary, KMS key policy versus the encryption checkbox, what "six years" really applies to, backup and restore scope, and the subprocessor chain nobody inventories.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/">HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket usually reads something like: &#8220;Legal wants to know if the analytics account is in scope.&#8221; So you open Security Hub. Green. You check the Config rules. Passing. Every bucket is encrypted, every volume is encrypted, MFA is on, CloudTrail is running in all Regions. You reply that the account is fine.</p>



<p class="wp-block-paragraph">Then someone points out that a nightly job has been copying a de-identified extract into that account for eight months, the de-identification script never removed the admission dates, and the account was spun up before anyone thought about the Business Associate Addendum. Nothing was misconfigured. Every control you built worked exactly as designed. And you have been out of compliance the entire time.</p>



<p class="wp-block-paragraph">That is the shape of most real failures here. Not a breach, not a misconfiguration, but a mismatch between the boundary your tooling checks and the boundary your obligation actually follows. This post covers HIPAA compliance on AWS organised by those gaps: where the contract stops, where encryption stops being a control, what &#8220;six years&#8221; genuinely applies to, and which parts of the estate people forget are in scope at all.</p>



<h2 class="wp-block-heading">Eligible is not compliant, and the difference is the whole job</h2>



<p class="wp-block-paragraph">AWS does not sell HIPAA compliance. It sells HIPAA <em>eligible</em> services, which is a genuinely different thing. Eligible means AWS has built the service so it can lawfully handle electronic protected health information and has agreed to cover it under a Business Associate Addendum. Compliant describes an entire system: your architecture, your key management, your access reviews, your policies, your staff, your vendors.</p>



<p class="wp-block-paragraph">Under the shared responsibility model, AWS secures the infrastructure. You secure everything you build on it. Nothing about signing the BAA transfers a single obligation off your side of the line. An unencrypted RDS instance, an overly broad IAM policy or an application that logs a patient identifier into stdout is your problem in exactly the same way it would be in a rack you own.</p>



<p class="wp-block-paragraph">People know this in the abstract. Where it bites is in the specifics below.</p>



<h2 class="wp-block-heading">Gap one: the BAA is a contract boundary, and nothing enforces it</h2>



<p class="wp-block-paragraph">This is the one I would fix first, because it is invisible to every security tool you own.</p>



<p class="wp-block-paragraph">The AWS BAA is self-service through AWS Artifact, at no extra cost. You can accept it for a single account, or, if you are in the management account of an AWS Organization, accept it once so that existing and future member accounts are covered. That organization-level option is the one worth using, because the per-account version quietly rots: someone creates a new account for a proof of concept, nobody repeats the Artifact step, and six months later that account is running something real.</p>



<p class="wp-block-paragraph">The second half of the boundary is the HIPAA Eligible Services Reference that AWS publishes. Only services on that list may create, receive, process, maintain or transmit ePHI under the BAA. The list is long, it changes, and some entries carry carve-outs where the service is eligible but a specific feature is not. Reading a service name on the list and assuming every feature inside it is covered is the kind of mistake that only surfaces during an audit.</p>



<p class="wp-block-paragraph">Here is the part worth internalising: <strong>there is no AWS control that stops you putting PHI into a non-eligible service.</strong> No API error, no Config rule out of the box, no GuardDuty finding. The eligible services list is a contractual construct. Your infrastructure has no idea it exists.</p>



<h3 class="wp-block-heading">Turning a contract boundary into a technical one</h3>



<p class="wp-block-paragraph">The mechanism that actually helps is Service Control Policies on the organizational unit that holds your PHI accounts. SCPs set the ceiling on what any principal in those accounts can do, including the root user, so they work as a guardrail rather than a suggestion.</p>



<p class="wp-block-paragraph">Start with the easy one. Pin the accounts to the Regions you have actually assessed, because data residency assumptions fall apart the moment someone launches something in a Region you never reviewed:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnapprovedRegions",
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "organizations:*",
        "route53:*",
        "cloudfront:*",
        "support:*",
        "sts:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The <code>NotAction</code> list matters. Global services are backed by endpoints in specific Regions, so denying them wholesale by Region locks you out of IAM and breaks Route 53 and CloudFront. Those entries are exemptions, not an allow-list.</p>



<p class="wp-block-paragraph">The harder one is restricting which services can be used at all. The same <code>NotAction</code> pattern works, with the services you have approved for PHI listed as the exemptions and everything else denied. It is effective and it is blunt: every new service anyone wants becomes a change request against the policy, and if you forget a dependency you find out through a failure in production. I would only reach for it on a dedicated PHI OU where the workload is well understood, not across a general-purpose organization.</p>



<p class="wp-block-paragraph">Whichever route you take, write down the approved service list somewhere a human reviews on a schedule, and diff it against the AWS reference periodically. That review is itself a compliance artefact.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap two: encryption is a checkbox, the key policy is the control</h2>



<p class="wp-block-paragraph">Almost every guide to HIPAA compliance on AWS tells you to encrypt at rest and in transit. Almost none of them explain why it is worth doing properly rather than minimally, so teams enable default encryption with an AWS-managed key, watch the Config rule turn green, and move on.</p>



<p class="wp-block-paragraph">The reason to care is the Breach Notification Rule. It applies to <em>unsecured</em> PHI, meaning PHI that has not been rendered unusable, unreadable or indecipherable through a method HHS has specified. HHS guidance points at NIST-validated encryption. If PHI is encrypted to that standard and the decryption keys were not compromised alongside it, an incident involving that data generally does not trigger the notification machinery at all. No individual letters, no HHS portal submission, no press release for a large incident.</p>



<p class="wp-block-paragraph">Read that second condition again, because it is where the architecture decision lives. The safe harbour depends on the keys not being compromised with the data. If your encryption key is one an attacker inherits automatically the moment they compromise a role in the account, you have encryption but you may not have the argument.</p>



<h3 class="wp-block-heading">What that means in practice</h3>



<ul class="wp-block-list">
<li>Use customer managed KMS keys for anything holding PHI, not AWS-managed keys. Only a customer managed key gives you a key policy you can write, and only a key policy lets you deny decryption independently of the resource policy.</li>

<li>Separate the key administrators from the key users. The people who can schedule deletion of a key should not be the people whose application role uses it every second.</li>

<li>Use a distinct key per data domain rather than one key for the whole account. Blast radius and audit trail both improve, and you get the ability to revoke access to one dataset without touching another.</li>

<li>Constrain key usage with the <code>kms:ViaService</code> condition so a key that exists to encrypt RDS storage cannot be used to decrypt something a role dragged into Lambda.</li>

<li>Turn on key rotation and leave it on. It costs nothing operationally and it is the kind of thing an assessor asks about by reflex.</li>
</ul>



<p class="wp-block-paragraph">Pull the current key policy before you assume it says what you think:</p>



<pre class="wp-block-code"><code>aws kms get-key-policy 
  --key-id alias/phi-rds 
  --policy-name default 
  --output text

# Find storage that slipped through unencrypted
aws ec2 describe-volumes 
  --filters Name=encrypted,Values=false 
  --query 'Volumes[].{Id:VolumeId,AZ:AvailabilityZone}' 
  --output table

aws rds describe-db-instances 
  --query 'DBInstances[?StorageEncrypted==`false`].DBInstanceIdentifier' 
  --output text</code></pre>



<p class="wp-block-paragraph">The RDS query is the important one, because RDS encryption cannot be enabled in place. If that command returns anything, the fix is a snapshot, an encrypted copy of the snapshot, a restore, and a cutover. Plan for downtime or a replication strategy. This is the single most common &#8220;we will fix it later&#8221; item I see, and later gets expensive.</p>



<p class="wp-block-paragraph">Also switch on EBS encryption by default in every Region you use, so the next instance somebody launches from a console wizard is not a new exception:</p>



<pre class="wp-block-code"><code>aws ec2 enable-ebs-encryption-by-default --region us-east-1
aws ec2 get-ebs-encryption-by-default --region us-east-1</code></pre>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap three: you have logs, but you may not have evidence</h2>



<p class="wp-block-paragraph">The Security Rule requires audit controls: mechanisms that record and examine activity in systems containing ePHI. It also requires you to regularly review records of information system activity. Both of those are about having and using the records.</p>



<p class="wp-block-paragraph">Now the correction, because this one is repeated everywhere and it is wrong in a way that costs money. You will read that HIPAA requires six years of audit logs. It does not. The six-year requirement sits in the documentation standard, and it applies to the policies, procedures and records of actions, activities and assessments that the Security Rule requires you to keep, retained for six years from creation or from when the document was last in effect, whichever is later. There is no clause anywhere in the Security Rule that names a retention period for CloudTrail events.</p>



<p class="wp-block-paragraph">What this actually means is more demanding, not less. You have to <em>decide</em> your log retention period, write it into a policy, justify it against your risk analysis, and then keep that policy for six years. And an assessor will hold you to the number you wrote. Setting a CloudWatch Logs retention of thirty days while your policy claims one year is a finding. Storing seven years of everything because a blog told you to, when your policy says two, is not compliance, it is just a bill.</p>



<p class="wp-block-paragraph">So: pick a period you can defend, make the infrastructure match it exactly, and treat any gap between policy and configuration as a defect.</p>



<h3 class="wp-block-heading">Making logs into evidence</h3>



<p class="wp-block-paragraph">Retention is only half of it. The other half is being able to show that the records were not altered. CloudTrail has log file validation for exactly this, and it is off unless you turn it on:</p>



<pre class="wp-block-code"><code>aws cloudtrail update-trail 
  --name org-phi-trail 
  --enable-log-file-validation

# Later, prove a window of logs is intact
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:us-east-1:111122223333:trail/org-phi-trail 
  --start-time "$(date -u -d '90 days ago' +%Y-%m-%dT%H:%M:%SZ)"</code></pre>



<p class="wp-block-paragraph">With validation enabled, CloudTrail writes signed digest files alongside the log files, and <code>validate-logs</code> checks them. The difference between &#8220;here are our logs&#8221; and &#8220;here are our logs, and here is a cryptographic check that nothing was modified or deleted&#8221; is the difference between an assertion and evidence.</p>



<p class="wp-block-paragraph">Put the archive bucket in a separate account that the workload accounts cannot write to or delete from, and apply S3 Object Lock in compliance mode for the retention window you committed to. Object Lock in compliance mode cannot be shortened or bypassed by anyone, including the root user, which is exactly the property you want and exactly the property that will hurt if you set the period carelessly. Test it in governance mode first.</p>



<p class="wp-block-paragraph">For the review obligation, a query interface matters more than raw storage. Athena over the CloudTrail bucket is the cheap default. If you want alerting and dashboards on top of access patterns, this is a natural place for a platform such as Grafana, Datadog or Splunk, and any of them will hold access records for you. Just remember that if those records contain PHI, that vendor needs a BAA too. See the subprocessor section below.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap four: backups, snapshots and the parts of scope people forget</h2>



<p class="wp-block-paragraph">The Security Rule&#8217;s contingency plan standard is not optional decoration. It requires a data backup plan, a disaster recovery plan and an emergency mode operation plan, plus testing and revision procedures. Most teams have the backups. Far fewer have the tested restore, and the tested restore is the part that gets asked about.</p>



<p class="wp-block-paragraph">Three things routinely go wrong here.</p>



<ol class="wp-block-list">
<li><strong>Copies leave the boundary.</strong> A cross-Region snapshot copy lands in a Region you did not assess. A cross-account copy for the DR account lands somewhere outside the OU your SCPs protect. The data is still PHI. The controls did not travel with it.</li>

<li><strong>Re-encryption changes the key, not just the copy.</strong> Copying an encrypted snapshot to another account requires a key the destination can use. It is easy to end up with a shared or less restrictive key protecting your backups than protects production, which inverts the risk model.</li>

<li><strong>The restore is never rehearsed.</strong> A backup you have never restored is a hypothesis. Schedule a restore into an isolated account, record the elapsed time, and file the result. That record is your evidence for the testing requirement, and it is the single easiest compliance artefact to produce for free.</li>
</ol>



<p class="wp-block-paragraph">While you are inventorying, remember the places PHI ends up without anyone deciding it should: application logs that include request bodies, database slow query logs capturing parameter values, support tickets with screenshots attached, CSV extracts in an analyst&#8217;s bucket, and non-production environments seeded from a production dump. That last one is the classic. If your staging database is a copy of production, staging is in scope, and staging is almost never built to the same standard.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap five: the business associate chain does not stop at AWS</h2>



<p class="wp-block-paragraph">Your BAA with AWS covers AWS. It covers nothing else in your stack.</p>



<p class="wp-block-paragraph">Every vendor that can create, receive, maintain or transmit PHI on your behalf is a business associate and needs an agreement. In a typical AWS estate that means the error tracker holding stack traces, the log aggregation platform, the APM tool, the transactional email provider, the customer support desk, the CI system if it ever touches a production dataset, and any AI or analytics service you have wired in.</p>



<p class="wp-block-paragraph">Build the inventory as a table with three columns: vendor, what PHI it can see, and whether a signed agreement exists. The third column is usually where the surprises are. Some vendors sign readily, some only on higher-priced tiers, and some decline entirely, at which point you have an architecture decision rather than a procurement one.</p>



<p class="wp-block-paragraph">One structural move that reduces this surface considerably: keep everything that does not need PHI out of the PHI accounts entirely. Your marketing site, your docs, your status page and your public API gateway for non-clinical traffic do not belong in a regulated account. Running them on ordinary infrastructure, whether that is a separate AWS account, a straightforward VPS from a host like InterServer, or a static site behind Cloudflare, shrinks the estate you have to assess, evidence and defend. Fewer things in scope is the cheapest compliance win available.</p>



<p class="wp-block-paragraph">For tracking the paperwork side, compliance automation platforms such as Vanta, Drata or Secureframe pull evidence from AWS on a schedule and keep the vendor register current. They are genuinely useful for the collection and reminder burden. They do not design your architecture, and I have seen teams treat a green dashboard in one of those tools as though it were an assessment. It is not. It is a checklist that knows what you told it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What is changing, and why &#8220;addressable&#8221; is a bad thing to build on</h2>



<p class="wp-block-paragraph">Since it was adopted, the Security Rule has split implementation specifications into <em>required</em> and <em>addressable</em>. Addressable never meant optional. It meant you assess whether the specification is reasonable and appropriate, and if not, you implement an equivalent alternative or document why neither is necessary. In practice, a lot of organisations turned the documented justification into the deliverable and skipped the control.</p>



<p class="wp-block-paragraph">HHS published a Notice of Proposed Rulemaking in the Federal Register in January 2025 that would remove that distinction, making implementation specifications required with limited exceptions, and would explicitly require encryption of ePHI at rest and in transit and multi-factor authentication, again with limited exceptions. The comment period closed in March 2025.</p>



<p class="wp-block-paragraph">Be precise about the status, because a lot of vendor content is not: <strong>this is a proposed rule and it is not final.</strong> The expected timeline for final action has slipped more than once, and the requirements could still change or be withdrawn. Nobody should be telling you a compliance deadline as though it were settled.</p>



<p class="wp-block-paragraph">What is worth taking from it is the direction of travel. If your current position depends on having documented that encryption or MFA was not reasonable and appropriate, that position is fragile regardless of what the final rule says. On AWS specifically, encryption at rest and MFA are both cheap and both already best practice. Building the architecture on an addressable deferral is an unforced risk.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the findings you will actually hit</h2>



<h3 class="wp-block-heading">&#8220;An assessor asked which accounts are in BAA scope and nobody could answer&#8221;</h3>



<p class="wp-block-paragraph">Sign in to AWS Artifact from the management account and check the organization agreements tab to see whether the BAA was accepted at the organization level or per account. If it is per account, list your accounts, work out which hold PHI, and confirm each one individually. Then move to the organization-level agreement so this question has one answer forever.</p>



<h3 class="wp-block-heading">&#8220;Config says the bucket is encrypted but we cannot prove who read the objects&#8221;</h3>



<p class="wp-block-paragraph">Bucket encryption and object-level access logging are unrelated. CloudTrail management events do not record S3 object reads by default. You need CloudTrail data events for that bucket, or S3 server access logging, and both cost money proportional to request volume. Enable data events selectively on the buckets that hold PHI rather than account-wide.</p>



<h3 class="wp-block-heading">&#8220;We enabled an SCP and production broke&#8221;</h3>



<p class="wp-block-paragraph">Almost always a Region deny catching a global service endpoint, or a service allow-list missing a dependency the workload calls indirectly. Check CloudTrail for <code>AccessDenied</code> events with an explicit deny from an SCP, and look at the service name in the event rather than the one you expected. Attach new SCPs to a test OU with a representative workload before the PHI OU.</p>



<h3 class="wp-block-heading">&#8220;Snapshot copy to the DR account fails with a KMS error&#8221;</h3>



<p class="wp-block-paragraph">The destination account cannot use the source key. The source key policy has to grant the destination principal permission to use it, and the copy has to specify a key the destination can decrypt with. Fix it by granting explicitly on a key you control, not by falling back to an AWS-managed key, which is the tempting shortcut and gives up the key policy control you needed.</p>



<h3 class="wp-block-heading">&#8220;CloudWatch Logs retention was never set&#8221;</h3>



<p class="wp-block-paragraph">New log groups default to never expiring, which is both a cost problem and a policy mismatch. Audit them with <code>aws logs describe-log-groups</code> and look for groups with no <code>retentionInDays</code> value, then set the period your policy specifies with <code>aws logs put-retention-policy</code>.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the signed BAA as the finish line rather than the prerequisite. It is the thing you need before the first byte of PHI arrives, not evidence that anything is configured correctly.</li>

<li>Assuming a service is fully eligible because its name appears on the list, without reading the feature-level carve-outs next to it.</li>

<li>Quoting &#8220;six years&#8221; as a log retention requirement, then either overspending on storage or writing a policy that contradicts the actual configuration.</li>

<li>Using AWS-managed KMS keys for PHI, which leaves no key policy to write and no independent revocation path.</li>

<li>Seeding staging or test environments from production data and then holding those environments to a lower standard.</li>

<li>Forgetting that the risk analysis is a required, recurring, documented activity, not a one-off spreadsheet from the year you launched.</li>

<li>Signing a BAA with AWS and none of the ten other vendors that can see the same data.</li>

<li>Letting a compliance automation dashboard stand in for an architecture review.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li><strong>Isolate PHI in its own accounts and its own OU.</strong> Account boundaries are the strongest isolation AWS offers, and they make the scope question answerable in one sentence.</li>

<li><strong>Accept the BAA at the organization level.</strong> It removes an ongoing manual step that fails silently.</li>

<li><strong>Customer managed keys, one per data domain, with split admin and usage roles.</strong> This is what makes the breach safe harbour argument defensible rather than theoretical.</li>

<li><strong>Write the retention period down first, configure second.</strong> Policy and infrastructure should agree exactly, in both directions.</li>

<li><strong>Ship audit logs to a separate account with Object Lock and CloudTrail validation enabled.</strong> Immutability and integrity are what turn logs into evidence.</li>

<li><strong>Keep PHI out of everything that does not need it.</strong> De-identify early, tokenise where you can, and route non-clinical traffic through infrastructure that is not in scope.</li>

<li><strong>Define everything in Terraform or OpenTofu.</strong> A reviewable, version-controlled definition of your controls is worth more to an assessor than any screenshot, and it stops drift being invisible.</li>

<li><strong>Rehearse the restore and the breach response.</strong> Both are required, both are tested by asking for the record, and both are cheap to evidence if you actually do them.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is AWS HIPAA compliant?</h3>



<p class="wp-block-paragraph">Not on its own, and the phrasing is the problem. AWS offers HIPAA-eligible services and will sign a Business Associate Addendum, which means you can build a compliant system on it. Compliance is a property of your whole environment, including configuration, policies, vendors and staff. No provider can sell it to you as a finished product.</p>



<h3 class="wp-block-heading">How do I sign a BAA with AWS?</h3>



<p class="wp-block-paragraph">Through AWS Artifact in the console. It is self-service and there is no additional charge. Accept it for an individual account under account agreements, or from the management account of an AWS Organization under organization agreements so all current and future member accounts are covered. It should be accepted by someone with authority to bind your organisation, and it must be in place before any PHI reaches AWS.</p>



<h3 class="wp-block-heading">Does HIPAA require six years of CloudTrail logs?</h3>



<p class="wp-block-paragraph">No. The six-year requirement is a documentation retention rule covering the policies, procedures and records the Security Rule requires you to maintain, kept for six years from creation or from when they were last in effect. The audit controls standard requires the mechanism to record and examine activity but does not name a retention period for the logs themselves. You set that period in your own policy, justify it, and make the configuration match.</p>



<h3 class="wp-block-heading">Which AWS services can I use with PHI?</h3>



<p class="wp-block-paragraph">Only those on the AWS HIPAA Eligible Services Reference, and only in accounts covered by your BAA. Check the list before adopting anything new, read the feature-level exclusions noted against individual services, and re-check periodically because entries are added over time. Nothing in AWS will stop you using a non-eligible service with PHI, so this has to be an explicit process on your side.</p>



<h3 class="wp-block-heading">If encrypted PHI is exposed, do I still have to report a breach?</h3>



<p class="wp-block-paragraph">Generally no, provided the encryption meets the standard in HHS guidance and the decryption keys were not compromised along with the data. The Breach Notification Rule applies to unsecured PHI, and properly encrypted data does not meet that definition. This is why key management, not just enabling encryption, is the part that determines whether the protection is real. You still document the incident and the assessment.</p>



<h3 class="wp-block-heading">Are the new HIPAA Security Rule requirements in force?</h3>



<p class="wp-block-paragraph">Not at the time of writing. The proposals to make all implementation specifications required and to mandate encryption and multi-factor authentication came from a Notice of Proposed Rulemaking published in January 2025. The comment period has closed, but no final rule has been issued and the timeline has moved. Treat any specific compliance deadline you see quoted with suspicion and check the current status directly.</p>



<h3 class="wp-block-heading">Is a HIPAA-compliant AWS environment expensive to run?</h3>



<p class="wp-block-paragraph">The controls themselves are mostly cheap. KMS keys, CloudTrail validation, Config rules and account separation cost very little. The real costs are log storage volume, CloudTrail data events on busy buckets, running non-production environments to production standard, and staff time on risk analysis and evidence collection. Reducing what is in scope is the most effective cost lever, which is another reason to keep non-clinical workloads out of the regulated accounts.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">HIPAA compliance on AWS fails at the seams, not at the controls. Your encryption works. Your IAM policies are tight. What goes wrong is that the obligation follows the data into an account nobody added to the BAA, a Region nobody assessed, a staging database seeded from production, a vendor nobody signed an agreement with, or a retention period nobody wrote down.</p>



<p class="wp-block-paragraph">So build the boundary technically rather than trusting it contractually. Isolate PHI into its own accounts, wrap those accounts in guardrails that make the contract boundary enforceable, own your keys so the encryption means something legally, and make your logs provable rather than merely present. Then write the whole thing down, because in this domain the documentation genuinely is part of the control.</p>



<p class="wp-block-paragraph">None of that is exotic engineering. It is ordinary AWS work applied to a boundary that no dashboard draws for you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Working on a healthcare workload on AWS?</h2>



<p class="wp-block-paragraph">This is the kind of work I do. If you are building or inheriting a PHI environment on AWS, I can help with:</p>



<ul class="wp-block-list">
<li><strong>Scope and boundary review:</strong> mapping which accounts, Regions, services and vendors actually touch PHI, and finding the ones nobody knew about.</li>

<li><strong>Account and OU design with enforceable guardrails:</strong> SCPs, organization-level BAA coverage, and Region and service restrictions that hold without breaking your workloads.</li>

<li><strong>KMS key architecture:</strong> customer managed keys per data domain, split administration and usage, and key policies written so the breach safe harbour argument stands up.</li>

<li><strong>Audit logging that produces evidence:</strong> centralised CloudTrail with log file validation, an isolated archive account with Object Lock, and retention that matches your written policy exactly.</li>

<li><strong>Backup, restore and contingency testing:</strong> encrypted cross-account copies that stay inside your boundary, plus rehearsed restores documented as compliance artefacts.</li>

<li><strong>Terraform or OpenTofu modules for the whole control set,</strong> so your posture is reviewable, repeatable and does not drift between audits.</li>
</ul>



<p class="wp-block-paragraph">If you would rather start with something concrete than a discovery call, send me a redacted account structure, an SCP that is causing trouble, or the output of a Config or Security Hub run, and I will tell you what I would look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/">HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
