<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>AWS KMS | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/aws-kms/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/aws-kms/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 07 Nov 2026 07:00:02 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>AWS KMS | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/aws-kms/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Your Admin Got Phished. Can They Still Delete Your AWS Backups?</title>
		<link>https://john-nessime.com/blog/cloud-security/aws-backup-vault-lock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 07 Nov 2026 07:00:02 +0000</pubDate>
				<category><![CDATA[Backup and Recovery]]></category>
		<category><![CDATA[Cloud Governance]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Backup]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Backup Retention]]></category>
		<category><![CDATA[Break-Glass Access]]></category>
		<category><![CDATA[Cross-Account Backup]]></category>
		<category><![CDATA[Disaster Recovery]]></category>
		<category><![CDATA[EBS Snapshots]]></category>
		<category><![CDATA[Immutable Storage]]></category>
		<category><![CDATA[Incident Response]]></category>
		<category><![CDATA[Logically Air-Gapped Vault]]></category>
		<category><![CDATA[Multi-Party Approval]]></category>
		<category><![CDATA[Ransomware Protection]]></category>
		<category><![CDATA[Restore Testing]]></category>
		<category><![CDATA[Service Control Policies]]></category>
		<category><![CDATA[WORM Storage]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=1134</guid>

					<description><![CDATA[<p>AWS Backup Vault Lock stops anyone deleting recovery points, root included. It doesn't stop a compromised admin deleting the KMS key, halting new backups or closing the account. Here's how to close each gap.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/aws-backup-vault-lock/">Your Admin Got Phished. Can They Still Delete Your AWS Backups?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">It usually comes up in a tabletop exercise, or right after a phishing report lands in someone&#8217;s inbox. A person asks the plain question: <em>if an attacker had our admin role for an afternoon, could they delete the backups?</em> The room goes quiet, because in most AWS accounts the answer is yes. The backups live in the same account as the workloads, and the role that can manage one can delete the other.</p>



<p class="wp-block-paragraph"><strong>AWS Backup Vault Lock</strong> is the feature built to change that answer. It puts a backup vault into a write-once, read-many state: recovery points can&#8217;t be deleted or have retention shortened, not even by root. It works. The catch is that the lock protects exactly one thing, the recovery points inside the vault, and a patient attacker with admin rights has several other ways to leave you without a usable restore.</p>



<p class="wp-block-paragraph">Below: how the two lock modes behave, the gaps a compromised admin can still use, a setup that closes them, and the problems you&#8217;ll hit once the lock is on.</p>



<h2 class="wp-block-heading">What AWS Backup Vault Lock actually protects</h2>



<p class="wp-block-paragraph">A vault lock does three things. It blocks deletion of recovery points in the vault. It blocks changes to their lifecycle, so nobody can shorten retention to make a backup expire early. And, if you set them, it enforces a minimum and maximum retention on every new backup or copy job that targets the vault. Jobs outside that range fail rather than land.</p>



<p class="wp-block-paragraph">Two details matter more than the documentation makes them look:</p>



<ul class="wp-block-list">
<li>The retention limits only apply to <strong>new</strong> jobs. Recovery points already in the vault when you lock it keep whatever lifecycle they had.</li>

<li>Each recovery point still expires on its own schedule. The lock guarantees a backup survives until its retention ends. It does not keep backups forever, and it does not keep new ones coming.</li>
</ul>



<h3 class="wp-block-heading">Governance mode vs compliance mode</h3>



<p class="wp-block-paragraph">There are two modes, and the difference is who can take the lock off.</p>



<ul class="wp-block-list">
<li><strong>Governance mode</strong> can be removed by any principal with the <code>backup:DeleteBackupVaultLockConfiguration</code> permission. It stops accidents and casual deletion. It does not stop an attacker holding a broad admin role, because that role almost certainly includes the permission.</li>

<li><strong>Compliance mode</strong> starts with a grace period (AWS calls it the cooling-off period) of at least three days. During that window you can still change or delete the lock. Once it ends, the lock is immutable. No user, no root account and not AWS can remove it or change it while the vault holds recovery points.</li>
</ul>



<p class="wp-block-paragraph">The API makes the choice in an unusual way. There&#8217;s no mode parameter. If you include <code>ChangeableForDays</code>, you get compliance mode. Leave it out and you get governance mode. That asymmetry is easy to get wrong in Terraform or CloudFormation, so read your plan output before applying.</p>



<pre class="wp-block-code"><code># Governance mode: no --changeable-for-days
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name prod-backups-locked 
  --min-retention-days 14 
  --max-retention-days 400

# Compliance mode: --changeable-for-days sets the grace period (minimum 3)
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name prod-backups-locked 
  --min-retention-days 14 
  --max-retention-days 400 
  --changeable-for-days 7</code></pre>



<p class="wp-block-paragraph">The numbers are placeholders. Minimum retention should come from how long an intrusion could go unnoticed; maximum from any rule that says data must be destroyed.</p>



<h2 class="wp-block-heading">Seven ways a compromised admin still beats a locked vault</h2>



<p class="wp-block-paragraph">A lock on the vault is necessary. It isn&#8217;t sufficient. Each of these is a separate path with its own control.</p>



<h3 class="wp-block-heading">1. The lock is in governance mode</h3>



<p class="wp-block-paragraph">If the attacker&#8217;s role can call <code>DeleteBackupVaultLockConfiguration</code>, a governance lock is one API call away from gone. Governance mode is still useful as a staging step and as protection against a tired engineer, but if your threat is a stolen admin session, it isn&#8217;t the control you think it is.</p>



<h3 class="wp-block-heading">2. The lock is still inside its grace period</h3>



<p class="wp-block-paragraph">Here&#8217;s a trap in the console and the API. <code>DescribeBackupVault</code> reports <code>"Locked": true</code> as soon as you apply a compliance lock, including during the grace period. Deletes of recovery points do fail during that time. But the lock itself can still be removed until the <code>LockDate</code> passes. A dashboard showing &#8220;Locked: true&#8221; is not proof of immutability. Check the date.</p>



<pre class="wp-block-code"><code>aws backup describe-backup-vault 
  --backup-vault-name prod-backups-locked 
  --query '{Locked:Locked,LockDate:LockDate,Min:MinRetentionDays,Max:MaxRetentionDays}'</code></pre>



<p class="wp-block-paragraph">If <code>LockDate</code> is in the future, you&#8217;re still in the window where an attacker (or you) can undo it.</p>



<h3 class="wp-block-heading">3. The attacker goes after the key, not the vault</h3>



<p class="wp-block-paragraph">This is the one that stays invisible until a restore fails. Vault Lock stops deletion of recovery points. It says nothing about the KMS keys that encrypt them. An encrypted backup whose key has been deleted is a very well-protected pile of noise.</p>



<p class="wp-block-paragraph">Two encryption behaviors matter here:</p>



<ul class="wp-block-list">
<li>Resources that support full AWS Backup management (S3, DynamoDB with advanced backup, EFS and others) are encrypted with the <strong>vault&#8217;s</strong> KMS key. Protect that key and you protect those backups.</li>

<li>Resources that don&#8217;t (EBS snapshots and RDS snapshots are the common ones) inherit the <strong>source resource&#8217;s</strong> key in the backup they create in the same account. The backup is only as safe as the key on the production volume or database.</li>
</ul>



<p class="wp-block-paragraph">A compromised admin with <code>kms:ScheduleKeyDeletion</code> can start the clock on that key. KMS enforces a waiting period before deletion, which gives you time to cancel, but only if something alerts you. <code>kms:DisableKey</code> is quieter and takes effect immediately, though it&#8217;s reversible. Put both behind explicit denies and alert on both.</p>



<h3 class="wp-block-heading">4. The attacker stops new backups and waits</h3>



<p class="wp-block-paragraph">The lock guarantees old backups survive until their retention ends. It does nothing to guarantee new backups get made. An attacker can edit the backup plan, delete a resource selection, or remove tags that a plan selects on, then simply wait. If your retention is seven days and nobody notices for eight, every clean copy has aged out on schedule, fully compliant with the lock.</p>



<p class="wp-block-paragraph">This is why minimum retention should be set against detection time, not against how much storage you&#8217;d like to pay for. It&#8217;s also why changes to backup plans need an alert, even if you allow them.</p>



<h3 class="wp-block-heading">5. The account is closed</h3>



<p class="wp-block-paragraph">If an account holding a backup vault is closed, AWS suspends it for 90 days with backups intact. If it isn&#8217;t reopened, AWS deletes the vault contents, <strong>even with a vault lock in place</strong>. The account itself is part of the attack surface.</p>



<p class="wp-block-paragraph">The practical answer is to keep the locked copies in an account whose root user and closure path are controlled by different people than your workload admins.</p>



<h3 class="wp-block-heading">6. Restores are blocked without deleting anything</h3>



<p class="wp-block-paragraph">EC2 backups are AMIs, and <code>ec2:DisableImage</code> can disable one. It stays in the locked vault, but you can&#8217;t restore from it until it&#8217;s re-enabled. Recoverable, but better found in a drill than mid-incident. Deny it alongside the backup actions.</p>



<h3 class="wp-block-heading">7. The backups were never restorable in the first place</h3>



<p class="wp-block-paragraph">Not an attack, but the same outcome. Snapshots of a database that ransomware had already encrypted are immutable garbage. Immutability protects the copy, not its usefulness. Only a timed restore tells you that.</p>



<h2 class="wp-block-heading">A setup that holds up, in the order I&#8217;d build it</h2>



<p class="wp-block-paragraph">For a small team this doesn&#8217;t need to be elaborate. Put the locked copy where the compromised role can&#8217;t reach, and make every remaining gap noisy.</p>



<ol class="wp-block-list">
<li><strong>Create a separate backup account in AWS Organizations.</strong> Workloads back up locally as they do now, then a copy rule sends recovery points to a vault in the backup account. The workload admin role has no access there.</li>

<li><strong>Give the backup vault a customer managed KMS key owned by the backup account.</strong> Cross-account copies are re-encrypted with the destination vault&#8217;s key, which breaks the dependency on the production key from gap 3. Check the source side too: backups encrypted with AWS managed keys such as <code>aws/ebs</code> or <code>aws/rds</code> can&#8217;t be copied across accounts, so those resources need a customer managed key.</li>

<li><strong>Lock the destination vault in governance mode first.</strong> Run at least one full backup and copy cycle, confirm jobs succeed with your retention settings, and do a test restore from the copy.</li>

<li><strong>Switch to compliance mode with a generous grace period.</strong> The minimum is three days. There&#8217;s no reason to use the minimum on your first lock. Put the <code>LockDate</code> in someone&#8217;s calendar, and review the configuration one more time before it passes.</li>

<li><strong>Apply service control policies</strong> that deny destructive backup and KMS actions to everyone except a named break-glass role (example below).</li>

<li><strong>Alert on the actions you allow.</strong> Plan changes, selection deletes and key state changes should page someone, because they&#8217;re exactly what gap 4 looks like.</li>

<li><strong>Schedule restore tests.</strong> AWS Backup has a restore testing feature that restores recovery points on a schedule and records how long it took. Use it, or run the restore by hand, but make it recurring.</li>
</ol>



<h3 class="wp-block-heading">The logically air-gapped vault option</h3>



<p class="wp-block-paragraph">AWS also offers a logically air-gapped vault, which bundles much of the above into one resource. It comes with a compliance-mode lock built in, uses an AWS owned key by default (a customer managed key is optional), and can be shared to other accounts through AWS RAM so you can restore from a clean account.</p>



<p class="wp-block-paragraph">Its strongest feature for this threat model is Multi-party approval. You set up an approval team in a separate recovery organization, backed by IAM Identity Center, and that team can authorize access to the vault even if the owning account is compromised or unreachable. That addresses gap 5 far better than anything you&#8217;d build by hand.</p>



<p class="wp-block-paragraph">The trade-off is setup effort and a second organization to govern. For one or two production accounts, a compliance-locked standard vault in a separate backup account is often enough. If auditors or your insurer ask about recovery from a fully compromised organization, the air-gapped vault is the cleaner answer.</p>



<h3 class="wp-block-heading">Service control policy for the backup guardrails</h3>



<p class="wp-block-paragraph">This SCP denies the actions that remove protection or quietly make backups unusable. It exempts one break-glass role. Adjust the role name to your own, and test it in a non-production OU first, because a bad SCP can lock out your own automation.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ProtectBackupsAndKeys",
      "Effect": "Deny",
      "Action": [
        "backup:DeleteBackupVault",
        "backup:DeleteBackupVaultLockConfiguration",
        "backup:DeleteRecoveryPoint",
        "backup:UpdateRecoveryPointLifecycle",
        "backup:PutBackupVaultAccessPolicy",
        "backup:DeleteBackupVaultAccessPolicy",
        "kms:ScheduleKeyDeletion",
        "kms:DisableKey",
        "ec2:DisableImage"
      ],
      "Resource": "*",
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalArn": "arn:aws:iam::*:role/BackupBreakGlass"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Two limits to know. SCPs don&#8217;t apply to the Organizations management account, which is one more reason the backups shouldn&#8217;t live there. And SCPs don&#8217;t restrict service-linked roles. Before you attach anything, paste the policy into my <a href="https://john-nessime.com/tools/aws-iam-policy-checker/">free AWS IAM Policy Checker</a> to catch wildcards and typos in action names.</p>



<p class="wp-block-paragraph">I&#8217;ve left <code>UpdateBackupPlan</code> and <code>DeleteBackupSelection</code> out of the deny on purpose. If your plans are managed through Terraform or CloudFormation, blocking them breaks your own pipeline. Restrict them to the pipeline&#8217;s role instead, and alert on every call.</p>



<h3 class="wp-block-heading">Alerting on the quiet changes</h3>



<p class="wp-block-paragraph">An EventBridge rule can match CloudTrail management events for the calls that matter. You need a CloudTrail trail recording management events for these to arrive. A pattern like this covers the backup side:</p>



<pre class="wp-block-code"><code>{
  "source": ["aws.backup"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventName": [
      "UpdateBackupPlan",
      "DeleteBackupPlan",
      "DeleteBackupSelection",
      "DeleteBackupVaultLockConfiguration",
      "PutBackupVaultLockConfiguration"
    ]
  }
}</code></pre>



<p class="wp-block-paragraph">Add a second rule with <code>"source": ["aws.kms"]</code> for <code>ScheduleKeyDeletion</code> and <code>DisableKey</code>. Route both to SNS or your paging tool, such as PagerDuty or Opsgenie. AWS Backup Audit Manager also has a control checking that resources sit in a backup plan with a Vault Lock, which catches resources nobody enrolled.</p>



<h2 class="wp-block-heading">Troubleshooting a locked vault</h2>



<h3 class="wp-block-heading">Backup jobs started failing right after the lock</h3>



<p class="wp-block-paragraph">Almost always a retention mismatch. Any job whose lifecycle is shorter than <code>MinRetentionDays</code> or longer than <code>MaxRetentionDays</code> fails. Check every backup rule and every copy rule that targets the vault, not just the obvious one.</p>



<p class="wp-block-paragraph">Continuous backups are the usual culprit. They can&#8217;t retain longer than 35 days, so a vault with a minimum retention above that will reject them. Send continuous backups to a separate vault with its own lock settings.</p>



<h3 class="wp-block-heading">Cross-account copy jobs fail</h3>



<p class="wp-block-paragraph">Check how the source is encrypted. Resources that aren&#8217;t fully managed by AWS Backup and use an AWS managed key can&#8217;t be copied to another account. You&#8217;ll need to move them to a customer managed key, which for EBS and RDS usually means copying a snapshot with a new key and restoring from it. Plan that as a change, not a quick fix.</p>



<h3 class="wp-block-heading">You can&#8217;t remove the lock</h3>



<p class="wp-block-paragraph">If <code>DeleteBackupVaultLockConfiguration</code> returns <code>InvalidRequestException</code>, the grace period has ended. There&#8217;s no support ticket that reverses it. The vault can be deleted only once it&#8217;s empty, which means waiting for every recovery point to expire.</p>



<h3 class="wp-block-heading">Storage keeps growing on a vault you want to retire</h3>



<p class="wp-block-paragraph">Look for recovery points with a retention of &#8220;Always&#8221;. AWS warns these become permanent once the grace period ends. Audit lifecycles before the lock date, not after.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating governance mode as ransomware protection when the admin role can remove it.</li>

<li>Trusting <code>"Locked": true</code> without checking whether <code>LockDate</code> has passed.</li>

<li>Locking the vault but leaving the KMS key deletable by the same role that was compromised.</li>

<li>Setting a short minimum retention to save storage, shorter than your realistic detection time.</li>

<li>Keeping the only locked copy in the same account, or in the Organizations management account.</li>

<li>Going straight to compliance mode with the three-day minimum grace period and an untested backup plan.</li>

<li>Never restoring from the locked copy, so the first real test happens during the incident.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth keeping</h2>



<ul class="wp-block-list">
<li>Keep locked copies in a dedicated backup account with its own KMS key and very few people who can reach it.</li>

<li>Keep the break-glass role&#8217;s credentials in a password manager such as 1Password or Bitwarden, behind hardware MFA keys like YubiKey.</li>

<li>Record restore times from every test, so you know your real recovery time rather than a guess.</li>

<li>Consider a third copy outside AWS entirely if a single-provider failure is in your risk register. Backblaze B2 and Wasabi both support object lock on their storage, and backup products such as Veeam and N2WS can manage that copy for you. That&#8217;s more to operate, so weigh it against what you&#8217;re protecting.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Does AWS Backup Vault Lock cost extra?</h3>



<p class="wp-block-paragraph">The lock itself has no separate charge. You pay normal AWS Backup storage, and because locked backups can&#8217;t be deleted early, you pay for their full retention.</p>



<h3 class="wp-block-heading">Can the root user delete backups in a compliance-mode locked vault?</h3>



<p class="wp-block-paragraph">No. Once the grace period ends, neither the root user nor AWS can delete recovery points or remove the lock while the vault contains recovery points. Closing the account is the exception: AWS deletes vault contents 90 days after closure if the account isn&#8217;t reopened.</p>



<h3 class="wp-block-heading">What&#8217;s the difference between governance and compliance mode?</h3>



<p class="wp-block-paragraph">Governance mode can be removed by anyone with the right IAM permission. Compliance mode becomes permanent after a grace period of at least three days. Use governance to test, compliance to protect.</p>



<h3 class="wp-block-heading">Is Vault Lock the same as S3 Object Lock?</h3>



<p class="wp-block-paragraph">No. Vault Lock covers AWS Backup vaults and their recovery points. S3 Object Lock covers objects in an S3 bucket. Neither is the older Glacier vault lock.</p>



<h3 class="wp-block-heading">Can I shorten retention after I lock the vault?</h3>



<p class="wp-block-paragraph">Not for recovery points already in a compliance-locked vault. You can change the retention in your backup plan for future backups, as long as the new value still falls within the vault&#8217;s minimum and maximum.</p>



<h3 class="wp-block-heading">Do I need a logically air-gapped vault, or is a locked standard vault enough?</h3>



<p class="wp-block-paragraph">A locked standard vault in a separate account covers most small and mid-size setups. The air-gapped vault adds Multi-party approval for recovery when the owning account or organization is compromised. If that scenario is in scope for you, it&#8217;s the better choice.</p>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">AWS Backup Vault Lock stops a compromised admin from deleting your recovery points. It doesn&#8217;t stop them from deleting the key, stopping new backups, or closing the account. Lock the vault in compliance mode, but put it in a separate account, deny key deletion, alert on plan changes, and prove the copy restores. The lock is the last line. Make sure it isn&#8217;t the only one.</p>



<p class="wp-block-paragraph">To put a number on an unrecoverable afternoon before the storage budget conversation, try the <a href="https://john-nessime.com/tools/downtime-cost-calculator/">Downtime Cost Calculator</a>.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your AWS backups?</h2>



<p class="wp-block-paragraph">This is the work I do on <a href="https://john-nessime.com/aws/backup-recovery/">AWS backup and recovery</a> engagements. I can help with:</p>



<ul class="wp-block-list">
<li>Checking if your current backups survive a compromised admin role, gap by gap as above.</li>

<li>Designing a separate backup account with cross-account copies and the KMS keys to match.</li>

<li>Moving EBS and RDS resources off AWS managed keys so their backups can be copied.</li>

<li>Writing and testing the SCPs and EventBridge alerts in a non-production OU before rollout.</li>

<li>Choosing between a locked standard vault and a logically air-gapped vault with Multi-party approval.</li>

<li>Running a timed restore from the locked copy and writing down what came back and how long it took.</li>
</ul>



<p class="wp-block-paragraph">Send me the output of <code>aws backup describe-backup-vault</code> for your main vault and a copy of one backup plan, and I&#8217;ll tell you which of the seven gaps are open.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/cloud-resilience-review/">See the Cloud Resilience Review</a></div>



<div class="wp-block-button is-style-outline is-style-outline--1"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/aws-backup-vault-lock/">Your Admin Got Phished. Can They Still Delete Your AWS Backups?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</title>
		<link>https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 03 Nov 2026 07:06:02 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Amazon ECS]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[AWS Secrets Manager]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[HashiCorp Vault]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Key Rotation]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Parameter Store]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Terraform]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=442</guid>

					<description><![CDATA[<p>Most teams pick between AWS Secrets Manager and SSM Parameter Store on price, then get bitten months later by rotation semantics, a shared throughput quota, or a recovery window that blocks a redeploy. Here are the four levers that actually decide it, per value, plus a decision procedure you can run against a real inventory.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/">AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pull request comment was one line: why is the database password in Parameter Store and the Stripe key in Secrets Manager?</p>



<p class="wp-block-paragraph">Nobody had a good answer. Both were added in a hurry, months apart, by different people, and each choice made sense on the day it was made. That is how most AWS accounts end up with secrets in two places. Not by design, by accumulation.</p>



<p class="wp-block-paragraph">The comparison usually gets framed as a pricing question: Parameter Store&#8217;s standard tier has no storage charge, Secrets Manager bills per secret, so use Parameter Store until somebody complains. That framing is wrong, and it stays invisible for months before it costs you anything. What separates these two services is what happens when a value <em>changes</em>, not what it costs to sit there.</p>



<p class="wp-block-paragraph">This post covers the failure modes that actually bite, the four levers that decide <strong>AWS Secrets Manager vs Parameter Store</strong> for any given value, a decision procedure you can run against a real inventory, and the arguments about this that do not survive contact with a running system.</p>



<h2 class="wp-block-heading">The two failure modes that actually bite</h2>



<h3 class="wp-block-heading">Rotation that succeeds while your application quietly breaks</h3>



<p class="wp-block-paragraph">Secrets Manager&#8217;s single user rotation strategy changes the password on the existing database user. AWS documents a short window between the password changing in the database and the secret being updated, during which calls using the rotated credentials can be denied. Retry logic covers that window.</p>



<p class="wp-block-paragraph">That is the documented gap. The one that catches people is quieter.</p>



<p class="wp-block-paragraph">Your application fetched the credential once at boot, cached it in a module-level variable, and never looked again. Rotation completes. The rotation function logs success. Nothing breaks. Then a container recycles, picks up the new value and works fine, while older containers keep running on connections opened before the change. It looks healthy right up until a connection pool churns and part of your fleet starts failing authentication against a service you have not deployed in weeks.</p>



<p class="wp-block-paragraph">The alternating users strategy exists for exactly this. Instead of changing one user&#8217;s password, Secrets Manager clones the user and alternates which one it updates, so a valid credential exists on both sides of every change. The price is a second database user and a separate superuser secret with permission to do the cloning. AWS recommends single user for most cases and for interactive or one-off credentials, and alternating users where the application needs high availability.</p>



<p class="wp-block-paragraph">The rule I use: if consumers cache and you rotate on a schedule, you need alternating users or an explicit refresh path. A shorter cache TTL narrows the window, it does not close it.</p>



<h3 class="wp-block-heading">A throughput ceiling you do not own</h3>



<p class="wp-block-paragraph">Parameter Store read throughput is a quota on the account and Region, not on the parameter and not on the caller. Every cold start, every task boot, every CI job in that account draws from the same budget. The default is sized for low-volume workloads, and there is an opt-in higher throughput setting that raises the ceiling for an additional charge.</p>



<p class="wp-block-paragraph">The failure presents as an application bug. A scale-out event or a chatty pipeline fans out, <code>GetParameter</code> starts returning <code>ThrottlingException</code>, and services fail at boot for reasons unrelated to the code anybody just shipped. Nothing in the console tells you that you are near the line.</p>



<p class="wp-block-paragraph">Both services have quotas. The difference is blast radius: Secrets Manager throttling is annoying, Parameter Store throttling is a shared resource problem where one team&#8217;s new batch job degrades another team&#8217;s deploys. Alarm on throttling for both before you need to. CloudWatch does it natively, and if your dashboards live in Grafana Cloud or a self-hosted Grafana, pull the metric across so the alert lands where people actually look.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where Parameter Store wins, and where it hurts</h2>



<p class="wp-block-paragraph">Parameter Store is not primarily a secrets store. It is a hierarchical configuration store that encrypts when you ask it to. Hold that framing and its behaviour becomes predictable.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong></p>



<ul class="wp-block-list">
<li><strong>Configuration that is not confidential.</strong> AMI IDs, feature flags, queue names, tuning values. Putting these in Secrets Manager means paying a per-secret charge for data with no confidentiality requirement.</li>



<li><strong>Hierarchies and bulk reads.</strong> Path-based naming plus <code>GetParametersByPath</code> pulls a whole subtree in one call, which is what a service wants at startup.</li>



<li><strong>No storage charge on the standard tier,</strong> so nobody is incentivized to cram unrelated values together to save money.</li>



<li><strong>Parameter policies and AWS public parameters</strong> live here, including read-only values AWS publishes for things like current AMI IDs.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it hurts:</strong></p>



<ul class="wp-block-list">
<li><strong>No rotation.</strong> You write the function, schedule it, and own verification and rollback yourself.</li>



<li><strong>CloudFormation cannot create an encrypted parameter.</strong> The <code>AWS::SSM::Parameter</code> resource accepts <code>String</code> and <code>StringList</code> only. You can reference an existing encrypted parameter through the <code>ssm-secure</code> dynamic reference, and only on the resource properties that support that pattern.</li>



<li><strong>No recovery window.</strong> Delete is delete, and parameter history does not survive deleting the parameter.</li>



<li><strong>Cross-account sharing requires the advanced tier,</strong> through Resource Access Manager.</li>
</ul>



<h2 class="wp-block-heading">Where AWS Secrets Manager wins, and where it hurts</h2>



<p class="wp-block-paragraph">Secrets Manager is a lifecycle service. Storage is the least interesting thing it does.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong></p>



<ul class="wp-block-list">
<li><strong>Managed rotation</strong> for Amazon RDS, Aurora, Redshift and DocumentDB, using AWS-published rotation functions and two documented strategies.</li>



<li><strong>Versioning with staging labels.</strong> <code>AWSCURRENT</code>, <code>AWSPENDING</code> and <code>AWSPREVIOUS</code> give rotation a defined state machine and give you a last known good version to fall back to.</li>



<li><strong>A recovery window on delete,</strong> between seven and thirty days, defaulting to thirty.</li>



<li><strong>Cross-region replication and resource policies</strong> as first-class features, so multi-Region and cross-account access are properties of the secret rather than things you build.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it hurts:</strong></p>



<ul class="wp-block-list">
<li><strong>Cost scales by multiplication.</strong> Storage bills per secret per month, prorated hourly, and every cross-region replica bills as its own secret. The multiplier is secrets times environments times Regions.</li>



<li><strong>The recovery window is a deployment trap.</strong> Destroy an environment, apply again, and the create fails because the name is still reserved.</li>



<li><strong>Managed rotation only covers databases.</strong> Anything else means writing your own function, so you carry the operational burden without the free tier.</li>



<li><strong>The pricing model pushes you toward bundling.</strong> One secret per value is the clean design and the expensive one, so teams pack fields into JSON and lose per-value IAM granularity.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Four levers that decide AWS Secrets Manager vs Parameter Store</h2>



<h3 class="wp-block-heading">1. Does the value have to change on a schedule?</h3>



<p class="wp-block-paragraph">If yes, and it is a credential for RDS, Aurora, Redshift or DocumentDB, use Secrets Manager and stop deliberating. You are not paying for storage. You are paying to avoid writing, testing and maintaining a rotation function, and to inherit a staging-label state machine that makes a failed rotation recoverable instead of an outage.</p>



<p class="wp-block-paragraph">If the value never changes on a schedule, an AMI ID, a webhook URL, a vendor key you replace manually once a year, then rotation machinery buys you nothing.</p>



<h3 class="wp-block-heading">2. How many independent gates guard the read path?</h3>



<p class="wp-block-paragraph">This is the lever people skip, and it matters more than the service choice.</p>



<p class="wp-block-paragraph">Both services encrypt through KMS. The question is whether KMS is a second, independent authorization decision or a formality. An encrypted parameter under the AWS managed key for Systems Manager is decrypted through that key on behalf of the service, so in practice the SSM permission is the gate. Encrypt under a customer managed key instead and the caller must also be allowed <code>kms:Decrypt</code> on that specific key. Now two policies in two services both have to say yes, and you can scope the key policy to a named set of roles independently of who may call Systems Manager. The first gate protects the API surface, the second protects the plaintext.</p>



<p class="wp-block-paragraph">The same split exists in Secrets Manager, between the AWS managed key and one you create.</p>



<pre class="wp-block-code"><code># Store an encrypted parameter under a customer managed key
# instead of the AWS managed default.
aws ssm put-parameter 
  --name "/prod/billing/db-password" 
  --type SecureString 
  --key-id "alias/prod-app-secrets" 
  --value "REPLACE_ME" 
  --overwrite

# Reading it now needs ssm:GetParameter AND
# kms:Decrypt on alias/prod-app-secrets.
aws ssm get-parameter 
  --name "/prod/billing/db-password" 
  --with-decryption 
  --query "Parameter.Value" 
  --output text</code></pre>



<p class="wp-block-paragraph">There is a second payoff. A customer managed key makes the decrypt call attributable in CloudTrail against a key policy you control, which turns &#8220;someone read a secret&#8221; into &#8220;this role decrypted this value with this key at this time&#8221;. That is usually what an auditor is asking for, and it is independent of which store you land on.</p>



<h3 class="wp-block-heading">3. What does the retrieval path look like under load?</h3>



<p class="wp-block-paragraph">Neither service is designed to be called on every request. Both punish you for it in different currencies: Parameter Store with a shared throughput ceiling, Secrets Manager with a per-call charge.</p>



<p class="wp-block-paragraph">The answer for both is caching, and AWS ships the mechanism rather than making you build it. The Secrets Manager Agent, now published under the name AWS Workload Credentials Provider, runs as a sidecar or local service, holds secrets in memory, and serves them to your application over a local HTTP endpoint, so the app makes one call instead of thousands and needs no AWS SDK dependency to do it. For Lambda, the AWS Parameters and Secrets Lambda Extension does the equivalent inside the execution environment and covers both services.</p>



<p class="wp-block-paragraph">Caching moves the problem rather than removing it. A cached credential is a stale credential during rotation, which loops straight back to lever one.</p>



<h3 class="wp-block-heading">4. How does the value get created and destroyed?</h3>



<p class="wp-block-paragraph">Creation first. CloudFormation cannot create an encrypted parameter, so if your pipeline is CloudFormation or CDK and the value must be encrypted from the moment it exists, Secrets Manager is the path of least resistance. The usual workarounds, a Lambda-backed custom resource or creating the parameter out of band, both move the secret outside the template that claims to own it. That is a real cost, not a detail.</p>



<p class="wp-block-paragraph">Destruction is where the surprise lives. Secrets Manager schedules deletion after a recovery window with a minimum of seven days and a default of thirty, and the name stays reserved for the duration.</p>



<pre class="wp-block-code"><code># Shorten the recovery window to the documented minimum.
aws secretsmanager delete-secret 
  --secret-id prod/billing/db 
  --recovery-window-in-days 7

# Or skip the window entirely. There is no undo after this.
aws secretsmanager delete-secret 
  --secret-id ephemeral/pr-1482/db 
  --force-delete-without-recovery</code></pre>



<p class="wp-block-paragraph">AWS notes the actual deletion runs as an asynchronous background process, so deleting and immediately recreating a secret with the same name needs backoff and retry even with the force flag. This is why per-branch environments built with Terraform or OpenTofu fail on their second run. Use the force flag deliberately, put a unique suffix in the name, or keep ephemeral values in Parameter Store where deletion is immediate.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Cost levers, without quoting numbers that will age</h2>



<p class="wp-block-paragraph">Published rates change. The billing mechanisms are stable, and the mechanisms are what let you predict the bill.</p>



<ul class="wp-block-list">
<li><strong>Parameter Store standard tier</strong> carries no storage charge. The advanced tier is charged per parameter, and the higher throughput setting is charged separately as an account and Region level switch.</li>



<li><strong>Secrets Manager</strong> charges per stored secret per month, prorated hourly, plus a charge per block of API calls. Replicas count as separate stored secrets.</li>



<li><strong>KMS</strong> charges a monthly fee per customer managed key plus a request charge on cryptographic operations. This applies to both services and it is the line most estimates forget.</li>



<li><strong>Rotation</strong> has no separate line item, but the rotation function bills as ordinary Lambda usage.</li>
</ul>



<p class="wp-block-paragraph">Two consequences. The Secrets Manager multiplier is not the number of secrets, it is secrets across environments across Regions, and replicas are the sneaky term because nobody remembers enabling them. And the advanced parameter tier is a one-way door: you can promote a standard parameter to advanced, but you cannot demote it, because reverting would truncate the value. Undoing that means delete and recreate.</p>



<p class="wp-block-paragraph">If you cannot see which term is growing, break the bill down by service and tag before you migrate anything. Cost Explorer handles the single-account case; for cross-account views without building the aggregation yourself, tools like Vantage or CloudZero will answer &#8220;is this secret count, replication, or retrieval frequency&#8221; in a few minutes. That is a better use of an afternoon than guessing.</p>



<h2 class="wp-block-heading">A decision procedure you can run today</h2>



<p class="wp-block-paragraph">Take an actual list of values and walk each one through this. It takes about a minute each.</p>



<ol class="wp-block-list">
<li><strong>Is it confidential at all?</strong> If no, Parameter Store, plain <code>String</code> type, done. Do not encrypt configuration for the aesthetics.</li>



<li><strong>Is it a database credential that should rotate on a schedule?</strong> Secrets Manager with managed rotation. Choose single user or alternating users based on whether consumers cache.</li>



<li><strong>Does it need scheduled rotation but is not a supported database?</strong> You are writing a function either way. Secrets Manager if you want versioning, staging labels and a recovery window; Parameter Store if the free tier matters more and you accept owning the lifecycle.</li>



<li><strong>Does it need to exist in more than one Region or account?</strong> Secrets Manager gives you replication and resource policies directly. Parameter Store shares across accounts through Resource Access Manager, advanced tier only.</li>



<li><strong>Does CloudFormation or CDK create it?</strong> Encrypted parameters are not creatable there, so it is Secrets Manager or an accepted out-of-band step.</li>



<li><strong>Is it read frequently?</strong> Cache it either way, then return to step two, because caching and rotation interact and that interaction is where outages come from.</li>



<li><strong>Everything else</strong> goes to Parameter Store, encrypted under a customer managed key.</li>
</ol>



<p class="wp-block-paragraph">Most accounts land with the large majority of values in Parameter Store and a short list of rotating database credentials in Secrets Manager. That split is healthy. What is not healthy is being unable to say why a particular value is where it is.</p>



<p class="wp-block-paragraph">Once you accept you will use both, the operational work is making them look like one system: one naming convention across both stores, IAM scoped to the path rather than the service, KMS keys separated by trust boundary rather than by service, and the rule written down where the next person will find it. Two lines in a repository README beats a policy document nobody opens.</p>



<h2 class="wp-block-heading">Arguments that do not survive contact</h2>



<p class="wp-block-paragraph"><strong>&#8220;Parameter Store is free, so use it for everything.&#8221;</strong> The standard tier has no storage charge, which is not the same as free. KMS requests bill, the higher throughput setting bills, advanced parameters bill, and the rotation function you write instead of using a managed one has a real cost in engineering hours and incident risk.</p>



<p class="wp-block-paragraph"><strong>&#8220;Secrets Manager is expensive.&#8221;</strong> Per secret it is small. The bill grows through multiplication, not unit price. Before migrating anything, count secrets across environments and Regions and find out how many replicas exist that nobody asked for. Deleting unused replicas is usually a bigger win than a migration project.</p>



<p class="wp-block-paragraph"><strong>&#8220;Just use Vault and skip the argument.&#8221;</strong> HashiCorp Vault is a better answer if you are multi-cloud, need dynamic short-lived credentials, or want secrets engines AWS does not offer. It also hands you a cluster to run, upgrade, seal and unseal, back up and monitor, plus a licensing decision. On a single-cloud AWS estate with a handful of rotating credentials, the AWS services are less capable and less work. Both statements are true at once, and which dominates depends on how much platform team you have.</p>



<p class="wp-block-paragraph"><strong>&#8220;It is encrypted at rest, so it is secure.&#8221;</strong> Encryption at rest protects against someone reading the storage layer, which is not the threat you have. Your threat is an over-permissive role and a compromised workload. That is an IAM policy and a KMS key policy question, and the answer is the same on either service.</p>



<p class="wp-block-paragraph"><strong>&#8220;We will add rotation later.&#8221;</strong> Rotation is not a feature flag, it is a coordination problem between the store, the target system and every consumer&#8217;s caching behaviour. Deciding that a credential will eventually rotate changes how you write the consumer today. Retrofitting it means touching every consumer anyway.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I migrate a value between the two without downtime?</h3>



<p class="wp-block-paragraph">Yes, with a dual read. Write the value to the new location, deploy consumers that read the new location and fall back to the old one, then use CloudTrail to confirm nothing still reads the old path before deleting it. Do not skip the CloudTrail step; the consumer you forgot is always a job that runs monthly. If the value passed through a local file during the move, erase that file properly rather than with <code>rm</code>, using <code>shred</code> on Linux or a secure deletion tool such as those from O&amp;O Software on Windows.</p>



<h3 class="wp-block-heading">Does Parameter Store support rotation?</h3>



<p class="wp-block-paragraph">Not as a built-in feature. You write a function and invoke it on a schedule, typically with an EventBridge rule. You also inherit everything managed rotation handles for you: verifying the new credential works before committing to it, rolling back cleanly when it does not, and giving consumers an overlap window.</p>



<h3 class="wp-block-heading">Which one should I use for third-party API keys?</h3>



<p class="wp-block-paragraph">It depends on whether you rotate them. Most vendor keys get rotated manually and rarely, in which case an encrypted parameter under a customer managed key is enough. If the provider supports programmatic key rotation and you want that automated, Secrets Manager&#8217;s version staging is what makes the swap safe.</p>



<h3 class="wp-block-heading">Can workloads outside AWS read from either service?</h3>



<p class="wp-block-paragraph">Both, over the API, with AWS credentials. That means a long-lived access key or a federated identity, and you should federate wherever the platform supports it. If part of your estate runs on VPS hosts at a provider like Contabo or InterServer, this is the boundary to think hardest about, because you have just placed an AWS credential on a machine whose network you do not control. Scope the role narrowly, apply condition keys where the network path allows, and treat that credential as the one most likely to leak.</p>



<h3 class="wp-block-heading">What happens if I delete something by accident?</h3>



<p class="wp-block-paragraph">Secrets Manager lets you restore during the recovery window, which is the single best argument for putting genuinely important credentials there. Parameter Store has no equivalent. Once the parameter is gone your options are whatever exists in state files, backups or someone&#8217;s terminal history, and none of those is a plan.</p>



<h3 class="wp-block-heading">Should ECS or EKS workloads pull secrets differently?</h3>



<p class="wp-block-paragraph">ECS can inject values from either service directly into a task&#8217;s environment, which removes SDK code from the container but reads the value at task start, so a rotation does not reach a running task. On EKS you have both the Secrets Store CSI driver with the AWS provider, which mounts values as files, and the agent sidecar pattern, which serves them over localhost and can refresh without restarting the pod. Choose based on whether your application can reload a credential mid-life. If it cannot, mounting and injection are equivalent, and rotation means a rollout.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>AWS Secrets Manager vs Parameter Store</strong> decision is a lifecycle decision wearing a pricing costume. What separates the two is not what they store, it is what they do when the value changes, when it gets deleted, when a Region falls over, and when ten thousand containers ask for it at once.</p>



<p class="wp-block-paragraph">Decide per value, not per project. Configuration and rarely-changing keys belong in Parameter Store. Database credentials that rotate on a schedule belong in Secrets Manager. Put a customer managed key in front of both so the second gate does real work, cache reads so you are not fighting a quota, and make sure whoever wrote the caching understands what happens the next time a rotation runs. If you cannot explain why a given value lives where it does, that is the value that will surprise you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your secrets setup?</h2>



<p class="wp-block-paragraph">Most of the work here is not choosing a service, it is untangling what already exists. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing an AWS account for where secrets actually live, which roles can decrypt them, and which values are duplicated across environments or replicated into Regions nobody uses</li>



<li>Designing a naming hierarchy and path-scoped IAM so one policy covers a service instead of one policy per value</li>



<li>Setting up managed rotation for RDS and Aurora credentials, including the alternating users pattern when the application caches and cannot be changed</li>



<li>Writing and testing custom rotation functions for non-database secrets, with the rollback path actually exercised rather than assumed</li>



<li>Moving values between the two stores with a dual-read cutover and CloudTrail verification that nothing is left reading the old path</li>



<li>Working out whether a growing Secrets Manager line item is secret count, replication or retrieval frequency, and fixing the one actually responsible</li>
</ul>



<p class="wp-block-paragraph">Send me a redacted IAM policy, a CloudTrail query result, or the output of listing your parameters by path, and I will tell you what I would change first and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/">AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</title>
		<link>https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 20 Oct 2026 06:00:04 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Append-Only Storage]]></category>
		<category><![CDATA[Audit Evidence]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Backup]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Document Retention]]></category>
		<category><![CDATA[Immutable Storage]]></category>
		<category><![CDATA[Legal Hold]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Service Control Policies]]></category>
		<category><![CDATA[WORM Storage]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=910</guid>

					<description><![CDATA[<p>Enabling S3 Object Lock on a bucket does not protect the objects already in it. This is how to build an audit evidence repository on AWS that survives the question auditors actually ask, covering retention modes you cannot undo, delete markers that hide protected versions, chain of custody, and the storage bill compliance mode makes permanent.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/">Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that ends a smooth audit call is never &#8220;do you have the logs&#8221;. It&#8217;s &#8220;show me why nobody could have edited them.&#8221;</p>



<p class="wp-block-paragraph">That is a different question, and most evidence buckets fail it quietly. Versioning is on, the bucket is encrypted, access logging is enabled, and there is still an admin role in the account that can call <code>DeleteObjectVersion</code> on every object in it. Nothing is wrong until someone asks you to prove a negative.</p>



<p class="wp-block-paragraph">An audit evidence repository on AWS is not a bucket with a good name. It is a write path, a retention decision you cannot take back, and a set of guardrails that survive the person who built them leaving the company. This post covers how to build one with S3 Object Lock and AWS Backup Vault Lock, and more usefully, the five ways I have seen the design go wrong in ways nobody notices until the evidence is needed or the bill arrives.</p>



<h2 class="wp-block-heading">What &#8220;immutable&#8221; actually buys you on AWS</h2>



<p class="wp-block-paragraph">Three separate mechanisms get called immutability, and they protect different things.</p>



<ul class="wp-block-list">
<li><strong>S3 Object Lock</strong> stops an object version from being overwritten or deleted for a defined period. It works only on versioned buckets, and once you enable it on a bucket you cannot disable it or suspend versioning again.</li>



<li><strong>AWS Backup Vault Lock</strong> does the same job for recovery points in a backup vault, with its own retention floor and ceiling enforced at the vault level.</li>



<li><strong>Integrity validation</strong>, such as CloudTrail&#8217;s digest files, proves a file has not changed since delivery. That is a separate property from being undeletable, and auditors increasingly ask for both.</li>
</ul>



<p class="wp-block-paragraph">Object Lock is the core of an audit evidence repository on AWS because evidence usually arrives as files: exported reports, signed approvals, config snapshots, log archives. Vault Lock matters when the evidence is a backup of a database or an EBS volume. Integrity validation is what turns &#8220;we kept it&#8221; into &#8220;we kept it and here is the hash chain&#8221;.</p>



<h2 class="wp-block-heading">Failure one: picking the retention mode you cannot walk back</h2>



<p class="wp-block-paragraph">Object Lock has two retention modes and the difference is not cosmetic. In governance mode, a caller who holds the <code>s3:BypassGovernanceRetention</code> permission can shorten or remove retention and delete the version. In compliance mode, <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html">nobody can</a>, including the account root user, and the retention period cannot be shortened or downgraded to governance.</p>



<p class="wp-block-paragraph">Read that second sentence again with a typo in mind. Set seven years where you meant seven days, in compliance mode, and the only exit AWS documents is closing the account. That is not a bug. It is the entire point of the mode. But it means the mode is a business decision wearing a technical costume.</p>



<p class="wp-block-paragraph">How I decide: compliance mode only where an external rule names a retention period and an external party will inspect it. Everything else starts in governance mode, with the bypass permission granted to nobody by default and denied outright in a service control policy. Governance mode with no principal holding the bypass right is functionally immutable for day-to-day purposes, and it leaves you a documented, auditable escape hatch that requires an explicit policy change to use.</p>



<p class="wp-block-paragraph">The bypass is not silent, either. Overriding governance retention requires the <code>x-amz-bypass-governance-retention:true</code> header on the request, which means the action shows up in CloudTrail as a distinctly shaped call. That is a good alarm to build.</p>



<pre class="wp-block-code"><code># Create the bucket with Object Lock on. Versioning is enabled automatically
# and cannot be suspended afterwards.
aws s3api create-bucket 
  --bucket my-evidence-archive 
  --object-lock-enabled-for-bucket 
  --region eu-west-1 
  --create-bucket-configuration LocationConstraint=eu-west-1

# Default retention for NEW objects written to the bucket.
aws s3api put-object-lock-configuration 
  --bucket my-evidence-archive 
  --object-lock-configuration 
  '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}}'</code></pre>



<p class="wp-block-paragraph">Start there. Run it for a quarter. Watch what actually lands in the bucket and how big it gets. Then decide whether compliance mode is worth its permanence.</p>



<h2 class="wp-block-heading">Failure two: retention that was never applied to the objects that matter</h2>



<p class="wp-block-paragraph">This is the invisible one, and it is the reason to read the rest of this post.</p>



<p class="wp-block-paragraph">Object Lock is enabled at the bucket level, but retention lives on the individual object version. A default retention rule applies to objects written after the rule exists. It does not reach backwards. Enable Object Lock on an existing bucket holding three years of evidence and you get a bucket that reports Object Lock as enabled while every object already in it carries no retention at all.</p>



<p class="wp-block-paragraph">The bucket console looks correct. The compliance checklist gets ticked. The old evidence is still deletable by anyone with delete permissions.</p>



<p class="wp-block-paragraph">Two things to do about it. First, backfill retention on existing versions, either object by object with <code>put-object-retention</code> or, for anything above a few thousand keys, with S3 Batch Operations, which exists precisely for applying retention at scale. Second, verify per object rather than per bucket.</p>



<pre class="wp-block-code"><code># Check what protection a specific version actually carries.
# Look for ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus.
aws s3api head-object 
  --bucket my-evidence-archive 
  --key 2024/q3/access-review-export.csv 
  --version-id "SOME_VERSION_ID"</code></pre>



<p class="wp-block-paragraph">The stronger fix is to stop trusting the writer to set retention correctly and enforce a floor in the bucket policy. S3 exposes the <code>s3:object-lock-remaining-retention-days</code> condition key for exactly this, so you can reject writes that ask for too short a lock.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RejectShortRetention",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-evidence-archive/*",
      "Condition": {
        "NumericLessThan": {
          "s3:object-lock-remaining-retention-days": "365"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The write path now fails loudly when someone gets it wrong, which is what you want. A rejected upload gets noticed. A silently unprotected object does not.</p>



<h2 class="wp-block-heading">Failure three: the delete that looks like a delete</h2>



<p class="wp-block-paragraph">Object Lock protects object versions. It does not stop new versions being created, and it does not stop delete markers being placed on top of a protected object.</p>



<p class="wp-block-paragraph">So a user with ordinary delete permissions can issue a delete against a locked object, get a success response, and make it vanish from every normal listing. The protected version is still there. Nothing was destroyed. But an auditor browsing the bucket sees nothing, and the person who ran the command believes the file is gone.</p>



<p class="wp-block-paragraph">Three practical consequences:</p>



<ul class="wp-block-list">
<li>Deny <code>s3:DeleteObject</code> on the evidence prefix for everyone except a break-glass role. Immutability is not a substitute for permissions, it is a backstop under them.</li>



<li>Any retrieval tooling you hand to an auditor must list object <em>versions</em>, not objects. <code>list-object-versions</code>, not <code>list-objects-v2</code>.</li>



<li>Lifecycle rules interact with this in a way that surprises people. An expiration rule on a versioned bucket adds a delete marker rather than removing data, and a rule that tries to permanently expire a noncurrent version will not remove a version still under retention. Your cleanup policy and your retention policy are two different clocks, and the lock always wins.</li>
</ul>



<h2 class="wp-block-heading">Failure four: evidence that arrives already unprovable</h2>



<p class="wp-block-paragraph">Immutability starts when the object lands. Everything before that is chain of custody, and it is where most repositories are weakest.</p>



<p class="wp-block-paragraph">If the evidence is CloudTrail, use the integrity validation CloudTrail already ships. With log file validation enabled, CloudTrail writes an hourly digest file containing hashes of the log files delivered in that hour, signs each digest with a per-region private key, and puts the digests in a separate folder in the same bucket. You can then verify the chain later.</p>



<pre class="wp-block-code"><code># Turn on digest files for an existing trail.
aws cloudtrail update-trail 
  --name org-audit-trail 
  --enable-log-file-validation

# Verify a time range later. Files must still be in the location
# CloudTrail delivered them to, or validation cannot run.
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-audit-trail 
  --start-time 2020-01-01T00:00:00Z</code></pre>



<p class="wp-block-paragraph">Two details worth knowing before you design around it. Validation only covers log files referenced by digest files, so a gap in digest coverage is a gap in your proof. And the files must stay where CloudTrail put them, which means copying logs into a tidier prefix breaks the built-in verification and hands you the job of writing your own checker.</p>



<p class="wp-block-paragraph">For evidence that is not CloudTrail, the equivalent is doing the work yourself: hash the artifact at the point of generation, record the hash somewhere separate from the artifact, and write both. A manifest per collection run, checksummed and locked alongside the files, is unglamorous and holds up.</p>



<p class="wp-block-paragraph">One constraint that catches people mid-build: a bucket with Object Lock enabled cannot be used as the destination for S3 server access logs. If your design had one bucket receiving both the evidence and its own access logs, it needs splitting.</p>



<h2 class="wp-block-heading">Failure five: the bill you cannot switch off</h2>



<p class="wp-block-paragraph">Compliance mode plus a long retention period plus archival storage classes is where cost surprises live, because every lever you would normally pull to reduce spend has been deliberately disabled.</p>



<p class="wp-block-paragraph">The mechanics that drive it, none of which are secret but all of which get discovered late:</p>



<ul class="wp-block-list">
<li><strong>Minimum storage durations.</strong> S3 Glacier Flexible Retrieval carries a 90 day minimum and S3 Glacier Deep Archive carries 180 days. Transition earlier than that and you are billed for the remainder of the minimum anyway.</li>



<li><strong>Per-object overhead in Glacier classes.</strong> AWS documents 40 KB of metadata overhead per object for Glacier Flexible Retrieval and Deep Archive, of which 8 KB is billed at S3 Standard rates and 32 KB at the destination rate. For a repository full of 2 KB JSON approvals, the overhead is the bill.</li>



<li><strong>Billing starts when the rule is satisfied.</strong> Lifecycle transitions to Glacier classes happen asynchronously, but you are charged at the destination rate, and the minimum duration clock starts, from the date the rule matched rather than the date the bytes physically moved.</li>



<li><strong>You cannot delete your way out.</strong> Under compliance mode, oversized or mistakenly retained objects stay until their retain-until date passes, whatever the finance conversation says.</li>
</ul>



<p class="wp-block-paragraph">The design response is to aggregate before you lock. Batch small artifacts into per-day or per-collection archives, checksum the archive, and lock the archive rather than thousands of tiny files. It costs you a little retrieval convenience and saves you a per-object overhead multiplier you cannot renegotiate later.</p>



<p class="wp-block-paragraph">Also pick the retrieval tier against the real deadline. Deep Archive is the cheapest place to keep evidence and one of the slowest to get it back. If a regulator or a legal team may need files at short notice, that trade is a bad one regardless of the storage line item.</p>



<h2 class="wp-block-heading">Putting the repository together</h2>



<p class="wp-block-paragraph">With the failure families in view, the build order is fairly short.</p>



<ol class="wp-block-list">
<li><strong>Separate the account.</strong> Evidence lives in its own AWS account, not a prefix in the production log bucket. The blast radius of a compromised production admin should not include the record of what they did.</li>



<li><strong>Write the service control policies first.</strong> At the organisation level, deny <code>s3:BypassGovernanceRetention</code>, deny <code>s3:PutBucketPolicy</code> and <code>s3:DeleteBucketPolicy</code> on the evidence bucket, and deny disabling the trail. SCPs apply to the account root too, which is the point.</li>



<li><strong>Create the bucket with Object Lock and a default governance retention.</strong> Block public access, enforce TLS with an <code>aws:SecureTransport</code> condition, and encrypt with a customer managed KMS key whose policy is written for this bucket alone.</li>



<li><strong>Enforce the retention floor in the bucket policy</strong> using the remaining-retention-days condition, so a misconfigured collector fails rather than writes something unprotected.</li>



<li><strong>Replicate to a second account or region.</strong> Object Lock works with S3 Replication, and the destination bucket must also have Object Lock enabled. This is your answer to region-level loss and to an account-level mistake.</li>



<li><strong>Add Vault Lock for backup-shaped evidence.</strong> Snapshots and database backups belong in a locked backup vault rather than being exported to S3 by hand.</li>



<li><strong>Alarm on the write path.</strong> A day with no new evidence objects is the failure that never pages anyone. Alert on absence, not just on errors.</li>
</ol>



<p class="wp-block-paragraph">Vault Lock deserves its own note because its compliance mode has a grace period that Object Lock does not. You choose a cooling-off window, which AWS requires to be at least three days, during which the lock can still be removed. After that the vault and its lock are immutable and cannot be changed or deleted by anyone, including AWS.</p>



<pre class="wp-block-code"><code># Compliance mode: the presence of --changeable-for-days is what makes it
# compliance rather than governance. Minimum value is 3 (72 hours).
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name evidence-vault 
  --min-retention-days 365 
  --max-retention-days 2555 
  --changeable-for-days 3</code></pre>



<p class="wp-block-paragraph">Use that window. Run a backup job through the vault, confirm the retention arithmetic against what the policy document actually requires, and only then let the clock run out.</p>



<p class="wp-block-paragraph">One more design question worth asking out loud: does every copy need to be inside AWS? A second, offline-ish copy on independent infrastructure answers a different threat than cross-region replication does, because it survives an account-level compromise. Providers like InterServer, Contabo and Hetzner sell cheap storage that can hold an encrypted archive, and several S3-compatible services implement Object Lock semantics of their own. Confirm the provider actually supports Object Lock rather than assuming an S3-compatible API implies it, because that varies.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">AccessDenied when deleting a version you are allowed to delete</h3>



<p class="wp-block-paragraph">Usually governance retention plus a missing bypass. The IAM permission <code>s3:BypassGovernanceRetention</code> and the explicit bypass on the request are both required. If the object is in compliance mode, there is no combination of permissions that will work, and the error is telling you the truth.</p>



<pre class="wp-block-code"><code># Only valid for governance mode, and only with the matching IAM permission.
aws s3api delete-object 
  --bucket my-evidence-archive 
  --key path/to/object 
  --version-id "SOME_VERSION_ID" 
  --bypass-governance-retention</code></pre>



<h3 class="wp-block-heading">Replication fails on an Object Lock bucket</h3>



<p class="wp-block-paragraph">The destination needs Object Lock enabled too. Replicated objects take the lock configuration of the source, so a destination without Object Lock cannot accept them. Check the destination bucket first, then the replication role&#8217;s permissions.</p>



<h3 class="wp-block-heading">Lifecycle rule appears to do nothing</h3>



<p class="wp-block-paragraph">Two likely causes. The versions it targets are still under retention, so expiration cannot remove them. Or the rule targets current versions on a versioned bucket, in which case it is adding delete markers and the data is still there as noncurrent versions, still being billed.</p>



<h3 class="wp-block-heading">validate-logs reports invalid or missing files</h3>



<p class="wp-block-paragraph">Confirm nothing moved the objects. Validation requires the log and digest files to be in the location CloudTrail delivered them to, and the command runs per region. A copy job that reorganised the prefix layout is the usual culprit.</p>



<h3 class="wp-block-heading">Object Lock cannot be enabled on the bucket</h3>



<p class="wp-block-paragraph">Versioning has to be enabled first, and Object Lock cannot be turned off later. If the bucket is currently a server access log destination, that has to move before Object Lock will work for you at all.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Enabling Object Lock on an existing bucket and assuming the objects already in it are now protected. They are not, until retention is applied to each version.</li>



<li>Choosing compliance mode on day one to look thorough, then discovering the retention period was wrong.</li>



<li>Treating Object Lock as an access control. It is a floor under your permissions, not a replacement for them.</li>



<li>Leaving the bypass permission attached to a general administrator role because it was needed once during testing.</li>



<li>Locking millions of tiny objects into Glacier classes and paying per-object overhead on all of them.</li>



<li>Storing evidence in the same account as the systems it describes, which makes the evidence only as trustworthy as that account&#8217;s admins.</li>



<li>Never testing retrieval, so the first restore attempt happens under audit pressure with an unfamiliar Glacier restore workflow.</li>
</ul>



<h2 class="wp-block-heading">Best practices for an audit evidence repository on AWS</h2>



<ul class="wp-block-list">
<li>Governance mode by default, compliance mode only where an external requirement names the period. Write down which rule drove the choice, in the same repository.</li>



<li>Enforce the retention floor in the bucket policy so the write path fails loudly instead of writing unprotected objects.</li>



<li>Use legal hold, not extended retention, for investigations. Legal holds have no expiry and are removed explicitly, which is the right shape for something that ends on an unknown date.</li>



<li>Aggregate small artifacts into archives with a checksummed manifest before locking them.</li>



<li>Replicate to a second account with Object Lock enabled, and check replication metrics rather than assuming.</li>



<li>Rehearse retrieval quarterly, including the Glacier restore step, and keep the output as evidence of the rehearsal.</li>



<li>Monitor for absence of writes. Tools like Amazon CloudWatch, Grafana or Datadog are all fine here. The alert that matters is &#8220;no evidence arrived today&#8221;.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I enable S3 Object Lock on a bucket that already exists?</h3>



<p class="wp-block-paragraph">Yes. S3 supports enabling Object Lock on existing buckets, and versioning must be enabled. The important caveat is that enabling it does not retroactively protect the objects already stored. You apply retention to those separately, per object or with S3 Batch Operations.</p>



<h3 class="wp-block-heading">What is the difference between governance mode and compliance mode?</h3>



<p class="wp-block-paragraph">Governance mode can be overridden by a principal holding <code>s3:BypassGovernanceRetention</code> who sends the bypass header. Compliance mode cannot be overridden by anyone, including the root user, and the retention period cannot be shortened or the mode downgraded.</p>



<h3 class="wp-block-heading">Does S3 Object Lock stop someone deleting the file?</h3>



<p class="wp-block-paragraph">It stops the protected version being destroyed. It does not stop a delete marker being placed on top, which hides the object from ordinary listings while the version remains. Keep delete permissions tight and list versions when you audit.</p>



<h3 class="wp-block-heading">Do I need Object Lock if I already have versioning and MFA delete?</h3>



<p class="wp-block-paragraph">They solve different problems. Versioning preserves history but a sufficiently privileged caller can still remove versions. Object Lock removes that possibility for the duration of the retention period, which is the property an auditor is asking about.</p>



<h3 class="wp-block-heading">How do I prove the evidence has not been modified?</h3>



<p class="wp-block-paragraph">For CloudTrail, enable log file integrity validation and keep the signed digest files with the logs, then run <code>aws cloudtrail validate-logs</code> for the period in question. For other artifacts, hash at generation time, store the manifest separately, and lock both.</p>



<h3 class="wp-block-heading">Can a lifecycle rule delete objects that are still locked?</h3>



<p class="wp-block-paragraph">No. Retention wins over lifecycle expiration. Plan the two together, because a cleanup rule that quietly does nothing for years is a cost problem you will not see until you look at storage growth.</p>



<h3 class="wp-block-heading">Is Vault Lock compliance mode reversible?</h3>



<p class="wp-block-paragraph">Only during the cooling-off window you set at creation, which must be at least three days. After the lock date, the vault and its lock cannot be changed or deleted by anyone, including AWS.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An audit evidence repository on AWS fails in a specific, quiet way: the bucket says Object Lock is enabled, and the objects inside carry no retention. Everything else in this post is downstream of that gap. Check protection at the object version, not at the bucket, and enforce the floor in the write path so the system tells you when it is wrong.</p>



<p class="wp-block-paragraph">Then pick the mode like the one-way door it is. Governance mode with a locked-down bypass gets you almost all of the protection and keeps a documented way back. Compliance mode is worth its permanence when a real external rule demands it, and expensive theatre when it does not.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or reviewing one of these?</h2>



<p class="wp-block-paragraph">Immutable retention is easy to enable and easy to get subtly wrong, and the mistakes are the kind you find out about at the worst moment. This is work I take on directly:</p>



<ul class="wp-block-list">
<li>Designing an evidence account with Object Lock, KMS, service control policies and a locked-down write path</li>



<li>Auditing an existing bucket to find object versions that carry no retention despite Object Lock being enabled</li>



<li>Backfilling retention across existing objects with S3 Batch Operations without locking in the wrong period</li>



<li>Modelling the long-run cost of a retention decision before compliance mode makes it permanent</li>



<li>Setting up cross-account replication, AWS Backup Vault Lock and CloudTrail integrity validation as one coherent design</li>



<li>Writing the retrieval runbook and rehearsing it, so the first restore is not during an audit</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion on what you already have, send me the bucket policy, the Object Lock configuration and the output of a <code>head-object</code> call on one old evidence file. That is usually enough to tell you where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/">See how I work on projects</a></div>



<div class="wp-block-button is-style-outline is-style-outline--2"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/">Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Secure Patient Document Portal on S3 and Cognito</title>
		<link>https://john-nessime.com/blog/cloud-computing/secure-patient-document-portal-s3-cognito/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 19 Oct 2026 06:00:12 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[ePHI]]></category>
		<category><![CDATA[Healthcare Cloud]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Identity Pools]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[MFA]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[Session Management]]></category>
		<category><![CDATA[Token Revocation]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=457</guid>

					<description><![CDATA[<p>Cognito for sign-in, S3 for the files, and a demo that passes on the first try. The problems start after that. Prefix isolation, presigned URL lifetimes, KMS as a second gate, CloudTrail data events, and token revocation that actually ends a session.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/secure-patient-document-portal-s3-cognito/">Building a Secure Patient Document Portal on S3 and Cognito</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The request usually arrives sideways. A clinic manager says something like: &#8220;we&#8217;re emailing scans as PDF attachments and our insurer flagged it, can you build us something?&#8221; Nobody says the words &#8220;patient portal.&#8221; They say &#8220;somewhere for patients to pick up their files.&#8221;</p>



<p class="wp-block-paragraph">So you sketch the obvious thing. Amazon Cognito for sign-in, Amazon S3 for the documents, a small front end in between. It&#8217;s three services and a weekend, and it works on the first try, which is exactly the problem. A <strong>secure patient document portal</strong> built this way passes every test you think to run, because the tests you think to run are all &#8220;can Alice download Alice&#8217;s file?&#8221; The one that matters is &#8220;what can Alice see about Bob?&#8221; and that one has an answer you won&#8217;t like.</p>



<p class="wp-block-paragraph">This post covers the parts that go wrong after the demo. Where to put the authorization boundary, how to scope the S3 policy so one patient can&#8217;t enumerate another, why your download links expire earlier than you configured, what encryption actually gates, and what your audit trail looks like on the day someone asks for it. Everything here is mechanism, not a compliance checklist. Getting a Business Associate Agreement in place is a separate conversation with your lawyer and your AWS account team.</p>



<h2 class="wp-block-heading">The failure that never pages you</h2>



<p class="wp-block-paragraph">Here&#8217;s the shape of it. You write an IAM policy that lets a signed-in user read objects under their own prefix. Something like <code>arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*</code>. You test it. Alice gets her file. Alice cannot get Bob&#8217;s file. Ship it.</p>



<p class="wp-block-paragraph">Then, because the front end needs to render a file list, someone adds <code>s3:ListBucket</code>. And <code>s3:ListBucket</code> is not an object-level action. It applies to the bucket ARN, not the object ARN, so the <code>${cognito-identity.amazonaws.com:sub}</code> substitution in your resource string does nothing for it. Unless you add a separate condition on the request, a signed-in patient can list every key in the bucket.</p>



<p class="wp-block-paragraph">They still can&#8217;t download anything. Every <code>GetObject</code> outside their prefix returns a denial. But they can read the key names. And key names in a clinic bucket are rarely neutral. <code>patients/8f2a.../oncology-referral-2024.pdf</code> tells you something about a person even if the bytes stay locked.</p>



<p class="wp-block-paragraph">Nothing alerts on this. No error appears in any log you&#8217;re watching, because nothing errored. The listing succeeded. That&#8217;s the failure mode worth building around: in a document portal, the metadata leak is the one that doesn&#8217;t announce itself.</p>



<p class="wp-block-paragraph">The fix is a condition key on the list action, not a resource string:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOwnPrefixOnly",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::clinic-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "patients/${cognito-identity.amazonaws.com:sub}/*"
          ]
        }
      }
    },
    {
      "Sid": "ReadOwnObjectsOnly",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*"
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Two statements, two different jobs. The first constrains what the caller may ask the bucket to enumerate. The second constrains what bytes the caller may pull. Write only the second one and you&#8217;ve secured the download path while leaving the discovery path wide open.</p>



<p class="wp-block-paragraph">One trap in that variable. <code>${cognito-identity.amazonaws.com:sub}</code> resolves to the identity ID from the Cognito <em>identity pool</em>, not the <code>sub</code> claim from the user pool. They are different values for the same human. If you name your prefixes with the user pool sub because that&#8217;s what your application code has handy, the policy will match nothing and every request will be denied, which at least fails loudly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Decide where the authorization boundary lives</h2>



<p class="wp-block-paragraph">There are two honest architectures here and the choice shapes everything downstream. Pick deliberately.</p>



<h3 class="wp-block-heading">Pattern A: browser holds AWS credentials</h3>



<p class="wp-block-paragraph">The user signs in to a Cognito user pool. The application exchanges that token with a Cognito identity pool for temporary AWS credentials, and the browser calls S3 directly using the AWS SDK.</p>



<p class="wp-block-paragraph">It&#8217;s less code and there&#8217;s no server to run. The authorization boundary is entirely the IAM role policy attached to the identity pool. That policy is the whole security model, so it had better be right, and it&#8217;s harder to reason about than application code because the enforcement happens in a service you can&#8217;t step through with a debugger.</p>



<p class="wp-block-paragraph">The real limitation is expressiveness. IAM policy conditions do not know that Bob&#8217;s referral was superseded, that Alice revoked consent last Tuesday, or that a document should only be visible after a clinician releases it. Anything with business logic in it has to be encoded as prefix structure or object tags, and that gets brittle fast.</p>



<h3 class="wp-block-heading">Pattern B: backend brokers every object</h3>



<p class="wp-block-paragraph">The browser never touches AWS credentials. It sends the Cognito access token to your own API. The API validates the token, checks its own records for who may see what, and returns a presigned URL for that one object. S3 is reachable only by the API&#8217;s execution role.</p>



<p class="wp-block-paragraph">You now have a place to put logic, a place to log intent alongside access, and a single choke point to rate limit. The cost is that you own a service, its uptime, and its dependencies.</p>



<p class="wp-block-paragraph">For a small clinic build, this is the one I reach for first. Not because Pattern A is insecure, but because the questions clients ask in month three (&#8220;can we hide results until the doctor reviews them?&#8221;) are answerable in Pattern B and require a re-architecture in Pattern A. The identity-pool approach earns its keep when access really is a pure function of who you are, which in healthcare it rarely stays.</p>



<p class="wp-block-paragraph">If you do want attribute-driven access without a backend, Cognito identity pools can map claims from the token onto principal tags on the temporary session, which you then reference in policy conditions. It works, and it is the right tool for coarse distinctions like patient versus staff. It is not the right tool for per-document consent state.</p>



<h2 class="wp-block-heading">Your download links expire before you think they do</h2>



<p class="wp-block-paragraph">This one generates support tickets on week two. You set a presigned URL to last a few hours. Patients report links dying in under one. Nothing in your code changed.</p>



<p class="wp-block-paragraph">A presigned URL carries a signature made with whatever credentials signed it, and it cannot outlive those credentials. AWS documents the ceilings clearly: with long-lived IAM user keys and Signature Version 4 the maximum is seven days; from the S3 console the range is one minute to twelve hours; with temporary credentials from an assumed role, the URL dies when the role session does, regardless of the expiry you asked for. A default <code>AssumeRole</code> session is one hour.</p>



<p class="wp-block-paragraph">Your Lambda or container signing those URLs is using temporary credentials. So the effective lifetime is the shorter of your requested expiry and the remaining time on that role session, and the remaining time depends on when the runtime last refreshed. That&#8217;s the source of the flapping. You can watch it yourself:</p>



<pre class="wp-block-code"><code># Sign a URL and inspect what was actually encoded into it.
aws s3 presign s3://clinic-docs/patients/example/scan.pdf --expires-in 3600

# The returned URL contains X-Amz-Date, X-Amz-Expires and X-Amz-Credential.
# X-Amz-Expires is the window you asked for. If X-Amz-Credential shows an
# ASIA... key id, you signed with temporary credentials and the real ceiling
# is the session, not the window.</code></pre>



<p class="wp-block-paragraph">The honest resolution is to stop fighting it. Sign short, on demand, when the user clicks. Fifteen minutes is generous for a download the patient is standing in front of. Long links are not a convenience feature in this context anyway, because a presigned URL is a bearer credential: whoever holds the string can fetch the object, signed in or not. Forwarding the email forwards the access. If your portal mails links, you have moved the authorization boundary into someone&#8217;s inbox.</p>



<p class="wp-block-paragraph">Mail a notification that says a document is ready and link to the portal login. Never mail the signed URL itself.</p>



<h2 class="wp-block-heading">Encryption that actually gates something</h2>



<p class="wp-block-paragraph">New S3 objects are encrypted at rest by default with S3-managed keys. That satisfies the sentence &#8220;data is encrypted at rest&#8221; and gates precisely nothing, because anyone with <code>s3:GetObject</code> gets plaintext back. It is a storage property, not an access control.</p>



<p class="wp-block-paragraph">Server-side encryption with an AWS KMS customer managed key is different in kind. Reading an object now requires two independent grants: the S3 permission and <code>kms:Decrypt</code> on the key. Those live in different policies and can be owned by different people. A bucket policy mistake no longer hands over plaintext on its own, and revoking the key grant cuts off reads without touching a single S3 permission.</p>



<p class="wp-block-paragraph">You also get a second audit stream. Every decrypt is a KMS API call, recorded separately from the S3 call, which is useful precisely because an attacker who tampers with one trail probably didn&#8217;t think about the other.</p>



<p class="wp-block-paragraph">The costs are real and worth stating. KMS charges per request, so a portal that fetches many small objects will notice. S3 Bucket Keys reduce that request volume substantially by caching a bucket-level key, at the price of coarser detail in the KMS audit records. For a document portal with modest traffic and high sensitivity, I&#8217;d take the per-request cost and keep the granularity. For a thumbnail-heavy application, the opposite.</p>



<h2 class="wp-block-heading">The audit trail you don&#8217;t have yet</h2>



<p class="wp-block-paragraph">Assume someone will eventually ask: who downloaded this specific file, and when? If you have not explicitly turned something on, the answer is that you don&#8217;t know.</p>



<p class="wp-block-paragraph">CloudTrail is on for management events by default, and management events are control-plane operations: creating the bucket, changing its policy, attaching a role. Object-level reads and writes are <em>data events</em>, and data events are off by default because of volume and cost. <code>GetObject</code> on a patient scan is a data event. Unless you configured a trail with data events for that bucket, no record of the download exists.</p>



<pre class="wp-block-code"><code># Log S3 object-level activity for one bucket on an existing trail.
# Advanced event selectors let you scope to a single bucket prefix rather
# than every object in the account.
aws cloudtrail put-event-selectors 
  --trail-name clinic-audit-trail 
  --advanced-event-selectors '[
    {
      "Name": "PHI bucket object access",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Data"]},
        {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
        {"Field": "resources.ARN", "StartsWith":
          ["arn:aws:s3:::clinic-docs/patients/"]}
      ]
    }
  ]'</code></pre>



<p class="wp-block-paragraph">S3 server access logs are the other option and they are not equivalent. They record HTTP-level request detail and are delivered on a best-effort basis with no delivery guarantee, which makes them useful for traffic analysis and weak as evidence. CloudTrail data events carry caller identity, which is the field you actually need when the question is &#8220;who.&#8221;</p>



<p class="wp-block-paragraph">Two things to sort out while you&#8217;re in there. Send the trail to a bucket in a different account, or at minimum a bucket the portal&#8217;s roles cannot write to, so that compromising the application doesn&#8217;t compromise its own log. And put a lifecycle rule on the log bucket, because data event volume grows with usage and nobody budgets for it. If you already run centralized dashboards on something like Grafana Cloud or CloudWatch, wire an alert to unusual read volume per identity now rather than after the incident.</p>



<h2 class="wp-block-heading">Cutting off access when someone leaves</h2>



<p class="wp-block-paragraph">Disabling a Cognito user does not immediately end their session, and this surprises people. Tokens are self-contained. An access token already in a browser stays cryptographically valid until it expires, typically about an hour.</p>



<p class="wp-block-paragraph">Refresh tokens are the longer exposure. The default is thirty days, and app clients can be configured anywhere from sixty minutes to ten years. Ten-year refresh tokens on a portal holding medical records is not a hypothetical misconfiguration; it&#8217;s a checkbox someone ticked to stop users complaining about re-login.</p>



<p class="wp-block-paragraph">Cognito gives you revocation. Token revocation is enabled by default on newly created app clients. <code>AdminUserGlobalSignOut</code> revokes all of a user&#8217;s refresh, ID, and access tokens from the server side, and the <code>/oauth2/revoke</code> endpoint kills one refresh token and everything descended from it.</p>



<pre class="wp-block-code"><code># Server-side, authorized with IAM credentials. Revokes the user's refresh,
# ID and access tokens.
aws cognito-idp admin-user-global-sign-out 
  --user-pool-id us-east-1_EXAMPLE 
  --username patient-example

# Confirm revocation is switched on for the app client before you rely on it.
aws cognito-idp describe-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --query 'UserPoolClient.EnableTokenRevocation'</code></pre>



<p class="wp-block-paragraph">Now the part that catches teams out. AWS documents that a revoked token still passes verification in a generic JWT library that only checks signature and expiry. Revocation is enforced by Cognito, not baked into the token. If your API validates tokens by fetching the JWKS and calling a verify function, a revoked token sails straight through until it expires on its own.</p>



<p class="wp-block-paragraph">Which means short access token lifetimes are load-bearing. Revocation shortens the tail; it doesn&#8217;t close the window. Refresh token rotation, available on the Essentials and Plus feature tiers, replaces the refresh token on each exchange and narrows the window a stolen one is useful for. If you&#8217;re paying for Plus for threat protection features, you already have it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the usual suspects</h2>



<h3 class="wp-block-heading">Every request returns access denied after login</h3>



<p class="wp-block-paragraph">Nine times out of ten the prefix in your policy variable doesn&#8217;t match the prefix in your bucket. Print the identity ID your session actually resolved to and compare it character by character against a real key. Remember the identity pool ID and the user pool sub are different values.</p>



<h3 class="wp-block-heading">Signature does not match on a presigned URL</h3>



<p class="wp-block-paragraph">Usually something modified the request after signing. A client library adding a header, a proxy rewriting the path, or an email gateway URL-encoding the query string. Compare the <code>X-Amz-SignedHeaders</code> list in the URL against what the client actually sent.</p>



<h3 class="wp-block-heading">Uploads succeed but downloads fail</h3>



<p class="wp-block-paragraph">Check the KMS key policy separately from the IAM policy. A role can hold <code>kms:GenerateDataKey</code> (enough to write) without <code>kms:Decrypt</code> (needed to read). The S3 error will not say the word KMS prominently, which is why this one burns an afternoon.</p>



<h3 class="wp-block-heading">Browser uploads fail before any request reaches S3</h3>



<p class="wp-block-paragraph">That&#8217;s CORS. The bucket needs a CORS configuration allowing your portal&#8217;s origin, the methods you use, and any headers the SDK sends. Browser preflight failures never appear in S3 logs, so check the browser console first.</p>



<h3 class="wp-block-heading">Nothing appears in CloudTrail</h3>



<p class="wp-block-paragraph">Data events are not shown in CloudTrail Event History. Query the destination bucket directly, with Athena or CloudTrail Lake, and allow a few minutes for delivery.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Granting <code>s3:ListBucket</code> without an <code>s3:prefix</code> condition, so patients can enumerate every key even though downloads are blocked.</li>

<li>Putting identifying information in object keys or file names. Assume the key will leak eventually and use opaque identifiers.</li>

<li>Emailing presigned URLs. The link is the credential, and inboxes get forwarded, backed up, and breached.</li>

<li>Extending refresh token lifetime to silence complaints about re-login, instead of implementing rotation.</li>

<li>Assuming default encryption means access control. It means the disk is encrypted, nothing more.</li>

<li>Enabling versioning without a lifecycle rule, so deleted documents linger indefinitely as noncurrent versions you forgot exist.</li>

<li>Skipping CloudTrail data events to save money, then discovering the gap during an access review.</li>
</ul>



<h2 class="wp-block-heading">Best practices for a secure patient document portal</h2>



<ul class="wp-block-list">
<li>Keep one bucket for patient documents and nothing else. Mixed-purpose buckets make policies ambiguous and audits slow.</li>

<li>Turn on S3 Block Public Access at the account level, not just the bucket. Account level survives someone creating a new bucket in a hurry.</li>

<li>Enable versioning and set a noncurrent version expiration rule, so an overwritten document is recoverable but not permanent.</li>

<li>Use a KMS customer managed key so decrypt authority is separable from S3 authority.</li>

<li>Require MFA for staff accounts. TOTP costs nothing to enable and staff accounts are the ones with breadth.</li>

<li>Sign download URLs on demand with short expiry, and always from a backend that logged why it signed.</li>

<li>Put the portal behind a WAF and rate limiting. Cloudflare in front of the application, or AWS WAF on the distribution, both work; the point is that credential stuffing against a login page is the most likely thing to actually happen to you.</li>

<li>Restrict administrative access to the AWS console and to any bastion by network as well as identity, whether that&#8217;s IAM Identity Center with condition keys or a VPN such as NordVPN or Surfshark for a small team without central identity.</li>

<li>Write down the retention period before launch, then encode it as a lifecycle rule. Retention decided later is retention never decided.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Are S3 and Cognito enough for HIPAA compliance?</h3>



<p class="wp-block-paragraph">Both are covered under AWS&#8217;s HIPAA eligibility program, and Cognito&#8217;s documentation states it is HIPAA-BAA eligible. That makes them usable building blocks, not compliance. You need a signed Business Associate Agreement with AWS, and the configuration, access reviews, training, and documented policies remain yours. Service eligibility is a prerequisite, not an outcome.</p>



<h3 class="wp-block-heading">Should I use a Cognito identity pool or a backend API?</h3>



<p class="wp-block-paragraph">Use an identity pool when access is a pure function of identity and the rules will not change. Use a backend when documents have states, when a clinician gates release, or when you need to log intent alongside access. Most clinic portals become the second within a year.</p>



<h3 class="wp-block-heading">How long should a presigned URL last?</h3>



<p class="wp-block-paragraph">Minutes, not hours. Sign at click time. The maximum with long-lived credentials is seven days, but that ceiling is irrelevant here because the link is a bearer credential and a longer window is a longer leak.</p>



<h3 class="wp-block-heading">Can patients upload documents back to the portal?</h3>



<p class="wp-block-paragraph">Yes, using presigned PUT or presigned POST. Presigned POST is worth knowing about because its policy document can constrain content type and size before the object lands, which presigned PUT cannot. Either way, write uploads into a quarantine prefix, scan them, then move them.</p>



<h3 class="wp-block-heading">Does enabling CloudTrail data events cost much?</h3>



<p class="wp-block-paragraph">Data events are billed per event recorded with no free tier, unlike management events. For a portal serving a few thousand downloads a month the amount is small; for a bucket also serving application assets it is not. Scope your event selector to the patient prefix rather than the whole account and the question mostly goes away.</p>



<h3 class="wp-block-heading">What happens to sessions when I disable a user?</h3>



<p class="wp-block-paragraph">Nothing immediate. Existing access tokens remain valid until they expire. Call <code>AdminUserGlobalSignOut</code> to revoke tokens, and keep access token lifetime short so the residual window is measured in minutes.</p>



<h3 class="wp-block-heading">Do I need CloudFront in front of the bucket?</h3>



<p class="wp-block-paragraph">Not for correctness. It helps when patients are geographically spread, when you want signed cookies for a set of objects rather than one URL per object, or when you want a single edge to attach a WAF to. For a single-clinic portal, direct presigned URLs are simpler and one fewer thing to misconfigure.</p>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">A signed-in token proves who someone is. It does not decide what they may reach. In a <strong>secure patient document portal</strong> the boundary is the policy condition, the KMS grant, and the code that decides whether to sign a URL at all, and every one of those is something you have to write on purpose.</p>



<p class="wp-block-paragraph">The demo passes either way. The difference only shows up when someone reads a key name they shouldn&#8217;t have seen, and nothing in your system considers that an error.</p>



<p class="wp-block-paragraph">If you build one of these, spend an hour before launch on exactly two things: sign in as a test patient and try to list the bucket, then ask your logs who downloaded a specific file last week. If either answer is uncomfortable, you&#8217;ve found your next task.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your portal?</h2>



<p class="wp-block-paragraph">This is the kind of build where the gap between working and safe is invisible from the outside. I help small teams close it.</p>



<ul class="wp-block-list">
<li>Reviewing S3 bucket policies and Cognito identity pool roles for prefix isolation gaps, including the list-versus-get split most policies miss.</li>

<li>Designing the boundary: identity pool credentials in the browser versus a backend that brokers presigned URLs, with the trade-offs written down for your stakeholders.</li>

<li>Setting up KMS customer managed keys, key policies, and grant separation so decrypt authority is not implied by storage authority.</li>

<li>Turning on scoped CloudTrail data events, building the Athena tables to query them, and alerting on abnormal per-identity read volume.</li>

<li>Hardening the Cognito side: MFA, token lifetimes, refresh token rotation, and a revocation path that actually ends sessions.</li>

<li>Writing the whole thing as Terraform or OpenTofu so the next change is a reviewed pull request rather than a console click.</li>
</ul>



<p class="wp-block-paragraph">Send me your bucket policy and your identity pool role, with the account IDs redacted, and I&#8217;ll tell you what a signed-in patient can actually see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/secure-patient-document-portal-s3-cognito/">Building a Secure Patient Document Portal on S3 and Cognito</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Property SaaS Document Storage: The Five Failures That Show Up in an Audit</title>
		<link>https://john-nessime.com/blog/cloud-security/property-saas-document-storage/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 28 Sep 2026 06:04:50 +0000</pubDate>
				<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Real Estate Technology]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[ABAC]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS STS]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[Cross-Tenant Leakage]]></category>
		<category><![CDATA[Crypto Shredding]]></category>
		<category><![CDATA[Data Minimization]]></category>
		<category><![CDATA[Document Retention]]></category>
		<category><![CDATA[Encryption Context]]></category>
		<category><![CDATA[File Upload Security]]></category>
		<category><![CDATA[GDPR]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[Object Storage]]></category>
		<category><![CDATA[Presigned POST]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[Property Management Software]]></category>
		<category><![CDATA[Real Estate Data]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[SaaS Architecture]]></category>
		<category><![CDATA[Secure Disposal]]></category>
		<category><![CDATA[Stored XSS]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=898</guid>

					<description><![CDATA[<p>Storing tenant IDs, leases and referencing packs is easy until someone asks who actually read them. Five failure families in property SaaS document storage, with the IAM policies, presigned URL mechanics, upload controls and deletion patterns that close each one.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/property-saas-document-storage/">Property SaaS Document Storage: The Five Failures That Show Up in an Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket that starts this is almost never &#8220;our document storage is insecure.&#8221; It&#8217;s a security questionnaire from a prospect, or a data subject access request, or an insurer asking one specific question: list everyone who has viewed this tenant&#8217;s ID document in the last twelve months.</p>



<p class="wp-block-paragraph">So you open the audit table. And you find it records who <em>requested</em> a link to the document. Not who opened it. The actual downloads never touched your application at all, because you handed out a signed storage URL and the browser went straight to the bucket. Your log is a record of intentions.</p>



<p class="wp-block-paragraph">That gap is typical of how property SaaS document storage goes wrong. The failures are rarely dramatic. Nobody dumps a bucket publicly. The guarantees you thought you had just turn out to be enforced in application code that ran once, long before the file was actually read. What follows is five failure families worth designing against when a platform stores tenant ID scans, signed leases, deposit certificates, right-to-rent evidence and referencing packs.</p>



<h2 class="wp-block-heading">What&#8217;s actually in the pile</h2>



<p class="wp-block-paragraph">Before architecture, be honest about the contents. A property platform&#8217;s document store is at least four risk classes wearing the same file extension:</p>



<ul class="wp-block-list">
<li><strong>Identity evidence.</strong> Passport and driving licence scans, visa documents, sometimes a selfie held next to the document. Highest blast radius, lowest genuine need for long retention.</li>



<li><strong>Contracts.</strong> Tenancy agreements, addenda, guarantor deeds, signed inventories. Legally driven retention, plus an integrity requirement: proving the file hasn&#8217;t changed matters more than hiding it.</li>



<li><strong>Financial and referencing material.</strong> Bank statements, payslips, credit reference reports. Sensitive, third-party sourced, and often covered by contractual restrictions from the referencing provider.</li>



<li><strong>Operational clutter.</strong> Inspection photos, safety certificates, maintenance invoices. Low sensitivity, high volume, and the thing that dominates your storage bill.</li>
</ul>



<p class="wp-block-paragraph">These want different retention clocks, different access rules and arguably different buckets. Treating them as one &#8220;documents&#8221; table is where most of the later pain originates.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure one: the access check and the access happen at different times</h2>



<p class="wp-block-paragraph">Presigned URLs are the right pattern. They keep the bucket private, take the transfer off your servers, and mean you never hand storage credentials to a browser. I reach for them first.</p>



<p class="wp-block-paragraph">The problem is what a presigned URL <em>is</em>. It&#8217;s a request you already authorised, signed, and then gave away. Anyone holding it can replay it until it expires. Your permission logic ran once and never runs again. Which means all of these keep working after the fact:</p>



<ul class="wp-block-list">
<li>A letting agent who left the company yesterday and whose account you disabled this morning.</li>



<li>A link forwarded into a WhatsApp group, pasted into a support ticket, or sitting in browser history on a shared office machine.</li>



<li>A document the tenant asked you to delete, if the object is still there behind a delete marker.</li>
</ul>



<h3 class="wp-block-heading">Make the URL die when the session does</h3>



<p class="wp-block-paragraph">Amazon&#8217;s own guidance is clearer on this than most people realise. A presigned URL created with temporary credentials expires when those credentials expire, <em>even if you asked for a longer expiry</em>. It also stops working if the signing credential is revoked, deleted or deactivated.</p>



<p class="wp-block-paragraph">That&#8217;s a lever, not a footnote. Sign with a long-lived IAM user key and the seven day maximum the SDK allows is a real seven days. Sign inside a short-lived assumed-role session and the URL cannot outlive that session, whatever number you passed in. So: never sign document URLs with static credentials. Assume a role per request, tag the session with the tenant, sign inside it.</p>



<pre class="wp-block-code"><code>import boto3

sts = boto3.client("sts")

# The tag key here must match the principal tag key used in the role's policy.
session = sts.assume_role(
    RoleArn="arn:aws:iam::111122223333:role/TenantDocumentAccess",
    RoleSessionName="doc-download",
    Tags=[{"Key": "TenantID", "Value": tenant_id}],
    DurationSeconds=900,
)["Credentials"]

s3 = boto3.client(
    "s3",
    aws_access_key_id=session["AccessKeyId"],
    aws_secret_access_key=session["SecretAccessKey"],
    aws_session_token=session["SessionToken"],
)

url = s3.generate_presigned_url(
    "get_object",
    Params={
        "Bucket": "prop-docs",
        "Key": f"tenants/{tenant_id}/identity/{doc_id}.pdf",
        "ResponseContentDisposition": 'attachment; filename="id-document.pdf"',
        "ResponseContentType": "application/octet-stream",
    },
    ExpiresIn=120,
)</code></pre>



<p class="wp-block-paragraph">Two things are doing work. <code>DurationSeconds</code> caps the real lifetime of the link regardless of <code>ExpiresIn</code>. And <code>ResponseContentDisposition</code> overrides how the object is served on this request, which matters for the serving problem further down.</p>



<p class="wp-block-paragraph">The trade-off is real: an STS call per download adds latency, adds a dependency, and STS has request limits you can hit at volume. AWS&#8217;s prescriptive guidance says as much. For identity documents I&#8217;d pay that cost. For inspection photos I wouldn&#8217;t.</p>



<h3 class="wp-block-heading">Log where the read actually happens</h3>



<p class="wp-block-paragraph">Your application log records issuance. To record retrieval you need logging at the storage layer: CloudTrail data events for object-level operations, or S3 server access logging. Data events are billed per event, so scope them to the identity and contract prefixes rather than the bucket holding thousands of inspection photos.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure two: isolation that lives in your ORM instead of your IAM policy</h2>



<p class="wp-block-paragraph">Ask most teams how tenant isolation works and you get a description of a WHERE clause. The key is built as <code>tenants/{tenant_id}/...</code> and the tenant ID comes from the session. Fine until one code path takes it from a request parameter, or a background job iterates without scoping, or someone writes an admin endpoint in a hurry.</p>



<p class="wp-block-paragraph">Move the boundary below your code. With attribute-based access control the tenant becomes a session tag and the IAM policy interpolates it into the resource ARN. One role serves every tenant, and the policy count doesn&#8217;t grow as you onboard.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadWriteOwnTenantPrefix",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::prop-docs/tenants/${aws:PrincipalTag/TenantID}/*"
    },
    {
      "Sid": "ListOnlyOwnTenantPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::prop-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": "tenants/${aws:PrincipalTag/TenantID}/*"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Two statements because S3 splits the problem. The first governs the object, where the tenant prefix is part of the resource ARN. The second governs listing, a bucket-level action, so the prefix has to be constrained through a condition instead. Skip it and a tenant can enumerate every key in the bucket even though they can&#8217;t read any of them, and key names in this domain leak plenty on their own.</p>



<p class="wp-block-paragraph">The failure mode worth internalising: if a code path forgets the session tag, <code>${aws:PrincipalTag/TenantID}</code> resolves to nothing and the statement doesn&#8217;t match. That&#8217;s a deny, which is what you want. But it only holds if the role has no broader fallback statement. One well-meaning <code>s3:GetObject</code> on the whole bucket, added to unblock a migration script, quietly turns the scheme into decoration.</p>



<h3 class="wp-block-heading">When to use a bucket per tenant instead</h3>



<p class="wp-block-paragraph">A bucket or an account per tenant is a genuinely harder boundary and I wouldn&#8217;t dismiss it. With a few dozen large agency customers it&#8217;s defensible and makes offboarding trivially provable. It stops being defensible once you onboard self-serve landlords, because bucket and account counts run into service quotas and the operational surface grows with every signup. Pick on tenant count, not on which sounds more secure. And note that ABAC is only as trustworthy as the tenant claim: if the caller can influence the value reaching the tagging call, you&#8217;ve built an elegant no-op.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure three: the upload and serve path</h2>



<p class="wp-block-paragraph">Uploads here come from people you have no relationship with yet. Prospective tenants, guarantors, contractors. That&#8217;s an untrusted channel by definition, and it produces two separate problems.</p>



<h3 class="wp-block-heading">Constrain the upload itself</h3>



<p class="wp-block-paragraph">A presigned PUT URL lets the holder write whatever they like to that key, at whatever size. Presigned POST is the better tool for public-facing uploads: the signature covers a policy document, and the policy carries conditions the browser cannot alter.</p>



<pre class="wp-block-code"><code>post = s3.generate_presigned_post(
    Bucket="prop-docs",
    Key=f"tenants/{tenant_id}/inbox/{upload_id}",
    Fields={"Content-Type": "application/pdf"},
    Conditions=[
        {"Content-Type": "application/pdf"},
        ["content-length-range", 1, 15728640],
    ],
    ExpiresIn=600,
)</code></pre>



<p class="wp-block-paragraph">The <code>content-length-range</code> condition is the one people skip, and it&#8217;s the one that stops a stranger using your referencing portal as free storage for a 40 GB file. A declared <code>Content-Type</code> is a claim, not a fact, so land uploads in a quarantine prefix, verify the actual bytes, and only then move them to the tenant prefix.</p>



<p class="wp-block-paragraph">Scan on that path too. ClamAV in a container is the zero-cost option and covers known malware; GuardDuty Malware Protection for S3 and hosted APIs such as Cloudmersive trade money for not maintaining it. What you&#8217;re protecting is mostly the letting agent who opens the attachment on a Windows machine, so this is a control for your customers rather than your servers.</p>



<h3 class="wp-block-heading">Never serve tenant documents from your application origin</h3>



<p class="wp-block-paragraph">This is the one that turns a document store into an account takeover. SVG and HTML are formats browsers execute. If a file uploaded by a prospective tenant is served inline from the same origin as your app, any script inside it runs with your application&#8217;s cookies and session context. A long stream of real advisories follows exactly this shape, particularly in self-hosted deployments where the object store sits on the app&#8217;s own domain.</p>



<p class="wp-block-paragraph">Three controls, and they stack rather than substitute:</p>



<ol class="wp-block-list">
<li><strong>Force download.</strong> Serve with <code>Content-Disposition: attachment</code> and <code>X-Content-Type-Options: nosniff</code>. Set these as object metadata at upload time <em>and</em> override them per request. The override protects links you mint; the stored metadata protects anything that bypasses your minting path.</li>



<li><strong>Use a separate origin.</strong> Put user content on a distinct domain, not a subdomain of the app. GitHub&#8217;s separate apex for user content is the reference implementation. Same-origin policy then contains the blast radius if control one is missed somewhere.</li>



<li><strong>Normalise where you can.</strong> Rendering an uploaded ID to a flattened image server side removes the executable content entirely. It costs CPU and fidelity. For identity documents, where you rarely need the original bytes back, it&#8217;s the right trade.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure four: deleting the document doesn&#8217;t delete the document</h2>



<p class="wp-block-paragraph">A tenant asks you to erase their passport scan. Your app sets a flag, the file disappears from the UI, and you reply confirming deletion. Then count the copies that still exist.</p>



<ul class="wp-block-list">
<li>The previous object versions, if versioning is on. A delete without a version ID just writes a delete marker.</li>



<li>Cross-region replicas with their own versioning and lifecycle rules, plus any backup or snapshot taken before the deletion.</li>



<li>Derivatives you generated and forgot: thumbnails, page previews, OCR text in a search index. Run a passport through Amazon Textract and the extracted fields are a copy of the sensitive data in a different shape.</li>



<li>Copies that left the platform. A document attached to a CRM record in something like GoHighLevel, forwarded by email, or downloaded to a letting agent&#8217;s laptop.</li>
</ul>



<p class="wp-block-paragraph">The ICO&#8217;s position is that a valid erasure request extends to backups as well as live systems, and that where backup data cannot be immediately overwritten it must be put beyond use: isolated, unused for anything else, deleted when that backup cycles. Workable, but only if you know which backup sets contain documents and how long they cycle. If you can&#8217;t answer that, you can&#8217;t answer the request.</p>



<h3 class="wp-block-heading">Crypto-shredding, and what it actually costs</h3>



<p class="wp-block-paragraph">The one technique that makes deletion provable across copies you can&#8217;t reach is a per-tenant KMS key. Encrypt that tenant&#8217;s objects under their own customer managed key; when they offboard, schedule the key for deletion. Every ciphertext copy anywhere, backups included, becomes undecryptable. You delete the data by deleting the ability to read it. The honest costs:</p>



<ul class="wp-block-list">
<li>KMS bills per key per month plus per request, so per-tenant keys are a real line item. Check current rates and quotas first: there&#8217;s an account limit on customer managed keys that a self-serve signup flow will eventually meet.</li>



<li>Key deletion has a mandatory waiting period. Crypto-shredding is not instant, and your erasure response has to say so.</li>



<li>It&#8217;s all-or-nothing per key. If a tenant asks you to erase one document rather than offboard, the key doesn&#8217;t help.</li>
</ul>



<h3 class="wp-block-heading">Object Lock fights erasure, on purpose</h3>



<p class="wp-block-paragraph">S3 Object Lock is tempting for signed tenancy agreements, because write-once storage is a strong integrity story. But compliance mode means nobody, root included, can delete the object until retention lapses. That is the feature. It is also a direct collision with an erasure request, and you cannot engineer your way out of it afterwards.</p>



<p class="wp-block-paragraph">My line: Object Lock on executed contracts, where a statutory retention basis overrides erasure, and never on identity documents or referencing material. Different buckets, different rules, decided before anything is written. For per-document erasure elsewhere, lifecycle rules that expire noncurrent versions on the sensitive prefixes are worth setting up before you need them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure five: storing what you never needed to store</h2>



<p class="wp-block-paragraph">The cheapest control on this list is not storing the file. The ICO is clear that UK GDPR sets no fixed retention periods and puts the burden on you to justify how long you keep personal data in identifiable form. That&#8217;s a design instruction, not a legal footnote.</p>



<p class="wp-block-paragraph">For identity documents, ask what your platform needs once the check is done. Usually it&#8217;s the result, not the evidence: document type, issuing country, expiry date, a check reference, who performed it and when. If your customer&#8217;s regulator requires the image itself, that&#8217;s a retention rule with a stated basis, which is a different conversation from keeping it because deleting felt risky.</p>



<p class="wp-block-paragraph">Practical version: give every document class a retention clock wired to a business event rather than upload date, and let lifecycle rules do the deleting. Tenancy end plus a defined period. Check completion plus a defined period. Not &#8220;whenever someone remembers.&#8221;</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the breakages you&#8217;ll actually hit</h2>



<h3 class="wp-block-heading">Everything broke after someone enabled S3 Bucket Keys</h3>



<p class="wp-block-paragraph">This one catches teams who did the sophisticated thing. If your IAM or KMS key policies use the object ARN as the encryption context to scope access, enabling an S3 Bucket Key changes that context to the <em>bucket</em> ARN and those policies stop matching. AWS documents it explicitly as something to fix before flipping the setting.</p>



<p class="wp-block-paragraph">The quieter side effect: KMS CloudTrail events then log the bucket ARN instead of the object ARN, and there are far fewer of them because S3 caches the bucket-level key. If you were treating KMS CloudTrail volume as a proxy for document access, that signal degrades without anything appearing to break. Bucket Keys are still worth enabling for the request cost reduction. Move the conditions first, then re-check what your dashboards count.</p>



<h3 class="wp-block-heading">The presigned URL expires far sooner than requested</h3>



<p class="wp-block-paragraph">Almost always the signing credential is a role session shorter than the expiry you asked for. This is documented behaviour, not a bug. Either raise the session duration within the role&#8217;s maximum, or accept the shorter window as the security property it is.</p>



<h3 class="wp-block-heading">Listing works, reading returns Access Denied</h3>



<p class="wp-block-paragraph">Classic split between the bucket-level and object-level statements. Check both exist and reference the same tag key spelling. Session tag keys are case sensitive, and a <code>tenantId</code> in the code against a <code>TenantID</code> in the policy fails silently as a non-match.</p>



<h3 class="wp-block-heading">Access denied only for encrypted objects</h3>



<p class="wp-block-paragraph">An SSE-KMS object needs two grants, not one. The S3 action on the object, and <code>kms:Decrypt</code> on the key it was encrypted under. In a per-tenant key setup that second grant is usually also tag-conditioned, so a missing or mismatched tag on the key itself produces a denial that looks like an S3 problem and isn&#8217;t.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Signing document URLs with a long-lived IAM user access key, which makes every link independently long-lived.</li>



<li>Granting the tenant role a broad bucket-wide read statement alongside the tag-scoped one, usually to unblock a one-off script that then stayed.</li>



<li>Serving user uploads from a subdomain of the application and assuming that counts as origin separation. It doesn&#8217;t for cookie scope.</li>



<li>Enabling versioning for safety and never adding a noncurrent version expiry rule, so every deletion is reversible forever.</li>



<li>Building deletion that handles the object but not the OCR output, the thumbnail, or the search index entry.</li>



<li>Putting Object Lock on the bucket that also holds identity documents.</li>
</ul>



<h2 class="wp-block-heading">Design rules worth holding to</h2>



<ol class="wp-block-list">
<li><strong>Separate buckets by risk class,</strong> not by feature. Identity, contracts, financial, operational. Different encryption, retention, logging and lock settings follow naturally from that split.</li>



<li><strong>Enforce isolation in IAM,</strong> with application checks as defence in depth rather than the boundary itself.</li>



<li><strong>Sign short, sign from a session.</strong> Minutes, not days, and always from assumed-role credentials so revocation actually revokes.</li>



<li><strong>Deny unencrypted writes at the bucket policy,</strong> using the server-side encryption request headers as conditions, so a misconfigured client fails loudly instead of writing plaintext.</li>



<li><strong>Give every document class an expiry clock</strong> tied to a business event, implemented in lifecycle rules rather than a cron job someone maintains.</li>



<li><strong>Write the deletion runbook before launch,</strong> listing every place a copy lands. If you can&#8217;t enumerate them, that&#8217;s the finding.</li>



<li><strong>Test cross-tenant access in CI.</strong> A negative test that asserts tenant A&#8217;s session cannot read tenant B&#8217;s key is worth more than any amount of policy review.</li>
</ol>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How long should a presigned URL for a tenant ID document last?</h3>



<p class="wp-block-paragraph">Short enough that the user clicks it immediately. A couple of minutes covers a click-to-download flow. If the UI needs longer, re-mint on demand rather than extend. And remember the ceiling is set by the signing credential, so a short assumed-role session enforces the limit even if a later code change passes a bigger number.</p>



<h3 class="wp-block-heading">Do I need AWS, or will Backblaze B2 or Cloudflare R2 do?</h3>



<p class="wp-block-paragraph">They all speak the S3 API and presigned URLs work on all of them, so the core patterns transfer. Wasabi, Backblaze B2, Cloudflare R2 and DigitalOcean Spaces are reasonable for the operational tier, and R2&#8217;s egress model is attractive if agents download a lot. They differ in the depth of the IAM story: session-tag ABAC and per-tenant KMS keys are AWS-specific. If isolation and crypto-shredding are hard requirements, keep the sensitive prefixes on S3 even when the bulk photos live somewhere cheaper.</p>



<h3 class="wp-block-heading">Can I store scanned passports on a normal VPS instead?</h3>



<p class="wp-block-paragraph">You can, and plenty of platforms run the app tier on a VPS from Contabo or InterServer while keeping documents in object storage. What you lose on local disk is the deletion story and the access log. Proving an image is gone from a filesystem, its backups and its snapshots is far harder than scheduling a KMS key for deletion. If documents stay local, budget real effort for secure disposal, including on staff machines where tools like O&amp;O SafeErase have a place.</p>



<h3 class="wp-block-heading">What about e-signed tenancy agreements?</h3>



<p class="wp-block-paragraph">With a provider like DocuSign, the executed document and its audit trail live there as well as in your platform. That&#8217;s a second copy under a second retention regime, and it belongs in your deletion runbook and data map. It also strengthens your integrity story, which is a fair reason to skip Object Lock on contracts you retrieve from the provider.</p>



<h3 class="wp-block-heading">Should the CDN cache tenant documents?</h3>



<p class="wp-block-paragraph">Not the sensitive classes. A CDN in front of private documents adds a cache that can serve content after your access decision changed, and header handling at the edge is a common source of the inline-rendering bug. CloudFront with origin access control and signed URLs is legitimate if you need the performance, but for identity documents I&#8217;d go straight to the bucket. For inspection photos, cache away.</p>



<h3 class="wp-block-heading">How do I prove any of this to a prospect&#8217;s security team?</h3>



<p class="wp-block-paragraph">Artefacts, not assertions. The bucket policy, the ABAC policy, CI output showing cross-tenant denial, a sample of storage-layer access logs, and a retention schedule with a clock per document class. That set answers most questionnaires without a call.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Good property SaaS document storage is mostly about closing the gap between when you check access and when access happens. Every failure family above is a version of that gap: the presigned link that outlives the permission, the isolation enforced once in code and never at the storage layer, the deletion that updates a flag while the bytes sit behind a delete marker.</p>



<p class="wp-block-paragraph">Close it by pushing the decision down: into IAM policies that evaluate on every request, into credentials short enough that revocation is real, into lifecycle rules that delete without anyone remembering to. Then the answer to &#8220;show us who read this tenant&#8217;s ID document&#8221; is a query, not a project.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your document storage?</h2>



<p class="wp-block-paragraph">Most of what I do on property and lettings platforms is this exact work. Things I&#8217;d help with here:</p>



<ul class="wp-block-list">
<li>Reviewing bucket and IAM policies for cross-tenant read paths, including the ones added to unblock a migration and never removed.</li>



<li>Moving tenant isolation from application code into session-tag ABAC, with a CI test that fails the build on cross-tenant access.</li>



<li>Rebuilding the upload path with presigned POST conditions, quarantine, content verification and scanning before anything lands in a tenant prefix.</li>



<li>Fixing the serving path so uploaded files can&#8217;t execute against your origin, including header handling through a CDN.</li>



<li>Designing retention clocks and lifecycle rules per document class, plus a deletion runbook that names every copy including derivatives and third-party systems.</li>



<li>Setting up storage-layer access logging scoped to the sensitive prefixes so you can actually answer an access request.</li>
</ul>



<p class="wp-block-paragraph">If it&#8217;s useful, send me your bucket policy and the IAM policy attached to your document role, with the account IDs redacted, and I&#8217;ll tell you what a determined tenant could reach.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/property-saas-document-storage/">Property SaaS Document Storage: The Five Failures That Show Up in an Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Amazon Macie PII Detection: The Buckets It Never Opened</title>
		<link>https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 06 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Data Firehose]]></category>
		<category><![CDATA[Amazon Macie]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Classification]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[Security Hub]]></category>
		<category><![CDATA[Sensitive Data Discovery]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=405</guid>

					<description><![CDATA[<p>A Macie bucket labeled "Not sensitive" often just means Macie never read it. Extensionless objects, unsupported storage classes, unreachable KMS keys and quota truncation all produce silence that looks identical to a clean result. Here's how to measure coverage, fix the four gaps, tune identifiers, and keep the bill honest.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email came from legal, not from security. A customer had exercised a data access request, someone pulled the export by hand, and it contained email addresses and phone numbers sitting in a bucket Macie had labeled <em>Not sensitive</em>. Nobody had done anything wrong. The console was telling the truth as it understood it. It just didn&#8217;t understand very much about that bucket.</p>



<p class="wp-block-paragraph">That is the failure mode worth internalizing before anything else: Amazon Macie PII detection reports on what it managed to read, not on what is there. Objects it never opened produce no findings, and no findings looks identical to a clean result. The gap between &#8220;we scanned this&#8221; and &#8220;we found nothing&#8221; is where most Macie deployments quietly fail their first audit.</p>



<p class="wp-block-paragraph">This post covers where that gap comes from and how to close it: the eligibility chain every object goes through, the four distinct reasons an object gets skipped, how to tune identifiers so the findings are worth reading, what actually drives the bill, and how to route results somewhere a human will act on them.</p>



<h2 class="wp-block-heading">The eligibility chain behind Amazon Macie PII detection</h2>



<p class="wp-block-paragraph">Every object passes the same gates before a byte gets inspected. The order matters, because each gate fails differently and needs a different fix.</p>



<ol class="wp-block-list">
<li>Is it in a general purpose S3 bucket? Directory buckets are out of scope entirely.</li>



<li>Is the storage class supported? Standard, Standard-IA, One Zone-IA, Intelligent-Tiering, Glacier Instant Retrieval and Reduced Redundancy are in. Glacier Deep Archive and S3 Express One Zone are not.</li>



<li>Does the object key carry a recognized file extension? This is a string check on the name, not a content sniff.</li>



<li>Can Macie retrieve and decrypt it, given the bucket policy, object ACL and encryption key?</li>



<li>Does the content parse, and stay inside the per-file quotas?</li>



<li>Do the active data identifiers match anything in it?</li>
</ol>



<p class="wp-block-paragraph">Only the last gate produces a finding. Everything before it produces silence. Macie does record why each object was skipped, but that lives in coverage data and object samples, not on the findings page most people look at.</p>



<p class="wp-block-paragraph">One detail explains a lot of confusion: when you first enable automated sensitive data discovery, every bucket gets a sensitivity score of 50 and the label <em>Not yet analyzed</em>. A bucket whose permissions block Macie stays there permanently. It never turns red. It sits mid-list, looking unremarkable next to buckets that genuinely were analyzed.</p>



<h2 class="wp-block-heading">Gap one: objects Macie never opened</h2>



<p class="wp-block-paragraph">Macie calls these <em>unclassifiable</em>. They fail the storage class or extension check, so no retrieval is attempted. This is the largest source of false confidence I&#8217;ve seen, and the cheapest to fix.</p>



<h3 class="wp-block-heading">Missing file extensions</h3>



<p class="wp-block-paragraph">Classifiability comes from the file name extension. A file full of customer records named <code>part-00003-a4f9</code> is invisible. The same bytes named <code>part-00003-a4f9.json</code> get inspected and produce findings.</p>



<p class="wp-block-paragraph">This bites hardest on streaming ingestion. Amazon Data Firehose writes to S3 without appending an extension unless compression, format conversion, or the explicit file extension setting adds one. An uncompressed JSON stream lands as a tree of extensionless objects. Athena and Glue read them fine, because they infer format from the table definition. Macie skips every one.</p>



<p class="wp-block-paragraph">Fix it upstream: set the S3 file extension on the delivery configuration, or enable GZIP compression or Parquet conversion, both of which append a recognized extension. If you can&#8217;t change the producer, the fallback is a copy-and-rename step into a scan prefix, which costs you storage plus a pipeline to maintain. Fixing the producer is almost always cheaper.</p>



<h3 class="wp-block-heading">Unsupported storage classes</h3>



<p class="wp-block-paragraph">A lifecycle rule transitioning old exports to Glacier Deep Archive quietly removes them from scope. That may be the right storage decision, but it needs to be a conscious one, because archived customer data is still customer data when a regulator asks.</p>



<p class="wp-block-paragraph">Macie exposes these counts per bucket, split by cause. This is the first query I run against a new account:</p>



<pre class="wp-block-code"><code># Buckets where Macie can't classify objects, broken down by cause
aws macie2 describe-buckets 
  --query 'buckets[?unclassifiableObjectCount.total &gt; `0`].{
      bucket: bucketName,
      classifiable: classifiableObjectCount,
      skipped_total: unclassifiableObjectCount.total,
      skipped_extension: unclassifiableObjectCount.fileType,
      skipped_storage_class: unclassifiableObjectCount.storageClass
    }' 
  --output table</code></pre>



<p class="wp-block-paragraph">Read the ratio, not the raw number. Forty thousand classifiable objects and twelve skipped is fine. Three classifiable and four hundred thousand skipped is a reporting artifact pretending to be a scan result, and that bucket&#8217;s sensitivity score means nothing.</p>



<h2 class="wp-block-heading">Gap two: objects it opened and couldn&#8217;t read</h2>



<p class="wp-block-paragraph">These are classification errors rather than eligibility failures. Macie selected the object, tried to fetch it, and was refused. Three causes, and they need three different people to fix them.</p>



<ul class="wp-block-list">
<li><strong>Customer-provided keys (SSE-C).</strong> Macie cannot supply the key material, so it cannot retrieve the object. No permissions fix exists. Re-encrypt with S3 managed or KMS keys if you want coverage.</li>



<li><strong>KMS key policy.</strong> For customer managed keys, the key policy must allow the Macie service-linked role, <code>AWSServiceRoleForAmazonMacie</code>, to decrypt. Cross-account buckets need the key owner to grant it, not the bucket owner. AWS publishes a permission analyzer script in the <code>aws-samples/amazon-macie-scripts</code> repository that enumerates every key Macie needs and generates the CLI commands to fix them.</li>



<li><strong>Restrictive bucket policies.</strong> An explicit <code>Deny</code> conditioned on source IP or VPC endpoint blocks Macie along with everything else. The working pattern excludes the service-linked role ARN from the deny using the <code>aws:PrincipalArn</code> condition key.</li>
</ul>



<p class="wp-block-paragraph">To see which objects Macie actually touched in a bucket, pull the object samples. It&#8217;s the closest thing to a scan log you get:</p>



<pre class="wp-block-code"><code># Objects automated discovery selected, with per-object status
aws macie2 list-resource-profile-artifacts 
  --resource-arn arn:aws:s3:::your-bucket-name</code></pre>



<p class="wp-block-paragraph">Anything with a status of <code>SKIPPED</code> is a lead. Take the key, check its encryption settings in S3, and you usually have your answer inside a minute.</p>



<h2 class="wp-block-heading">Gap three: it read the file and nothing matched</h2>



<p class="wp-block-paragraph">Macie ships managed data identifiers covering common PII, financial data and credentials across many countries. Automated discovery uses a recommended subset by default rather than all of them. Sensible, but it is a default, and it will miss things that matter to you.</p>



<p class="wp-block-paragraph"><strong>Add the managed identifiers for your actual jurisdictions.</strong> If you hold records for customers in a country whose national ID identifier isn&#8217;t in the recommended set, you&#8217;re scanning for the wrong things with perfect efficiency. Pull the current list with <code>aws macie2 list-managed-data-identifiers</code> and compare it against where your customers live.</p>



<p class="wp-block-paragraph"><strong>Write custom identifiers for what only you know is sensitive.</strong> Internal account numbers, case references, employee IDs. A custom identifier is a regex plus optional keywords and a proximity rule, and the keywords are what save you. A bare <code>[0-9]{8}</code> matches timestamps, order totals and row counts, and you drown. The same regex with keywords and a match distance only fires when the number sits near a word that gives it meaning.</p>



<p class="wp-block-paragraph">Test the pattern before it goes near a job. This runs the criteria against sample text and returns match counts without creating anything:</p>



<pre class="wp-block-code"><code># Dry-run a custom identifier before creating it
aws macie2 test-custom-data-identifier 
  --regex 'ACC-[0-9]{4}-[0-9]{4}' 
  --keywords account customer acct 
  --maximum-match-distance 30 
  --sample-text 'customer account ACC-4821-9930 updated; total 1234-5678'</code></pre>



<p class="wp-block-paragraph">Run it against real samples, including the awkward ones, then against a file you know is clean and confirm zero. A pattern you only tested on positives is a pattern you haven&#8217;t tested.</p>



<p class="wp-block-paragraph">The mirror image is allow lists, which tell Macie to ignore specific text or patterns: your support inbox address, your published press contact, the seeded test records in everyone&#8217;s staging fixtures. Without them the same non-issues appear in every scan until the team stops reading findings. Allow lists accept predefined text or regex, and the per-account and per-job quotas are tight enough that you should curate rather than dump.</p>



<h2 class="wp-block-heading">Gap four: quota truncation inside large files</h2>



<p class="wp-block-paragraph">Subtle, because the object <em>is</em> analyzed. Just not completely, and a partial result reports like a full one.</p>



<ul class="wp-block-list">
<li>Per-file size quotas differ by format. Exceed the quota for a type and Macie analyzes none of that file, not part of it.</li>



<li>Archives have separate ceilings for nesting depth, extracted bytes and file count. If the metadata shows a breach up front, nothing is extracted. Cross a limit partway through and Macie stops, reporting only what it processed.</li>



<li>JSON and JSON Lines have a nested depth limit. Past it, the file is not analyzed at all.</li>



<li>Full names and mailing addresses cap out per file. After the cap Macie stops counting and stops reporting locations for that type, so the occurrence count is a floor, never a total.</li>
</ul>



<p class="wp-block-paragraph">Findings carry a status reason naming the limit that was hit, with values like <code>FILE_EXCEEDS_SIZE_LIMIT</code>, <code>ARCHIVE_NESTING_LEVEL_OVER_LIMIT</code> and <code>JSON_NESTING_LEVEL_OVER_LIMIT</code>. Filter on those before declaring a data lake of large compressed exports clean. Check current quota values in the Macie documentation rather than memorizing them, since they differ per format.</p>



<h2 class="wp-block-heading">Automated discovery or a discovery job?</h2>



<p class="wp-block-paragraph"><strong>Automated sensitive data discovery</strong> evaluates your inventory daily and samples representative objects across as many buckets as it can, grouping by metadata like prefix, extension and last-modified date. It is breadth-first and it does not re-analyze unchanged objects. Its job is to tell you which buckets deserve attention. It is not designed to prove a bucket is clean, and reading it that way is the root of most Macie misunderstandings.</p>



<p class="wp-block-paragraph"><strong>Sensitive data discovery jobs</strong> are the depth tool. You choose the buckets, scope by prefix, extension, size or object tag, choose the identifiers, and run once or on a schedule. This is what you point at the bucket automated discovery just flagged.</p>



<pre class="wp-block-code"><code># One-time job over a single prefix, recommended identifiers, 30% sampling
aws macie2 create-classification-job 
  --job-type ONE_TIME 
  --name "exports-prefix-review" 
  --managed-data-identifier-selector RECOMMENDED 
  --sampling-percentage 30 
  --s3-job-definition '{
    "bucketDefinitions": [
      {"accountId": "111122223333", "buckets": ["your-bucket-name"]}
    ],
    "scoping": {
      "includes": {
        "and": [
          {"simpleScopeTerm": {
             "comparator": "STARTS_WITH",
             "key": "OBJECT_KEY",
             "values": ["exports/"]
          }}
        ]
      }
    }
  }'</code></pre>



<p class="wp-block-paragraph">One detail about <code>--sampling-percentage</code> that trips people up: it selects a random percentage of eligible <em>objects</em> and then analyzes each selected object completely. It does not read a fraction of each file. For a bucket where every object comes from one pipeline, 20 or 30 percent tells you what you need. For heterogeneous uploads, sampling is a coin flip and you want the full pass.</p>



<h2 class="wp-block-heading">What actually drives the bill</h2>



<p class="wp-block-paragraph">Macie bills on three dimensions, and the one that surprises people is not the one they budget for.</p>



<ul class="wp-block-list">
<li><strong>Bucket evaluation.</strong> Per general purpose bucket monitored, prorated daily, up to the account ceiling. Predictable, driven by bucket count rather than data volume.</li>



<li><strong>Object monitoring.</strong> Per object tracked while automated discovery is enabled. This is inventory bookkeeping, not content reading, and it runs whether or not anything gets scanned that day. On an account with a log archive holding hundreds of millions of tiny objects, it can dwarf the other two.</li>



<li><strong>Data inspected.</strong> Per GB actually analyzed, covering both automated discovery and jobs.</li>
</ul>



<p class="wp-block-paragraph">So excluding buckets from automated discovery is a genuine cost lever, not just noise reduction. Log archives, CloudTrail destinations, build artifact stores and backup targets are usually safe exclusions. Exclude for the right reason though: a backup bucket holding database dumps of your customer table is exactly what you want scanned, however boring its name sounds.</p>



<p class="wp-block-paragraph">The three dimensions appear as distinct usage types in the Cost and Usage Report, so they split cleanly. A FinOps platform like Vantage or CloudZero, or just a Cost Explorer view grouped by usage type, tells you within a day whether object monitoring or inspection is driving the number. That changes the fix entirely: object monitoring is solved by scoping, inspection by sampling and scheduling.</p>



<h2 class="wp-block-heading">Getting findings somewhere a human sees them</h2>



<p class="wp-block-paragraph">Findings that only exist in the Macie console get read during onboarding week and never again. Two things are worth wiring up on day one.</p>



<p class="wp-block-paragraph"><strong>Publish findings outward.</strong> Macie emits findings to EventBridge and integrates with Security Hub. EventBridge earns its keep because you can filter on severity and finding type and route only what matters, whether that&#8217;s a Slack channel, a ticket, or a Lambda that tags the bucket. If you already centralize alerts in Grafana Cloud or similar, that path keeps sensitive data findings next to everything else on call sees.</p>



<p class="wp-block-paragraph"><strong>Configure the discovery results repository.</strong> Findings tell you what was found and where. The detailed discovery results, including analysis logs for objects where nothing was found, need an S3 bucket and a KMS key configured to persist. Nothing is retained long term until you set this up, and &#8220;we scanned it and found nothing&#8221; is exactly the record an auditor asks for.</p>



<p class="wp-block-paragraph">One operational note on the reveal feature, which retrieves sample occurrences so a human can confirm a finding is real. It needs a customer managed KMS key and it&#8217;s genuinely useful for triage. It also means someone is now looking at live customer PII, which belongs in your access model and audit trail. Same goes if an engineer downloads an object to investigate locally: that copy is unmanaged PII on a laptop and needs a secure deletion step, whether that&#8217;s a wipe tool from a vendor like O&amp;O Software or a documented and enforced process. Findings triage has a habit of creating the exposure it was meant to prevent.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting a bucket that reports nothing</h2>



<p class="wp-block-paragraph">When Amazon Macie PII detection reports nothing for a bucket you have doubts about, work down this in order. Each step rules out one gate, and stopping at the first hit saves you the rest.</p>



<ol class="wp-block-list">
<li><strong>Check the sensitivity label.</strong> <em>Not yet analyzed</em> at score 50 means nothing was analyzed. Do not read that as clean.</li>



<li><strong>Check the coverage page.</strong> Macie names the issue directly: access denied, invalid encryption, invalid KMS key, permission denied, or unclassifiable.</li>



<li><strong>Compare classifiable and unclassifiable counts</strong> with the <code>describe-buckets</code> query above.</li>



<li><strong>List a few object keys.</strong> Run <code>aws s3 ls</code> against a prefix and look for extensions. Extensionless keys explain the whole thing.</li>



<li><strong>Check the storage class.</strong> A lifecycle rule may have moved everything somewhere Macie doesn&#8217;t read.</li>



<li><strong>Check encryption.</strong> SSE-C is unfixable without re-encryption. A customer managed KMS key is fixable with a key policy change.</li>



<li><strong>Check object samples</strong> with <code>list-resource-profile-artifacts</code> to see exactly what was selected and skipped.</li>



<li><strong>Only now question the identifiers.</strong> Run a targeted job with a custom identifier for the data you expect and see whether it fires.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a low sensitivity score as evidence of no PII. It reflects what was found <em>and</em> how much was analyzed, so low coverage produces a low score.</li>



<li>Enabling Macie in one region and assuming account-wide coverage. Settings and results are per region, every time.</li>



<li>Running default identifiers in a business operating outside the jurisdictions the recommended set covers.</li>



<li>Building custom identifiers from regex alone, then abandoning Macie over the false positive volume.</li>



<li>Skipping the discovery results repository, then having no evidence trail when someone asks what was scanned and when.</li>



<li>Excluding buckets by name pattern without checking contents. Backup and export buckets are frequently the highest-risk ones you own.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Measure coverage before findings. A coverage report is the first deliverable of a Macie rollout, not an afterthought.</li>



<li>Fix extensions at the producer rather than building a rename pipeline downstream.</li>



<li>Run the KMS permission analyzer script once per account, then again whenever a new customer managed key appears.</li>



<li>Use automated discovery for breadth and targeted jobs for depth, letting the first choose targets for the second.</li>



<li>Test every custom identifier against both a positive and a negative sample before it reaches a job.</li>



<li>Curate allow lists early. The cost of a noisy first month is a team that stops reading findings.</li>



<li>Re-check coverage after any change to bucket policies, KMS keys, lifecycle rules or ingestion pipelines. All four silently remove data from scope.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Macie scan anything other than S3?</h3>



<p class="wp-block-paragraph">No. Macie analyzes objects in S3 general purpose buckets only. The documented workaround is bringing data to it: export RDS or Aurora snapshots to S3 in Parquet, or export a DynamoDB table to S3, then run a discovery job against the export. That works, but the export is now a second copy of your sensitive data with its own encryption and deletion requirements.</p>



<h3 class="wp-block-heading">Does Macie scan new objects as they land?</h3>



<p class="wp-block-paragraph">Not per object. Automated discovery evaluates inventory on a daily cycle and prioritizes objects that are new or recently changed, so fresh data moves up the queue rather than triggering an immediate scan. If you need scanning tied to arrival, drive a discovery job from S3 event notifications through EventBridge and accept that you own that orchestration.</p>



<h3 class="wp-block-heading">Why does a bucket I know has PII show a low sensitivity score?</h3>



<p class="wp-block-paragraph">Almost always coverage rather than detection. Check whether the objects are classifiable, whether Macie can decrypt them, and whether the label reads <em>Not yet analyzed</em>. If Macie genuinely read them and found nothing, the next suspect is the identifier set, and a targeted job with a custom identifier will tell you in one run.</p>



<h3 class="wp-block-heading">Do I need both automated discovery and discovery jobs?</h3>



<p class="wp-block-paragraph">For most teams, yes. Automated discovery gives you the map at predictable cost. Jobs give you proof for a specific bucket at a specific time, which is what compliance evidence actually requires. Running only jobs means you never discover the bucket nobody told you about.</p>



<h3 class="wp-block-heading">How do I cut false positives without missing real PII?</h3>



<p class="wp-block-paragraph">In order of preference: add keywords and a proximity rule so matches need context; use allow lists for specific known-benign values like your published contact details and test fixtures; and only then remove managed identifiers that are consistently wrong for your data. Removing identifiers is the bluntest option and the easiest to regret, so leave it last.</p>



<h3 class="wp-block-heading">Does Macie work across accounts in an organization?</h3>



<p class="wp-block-paragraph">Yes, through AWS Organizations with a delegated administrator. Design around this: the administrator&#8217;s automated discovery settings apply to member accounts, members can&#8217;t change them, and members see coverage and sensitivity data for their own buckets but not the sensitive data findings themselves. That shapes who can actually remediate what.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Amazon Macie PII detection is only as good as its coverage, and coverage fails silently by design. An object with the wrong extension, the wrong storage class, or an unreachable encryption key produces no finding, and no finding renders identically to a clean result. Before you hand a Macie report to anyone who will make a decision from it, put the coverage numbers next to it: how many objects were classifiable, how many were analyzed, how many were skipped and why.</p>



<p class="wp-block-paragraph">Do that once and the rest of the service becomes straightforward. Skip it and you&#8217;re maintaining an expensive dashboard reporting on whichever subset of your data happened to be readable.</p>



<h2 class="wp-block-heading">Need help getting Macie to actually see your data?</h2>



<p class="wp-block-paragraph">Most of the work in a Macie rollout isn&#8217;t turning it on. It&#8217;s the unglamorous part: proving what got scanned, fixing the reasons things didn&#8217;t, and making the output land somewhere a human acts on it. That&#8217;s the part I help with.</p>



<ul class="wp-block-list">
<li>Coverage audits across your S3 estate, with per-bucket classifiable, analyzed and skipped counts and the specific cause of each gap</li>



<li>Fixing those causes: KMS key policies, bucket policies blocking the service-linked role, lifecycle rules archiving data out of scope, pipelines writing extensionless objects</li>



<li>Custom data identifiers and allow lists tuned against your real data and tested on positive and negative samples first</li>



<li>Cost shaping: working out whether object monitoring or data inspection drives your bill, and scoping discovery so the number is defensible</li>



<li>Findings pipelines through EventBridge or Security Hub into the alerting and ticketing you already run, with severity filtering that keeps signal above noise</li>



<li>Multi-account setup under AWS Organizations, including the delegated administrator model and who can see and remediate what</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me the output of the <code>describe-buckets</code> query above, or a screenshot of your coverage page, and I&#8217;ll tell you what&#8217;s actually being scanned.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Secure Client Document Vault Your Accounting Firm Can Defend</title>
		<link>https://john-nessime.com/blog/cloud-security/secure-client-document-vault-accounting-firm/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 28 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Accounting Technology]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Accounting Firms]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[Client Portal]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Crypto Shredding]]></category>
		<category><![CDATA[Data Classification]]></category>
		<category><![CDATA[Envelope Encryption]]></category>
		<category><![CDATA[GLBA Safeguards Rule]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[MFA]]></category>
		<category><![CDATA[Offboarding]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Secure Disposal]]></category>
		<category><![CDATA[WISP]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=544</guid>

					<description><![CDATA[<p>Most accounting-firm document vaults do not fail at the bucket. They fail at the edges: a download link that outlives the engagement, a KMS key policy as wide as the bucket policy, a retention lock that cannot be undone, and an audit trail nobody switched on. Six failure families, and the control that closes each.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/secure-client-document-vault-accounting-firm/">Building a Secure Client Document Vault Your Accounting Firm Can Defend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket usually reads like this: <em>client says the download link expired, can you send a new one.</em> Thirty seconds of work. Generate a fresh link, paste it into a reply, close the ticket.</p>



<p class="wp-block-paragraph">Nobody looks at the old link. If it was signed with a long expiry and mailed out, it is still live: in the client&#8217;s inbox, in the inbox of the bookkeeper on the CC line, and in whatever mail archive that firm runs. It keeps working for anyone holding the URL until the clock runs out, because a presigned S3 URL is a bearer token. AWS says exactly that in its own documentation. The URL does not know who you are. It knows that you have it.</p>



<p class="wp-block-paragraph">That is the shape of most document vault failures at a small accounting firm. Not a wide-open bucket. Not a stolen partner password. An ordinary delivery mechanism that leaks quietly and writes no log line that looks wrong six months later.</p>



<p class="wp-block-paragraph">This post covers how to build a <strong>secure client document vault</strong> for an accounting practice: what the regulation actually demands, the six failure families that decide whether it holds, and the control that closes each one. The examples are AWS because that is where most of these land, but the failure families are platform independent.</p>



<h2 class="wp-block-heading">Let the regulation write your requirements</h2>



<p class="wp-block-paragraph">Most firms treat compliance as paperwork done after the build. That is backwards. For a US accounting or tax practice, the FTC Safeguards Rule at 16 CFR Part 314 is the closest thing you will get to a free requirements document.</p>



<p class="wp-block-paragraph">It treats tax preparers, bookkeepers and accounting firms as financial institutions under the Gramm-Leach-Bliley Act. Size does not get you out of it. The parts that translate directly into architecture:</p>



<ul class="wp-block-list">
<li><strong>Access controls</strong> limiting each user to the customer information they need, and limiting customers to their own information.</li>



<li><strong>Encryption</strong> in transit over external networks and at rest, with compensating controls only where encryption is infeasible and your Qualified Individual signs off in writing.</li>



<li><strong>Multi-factor authentication</strong> for any individual accessing any information system, subject to the same written-approval exception.</li>



<li><strong>Secure disposal</strong>, with an outer bound of two years after last use unless retention is required by law or necessary for legitimate business purposes.</li>



<li><strong>Logging of authorized user activity</strong>, plus a named Qualified Individual who owns the program.</li>
</ul>



<p class="wp-block-paragraph">One more clause should shape your design more than any other. The Rule requires notifying the FTC within thirty days of discovering a security event involving the unencrypted information of at least five hundred consumers, and it counts encrypted information as unencrypted if the key was accessed by an unauthorized person.</p>



<p class="wp-block-paragraph">Read that twice. Encryption only helps at the breach-analysis stage if the key was genuinely out of reach. That is the entire argument for treating key custody as a control separate from storage.</p>



<p class="wp-block-paragraph">Separately, IRS Publication 4557 expects paid preparers to keep a Written Information Security Plan, and PTIN renewal asks you to confirm you have one. Treat the WISP as the document describing the vault you built, not a substitute for building it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure one: the link that outlives the engagement</h2>



<p class="wp-block-paragraph">Presigned URLs are the sensible answer to &#8220;let the client download this without an AWS account&#8221;. The problem is their expiry semantics, which almost nobody has read.</p>



<p class="wp-block-paragraph">Two clocks apply and the shorter wins: the expiry you asked for, and the lifetime of the credential you signed with. Long-lived IAM user credentials under Signature Version 4 reach seven days. A role session from <code>AssumeRole</code> dies when the session ends, one hour by default, regardless of what you requested. An EC2 instance profile is bound by metadata credential rotation. The console caps at twelve hours.</p>



<p class="wp-block-paragraph">So the two symptoms are opposites and both are common. Links dying early is a credential problem. Links living too long is a policy problem, and that is the one that hurts.</p>



<p class="wp-block-paragraph">The fix is not a smaller number. It is changing what goes in the email.</p>



<ol class="wp-block-list">
<li>Email a notification, not the file link. The client logs in with MFA and the portal mints a URL at click time.</li>



<li>Keep that expiry in minutes. The URL only has to survive a redirect and a download.</li>



<li>Scope every URL to one object and one verb.</li>



<li>If you truly need a long-lived share, use CloudFront signed URLs or signed cookies, where you can revoke by rotating the key group without touching the object.</li>
</ol>



<pre class="wp-block-code"><code># Sign a download URL valid for five minutes.
# --expires-in takes seconds; the default is 3600.
aws s3 presign s3://firm-client-vault/clients/acme-llc/workpapers.pdf 
    --expires-in 300</code></pre>



<p class="wp-block-paragraph">Run that under an assumed role, wait past the session expiry, and watch it fail well before five minutes. That is behaviour worth seeing once in a test rather than discovering during an incident review.</p>



<h2 class="wp-block-heading">Failure two: everyone inside the firm sees everything</h2>



<p class="wp-block-paragraph">This shows up in nearly every small-firm build, and the reasoning is always the same. Nine people, everyone helps with everything during filing season, and compartmentalising access is friction exactly when friction is expensive.</p>



<p class="wp-block-paragraph">The case for flat access is real, so say it out loud: a scoped model people fight gets worked around, and a workaround is worse than the flat model you started with. If your access control makes a senior accountant email a PDF to themselves at 11pm, you have made things less safe.</p>



<p class="wp-block-paragraph">So scope by engagement rather than by person, and automate the grant so nobody files a request. Make the key prefix the boundary, then grant the prefix instead of the bucket. Two statements, two jobs: the first controls what someone can enumerate, the second what they can fetch.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOnlyThisClient",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::firm-client-vault",
      "Condition": {
        "StringLike": { "s3:prefix": "clients/acme-llc/*" }
      }
    },
    {
      "Sid": "ReadWriteThisClient",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::firm-client-vault/clients/acme-llc/*"
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Without the prefix condition on <code>ListBucket</code>, a scoped user still gets a full inventory of your client list. Client names are confidential in this business. A directory listing that reads like a book of business is a disclosure even when no file opens.</p>



<p class="wp-block-paragraph">The other half of this family is standing administrative access. Somebody has to fix things, so somebody ends up with a permanent <code>s3:*</code> on the whole bucket. That principal is the highest-value target in the firm and usually has the weakest MFA, because it is a service account nobody wants to touch.</p>



<p class="wp-block-paragraph">Replace it with an assumable break-glass role: MFA required, short session, and an alert on every assumption. The alert is the point. Emergency access used twice a year is fine. Used weekly, it means your normal path is broken.</p>



<h2 class="wp-block-heading">Failure three: encrypted at rest, key sitting right next to it</h2>



<p class="wp-block-paragraph">Default bucket encryption gets ticked in week one and forgotten. It satisfies the letter of &#8220;encrypted at rest&#8221; and protects against exactly one threat: somebody walking out of a data centre with a disk. That is not your threat.</p>



<p class="wp-block-paragraph">Go back to the notification clause. If an attacker reaches a principal that can both read the object and use the key, your data is unencrypted for reporting purposes. The question is not &#8220;was the bucket encrypted&#8221;. It is &#8220;could whoever got in also use the key&#8221;.</p>



<ul class="wp-block-list">
<li>Use a customer managed KMS key. Only that gives you a key policy you control.</li>



<li>Name the principals allowed to call <code>Decrypt</code>, and keep that list shorter than the list allowed to call <code>GetObject</code>. Two different lists means an attacker needs two wins.</li>



<li>Use encryption context so grants can be conditioned per client, and so CloudTrail entries are readable later.</li>



<li>Enable S3 Bucket Keys to cut KMS request volume. KMS bills per API call, so a vault serving many small reads generates a surprising line item.</li>
</ul>



<p class="wp-block-paragraph">Encryption in transit needs its own control, because IAM policies say nothing about the connection. A bucket policy does:</p>



<pre class="wp-block-code"><code>{
  "Sid": "DenyPlaintextTransport",
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": [
    "arn:aws:s3:::firm-client-vault",
    "arn:aws:s3:::firm-client-vault/*"
  ],
  "Condition": {
    "Bool": { "aws:SecureTransport": "false" }
  }
}</code></pre>



<p class="wp-block-paragraph">An explicit deny beats every allow, including one you add by mistake later. Five lines that permanently remove a category of misconfiguration, which makes it the best value item on this page.</p>



<h2 class="wp-block-heading">Failure four: retention rules that fight your disposal duty</h2>



<p class="wp-block-paragraph">Here is the trap, and it is why I would not let an accounting firm switch on Object Lock without a conversation first.</p>



<p class="wp-block-paragraph">Object Lock has two retention modes. In governance mode, a principal holding <code>s3:BypassGovernanceRetention</code> can remove or shorten the lock. In compliance mode, nobody can. Not an administrator, not the account root user. AWS is explicit that the only way to delete a compliance-locked object before its retention date is to close the AWS account.</p>



<p class="wp-block-paragraph">Set that against the Safeguards Rule&#8217;s two-year outer bound on disposal of information you no longer need. Apply a blanket seven-year compliance-mode retention to everything landing in the vault and you have written a promise you cannot revoke, on documents you may later be obliged to destroy.</p>



<ol class="wp-block-list">
<li>Classify by document type before client. Filed returns, engagement letters and supporting workpapers have real statutory retention arguments. Raw intake uploads and scratch scans usually do not.</li>



<li>Default to governance mode. It stops accidental and casual deletion, which is what you need, while keeping a documented and audited escape hatch.</li>



<li>Give <code>s3:BypassGovernanceRetention</code> to almost nobody and alert on every use through CloudTrail.</li>



<li>Reserve compliance mode for records where an external mandate genuinely requires immutability and you chose the period deliberately.</li>



<li>Use legal hold, not a retention period, for litigation. A legal hold has no end date and stays on until explicitly removed, which is right when you cannot know how long a matter runs.</li>
</ol>



<p class="wp-block-paragraph">Two details catch people out. Object Lock requires versioning and normally has to be enabled at bucket creation, so it is a design decision rather than a retrofit. And lifecycle expiration will not delete a version still under retention, so lifecycle and Object Lock are two layers that must agree, not alternatives.</p>



<pre class="wp-block-code"><code># What lock configuration is this bucket actually running?
aws s3api get-object-lock-configuration --bucket firm-client-vault

# Freeze one object version for a matter with no known end date.
aws s3api put-object-legal-hold 
    --bucket firm-client-vault 
    --key clients/acme-llc/2023/return.pdf 
    --legal-hold Status=ON</code></pre>



<p class="wp-block-paragraph">For deletion itself, crypto-shredding is worth knowing. If a client&#8217;s documents sit under a per-client KMS key, scheduling that key for deletion renders every object unreadable without touching the objects. Clean, fast, provable, and irreversible once the waiting period ends, so it belongs behind two-person approval rather than in a script.</p>



<h2 class="wp-block-heading">Failure five: no evidence when somebody asks</h2>



<p class="wp-block-paragraph">Every control above is worth nothing if you cannot show it held. The Rule expects logging of authorized user activity and an annual written report from the Qualified Individual.</p>



<p class="wp-block-paragraph">The trap is that CloudTrail does not record S3 object-level reads and writes by default. Management events cover things like changing a bucket policy. Data events cover the actual <code>GetObject</code> calls, and they are opt-in and billed per event. Firms enable CloudTrail, see a trail in the console, assume they have a record of file access, then find during an incident that they have nothing at object level.</p>



<p class="wp-block-paragraph">Turn on data events for the vault bucket. At firm scale the volume is manageable, and it is the difference between &#8220;we believe two people opened that file&#8221; and being able to show it.</p>



<p class="wp-block-paragraph">Then decide where logs live. Access records for client financial data are themselves sensitive, so keeping them in the same account under the same administrators defeats the purpose. A separate logging account, or at minimum a separate bucket with its own key and retention, is the shape you want. If you already run Grafana Cloud or similar for infrastructure monitoring, pointing vault access logs there gives you dashboards and alerting without standing anything new up.</p>



<p class="wp-block-paragraph">The alerts worth having are boring and few: any use of the break-glass role, any change to the bucket or key policy, any bulk download outside normal working patterns, and any failed decrypt. Four alerts that fire almost never beat forty that everyone mutes.</p>



<h2 class="wp-block-heading">Failure six: the two ends of the pipeline</h2>



<p class="wp-block-paragraph">A vault only protects documents that are in it. The two ends are where firms quietly undo everything.</p>



<h3 class="wp-block-heading">Intake</h3>



<p class="wp-block-paragraph">If clients can still email a W-2 as an attachment, they will, because it is easier. Each attachment becomes customer information sitting in a mailbox, a sent-items folder and a mail archive, each with a different retention policy and access model than your vault.</p>



<p class="wp-block-paragraph">A policy memo will not fix this. Make the vault path faster than the email path: an upload link that works on a phone, no account creation for a one-off drop, and a mail rule that returns certain attachment types with the upload link instead. Then sweep the mailboxes you already have, because that backlog is real customer information and it counts.</p>



<h3 class="wp-block-heading">Offboarding</h3>



<p class="wp-block-paragraph">Seasonal staff are normal here, so offboarding happens in batches and gets rushed. Disabling the SSO login is the part everyone does. The parts that get missed:</p>



<ul class="wp-block-list">
<li>Active role sessions and refresh tokens, which survive the account being disabled until they expire on their own.</li>



<li>Presigned URLs that person generated, which stay valid because they are bound to the signing credential, not the login session.</li>



<li>Named principals still sitting in the KMS key policy.</li>



<li>Local copies on laptops and external drives. When hardware is retired, a certified erase tool such as O&amp;O SafeErase is the difference between disposal and hoping.</li>
</ul>



<p class="wp-block-paragraph">The same applies to service providers. The Rule makes you responsible for selecting providers capable of maintaining appropriate safeguards and holding them to it contractually. If an outsourced bookkeeping team touches your document workflow, that relationship is in scope, and direct network access means they need MFA too. Cloudflare Zero Trust or a business VPN such as NordLayer in front of the portal gives you somewhere to enforce that without rebuilding the application.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting a secure client document vault</h2>



<ul class="wp-block-list">
<li><strong>Links expire far earlier than the expiry you set.</strong> You signed with temporary credentials that expired first. Sign from a principal whose credential outlives the link, or shorten the link and mint on demand.</li>



<li><strong>Uploads fail with access denied despite an allow on PutObject.</strong> With a customer managed key, both the IAM policy and the key policy must permit the caller. Check <code>kms:GenerateDataKey</code>, not just S3 actions.</li>



<li><strong>A file will not delete and no policy seems to block it.</strong> Look for Object Lock retention or a legal hold on that version. They are independent and either one alone blocks the delete.</li>



<li><strong>CloudTrail shows the bucket being created but no file access.</strong> Data events are not enabled. Management events will never show <code>GetObject</code>.</li>



<li><strong>A scoped user can see every client folder name.</strong> The prefix condition is missing from <code>ListBucket</code>, or was applied to the object ARN instead of the bucket ARN.</li>



<li><strong>Lifecycle rules appear to do nothing.</strong> Versions under retention are not expired by lifecycle. Confirm lock state before assuming the rule is wrong.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes, and what to do instead</h2>



<ul class="wp-block-list">
<li><strong>Emailing the download link.</strong> Email a notification that a document is waiting, and mint the link after the client authenticates.</li>



<li><strong>Treating default bucket encryption as finished.</strong> Use a customer managed key and keep the decrypt list shorter than the read list.</li>



<li><strong>Granting bucket-wide list permissions.</strong> Condition <code>ListBucket</code> on the client prefix, or the roster leaks even when no document opens.</li>



<li><strong>Applying compliance-mode Object Lock broadly because it sounds safer.</strong> Write the retention schedule per document class first, get partners to sign it, then configure to match.</li>



<li><strong>Keeping a standing full-access principal.</strong> Replace it with an MFA-gated break-glass role that alerts on every assumption.</li>



<li><strong>Building a portal and leaving email intake open.</strong> Both channels then carry client data and only one is governed. Close the mail path and sweep the backlog.</li>



<li><strong>Storing access logs beside the data they describe.</strong> Ship S3 data events somewhere with different administrators and different keys.</li>



<li><strong>Never testing any of it.</strong> Verify a restore and a scoped-access denial quarterly. An unverified control is an assumption.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does a small accounting firm really fall under the FTC Safeguards Rule?</h3>



<p class="wp-block-paragraph">Almost certainly, if it prepares returns or handles client financial data in the United States. The Rule&#8217;s definition of a financial institution covers activities financial in nature, and the FTC&#8217;s own examples include tax preparation firms. A narrow exception at 16 CFR 314.6 covers institutions holding information on fewer than five thousand consumers, but it exempts only four specific provisions. Encryption, MFA, access controls and disposal still apply. Confirm your own position with counsel, not a blog post.</p>



<h3 class="wp-block-heading">Should I build this or buy a client portal product?</h3>



<p class="wp-block-paragraph">Buy, if the product genuinely does what you need. A firm with no engineering capacity is safer on a maintained product than a bespoke stack nobody patches. Build when you need controls the product will not give you: per-client key isolation, your own retention logic, or integration with a pipeline you already run. The honest test is whether someone will still own the system in a year.</p>



<h3 class="wp-block-heading">Can I self-host a secure client document vault instead of using AWS?</h3>



<p class="wp-block-paragraph">Yes. Nextcloud or similar on a well-run VPS from a provider such as InterServer or Contabo can meet the same requirements, and it is often simpler to reason about. What you take on is patching, backups, TLS renewal and log retention. The regulation does not care where the data sits. It cares that the controls exist and that you can show they held.</p>



<h3 class="wp-block-heading">How long should client documents be retained?</h3>



<p class="wp-block-paragraph">There is no single number, which is why a schedule beats a default. Tax and professional standards drive minimums for filed returns and engagement documentation, and those vary by jurisdiction and engagement type. The Safeguards Rule pushes the other way. Write both directions down per document class, then configure to match.</p>



<h3 class="wp-block-heading">Is a presigned URL safe to send to a client?</h3>



<p class="wp-block-paragraph">It is safe to use, not safe to send. Treat it as a bearer token with a short life, minted after the client authenticates and consumed immediately. What travels over email should be a notification, not the credential.</p>



<h3 class="wp-block-heading">Does encrypting the bucket mean I avoid breach notification?</h3>



<p class="wp-block-paragraph">Only if the key stayed out of the attacker&#8217;s hands. The Rule treats encrypted customer information as unencrypted where the key was accessed by an unauthorized person. That is the whole argument for keeping key permissions narrower than storage permissions.</p>



<h3 class="wp-block-heading">What is the minimum viable version for a two-person practice?</h3>



<p class="wp-block-paragraph">One bucket with per-client prefixes, a customer managed key, an explicit deny on non-TLS transport, MFA on every human account, short mint-on-click links, S3 data events enabled, and a written retention schedule. That is a weekend of work and it closes five of the six failure families above.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A secure client document vault is not defined by the storage you picked. It is defined by how narrow the path is between an attacker and a readable file, and by your ability to prove afterwards how many people walked it.</p>



<p class="wp-block-paragraph">Most firms get the storage right and lose at the edges: a link that outlives the engagement, a key policy as wide as the bucket policy, a retention setting that cannot be undone, an audit trail nobody turned on. Fix those four and the rest is ordinary engineering.</p>



<h2 class="wp-block-heading">Where I can help</h2>



<p class="wp-block-paragraph">I work with small firms and the MSPs that support them on exactly this kind of build. Things I take on regularly:</p>



<ul class="wp-block-list">
<li>Designing and building the vault itself: bucket layout, per-client prefix isolation, KMS key policy, and the portal glue that mints short-lived links.</li>



<li>Reviewing an existing setup against the Safeguards Rule&#8217;s technical elements and producing a gap list your Qualified Individual can work from.</li>



<li>Untangling retention: turning a vague policy into a per-document-class schedule, implemented with lifecycle rules, Object Lock and legal holds that do not conflict.</li>



<li>Standing up the evidence layer: S3 data events, a separate logging account, dashboards, and the small set of alerts worth waking up for.</li>



<li>Closing the email intake path without making life harder for clients, including mobile upload flows and a sweep of the historic mailbox backlog.</li>



<li>Offboarding and access reviews: break-glass roles, session revocation, key policy hygiene, and a repeatable quarterly check.</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send the thing itself: a bucket policy, an IAM policy, a CloudTrail event selector, or a description of how documents reach your clients today. It is much easier to say something useful about a real configuration than about a general question.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/secure-client-document-vault-accounting-firm/">Building a Secure Client Document Vault Your Accounting Firm Can Defend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</title>
		<link>https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Comprehend]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=272</guid>

					<description><![CDATA[<p>A confident answer with a citation from a matter the reader was walled off from. Nothing crashed, nothing alerted. Building legal document intelligence on AWS with Textract, Bedrock and OpenSearch is the easy half; the hard half is tenant isolation, verified deletion, S3 Object Lock holds, what leaves the account, and an audit trail that names the actual user.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo was going well until someone asked where the third citation came from.</p>



<p class="wp-block-paragraph">The answer on screen was good. Fluent, specific, correctly hedged, three sources listed underneath it. Then a partner in the room asked a simple question: which matter is source three from? Nobody could answer it in the room, and when we went and looked, it was from a matter that half the people watching the demo were formally walled off from.</p>



<p class="wp-block-paragraph">Nothing had crashed. No alarm fired. The retrieval layer had done exactly what a similarity search does, which is return the nearest vectors, and the nearest vectors did not care about the ethical wall. That is the thing about building a legal document intelligence platform on AWS: the failures that matter almost never look like failures. They look like a confident answer with a citation attached.</p>



<p class="wp-block-paragraph">This post is about the parts of that build that are hard, organised around the five boundaries that actually have to hold. The extraction pipeline is the easy half. I will cover it, but quickly, because the AWS documentation is good and the failure modes are visible. The other four boundaries fail quietly, and that is where the engineering goes.</p>



<h2 class="wp-block-heading">The shape of the thing</h2>



<p class="wp-block-paragraph">Before the boundaries, the skeleton. A legal document intelligence platform on AWS almost always ends up looking like this, whether you plan it or arrive at it:</p>



<ul class="wp-block-list">
<li>Documents land in Amazon S3, one prefix per matter, versioning on.</li>



<li>An S3 event triggers AWS Lambda, which starts an asynchronous Amazon Textract job.</li>



<li>Textract output lands back in S3 as JSON. Something normalises it into text plus page and bounding-box coordinates.</li>



<li>Amazon Comprehend or a foundation model classifies the document and pulls entities: parties, dates, governing law, clause types.</li>



<li>Chunks get embedded and written to a vector store, usually Amazon OpenSearch Serverless behind Amazon Bedrock Knowledge Bases.</li>



<li>An application layer, typically API Gateway plus Lambda, takes a question, retrieves, and calls a model on Amazon Bedrock.</li>
</ul>



<p class="wp-block-paragraph">AWS Step Functions is worth reaching for once you have more than three stages, because Textract jobs are long-running and Lambda timeouts are not a retry strategy. That is the whole architecture. You can stand it up in a fortnight. Then you spend six months on everything below.</p>



<h2 class="wp-block-heading">Boundary one: isolation, and why metadata filtering is not optional</h2>



<p class="wp-block-paragraph">This is the one from the demo, and it is the one that ends engagements.</p>



<p class="wp-block-paragraph">A vector index has no native concept of a matter, a client, or an ethical wall. If you pool every document into one index and rely on the prompt to keep things separate, you have built a system where a well-phrased question can pull privileged material across a wall. Prompt instructions are not an access control. They are a suggestion that usually works.</p>



<p class="wp-block-paragraph">There are two shapes that do work, and the choice between them is a real trade-off rather than a best practice.</p>



<h3 class="wp-block-heading">Pooled index with server-side filtering</h3>



<p class="wp-block-paragraph">Every chunk carries a <code>matter_id</code> and a <code>client_id</code> as metadata. Every retrieval call attaches a filter. With an S3 data source in Bedrock Knowledge Bases, the metadata comes from a sidecar JSON file that sits next to the document and carries a <code>metadataAttributes</code> object.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "matter_id": "M-4417",
    "client_id": "C-108",
    "doc_type": "engagement_letter",
    "privileged": true
  }
}</code></pre>



<p class="wp-block-paragraph">The retrieval call then narrows the search before the model ever sees a chunk:</p>



<pre class="wp-block-code"><code>"retrievalConfiguration": {
  "vectorSearchConfiguration": {
    "filter": {
      "equals": { "key": "matter_id", "value": "M-4417" }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two rules make this safe, and both are the kind of thing that gets skipped under deadline. First, the filter value is derived on the server from the authenticated caller&#8217;s claims, never accepted from the client. If the browser can send you a <code>matter_id</code>, the browser can send you a different one. Second, the filter is applied by code that no feature request can bypass. A helper that builds every retrieval request, and a code review rule that no other path may call the retrieve API directly.</p>



<p class="wp-block-paragraph">The trap here is ordering. You cannot filter on an attribute the index does not have. Add <code>matter_id</code> after ingestion and every chunk already in the index is invisible to that filter, which means it either returns for everyone or for no one, depending on how your filter is written. Design the metadata schema before the first ingestion run, not after the first demo.</p>



<h3 class="wp-block-heading">Separate collection per client</h3>



<p class="wp-block-paragraph">The heavier option. A separate OpenSearch Serverless collection per client, which buys you a separate AWS KMS key per client, separate index settings, and a failure mode where a bug in the filtering logic cannot reach across clients at all.</p>



<p class="wp-block-paragraph">It costs you sprawl. Every collection is a resource to provision, monitor, patch policy on, and eventually delete. Ingestion jobs run per data source per knowledge base, so freshness guarantees fragment. For a firm with twelve institutional clients this is fine. For a platform onboarding a hundred small clients it becomes the main operational burden of the product.</p>



<p class="wp-block-paragraph">My default is pooled with strict server-side filtering, and per-client collections only where the client&#8217;s own contract demands a dedicated encryption key. If you cannot say out loud which of those two you are running, you are running neither properly.</p>



<h2 class="wp-block-heading">Boundary two: extraction, and the confidence score everyone ignores</h2>



<p class="wp-block-paragraph">Textract returns a confidence score between 0 and 100 with each prediction. AWS is explicit in its own best-practice guidance that applications sensitive to detection errors should enforce a minimum threshold and route anything below it for human scrutiny, and that the right threshold depends entirely on how the output gets used.</p>



<p class="wp-block-paragraph">Legal documents sit at the harsh end of that. A misread date on an archival scan is a curiosity. A misread date on a limitation period is a problem with a name on it.</p>



<p class="wp-block-paragraph">Practical notes from this layer:</p>



<ul class="wp-block-list">
<li>Use the asynchronous operations for anything multipage. <code>StartDocumentAnalysis</code> and <code>GetDocumentAnalysis</code> handle long PDFs and TIFFs; the synchronous <code>AnalyzeDocument</code> path is for single pages and will fight you on a 400-page bundle.</li>



<li>Textract&#8217;s Queries feature earns its place on structured instruments. Instead of extracting everything and grepping, you ask the document a direct question and get a scoped answer with its own confidence.</li>



<li>Keep the page number and bounding box for every chunk. When a lawyer asks where a sentence came from, &#8220;page 14, second column&#8221; is an answer. &#8220;It&#8217;s in the corpus somewhere&#8221; is not, and the platform loses credibility the first time you say it.</li>



<li>Amazon Augmented AI wires low-confidence predictions into a human review queue with a private workforce, so the reviewers are your people rather than an anonymous pool. For privileged material that distinction is the entire point.</li>
</ul>



<p class="wp-block-paragraph">What I would skip early: building a custom entity recognition model. Comprehend supports custom entity recognition and it is genuinely useful, but you need labelled data you do not have yet, and a foundation model with a decent prompt gets you far enough to find out which entities the users actually care about. Train the custom model once the requirement has stopped moving.</p>



<h2 class="wp-block-heading">Boundary three: retention, where deletion is a distributed problem</h2>



<p class="wp-block-paragraph">Here is the failure that will not show up in any test suite you write.</p>



<p class="wp-block-paragraph">A client asks for a document to be removed. Someone deletes the S3 object. The ticket closes. The document is still fully searchable, because its chunks are sitting in the vector index and will stay there until the data source is re-synced. Meanwhile the Textract JSON output is in a second bucket, the extracted text may be in a database, and if model invocation logging is on, whole passages of the document are sitting in CloudWatch Logs or another S3 bucket entirely.</p>



<p class="wp-block-paragraph">One delete, at least four places holding a copy. Write the deletion path as a real workflow with an assertion at the end, not as a single API call:</p>



<ol class="wp-block-list">
<li>Delete the source object and, if versioning is on, its versions.</li>



<li>Delete the derived artefacts: Textract JSON, normalised text, any thumbnails or page images.</li>



<li>Trigger an ingestion job so the knowledge base drops the orphaned chunks.</li>



<li>Confirm removal by querying the index for the document identifier and expecting nothing back.</li>



<li>Deal with the logs, which means either a retention policy short enough to make the problem expire or a deliberate decision, written down, that logs are out of scope.</li>
</ol>



<p class="wp-block-paragraph">Step four is the one people leave out, and it is the only step that actually proves anything.</p>



<h3 class="wp-block-heading">The opposite problem: things that must not be deleted</h3>



<p class="wp-block-paragraph">Legal work has the reverse requirement too. When a matter goes into litigation hold, the documents need to survive an administrator with delete permissions and a bad afternoon.</p>



<p class="wp-block-paragraph">S3 Object Lock is the mechanism, and it has two independent controls that people routinely conflate. A retention period protects an object version until a fixed date, in either governance mode, which privileged users can override, or compliance mode, which nobody can override and where the period cannot be shortened. A legal hold has no date at all. It stays until someone with <code>s3:PutObjectLegalHold</code> explicitly removes it. The two can be active at once, and while either is active the object version cannot be deleted or overwritten.</p>



<pre class="wp-block-code"><code># Place an indefinite hold on a specific object version
aws s3api put-object-legal-hold 
  --bucket matters-archive 
  --key M-4417/exhibit-c.pdf 
  --version-id 3sL7f2Qz9pXvB1kR 
  --legal-hold Status=ON</code></pre>



<p class="wp-block-paragraph">Two things bite here. Object Lock requires versioning and turns it on automatically, and once enabled on a bucket you cannot turn it off or suspend versioning again. And compliance mode is genuinely permanent: if you set a seven-year retention when you meant seven days, that is the answer, for everyone, including the account root. Test in governance mode first. The bypass path exists and is deliberately awkward, requiring both the <code>s3:BypassGovernanceRetention</code> permission and an explicit <code>x-amz-bypass-governance-retention:true</code> header on the request, which is exactly the level of friction you want on that operation.</p>



<p class="wp-block-paragraph">Design the hold model before you design the deletion model, because holds win. A deletion request that collides with an active hold is a legal question, not an engineering one, and the platform&#8217;s job is to surface the collision clearly rather than resolve it silently.</p>



<h2 class="wp-block-heading">Boundary four: disclosure, or what leaves the account</h2>



<p class="wp-block-paragraph">This is the boundary the client&#8217;s general counsel will ask about, usually in writing, usually before signature. Three separate controls, and they are not interchangeable.</p>



<h3 class="wp-block-heading">The AI services opt-out policy</h3>



<p class="wp-block-paragraph">AWS Organizations has a governance policy type that opts your accounts out of having content processed by certain AI services stored and used for service improvement. The list includes Amazon Textract and Amazon Comprehend, which are precisely the two doing the reading in this architecture. The policy type has to be enabled at the organisation root before you can attach anything:</p>



<pre class="wp-block-code"><code>aws organizations enable-policy-type 
  --root-id r-example 
  --policy-type AISERVICES_OPT_OUT_POLICY</code></pre>



<p class="wp-block-paragraph">Read the AWS documentation on this one carefully rather than taking my summary as gospel, because there is a caveat in it that matters: the services may still need to store your data operationally even when you have opted out of it being used for improvement. Opting out is not the same as the data never existing outside your account. Say that plainly in the client conversation. It is a much better position than being asked about it later.</p>



<h3 class="wp-block-heading">Bedrock&#8217;s data position, and the log that undoes it</h3>



<p class="wp-block-paragraph">Amazon Bedrock&#8217;s published position is that inputs and outputs are not shared with third-party model providers and are not used to train the base models, and that fine-tuning operates on a private copy. That is a strong starting point for privileged content and it is the main reason Bedrock rather than a direct provider API shows up in these builds.</p>



<p class="wp-block-paragraph">Then there is model invocation logging. It captures full prompt and response payloads to CloudWatch Logs or S3, and you want it on, because without it you cannot reconstruct what the system told someone. But understand what you have just built: a log that contains privileged document text, at a per-region setting, in a destination that probably has looser access controls than the document store it came from. Encrypt it with the same KMS key discipline, restrict it harder than you think you need to, and set a retention period on purpose.</p>



<p class="wp-block-paragraph">Worth separating in your head: CloudTrail records that an API call happened and who made it. Model invocation logging records what was in it. You need both, for different questions, and only one of them contains client confidences.</p>



<h3 class="wp-block-heading">Network path</h3>



<p class="wp-block-paragraph">Interface VPC endpoints via AWS PrivateLink keep traffic to Bedrock, Textract and the rest on the AWS network rather than out through an internet gateway or NAT. Add a gateway endpoint for S3 while you are there.</p>



<p class="wp-block-paragraph">Endpoint policies are the part that gets forgotten. An endpoint without a policy is a private path to the whole service, including into accounts that are not yours. Scope it to the operations and resources you actually use, and pair it with an S3 bucket policy that rejects requests not arriving through your endpoint. The first protects what leaves; the second protects what can be reached.</p>



<h3 class="wp-block-heading">The parts that are not on the architecture diagram</h3>



<p class="wp-block-paragraph">Two of these have caught people out badly, and neither is an AWS control.</p>



<p class="wp-block-paragraph">The first is the development environment. You do not want real client documents on a laptop or in a scratch account, so build the parsing and chunking code against synthetic documents on a small separate box. A cheap VPS from a provider like Contabo or InterServer is fine for this, and the separation is worth more than the convenience of iterating in the production account.</p>



<p class="wp-block-paragraph">The second is the reviewers. If your human review loop involves contractors working remotely, their network path and their disks are part of your boundary whether or not you drew them. A managed VPN such as NordVPN or Surfshark handles the network side, and when a matter closes and local copies have to go, a dedicated erasure tool from something like O&amp;O Software does what dragging a folder to the bin does not. Unglamorous, and it is the layer that gets audited.</p>



<h2 class="wp-block-heading">Boundary five: evidence, because someone will ask</h2>



<p class="wp-block-paragraph">At some point the question stops being &#8220;does it work&#8221; and becomes &#8220;who saw what, and when&#8221;. If you cannot answer that from stored records, the platform is not defensible regardless of how good the answers are.</p>



<ul class="wp-block-list">
<li>Turn on CloudTrail data events for the document buckets. They are off by default, they are billed separately from management events, and they are the only way to see individual object-level reads.</li>



<li>Log the retrieval, not just the generation. Store which chunks came back and which filter was applied. When somebody asks whether the wall held on a specific query, this record is the answer and there is no reconstructing it later.</li>



<li>Carry the end user&#8217;s identity through to the audit record. A Lambda execution role in the logs tells you the platform did something. It does not tell you who asked.</li>



<li>Alarm on the filter, not just on errors. A retrieval that ran without a matter filter should page someone, even though it returned HTTP 200 and a perfectly good answer.</li>
</ul>



<p class="wp-block-paragraph">That last one is the closest thing to a single takeaway in this post. The dangerous events in this architecture are successful ones.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the things that will go wrong</h2>



<h3 class="wp-block-heading">Retrieval returns nothing after you add a filter</h3>



<p class="wp-block-paragraph">Nearly always a metadata problem rather than a query problem. Either the sidecar metadata file was missing at ingestion, so the chunks carry no attribute to match, or the attribute was added after the chunks were indexed. Check whether the attribute exists on a known chunk before you touch the query. If it does not, you need a re-sync, not a different filter expression.</p>



<h3 class="wp-block-heading">Textract returns text but the layout is scrambled</h3>



<p class="wp-block-paragraph">Common on two-column pleadings and on documents with headers and footers on every page. Raw text detection reads in a reading order that is not always yours. Use the layout and table features rather than plain detection, and hold onto the geometry so you can reassemble columns yourself if the default order is wrong for that document class.</p>



<h3 class="wp-block-heading">The model cites a document that does not exist</h3>



<p class="wp-block-paragraph">Usually the citation is being generated rather than passed through. If document names are being written by the model instead of read from the retrieval response, it will invent plausible ones. Build citations from the retrieval result metadata in your application code and never from the generated text. Bedrock Guardrails helps with contextual grounding, but the structural fix is not asking the model to produce the reference in the first place.</p>



<h3 class="wp-block-heading">Ingestion is slow and nobody knows where</h3>



<p class="wp-block-paragraph">Instrument per stage before you optimise anything. In most of these pipelines the time is in Textract for large scanned bundles, and the fix is parallelism at the document level rather than tuning the extraction itself. Step Functions with a distributed map over documents gets you there without a queue you have to babysit.</p>



<h3 class="wp-block-heading">Costs climb faster than volume</h3>



<p class="wp-block-paragraph">Look at re-processing first. A pipeline that re-extracts a document every time anything downstream changes will quietly multiply your Textract bill against a static corpus. Key the extraction cache on the object&#8217;s content hash, not its path, and make re-ingestion an explicit action. The billing mechanisms differ per service, so read the current pricing pages rather than trusting a number from a blog post.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating the system prompt as an access control.</strong> It is a formatting instruction that happens to look like a rule.</li>



<li><strong>Taking the tenant identifier from the request body.</strong> If the client can name the matter, the client can name someone else&#8217;s.</li>



<li><strong>Designing the metadata schema after the first ingestion run.</strong> Retrofitting a filterable attribute means re-indexing the whole corpus.</li>



<li><strong>Assuming a deleted S3 object is gone from the platform.</strong> The chunks, the extraction output and the invocation logs all outlive it.</li>



<li><strong>Setting compliance-mode retention without testing in governance mode.</strong> There is no support ticket that fixes a seven-year mistake.</li>



<li><strong>Enabling model invocation logging without treating the log as privileged.</strong> You have just made a second copy of the documents in a less protected place.</li>



<li><strong>Building VPC endpoints and leaving the default policy on them.</strong> A private path with no policy is still a path.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>One retrieval helper, server-side filter injection, and a review rule that no other code path calls the retrieve API directly.</li>



<li>Metadata schema agreed and frozen before the first production ingestion. Include the fields you might filter on later, even if unused today.</li>



<li>Page-level provenance on every chunk, surfaced in the interface. It builds trust faster than any accuracy improvement.</li>



<li>A confidence threshold chosen per document class, with a human review path for everything under it.</li>



<li>Deletion implemented as a verified workflow across every store, ending in an assertion that the content is actually unretrievable.</li>



<li>Object Lock legal holds for matters under litigation hold, tested in governance mode before anything runs in compliance mode.</li>



<li>Customer-managed KMS keys across S3, the vector store and the logs, so key access is a control you can actually revoke.</li>



<li>An alarm on unfiltered retrievals, because the failure mode is a success response.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a legal document intelligence platform on AWS safe for privileged material?</h3>



<p class="wp-block-paragraph">The services support it. Bedrock&#8217;s stated position is that inputs and outputs are not shared with model providers or used to train base models, PrivateLink keeps traffic off the public internet, and KMS covers encryption at rest. What determines safety is your own configuration: tenant isolation, log handling and deletion behaviour. The platform is exactly as confidential as its weakest copy of the text.</p>



<h3 class="wp-block-heading">Should I use Bedrock Knowledge Bases or build retrieval myself?</h3>



<p class="wp-block-paragraph">Start with Knowledge Bases. It handles chunking, embedding, sync and metadata filtering, and those are weeks of work with no differentiation in them. Build your own when you need retrieval behaviour it does not expose, such as unusual re-ranking or per-tenant chunking strategies. Going custom on day one usually means reimplementing the managed service badly.</p>



<h3 class="wp-block-heading">How do I stop one client&#8217;s documents surfacing in another client&#8217;s answers?</h3>



<p class="wp-block-paragraph">Tag every chunk with a tenant identifier at ingestion, and inject the matching filter server-side on every retrieval from the authenticated caller&#8217;s identity. Never accept the identifier from the client. If the contract requires per-client encryption keys, use a separate collection per client instead, accepting the extra operational load.</p>



<h3 class="wp-block-heading">Does deleting a document from S3 remove it from the search index?</h3>



<p class="wp-block-paragraph">No. The chunks stay in the vector store until an ingestion job runs and reconciles the data source. Until then the document is deleted and still fully searchable, which is the worst combination available. Always finish a deletion by querying the index and confirming nothing comes back.</p>



<h3 class="wp-block-heading">What is the difference between an S3 legal hold and a retention period?</h3>



<p class="wp-block-paragraph">A retention period runs until a fixed date and comes in governance mode, which privileged users can override, or compliance mode, which nobody can. A legal hold has no end date and stays until someone explicitly removes it. They are independent, they can both be active on the same object version, and while either is active the object cannot be deleted or overwritten.</p>



<h3 class="wp-block-heading">Do I need human review, or is the model accurate enough?</h3>



<p class="wp-block-paragraph">Accuracy is the wrong frame. The question is what happens to a wrong answer downstream. Where output feeds a decision with consequences, you want a confidence threshold and a review queue, and Amazon Augmented AI with a private workforce gives you that without building the review tooling yourself. Where output is a search aid a person will verify anyway, review is friction you do not need.</p>



<h3 class="wp-block-heading">Can I run this in a single AWS account?</h3>



<p class="wp-block-paragraph">Technically yes, and for a pilot it is reasonable. It gets uncomfortable once you have production documents and a development environment in the same place, because an IAM mistake has nowhere to stop. Separate accounts under Organizations also gives you the policy layer, including the AI services opt-out policy, which only exists at the organisation level.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A legal document intelligence platform on AWS is not hard to build. Textract reads, Bedrock reasons, OpenSearch retrieves, and the tutorial version works on the first afternoon.</p>



<p class="wp-block-paragraph">What is hard is that its failures are silent and well-formed. The cross-matter citation, the deleted document that still answers questions, the privileged passage sitting in a log bucket, the retrieval that ran without a filter and returned a beautiful paragraph. None of these throw an error. All of them are the kind of thing that ends a client relationship.</p>



<p class="wp-block-paragraph">So build the boundaries first and the features second, and make sure every one of them is something you can prove from a stored record rather than something you believe about the code. If you can only take one habit from this: alarm on the successful requests that should not have been possible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your document pipeline?</h2>



<p class="wp-block-paragraph">I work on AWS document processing and retrieval systems where the confidentiality requirements are real. Typical things I get called in for:</p>



<ul class="wp-block-list">
<li>Reviewing tenant isolation in an existing RAG setup and finding the paths where the filter can be bypassed.</li>



<li>Designing the metadata and chunking schema before ingestion, so filtering works without a re-index later.</li>



<li>Building Textract pipelines with confidence thresholds, human review routing and page-level provenance.</li>



<li>Implementing verified deletion across S3, derived artefacts, the vector index and logs, with a proof step at the end.</li>



<li>Setting up retention and legal hold with S3 Object Lock, including the governance-mode rehearsal before compliance mode.</li>



<li>Locking down the network and logging path: VPC endpoints with real policies, KMS key separation, and audit records that name the actual user.</li>
</ul>



<p class="wp-block-paragraph">If you have an architecture diagram, a Step Functions definition or a retrieval request you are unsure about, send it over and I will tell you what I would change and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AWS Cross-Account Data Sharing Done Safely: Share Access, Not Copies</title>
		<link>https://john-nessime.com/blog/cloud-computing/aws-cross-account-data-sharing/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 22 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[AWS RAM]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Sharing]]></category>
		<category><![CDATA[External ID]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[IAM Access Analyzer]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Multi-Account Strategy]]></category>
		<category><![CDATA[Resource Control Policies]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=445</guid>

					<description><![CDATA[<p>Adding another account to a bucket policy takes one line and delegates far more than most people realize. A practical look at AWS cross-account data sharing: the delegation model, the KMS gaps, object ownership, references versus copies, and the guardrails that catch the mistake you will eventually make.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-cross-account-data-sharing/">AWS Cross-Account Data Sharing Done Safely: Share Access, Not Copies</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The request usually arrives as one sentence in Slack. &#8220;Analytics moved to their own account, can you just add them to the bucket policy?&#8221;</p>



<p class="wp-block-paragraph">It is a one-line change. It works on the first try. And it is the grant most likely to still be sitting there long after that analytics project shipped, covering a wider set of identities than the person who wrote it ever had in mind.</p>



<p class="wp-block-paragraph">That asymmetry is the whole problem. A denied cross-account request is a nuisance: someone pings you, you read the policy, you fix it before lunch. A cross-account grant that succeeds, keeps succeeding, and quietly reaches further than intended produces no error, no alert, and no ticket. It surfaces months later during an audit, when someone asks who has been reading that prefix and the honest answer is &#8220;anyone the other account decided to let in.&#8221;</p>



<p class="wp-block-paragraph">This post covers AWS cross-account data sharing as a set of failure families rather than a service tour. How the delegation model actually behaves, where encryption keys break or widen access without warning, who ends up owning the bytes after a handoff, why a copy is worse than a reference, and which guardrails catch the mistake you are eventually going to make. Real policy snippets and commands throughout, plus the denials that are hardest to read.</p>



<h2 class="wp-block-heading">AWS cross-account data sharing is a delegation, not a permission</h2>



<p class="wp-block-paragraph">Start with the mechanic that everything else hangs off. For a request that crosses an account boundary, two separate accounts have to say yes. The resource-based policy in the account that owns the data has to allow the caller, and an identity-based policy in the calling account has to allow the action. Neither side alone is enough.</p>



<p class="wp-block-paragraph">That is why &#8220;I added them to the bucket policy and it still fails&#8221; is such a routine ticket. The producer did their half. Nobody attached the matching IAM policy on the consumer side.</p>



<p class="wp-block-paragraph">Now the part that bites. Look at how most cross-account bucket policies get written:</p>



<pre class="wp-block-code"><code>{
  "Sid": "AllowAnalyticsAccount",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ]
}</code></pre>



<p class="wp-block-paragraph">Despite how it reads, that <code>:root</code> suffix does not mean the root user. It means the account. You have not granted access to a principal at all. You have delegated the guest list to the administrators of account 111122223333, and they can hand that access to any principal they like, including a role somebody creates next quarter for a purpose you will never hear about.</p>



<p class="wp-block-paragraph">Sometimes that delegation is exactly what you want, because the consumer team manages its own roles and you do not want to be in the loop for every change. Usually you want the narrower version:</p>



<pre class="wp-block-code"><code>{
  "Sid": "AllowAnalyticsRoleOnly",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ],
  "Condition": {
    "StringEquals": { "aws:PrincipalOrgID": "o-abc123def4" }
  }
}</code></pre>



<p class="wp-block-paragraph">Both halves earn their place. The named ARN stops the wrong role in the right account. The organization condition stops the right role name in the wrong account, which matters if the statement is ever copied, templated, or widened by someone in a hurry. One protects against internal drift, the other against the policy escaping your organization entirely.</p>



<p class="wp-block-paragraph">Be clear-eyed about the trade-off, though. Naming ARNs is precise and brittle: every new consumer role is a policy change, a pull request, and a review. Organization-wide conditions scale beautifully and narrow nothing inside a member account. If your organization contains twelve accounts and forty teams, <code>aws:PrincipalOrgID</code> on its own is a fairly wide door.</p>



<h3 class="wp-block-heading">The grant that stops working after a rebuild</h3>



<p class="wp-block-paragraph">When you name a role ARN as a principal, IAM stores the role&#8217;s internal unique ID behind the scenes. Delete that role and the policy will start displaying an opaque <code>AROA...</code> string where the ARN used to be, because the ARN no longer resolves to anything. Recreate a role with the identical name and the access does not come back, because the new role has a new unique ID.</p>



<p class="wp-block-paragraph">A <code>terraform destroy</code> followed by <code>terraform apply</code> on the consumer side reproduces this perfectly. If you are staring at a bucket policy that contains a principal starting with AROA, that is your answer. Rewrite the statement with the current ARN.</p>



<h2 class="wp-block-heading">The encryption boundary nobody tests until it is live</h2>



<p class="wp-block-paragraph">Second failure family: the key. Encrypting shared data is the easy decision. What people miss is that a KMS key is a second, fully independent authorization surface, with its own resource policy, evaluated separately from the bucket.</p>



<ul class="wp-block-list">
<li>Downloading an object encrypted with SSE-KMS requires <code>kms:Decrypt</code> on the key.</li>

<li>Uploading one requires <code>kms:GenerateDataKey</code>.</li>

<li>A multipart upload requires both.</li>
</ul>



<p class="wp-block-paragraph">The AWS managed key for S3 cannot carry you here, because you cannot edit its key policy. Sharing KMS-encrypted objects across accounts means a customer managed key, with a statement naming the consumer:</p>



<pre class="wp-block-code"><code>{
  "Sid": "AllowAnalyticsRoleToDecrypt",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*",
  "Condition": {
    "StringEquals": { "kms:ViaService": "s3.us-east-1.amazonaws.com" }
  }
}</code></pre>



<p class="wp-block-paragraph">The <code>kms:ViaService</code> condition is cheap insurance. It restricts use of the key to requests that arrive through S3 in that region, so a leaked set of consumer credentials cannot call KMS directly and start decrypting ciphertext it obtained some other way. And the two-key rule applies to the key exactly as it does to the bucket: the consumer&#8217;s own IAM policy has to allow the KMS actions too.</p>



<p class="wp-block-paragraph">Now the sharp edge, which is worth reading twice. If a KMS key is specified by <em>alias</em> rather than by fully qualified ARN, the alias is resolved in the requester&#8217;s account. In a cross-account write that means data landing in your bucket can end up encrypted under a key belonging to the writer. You own the object and cannot read it, and no error is raised at write time. Use full key ARNs in anything that crosses an account boundary.</p>



<h2 class="wp-block-heading">Who owns the bytes after the handoff</h2>



<p class="wp-block-paragraph">Third family: ownership, and it depends entirely on which direction the data moves.</p>



<p class="wp-block-paragraph">New buckets are created with Object Ownership set to Bucket owner enforced. ACLs are off, and the bucket owner owns every object written into the bucket regardless of who wrote it. That default removed the single most confusing trap in S3 sharing.</p>



<p class="wp-block-paragraph">Buckets created before that default are the ones to check. If a bucket still sits on the Object writer setting, an object uploaded by another account is owned by that account, and you, holding the bucket, may not be able to read it. Nothing fails at write time. It fails later, when a job in your own account tries to read your own bucket and gets a 403.</p>



<p class="wp-block-paragraph">Check and fix it with the ownership controls API:</p>



<pre class="wp-block-code"><code>aws s3api get-bucket-ownership-controls --bucket shared-events-bucket

aws s3api put-bucket-ownership-controls 
  --bucket shared-events-bucket 
  --ownership-controls '{"Rules":[{"ObjectOwnership":"BucketOwnerEnforced"}]}'</code></pre>



<p class="wp-block-paragraph">Order matters when you make that change on a live bucket. Once ACLs are disabled, any PUT that still carries an ACL is rejected with <code>AccessControlListNotSupported</code>. Requests that send no ACL, or send <code>bucket-owner-full-control</code>, keep working. So the sequence is:</p>



<ol class="wp-block-list">
<li>Find every writer that sets an ACL, including old scripts using <code>--acl</code> and any SDK call passing an ACL argument.</li>

<li>Update those clients so they stop sending one.</li>

<li>Confirm from CloudTrail that ACL-bearing PUTs have stopped.</li>

<li>Flip the bucket to Bucket owner enforced.</li>
</ol>



<p class="wp-block-paragraph">Given a free choice of design, I would rather the consumer pull from a bucket I own than push into a bucket I do not. Pull keeps ownership, lifecycle rules, encryption configuration, and access logging in one account, under one team&#8217;s control. Push scatters them.</p>



<h2 class="wp-block-heading">Copies you cannot recall</h2>



<p class="wp-block-paragraph">Fourth family, and the one with the longest tail. There is a large difference between granting a reference and shipping a copy. A reference can be revoked on a Tuesday afternoon. A copy is simply gone.</p>



<p class="wp-block-paragraph">Three reference-shaped patterns are worth knowing before you reach for a bulk export.</p>



<h3 class="wp-block-heading">S3 Access Points for object-level sharing</h3>



<p class="wp-block-paragraph">Instead of a bucket policy that grows a new statement every time a team asks for something, you write the bucket policy once to delegate to access points, then give each consumer their own access point with its own focused policy. The delegation is expressed with the <code>s3:DataAccessPointAccount</code> condition. Access points can also be owned by the consumer&#8217;s account, so they manage their own detail without touching your bucket policy.</p>



<p class="wp-block-paragraph">The gotcha: when a request goes through an access point, both the access point policy and the underlying bucket policy have to allow it. A 403 here is easy to misdiagnose because the bucket policy looks correct and is correct. The access point is the layer that said no.</p>



<h3 class="wp-block-heading">Lake Formation for table-level sharing</h3>



<p class="wp-block-paragraph">When the thing being shared is a table rather than a prefix, Lake Formation is the better fit, because it lets you grant on databases, tables, columns, and row filters instead of on object paths. Cross-account grants ride on AWS Resource Access Manager, so the recipient gets an invitation that a data lake administrator has to accept.</p>



<pre class="wp-block-code"><code>aws ram get-resource-share-invitations

aws ram accept-resource-share-invitation 
  --resource-share-invitation-arn arn:aws:ram:us-east-1:111122223333:resource-share-invitation/EXAMPLE</code></pre>



<p class="wp-block-paragraph">After the invitation is accepted, the shared database and tables are visible but still not queryable from Athena or Redshift Spectrum until someone creates a <em>resource link</em>, which behaves roughly like a symlink in the recipient&#8217;s Data Catalog. Missing resource links account for most of the &#8220;the share went through but I cannot see the table&#8221; messages.</p>



<p class="wp-block-paragraph">One setting to check on the producer side is the cross-account data sharing version. Newer versions collapse many grants into far fewer RAM resource shares, which matters once you are sharing at any scale, and they allow granting directly to IAM principals in another account rather than only at account level. Read the current version with:</p>



<pre class="wp-block-code"><code>aws lakeformation get-data-lake-settings</code></pre>



<h3 class="wp-block-heading">When a copy really is the answer</h3>



<p class="wp-block-paragraph">Sometimes the partner genuinely needs the data on their own infrastructure: a warehouse you have no reach into, a VPS at a provider like Contabo or InterServer, or in the worst case an analyst&#8217;s laptop. That is a legitimate outcome, but be honest with yourself and with them that your controls stop at the handoff.</p>



<p class="wp-block-paragraph">Practical hygiene for that case: scope the extract to the columns actually needed rather than the whole table, put an expiry date in the agreement rather than in your head, and write down what happens to the copy when the engagement ends. If the copy lives on machines you or the client control, verified erasure with something like O&amp;O SafeErase reads a great deal better in an audit than &#8220;we deleted the folder.&#8221; And if the recipient has no AWS account at all, a time-limited presigned URL is usually a better answer than minting an IAM user with long-lived keys.</p>



<h2 class="wp-block-heading">The guardrail that catches the mistake you will make</h2>



<p class="wp-block-paragraph">Everything above describes what a careful engineer does per grant. Guardrails are what happens when somebody is not careful, which over a long enough window includes all of us.</p>



<p class="wp-block-paragraph">Resource control policies are the strongest lever if you run AWS Organizations. An RCP sets a ceiling on what any resource-based policy inside the organization can grant. The canonical statement is &#8220;no principal outside my organization may access S3 buckets in my accounts, regardless of what an individual bucket policy says.&#8221; That inverts the risk model: a careless bucket policy is no longer an exposure, it is just an ineffective statement.</p>



<p class="wp-block-paragraph">Two limits are worth stating plainly. RCPs cover a specific set of services rather than everything in AWS, so check the current supported list rather than assuming coverage. And they constrain your resources, not a partner&#8217;s behavior once data has legitimately left. An RCP is a wall around your side of the fence.</p>



<p class="wp-block-paragraph">Service control policies do the mirror-image job on the consumer side, limiting which external accounts your own principals are allowed to assume roles into. Account-level S3 Block Public Access remains a sensible floor underneath both.</p>



<p class="wp-block-paragraph">For sharing with a third party rather than an internal account, the external ID is not optional in my book:</p>



<pre class="wp-block-code"><code>{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::444455556666:role/vendor-connector" },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringEquals": { "sts:ExternalId": "a-value-the-vendor-generated-for-you" }
  }
}</code></pre>



<p class="wp-block-paragraph">The external ID exists because of the confused deputy problem. A vendor holds credentials that can assume roles across many customers, and without a per-customer secret in the condition, customer A can trick the vendor into pointing that access at customer B. If a vendor asks you to create a role for them with no external ID and no conditions at all, that is worth a conversation before it is worth a role.</p>



<h2 class="wp-block-heading">Verifying that the grant is what you think it is</h2>



<p class="wp-block-paragraph">Reading policies is not the same as knowing what they permit. IAM Access Analyzer answers the question &#8220;what is currently reachable from outside my zone of trust&#8221; using automated reasoning rather than pattern matching, which is why it catches combinations a human reviewer skims past.</p>



<pre class="wp-block-code"><code>aws accessanalyzer create-analyzer 
  --analyzer-name external-access 
  --type ACCOUNT

aws accessanalyzer list-findings 
  --analyzer-arn arn:aws:access-analyzer:us-east-1:999988887777:analyzer/external-access</code></pre>



<p class="wp-block-paragraph">Scope the zone of trust to your organization where you have one, then archive the findings that represent intentional shares, so that a new entry in the list actually means something.</p>



<p class="wp-block-paragraph">The same service exposes custom policy checks you can run in a pipeline before a policy change merges, including checks for public access, for specific actions being granted, and for whether an updated policy grants more than the one it replaces. One thing to understand about them: they evaluate the document you hand them and nothing else. They do not read your account state, so a public-access check can fail on a policy in an account that has Block Public Access enabled. For a merge gate that is correct behavior, because the policy should not be relying on a distant safety net.</p>



<p class="wp-block-paragraph">Finally, CloudTrail tells you who actually used a grant, which is the input to the only cleanup that ever really works: delete the statements nobody has exercised. Routing that history into CloudWatch, or into a Grafana Cloud dashboard alongside the rest of your platform telemetry, turns &#8220;is this still needed&#8221; from an argument into a query.</p>



<h2 class="wp-block-heading">Troubleshooting the denials that are hard to read</h2>



<ul class="wp-block-list">
<li><strong>You can list the bucket but every GET returns 403.</strong> That pattern points at KMS, not S3. Listing does not touch the key; reading does. Check <code>kms:Decrypt</code> on both the key policy and the consumer&#8217;s identity policy.</li>

<li><strong>403 through an access point while the bucket policy is clearly correct.</strong> The access point has its own policy and it also has to allow the calling principal. Read that one next.</li>

<li><strong>AssumeRole is denied although the trust policy names the caller.</strong> Cross-account needs both halves: the caller&#8217;s identity policy must permit <code>sts:AssumeRole</code> against that role ARN. Then check the external ID value and any SCP or permissions boundary in the calling account.</li>

<li><strong>A principal in the policy displays as AROA followed by random characters.</strong> The referenced role was deleted. Recreating it under the same name will not restore access.</li>

<li><strong>A Lake Formation share appears in the catalog but queries fail.</strong> Usually a missing resource link in the recipient account, or a RAM invitation that was never accepted.</li>

<li><strong>Access worked yesterday and stopped today with nothing changed locally.</strong> Look at the organization layer before re-reading the bucket policy. A newly attached RCP or SCP denies in a way that no amount of resource-policy editing will fix.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Granting to an account principal when a role ARN would have done, then never revisiting it.</li>

<li>Treating <code>aws:PrincipalOrgID</code> as least privilege. It narrows the organization, not the account.</li>

<li>Sharing a bucket and forgetting the key, then debugging S3 for an hour over a KMS denial.</li>

<li>Referring to KMS keys by alias in cross-account configuration instead of by full ARN.</li>

<li>Letting a partner push into your bucket without confirming the ownership setting first.</li>

<li>Exporting a full table when the consumer needed four columns, because the export was easier to write.</li>

<li>Creating a vendor role with no external ID and no conditions because the vendor&#8217;s setup guide said to.</li>

<li>Granting access with no expiry, no owner, and no note in the policy Sid explaining why it exists.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Give every cross-account grant a descriptive <code>Sid</code>. It is the only free documentation slot IAM offers, and future-you will read it.</li>

<li>Prefer one dedicated consumer role per relationship over a shared role used by several integrations. Revocation becomes a single delete.</li>

<li>Pair a specific principal with a broad condition, rather than relying on either alone.</li>

<li>Put policy checks in CI so a widened grant fails a pull request instead of a quarterly review.</li>

<li>Keep an inventory of active shares with an owner and a review date, even if it is a table in a repository README.</li>

<li>Enable an external access analyzer scoped to your organization and treat new findings as work, not noise.</li>

<li>Watch cross-region and internet data transfer when a consumer sits somewhere unexpected. Cost visibility tooling such as Vantage or CloudZero will attribute it faster than reading a bill.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the safest way to share an S3 bucket with another AWS account?</h3>



<p class="wp-block-paragraph">Name a specific role ARN in the bucket policy, add an organization or account condition alongside it, scope the resource to a prefix rather than the whole bucket, and make sure the consumer&#8217;s identity policy is equally narrow. If the bucket is encrypted with a customer managed key, grant on the key as well. If the list of consumers is going to grow, move to access points rather than growing the bucket policy.</p>



<h3 class="wp-block-heading">Do I need both a bucket policy and an IAM policy?</h3>



<p class="wp-block-paragraph">For cross-account access, yes. The resource-based policy in the owning account and the identity-based policy in the calling account both have to allow the action. That is the single most common reason a correct-looking bucket policy still returns Access Denied.</p>



<h3 class="wp-block-heading">Why do I get Access Denied when the bucket policy clearly allows the other account?</h3>



<p class="wp-block-paragraph">Work through it in this order: the consumer&#8217;s identity policy, then the KMS key policy if the objects are encrypted with a customer managed key, then any access point policy in the path, then organization-level SCPs and RCPs. Denials from the organization layer are the ones people waste the most time on, because nothing in either account&#8217;s policies looks wrong.</p>



<h3 class="wp-block-heading">Can I share data with an account outside my AWS Organization?</h3>



<p class="wp-block-paragraph">You can, and organization conditions will not help you there. Use a named role ARN, an external ID on any assumed role, a tightly scoped resource, and a defined end date. Also confirm whether an RCP in your organization already blocks external principals for that service, because that is a common reason a partner grant refuses to work no matter how carefully it is written.</p>



<h3 class="wp-block-heading">Should I use S3 Access Points or Lake Formation?</h3>



<p class="wp-block-paragraph">It depends on the unit of sharing. If the consumer wants objects under a prefix, access points are simpler and cheaper to operate. If the consumer wants a table, and especially if you need column or row filtering, Lake Formation is the right tool and access points would force you to fake fine-grained control with prefix layout. Plenty of environments run both.</p>



<h3 class="wp-block-heading">How do I revoke cross-account access cleanly?</h3>



<p class="wp-block-paragraph">Remove the statement from the resource policy, remove the KMS key grant, and check for any access point, RAM share, or resource link that also carries permission. Then confirm with CloudTrail that the principal has stopped appearing. Revocation feels done long before it is done, which is exactly why an inventory of active shares pays for itself.</p>



<h3 class="wp-block-heading">Does cross-account access change anything about cost?</h3>



<p class="wp-block-paragraph">The request charges land on the bucket owner by default, and data transfer depends on whether the consumer is in the same region. A consumer that quietly starts scanning a large prefix from another region is a billing event as much as a security one, which is a good argument for reviewing access patterns and spend together rather than in separate meetings.</p>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Getting AWS cross-account data sharing right is less about knowing which service to reach for and more about noticing which grants are quietly larger than they read. An account principal delegates your guest list. An organization condition narrows the wrong dimension. A copy leaves your perimeter permanently and no policy edit brings it back.</p>



<p class="wp-block-paragraph">Grant references rather than copies, name principals rather than accounts, remember that the key is a second door, and put something in place that watches the perimeter for you. The cross-account grant worth worrying about is never the one that failed. It is the one that worked, and kept working, long after anybody was still looking at it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your cross-account setup?</h2>



<p class="wp-block-paragraph">Most of the cross-account problems I see are not exotic. They are a policy written in a hurry two years ago that nobody has read since. If that sounds familiar, here is where I can help:</p>



<ul class="wp-block-list">
<li>Auditing existing bucket, key, and role trust policies to find grants that are wider than intended, and rewriting them to named principals with conditions.</li>

<li>Designing a producer and consumer pattern for a specific data handoff, including whether access points, Lake Formation, or a plain scoped role is the right shape.</li>

<li>Setting up IAM Access Analyzer with a sensible zone of trust and archive rules, so the findings list stays worth reading.</li>

<li>Adding policy checks to your pipeline so a widened grant blocks a merge rather than surviving to production.</li>

<li>Building an organization-level perimeter with resource control policies and service control policies without breaking the integrations you already depend on.</li>

<li>Untangling stubborn cross-account denials, including the KMS, access point, and organization-layer cases that hide well.</li>
</ul>



<p class="wp-block-paragraph">Send me a bucket policy, a trust policy, or the CloudTrail entry for the denial you cannot explain, and I will tell you what it actually permits.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-cross-account-data-sharing/">AWS Cross-Account Data Sharing Done Safely: Share Access, Not Copies</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Where PHI Escapes: Building a Secure Medical Lab Data Pipeline on AWS</title>
		<link>https://john-nessime.com/blog/cloud-computing/medical-lab-data-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 21 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Comprehend Medical]]></category>
		<category><![CDATA[Amazon Macie]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS HealthLake]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[Business Associate Agreement]]></category>
		<category><![CDATA[CLIA]]></category>
		<category><![CDATA[Clinical Data]]></category>
		<category><![CDATA[Crypto Shredding]]></category>
		<category><![CDATA[Data Classification]]></category>
		<category><![CDATA[De-identification]]></category>
		<category><![CDATA[Dead Letter Queue]]></category>
		<category><![CDATA[ePHI]]></category>
		<category><![CDATA[FHIR]]></category>
		<category><![CDATA[Healthcare Cloud]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[HL7 v2]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Multi-Account Strategy]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Pseudonymization]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=455</guid>

					<description><![CDATA[<p>Encryption is the part everyone gets right. The PHI sitting in your logs, dead letter queues and object key names is the part that fails an audit. A boundary-by-boundary guide to building a medical lab data pipeline on AWS that survives review.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/medical-lab-data-pipeline-aws/">Where PHI Escapes: Building a Secure Medical Lab Data Pipeline on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that breaks most healthcare data designs is not technical. It is an auditor asking you to list every place protected health information is stored. Someone points at the encrypted S3 bucket, the encrypted RDS instance, the encrypted backups, and says &#8220;those three.&#8221; Then you grep CloudWatch Logs and find an accession number, a date of birth, and a patient surname sitting inside a Python traceback with no retention policy set on the log group.</p>



<p class="wp-block-paragraph">That is the failure mode that actually bites when you build a medical lab data pipeline on AWS. Not the bucket you thought about. The seven places PHI ended up because a payload got logged, an error got queued, or a filename contained a medical record number.</p>



<p class="wp-block-paragraph">This post walks through the design boundary by boundary: where lab data crosses from one trust zone into another, what leaks at each crossing, and which control closes it. It is aimed at engineers who have been handed an HL7 feed and a compliance checklist and told to make them meet. I am an engineer and not your compliance counsel, so treat the regulatory references here as pointers to read the primary text yourself.</p>



<h2 class="wp-block-heading">Why laboratory data is a harder pipeline than it looks</h2>



<p class="wp-block-paragraph">Lab data has three properties that most ETL work does not.</p>



<p class="wp-block-paragraph">First, results get corrected. A preliminary result goes out, the analyzer is recalibrated, and a corrected result follows hours or days later. Your pipeline is not append-only in the way a clickstream is. It has to carry amendment semantics all the way through, and an analytics table that silently keeps the first value is worse than no table at all.</p>



<p class="wp-block-paragraph">Second, the wire format is old and the transport is older. Most laboratory information systems still emit HL7 version 2 messages, pipe-delimited segments carried over the Minimum Lower Layer Protocol, which is framed bytes over a raw TCP socket. AWS documents MLLP as the common transport standard for HL7 v2 interoperability in its Healthcare Industry Lens. It has no built-in authentication and no built-in encryption. Whatever security it has, you bolt on around it.</p>



<p class="wp-block-paragraph">Third, the identifiers are load-bearing. Accession numbers, specimen IDs and medical record numbers are how the lab reconciles anything, so PHI is not a column you can drop. It is structural, and it will try to escape into every metadata surface you own.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Boundary one: getting messages out of the lab</h2>



<p class="wp-block-paragraph">The instrument or LIS sits on a network you probably do not control, run by a vendor who may not let you touch the configuration. That constrains the design more than anything on the AWS side.</p>



<p class="wp-block-paragraph">You get three realistic shapes, and the LIS vendor usually picks for you.</p>



<ul class="wp-block-list">
<li><strong>Live MLLP over a private link.</strong> An interface engine in the lab holds the TCP session and forwards over a Site-to-Site VPN or Direct Connect into a private subnet. Lowest latency, highest operational burden, because a dropped socket is a clinical incident and not a retry.</li>

<li><strong>Batch file drop over SFTP.</strong> The LIS writes flat files on a schedule and something picks them up. Far easier to reason about, far easier to audit, and acceptable whenever the downstream use is analytics rather than result delivery.</li>

<li><strong>An outbound API push.</strong> Newer systems will POST to an endpoint you expose. Best case, because you control authentication and you get a response code back, but rarest in practice.</li>
</ul>



<p class="wp-block-paragraph">Whichever shape you get, the same rule applies at this boundary: <strong>terminate the untrusted protocol as early as possible and convert to something you can authorize.</strong> A raw MLLP socket carries no identity. The moment those bytes land, wrap them in a message you can attribute, sign, and trace.</p>



<p class="wp-block-paragraph">Two things go wrong here. First, people treat a network tunnel as authentication. A VPN tells you the packets came from the lab&#8217;s network. It does not tell you which system sent them, or that the sender was authorized to send that patient&#8217;s results. Second, people assume consumer VPN products fill the gap. Tools like NordVPN or Surfshark are fine for your own admin laptop on untrusted wifi, but they carry no business associate agreement and they are not a substitute for AWS Site-to-Site VPN or AWS Client VPN on a clinical path.</p>



<p class="wp-block-paragraph">One piece of housekeeping that is cheap to do and expensive to skip: the business associate addendum. AWS makes this self-service through AWS Artifact, and it can be accepted across an entire organization. Only services on the current HIPAA Eligible Services Reference may hold PHI, and that list changes, so check it against the services you actually chose rather than your memory of it.</p>



<h2 class="wp-block-heading">Boundary two: the landing zone</h2>



<p class="wp-block-paragraph">Raw messages land in S3. This part is well documented, so I will focus on the parts people configure once and never verify.</p>



<h3 class="wp-block-heading">Make the bucket policy do the enforcing</h3>



<p class="wp-block-paragraph">Default bucket encryption means objects get encrypted. It does not mean a caller cannot write an object with different settings, and it does not stop a plaintext HTTP request. Two explicit denials cover the gap. The first protects the transport, the second protects the storage:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::lab-raw-inbound",
        "arn:aws:s3:::lab-raw-inbound/*"
      ],
      "Condition": {
        "Bool": { "aws:SecureTransport": "false" }
      }
    },
    {
      "Sid": "DenyWrongKey",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::lab-raw-inbound/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption-aws-kms-key-id":
            "arn:aws:kms:REGION:ACCOUNT:key/KEY-ID"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Use a customer managed KMS key, not the AWS managed one. The reason is not stronger cryptography, it is the key policy. A customer managed key gives you a second, independent authorization surface: even a principal with broad S3 permissions cannot read the object if the key policy does not grant them decrypt. That separation is what makes a blast radius argument credible to an assessor, and it is what makes crypto shredding possible later.</p>



<h3 class="wp-block-heading">Your object keys are metadata, and metadata leaks</h3>



<p class="wp-block-paragraph">This is the one I would put on a poster. Object contents are encrypted. Object <em>keys</em> are not. They show up in CloudTrail data events, S3 server access logs, S3 Inventory reports, bucket listings, Lambda event payloads, and every console screenshot anyone pastes into a ticket.</p>



<pre class="wp-block-code"><code># Leaks PHI into six places that are not the bucket
raw/2024/ORU_MRN4471902_SMITH_JOHN_A1C.hl7

# Same object, no identifiers in the key path
raw/dt=&lt;partition&gt;/src=lis01/msg=01HXYZ...ULID.hl7</code></pre>



<p class="wp-block-paragraph">Use an opaque, sortable identifier and keep the mapping from identifier to patient inside an encrypted store you actually control. Partition on ingest date and source system, never on anything derived from the patient. This one decision removes more PHI surface than any other single change in the pipeline.</p>



<h3 class="wp-block-heading">Immutability, and the trap inside it</h3>



<p class="wp-block-paragraph">Clinical laboratories in the United States operating under CLIA have record retention obligations set out in 42 CFR 493.1105. The floor for test reports is at least two years after the date of reporting, with pathology test reports at ten years, and several state requirements sit above the federal floor. Retention is therefore not something you invent; go and read the section, then check your state.</p>



<p class="wp-block-paragraph">S3 Object Lock gives you write-once-read-many enforcement, and it is the right tool. But understand the two modes before you commit, because one of them is genuinely irreversible.</p>



<ul class="wp-block-list">
<li><strong>Governance mode</strong> blocks deletion for most principals, but anyone holding <code>s3:BypassGovernanceRetention</code> can override it by sending the <code>x-amz-bypass-governance-retention:true</code> header. Note the S3 console includes that header by default.</li>

<li><strong>Compliance mode</strong> cannot be overridden by anyone, including the account root. Per AWS documentation, the only way to delete an object under compliance retention before its date expires is to delete the AWS account.</li>
</ul>



<p class="wp-block-paragraph">A typo in a retention date under compliance mode is permanent, and you pay to store the mistake for its full term. Object Lock also has to be enabled at bucket creation, alongside versioning. Start in governance mode, prove the retention values against real traffic, then move to compliance mode deliberately rather than as a default.</p>



<pre class="wp-block-code"><code># Indefinite hold for a specific object version, e.g. under litigation
aws s3api put-object-legal-hold 
  --bucket lab-raw-inbound 
  --key raw/dt=.../msg=01HXYZ.hl7 
  --legal-hold Status=ON</code></pre>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Boundary three: where PHI escapes into your observability</h2>



<p class="wp-block-paragraph">Here is the section that justifies the post. Everything above is on the compliance checklist. This is the part that is invisible until an audit or an incident makes it visible.</p>



<p class="wp-block-paragraph">Your operational plane is a PHI store you did not declare. Specifically:</p>



<ul class="wp-block-list">
<li><strong>Application logs.</strong> One <code>print(record)</code> left in a parser during debugging writes patient demographics into CloudWatch Logs, where the default is often no expiry at all.</li>

<li><strong>Exception traces.</strong> Unhandled parse errors frequently include the offending segment in the message. That segment is the PHI.</li>

<li><strong>Step Functions execution history.</strong> If you pass message bodies between states as payloads, that content is retained in the execution record and readable by anyone who can view executions.</li>

<li><strong>Dead letter queues.</strong> The whole point of a DLQ is to keep the failed message. A DLQ full of unparseable HL7 is a PHI archive with a different access policy than the bucket you designed so carefully.</li>

<li><strong>Third-party monitoring.</strong> Anything shipped to an external platform leaves your boundary. Grafana Cloud, Datadog and equivalents are perfectly reasonable choices for infrastructure telemetry, but a log line containing PHI going to a vendor without an executed BAA is a disclosure, not a metric.</li>

<li><strong>Notifications.</strong> An SNS topic that emails an on-call engineer &#8220;failed to process record for J. Smith, DOB &#8230;&#8221; has just sent PHI to a mail server nobody reviewed.</li>
</ul>



<h3 class="wp-block-heading">The pattern that fixes it: pass pointers, not payloads</h3>



<p class="wp-block-paragraph">Adopt a single rule and enforce it in review: <strong>PHI moves by reference between components, never by value.</strong></p>



<ol class="wp-block-list">
<li>The ingest function writes the raw message to the encrypted bucket and emits only an opaque object key plus a message ID.</li>

<li>Every downstream state, queue and event carries that pointer. Nothing carries the record body.</li>

<li>Error handling logs the pointer and a failure class, never the content that failed.</li>

<li>A DLQ therefore contains pointers to failures, and reprocessing means re-reading from the bucket under the same key policy as everything else.</li>
</ol>



<p class="wp-block-paragraph">The cost of this pattern is a slightly noisier debugging experience. You cannot read the failing record straight out of the queue. You have to go and fetch it, with credentials, and that fetch is logged. That inconvenience is the control working.</p>



<h3 class="wp-block-heading">Then close the log surface itself</h3>



<p class="wp-block-paragraph">Log groups created implicitly by Lambda and other services often retain data indefinitely unless you set a policy. Find the ones nobody configured:</p>



<pre class="wp-block-code"><code># List log groups with no retention configured
aws logs describe-log-groups 
  --query 'logGroups[?retentionInDays==null].logGroupName' 
  --output text

# Set an explicit retention on one of them
aws logs put-retention-policy 
  --log-group-name /aws/lambda/hl7-ingest 
  --retention-in-days 30</code></pre>



<p class="wp-block-paragraph">Run the first command as a scheduled check, not a one-off. New functions create new log groups, and the default comes back every time someone deploys.</p>



<p class="wp-block-paragraph">Amazon Macie is worth pointing at your buckets as a detective backstop. It will not stop a leak, but it will tell you when something started landing where it should not.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Boundary four: from identified data to analytics</h2>



<p class="wp-block-paragraph">At some point someone wants to query the data. This is the crossing where teams are most likely to convince themselves they have done something they have not.</p>



<p class="wp-block-paragraph">Amazon Comprehend Medical will detect PHI entities in unstructured clinical text through its <code>DetectPHI</code> operation, returning entity types such as name, address, identifier and date, each with a confidence score. It is a genuinely useful tool for narrative fields: pathology comments, specimen notes, microbiology free text.</p>



<pre class="wp-block-code"><code>aws comprehendmedical detect-phi 
  --text "Specimen received from the referring clinic on the stated date."</code></pre>



<p class="wp-block-paragraph">Now the part the marketing pages underplay, straight from the AWS developer guide: the entities Comprehend Medical detects <em>do not map one to one</em> to the identifier list specified by the Safe Harbor method, and AWS explicitly recommends additional human review or other methods to confirm accuracy for compliance use cases.</p>



<p class="wp-block-paragraph">Read that as: PHI detection is not de-identification. Under the HIPAA Privacy Rule, de-identification has two defined routes, Safe Harbor and Expert Determination. An NLP model with confidence scores is an input to either, not a substitute for either. If your analytics tier is meant to hold de-identified data, someone has to own that determination, and it will not be your pipeline.</p>



<p class="wp-block-paragraph">What works structurally:</p>



<ul class="wp-block-list">
<li>Put the analytics tier in a <strong>separate AWS account</strong> with its own KMS key. Not a separate bucket. A separate account, so the blast radius argument is enforced by IAM boundaries rather than by convention.</li>

<li>Make the flow <strong>one directional</strong>. The de-identification job reads from the identified side and writes to the analytics side. Nothing in the analytics account holds permissions pointing back.</li>

<li><strong>Replace identifiers with pseudonyms</strong> rather than deleting them, keeping the crosswalk in the identified account. This preserves your ability to join across results and to re-identify under a documented process, which the research use case usually needs.</li>

<li>Watch <strong>quasi-identifiers</strong>. Rare test panels, unusual reference ranges and precise timestamps re-identify people in small populations even after the obvious fields are gone. This is a statistics problem, not an IAM problem.</li>

<li>Use <strong>Lake Formation</strong> or equivalent for column and row filtering if analysts need partial access, so the restriction lives with the catalog rather than in each query tool.</li>
</ul>



<h2 class="wp-block-heading">Boundary five: sending results back out</h2>



<p class="wp-block-paragraph">Delivery is where clinical correctness and security pull against each other, and security usually loses quietly.</p>



<p class="wp-block-paragraph">The correction problem is the sharp edge. HL7 result messages carry a status that distinguishes preliminary, final and corrected results. If your delivery layer is idempotent on message ID alone, a corrected result with a new ID for the same observation will happily land beside the original instead of superseding it. Two conflicting values, both marked delivered, nobody alerted.</p>



<p class="wp-block-paragraph">Build the idempotency key from the identity of the <em>observation</em>, which typically means the placer or filler order identifier combined with the specific observation identifier, and carry the result status as a first-class field so a correction overwrites rather than appends. Keep every version, expose the current one.</p>



<p class="wp-block-paragraph">For the outbound path itself:</p>



<ul class="wp-block-list">
<li>Presigned URLs are convenient for report PDFs and they are also a bearer credential. Anyone holding the link has the document. Keep expiry short and treat generation as an auditable event.</li>

<li>If you expose a provider-facing portal, put a WAF in front of it. AWS WAF and Cloudflare are both credible here; the deciding factor is usually where the rest of your edge already lives, not the rule engines.</li>

<li>Never email a result. Email a notification that a result is available, behind authentication. The distinction sounds pedantic right up until someone forwards a mailbox.</li>
</ul>



<h2 class="wp-block-heading">Boundary six: deletion, and the conflict nobody plans for</h2>



<p class="wp-block-paragraph">Two obligations point in opposite directions. Retention rules say keep the record. Privacy rights and internal policy say be able to remove data. If you set compliance-mode Object Lock across everything, you have chosen one side without noticing.</p>



<p class="wp-block-paragraph">Resolve it by classifying before you lock. Records that fall under laboratory retention obligations go into the immutable tier with a retention period derived from the applicable rule. Derived artifacts, caches, intermediate parquet, analytics extracts and enrichment outputs do not belong there. They belong in buckets with lifecycle rules, and they are what you actually delete.</p>



<p class="wp-block-paragraph">Crypto shredding covers the middle ground: if a data set is encrypted under a key used for nothing else, scheduling that key for deletion renders the ciphertext unrecoverable without touching the objects. It is clean, it is verifiable, and it only works if you planned the key granularity up front. Retrofitting per-tenant or per-cohort keys onto a pipeline that used one key for everything is a full re-encryption exercise.</p>



<p class="wp-block-paragraph">Do not forget the endpoints. Instrument workstations, the interface engine box, and the laptop somebody used to test the parser all accumulate copies. Cloud controls do nothing for physical media, and secure erase tooling such as O&amp;O SafeErase exists for exactly that step in a decommissioning runbook.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting a medical lab data pipeline on AWS</h2>



<h3 class="wp-block-heading">Messages arrive but nothing appears downstream</h3>



<p class="wp-block-paragraph">Check the DLQ first, then check whether the DLQ itself has a consumer. A surprising number of pipelines have a correctly configured dead letter queue that nobody monitors, so failures accumulate silently and the only symptom is a gap in the data. Alarm on DLQ depth greater than zero, not on a threshold.</p>



<h3 class="wp-block-heading">Access denied from a Lambda that has the right IAM policy</h3>



<p class="wp-block-paragraph">With a customer managed KMS key, IAM permission on S3 is only half the grant. The function&#8217;s role also needs decrypt permission on the key, and the key policy has to allow it. If you are crossing accounts, both the key policy and the role policy must permit the action. This is the intended behavior of the two-lock design, and it is the single most common support question on any pipeline built this way.</p>



<h3 class="wp-block-heading">Deletion fails on an object you are certain you should be able to delete</h3>



<p class="wp-block-paragraph">Check for a legal hold before you check retention. A legal hold has no expiry and is independent of the retention period, so an object can be past its retention date and still undeletable. Also confirm which mode the retention uses, because governance and compliance produce the same error to a caller without bypass permission.</p>



<h3 class="wp-block-heading">Counts do not reconcile with the LIS</h3>



<p class="wp-block-paragraph">Compare on observations, not messages. One message can carry several results, corrections create additional messages for the same observation, and a naive message count will diverge from the LIS by exactly the amount that matters clinically. Reconcile daily and alert on drift rather than investigating at quarter end.</p>



<h3 class="wp-block-heading">Timestamps drift between systems</h3>



<p class="wp-block-paragraph">HL7 v2 timestamps do not always carry a timezone offset, and analyzers are often set to local time with no daylight saving handling. Capture the offset at ingest from the source system configuration and store everything in UTC with the original string preserved. Recovering an ambiguous timestamp after the fact is unpleasant, and around a daylight saving transition it may not be possible at all.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating &#8220;HIPAA eligible&#8221; as &#8220;compliant.&#8221;</strong> Eligibility means AWS will cover the service under the BAA. Configuration is still entirely yours.</li>

<li><strong>Putting identifiers in object keys, queue names, or log group names.</strong> Encrypted contents, plaintext metadata.</li>

<li><strong>Enabling compliance-mode Object Lock before validating retention values.</strong> There is no undo, and no support ticket that fixes it.</li>

<li><strong>Assuming automated PHI detection equals de-identification.</strong> AWS itself says the entity list does not map one to one to Safe Harbor identifiers.</li>

<li><strong>Running the analytics tier in the same account as identified data.</strong> A bucket boundary is a policy away from collapsing. An account boundary is not.</li>

<li><strong>Shipping application logs to an external platform without checking what the log lines contain.</strong> Test it by grepping your own log group for a known test patient name.</li>

<li><strong>Ignoring corrected results until an analyst notices.</strong> Amendment handling is a day-one requirement, not a phase two feature.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li><strong>Write the data flow diagram first and mark every trust boundary crossing.</strong> Every arrow that crosses one needs a named control. This exercise finds more problems than any scanner.</li>

<li><strong>Encode the eligible service list as policy.</strong> Service control policies at the organization level stop someone from putting PHI into a service you never assessed.</li>

<li><strong>Keep VPC endpoints on the PHI path.</strong> Interface and gateway endpoints keep traffic to AWS services off the public internet and give you an endpoint policy as an extra choke point. Check which of your chosen services offer one.</li>

<li><strong>Build with synthetic HL7 from day one.</strong> Nobody should need production PHI to develop a parser. A cheap VPS from Contabo or InterServer is fine for a synthetic-data development box, and keeping that environment entirely outside the PHI boundary is the point.</li>

<li><strong>Alarm on absence.</strong> A feed that stops is more dangerous than a feed that errors, because errors are loud. Alert when expected message volume for a source drops below its floor.</li>

<li><strong>Test restores, not backups.</strong> Restore into an isolated account, confirm the KMS grants work there, and document how long it took.</li>

<li><strong>Review CloudTrail for data events on the PHI buckets.</strong> Object-level logging costs money and is the only record of who read what.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a BAA with AWS before I start building?</h3>



<p class="wp-block-paragraph">Before PHI touches the environment, yes. You can architect and test with synthetic data first. AWS provides the business associate addendum through AWS Artifact as a self-service acceptance that can be applied across an organization, and only services listed on the HIPAA Eligible Services Reference may process, store or transmit PHI.</p>



<h3 class="wp-block-heading">Should I use AWS HealthLake or build on S3, Glue and Athena?</h3>



<p class="wp-block-paragraph">HealthLake is a managed FHIR datastore with FHIR APIs, medical NLP and query built in, and it is the shorter path if your consumers speak FHIR. It does not natively ingest HL7 v2; AWS points to partner tooling or your own transformation for non-FHIR input. If your consumers are analysts with SQL and the data is structured lab results rather than clinical documents, an S3 and Athena lakehouse is simpler and easier to keep inside a boundary you control. Decide on the consumer, not the source.</p>



<h3 class="wp-block-heading">How long do I have to keep laboratory records?</h3>



<p class="wp-block-paragraph">For CLIA-regulated laboratories in the United States, 42 CFR 493.1105 sets the federal floor, including at least two years for test reports after the date of reporting and at least ten years for pathology test reports. State law frequently exceeds these minimums, so the applicable period is whichever is longer for your jurisdiction. Read the regulation and confirm with your compliance lead before you encode a number into a retention policy.</p>



<h3 class="wp-block-heading">Can Amazon Comprehend Medical de-identify data for me?</h3>



<p class="wp-block-paragraph">It can detect PHI entities and give you confidence scores, which is a strong starting point for redaction workflows. It does not by itself satisfy HIPAA&#8217;s de-identification standard. AWS documents that the detected entities do not map one to one to the Safe Harbor identifier list and recommends human review for compliance use cases. Treat it as a detector feeding a documented Safe Harbor or Expert Determination process.</p>



<h3 class="wp-block-heading">Is S3 Object Lock compliance mode required for lab records?</h3>



<p class="wp-block-paragraph">Not inherently. The argument for compliance mode is a threat model in which a compromised or malicious administrator could remove retention, not a regulation that names the mode. Governance mode plus tightly controlled bypass permissions and strong audit logging is a defensible position for many laboratories. Choose deliberately and write down the reasoning, because compliance mode commits you to the storage cost for the full term with no exit.</p>



<h3 class="wp-block-heading">Where does PHI most often leak in a pipeline that looks correctly configured?</h3>



<p class="wp-block-paragraph">Logs, dead letter queues, workflow execution histories, notification messages and object key names. All five are metadata surfaces that sit outside the storage layer everyone reviews. Grep your own log groups for a known test patient identifier and you will find out quickly whether your pipeline has the problem.</p>



<h3 class="wp-block-heading">Do I need a dedicated AWS account just for the lab pipeline?</h3>



<p class="wp-block-paragraph">Separate accounts for identified and de-identified data are worth it even at small scale, because the boundary is then enforced by IAM rather than by naming conventions. Whether the PHI workload also needs isolating from your other production workloads depends on who holds admin access there. If the answer is &#8220;broadly the same people,&#8221; separate it.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A secure medical lab data pipeline on AWS is not defined by the encryption on its storage layer. Encryption is the easy part, and it is the part everyone gets right. What separates a design that survives an audit from one that does not is whether PHI can travel by value through your operational plane: into logs, into queue bodies, into workflow histories, into filenames, into alert emails.</p>



<p class="wp-block-paragraph">Move PHI by reference, keep identifiers out of every metadata surface, and put an account boundary between identified and de-identified data. Do those three things and the rest of the compliance work becomes documentation rather than redesign.</p>



<h2 class="wp-block-heading">Work with me on healthcare data pipelines</h2>



<p class="wp-block-paragraph">I design and review AWS data platforms for regulated workloads, and lab data is a specific enough problem that generic cloud advice tends to miss it. Things I can help with:</p>



<ul class="wp-block-list">
<li>Mapping every trust boundary in an existing pipeline and finding the PHI surfaces nobody documented, including logs, queues and object key schemes.</li>

<li>Designing HL7 v2 or file-drop ingestion into S3 with KMS key separation, VPC endpoints and bucket policies that enforce rather than suggest.</li>

<li>Building the identified and de-identified account split, including the one-way transformation job and the pseudonym crosswalk.</li>

<li>Getting amendment and correction handling right so corrected results supersede rather than duplicate, all the way to the analytics tables.</li>

<li>Retention and deletion design: Object Lock modes, lifecycle policies, crypto shredding key granularity, and the runbook that ties them together.</li>

<li>Observability that is useful without being a disclosure risk, including CloudWatch retention hygiene and safe alerting patterns.</li>
</ul>



<p class="wp-block-paragraph">If you have an architecture diagram, a Terraform plan, or a redacted sample of the messages you are receiving, send it over and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/medical-lab-data-pipeline-aws/">Where PHI Escapes: Building a Secure Medical Lab Data Pipeline on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
