<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Compliance | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/compliance/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/compliance/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 20 Oct 2026 06:00:04 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Compliance | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/compliance/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</title>
		<link>https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 20 Oct 2026 06:00:04 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Append-Only Storage]]></category>
		<category><![CDATA[Audit Evidence]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Backup]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Document Retention]]></category>
		<category><![CDATA[Immutable Storage]]></category>
		<category><![CDATA[Legal Hold]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Service Control Policies]]></category>
		<category><![CDATA[WORM Storage]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=910</guid>

					<description><![CDATA[<p>Enabling S3 Object Lock on a bucket does not protect the objects already in it. This is how to build an audit evidence repository on AWS that survives the question auditors actually ask, covering retention modes you cannot undo, delete markers that hide protected versions, chain of custody, and the storage bill compliance mode makes permanent.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/">Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that ends a smooth audit call is never &#8220;do you have the logs&#8221;. It&#8217;s &#8220;show me why nobody could have edited them.&#8221;</p>



<p class="wp-block-paragraph">That is a different question, and most evidence buckets fail it quietly. Versioning is on, the bucket is encrypted, access logging is enabled, and there is still an admin role in the account that can call <code>DeleteObjectVersion</code> on every object in it. Nothing is wrong until someone asks you to prove a negative.</p>



<p class="wp-block-paragraph">An audit evidence repository on AWS is not a bucket with a good name. It is a write path, a retention decision you cannot take back, and a set of guardrails that survive the person who built them leaving the company. This post covers how to build one with S3 Object Lock and AWS Backup Vault Lock, and more usefully, the five ways I have seen the design go wrong in ways nobody notices until the evidence is needed or the bill arrives.</p>



<h2 class="wp-block-heading">What &#8220;immutable&#8221; actually buys you on AWS</h2>



<p class="wp-block-paragraph">Three separate mechanisms get called immutability, and they protect different things.</p>



<ul class="wp-block-list">
<li><strong>S3 Object Lock</strong> stops an object version from being overwritten or deleted for a defined period. It works only on versioned buckets, and once you enable it on a bucket you cannot disable it or suspend versioning again.</li>



<li><strong>AWS Backup Vault Lock</strong> does the same job for recovery points in a backup vault, with its own retention floor and ceiling enforced at the vault level.</li>



<li><strong>Integrity validation</strong>, such as CloudTrail&#8217;s digest files, proves a file has not changed since delivery. That is a separate property from being undeletable, and auditors increasingly ask for both.</li>
</ul>



<p class="wp-block-paragraph">Object Lock is the core of an audit evidence repository on AWS because evidence usually arrives as files: exported reports, signed approvals, config snapshots, log archives. Vault Lock matters when the evidence is a backup of a database or an EBS volume. Integrity validation is what turns &#8220;we kept it&#8221; into &#8220;we kept it and here is the hash chain&#8221;.</p>



<h2 class="wp-block-heading">Failure one: picking the retention mode you cannot walk back</h2>



<p class="wp-block-paragraph">Object Lock has two retention modes and the difference is not cosmetic. In governance mode, a caller who holds the <code>s3:BypassGovernanceRetention</code> permission can shorten or remove retention and delete the version. In compliance mode, <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html">nobody can</a>, including the account root user, and the retention period cannot be shortened or downgraded to governance.</p>



<p class="wp-block-paragraph">Read that second sentence again with a typo in mind. Set seven years where you meant seven days, in compliance mode, and the only exit AWS documents is closing the account. That is not a bug. It is the entire point of the mode. But it means the mode is a business decision wearing a technical costume.</p>



<p class="wp-block-paragraph">How I decide: compliance mode only where an external rule names a retention period and an external party will inspect it. Everything else starts in governance mode, with the bypass permission granted to nobody by default and denied outright in a service control policy. Governance mode with no principal holding the bypass right is functionally immutable for day-to-day purposes, and it leaves you a documented, auditable escape hatch that requires an explicit policy change to use.</p>



<p class="wp-block-paragraph">The bypass is not silent, either. Overriding governance retention requires the <code>x-amz-bypass-governance-retention:true</code> header on the request, which means the action shows up in CloudTrail as a distinctly shaped call. That is a good alarm to build.</p>



<pre class="wp-block-code"><code># Create the bucket with Object Lock on. Versioning is enabled automatically
# and cannot be suspended afterwards.
aws s3api create-bucket 
  --bucket my-evidence-archive 
  --object-lock-enabled-for-bucket 
  --region eu-west-1 
  --create-bucket-configuration LocationConstraint=eu-west-1

# Default retention for NEW objects written to the bucket.
aws s3api put-object-lock-configuration 
  --bucket my-evidence-archive 
  --object-lock-configuration 
  '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}}'</code></pre>



<p class="wp-block-paragraph">Start there. Run it for a quarter. Watch what actually lands in the bucket and how big it gets. Then decide whether compliance mode is worth its permanence.</p>



<h2 class="wp-block-heading">Failure two: retention that was never applied to the objects that matter</h2>



<p class="wp-block-paragraph">This is the invisible one, and it is the reason to read the rest of this post.</p>



<p class="wp-block-paragraph">Object Lock is enabled at the bucket level, but retention lives on the individual object version. A default retention rule applies to objects written after the rule exists. It does not reach backwards. Enable Object Lock on an existing bucket holding three years of evidence and you get a bucket that reports Object Lock as enabled while every object already in it carries no retention at all.</p>



<p class="wp-block-paragraph">The bucket console looks correct. The compliance checklist gets ticked. The old evidence is still deletable by anyone with delete permissions.</p>



<p class="wp-block-paragraph">Two things to do about it. First, backfill retention on existing versions, either object by object with <code>put-object-retention</code> or, for anything above a few thousand keys, with S3 Batch Operations, which exists precisely for applying retention at scale. Second, verify per object rather than per bucket.</p>



<pre class="wp-block-code"><code># Check what protection a specific version actually carries.
# Look for ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus.
aws s3api head-object 
  --bucket my-evidence-archive 
  --key 2024/q3/access-review-export.csv 
  --version-id "SOME_VERSION_ID"</code></pre>



<p class="wp-block-paragraph">The stronger fix is to stop trusting the writer to set retention correctly and enforce a floor in the bucket policy. S3 exposes the <code>s3:object-lock-remaining-retention-days</code> condition key for exactly this, so you can reject writes that ask for too short a lock.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RejectShortRetention",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-evidence-archive/*",
      "Condition": {
        "NumericLessThan": {
          "s3:object-lock-remaining-retention-days": "365"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The write path now fails loudly when someone gets it wrong, which is what you want. A rejected upload gets noticed. A silently unprotected object does not.</p>



<h2 class="wp-block-heading">Failure three: the delete that looks like a delete</h2>



<p class="wp-block-paragraph">Object Lock protects object versions. It does not stop new versions being created, and it does not stop delete markers being placed on top of a protected object.</p>



<p class="wp-block-paragraph">So a user with ordinary delete permissions can issue a delete against a locked object, get a success response, and make it vanish from every normal listing. The protected version is still there. Nothing was destroyed. But an auditor browsing the bucket sees nothing, and the person who ran the command believes the file is gone.</p>



<p class="wp-block-paragraph">Three practical consequences:</p>



<ul class="wp-block-list">
<li>Deny <code>s3:DeleteObject</code> on the evidence prefix for everyone except a break-glass role. Immutability is not a substitute for permissions, it is a backstop under them.</li>



<li>Any retrieval tooling you hand to an auditor must list object <em>versions</em>, not objects. <code>list-object-versions</code>, not <code>list-objects-v2</code>.</li>



<li>Lifecycle rules interact with this in a way that surprises people. An expiration rule on a versioned bucket adds a delete marker rather than removing data, and a rule that tries to permanently expire a noncurrent version will not remove a version still under retention. Your cleanup policy and your retention policy are two different clocks, and the lock always wins.</li>
</ul>



<h2 class="wp-block-heading">Failure four: evidence that arrives already unprovable</h2>



<p class="wp-block-paragraph">Immutability starts when the object lands. Everything before that is chain of custody, and it is where most repositories are weakest.</p>



<p class="wp-block-paragraph">If the evidence is CloudTrail, use the integrity validation CloudTrail already ships. With log file validation enabled, CloudTrail writes an hourly digest file containing hashes of the log files delivered in that hour, signs each digest with a per-region private key, and puts the digests in a separate folder in the same bucket. You can then verify the chain later.</p>



<pre class="wp-block-code"><code># Turn on digest files for an existing trail.
aws cloudtrail update-trail 
  --name org-audit-trail 
  --enable-log-file-validation

# Verify a time range later. Files must still be in the location
# CloudTrail delivered them to, or validation cannot run.
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-audit-trail 
  --start-time 2020-01-01T00:00:00Z</code></pre>



<p class="wp-block-paragraph">Two details worth knowing before you design around it. Validation only covers log files referenced by digest files, so a gap in digest coverage is a gap in your proof. And the files must stay where CloudTrail put them, which means copying logs into a tidier prefix breaks the built-in verification and hands you the job of writing your own checker.</p>



<p class="wp-block-paragraph">For evidence that is not CloudTrail, the equivalent is doing the work yourself: hash the artifact at the point of generation, record the hash somewhere separate from the artifact, and write both. A manifest per collection run, checksummed and locked alongside the files, is unglamorous and holds up.</p>



<p class="wp-block-paragraph">One constraint that catches people mid-build: a bucket with Object Lock enabled cannot be used as the destination for S3 server access logs. If your design had one bucket receiving both the evidence and its own access logs, it needs splitting.</p>



<h2 class="wp-block-heading">Failure five: the bill you cannot switch off</h2>



<p class="wp-block-paragraph">Compliance mode plus a long retention period plus archival storage classes is where cost surprises live, because every lever you would normally pull to reduce spend has been deliberately disabled.</p>



<p class="wp-block-paragraph">The mechanics that drive it, none of which are secret but all of which get discovered late:</p>



<ul class="wp-block-list">
<li><strong>Minimum storage durations.</strong> S3 Glacier Flexible Retrieval carries a 90 day minimum and S3 Glacier Deep Archive carries 180 days. Transition earlier than that and you are billed for the remainder of the minimum anyway.</li>



<li><strong>Per-object overhead in Glacier classes.</strong> AWS documents 40 KB of metadata overhead per object for Glacier Flexible Retrieval and Deep Archive, of which 8 KB is billed at S3 Standard rates and 32 KB at the destination rate. For a repository full of 2 KB JSON approvals, the overhead is the bill.</li>



<li><strong>Billing starts when the rule is satisfied.</strong> Lifecycle transitions to Glacier classes happen asynchronously, but you are charged at the destination rate, and the minimum duration clock starts, from the date the rule matched rather than the date the bytes physically moved.</li>



<li><strong>You cannot delete your way out.</strong> Under compliance mode, oversized or mistakenly retained objects stay until their retain-until date passes, whatever the finance conversation says.</li>
</ul>



<p class="wp-block-paragraph">The design response is to aggregate before you lock. Batch small artifacts into per-day or per-collection archives, checksum the archive, and lock the archive rather than thousands of tiny files. It costs you a little retrieval convenience and saves you a per-object overhead multiplier you cannot renegotiate later.</p>



<p class="wp-block-paragraph">Also pick the retrieval tier against the real deadline. Deep Archive is the cheapest place to keep evidence and one of the slowest to get it back. If a regulator or a legal team may need files at short notice, that trade is a bad one regardless of the storage line item.</p>



<h2 class="wp-block-heading">Putting the repository together</h2>



<p class="wp-block-paragraph">With the failure families in view, the build order is fairly short.</p>



<ol class="wp-block-list">
<li><strong>Separate the account.</strong> Evidence lives in its own AWS account, not a prefix in the production log bucket. The blast radius of a compromised production admin should not include the record of what they did.</li>



<li><strong>Write the service control policies first.</strong> At the organisation level, deny <code>s3:BypassGovernanceRetention</code>, deny <code>s3:PutBucketPolicy</code> and <code>s3:DeleteBucketPolicy</code> on the evidence bucket, and deny disabling the trail. SCPs apply to the account root too, which is the point.</li>



<li><strong>Create the bucket with Object Lock and a default governance retention.</strong> Block public access, enforce TLS with an <code>aws:SecureTransport</code> condition, and encrypt with a customer managed KMS key whose policy is written for this bucket alone.</li>



<li><strong>Enforce the retention floor in the bucket policy</strong> using the remaining-retention-days condition, so a misconfigured collector fails rather than writes something unprotected.</li>



<li><strong>Replicate to a second account or region.</strong> Object Lock works with S3 Replication, and the destination bucket must also have Object Lock enabled. This is your answer to region-level loss and to an account-level mistake.</li>



<li><strong>Add Vault Lock for backup-shaped evidence.</strong> Snapshots and database backups belong in a locked backup vault rather than being exported to S3 by hand.</li>



<li><strong>Alarm on the write path.</strong> A day with no new evidence objects is the failure that never pages anyone. Alert on absence, not just on errors.</li>
</ol>



<p class="wp-block-paragraph">Vault Lock deserves its own note because its compliance mode has a grace period that Object Lock does not. You choose a cooling-off window, which AWS requires to be at least three days, during which the lock can still be removed. After that the vault and its lock are immutable and cannot be changed or deleted by anyone, including AWS.</p>



<pre class="wp-block-code"><code># Compliance mode: the presence of --changeable-for-days is what makes it
# compliance rather than governance. Minimum value is 3 (72 hours).
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name evidence-vault 
  --min-retention-days 365 
  --max-retention-days 2555 
  --changeable-for-days 3</code></pre>



<p class="wp-block-paragraph">Use that window. Run a backup job through the vault, confirm the retention arithmetic against what the policy document actually requires, and only then let the clock run out.</p>



<p class="wp-block-paragraph">One more design question worth asking out loud: does every copy need to be inside AWS? A second, offline-ish copy on independent infrastructure answers a different threat than cross-region replication does, because it survives an account-level compromise. Providers like InterServer, Contabo and Hetzner sell cheap storage that can hold an encrypted archive, and several S3-compatible services implement Object Lock semantics of their own. Confirm the provider actually supports Object Lock rather than assuming an S3-compatible API implies it, because that varies.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">AccessDenied when deleting a version you are allowed to delete</h3>



<p class="wp-block-paragraph">Usually governance retention plus a missing bypass. The IAM permission <code>s3:BypassGovernanceRetention</code> and the explicit bypass on the request are both required. If the object is in compliance mode, there is no combination of permissions that will work, and the error is telling you the truth.</p>



<pre class="wp-block-code"><code># Only valid for governance mode, and only with the matching IAM permission.
aws s3api delete-object 
  --bucket my-evidence-archive 
  --key path/to/object 
  --version-id "SOME_VERSION_ID" 
  --bypass-governance-retention</code></pre>



<h3 class="wp-block-heading">Replication fails on an Object Lock bucket</h3>



<p class="wp-block-paragraph">The destination needs Object Lock enabled too. Replicated objects take the lock configuration of the source, so a destination without Object Lock cannot accept them. Check the destination bucket first, then the replication role&#8217;s permissions.</p>



<h3 class="wp-block-heading">Lifecycle rule appears to do nothing</h3>



<p class="wp-block-paragraph">Two likely causes. The versions it targets are still under retention, so expiration cannot remove them. Or the rule targets current versions on a versioned bucket, in which case it is adding delete markers and the data is still there as noncurrent versions, still being billed.</p>



<h3 class="wp-block-heading">validate-logs reports invalid or missing files</h3>



<p class="wp-block-paragraph">Confirm nothing moved the objects. Validation requires the log and digest files to be in the location CloudTrail delivered them to, and the command runs per region. A copy job that reorganised the prefix layout is the usual culprit.</p>



<h3 class="wp-block-heading">Object Lock cannot be enabled on the bucket</h3>



<p class="wp-block-paragraph">Versioning has to be enabled first, and Object Lock cannot be turned off later. If the bucket is currently a server access log destination, that has to move before Object Lock will work for you at all.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Enabling Object Lock on an existing bucket and assuming the objects already in it are now protected. They are not, until retention is applied to each version.</li>



<li>Choosing compliance mode on day one to look thorough, then discovering the retention period was wrong.</li>



<li>Treating Object Lock as an access control. It is a floor under your permissions, not a replacement for them.</li>



<li>Leaving the bypass permission attached to a general administrator role because it was needed once during testing.</li>



<li>Locking millions of tiny objects into Glacier classes and paying per-object overhead on all of them.</li>



<li>Storing evidence in the same account as the systems it describes, which makes the evidence only as trustworthy as that account&#8217;s admins.</li>



<li>Never testing retrieval, so the first restore attempt happens under audit pressure with an unfamiliar Glacier restore workflow.</li>
</ul>



<h2 class="wp-block-heading">Best practices for an audit evidence repository on AWS</h2>



<ul class="wp-block-list">
<li>Governance mode by default, compliance mode only where an external requirement names the period. Write down which rule drove the choice, in the same repository.</li>



<li>Enforce the retention floor in the bucket policy so the write path fails loudly instead of writing unprotected objects.</li>



<li>Use legal hold, not extended retention, for investigations. Legal holds have no expiry and are removed explicitly, which is the right shape for something that ends on an unknown date.</li>



<li>Aggregate small artifacts into archives with a checksummed manifest before locking them.</li>



<li>Replicate to a second account with Object Lock enabled, and check replication metrics rather than assuming.</li>



<li>Rehearse retrieval quarterly, including the Glacier restore step, and keep the output as evidence of the rehearsal.</li>



<li>Monitor for absence of writes. Tools like Amazon CloudWatch, Grafana or Datadog are all fine here. The alert that matters is &#8220;no evidence arrived today&#8221;.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I enable S3 Object Lock on a bucket that already exists?</h3>



<p class="wp-block-paragraph">Yes. S3 supports enabling Object Lock on existing buckets, and versioning must be enabled. The important caveat is that enabling it does not retroactively protect the objects already stored. You apply retention to those separately, per object or with S3 Batch Operations.</p>



<h3 class="wp-block-heading">What is the difference between governance mode and compliance mode?</h3>



<p class="wp-block-paragraph">Governance mode can be overridden by a principal holding <code>s3:BypassGovernanceRetention</code> who sends the bypass header. Compliance mode cannot be overridden by anyone, including the root user, and the retention period cannot be shortened or the mode downgraded.</p>



<h3 class="wp-block-heading">Does S3 Object Lock stop someone deleting the file?</h3>



<p class="wp-block-paragraph">It stops the protected version being destroyed. It does not stop a delete marker being placed on top, which hides the object from ordinary listings while the version remains. Keep delete permissions tight and list versions when you audit.</p>



<h3 class="wp-block-heading">Do I need Object Lock if I already have versioning and MFA delete?</h3>



<p class="wp-block-paragraph">They solve different problems. Versioning preserves history but a sufficiently privileged caller can still remove versions. Object Lock removes that possibility for the duration of the retention period, which is the property an auditor is asking about.</p>



<h3 class="wp-block-heading">How do I prove the evidence has not been modified?</h3>



<p class="wp-block-paragraph">For CloudTrail, enable log file integrity validation and keep the signed digest files with the logs, then run <code>aws cloudtrail validate-logs</code> for the period in question. For other artifacts, hash at generation time, store the manifest separately, and lock both.</p>



<h3 class="wp-block-heading">Can a lifecycle rule delete objects that are still locked?</h3>



<p class="wp-block-paragraph">No. Retention wins over lifecycle expiration. Plan the two together, because a cleanup rule that quietly does nothing for years is a cost problem you will not see until you look at storage growth.</p>



<h3 class="wp-block-heading">Is Vault Lock compliance mode reversible?</h3>



<p class="wp-block-paragraph">Only during the cooling-off window you set at creation, which must be at least three days. After the lock date, the vault and its lock cannot be changed or deleted by anyone, including AWS.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An audit evidence repository on AWS fails in a specific, quiet way: the bucket says Object Lock is enabled, and the objects inside carry no retention. Everything else in this post is downstream of that gap. Check protection at the object version, not at the bucket, and enforce the floor in the write path so the system tells you when it is wrong.</p>



<p class="wp-block-paragraph">Then pick the mode like the one-way door it is. Governance mode with a locked-down bypass gets you almost all of the protection and keeps a documented way back. Compliance mode is worth its permanence when a real external rule demands it, and expensive theatre when it does not.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or reviewing one of these?</h2>



<p class="wp-block-paragraph">Immutable retention is easy to enable and easy to get subtly wrong, and the mistakes are the kind you find out about at the worst moment. This is work I take on directly:</p>



<ul class="wp-block-list">
<li>Designing an evidence account with Object Lock, KMS, service control policies and a locked-down write path</li>



<li>Auditing an existing bucket to find object versions that carry no retention despite Object Lock being enabled</li>



<li>Backfilling retention across existing objects with S3 Batch Operations without locking in the wrong period</li>



<li>Modelling the long-run cost of a retention decision before compliance mode makes it permanent</li>



<li>Setting up cross-account replication, AWS Backup Vault Lock and CloudTrail integrity validation as one coherent design</li>



<li>Writing the retrieval runbook and rehearsing it, so the first restore is not during an audit</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion on what you already have, send me the bucket policy, the Object Lock configuration and the output of a <code>head-object</code> call on one old evidence file. That is usually enough to tell you where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/">See how I work on projects</a></div>



<div class="wp-block-button is-style-outline is-style-outline--1"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/">Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Amazon Macie PII Detection: The Buckets It Never Opened</title>
		<link>https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 06 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Data Firehose]]></category>
		<category><![CDATA[Amazon Macie]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Classification]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[Security Hub]]></category>
		<category><![CDATA[Sensitive Data Discovery]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=405</guid>

					<description><![CDATA[<p>A Macie bucket labeled "Not sensitive" often just means Macie never read it. Extensionless objects, unsupported storage classes, unreachable KMS keys and quota truncation all produce silence that looks identical to a clean result. Here's how to measure coverage, fix the four gaps, tune identifiers, and keep the bill honest.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email came from legal, not from security. A customer had exercised a data access request, someone pulled the export by hand, and it contained email addresses and phone numbers sitting in a bucket Macie had labeled <em>Not sensitive</em>. Nobody had done anything wrong. The console was telling the truth as it understood it. It just didn&#8217;t understand very much about that bucket.</p>



<p class="wp-block-paragraph">That is the failure mode worth internalizing before anything else: Amazon Macie PII detection reports on what it managed to read, not on what is there. Objects it never opened produce no findings, and no findings looks identical to a clean result. The gap between &#8220;we scanned this&#8221; and &#8220;we found nothing&#8221; is where most Macie deployments quietly fail their first audit.</p>



<p class="wp-block-paragraph">This post covers where that gap comes from and how to close it: the eligibility chain every object goes through, the four distinct reasons an object gets skipped, how to tune identifiers so the findings are worth reading, what actually drives the bill, and how to route results somewhere a human will act on them.</p>



<h2 class="wp-block-heading">The eligibility chain behind Amazon Macie PII detection</h2>



<p class="wp-block-paragraph">Every object passes the same gates before a byte gets inspected. The order matters, because each gate fails differently and needs a different fix.</p>



<ol class="wp-block-list">
<li>Is it in a general purpose S3 bucket? Directory buckets are out of scope entirely.</li>



<li>Is the storage class supported? Standard, Standard-IA, One Zone-IA, Intelligent-Tiering, Glacier Instant Retrieval and Reduced Redundancy are in. Glacier Deep Archive and S3 Express One Zone are not.</li>



<li>Does the object key carry a recognized file extension? This is a string check on the name, not a content sniff.</li>



<li>Can Macie retrieve and decrypt it, given the bucket policy, object ACL and encryption key?</li>



<li>Does the content parse, and stay inside the per-file quotas?</li>



<li>Do the active data identifiers match anything in it?</li>
</ol>



<p class="wp-block-paragraph">Only the last gate produces a finding. Everything before it produces silence. Macie does record why each object was skipped, but that lives in coverage data and object samples, not on the findings page most people look at.</p>



<p class="wp-block-paragraph">One detail explains a lot of confusion: when you first enable automated sensitive data discovery, every bucket gets a sensitivity score of 50 and the label <em>Not yet analyzed</em>. A bucket whose permissions block Macie stays there permanently. It never turns red. It sits mid-list, looking unremarkable next to buckets that genuinely were analyzed.</p>



<h2 class="wp-block-heading">Gap one: objects Macie never opened</h2>



<p class="wp-block-paragraph">Macie calls these <em>unclassifiable</em>. They fail the storage class or extension check, so no retrieval is attempted. This is the largest source of false confidence I&#8217;ve seen, and the cheapest to fix.</p>



<h3 class="wp-block-heading">Missing file extensions</h3>



<p class="wp-block-paragraph">Classifiability comes from the file name extension. A file full of customer records named <code>part-00003-a4f9</code> is invisible. The same bytes named <code>part-00003-a4f9.json</code> get inspected and produce findings.</p>



<p class="wp-block-paragraph">This bites hardest on streaming ingestion. Amazon Data Firehose writes to S3 without appending an extension unless compression, format conversion, or the explicit file extension setting adds one. An uncompressed JSON stream lands as a tree of extensionless objects. Athena and Glue read them fine, because they infer format from the table definition. Macie skips every one.</p>



<p class="wp-block-paragraph">Fix it upstream: set the S3 file extension on the delivery configuration, or enable GZIP compression or Parquet conversion, both of which append a recognized extension. If you can&#8217;t change the producer, the fallback is a copy-and-rename step into a scan prefix, which costs you storage plus a pipeline to maintain. Fixing the producer is almost always cheaper.</p>



<h3 class="wp-block-heading">Unsupported storage classes</h3>



<p class="wp-block-paragraph">A lifecycle rule transitioning old exports to Glacier Deep Archive quietly removes them from scope. That may be the right storage decision, but it needs to be a conscious one, because archived customer data is still customer data when a regulator asks.</p>



<p class="wp-block-paragraph">Macie exposes these counts per bucket, split by cause. This is the first query I run against a new account:</p>



<pre class="wp-block-code"><code># Buckets where Macie can't classify objects, broken down by cause
aws macie2 describe-buckets 
  --query 'buckets[?unclassifiableObjectCount.total &gt; `0`].{
      bucket: bucketName,
      classifiable: classifiableObjectCount,
      skipped_total: unclassifiableObjectCount.total,
      skipped_extension: unclassifiableObjectCount.fileType,
      skipped_storage_class: unclassifiableObjectCount.storageClass
    }' 
  --output table</code></pre>



<p class="wp-block-paragraph">Read the ratio, not the raw number. Forty thousand classifiable objects and twelve skipped is fine. Three classifiable and four hundred thousand skipped is a reporting artifact pretending to be a scan result, and that bucket&#8217;s sensitivity score means nothing.</p>



<h2 class="wp-block-heading">Gap two: objects it opened and couldn&#8217;t read</h2>



<p class="wp-block-paragraph">These are classification errors rather than eligibility failures. Macie selected the object, tried to fetch it, and was refused. Three causes, and they need three different people to fix them.</p>



<ul class="wp-block-list">
<li><strong>Customer-provided keys (SSE-C).</strong> Macie cannot supply the key material, so it cannot retrieve the object. No permissions fix exists. Re-encrypt with S3 managed or KMS keys if you want coverage.</li>



<li><strong>KMS key policy.</strong> For customer managed keys, the key policy must allow the Macie service-linked role, <code>AWSServiceRoleForAmazonMacie</code>, to decrypt. Cross-account buckets need the key owner to grant it, not the bucket owner. AWS publishes a permission analyzer script in the <code>aws-samples/amazon-macie-scripts</code> repository that enumerates every key Macie needs and generates the CLI commands to fix them.</li>



<li><strong>Restrictive bucket policies.</strong> An explicit <code>Deny</code> conditioned on source IP or VPC endpoint blocks Macie along with everything else. The working pattern excludes the service-linked role ARN from the deny using the <code>aws:PrincipalArn</code> condition key.</li>
</ul>



<p class="wp-block-paragraph">To see which objects Macie actually touched in a bucket, pull the object samples. It&#8217;s the closest thing to a scan log you get:</p>



<pre class="wp-block-code"><code># Objects automated discovery selected, with per-object status
aws macie2 list-resource-profile-artifacts 
  --resource-arn arn:aws:s3:::your-bucket-name</code></pre>



<p class="wp-block-paragraph">Anything with a status of <code>SKIPPED</code> is a lead. Take the key, check its encryption settings in S3, and you usually have your answer inside a minute.</p>



<h2 class="wp-block-heading">Gap three: it read the file and nothing matched</h2>



<p class="wp-block-paragraph">Macie ships managed data identifiers covering common PII, financial data and credentials across many countries. Automated discovery uses a recommended subset by default rather than all of them. Sensible, but it is a default, and it will miss things that matter to you.</p>



<p class="wp-block-paragraph"><strong>Add the managed identifiers for your actual jurisdictions.</strong> If you hold records for customers in a country whose national ID identifier isn&#8217;t in the recommended set, you&#8217;re scanning for the wrong things with perfect efficiency. Pull the current list with <code>aws macie2 list-managed-data-identifiers</code> and compare it against where your customers live.</p>



<p class="wp-block-paragraph"><strong>Write custom identifiers for what only you know is sensitive.</strong> Internal account numbers, case references, employee IDs. A custom identifier is a regex plus optional keywords and a proximity rule, and the keywords are what save you. A bare <code>[0-9]{8}</code> matches timestamps, order totals and row counts, and you drown. The same regex with keywords and a match distance only fires when the number sits near a word that gives it meaning.</p>



<p class="wp-block-paragraph">Test the pattern before it goes near a job. This runs the criteria against sample text and returns match counts without creating anything:</p>



<pre class="wp-block-code"><code># Dry-run a custom identifier before creating it
aws macie2 test-custom-data-identifier 
  --regex 'ACC-[0-9]{4}-[0-9]{4}' 
  --keywords account customer acct 
  --maximum-match-distance 30 
  --sample-text 'customer account ACC-4821-9930 updated; total 1234-5678'</code></pre>



<p class="wp-block-paragraph">Run it against real samples, including the awkward ones, then against a file you know is clean and confirm zero. A pattern you only tested on positives is a pattern you haven&#8217;t tested.</p>



<p class="wp-block-paragraph">The mirror image is allow lists, which tell Macie to ignore specific text or patterns: your support inbox address, your published press contact, the seeded test records in everyone&#8217;s staging fixtures. Without them the same non-issues appear in every scan until the team stops reading findings. Allow lists accept predefined text or regex, and the per-account and per-job quotas are tight enough that you should curate rather than dump.</p>



<h2 class="wp-block-heading">Gap four: quota truncation inside large files</h2>



<p class="wp-block-paragraph">Subtle, because the object <em>is</em> analyzed. Just not completely, and a partial result reports like a full one.</p>



<ul class="wp-block-list">
<li>Per-file size quotas differ by format. Exceed the quota for a type and Macie analyzes none of that file, not part of it.</li>



<li>Archives have separate ceilings for nesting depth, extracted bytes and file count. If the metadata shows a breach up front, nothing is extracted. Cross a limit partway through and Macie stops, reporting only what it processed.</li>



<li>JSON and JSON Lines have a nested depth limit. Past it, the file is not analyzed at all.</li>



<li>Full names and mailing addresses cap out per file. After the cap Macie stops counting and stops reporting locations for that type, so the occurrence count is a floor, never a total.</li>
</ul>



<p class="wp-block-paragraph">Findings carry a status reason naming the limit that was hit, with values like <code>FILE_EXCEEDS_SIZE_LIMIT</code>, <code>ARCHIVE_NESTING_LEVEL_OVER_LIMIT</code> and <code>JSON_NESTING_LEVEL_OVER_LIMIT</code>. Filter on those before declaring a data lake of large compressed exports clean. Check current quota values in the Macie documentation rather than memorizing them, since they differ per format.</p>



<h2 class="wp-block-heading">Automated discovery or a discovery job?</h2>



<p class="wp-block-paragraph"><strong>Automated sensitive data discovery</strong> evaluates your inventory daily and samples representative objects across as many buckets as it can, grouping by metadata like prefix, extension and last-modified date. It is breadth-first and it does not re-analyze unchanged objects. Its job is to tell you which buckets deserve attention. It is not designed to prove a bucket is clean, and reading it that way is the root of most Macie misunderstandings.</p>



<p class="wp-block-paragraph"><strong>Sensitive data discovery jobs</strong> are the depth tool. You choose the buckets, scope by prefix, extension, size or object tag, choose the identifiers, and run once or on a schedule. This is what you point at the bucket automated discovery just flagged.</p>



<pre class="wp-block-code"><code># One-time job over a single prefix, recommended identifiers, 30% sampling
aws macie2 create-classification-job 
  --job-type ONE_TIME 
  --name "exports-prefix-review" 
  --managed-data-identifier-selector RECOMMENDED 
  --sampling-percentage 30 
  --s3-job-definition '{
    "bucketDefinitions": [
      {"accountId": "111122223333", "buckets": ["your-bucket-name"]}
    ],
    "scoping": {
      "includes": {
        "and": [
          {"simpleScopeTerm": {
             "comparator": "STARTS_WITH",
             "key": "OBJECT_KEY",
             "values": ["exports/"]
          }}
        ]
      }
    }
  }'</code></pre>



<p class="wp-block-paragraph">One detail about <code>--sampling-percentage</code> that trips people up: it selects a random percentage of eligible <em>objects</em> and then analyzes each selected object completely. It does not read a fraction of each file. For a bucket where every object comes from one pipeline, 20 or 30 percent tells you what you need. For heterogeneous uploads, sampling is a coin flip and you want the full pass.</p>



<h2 class="wp-block-heading">What actually drives the bill</h2>



<p class="wp-block-paragraph">Macie bills on three dimensions, and the one that surprises people is not the one they budget for.</p>



<ul class="wp-block-list">
<li><strong>Bucket evaluation.</strong> Per general purpose bucket monitored, prorated daily, up to the account ceiling. Predictable, driven by bucket count rather than data volume.</li>



<li><strong>Object monitoring.</strong> Per object tracked while automated discovery is enabled. This is inventory bookkeeping, not content reading, and it runs whether or not anything gets scanned that day. On an account with a log archive holding hundreds of millions of tiny objects, it can dwarf the other two.</li>



<li><strong>Data inspected.</strong> Per GB actually analyzed, covering both automated discovery and jobs.</li>
</ul>



<p class="wp-block-paragraph">So excluding buckets from automated discovery is a genuine cost lever, not just noise reduction. Log archives, CloudTrail destinations, build artifact stores and backup targets are usually safe exclusions. Exclude for the right reason though: a backup bucket holding database dumps of your customer table is exactly what you want scanned, however boring its name sounds.</p>



<p class="wp-block-paragraph">The three dimensions appear as distinct usage types in the Cost and Usage Report, so they split cleanly. A FinOps platform like Vantage or CloudZero, or just a Cost Explorer view grouped by usage type, tells you within a day whether object monitoring or inspection is driving the number. That changes the fix entirely: object monitoring is solved by scoping, inspection by sampling and scheduling.</p>



<h2 class="wp-block-heading">Getting findings somewhere a human sees them</h2>



<p class="wp-block-paragraph">Findings that only exist in the Macie console get read during onboarding week and never again. Two things are worth wiring up on day one.</p>



<p class="wp-block-paragraph"><strong>Publish findings outward.</strong> Macie emits findings to EventBridge and integrates with Security Hub. EventBridge earns its keep because you can filter on severity and finding type and route only what matters, whether that&#8217;s a Slack channel, a ticket, or a Lambda that tags the bucket. If you already centralize alerts in Grafana Cloud or similar, that path keeps sensitive data findings next to everything else on call sees.</p>



<p class="wp-block-paragraph"><strong>Configure the discovery results repository.</strong> Findings tell you what was found and where. The detailed discovery results, including analysis logs for objects where nothing was found, need an S3 bucket and a KMS key configured to persist. Nothing is retained long term until you set this up, and &#8220;we scanned it and found nothing&#8221; is exactly the record an auditor asks for.</p>



<p class="wp-block-paragraph">One operational note on the reveal feature, which retrieves sample occurrences so a human can confirm a finding is real. It needs a customer managed KMS key and it&#8217;s genuinely useful for triage. It also means someone is now looking at live customer PII, which belongs in your access model and audit trail. Same goes if an engineer downloads an object to investigate locally: that copy is unmanaged PII on a laptop and needs a secure deletion step, whether that&#8217;s a wipe tool from a vendor like O&amp;O Software or a documented and enforced process. Findings triage has a habit of creating the exposure it was meant to prevent.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting a bucket that reports nothing</h2>



<p class="wp-block-paragraph">When Amazon Macie PII detection reports nothing for a bucket you have doubts about, work down this in order. Each step rules out one gate, and stopping at the first hit saves you the rest.</p>



<ol class="wp-block-list">
<li><strong>Check the sensitivity label.</strong> <em>Not yet analyzed</em> at score 50 means nothing was analyzed. Do not read that as clean.</li>



<li><strong>Check the coverage page.</strong> Macie names the issue directly: access denied, invalid encryption, invalid KMS key, permission denied, or unclassifiable.</li>



<li><strong>Compare classifiable and unclassifiable counts</strong> with the <code>describe-buckets</code> query above.</li>



<li><strong>List a few object keys.</strong> Run <code>aws s3 ls</code> against a prefix and look for extensions. Extensionless keys explain the whole thing.</li>



<li><strong>Check the storage class.</strong> A lifecycle rule may have moved everything somewhere Macie doesn&#8217;t read.</li>



<li><strong>Check encryption.</strong> SSE-C is unfixable without re-encryption. A customer managed KMS key is fixable with a key policy change.</li>



<li><strong>Check object samples</strong> with <code>list-resource-profile-artifacts</code> to see exactly what was selected and skipped.</li>



<li><strong>Only now question the identifiers.</strong> Run a targeted job with a custom identifier for the data you expect and see whether it fires.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a low sensitivity score as evidence of no PII. It reflects what was found <em>and</em> how much was analyzed, so low coverage produces a low score.</li>



<li>Enabling Macie in one region and assuming account-wide coverage. Settings and results are per region, every time.</li>



<li>Running default identifiers in a business operating outside the jurisdictions the recommended set covers.</li>



<li>Building custom identifiers from regex alone, then abandoning Macie over the false positive volume.</li>



<li>Skipping the discovery results repository, then having no evidence trail when someone asks what was scanned and when.</li>



<li>Excluding buckets by name pattern without checking contents. Backup and export buckets are frequently the highest-risk ones you own.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Measure coverage before findings. A coverage report is the first deliverable of a Macie rollout, not an afterthought.</li>



<li>Fix extensions at the producer rather than building a rename pipeline downstream.</li>



<li>Run the KMS permission analyzer script once per account, then again whenever a new customer managed key appears.</li>



<li>Use automated discovery for breadth and targeted jobs for depth, letting the first choose targets for the second.</li>



<li>Test every custom identifier against both a positive and a negative sample before it reaches a job.</li>



<li>Curate allow lists early. The cost of a noisy first month is a team that stops reading findings.</li>



<li>Re-check coverage after any change to bucket policies, KMS keys, lifecycle rules or ingestion pipelines. All four silently remove data from scope.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Macie scan anything other than S3?</h3>



<p class="wp-block-paragraph">No. Macie analyzes objects in S3 general purpose buckets only. The documented workaround is bringing data to it: export RDS or Aurora snapshots to S3 in Parquet, or export a DynamoDB table to S3, then run a discovery job against the export. That works, but the export is now a second copy of your sensitive data with its own encryption and deletion requirements.</p>



<h3 class="wp-block-heading">Does Macie scan new objects as they land?</h3>



<p class="wp-block-paragraph">Not per object. Automated discovery evaluates inventory on a daily cycle and prioritizes objects that are new or recently changed, so fresh data moves up the queue rather than triggering an immediate scan. If you need scanning tied to arrival, drive a discovery job from S3 event notifications through EventBridge and accept that you own that orchestration.</p>



<h3 class="wp-block-heading">Why does a bucket I know has PII show a low sensitivity score?</h3>



<p class="wp-block-paragraph">Almost always coverage rather than detection. Check whether the objects are classifiable, whether Macie can decrypt them, and whether the label reads <em>Not yet analyzed</em>. If Macie genuinely read them and found nothing, the next suspect is the identifier set, and a targeted job with a custom identifier will tell you in one run.</p>



<h3 class="wp-block-heading">Do I need both automated discovery and discovery jobs?</h3>



<p class="wp-block-paragraph">For most teams, yes. Automated discovery gives you the map at predictable cost. Jobs give you proof for a specific bucket at a specific time, which is what compliance evidence actually requires. Running only jobs means you never discover the bucket nobody told you about.</p>



<h3 class="wp-block-heading">How do I cut false positives without missing real PII?</h3>



<p class="wp-block-paragraph">In order of preference: add keywords and a proximity rule so matches need context; use allow lists for specific known-benign values like your published contact details and test fixtures; and only then remove managed identifiers that are consistently wrong for your data. Removing identifiers is the bluntest option and the easiest to regret, so leave it last.</p>



<h3 class="wp-block-heading">Does Macie work across accounts in an organization?</h3>



<p class="wp-block-paragraph">Yes, through AWS Organizations with a delegated administrator. Design around this: the administrator&#8217;s automated discovery settings apply to member accounts, members can&#8217;t change them, and members see coverage and sensitivity data for their own buckets but not the sensitive data findings themselves. That shapes who can actually remediate what.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Amazon Macie PII detection is only as good as its coverage, and coverage fails silently by design. An object with the wrong extension, the wrong storage class, or an unreachable encryption key produces no finding, and no finding renders identically to a clean result. Before you hand a Macie report to anyone who will make a decision from it, put the coverage numbers next to it: how many objects were classifiable, how many were analyzed, how many were skipped and why.</p>



<p class="wp-block-paragraph">Do that once and the rest of the service becomes straightforward. Skip it and you&#8217;re maintaining an expensive dashboard reporting on whichever subset of your data happened to be readable.</p>



<h2 class="wp-block-heading">Need help getting Macie to actually see your data?</h2>



<p class="wp-block-paragraph">Most of the work in a Macie rollout isn&#8217;t turning it on. It&#8217;s the unglamorous part: proving what got scanned, fixing the reasons things didn&#8217;t, and making the output land somewhere a human acts on it. That&#8217;s the part I help with.</p>



<ul class="wp-block-list">
<li>Coverage audits across your S3 estate, with per-bucket classifiable, analyzed and skipped counts and the specific cause of each gap</li>



<li>Fixing those causes: KMS key policies, bucket policies blocking the service-linked role, lifecycle rules archiving data out of scope, pipelines writing extensionless objects</li>



<li>Custom data identifiers and allow lists tuned against your real data and tested on positive and negative samples first</li>



<li>Cost shaping: working out whether object monitoring or data inspection drives your bill, and scoping discovery so the number is defensible</li>



<li>Findings pipelines through EventBridge or Security Hub into the alerting and ticketing you already run, with severity filtering that keeps signal above noise</li>



<li>Multi-account setup under AWS Organizations, including the delegated administrator model and who can see and remediate what</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me the output of the <code>describe-buckets</code> query above, or a screenshot of your coverage page, and I&#8217;ll tell you what&#8217;s actually being scanned.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Cognito MFA for Healthcare Apps: The Settings That Look Compliant but Aren&#8217;t</title>
		<link>https://john-nessime.com/blog/cloud-computing/cognito-mfa-healthcare-apps/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 30 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Account Recovery]]></category>
		<category><![CDATA[Adaptive Authentication]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Artifact]]></category>
		<category><![CDATA[Business Associate Agreement]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[ePHI]]></category>
		<category><![CDATA[Healthcare Cloud]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[Identity Propagation]]></category>
		<category><![CDATA[MFA]]></category>
		<category><![CDATA[OIDC]]></category>
		<category><![CDATA[Passkeys]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[Session Management]]></category>
		<category><![CDATA[TOTP]]></category>
		<category><![CDATA[WebAuthn]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=348</guid>

					<description><![CDATA[<p>A user pool set to "MFA: Required" can still let clinicians in on a password alone. This is a practical guide to Cognito MFA for healthcare apps: why federated users bypass enforcement, how the account recovery collision locks people out, why required MFA and adaptive authentication cannot coexist, and how to make session lifetimes match what your documentation claims.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cognito-mfa-healthcare-apps/">Cognito MFA for Healthcare Apps: The Settings That Look Compliant but Aren&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The finding came back from a penetration test on a patient portal: the tester signed in as a clinical user with nothing but a username and password. No second factor. No prompt. Straight into a screen with appointment history on it.</p>



<p class="wp-block-paragraph">The user pool was set to <code>MFA: Required</code>. Somebody had screenshotted that console page for the security questionnaire. The screenshot was accurate. It just didn&#8217;t mean what everyone assumed it meant.</p>



<p class="wp-block-paragraph">That gap is the whole problem with Cognito MFA for healthcare apps. The service gives you a setting that reads like a guarantee, and then quietly carves out entire classes of users, sessions, and sign-in paths that the setting never touches. Nothing errors. Nothing logs a warning. Your compliance evidence looks fine right up until someone actually tests it.</p>



<p class="wp-block-paragraph">This post covers what the MFA switch in Amazon Cognito actually controls, the five failure families that show up in real healthcare deployments, how to wire a user pool so the enforcement matches the claim, and what to check when a clinician calls the help desk because they can&#8217;t get in.</p>



<h2 class="wp-block-heading">What HIPAA actually requires from MFA right now</h2>



<p class="wp-block-paragraph">Worth getting this straight before touching any configuration, because a lot of vendor content is confidently wrong about it.</p>



<p class="wp-block-paragraph">The HIPAA Security Rule as it stands today does not name multi-factor authentication. It requires person or entity authentication, and it leaves the mechanism to your risk analysis. MFA is how most people satisfy that in practice, but the rule does not spell it out.</p>



<p class="wp-block-paragraph">What changes the picture is the Notice of Proposed Rulemaking that HHS published in the Federal Register on January 6, 2025. It would make MFA an explicit requirement for access to systems handling ePHI, with narrow exceptions, and it would remove the &#8220;addressable&#8221; escape hatch that currently lets organizations document a control as infeasible. This one matters enough to name the date, because timelines have moved: the comment period closed in March 2025, the rule is still proposed, and the Unified Agenda has pushed anticipated final action out to 2027. That estimate is not binding and has already slipped once.</p>



<p class="wp-block-paragraph">Practical read: build for the proposed rule now, but don&#8217;t tell a client the mandate is already law. It isn&#8217;t, and auditors know the difference.</p>



<p class="wp-block-paragraph">On the AWS side, Amazon Cognito is on the HIPAA Eligible Services list and is covered by the AWS Business Associate Agreement. Eligibility is a starting line, not a finish line. You still have to accept the BAA in each AWS account that touches PHI, and you still have to configure the service correctly. Nothing about creating a user pool makes it compliant.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: half your users never see the MFA prompt</h2>



<p class="wp-block-paragraph">This is the one that produced the pen test finding, and it&#8217;s the most common serious gap I run into.</p>



<p class="wp-block-paragraph">Cognito user pool MFA applies to <em>local</em> users. Those are the accounts that live in the user pool directory and authenticate against it. For <em>federated</em> users, the ones arriving through SAML or OIDC from a hospital identity provider, Okta, Entra ID, or an in-house ADFS, Cognito hands the entire authentication decision to the upstream IdP. It does not add a second factor. It cannot. It never sees the password.</p>



<p class="wp-block-paragraph">So a portal with patients as local users and staff federated from a hospital directory has exactly the split you&#8217;d fear: patients get MFA, clinicians don&#8217;t, unless the hospital IdP is enforcing it independently. And the hospital IdP is usually somebody else&#8217;s system, run by a team you don&#8217;t have a Slack channel with.</p>



<p class="wp-block-paragraph">There&#8217;s a second version of this. The app client, not the user pool, controls which authentication flows are permitted. If a legacy mobile client is still allowed to use a flow you thought you&#8217;d retired, that path stays open regardless of what the pool-level MFA setting says.</p>



<p class="wp-block-paragraph">What to do about it:</p>



<ul class="wp-block-list"><li>Inventory every app client attached to the pool. Not the ones you remember creating. Run <code>list-user-pool-clients</code> and account for all of them.</li><li>For each federated IdP, get written confirmation of what the upstream enforces. A trust relationship is not a control. Put the answer in your risk analysis where an auditor will find it.</li><li>Treat federation as a boundary you inherit risk across, and say so out loud in design reviews. This is the single most useful sentence you can contribute to a healthcare identity project.</li></ul>



<h2 class="wp-block-heading">Failure family two: the recovery attribute collision</h2>



<p class="wp-block-paragraph">This one is genuinely subtle, and it doesn&#8217;t bite during development. It bites during rollout, at scale, to the people least able to tolerate it.</p>



<p class="wp-block-paragraph">Cognito refuses to let a user&#8217;s MFA delivery method be the same channel as their account recovery method. The logic is sound: if a password reset code and a second factor both land in the same inbox, an attacker with that inbox has everything, and the second factor is decoration.</p>



<p class="wp-block-paragraph">The consequence is where it gets sharp. If MFA is required, a user has only an email address and no phone number, email is your account recovery channel, and they have not registered a TOTP authenticator, then that user <strong>cannot complete sign-in at all</strong>. Cognito returns an error. They are not locked out temporarily. There is no path forward from the login screen.</p>



<p class="wp-block-paragraph">Now think about who that describes in a healthcare deployment. Part-time nursing staff enrolled from an HR export that had work email but no mobile number. Older patients who gave an email at registration and nothing else. Contractors. Locums. It is never a small number, and they all arrive at the help desk in the same week.</p>



<p class="wp-block-paragraph">The same rule cuts the other way on password reset. A user whose preferred MFA is email cannot receive a reset code by email. If they have no phone number, self-service recovery is gone, and someone with administrative credentials has to intervene for every single one.</p>



<p class="wp-block-paragraph">Three ways out, in the order I&#8217;d reach for them:</p>



<ul class="wp-block-list"><li><strong>Make TOTP the default factor.</strong> An authenticator app doesn&#8217;t consume either the email or phone attribute, so the collision never occurs. This is the one I reach for first in staff-facing apps.</li><li><strong>Require both email and phone_number as sign-up attributes.</strong> When a user has both, Cognito automatically routes reset codes to whichever channel isn&#8217;t their MFA factor. Clean, but it forces a phone number on people who may not want to give one.</li><li><strong>Configure a recovery fallback.</strong> Set account recovery to prefer email but fall back to SMS, so users who aren&#8217;t eligible for email recovery still have a route.</li></ul>



<p class="wp-block-paragraph">Whatever you choose, run a query against your existing user records for accounts missing a phone number before you flip MFA to required. Finding those users in a report is cheap. Finding them through a help desk queue is not.</p>



<h2 class="wp-block-heading">Failure family three: required MFA and adaptive authentication are mutually exclusive</h2>



<p class="wp-block-paragraph">This trips up almost every security-conscious design I review, because both features are things a healthcare client will ask for by name.</p>



<p class="wp-block-paragraph">Cognito&#8217;s threat protection includes adaptive authentication: risk scoring on each sign-in, with automated responses like requiring MFA or blocking outright. To use those automated responses in full-function mode, <strong>MFA must be set to optional in the user pool</strong>. That is a documented constraint, not a bug. The mechanism only makes sense if there&#8217;s a state where MFA is not already being demanded.</p>



<p class="wp-block-paragraph">So you cannot check both boxes. Either MFA is required for everyone and adaptive responses have nothing to escalate to, or MFA is optional and risk scoring can escalate, which means low-risk sign-ins may proceed on a password alone.</p>



<p class="wp-block-paragraph">For anything touching ePHI, I&#8217;d take required MFA and give up the adaptive escalation. A control that always applies beats a control that usually applies, and &#8220;usually&#8221; is a hard thing to defend in an audit.</p>



<p class="wp-block-paragraph">You don&#8217;t have to give up the visibility, though. Threat protection runs in audit-only mode alongside required MFA. You still get risk signals, device fingerprinting, and compromised credential detection surfaced in CloudWatch and in the user event logs, without the enforcement conflict. That&#8217;s a genuinely good trade: enforcement from the required setting, detection from threat protection.</p>



<p class="wp-block-paragraph">Two things to know before you budget for it. Threat protection lives in the Plus feature tier, and audit-only mode is billed the same as full-function. There&#8217;s no free observation period. Export those events somewhere you&#8217;ll actually look at them, whether that&#8217;s a CloudWatch Logs destination feeding your SIEM, or a platform like Datadog or Grafana Cloud if that&#8217;s where your on-call already lives.</p>



<h2 class="wp-block-heading">Failure family four: the session outlives the shift</h2>



<p class="wp-block-paragraph">MFA protects the moment of sign-in. It does nothing about the eight hours after it. In healthcare that gap is unusually expensive, because the workstation is shared, the browser stays open, and the next person to touch the keyboard is a different human being.</p>



<p class="wp-block-paragraph">Cognito&#8217;s defaults are built for consumer apps. Access and ID tokens last one hour. Refresh tokens last thirty days. Neither default is wrong in general and both are wrong for a clinical workstation.</p>



<p class="wp-block-paragraph">The ranges you have to work with:</p>



<ul class="wp-block-list"><li>Access token and ID token: five minutes to one day, set per app client.</li><li>Refresh token: sixty minutes to ten years, set per app client. It cannot be shorter than the access token lifetime.</li></ul>



<p class="wp-block-paragraph">Now the part that catches people, and the reason I put this failure family on the list. <strong>If you use managed login, Cognito sets a browser session cookie valid for one hour.</strong> Setting your access token to fifteen minutes does not shorten the re-authentication window. During that hour the user&#8217;s session refreshes without presenting credentials again, because the cookie is still good. Teams configure short tokens, write &#8220;15-minute session timeout&#8221; in the security documentation, and the observable behavior is an hour.</p>



<p class="wp-block-paragraph">The second trap is revocation. Revoking a token, or a global sign-out, invalidates refresh tokens. It does not invalidate an access token that has already been issued. That token stays valid until it expires, and any API accepting it will keep accepting it. So the practical blast radius of a compromised session is your access token TTL, not the moment you clicked &#8220;sign out this user.&#8221;</p>



<p class="wp-block-paragraph">That&#8217;s the argument for short access tokens: it is the only lever that shortens the window between &#8220;we revoked access&#8221; and &#8220;access actually stopped.&#8221;</p>



<pre class="wp-block-code"><code># Tighten an app client used by clinical staff.
# Access tokens live 15 minutes; refresh tokens expire at the end of a shift.
aws cognito-idp update-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --access-token-validity 15 
  --id-token-validity 15 
  --refresh-token-validity 9 
  --token-validity-units AccessToken=minutes,IdToken=minutes,RefreshToken=hours 
  --enable-token-revocation</code></pre>



<p class="wp-block-paragraph">Refresh token rotation is worth turning on where your client can support it. Each refresh returns a new refresh token and invalidates the old one, so a stolen refresh token is usable once at most. Two caveats that bite in practice: rotation does not extend the original lifetime, so a nine-hour refresh token still dies nine hours after sign-in no matter how often it rotated, and app clients with rotation enabled cannot use the older <code>REFRESH_TOKEN_AUTH</code> flow. That second one has broken more than one mobile release. Rotation requires the Essentials tier or higher.</p>



<p class="wp-block-paragraph">None of this replaces workstation-level controls. Screen lock policy, OS session timeouts, and endpoint hardening tooling like the utilities from O&amp;O Software do work your identity layer cannot do from the outside. If staff reach the admin console remotely, a managed VPN such as NordLayer or Surfshark reduces network exposure, but be honest in your documentation about what that is. It is not an authentication control, and treating it as one is exactly the reasoning auditors push back on.</p>



<h2 class="wp-block-heading">Failure family five: the factor you picked is phishable</h2>



<p class="wp-block-paragraph">Cognito offers SMS, email, and TOTP as second factors, plus passkeys as a first factor under specific conditions. They are not equivalent, and the console presents them as though they were.</p>



<p class="wp-block-paragraph"><strong>SMS</strong> is the weakest. SIM swap and SS7 interception are real, and an SMS code is trivially relayed through a phishing proxy. It has one genuine advantage worth naming: enrollment friction is near zero, and for elderly or low-technology-comfort patient populations that is not a small thing. An SMS factor that gets used beats a TOTP factor that gets abandoned. Just don&#8217;t put clinical staff behind it. There&#8217;s also an operational trap: if your account is in the Amazon SNS SMS sandbox in the relevant region, you must verify each destination number before delivery works, which turns a smooth rollout into a stalled one.</p>



<p class="wp-block-paragraph"><strong>Email OTP</strong> is roughly SMS-equivalent in phishing resistance and worse if the mailbox is on the same laptop as the session you&#8217;re protecting. It needs the Essentials tier or higher and a configured Amazon SES sending identity. Useful as a fallback, weak as a primary.</p>



<p class="wp-block-paragraph"><strong>TOTP</strong> is the sensible default for staff. It&#8217;s phishable, the code can be relayed in real time by an attacker-in-the-middle, but it&#8217;s immune to SIM swap and it doesn&#8217;t collide with your account recovery channel. Any standard authenticator works. If your organization already uses a password manager with built-in TOTP, such as 1Password or Bitwarden, enrollment gets meaningfully easier, at the cost of both factors living in one vault. That&#8217;s a trade-off to make deliberately rather than by accident.</p>



<p class="wp-block-paragraph"><strong>Passkeys</strong> are the only phishing-resistant option here, and their integration has a shape you need to understand. A passkey cannot be a second factor on top of a password in Cognito. It can satisfy the MFA requirement on its own, as a first factor, when you set <code>FactorConfiguration</code> to <code>MULTI_FACTOR_WITH_USER_VERIFICATION</code> in the pool&#8217;s WebAuthn configuration. The user verification step, biometric or device PIN, is what makes it count as multi-factor.</p>



<p class="wp-block-paragraph">That&#8217;s the path I&#8217;d push for privileged accounts: administrators, anyone with bulk record export, anyone who can change the pool configuration itself. Platform authenticators cover most staff; hardware keys like YubiKey are worth the cost for the handful of accounts whose compromise ends the conversation.</p>



<p class="wp-block-paragraph">One hard constraint to plan around: you cannot set MFA to required in a pool that allows email or SMS one-time passwords as a <em>first</em> factor. Passwordless OTP sign-in and required MFA are incompatible. Passkeys with user verification are the documented exception.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring up Cognito MFA for healthcare apps</h2>



<p class="wp-block-paragraph">An order of operations that avoids locking people out. Do this in a non-production pool first, with synthetic users only.</p>



<ol class="wp-block-list"><li><strong>Accept the BAA</strong> in the AWS account through AWS Artifact, before any real user data lands in the pool. The BAA is per-account and is not inherited by new accounts.</li><li><strong>Audit your user records</strong> for accounts missing <code>phone_number</code> or <code>email</code>. This is the population that will fail after you enable required MFA.</li><li><strong>Decide the recovery configuration</strong> and set it explicitly, with a fallback channel, so the MFA factor and the recovery channel can never be the same for any user.</li><li><strong>Enable MFA methods with the pool set to optional first.</strong> Let users enroll voluntarily. Track enrollment coverage as a number you report weekly.</li><li><strong>Flip to required only when coverage is high</strong> and the stragglers have a support path. Managed login will prompt unenrolled users to set up a factor at next sign-in, but only when MFA is required.</li><li><strong>Tighten token lifetimes per app client</strong>, then verify the observable behavior rather than the configuration. If you use managed login, remember the one-hour cookie.</li><li><strong>Turn on threat protection in audit-only mode</strong> and stream the events out to wherever you review security signal.</li></ol>



<p class="wp-block-paragraph">The core enforcement setting, once you&#8217;re ready:</p>



<pre class="wp-block-code"><code># Require MFA pool-wide, TOTP as the available method.
aws cognito-idp set-user-pool-mfa-config 
  --user-pool-id us-east-1_EXAMPLE 
  --mfa-configuration ON 
  --software-token-mfa-configuration Enabled=true

# Set a specific user's preferred factor administratively.
aws cognito-idp admin-set-user-mfa-preference 
  --user-pool-id us-east-1_EXAMPLE 
  --username clinician@example.org 
  --software-token-mfa-settings Enabled=true,PreferredMfa=true</code></pre>



<p class="wp-block-paragraph">Note that <code>admin-set-user-mfa-preference</code> does not reset an existing TOTP registration. To issue a new authenticator secret for a user who lost their phone, you go through <code>AssociateSoftwareToken</code> and <code>VerifySoftwareToken</code>. Build that runbook before you need it, because you will need it, and improvising an identity-proofing procedure at 2pm on a Tuesday is how bad help desk habits start.</p>



<p class="wp-block-paragraph">A staging pool holding only synthetic patients doesn&#8217;t need to live inside the BAA account, and there&#8217;s no reason a test environment should carry production&#8217;s cost structure. A plain VPS from InterServer or Contabo is fine for the app tier in front of a throwaway pool, provided you are certain no real PHI can reach it. That certainty is the whole condition, and it&#8217;s worth writing down rather than assuming.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">A user gets an error at sign-in and never reaches an MFA prompt</h3>



<p class="wp-block-paragraph">Almost always the recovery collision. Check whether the account has both <code>email</code> and <code>phone_number</code>, and compare the populated attribute against the pool&#8217;s account recovery setting. If the only available attribute is also the recovery channel, and there&#8217;s no TOTP registration, the user has no eligible factor. Register TOTP for them or add the missing attribute.</p>



<h3 class="wp-block-heading">Sign-in succeeds without MFA even though the pool says required</h3>



<p class="wp-block-paragraph">Work backward from the user, not from the pool. Is this a federated user? Then Cognito is not in the authentication path and pool MFA does not apply. Is it their first sign-in after account creation? Cognito treats the initial verification message as the second factor on that first sign-in and prompts for factor registration afterward. Is a different app client involved with different permitted flows?</p>



<h3 class="wp-block-heading">Password reset stopped working after enabling MFA</h3>



<p class="wp-block-paragraph">Expected behavior, not a regression. The reset code cannot go to the same channel as the user&#8217;s MFA factor. Users with only one contact attribute lose self-service recovery. Add the second attribute or configure a fallback recovery mechanism.</p>



<h3 class="wp-block-heading">SMS codes never arrive</h3>



<p class="wp-block-paragraph">Check the Amazon SNS SMS sandbox status in the region holding your SNS resources. In the sandbox, destination numbers must be verified in SNS before delivery. Also confirm the IAM role Cognito assumes for SNS is present and correctly scoped.</p>



<h3 class="wp-block-heading">Repeated MFA failures lock a user out</h3>



<p class="wp-block-paragraph">After five unsuccessful MFA code attempts, Cognito starts an exponential timeout lockout. It resolves on its own. Most real cases are clock drift on the user&#8217;s phone breaking TOTP validation, so check that before assuming an attack.</p>



<h3 class="wp-block-heading">Tokens keep working after you revoked them</h3>



<p class="wp-block-paragraph">Correct and by design. Revocation kills refresh tokens; already-issued access tokens survive to their expiry. If that window is unacceptable for your risk profile, shorten the access token TTL. There is no other lever short of checking revocation on every API call.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Treating the pool-level MFA setting as coverage for federated users. It is not, and this is the gap most likely to survive to production.</li><li>Storing PHI in Cognito custom attributes. Diagnosis codes, MRNs, and care team assignments do not belong in a directory record that gets copied into ID token claims, which then land in browser storage, proxy logs, and error reports. Keep the identity layer to identity.</li><li>Enabling required MFA in one deploy without an enrollment ramp. You will manufacture a support incident.</li><li>Assuming a short access token gives a short session when managed login is in play. The one-hour cookie governs re-authentication.</li><li>Believing revocation is immediate.</li><li>Leaving threat protection off because required MFA is on. Audit mode and required MFA coexist, and the detection signal is separately valuable.</li><li>Enabling refresh token rotation without checking whether your client uses <code>REFRESH_TOKEN_AUTH</code>. That combination is not permitted and it breaks at runtime.</li></ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list"><li>Separate app clients for patient-facing and staff-facing applications, with independently tuned token lifetimes. Staff sessions should not inherit consumer defaults.</li><li>TOTP as the baseline factor for anyone with access to more than their own record.</li><li>Passkeys with user verification for administrators and bulk-export roles.</li><li>Both <code>email</code> and <code>phone_number</code> collected at enrollment where the population allows it, purely to keep recovery working.</li><li>Access token TTL short enough that revocation means something. Fifteen minutes is a reasonable starting point for clinical apps.</li><li>Threat protection in audit mode, with events exported and actually reviewed.</li><li>Manage the pool in Terraform or OpenTofu so MFA settings, recovery configuration, and token lifetimes are reviewable in a diff instead of discovered in the console during an audit.</li><li>Test the MFA path in CI against a non-production pool. Configuration drift in an identity provider is silent by nature.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is Amazon Cognito HIPAA compliant?</h3>



<p class="wp-block-paragraph">No service is HIPAA compliant by itself. Cognito is HIPAA <em>eligible</em> and covered under the AWS Business Associate Agreement, which means you may process PHI with it once you have accepted the BAA in that account and configured it appropriately. Compliance is a property of your whole system and your documentation, not of the service.</p>



<h3 class="wp-block-heading">Does HIPAA require MFA today?</h3>



<p class="wp-block-paragraph">Not explicitly. The current Security Rule requires person or entity authentication and leaves the method to your risk analysis. The proposed 2025 update would make MFA an express requirement, but it remains proposed and the timeline for final action has moved more than once.</p>



<h3 class="wp-block-heading">Can I use required MFA and adaptive authentication together?</h3>



<p class="wp-block-paragraph">Not in full-function mode. Adaptive authentication&#8217;s automated responses require MFA to be optional in the user pool. You can run threat protection in audit-only mode alongside required MFA, which keeps the detection signal without the conflict.</p>



<h3 class="wp-block-heading">Which MFA factor should healthcare apps use?</h3>



<p class="wp-block-paragraph">TOTP as the baseline for staff, passkeys with user verification for privileged accounts. SMS is defensible for patient populations where enrollment friction genuinely determines whether MFA gets used at all, but it should not protect clinical access.</p>



<h3 class="wp-block-heading">Does Cognito MFA apply to users signing in through our hospital SSO?</h3>



<p class="wp-block-paragraph">No. For federated users arriving via SAML or OIDC, Cognito delegates authentication entirely to the upstream identity provider and does not add a second factor. Enforcement has to happen at the IdP, and you should get that in writing.</p>



<h3 class="wp-block-heading">How short should session timeouts be for a clinical application?</h3>



<p class="wp-block-paragraph">Set access tokens short, commonly around fifteen minutes, and refresh tokens to roughly a shift length rather than the thirty-day default. Then verify what users actually experience, because managed login&#8217;s one-hour browser cookie can keep a session alive past your token settings.</p>



<h3 class="wp-block-heading">Can I store patient identifiers in Cognito user attributes?</h3>



<p class="wp-block-paragraph">You can, and I would avoid it. Custom attributes flow into ID token claims, which end up in browser storage, application logs, and error tracking. Keep an opaque internal identifier in the token and resolve clinical data from your application database behind its own authorization check.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you remember one thing about Cognito MFA for healthcare apps, make it this: the MFA setting is a claim about local users at the moment of sign-in, and nothing else. It says nothing about your federated clinicians, nothing about the session that continues for the next eight hours, and nothing about whether the factor you chose survives a phishing proxy.</p>



<p class="wp-block-paragraph">Verify enforcement by observing behavior, per user population, per app client. Not by screenshotting the console page. That screenshot is what got flagged in the pen test at the top of this post, and it was completely accurate.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with Cognito and healthcare authentication?</h2>



<p class="wp-block-paragraph">I work with teams building and hardening authentication on AWS, particularly where a compliance obligation makes &#8220;it works&#8221; insufficient. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing an existing user pool for the gaps in this post: federated coverage, recovery collisions, token lifetimes, app client sprawl</li><li>Designing an MFA rollout that won&#8217;t generate a help desk queue, including the pre-flight query against your user records and the enrollment ramp</li><li>Implementing passkeys with user verification for privileged accounts, alongside TOTP for general staff</li><li>Moving user pool configuration into Terraform or OpenTofu so identity settings are reviewable and drift is visible</li><li>Wiring threat protection events into your existing monitoring and building the alerts that are worth waking someone for</li><li>Writing the technical evidence that maps your configuration to your risk analysis, in language an auditor accepts</li></ul>



<p class="wp-block-paragraph">If you have a user pool that isn&#8217;t behaving the way the console suggests it should, send me the output of <code>describe-user-pool</code> and <code>describe-user-pool-client</code> with the identifiers stripped, and I&#8217;ll tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cognito-mfa-healthcare-apps/">Cognito MFA for Healthcare Apps: The Settings That Look Compliant but Aren&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Choosing an AWS Region for a MENA Client: Latency Is the Last Question</title>
		<link>https://john-nessime.com/blog/cloud-computing/aws-region-for-mena/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 26 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Amazon CloudFront]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Control Tower]]></category>
		<category><![CDATA[AWS Regions]]></category>
		<category><![CDATA[Cross-Border Data Transfer]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Data Transfer Costs]]></category>
		<category><![CDATA[Disaster Recovery]]></category>
		<category><![CDATA[MENA]]></category>
		<category><![CDATA[Multi-AZ]]></category>
		<category><![CDATA[Parameter Store]]></category>
		<category><![CDATA[PDPL]]></category>
		<category><![CDATA[Region Selection]]></category>
		<category><![CDATA[Service Control Policies]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=548</guid>

					<description><![CDATA[<p>Latency is the constraint everyone reaches for first when choosing an AWS region for MENA clients, and it is almost always the one that matters least. Residency law eliminates candidates, service availability eliminates more, and latency only picks between whatever survives. A practical walkthrough of that filter, with the commands to check service availability and region opt-in status programmatically, the guardrail that keeps the decision true, and the things that break in the first few weeks after you commit.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-region-for-mena/">Choosing an AWS Region for a MENA Client: Latency Is the Last Question</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone says it on the kickoff call, usually in the first ten minutes. &#8220;Bahrain, right? It&#8217;s the closest one.&#8221; Everybody nods, the Terraform gets written, and six weeks later the client&#8217;s lawyer asks a question nobody prepared for: where exactly is the personal data of our Egyptian users sitting right now?</p>



<p class="wp-block-paragraph">That is the moment the region choice stops being a performance question and turns into an expensive one. Choosing an AWS region for MENA workloads looks like a latency problem and almost never is. Latency is the constraint you can measure in an afternoon and fix with a CDN. Residency is the constraint that can force you to rebuild.</p>



<p class="wp-block-paragraph">This post walks through the order I work in when picking an AWS region for MENA clients: which constraints bind hard, which ones bend, how to check service availability programmatically instead of trusting a table someone screenshotted, and what tends to break after the decision is made.</p>



<h2 class="wp-block-heading">Work the constraints in the order they bind</h2>



<p class="wp-block-paragraph">Region selection is a filter, not a scoring exercise. You are not hunting for the best average across four categories. You are eliminating regions that cannot work, in the order that matters, then picking from whatever survives.</p>



<p class="wp-block-paragraph">The order I use:</p>



<ol class="wp-block-list">
<li><strong>Residency and regulatory obligations.</strong> Not engineerable. Either the law permits the data to sit there or it does not.</li>



<li><strong>Service availability.</strong> Not engineerable cheaply. If the region lacks something your design needs, you change the design or change the region.</li>



<li><strong>Resilience and blast radius.</strong> Partly engineerable, bounded by which regions are near enough to pair with.</li>



<li><strong>Latency.</strong> Very engineerable. Edge caching, connection reuse and smaller payloads beat a few hundred kilometres of fibre most of the time.</li>
</ol>



<p class="wp-block-paragraph">Most guides run this list backwards, because latency is the fun part and the one with a nice map. Running it backwards is how you end up with a beautifully fast architecture that legal makes you move.</p>



<h2 class="wp-block-heading">Constraint one: residency, and the region that does not exist</h2>



<p class="wp-block-paragraph">Here is the thing that catches people who have only worked with EU or US clients. In Europe you can usually satisfy a residency requirement by picking a region in the right country. Across MENA, that option frequently is not on the table, because AWS does not have an infrastructure region in most MENA countries.</p>



<p class="wp-block-paragraph">Check the current region table before you promise anything. The list changes, and the geography column is what a regulator cares about, not the region&#8217;s marketing name. The programmatic version:</p>



<pre class="wp-block-code"><code># List every region your account can see, with opt-in status
aws account list-regions 
  --query 'Regions[*].[RegionName,RegionOptStatus]' 
  --output table

# Confirm the physical country AWS records for a region
aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/regions/me-central-1 
  --region us-east-1 
  --output table</code></pre>



<p class="wp-block-paragraph">That second command returns the metadata AWS publishes for the region, including a geolocation country code. Better source than a blog post, this one included, because it comes from the same data the console uses.</p>



<p class="wp-block-paragraph">One caution on the path itself: the <code>global-infrastructure</code> parameter tree is only queryable from a subset of regions, so pin <code>--region us-east-1</code> on those calls rather than assuming your default profile region will work.</p>



<h3 class="wp-block-heading">Announced is not the same as available</h3>



<p class="wp-block-paragraph">AWS announces regions well ahead of general availability, and those announcements get quoted back at you by sales teams and clients as though the capacity already exists. Treat an announcement as a roadmap item. If the region code does not appear in the table your account can query, you cannot deploy to it, and you certainly cannot write a residency clause around it. That gap matters in the Gulf specifically, where announced-but-not-yet-open regions have been part of the conversation for a while.</p>



<h3 class="wp-block-heading">Residency and localisation are not the same requirement</h3>



<p class="wp-block-paragraph">The word &#8220;residency&#8221; gets used for at least three different obligations, and clients rarely distinguish between them. Before you look at a single region, get the client&#8217;s counsel to tell you which one applies:</p>



<ul class="wp-block-list">
<li><strong>Hard localisation.</strong> The data must sit in-country, full stop. With no region in that country, no AWS region satisfies this and you are into a local provider, on-premises kit, or a hybrid split.</li>



<li><strong>Conditional transfer.</strong> Data may leave if a condition is met: regulator authorisation, an adequacy finding, explicit consent, a specific contractual basis. This is the common case, and it is more paperwork problem than architecture problem.</li>



<li><strong>Sector rules on top.</strong> Finance, health and government workloads usually add a supervisor with its own cloud rules, which may name approved jurisdictions or require notice before you move anything.</li>
</ul>



<p class="wp-block-paragraph">Egypt is the clearest illustration of why this matters to an engineer. Its Personal Data Protection Law and the executive regulations under it set up a regime where moving personal data out of the country generally needs a licence or permit from the supervisory authority, rather than a self-assessment you document internally and move on from. There is no AWS region in Egypt. So for an Egyptian client with personal data in scope, picking a region does not answer the residency question. It restates it: which foreign region will we be authorised to transfer into, and who is filing that application?</p>



<p class="wp-block-paragraph">I am not a lawyer and neither are you. What you can do is refuse to let the region be chosen before that answer exists, because picking first and asking later is how the rebuild happens.</p>



<h3 class="wp-block-heading">Enforce the decision in the account, not in the runbook</h3>



<p class="wp-block-paragraph">Once the approved regions are settled, stop treating them as a convention. A convention survives until someone opens the console in the wrong region at eleven at night.</p>



<p class="wp-block-paragraph">The mechanism is the <code>aws:RequestedRegion</code> condition key, applied either as an IAM policy or, better, as a Service Control Policy across the organisation so it cannot be edited away by an account admin. AWS Control Tower ships a managed version of the same idea as its Region deny control if you are running a landing zone.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyOutsideApprovedRegions",
      "Effect": "Deny",
      "NotAction": [
        "cloudfront:*",
        "iam:*",
        "organizations:*",
        "route53:*",
        "support:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": [
            "me-central-1"
          ]
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The <code>NotAction</code> block is the part people get wrong, and it is worth understanding rather than copying. Several AWS services are global with a single control-plane endpoint that lives in <code>us-east-1</code>. A blanket deny on everything outside your approved region catches those calls too, and locks you out of IAM. Exempting them by action prefix is what keeps the policy usable. Extend the list to match what the client actually runs, and test it in a sandbox account before it goes anywhere near production.</p>



<p class="wp-block-paragraph">One wrinkle worth checking: managed inference services can route requests to other regions within a geography to absorb load. Sensible availability feature, simultaneous residency problem. If generative AI is in scope, confirm the routing behaviour explicitly rather than assuming a request to a regional endpoint stays there.</p>



<h2 class="wp-block-heading">Constraint two: service availability, where the nasty surprises live</h2>



<p class="wp-block-paragraph">Not every AWS service exists in every region, and the gap is widest in the newest and smallest ones. This is the failure mode that bites hardest, because it is invisible during design and obvious during implementation. The diagram gets approved, the Terraform plan runs, and one resource type throws an error that reads like a permissions problem but is really a &#8220;this does not exist here&#8221; problem.</p>



<p class="wp-block-paragraph">Do not check this against the services page by eye. Check it in a script, at design time, against the same source AWS uses:</p>



<pre class="wp-block-code"><code># Is a given service available in a given region?
# An empty result means no.
aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/services/athena/regions/me-central-1 
  --region us-east-1 
  --output text

# Every service AWS records for a region
aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/regions/me-central-1/services 
  --region us-east-1 
  --query 'Parameters[].Name' 
  --output text

# The canonical list of service identifiers to test against
aws ssm get-parameters-by-path 
  --path /aws/service/global-infrastructure/services 
  --region us-east-1 
  --query 'Parameters[].Name | sort(@)' 
  --output text</code></pre>



<p class="wp-block-paragraph">The identifiers in that tree are short API names, not display names, which trips people up. Pull the full list and grep it rather than guessing how a service name is spelled.</p>



<p class="wp-block-paragraph">Two refinements make this useful rather than a party trick:</p>



<ul class="wp-block-list">
<li><strong>Write the check as a matrix, not a lookup.</strong> Put the services your design depends on in a file, loop over the candidate regions, print a grid. You want the gaps visible before any infrastructure code exists.</li>



<li><strong>Presence is not feature parity.</strong> The parameter tree says a service exists in a region. It says nothing about which instance families, engine versions, storage classes or model options are offered there, and newer regions routinely carry a narrower menu. Where the specific variant matters, check that service&#8217;s own regional documentation too.</li>
</ul>



<h3 class="wp-block-heading">Opt-in regions and the quiet failures they cause</h3>



<p class="wp-block-paragraph">Every AWS region introduced in recent years, including all of the Middle East regions, is disabled by default and must be enabled on the account before you can use it. Small piece of admin, disproportionate number of confusing failures.</p>



<p class="wp-block-paragraph">The failure shape is always the same: it works in your account and fails in the client&#8217;s, or works in the management account and fails in a member account created last week. Enablement is per-account and not instantaneous, so any automated account vending process has to handle it.</p>



<pre class="wp-block-code"><code>aws account get-region-opt-status --region-name me-central-1</code></pre>



<p class="wp-block-paragraph">Run that against every account in scope as a pre-flight check, not as a debugging step after a pipeline fails.</p>



<h2 class="wp-block-heading">Constraint three: resilience and concentration risk</h2>



<p class="wp-block-paragraph">Every AWS region carries at least three Availability Zones, and the Middle East regions are three-AZ regions. That is enough for a properly designed multi-AZ deployment, and multi-AZ should be your default for anything the client would notice going down.</p>



<p class="wp-block-paragraph">The subtlety is what you pair the region with. Cross-region DR inside MENA pairs two regions that sit closer together than a European or North American pair would, and that share more exposure to the same events: a cut on a common submarine route, a regulatory action, a physical incident. That is not an argument against a MENA pair. It is an argument for saying plainly what the DR plan protects against, because a nearby second region covers a regional AWS failure and not much wider than that.</p>



<p class="wp-block-paragraph">Two practical points on AZ handling that catch people:</p>



<ul class="wp-block-list">
<li><strong>AZ names are per-account.</strong> The letter suffix on a zone name does not map to the same physical zone across two different accounts. If you are coordinating placement between accounts, use the zone ID rather than the name. <code>aws ec2 describe-availability-zones --region me-central-1</code> returns both.</li>



<li><strong>Capacity is not uniform.</strong> Smaller regions have less headroom in specific instance families, and you can hit an insufficient-capacity error on a launch that would have succeeded in a large region. Design for instance-family flexibility, and if the workload is steady state, talk to the client about a capacity commitment rather than discovering the ceiling during a launch window.</li>
</ul>



<h2 class="wp-block-heading">Constraint four: latency, finally</h2>



<p class="wp-block-paragraph">By the time residency and service availability have done their work you often have one candidate left, and the latency question answers itself. When you do still have a choice, measure properly.</p>



<p class="wp-block-paragraph">Measuring from your own machine tells you about your own connection, which is not what you care about. Regional routing is often counter-intuitive, and traffic between two Gulf cities does not necessarily take the short path. You want a measurement from the network the users are actually on. Options, in rough order of how much I trust them:</p>



<ul class="wp-block-list">
<li><strong>Real user monitoring on an existing property.</strong> If the client already has a site or app with users in that market, instrument it and read the field data. Nothing synthetic beats this.</li>



<li><strong>A small instance in each candidate region, probed from the target country.</strong> Cheap, quick, measures the actual path rather than a claim about it.</li>



<li><strong>A commercial VPN endpoint in the target country</strong> when you have no presence there. NordVPN or Surfshark will sanity-check the relative ordering of two regions, but treat the absolute numbers with suspicion: you are measuring a path through the provider&#8217;s network, not a real user&#8217;s path.</li>



<li><strong>Published latency matrices.</strong> Fine for a first pass. Not evidence.</li>
</ul>



<h3 class="wp-block-heading">Split the problem before you optimise the region</h3>



<p class="wp-block-paragraph">Most of what users experience as slowness is not round-trip time to the origin. It is uncached static assets, chatty APIs, TLS handshakes on cold connections, and payloads bigger than they need to be. A CDN fixes a large share of that whatever region the origin sits in, and the edge network is far denser across MENA than the region footprint is. Amazon CloudFront is the obvious in-stack choice; Cloudflare is worth a look when the client wants DNS, WAF and edge in one place, or when the origin is not on AWS at all.</p>



<p class="wp-block-paragraph">The caveat matters though. Caching at the edge puts copies of responses in edge locations worldwide. For static assets that is fine. For authenticated responses carrying personal data it is a residency question of its own, and the answer is usually to scope caching tightly and leave dynamic personal responses uncached.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What breaks after you have chosen</h2>



<p class="wp-block-paragraph">A short troubleshooting list for the first few weeks, drawn from the things that reliably go wrong.</p>



<ul class="wp-block-list">
<li><strong>Access denied on a service you definitely have permissions for.</strong> Check the region deny SCP first, then check whether the call is hitting a global endpoint you forgot to exempt. The error will not tell you an SCP is responsible.</li>



<li><strong>A certificate that will not attach to your distribution.</strong> Certificates used with CloudFront must be requested in <code>us-east-1</code> regardless of where the origin lives. This looks like a residency violation to a nervous auditor and is not one, since a certificate holds no personal data. Have the explanation ready.</li>



<li><strong>Terraform plans that differ between engineers.</strong> Someone has a different default region in their profile. Pin it in the provider block and stop relying on environment configuration.</li>



<li><strong>An unexpected data transfer line on the bill.</strong> Cross-region traffic is charged, and replication, backups and cross-region reads all generate it quietly. Get cost allocation tags working before the first full month closes so you can attribute it rather than argue about it.</li>



<li><strong>A service that exists but behaves differently.</strong> Quotas start lower in newer regions and some features lag. Check the quota console before assuming your code is wrong.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Choosing the region on the kickoff call, before anyone has asked counsel what the transfer basis is.</li>



<li>Treating proximity as the residency answer. A region in a neighbouring country is still a cross-border transfer.</li>



<li>Copying a region deny policy from a blog without adapting the global-service exemptions to the client&#8217;s real service inventory.</li>



<li>Checking service availability once at the start and never again as the design grows new components.</li>



<li>Forgetting the observability stack. Logs and metrics carry personal data more often than people expect, and a monitoring platform hosted in another jurisdiction is a transfer. Grafana Cloud and the AWS-native options both let you choose where data lands, so choose deliberately.</li>



<li>Assuming this has to be AWS at all. For a low-sensitivity workload with no compliance surface, a VPS from Contabo or InterServer may beat a thin footprint in an expensive region. Say so if it is true.</li>
</ul>



<h2 class="wp-block-heading">How I would pick an AWS region for MENA</h2>



<ol class="wp-block-list">
<li>Get the in-scope data classes in writing. Personal data, health data, financial records and anonymised telemetry carry different rules.</li>



<li>Ask counsel which residency shape applies and whether a sector supervisor sits on top. Eliminate the regions that fail it. If nothing survives, have the hybrid conversation now rather than in month four.</li>



<li>Run the availability matrix against the survivors and eliminate again. If more than one region is left, measure latency from the target country and pick.</li>



<li>Write down the choice and the reasoning, enable the region on every account in scope, and apply the guardrail that same week.</li>
</ol>



<p class="wp-block-paragraph">That last step is the one people skip. A region decision without a written rationale gets relitigated every time a new engineer joins, and a region decision without a guardrail quietly stops being true.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Which AWS region is best for a client in the UAE?</h3>



<p class="wp-block-paragraph">If in-country storage is required, the UAE region is the only AWS option that puts data inside the country. If the requirement is softer, Bahrain is a fair alternative and has been open longer, which usually means broader service coverage. Run the availability matrix against both before committing.</p>



<h3 class="wp-block-heading">Is there an AWS region in Egypt or Saudi Arabia?</h3>



<p class="wp-block-paragraph">Check the region table rather than trusting any article, this one included. AWS has publicly announced intent to build in the area and those announcements circulate widely, but only regions appearing in the table your account can query are deployable. Where there is no region, the real question is which nearby region you are permitted to transfer into.</p>



<h3 class="wp-block-heading">Does choosing a nearby AWS region satisfy MENA data residency law?</h3>



<p class="wp-block-paragraph">Not by itself. Several MENA data protection regimes treat any movement of personal data across national borders as a transfer needing a legal basis, and some require prior authorisation from a regulator. Geographic closeness carries no legal weight. Get the transfer basis confirmed before the region is fixed.</p>



<h3 class="wp-block-heading">How do I check whether a specific AWS service is available in a region?</h3>



<p class="wp-block-paragraph">Query the global infrastructure tree in Systems Manager Parameter Store. An empty result for a service and region pair means it is not offered there. It is the most reliable programmatic source, but it only tells you the service exists, not which features or instance types are available.</p>



<h3 class="wp-block-heading">Why do I get access denied in a Middle East region even as an administrator?</h3>



<p class="wp-block-paragraph">Two likely causes: the region is opt-in and has not been enabled on that account, or an SCP is denying requests outside an approved region list. Administrator permissions do not override an SCP, which is rather the point of one.</p>



<h3 class="wp-block-heading">How hard is it to change region later?</h3>



<p class="wp-block-paragraph">Harder than a move between accounts and much harder than people estimate. Every region-scoped identifier changes, stateful services need a data move with a cutover, and any regional endpoint baked into client configuration has to be coordinated. Budget it as a project, not a task. Which is exactly why the upfront decision deserves the care.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Choosing an AWS region for MENA clients is a legal decision with a technical implementation, not the reverse. Residency eliminates candidates, service availability eliminates more, latency picks between whatever survives. Run the filter in that order and you will sometimes land on a region that is not the closest one, which is fine. Run it backwards and you get a fast architecture in the wrong jurisdiction, which no amount of edge caching fixes.</p>



<p class="wp-block-paragraph">Get the transfer basis in writing, enforce the decision with a guardrail rather than a convention, and record why you chose what you chose.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help choosing or defending a region decision?</h2>



<p class="wp-block-paragraph">This is the kind of work where an outside opinion early saves a rebuild later. Things I can help with:</p>



<ul class="wp-block-list">
<li>Running a service availability matrix across candidate regions for your actual architecture, so the gaps surface before any infrastructure code is written</li>



<li>Writing and testing a region deny SCP with the global-service exemptions tuned to your service inventory, without locking anyone out</li>



<li>Building a repeatable latency measurement from your target markets rather than from a vendor&#8217;s map</li>



<li>Designing the CDN and caching split so static content goes to the edge and personal data does not</li>



<li>Reviewing an existing deployment for accidental cross-region data flows in backups, replication, logging and monitoring</li>



<li>Producing the written region rationale your client&#8217;s auditor or counsel is going to ask for</li>
</ul>



<p class="wp-block-paragraph">Send me the architecture diagram, a Terraform provider block, or the output of your region opt-in check, and I will tell you what I would look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-region-for-mena/">Choosing an AWS Region for a MENA Client: Latency Is the Last Question</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</title>
		<link>https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Comprehend]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=272</guid>

					<description><![CDATA[<p>A confident answer with a citation from a matter the reader was walled off from. Nothing crashed, nothing alerted. Building legal document intelligence on AWS with Textract, Bedrock and OpenSearch is the easy half; the hard half is tenant isolation, verified deletion, S3 Object Lock holds, what leaves the account, and an audit trail that names the actual user.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo was going well until someone asked where the third citation came from.</p>



<p class="wp-block-paragraph">The answer on screen was good. Fluent, specific, correctly hedged, three sources listed underneath it. Then a partner in the room asked a simple question: which matter is source three from? Nobody could answer it in the room, and when we went and looked, it was from a matter that half the people watching the demo were formally walled off from.</p>



<p class="wp-block-paragraph">Nothing had crashed. No alarm fired. The retrieval layer had done exactly what a similarity search does, which is return the nearest vectors, and the nearest vectors did not care about the ethical wall. That is the thing about building a legal document intelligence platform on AWS: the failures that matter almost never look like failures. They look like a confident answer with a citation attached.</p>



<p class="wp-block-paragraph">This post is about the parts of that build that are hard, organised around the five boundaries that actually have to hold. The extraction pipeline is the easy half. I will cover it, but quickly, because the AWS documentation is good and the failure modes are visible. The other four boundaries fail quietly, and that is where the engineering goes.</p>



<h2 class="wp-block-heading">The shape of the thing</h2>



<p class="wp-block-paragraph">Before the boundaries, the skeleton. A legal document intelligence platform on AWS almost always ends up looking like this, whether you plan it or arrive at it:</p>



<ul class="wp-block-list">
<li>Documents land in Amazon S3, one prefix per matter, versioning on.</li>



<li>An S3 event triggers AWS Lambda, which starts an asynchronous Amazon Textract job.</li>



<li>Textract output lands back in S3 as JSON. Something normalises it into text plus page and bounding-box coordinates.</li>



<li>Amazon Comprehend or a foundation model classifies the document and pulls entities: parties, dates, governing law, clause types.</li>



<li>Chunks get embedded and written to a vector store, usually Amazon OpenSearch Serverless behind Amazon Bedrock Knowledge Bases.</li>



<li>An application layer, typically API Gateway plus Lambda, takes a question, retrieves, and calls a model on Amazon Bedrock.</li>
</ul>



<p class="wp-block-paragraph">AWS Step Functions is worth reaching for once you have more than three stages, because Textract jobs are long-running and Lambda timeouts are not a retry strategy. That is the whole architecture. You can stand it up in a fortnight. Then you spend six months on everything below.</p>



<h2 class="wp-block-heading">Boundary one: isolation, and why metadata filtering is not optional</h2>



<p class="wp-block-paragraph">This is the one from the demo, and it is the one that ends engagements.</p>



<p class="wp-block-paragraph">A vector index has no native concept of a matter, a client, or an ethical wall. If you pool every document into one index and rely on the prompt to keep things separate, you have built a system where a well-phrased question can pull privileged material across a wall. Prompt instructions are not an access control. They are a suggestion that usually works.</p>



<p class="wp-block-paragraph">There are two shapes that do work, and the choice between them is a real trade-off rather than a best practice.</p>



<h3 class="wp-block-heading">Pooled index with server-side filtering</h3>



<p class="wp-block-paragraph">Every chunk carries a <code>matter_id</code> and a <code>client_id</code> as metadata. Every retrieval call attaches a filter. With an S3 data source in Bedrock Knowledge Bases, the metadata comes from a sidecar JSON file that sits next to the document and carries a <code>metadataAttributes</code> object.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "matter_id": "M-4417",
    "client_id": "C-108",
    "doc_type": "engagement_letter",
    "privileged": true
  }
}</code></pre>



<p class="wp-block-paragraph">The retrieval call then narrows the search before the model ever sees a chunk:</p>



<pre class="wp-block-code"><code>"retrievalConfiguration": {
  "vectorSearchConfiguration": {
    "filter": {
      "equals": { "key": "matter_id", "value": "M-4417" }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two rules make this safe, and both are the kind of thing that gets skipped under deadline. First, the filter value is derived on the server from the authenticated caller&#8217;s claims, never accepted from the client. If the browser can send you a <code>matter_id</code>, the browser can send you a different one. Second, the filter is applied by code that no feature request can bypass. A helper that builds every retrieval request, and a code review rule that no other path may call the retrieve API directly.</p>



<p class="wp-block-paragraph">The trap here is ordering. You cannot filter on an attribute the index does not have. Add <code>matter_id</code> after ingestion and every chunk already in the index is invisible to that filter, which means it either returns for everyone or for no one, depending on how your filter is written. Design the metadata schema before the first ingestion run, not after the first demo.</p>



<h3 class="wp-block-heading">Separate collection per client</h3>



<p class="wp-block-paragraph">The heavier option. A separate OpenSearch Serverless collection per client, which buys you a separate AWS KMS key per client, separate index settings, and a failure mode where a bug in the filtering logic cannot reach across clients at all.</p>



<p class="wp-block-paragraph">It costs you sprawl. Every collection is a resource to provision, monitor, patch policy on, and eventually delete. Ingestion jobs run per data source per knowledge base, so freshness guarantees fragment. For a firm with twelve institutional clients this is fine. For a platform onboarding a hundred small clients it becomes the main operational burden of the product.</p>



<p class="wp-block-paragraph">My default is pooled with strict server-side filtering, and per-client collections only where the client&#8217;s own contract demands a dedicated encryption key. If you cannot say out loud which of those two you are running, you are running neither properly.</p>



<h2 class="wp-block-heading">Boundary two: extraction, and the confidence score everyone ignores</h2>



<p class="wp-block-paragraph">Textract returns a confidence score between 0 and 100 with each prediction. AWS is explicit in its own best-practice guidance that applications sensitive to detection errors should enforce a minimum threshold and route anything below it for human scrutiny, and that the right threshold depends entirely on how the output gets used.</p>



<p class="wp-block-paragraph">Legal documents sit at the harsh end of that. A misread date on an archival scan is a curiosity. A misread date on a limitation period is a problem with a name on it.</p>



<p class="wp-block-paragraph">Practical notes from this layer:</p>



<ul class="wp-block-list">
<li>Use the asynchronous operations for anything multipage. <code>StartDocumentAnalysis</code> and <code>GetDocumentAnalysis</code> handle long PDFs and TIFFs; the synchronous <code>AnalyzeDocument</code> path is for single pages and will fight you on a 400-page bundle.</li>



<li>Textract&#8217;s Queries feature earns its place on structured instruments. Instead of extracting everything and grepping, you ask the document a direct question and get a scoped answer with its own confidence.</li>



<li>Keep the page number and bounding box for every chunk. When a lawyer asks where a sentence came from, &#8220;page 14, second column&#8221; is an answer. &#8220;It&#8217;s in the corpus somewhere&#8221; is not, and the platform loses credibility the first time you say it.</li>



<li>Amazon Augmented AI wires low-confidence predictions into a human review queue with a private workforce, so the reviewers are your people rather than an anonymous pool. For privileged material that distinction is the entire point.</li>
</ul>



<p class="wp-block-paragraph">What I would skip early: building a custom entity recognition model. Comprehend supports custom entity recognition and it is genuinely useful, but you need labelled data you do not have yet, and a foundation model with a decent prompt gets you far enough to find out which entities the users actually care about. Train the custom model once the requirement has stopped moving.</p>



<h2 class="wp-block-heading">Boundary three: retention, where deletion is a distributed problem</h2>



<p class="wp-block-paragraph">Here is the failure that will not show up in any test suite you write.</p>



<p class="wp-block-paragraph">A client asks for a document to be removed. Someone deletes the S3 object. The ticket closes. The document is still fully searchable, because its chunks are sitting in the vector index and will stay there until the data source is re-synced. Meanwhile the Textract JSON output is in a second bucket, the extracted text may be in a database, and if model invocation logging is on, whole passages of the document are sitting in CloudWatch Logs or another S3 bucket entirely.</p>



<p class="wp-block-paragraph">One delete, at least four places holding a copy. Write the deletion path as a real workflow with an assertion at the end, not as a single API call:</p>



<ol class="wp-block-list">
<li>Delete the source object and, if versioning is on, its versions.</li>



<li>Delete the derived artefacts: Textract JSON, normalised text, any thumbnails or page images.</li>



<li>Trigger an ingestion job so the knowledge base drops the orphaned chunks.</li>



<li>Confirm removal by querying the index for the document identifier and expecting nothing back.</li>



<li>Deal with the logs, which means either a retention policy short enough to make the problem expire or a deliberate decision, written down, that logs are out of scope.</li>
</ol>



<p class="wp-block-paragraph">Step four is the one people leave out, and it is the only step that actually proves anything.</p>



<h3 class="wp-block-heading">The opposite problem: things that must not be deleted</h3>



<p class="wp-block-paragraph">Legal work has the reverse requirement too. When a matter goes into litigation hold, the documents need to survive an administrator with delete permissions and a bad afternoon.</p>



<p class="wp-block-paragraph">S3 Object Lock is the mechanism, and it has two independent controls that people routinely conflate. A retention period protects an object version until a fixed date, in either governance mode, which privileged users can override, or compliance mode, which nobody can override and where the period cannot be shortened. A legal hold has no date at all. It stays until someone with <code>s3:PutObjectLegalHold</code> explicitly removes it. The two can be active at once, and while either is active the object version cannot be deleted or overwritten.</p>



<pre class="wp-block-code"><code># Place an indefinite hold on a specific object version
aws s3api put-object-legal-hold 
  --bucket matters-archive 
  --key M-4417/exhibit-c.pdf 
  --version-id 3sL7f2Qz9pXvB1kR 
  --legal-hold Status=ON</code></pre>



<p class="wp-block-paragraph">Two things bite here. Object Lock requires versioning and turns it on automatically, and once enabled on a bucket you cannot turn it off or suspend versioning again. And compliance mode is genuinely permanent: if you set a seven-year retention when you meant seven days, that is the answer, for everyone, including the account root. Test in governance mode first. The bypass path exists and is deliberately awkward, requiring both the <code>s3:BypassGovernanceRetention</code> permission and an explicit <code>x-amz-bypass-governance-retention:true</code> header on the request, which is exactly the level of friction you want on that operation.</p>



<p class="wp-block-paragraph">Design the hold model before you design the deletion model, because holds win. A deletion request that collides with an active hold is a legal question, not an engineering one, and the platform&#8217;s job is to surface the collision clearly rather than resolve it silently.</p>



<h2 class="wp-block-heading">Boundary four: disclosure, or what leaves the account</h2>



<p class="wp-block-paragraph">This is the boundary the client&#8217;s general counsel will ask about, usually in writing, usually before signature. Three separate controls, and they are not interchangeable.</p>



<h3 class="wp-block-heading">The AI services opt-out policy</h3>



<p class="wp-block-paragraph">AWS Organizations has a governance policy type that opts your accounts out of having content processed by certain AI services stored and used for service improvement. The list includes Amazon Textract and Amazon Comprehend, which are precisely the two doing the reading in this architecture. The policy type has to be enabled at the organisation root before you can attach anything:</p>



<pre class="wp-block-code"><code>aws organizations enable-policy-type 
  --root-id r-example 
  --policy-type AISERVICES_OPT_OUT_POLICY</code></pre>



<p class="wp-block-paragraph">Read the AWS documentation on this one carefully rather than taking my summary as gospel, because there is a caveat in it that matters: the services may still need to store your data operationally even when you have opted out of it being used for improvement. Opting out is not the same as the data never existing outside your account. Say that plainly in the client conversation. It is a much better position than being asked about it later.</p>



<h3 class="wp-block-heading">Bedrock&#8217;s data position, and the log that undoes it</h3>



<p class="wp-block-paragraph">Amazon Bedrock&#8217;s published position is that inputs and outputs are not shared with third-party model providers and are not used to train the base models, and that fine-tuning operates on a private copy. That is a strong starting point for privileged content and it is the main reason Bedrock rather than a direct provider API shows up in these builds.</p>



<p class="wp-block-paragraph">Then there is model invocation logging. It captures full prompt and response payloads to CloudWatch Logs or S3, and you want it on, because without it you cannot reconstruct what the system told someone. But understand what you have just built: a log that contains privileged document text, at a per-region setting, in a destination that probably has looser access controls than the document store it came from. Encrypt it with the same KMS key discipline, restrict it harder than you think you need to, and set a retention period on purpose.</p>



<p class="wp-block-paragraph">Worth separating in your head: CloudTrail records that an API call happened and who made it. Model invocation logging records what was in it. You need both, for different questions, and only one of them contains client confidences.</p>



<h3 class="wp-block-heading">Network path</h3>



<p class="wp-block-paragraph">Interface VPC endpoints via AWS PrivateLink keep traffic to Bedrock, Textract and the rest on the AWS network rather than out through an internet gateway or NAT. Add a gateway endpoint for S3 while you are there.</p>



<p class="wp-block-paragraph">Endpoint policies are the part that gets forgotten. An endpoint without a policy is a private path to the whole service, including into accounts that are not yours. Scope it to the operations and resources you actually use, and pair it with an S3 bucket policy that rejects requests not arriving through your endpoint. The first protects what leaves; the second protects what can be reached.</p>



<h3 class="wp-block-heading">The parts that are not on the architecture diagram</h3>



<p class="wp-block-paragraph">Two of these have caught people out badly, and neither is an AWS control.</p>



<p class="wp-block-paragraph">The first is the development environment. You do not want real client documents on a laptop or in a scratch account, so build the parsing and chunking code against synthetic documents on a small separate box. A cheap VPS from a provider like Contabo or InterServer is fine for this, and the separation is worth more than the convenience of iterating in the production account.</p>



<p class="wp-block-paragraph">The second is the reviewers. If your human review loop involves contractors working remotely, their network path and their disks are part of your boundary whether or not you drew them. A managed VPN such as NordVPN or Surfshark handles the network side, and when a matter closes and local copies have to go, a dedicated erasure tool from something like O&amp;O Software does what dragging a folder to the bin does not. Unglamorous, and it is the layer that gets audited.</p>



<h2 class="wp-block-heading">Boundary five: evidence, because someone will ask</h2>



<p class="wp-block-paragraph">At some point the question stops being &#8220;does it work&#8221; and becomes &#8220;who saw what, and when&#8221;. If you cannot answer that from stored records, the platform is not defensible regardless of how good the answers are.</p>



<ul class="wp-block-list">
<li>Turn on CloudTrail data events for the document buckets. They are off by default, they are billed separately from management events, and they are the only way to see individual object-level reads.</li>



<li>Log the retrieval, not just the generation. Store which chunks came back and which filter was applied. When somebody asks whether the wall held on a specific query, this record is the answer and there is no reconstructing it later.</li>



<li>Carry the end user&#8217;s identity through to the audit record. A Lambda execution role in the logs tells you the platform did something. It does not tell you who asked.</li>



<li>Alarm on the filter, not just on errors. A retrieval that ran without a matter filter should page someone, even though it returned HTTP 200 and a perfectly good answer.</li>
</ul>



<p class="wp-block-paragraph">That last one is the closest thing to a single takeaway in this post. The dangerous events in this architecture are successful ones.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the things that will go wrong</h2>



<h3 class="wp-block-heading">Retrieval returns nothing after you add a filter</h3>



<p class="wp-block-paragraph">Nearly always a metadata problem rather than a query problem. Either the sidecar metadata file was missing at ingestion, so the chunks carry no attribute to match, or the attribute was added after the chunks were indexed. Check whether the attribute exists on a known chunk before you touch the query. If it does not, you need a re-sync, not a different filter expression.</p>



<h3 class="wp-block-heading">Textract returns text but the layout is scrambled</h3>



<p class="wp-block-paragraph">Common on two-column pleadings and on documents with headers and footers on every page. Raw text detection reads in a reading order that is not always yours. Use the layout and table features rather than plain detection, and hold onto the geometry so you can reassemble columns yourself if the default order is wrong for that document class.</p>



<h3 class="wp-block-heading">The model cites a document that does not exist</h3>



<p class="wp-block-paragraph">Usually the citation is being generated rather than passed through. If document names are being written by the model instead of read from the retrieval response, it will invent plausible ones. Build citations from the retrieval result metadata in your application code and never from the generated text. Bedrock Guardrails helps with contextual grounding, but the structural fix is not asking the model to produce the reference in the first place.</p>



<h3 class="wp-block-heading">Ingestion is slow and nobody knows where</h3>



<p class="wp-block-paragraph">Instrument per stage before you optimise anything. In most of these pipelines the time is in Textract for large scanned bundles, and the fix is parallelism at the document level rather than tuning the extraction itself. Step Functions with a distributed map over documents gets you there without a queue you have to babysit.</p>



<h3 class="wp-block-heading">Costs climb faster than volume</h3>



<p class="wp-block-paragraph">Look at re-processing first. A pipeline that re-extracts a document every time anything downstream changes will quietly multiply your Textract bill against a static corpus. Key the extraction cache on the object&#8217;s content hash, not its path, and make re-ingestion an explicit action. The billing mechanisms differ per service, so read the current pricing pages rather than trusting a number from a blog post.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating the system prompt as an access control.</strong> It is a formatting instruction that happens to look like a rule.</li>



<li><strong>Taking the tenant identifier from the request body.</strong> If the client can name the matter, the client can name someone else&#8217;s.</li>



<li><strong>Designing the metadata schema after the first ingestion run.</strong> Retrofitting a filterable attribute means re-indexing the whole corpus.</li>



<li><strong>Assuming a deleted S3 object is gone from the platform.</strong> The chunks, the extraction output and the invocation logs all outlive it.</li>



<li><strong>Setting compliance-mode retention without testing in governance mode.</strong> There is no support ticket that fixes a seven-year mistake.</li>



<li><strong>Enabling model invocation logging without treating the log as privileged.</strong> You have just made a second copy of the documents in a less protected place.</li>



<li><strong>Building VPC endpoints and leaving the default policy on them.</strong> A private path with no policy is still a path.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>One retrieval helper, server-side filter injection, and a review rule that no other code path calls the retrieve API directly.</li>



<li>Metadata schema agreed and frozen before the first production ingestion. Include the fields you might filter on later, even if unused today.</li>



<li>Page-level provenance on every chunk, surfaced in the interface. It builds trust faster than any accuracy improvement.</li>



<li>A confidence threshold chosen per document class, with a human review path for everything under it.</li>



<li>Deletion implemented as a verified workflow across every store, ending in an assertion that the content is actually unretrievable.</li>



<li>Object Lock legal holds for matters under litigation hold, tested in governance mode before anything runs in compliance mode.</li>



<li>Customer-managed KMS keys across S3, the vector store and the logs, so key access is a control you can actually revoke.</li>



<li>An alarm on unfiltered retrievals, because the failure mode is a success response.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a legal document intelligence platform on AWS safe for privileged material?</h3>



<p class="wp-block-paragraph">The services support it. Bedrock&#8217;s stated position is that inputs and outputs are not shared with model providers or used to train base models, PrivateLink keeps traffic off the public internet, and KMS covers encryption at rest. What determines safety is your own configuration: tenant isolation, log handling and deletion behaviour. The platform is exactly as confidential as its weakest copy of the text.</p>



<h3 class="wp-block-heading">Should I use Bedrock Knowledge Bases or build retrieval myself?</h3>



<p class="wp-block-paragraph">Start with Knowledge Bases. It handles chunking, embedding, sync and metadata filtering, and those are weeks of work with no differentiation in them. Build your own when you need retrieval behaviour it does not expose, such as unusual re-ranking or per-tenant chunking strategies. Going custom on day one usually means reimplementing the managed service badly.</p>



<h3 class="wp-block-heading">How do I stop one client&#8217;s documents surfacing in another client&#8217;s answers?</h3>



<p class="wp-block-paragraph">Tag every chunk with a tenant identifier at ingestion, and inject the matching filter server-side on every retrieval from the authenticated caller&#8217;s identity. Never accept the identifier from the client. If the contract requires per-client encryption keys, use a separate collection per client instead, accepting the extra operational load.</p>



<h3 class="wp-block-heading">Does deleting a document from S3 remove it from the search index?</h3>



<p class="wp-block-paragraph">No. The chunks stay in the vector store until an ingestion job runs and reconciles the data source. Until then the document is deleted and still fully searchable, which is the worst combination available. Always finish a deletion by querying the index and confirming nothing comes back.</p>



<h3 class="wp-block-heading">What is the difference between an S3 legal hold and a retention period?</h3>



<p class="wp-block-paragraph">A retention period runs until a fixed date and comes in governance mode, which privileged users can override, or compliance mode, which nobody can. A legal hold has no end date and stays until someone explicitly removes it. They are independent, they can both be active on the same object version, and while either is active the object cannot be deleted or overwritten.</p>



<h3 class="wp-block-heading">Do I need human review, or is the model accurate enough?</h3>



<p class="wp-block-paragraph">Accuracy is the wrong frame. The question is what happens to a wrong answer downstream. Where output feeds a decision with consequences, you want a confidence threshold and a review queue, and Amazon Augmented AI with a private workforce gives you that without building the review tooling yourself. Where output is a search aid a person will verify anyway, review is friction you do not need.</p>



<h3 class="wp-block-heading">Can I run this in a single AWS account?</h3>



<p class="wp-block-paragraph">Technically yes, and for a pilot it is reasonable. It gets uncomfortable once you have production documents and a development environment in the same place, because an IAM mistake has nowhere to stop. Separate accounts under Organizations also gives you the policy layer, including the AI services opt-out policy, which only exists at the organisation level.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A legal document intelligence platform on AWS is not hard to build. Textract reads, Bedrock reasons, OpenSearch retrieves, and the tutorial version works on the first afternoon.</p>



<p class="wp-block-paragraph">What is hard is that its failures are silent and well-formed. The cross-matter citation, the deleted document that still answers questions, the privileged passage sitting in a log bucket, the retrieval that ran without a filter and returned a beautiful paragraph. None of these throw an error. All of them are the kind of thing that ends a client relationship.</p>



<p class="wp-block-paragraph">So build the boundaries first and the features second, and make sure every one of them is something you can prove from a stored record rather than something you believe about the code. If you can only take one habit from this: alarm on the successful requests that should not have been possible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your document pipeline?</h2>



<p class="wp-block-paragraph">I work on AWS document processing and retrieval systems where the confidentiality requirements are real. Typical things I get called in for:</p>



<ul class="wp-block-list">
<li>Reviewing tenant isolation in an existing RAG setup and finding the paths where the filter can be bypassed.</li>



<li>Designing the metadata and chunking schema before ingestion, so filtering works without a re-index later.</li>



<li>Building Textract pipelines with confidence thresholds, human review routing and page-level provenance.</li>



<li>Implementing verified deletion across S3, derived artefacts, the vector index and logs, with a proof step at the end.</li>



<li>Setting up retention and legal hold with S3 Object Lock, including the governance-mode rehearsal before compliance mode.</li>



<li>Locking down the network and logging path: VPC endpoints with real policies, KMS key separation, and audit records that name the actual user.</li>
</ul>



<p class="wp-block-paragraph">If you have an architecture diagram, a Step Functions definition or a retrieval request you are unsure about, send it over and I will tell you what I would change and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Customer Sentiment Analysis From CRM and Support Data: Building a Score You Can Actually Trust</title>
		<link>https://john-nessime.com/blog/technical-guides/customer-sentiment-analysis-crm-support-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 22 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Comprehend]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[CRM Analytics]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[EU AI Act]]></category>
		<category><![CDATA[NLP]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Sentiment Analysis]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Zendesk]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=250</guid>

					<description><![CDATA[<p>Most customer sentiment analysis pipelines run perfectly and still produce a number nobody should act on. Here are the failure families that cause it, from labels stamped on the wrong message to averages taken over a scale that was never numeric, plus a pipeline shape and a legal boundary worth knowing before you ship.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/customer-sentiment-analysis-crm-support-data/">Customer Sentiment Analysis From CRM and Support Data: Building a Score You Can Actually Trust</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The CX lead pings you on a Thursday afternoon: &#8220;Sentiment has been sitting at 3.2 for four months. Is the pipeline broken?&#8221;</p>



<p class="wp-block-paragraph">You check. Every extraction job ran. Every ticket got a label. No rows dropped, no schema drift, no dead letters. Nothing is broken. The number is just useless, and it has been useless since the day it shipped.</p>



<p class="wp-block-paragraph">That is the normal outcome for customer sentiment analysis built on CRM and support data. The connectors work, the model returns labels, the dashboard renders on schedule, and the output still cannot tell anyone whether customers are getting happier or angrier. These are not pipeline failures. They are measurement failures, which is exactly why your monitoring never catches them.</p>



<p class="wp-block-paragraph">This post walks through the failure families I check first when a sentiment programme is producing numbers nobody trusts: where the label gets stamped, what happens when you average an ordinal scale, who is missing from the data entirely, why a general-purpose model misreads support English, where the CRM join breaks, and one regulatory boundary that catches more teams than it should. Then a pipeline shape that holds up, and the checks worth running before anyone puts this on a slide.</p>



<h2 class="wp-block-heading">What the score is actually measuring</h2>



<p class="wp-block-paragraph">Before anything else, get specific about the unit of analysis. &#8220;Customer sentiment&#8221; is a phrase people use for at least four different measurements:</p>



<ul class="wp-block-list">
<li><strong>Message sentiment</strong> — the tone of one comment in one thread.</li>

<li><strong>Ticket sentiment</strong> — usually the first inbound message, sometimes a rollup.</li>

<li><strong>Contact sentiment</strong> — one human&#8217;s tone across all their tickets.</li>

<li><strong>Account sentiment</strong> — the thing the business actually cares about, and the one nobody computes correctly.</li>
</ul>



<p class="wp-block-paragraph">Most stacks compute the second and label it the fourth. That single substitution causes more bad decisions than any model choice you will make downstream. A model with mediocre accuracy applied at the right unit beats an excellent model applied at the wrong one.</p>



<h2 class="wp-block-heading">Failure one: the label is stamped at the wrong moment</h2>



<p class="wp-block-paragraph">This is the expensive one, and it is invisible from the pipeline side because nothing errors.</p>



<p class="wp-block-paragraph">Zendesk&#8217;s intelligent triage is the clearest example because the documentation is honest about it. Zendesk states that most intelligent triage classifications are based on a ticket&#8217;s first message only, with an admin-configurable option to update topic and sentiment from the latest end-user message. If nobody turned that option on, your &#8220;customer sentiment&#8221; field is a permanent record of how annoyed someone was in their opening paragraph.</p>



<p class="wp-block-paragraph">Think about what that does to a trend line. The opening message of a support ticket is negative almost by definition, because people do not open tickets to say things are going well. What you want to know is whether the interaction <em>ended</em> well. A ticket that opens furious and closes with a thank-you is a support win, and a first-message-only score records it as a loss forever.</p>



<p class="wp-block-paragraph">Two consequences follow that people miss:</p>



<ul class="wp-block-list">
<li>Your sentiment metric is structurally incapable of moving in response to anything the support team does. Hire better agents, cut response time in half, rewrite your macros: the number will not budge, because it was set before any of that happened.</li>

<li>It <em>will</em> move in response to marketing, onboarding and product changes upstream, because those change who opens tickets and how angry they are on arrival. So the metric drifts for reasons the team watching it cannot influence.</li>
</ul>



<p class="wp-block-paragraph">Zendesk also documents that when an agent corrects the sentiment field on a ticket, the correction does not retrain the model. Worth knowing before you build a human-review workflow on the assumption that it feeds back into anything. It gives you a cleaner column, not a better classifier.</p>



<p class="wp-block-paragraph">The fix is to decide your stamping policy explicitly and store it. Score the first inbound message, score the last inbound message, and store both, along with the delta. The delta is the interesting column. Sentiment recovery within a thread is a far better signal of support quality than absolute sentiment ever was.</p>



<h2 class="wp-block-heading">Failure two: averaging a scale that was never numeric</h2>



<p class="wp-block-paragraph">Zendesk exposes sentiment as five ordered values from Very negative to Very positive, and Zendesk Explore ships a prebuilt report for average customer sentiment on a one-to-five scale. Amazon Comprehend returns one of <code>POSITIVE</code>, <code>NEGATIVE</code>, <code>NEUTRAL</code> or <code>MIXED</code> plus per-class confidence scores. Google Cloud Natural Language returns a polarity score and a magnitude. Azure AI Language returns document- and sentence-level labels with confidences.</p>



<p class="wp-block-paragraph">Only some of those are numbers, and none of the label sets are interval scales. The gap between Very negative and Negative is not the same size as the gap between Neutral and Positive, and there is no defensible reason to assume it is. So when you compute <code>AVG(sentiment)</code>, you are doing arithmetic on rank labels and getting a value that has no units.</p>



<p class="wp-block-paragraph">That would be a pedantic complaint if it were only a purity issue. It is not, because support sentiment distributions are usually bimodal. People write in angry or they write in grateful. Very few tickets are genuinely neutral. Take the mean of a two-humped distribution and you land in the valley between the humps, in a bucket where almost nobody actually is. Worse, the mean is stable there: the humps can grow, shrink and swap size while the average barely moves. That is your flat 3.2.</p>



<p class="wp-block-paragraph">Use share metrics instead. They are ordinal-safe, they move, and they are directly actionable.</p>



<pre class="wp-block-code"><code>-- Share of negative tickets, weighted by account value.
-- Ordinal-safe: no arithmetic on the label itself, only counting.
SELECT
    date_trunc('week', t.created_at)                      AS week,
    COUNT(*)                                              AS tickets,
    COUNT(*) FILTER (
        WHERE t.sentiment IN ('very_negative', 'negative')
    )                                                     AS negative_tickets,
    ROUND(
        100.0 * COUNT(*) FILTER (
            WHERE t.sentiment IN ('very_negative', 'negative')
        ) / NULLIF(COUNT(*), 0)
    , 1)                                                  AS pct_negative,
    SUM(a.arr) FILTER (
        WHERE t.sentiment IN ('very_negative', 'negative')
    )                                                     AS arr_behind_negative
FROM   support_tickets  t
JOIN   crm_contacts     c ON c.id = t.contact_id
JOIN   crm_accounts     a ON a.id = c.account_id
WHERE  t.created_at &gt;= date_trunc('week', CURRENT_DATE) - INTERVAL '26 weeks'
GROUP  BY 1
ORDER  BY 1;</code></pre>



<p class="wp-block-paragraph">Three things are happening there worth calling out. <code>FILTER</code> counts rows in a category without touching the label as a number. <code>NULLIF</code> stops a zero-ticket week from throwing a division error and silently killing the job. And the last column is the one executives will actually read, because a percentage of tickets is a support metric while revenue sitting behind negative sentiment is a business metric.</p>



<h2 class="wp-block-heading">Failure three: the people who left never showed up in the data</h2>



<p class="wp-block-paragraph">Support data only contains customers who chose to contact you. That is a selection effect, not a sample.</p>



<p class="wp-block-paragraph">The dangerous version is this: negative sentiment falls, everyone celebrates, and the actual cause is that the angriest cohort stopped writing in because they had already decided to leave. Silence reads as improvement. In a quarter where churn is rising, a falling negative-sentiment share is a warning sign, not a win.</p>



<p class="wp-block-paragraph">You cannot fix this inside the support dataset, which is precisely why the CRM join matters. Two guards worth building:</p>



<ul class="wp-block-list">
<li><strong>Contact rate as a companion metric.</strong> Tickets per active account per month, tracked alongside sentiment. Sentiment improving while contact rate collapses is a disengagement pattern, not a satisfaction one.</li>

<li><strong>Went-quiet detection.</strong> Accounts that used to file tickets regularly and have filed none in a defined window. In most books of business this list is a better churn predictor than the sentiment score itself, and it costs one window function to produce.</li>
</ul>



<h2 class="wp-block-heading">Failure four: the model learned someone else&#8217;s language</h2>



<p class="wp-block-paragraph">General-purpose sentiment APIs are trained largely on review-style and social text. Support English is a different dialect, and the mismatches are systematic rather than random.</p>



<ul class="wp-block-list">
<li><strong>Problem statements are not complaints.</strong> &#8220;The export fails with a 500&#8221; is a factual bug report. Review-trained models frequently score it negative because failure vocabulary dominates.</li>

<li><strong>Politeness masks severity.</strong> A calmly worded message from an enterprise admin saying the production sync has been down since Tuesday is your worst ticket of the week and will score close to neutral.</li>

<li><strong>Domain vocabulary inverts.</strong> &#8220;Killed the process&#8221;, &#8220;the job died&#8221;, &#8220;aborted the run&#8221; are neutral technical descriptions carrying heavy negative weight in general corpora.</li>

<li><strong>Your own text pollutes the corpus.</strong> Auto-acknowledgements, signatures, quoted reply chains and CSAT survey footers all get scored if you feed the raw comment body. Quoted history is the big one: score a five-reply thread naively and you have scored message one five times.</li>

<li><strong>Non-native English reads flat.</strong> Simpler sentence structure and less emotional vocabulary push scores toward neutral for a whole segment of your customer base, which quietly biases any regional comparison.</li>
</ul>



<p class="wp-block-paragraph">Independent evaluations of the major cloud sentiment APIs on general text tend to land well below the accuracy figures in vendor marketing, and the providers frequently disagree with each other on the same input. Treat published benchmark numbers as irrelevant to your decision. The only number that matters is accuracy on a few hundred of your own tickets, hand-labelled by someone who understands your product.</p>



<h3 class="wp-block-heading">Building the evaluation set</h3>



<ol class="wp-block-list">
<li>Pull a stratified random sample of tickets: across channels, across regions, across account tiers. Do not sample only recent tickets, or you bake in whatever was happening that month.</li>

<li>Have two people label independently, using a written rubric with worked examples of each class.</li>

<li>Measure agreement between the humans first. If your two support leads disagree on a third of the tickets, the task is underspecified and no model will fix that. Tighten the rubric and relabel.</li>

<li>Only then score the candidate models against the human labels, and look at the confusion matrix rather than accuracy. Where a model confuses classes tells you whether it is usable.</li>

<li>Freeze that set. It becomes your regression test when a vendor silently updates a model version.</li>
</ol>



<p class="wp-block-paragraph">Step three is the one teams skip, and it is the one that saves the project. Human agreement is the ceiling on model performance. There is no point chasing 90% against labels your own experts only agree on 70% of the time.</p>



<h2 class="wp-block-heading">Failure five: the CRM join is where the value and the breakage both live</h2>



<p class="wp-block-paragraph">A sentiment label with no account attached is trivia. Two hundred irritated free-tier users and one irritated account paying for a third of your renewals are not the same event, and the raw ticket table cannot tell them apart.</p>



<p class="wp-block-paragraph">The join from helpdesk to CRM is almost always on email, and email is a terrible key. Expect all of this:</p>



<ul class="wp-block-list">
<li>Shared inboxes such as ops@ or billing@, mapping to no single contact or to several.</li>

<li>Personal addresses used by people who work at a customer, invisible to any domain-matching logic.</li>

<li>Contacts who left and were deleted in Salesforce, HubSpot or Zoho CRM, orphaning every historical ticket they filed.</li>

<li>Account hierarchies, where the subsidiary files the ticket and the parent holds the contract.</li>

<li>Mergers and renames, which quietly rewrite history if your warehouse dimension is not versioned.</li>
</ul>



<p class="wp-block-paragraph">The engineering answer is boring and effective: treat unmatched tickets as a first-class, monitored quantity rather than something the join silently discards. Track match rate as a data quality metric with an alert threshold. When it drops, something changed in the CRM, and you want to know that before the number on the dashboard shifts for a reason nobody can explain.</p>



<p class="wp-block-paragraph">Also version the account dimension. If you overwrite the tier or the ARR in place, then every historical aggregate silently rewrites itself the next time someone upgrades, and last quarter&#8217;s report stops reproducing.</p>



<h2 class="wp-block-heading">The legal boundary that catches sentiment projects</h2>



<p class="wp-block-paragraph">This one is worth reading carefully if you have EU customers or EU staff, because the line does not fall where most people assume, and a lot of contact-centre software sells both sides of it in one bundle.</p>



<p class="wp-block-paragraph">Article 5(1)(f) of the EU AI Act prohibits AI systems that infer emotions of a person in the workplace or in education settings. The prohibitions in Article 5 have applied since 2 February 2025, and breaches sit in the Act&#8217;s highest penalty band. What matters for a sentiment project is the scope, which is narrow on two axes at once:</p>



<ul class="wp-block-list">
<li>It applies to emotion inference <strong>from biometric data</strong> — faces, voiceprints, physiological signals. The European Commission&#8217;s guidance takes the position that inferring emotion from written text is not based on biometric data and therefore falls outside the prohibition.</li>

<li>It applies to <strong>workplace and education contexts</strong>. Your customers writing into a helpdesk are neither.</li>
</ul>



<p class="wp-block-paragraph">So scoring the text of customer tickets and CRM notes generally sits outside the ban. Running voice-based emotion or stress analysis on your own support agents&#8217; calls, for coaching or quality scoring, is the case the prohibition was written for. Plenty of platforms ship customer-facing text sentiment and agent-facing voice sentiment as adjacent features in the same product, and the pitch does not distinguish them. If you are the deployer, the compliance obligation is yours, and a vendor&#8217;s assurance is not a defence.</p>



<p class="wp-block-paragraph">Separately, Article 50 carries transparency obligations for emotion recognition systems where they remain lawful. And none of this displaces GDPR: ticket bodies are full of personal data, and shipping them to a third-party API is a processing decision that needs a basis, a processor agreement and a residency answer.</p>



<p class="wp-block-paragraph">I am an engineer, not a lawyer. Treat the above as the shape of the question to take to counsel, not as advice.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A pipeline shape that survives contact</h2>



<p class="wp-block-paragraph">Nothing exotic. The discipline is in what you store, not what you call.</p>



<ol class="wp-block-list">
<li><strong>Extract incrementally</strong> from the helpdesk and CRM on an updated-at cursor, not a full reload. Land raw payloads immutably before you transform anything, so a bad model run is re-runnable without re-hitting the source API.</li>

<li><strong>Clean the text.</strong> Strip quoted reply chains, signatures, automated acknowledgements and survey footers. Keep the cleaned text as its own column so a scoring bug is debuggable later.</li>

<li><strong>Redact personal data</strong> before it leaves your boundary. Regex handles the structured patterns: emails, phone numbers, card and account numbers, order IDs. Named-entity detection handles names and addresses. Amazon Comprehend exposes <code>DetectPiiEntities</code> and <code>ContainsPiiEntities</code> for this if you are already in AWS.</li>

<li><strong>Score in batches</strong>, with the model identifier and version written into every row.</li>

<li><strong>Aggregate at account level</strong> with share metrics and value weighting, against a versioned account dimension.</li>

<li><strong>Alert on movement, not on level.</strong> Nobody acts on &#8220;sentiment is 3.2&#8221;. People act on &#8220;this named account&#8217;s negative share tripled in two weeks&#8221;.</li>
</ol>



<p class="wp-block-paragraph">Step four&#8217;s version column is the one people leave out and regret. Managed sentiment endpoints get retrained. When labels shift on unchanged text, the version column is the difference between a five-minute explanation and a week of forensics.</p>



<p class="wp-block-paragraph">Here is what a batch call looks like against Amazon Comprehend, which is a reasonable default if your warehouse already lives in AWS:</p>



<pre class="wp-block-code"><code>import boto3

comprehend = boto3.client("comprehend", region_name="eu-west-1")

# BatchDetectSentiment accepts a maximum of 25 documents per call,
# and DetectSentiment caps each document at 5 KB of UTF-8 text.
# Chunk to both limits or the call fails for the whole batch.
BATCH_SIZE = 25
MAX_BYTES  = 5000

def score_batch(texts):
    payload = [t.encode("utf-8")[:MAX_BYTES].decode("utf-8", "ignore")
               for t in texts[:BATCH_SIZE]]
    resp = comprehend.batch_detect_sentiment(
        TextList=payload,
        LanguageCode="en",
    )
    # ResultList is index-aligned to the input; ErrorList holds
    # per-document failures. Read both or you lose rows silently.
    return resp["ResultList"], resp["ErrorList"]</code></pre>



<p class="wp-block-paragraph">The truncation on the encoded bytes rather than the character count is deliberate: the limit is measured in bytes, and a ticket full of accented characters or CJK text hits it much earlier than the character count suggests. The <code>ErrorList</code> read matters for the same reason a dead letter queue matters. A partial batch success that you never inspect is data loss that looks like a clean run.</p>



<p class="wp-block-paragraph">For a quick sanity check on a single string before wiring any of this up, the CLI is faster than writing code:</p>



<pre class="wp-block-code"><code>aws comprehend detect-sentiment 
    --region eu-west-1 
    --language-code "en" 
    --text "Still waiting on the refund you promised last week."

{
    "SentimentScore": {
        "Mixed": 0.0033542951568961143,
        "Positive": 0.0086313202530145600,
        "Neutral": 0.1495875907897949,
        "Negative": 0.8384268 
    },
    "Sentiment": "NEGATIVE",
    "LanguageCode": "en"
}</code></pre>



<p class="wp-block-paragraph">Note that the response carries per-class confidences, not just the winning label. Store the whole <code>SentimentScore</code> object. A ticket scored NEGATIVE at 0.51 and one scored NEGATIVE at 0.98 are very different tickets, and the label alone throws that away. Confidence is also what lets you route only the ambiguous cases to a more expensive model or a human, instead of paying for everything.</p>



<h3 class="wp-block-heading">Choosing where the model runs</h3>



<p class="wp-block-paragraph">Three broad options, and the trade-offs are real in both directions.</p>



<ul class="wp-block-list">
<li><strong>Native helpdesk features.</strong> Zendesk intelligent triage, Salesforce Einstein, HubSpot and Zoho CRM all ship some form of built-in sentiment. Zero integration work, labels land on the ticket where agents can see them, and routing rules can consume them directly. You get the vendor&#8217;s definition of sentiment, no control over stamping policy beyond the toggles they expose, and data that is awkward to move.</li>

<li><strong>Managed NLP APIs.</strong> Amazon Comprehend, Azure AI Language and Google Cloud Natural Language. Per-character billing, no infrastructure, useful extras such as aspect-level or targeted sentiment. Comprehend&#8217;s targeted sentiment, which attaches sentiment to specific entities in the text, is English-only, and that constraint bites hard if your support volume is multilingual. Bill by volume of text, so a batch backfill of five years of tickets is a genuinely different cost event from steady-state scoring.</li>

<li><strong>Self-hosted transformers.</strong> A fine-tuned encoder model from the Hugging Face ecosystem, running on your own box. Best accuracy on your domain once tuned, no per-call cost, no text leaving your perimeter, which resolves most residency arguments before they start. In exchange you own model serving, evaluation and drift. A mid-tier dedicated server from a provider like Contabo or Hetzner runs CPU inference for batch workloads perfectly well, and batch is what this workload is. You do not need a GPU to score yesterday&#8217;s tickets overnight.</li>
</ul>



<p class="wp-block-paragraph">If you have no strong residency constraint and a small volume, start with a managed API. The self-hosted path is worth it when you have enough labelled data to fine-tune, or a legal reason the text cannot leave. Not before.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Sentiment is flat regardless of what the team does</h3>



<p class="wp-block-paragraph">Two likely causes. Either you are averaging an ordinal scale over a bimodal distribution, or you are scoring first messages only. Check the distribution shape first, since it takes one query. If it has two humps, switch to share metrics before you touch anything else.</p>



<h3 class="wp-block-heading">Everything scores neutral</h3>



<p class="wp-block-paragraph">Usually a text-cleaning problem. If quoted reply chains, signatures and boilerplate survive into the scored text, the actual message is diluted by noise and the model regresses to neutral. Print the exact string you are sending for twenty tickets and read them. This bug is invisible in aggregate and obvious in the raw payload.</p>



<h3 class="wp-block-heading">The trend shifted overnight with no matching business event</h3>



<p class="wp-block-paragraph">Check the model version column and the CRM match rate on the same day. A managed endpoint retrain and a CRM field change both produce exactly this signature. If you did not store the version, this is the moment you start.</p>



<h3 class="wp-block-heading">One region always looks happier</h3>



<p class="wp-block-paragraph">Before concluding anything about that region, check the language distribution. Non-native English and machine-translated tickets both drift toward neutral, which shows up as &#8220;less negative&#8221; in a share metric. Compare like with like by language, or score in the original language where the provider supports it.</p>



<h3 class="wp-block-heading">Batch scoring jobs fail intermittently</h3>



<p class="wp-block-paragraph">Almost always the document size limit, hit by one oversized ticket with a pasted log file or stack trace in the body. Truncate on encoded byte length before the call and check the per-document error list rather than assuming a successful HTTP response means every document scored.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Reporting an average of an ordinal label set and calling it a score.</li>

<li>Scoring the first message and describing the result as customer satisfaction.</li>

<li>Treating unmatched tickets as an inner-join casualty instead of a monitored metric.</li>

<li>Sending raw ticket bodies to a third-party API with no redaction step.</li>

<li>Discarding confidence scores and keeping only the winning label.</li>

<li>Trusting a vendor accuracy figure over a few hundred of your own hand-labelled tickets.</li>

<li>Overwriting the account dimension in place, so historical reports quietly stop reproducing.</li>

<li>Assuming a helpdesk vendor&#8217;s compliance posture transfers to you as the deployer.</li>
</ul>



<h2 class="wp-block-heading">Best practices for customer sentiment analysis</h2>



<ul class="wp-block-list">
<li>Write down the unit of analysis before writing any code, and make sure the dashboard label matches it.</li>

<li>Store first-message sentiment, last-message sentiment and the delta. Report the delta.</li>

<li>Use share-of-negative and value-weighted share, never a mean over labels.</li>

<li>Keep a frozen, hand-labelled evaluation set and re-run it on a schedule as a regression test.</li>

<li>Persist the model identifier, model version and full confidence distribution on every scored row.</li>

<li>Redact personal data before the text crosses a network boundary, and log what was redacted, not the values.</li>

<li>Monitor CRM match rate, contact rate and language mix alongside the sentiment number itself.</li>

<li>Alert on named-account movement, not on the global level.</li>

<li>Know which side of the biometric line each feature you enable sits on, and put the answer in writing.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is customer sentiment analysis accurate enough to act on?</h3>



<p class="wp-block-paragraph">For prioritisation and trend detection, generally yes. For individual decisions about a specific customer, no. The reliable pattern is to use it as a filter that surfaces conversations for a human to read, rather than as a verdict. Any single ticket&#8217;s label should be treated as a hint, not a fact.</p>



<h3 class="wp-block-heading">Should I use my helpdesk&#8217;s built-in sentiment or build my own pipeline?</h3>



<p class="wp-block-paragraph">Start with the built-in feature. It is free or cheap, it lands where agents work, and it tells you within a month whether anyone actually uses sentiment for anything. Build your own when you need a different stamping policy, cross-system aggregation, account-value weighting, or control over where the text is processed. Those are the four reasons that justify the effort.</p>



<h3 class="wp-block-heading">Can I use an LLM instead of a dedicated sentiment API?</h3>



<p class="wp-block-paragraph">You can, and the quality on nuance, sarcasm and domain vocabulary is usually better because you can describe your product and your customers in the prompt. The costs are latency, price per token at backfill scale, and output stability. Pin the model version, constrain the output to a fixed label set, and set temperature to zero, or you will get labels that drift between runs on identical text. A pragmatic middle ground is a cheap classifier by default with an LLM as a fallback on low-confidence cases.</p>



<h3 class="wp-block-heading">How much historical data do I need before the trend means anything?</h3>



<p class="wp-block-paragraph">Enough to cover your seasonality, which for most businesses means at least a full year. Support volume and tone move with billing cycles, product releases, holidays and renewal periods. A quarter of data will show you a trend that is really just the calendar.</p>



<h3 class="wp-block-heading">Does sentiment analysis on support tickets fall under the EU AI Act ban?</h3>



<p class="wp-block-paragraph">The Article 5(1)(f) prohibition targets emotion inference from biometric data in workplace and education contexts, and Commission guidance treats text-based sentiment as outside that scope. Scoring customer-written text is generally not caught. Voice or facial emotion analysis applied to your own employees is squarely the thing it prohibits. Transparency duties and GDPR obligations still apply regardless, and this is a question for your counsel rather than your engineer.</p>



<h3 class="wp-block-heading">What is a good target for share of negative tickets?</h3>



<p class="wp-block-paragraph">There is no cross-industry benchmark worth quoting, because the number depends entirely on your product category, your support channels and how the model was calibrated. Your own baseline is the only meaningful comparison. Measure for a full cycle, then set targets against that.</p>



<h3 class="wp-block-heading">How do I handle multilingual support volume?</h3>



<p class="wp-block-paragraph">Detect the language, store it as a column, and segment every report by it. Do not translate to English and then score, since translation flattens emotional register and adds a second error source. Check which of your provider&#8217;s features are English-only before designing around them; some advanced modes, including Comprehend&#8217;s targeted sentiment, are narrower than the base sentiment endpoint.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A customer sentiment analysis pipeline that runs cleanly every night and produces a number nobody can act on is not a partially working system. It is a broken one, and the breakage is upstream of anything your monitoring can see.</p>



<p class="wp-block-paragraph">So before optimising a model or shopping for a platform, answer three questions in writing: which message gets scored, what unit the number describes, and which accounts it is weighted by. Get those right with a mediocre classifier and you have something useful. Get them wrong with the best model on the market and you have a very accurate description of how annoyed people were in their opening sentence.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help making this work on your data?</h2>



<p class="wp-block-paragraph">I build and fix the data plumbing behind customer analytics. On this particular problem, that usually means:</p>



<ul class="wp-block-list">
<li>Auditing an existing sentiment metric to find out why it never moves, and rebuilding it on share and delta metrics that do.</li>

<li>Building incremental extraction from Zendesk, Salesforce, HubSpot or Zoho CRM into a warehouse, with cursor state, retries and dead-letter handling that survives an API outage.</li>

<li>Fixing the helpdesk-to-CRM identity join, including shared inboxes, account hierarchies and a versioned account dimension so historical reports keep reproducing.</li>

<li>Standing up a redaction layer so ticket text is scrubbed of personal data before it reaches any third-party model endpoint.</li>

<li>Running a proper evaluation: stratified sample, dual human labelling, inter-rater agreement, confusion matrices across candidate models, and a frozen regression set.</li>

<li>Setting up account-level alerting in Grafana or your BI tool that fires on movement in named accounts rather than on a global average.</li>
</ul>



<p class="wp-block-paragraph">If you have a dashboard that has been flat for months, send me the query behind it and a screenshot of the distribution. That is usually enough to tell you what is wrong before we talk about scope.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/customer-sentiment-analysis-crm-support-data/">Customer Sentiment Analysis From CRM and Support Data: Building a Score You Can Actually Trust</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</title>
		<link>https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 25 Jul 2026 12:04:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Config]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[Healthcare Cloud]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[Restore Testing]]></category>
		<category><![CDATA[VPC]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=184</guid>

					<description><![CDATA[<p>A working engineer's guide to HIPAA compliance on AWS, organised by the gap between the control you configured and the obligation you actually carry. Covers BAA account scope, the eligible services list as a contract boundary, KMS key policy versus the encryption checkbox, what "six years" really applies to, backup and restore scope, and the subprocessor chain nobody inventories.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/">HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket usually reads something like: &#8220;Legal wants to know if the analytics account is in scope.&#8221; So you open Security Hub. Green. You check the Config rules. Passing. Every bucket is encrypted, every volume is encrypted, MFA is on, CloudTrail is running in all Regions. You reply that the account is fine.</p>



<p class="wp-block-paragraph">Then someone points out that a nightly job has been copying a de-identified extract into that account for eight months, the de-identification script never removed the admission dates, and the account was spun up before anyone thought about the Business Associate Addendum. Nothing was misconfigured. Every control you built worked exactly as designed. And you have been out of compliance the entire time.</p>



<p class="wp-block-paragraph">That is the shape of most real failures here. Not a breach, not a misconfiguration, but a mismatch between the boundary your tooling checks and the boundary your obligation actually follows. This post covers HIPAA compliance on AWS organised by those gaps: where the contract stops, where encryption stops being a control, what &#8220;six years&#8221; genuinely applies to, and which parts of the estate people forget are in scope at all.</p>



<h2 class="wp-block-heading">Eligible is not compliant, and the difference is the whole job</h2>



<p class="wp-block-paragraph">AWS does not sell HIPAA compliance. It sells HIPAA <em>eligible</em> services, which is a genuinely different thing. Eligible means AWS has built the service so it can lawfully handle electronic protected health information and has agreed to cover it under a Business Associate Addendum. Compliant describes an entire system: your architecture, your key management, your access reviews, your policies, your staff, your vendors.</p>



<p class="wp-block-paragraph">Under the shared responsibility model, AWS secures the infrastructure. You secure everything you build on it. Nothing about signing the BAA transfers a single obligation off your side of the line. An unencrypted RDS instance, an overly broad IAM policy or an application that logs a patient identifier into stdout is your problem in exactly the same way it would be in a rack you own.</p>



<p class="wp-block-paragraph">People know this in the abstract. Where it bites is in the specifics below.</p>



<h2 class="wp-block-heading">Gap one: the BAA is a contract boundary, and nothing enforces it</h2>



<p class="wp-block-paragraph">This is the one I would fix first, because it is invisible to every security tool you own.</p>



<p class="wp-block-paragraph">The AWS BAA is self-service through AWS Artifact, at no extra cost. You can accept it for a single account, or, if you are in the management account of an AWS Organization, accept it once so that existing and future member accounts are covered. That organization-level option is the one worth using, because the per-account version quietly rots: someone creates a new account for a proof of concept, nobody repeats the Artifact step, and six months later that account is running something real.</p>



<p class="wp-block-paragraph">The second half of the boundary is the HIPAA Eligible Services Reference that AWS publishes. Only services on that list may create, receive, process, maintain or transmit ePHI under the BAA. The list is long, it changes, and some entries carry carve-outs where the service is eligible but a specific feature is not. Reading a service name on the list and assuming every feature inside it is covered is the kind of mistake that only surfaces during an audit.</p>



<p class="wp-block-paragraph">Here is the part worth internalising: <strong>there is no AWS control that stops you putting PHI into a non-eligible service.</strong> No API error, no Config rule out of the box, no GuardDuty finding. The eligible services list is a contractual construct. Your infrastructure has no idea it exists.</p>



<h3 class="wp-block-heading">Turning a contract boundary into a technical one</h3>



<p class="wp-block-paragraph">The mechanism that actually helps is Service Control Policies on the organizational unit that holds your PHI accounts. SCPs set the ceiling on what any principal in those accounts can do, including the root user, so they work as a guardrail rather than a suggestion.</p>



<p class="wp-block-paragraph">Start with the easy one. Pin the accounts to the Regions you have actually assessed, because data residency assumptions fall apart the moment someone launches something in a Region you never reviewed:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnapprovedRegions",
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "organizations:*",
        "route53:*",
        "cloudfront:*",
        "support:*",
        "sts:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The <code>NotAction</code> list matters. Global services are backed by endpoints in specific Regions, so denying them wholesale by Region locks you out of IAM and breaks Route 53 and CloudFront. Those entries are exemptions, not an allow-list.</p>



<p class="wp-block-paragraph">The harder one is restricting which services can be used at all. The same <code>NotAction</code> pattern works, with the services you have approved for PHI listed as the exemptions and everything else denied. It is effective and it is blunt: every new service anyone wants becomes a change request against the policy, and if you forget a dependency you find out through a failure in production. I would only reach for it on a dedicated PHI OU where the workload is well understood, not across a general-purpose organization.</p>



<p class="wp-block-paragraph">Whichever route you take, write down the approved service list somewhere a human reviews on a schedule, and diff it against the AWS reference periodically. That review is itself a compliance artefact.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap two: encryption is a checkbox, the key policy is the control</h2>



<p class="wp-block-paragraph">Almost every guide to HIPAA compliance on AWS tells you to encrypt at rest and in transit. Almost none of them explain why it is worth doing properly rather than minimally, so teams enable default encryption with an AWS-managed key, watch the Config rule turn green, and move on.</p>



<p class="wp-block-paragraph">The reason to care is the Breach Notification Rule. It applies to <em>unsecured</em> PHI, meaning PHI that has not been rendered unusable, unreadable or indecipherable through a method HHS has specified. HHS guidance points at NIST-validated encryption. If PHI is encrypted to that standard and the decryption keys were not compromised alongside it, an incident involving that data generally does not trigger the notification machinery at all. No individual letters, no HHS portal submission, no press release for a large incident.</p>



<p class="wp-block-paragraph">Read that second condition again, because it is where the architecture decision lives. The safe harbour depends on the keys not being compromised with the data. If your encryption key is one an attacker inherits automatically the moment they compromise a role in the account, you have encryption but you may not have the argument.</p>



<h3 class="wp-block-heading">What that means in practice</h3>



<ul class="wp-block-list">
<li>Use customer managed KMS keys for anything holding PHI, not AWS-managed keys. Only a customer managed key gives you a key policy you can write, and only a key policy lets you deny decryption independently of the resource policy.</li>

<li>Separate the key administrators from the key users. The people who can schedule deletion of a key should not be the people whose application role uses it every second.</li>

<li>Use a distinct key per data domain rather than one key for the whole account. Blast radius and audit trail both improve, and you get the ability to revoke access to one dataset without touching another.</li>

<li>Constrain key usage with the <code>kms:ViaService</code> condition so a key that exists to encrypt RDS storage cannot be used to decrypt something a role dragged into Lambda.</li>

<li>Turn on key rotation and leave it on. It costs nothing operationally and it is the kind of thing an assessor asks about by reflex.</li>
</ul>



<p class="wp-block-paragraph">Pull the current key policy before you assume it says what you think:</p>



<pre class="wp-block-code"><code>aws kms get-key-policy 
  --key-id alias/phi-rds 
  --policy-name default 
  --output text

# Find storage that slipped through unencrypted
aws ec2 describe-volumes 
  --filters Name=encrypted,Values=false 
  --query 'Volumes[].{Id:VolumeId,AZ:AvailabilityZone}' 
  --output table

aws rds describe-db-instances 
  --query 'DBInstances[?StorageEncrypted==`false`].DBInstanceIdentifier' 
  --output text</code></pre>



<p class="wp-block-paragraph">The RDS query is the important one, because RDS encryption cannot be enabled in place. If that command returns anything, the fix is a snapshot, an encrypted copy of the snapshot, a restore, and a cutover. Plan for downtime or a replication strategy. This is the single most common &#8220;we will fix it later&#8221; item I see, and later gets expensive.</p>



<p class="wp-block-paragraph">Also switch on EBS encryption by default in every Region you use, so the next instance somebody launches from a console wizard is not a new exception:</p>



<pre class="wp-block-code"><code>aws ec2 enable-ebs-encryption-by-default --region us-east-1
aws ec2 get-ebs-encryption-by-default --region us-east-1</code></pre>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap three: you have logs, but you may not have evidence</h2>



<p class="wp-block-paragraph">The Security Rule requires audit controls: mechanisms that record and examine activity in systems containing ePHI. It also requires you to regularly review records of information system activity. Both of those are about having and using the records.</p>



<p class="wp-block-paragraph">Now the correction, because this one is repeated everywhere and it is wrong in a way that costs money. You will read that HIPAA requires six years of audit logs. It does not. The six-year requirement sits in the documentation standard, and it applies to the policies, procedures and records of actions, activities and assessments that the Security Rule requires you to keep, retained for six years from creation or from when the document was last in effect, whichever is later. There is no clause anywhere in the Security Rule that names a retention period for CloudTrail events.</p>



<p class="wp-block-paragraph">What this actually means is more demanding, not less. You have to <em>decide</em> your log retention period, write it into a policy, justify it against your risk analysis, and then keep that policy for six years. And an assessor will hold you to the number you wrote. Setting a CloudWatch Logs retention of thirty days while your policy claims one year is a finding. Storing seven years of everything because a blog told you to, when your policy says two, is not compliance, it is just a bill.</p>



<p class="wp-block-paragraph">So: pick a period you can defend, make the infrastructure match it exactly, and treat any gap between policy and configuration as a defect.</p>



<h3 class="wp-block-heading">Making logs into evidence</h3>



<p class="wp-block-paragraph">Retention is only half of it. The other half is being able to show that the records were not altered. CloudTrail has log file validation for exactly this, and it is off unless you turn it on:</p>



<pre class="wp-block-code"><code>aws cloudtrail update-trail 
  --name org-phi-trail 
  --enable-log-file-validation

# Later, prove a window of logs is intact
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:us-east-1:111122223333:trail/org-phi-trail 
  --start-time "$(date -u -d '90 days ago' +%Y-%m-%dT%H:%M:%SZ)"</code></pre>



<p class="wp-block-paragraph">With validation enabled, CloudTrail writes signed digest files alongside the log files, and <code>validate-logs</code> checks them. The difference between &#8220;here are our logs&#8221; and &#8220;here are our logs, and here is a cryptographic check that nothing was modified or deleted&#8221; is the difference between an assertion and evidence.</p>



<p class="wp-block-paragraph">Put the archive bucket in a separate account that the workload accounts cannot write to or delete from, and apply S3 Object Lock in compliance mode for the retention window you committed to. Object Lock in compliance mode cannot be shortened or bypassed by anyone, including the root user, which is exactly the property you want and exactly the property that will hurt if you set the period carelessly. Test it in governance mode first.</p>



<p class="wp-block-paragraph">For the review obligation, a query interface matters more than raw storage. Athena over the CloudTrail bucket is the cheap default. If you want alerting and dashboards on top of access patterns, this is a natural place for a platform such as Grafana, Datadog or Splunk, and any of them will hold access records for you. Just remember that if those records contain PHI, that vendor needs a BAA too. See the subprocessor section below.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap four: backups, snapshots and the parts of scope people forget</h2>



<p class="wp-block-paragraph">The Security Rule&#8217;s contingency plan standard is not optional decoration. It requires a data backup plan, a disaster recovery plan and an emergency mode operation plan, plus testing and revision procedures. Most teams have the backups. Far fewer have the tested restore, and the tested restore is the part that gets asked about.</p>



<p class="wp-block-paragraph">Three things routinely go wrong here.</p>



<ol class="wp-block-list">
<li><strong>Copies leave the boundary.</strong> A cross-Region snapshot copy lands in a Region you did not assess. A cross-account copy for the DR account lands somewhere outside the OU your SCPs protect. The data is still PHI. The controls did not travel with it.</li>

<li><strong>Re-encryption changes the key, not just the copy.</strong> Copying an encrypted snapshot to another account requires a key the destination can use. It is easy to end up with a shared or less restrictive key protecting your backups than protects production, which inverts the risk model.</li>

<li><strong>The restore is never rehearsed.</strong> A backup you have never restored is a hypothesis. Schedule a restore into an isolated account, record the elapsed time, and file the result. That record is your evidence for the testing requirement, and it is the single easiest compliance artefact to produce for free.</li>
</ol>



<p class="wp-block-paragraph">While you are inventorying, remember the places PHI ends up without anyone deciding it should: application logs that include request bodies, database slow query logs capturing parameter values, support tickets with screenshots attached, CSV extracts in an analyst&#8217;s bucket, and non-production environments seeded from a production dump. That last one is the classic. If your staging database is a copy of production, staging is in scope, and staging is almost never built to the same standard.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap five: the business associate chain does not stop at AWS</h2>



<p class="wp-block-paragraph">Your BAA with AWS covers AWS. It covers nothing else in your stack.</p>



<p class="wp-block-paragraph">Every vendor that can create, receive, maintain or transmit PHI on your behalf is a business associate and needs an agreement. In a typical AWS estate that means the error tracker holding stack traces, the log aggregation platform, the APM tool, the transactional email provider, the customer support desk, the CI system if it ever touches a production dataset, and any AI or analytics service you have wired in.</p>



<p class="wp-block-paragraph">Build the inventory as a table with three columns: vendor, what PHI it can see, and whether a signed agreement exists. The third column is usually where the surprises are. Some vendors sign readily, some only on higher-priced tiers, and some decline entirely, at which point you have an architecture decision rather than a procurement one.</p>



<p class="wp-block-paragraph">One structural move that reduces this surface considerably: keep everything that does not need PHI out of the PHI accounts entirely. Your marketing site, your docs, your status page and your public API gateway for non-clinical traffic do not belong in a regulated account. Running them on ordinary infrastructure, whether that is a separate AWS account, a straightforward VPS from a host like InterServer, or a static site behind Cloudflare, shrinks the estate you have to assess, evidence and defend. Fewer things in scope is the cheapest compliance win available.</p>



<p class="wp-block-paragraph">For tracking the paperwork side, compliance automation platforms such as Vanta, Drata or Secureframe pull evidence from AWS on a schedule and keep the vendor register current. They are genuinely useful for the collection and reminder burden. They do not design your architecture, and I have seen teams treat a green dashboard in one of those tools as though it were an assessment. It is not. It is a checklist that knows what you told it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What is changing, and why &#8220;addressable&#8221; is a bad thing to build on</h2>



<p class="wp-block-paragraph">Since it was adopted, the Security Rule has split implementation specifications into <em>required</em> and <em>addressable</em>. Addressable never meant optional. It meant you assess whether the specification is reasonable and appropriate, and if not, you implement an equivalent alternative or document why neither is necessary. In practice, a lot of organisations turned the documented justification into the deliverable and skipped the control.</p>



<p class="wp-block-paragraph">HHS published a Notice of Proposed Rulemaking in the Federal Register in January 2025 that would remove that distinction, making implementation specifications required with limited exceptions, and would explicitly require encryption of ePHI at rest and in transit and multi-factor authentication, again with limited exceptions. The comment period closed in March 2025.</p>



<p class="wp-block-paragraph">Be precise about the status, because a lot of vendor content is not: <strong>this is a proposed rule and it is not final.</strong> The expected timeline for final action has slipped more than once, and the requirements could still change or be withdrawn. Nobody should be telling you a compliance deadline as though it were settled.</p>



<p class="wp-block-paragraph">What is worth taking from it is the direction of travel. If your current position depends on having documented that encryption or MFA was not reasonable and appropriate, that position is fragile regardless of what the final rule says. On AWS specifically, encryption at rest and MFA are both cheap and both already best practice. Building the architecture on an addressable deferral is an unforced risk.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the findings you will actually hit</h2>



<h3 class="wp-block-heading">&#8220;An assessor asked which accounts are in BAA scope and nobody could answer&#8221;</h3>



<p class="wp-block-paragraph">Sign in to AWS Artifact from the management account and check the organization agreements tab to see whether the BAA was accepted at the organization level or per account. If it is per account, list your accounts, work out which hold PHI, and confirm each one individually. Then move to the organization-level agreement so this question has one answer forever.</p>



<h3 class="wp-block-heading">&#8220;Config says the bucket is encrypted but we cannot prove who read the objects&#8221;</h3>



<p class="wp-block-paragraph">Bucket encryption and object-level access logging are unrelated. CloudTrail management events do not record S3 object reads by default. You need CloudTrail data events for that bucket, or S3 server access logging, and both cost money proportional to request volume. Enable data events selectively on the buckets that hold PHI rather than account-wide.</p>



<h3 class="wp-block-heading">&#8220;We enabled an SCP and production broke&#8221;</h3>



<p class="wp-block-paragraph">Almost always a Region deny catching a global service endpoint, or a service allow-list missing a dependency the workload calls indirectly. Check CloudTrail for <code>AccessDenied</code> events with an explicit deny from an SCP, and look at the service name in the event rather than the one you expected. Attach new SCPs to a test OU with a representative workload before the PHI OU.</p>



<h3 class="wp-block-heading">&#8220;Snapshot copy to the DR account fails with a KMS error&#8221;</h3>



<p class="wp-block-paragraph">The destination account cannot use the source key. The source key policy has to grant the destination principal permission to use it, and the copy has to specify a key the destination can decrypt with. Fix it by granting explicitly on a key you control, not by falling back to an AWS-managed key, which is the tempting shortcut and gives up the key policy control you needed.</p>



<h3 class="wp-block-heading">&#8220;CloudWatch Logs retention was never set&#8221;</h3>



<p class="wp-block-paragraph">New log groups default to never expiring, which is both a cost problem and a policy mismatch. Audit them with <code>aws logs describe-log-groups</code> and look for groups with no <code>retentionInDays</code> value, then set the period your policy specifies with <code>aws logs put-retention-policy</code>.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the signed BAA as the finish line rather than the prerequisite. It is the thing you need before the first byte of PHI arrives, not evidence that anything is configured correctly.</li>

<li>Assuming a service is fully eligible because its name appears on the list, without reading the feature-level carve-outs next to it.</li>

<li>Quoting &#8220;six years&#8221; as a log retention requirement, then either overspending on storage or writing a policy that contradicts the actual configuration.</li>

<li>Using AWS-managed KMS keys for PHI, which leaves no key policy to write and no independent revocation path.</li>

<li>Seeding staging or test environments from production data and then holding those environments to a lower standard.</li>

<li>Forgetting that the risk analysis is a required, recurring, documented activity, not a one-off spreadsheet from the year you launched.</li>

<li>Signing a BAA with AWS and none of the ten other vendors that can see the same data.</li>

<li>Letting a compliance automation dashboard stand in for an architecture review.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li><strong>Isolate PHI in its own accounts and its own OU.</strong> Account boundaries are the strongest isolation AWS offers, and they make the scope question answerable in one sentence.</li>

<li><strong>Accept the BAA at the organization level.</strong> It removes an ongoing manual step that fails silently.</li>

<li><strong>Customer managed keys, one per data domain, with split admin and usage roles.</strong> This is what makes the breach safe harbour argument defensible rather than theoretical.</li>

<li><strong>Write the retention period down first, configure second.</strong> Policy and infrastructure should agree exactly, in both directions.</li>

<li><strong>Ship audit logs to a separate account with Object Lock and CloudTrail validation enabled.</strong> Immutability and integrity are what turn logs into evidence.</li>

<li><strong>Keep PHI out of everything that does not need it.</strong> De-identify early, tokenise where you can, and route non-clinical traffic through infrastructure that is not in scope.</li>

<li><strong>Define everything in Terraform or OpenTofu.</strong> A reviewable, version-controlled definition of your controls is worth more to an assessor than any screenshot, and it stops drift being invisible.</li>

<li><strong>Rehearse the restore and the breach response.</strong> Both are required, both are tested by asking for the record, and both are cheap to evidence if you actually do them.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is AWS HIPAA compliant?</h3>



<p class="wp-block-paragraph">Not on its own, and the phrasing is the problem. AWS offers HIPAA-eligible services and will sign a Business Associate Addendum, which means you can build a compliant system on it. Compliance is a property of your whole environment, including configuration, policies, vendors and staff. No provider can sell it to you as a finished product.</p>



<h3 class="wp-block-heading">How do I sign a BAA with AWS?</h3>



<p class="wp-block-paragraph">Through AWS Artifact in the console. It is self-service and there is no additional charge. Accept it for an individual account under account agreements, or from the management account of an AWS Organization under organization agreements so all current and future member accounts are covered. It should be accepted by someone with authority to bind your organisation, and it must be in place before any PHI reaches AWS.</p>



<h3 class="wp-block-heading">Does HIPAA require six years of CloudTrail logs?</h3>



<p class="wp-block-paragraph">No. The six-year requirement is a documentation retention rule covering the policies, procedures and records the Security Rule requires you to maintain, kept for six years from creation or from when they were last in effect. The audit controls standard requires the mechanism to record and examine activity but does not name a retention period for the logs themselves. You set that period in your own policy, justify it, and make the configuration match.</p>



<h3 class="wp-block-heading">Which AWS services can I use with PHI?</h3>



<p class="wp-block-paragraph">Only those on the AWS HIPAA Eligible Services Reference, and only in accounts covered by your BAA. Check the list before adopting anything new, read the feature-level exclusions noted against individual services, and re-check periodically because entries are added over time. Nothing in AWS will stop you using a non-eligible service with PHI, so this has to be an explicit process on your side.</p>



<h3 class="wp-block-heading">If encrypted PHI is exposed, do I still have to report a breach?</h3>



<p class="wp-block-paragraph">Generally no, provided the encryption meets the standard in HHS guidance and the decryption keys were not compromised along with the data. The Breach Notification Rule applies to unsecured PHI, and properly encrypted data does not meet that definition. This is why key management, not just enabling encryption, is the part that determines whether the protection is real. You still document the incident and the assessment.</p>



<h3 class="wp-block-heading">Are the new HIPAA Security Rule requirements in force?</h3>



<p class="wp-block-paragraph">Not at the time of writing. The proposals to make all implementation specifications required and to mandate encryption and multi-factor authentication came from a Notice of Proposed Rulemaking published in January 2025. The comment period has closed, but no final rule has been issued and the timeline has moved. Treat any specific compliance deadline you see quoted with suspicion and check the current status directly.</p>



<h3 class="wp-block-heading">Is a HIPAA-compliant AWS environment expensive to run?</h3>



<p class="wp-block-paragraph">The controls themselves are mostly cheap. KMS keys, CloudTrail validation, Config rules and account separation cost very little. The real costs are log storage volume, CloudTrail data events on busy buckets, running non-production environments to production standard, and staff time on risk analysis and evidence collection. Reducing what is in scope is the most effective cost lever, which is another reason to keep non-clinical workloads out of the regulated accounts.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">HIPAA compliance on AWS fails at the seams, not at the controls. Your encryption works. Your IAM policies are tight. What goes wrong is that the obligation follows the data into an account nobody added to the BAA, a Region nobody assessed, a staging database seeded from production, a vendor nobody signed an agreement with, or a retention period nobody wrote down.</p>



<p class="wp-block-paragraph">So build the boundary technically rather than trusting it contractually. Isolate PHI into its own accounts, wrap those accounts in guardrails that make the contract boundary enforceable, own your keys so the encryption means something legally, and make your logs provable rather than merely present. Then write the whole thing down, because in this domain the documentation genuinely is part of the control.</p>



<p class="wp-block-paragraph">None of that is exotic engineering. It is ordinary AWS work applied to a boundary that no dashboard draws for you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Working on a healthcare workload on AWS?</h2>



<p class="wp-block-paragraph">This is the kind of work I do. If you are building or inheriting a PHI environment on AWS, I can help with:</p>



<ul class="wp-block-list">
<li><strong>Scope and boundary review:</strong> mapping which accounts, Regions, services and vendors actually touch PHI, and finding the ones nobody knew about.</li>

<li><strong>Account and OU design with enforceable guardrails:</strong> SCPs, organization-level BAA coverage, and Region and service restrictions that hold without breaking your workloads.</li>

<li><strong>KMS key architecture:</strong> customer managed keys per data domain, split administration and usage, and key policies written so the breach safe harbour argument stands up.</li>

<li><strong>Audit logging that produces evidence:</strong> centralised CloudTrail with log file validation, an isolated archive account with Object Lock, and retention that matches your written policy exactly.</li>

<li><strong>Backup, restore and contingency testing:</strong> encrypted cross-account copies that stay inside your boundary, plus rehearsed restores documented as compliance artefacts.</li>

<li><strong>Terraform or OpenTofu modules for the whole control set,</strong> so your posture is reviewable, repeatable and does not drift between audits.</li>
</ul>



<p class="wp-block-paragraph">If you would rather start with something concrete than a discovery call, send me a redacted account structure, an SCP that is causing trouble, or the output of a Config or Security Hub run, and I will tell you what I would look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/">HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
