<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Encryption | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/encryption/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/encryption/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Mon, 19 Oct 2026 06:00:12 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Encryption | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/encryption/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Building a Secure Patient Document Portal on S3 and Cognito</title>
		<link>https://john-nessime.com/blog/cloud-computing/secure-patient-document-portal-s3-cognito/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 19 Oct 2026 06:00:12 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[ePHI]]></category>
		<category><![CDATA[Healthcare Cloud]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Identity Pools]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[MFA]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[Session Management]]></category>
		<category><![CDATA[Token Revocation]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=457</guid>

					<description><![CDATA[<p>Cognito for sign-in, S3 for the files, and a demo that passes on the first try. The problems start after that. Prefix isolation, presigned URL lifetimes, KMS as a second gate, CloudTrail data events, and token revocation that actually ends a session.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/secure-patient-document-portal-s3-cognito/">Building a Secure Patient Document Portal on S3 and Cognito</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The request usually arrives sideways. A clinic manager says something like: &#8220;we&#8217;re emailing scans as PDF attachments and our insurer flagged it, can you build us something?&#8221; Nobody says the words &#8220;patient portal.&#8221; They say &#8220;somewhere for patients to pick up their files.&#8221;</p>



<p class="wp-block-paragraph">So you sketch the obvious thing. Amazon Cognito for sign-in, Amazon S3 for the documents, a small front end in between. It&#8217;s three services and a weekend, and it works on the first try, which is exactly the problem. A <strong>secure patient document portal</strong> built this way passes every test you think to run, because the tests you think to run are all &#8220;can Alice download Alice&#8217;s file?&#8221; The one that matters is &#8220;what can Alice see about Bob?&#8221; and that one has an answer you won&#8217;t like.</p>



<p class="wp-block-paragraph">This post covers the parts that go wrong after the demo. Where to put the authorization boundary, how to scope the S3 policy so one patient can&#8217;t enumerate another, why your download links expire earlier than you configured, what encryption actually gates, and what your audit trail looks like on the day someone asks for it. Everything here is mechanism, not a compliance checklist. Getting a Business Associate Agreement in place is a separate conversation with your lawyer and your AWS account team.</p>



<h2 class="wp-block-heading">The failure that never pages you</h2>



<p class="wp-block-paragraph">Here&#8217;s the shape of it. You write an IAM policy that lets a signed-in user read objects under their own prefix. Something like <code>arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*</code>. You test it. Alice gets her file. Alice cannot get Bob&#8217;s file. Ship it.</p>



<p class="wp-block-paragraph">Then, because the front end needs to render a file list, someone adds <code>s3:ListBucket</code>. And <code>s3:ListBucket</code> is not an object-level action. It applies to the bucket ARN, not the object ARN, so the <code>${cognito-identity.amazonaws.com:sub}</code> substitution in your resource string does nothing for it. Unless you add a separate condition on the request, a signed-in patient can list every key in the bucket.</p>



<p class="wp-block-paragraph">They still can&#8217;t download anything. Every <code>GetObject</code> outside their prefix returns a denial. But they can read the key names. And key names in a clinic bucket are rarely neutral. <code>patients/8f2a.../oncology-referral-2024.pdf</code> tells you something about a person even if the bytes stay locked.</p>



<p class="wp-block-paragraph">Nothing alerts on this. No error appears in any log you&#8217;re watching, because nothing errored. The listing succeeded. That&#8217;s the failure mode worth building around: in a document portal, the metadata leak is the one that doesn&#8217;t announce itself.</p>



<p class="wp-block-paragraph">The fix is a condition key on the list action, not a resource string:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOwnPrefixOnly",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::clinic-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "patients/${cognito-identity.amazonaws.com:sub}/*"
          ]
        }
      }
    },
    {
      "Sid": "ReadOwnObjectsOnly",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*"
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Two statements, two different jobs. The first constrains what the caller may ask the bucket to enumerate. The second constrains what bytes the caller may pull. Write only the second one and you&#8217;ve secured the download path while leaving the discovery path wide open.</p>



<p class="wp-block-paragraph">One trap in that variable. <code>${cognito-identity.amazonaws.com:sub}</code> resolves to the identity ID from the Cognito <em>identity pool</em>, not the <code>sub</code> claim from the user pool. They are different values for the same human. If you name your prefixes with the user pool sub because that&#8217;s what your application code has handy, the policy will match nothing and every request will be denied, which at least fails loudly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Decide where the authorization boundary lives</h2>



<p class="wp-block-paragraph">There are two honest architectures here and the choice shapes everything downstream. Pick deliberately.</p>



<h3 class="wp-block-heading">Pattern A: browser holds AWS credentials</h3>



<p class="wp-block-paragraph">The user signs in to a Cognito user pool. The application exchanges that token with a Cognito identity pool for temporary AWS credentials, and the browser calls S3 directly using the AWS SDK.</p>



<p class="wp-block-paragraph">It&#8217;s less code and there&#8217;s no server to run. The authorization boundary is entirely the IAM role policy attached to the identity pool. That policy is the whole security model, so it had better be right, and it&#8217;s harder to reason about than application code because the enforcement happens in a service you can&#8217;t step through with a debugger.</p>



<p class="wp-block-paragraph">The real limitation is expressiveness. IAM policy conditions do not know that Bob&#8217;s referral was superseded, that Alice revoked consent last Tuesday, or that a document should only be visible after a clinician releases it. Anything with business logic in it has to be encoded as prefix structure or object tags, and that gets brittle fast.</p>



<h3 class="wp-block-heading">Pattern B: backend brokers every object</h3>



<p class="wp-block-paragraph">The browser never touches AWS credentials. It sends the Cognito access token to your own API. The API validates the token, checks its own records for who may see what, and returns a presigned URL for that one object. S3 is reachable only by the API&#8217;s execution role.</p>



<p class="wp-block-paragraph">You now have a place to put logic, a place to log intent alongside access, and a single choke point to rate limit. The cost is that you own a service, its uptime, and its dependencies.</p>



<p class="wp-block-paragraph">For a small clinic build, this is the one I reach for first. Not because Pattern A is insecure, but because the questions clients ask in month three (&#8220;can we hide results until the doctor reviews them?&#8221;) are answerable in Pattern B and require a re-architecture in Pattern A. The identity-pool approach earns its keep when access really is a pure function of who you are, which in healthcare it rarely stays.</p>



<p class="wp-block-paragraph">If you do want attribute-driven access without a backend, Cognito identity pools can map claims from the token onto principal tags on the temporary session, which you then reference in policy conditions. It works, and it is the right tool for coarse distinctions like patient versus staff. It is not the right tool for per-document consent state.</p>



<h2 class="wp-block-heading">Your download links expire before you think they do</h2>



<p class="wp-block-paragraph">This one generates support tickets on week two. You set a presigned URL to last a few hours. Patients report links dying in under one. Nothing in your code changed.</p>



<p class="wp-block-paragraph">A presigned URL carries a signature made with whatever credentials signed it, and it cannot outlive those credentials. AWS documents the ceilings clearly: with long-lived IAM user keys and Signature Version 4 the maximum is seven days; from the S3 console the range is one minute to twelve hours; with temporary credentials from an assumed role, the URL dies when the role session does, regardless of the expiry you asked for. A default <code>AssumeRole</code> session is one hour.</p>



<p class="wp-block-paragraph">Your Lambda or container signing those URLs is using temporary credentials. So the effective lifetime is the shorter of your requested expiry and the remaining time on that role session, and the remaining time depends on when the runtime last refreshed. That&#8217;s the source of the flapping. You can watch it yourself:</p>



<pre class="wp-block-code"><code># Sign a URL and inspect what was actually encoded into it.
aws s3 presign s3://clinic-docs/patients/example/scan.pdf --expires-in 3600

# The returned URL contains X-Amz-Date, X-Amz-Expires and X-Amz-Credential.
# X-Amz-Expires is the window you asked for. If X-Amz-Credential shows an
# ASIA... key id, you signed with temporary credentials and the real ceiling
# is the session, not the window.</code></pre>



<p class="wp-block-paragraph">The honest resolution is to stop fighting it. Sign short, on demand, when the user clicks. Fifteen minutes is generous for a download the patient is standing in front of. Long links are not a convenience feature in this context anyway, because a presigned URL is a bearer credential: whoever holds the string can fetch the object, signed in or not. Forwarding the email forwards the access. If your portal mails links, you have moved the authorization boundary into someone&#8217;s inbox.</p>



<p class="wp-block-paragraph">Mail a notification that says a document is ready and link to the portal login. Never mail the signed URL itself.</p>



<h2 class="wp-block-heading">Encryption that actually gates something</h2>



<p class="wp-block-paragraph">New S3 objects are encrypted at rest by default with S3-managed keys. That satisfies the sentence &#8220;data is encrypted at rest&#8221; and gates precisely nothing, because anyone with <code>s3:GetObject</code> gets plaintext back. It is a storage property, not an access control.</p>



<p class="wp-block-paragraph">Server-side encryption with an AWS KMS customer managed key is different in kind. Reading an object now requires two independent grants: the S3 permission and <code>kms:Decrypt</code> on the key. Those live in different policies and can be owned by different people. A bucket policy mistake no longer hands over plaintext on its own, and revoking the key grant cuts off reads without touching a single S3 permission.</p>



<p class="wp-block-paragraph">You also get a second audit stream. Every decrypt is a KMS API call, recorded separately from the S3 call, which is useful precisely because an attacker who tampers with one trail probably didn&#8217;t think about the other.</p>



<p class="wp-block-paragraph">The costs are real and worth stating. KMS charges per request, so a portal that fetches many small objects will notice. S3 Bucket Keys reduce that request volume substantially by caching a bucket-level key, at the price of coarser detail in the KMS audit records. For a document portal with modest traffic and high sensitivity, I&#8217;d take the per-request cost and keep the granularity. For a thumbnail-heavy application, the opposite.</p>



<h2 class="wp-block-heading">The audit trail you don&#8217;t have yet</h2>



<p class="wp-block-paragraph">Assume someone will eventually ask: who downloaded this specific file, and when? If you have not explicitly turned something on, the answer is that you don&#8217;t know.</p>



<p class="wp-block-paragraph">CloudTrail is on for management events by default, and management events are control-plane operations: creating the bucket, changing its policy, attaching a role. Object-level reads and writes are <em>data events</em>, and data events are off by default because of volume and cost. <code>GetObject</code> on a patient scan is a data event. Unless you configured a trail with data events for that bucket, no record of the download exists.</p>



<pre class="wp-block-code"><code># Log S3 object-level activity for one bucket on an existing trail.
# Advanced event selectors let you scope to a single bucket prefix rather
# than every object in the account.
aws cloudtrail put-event-selectors 
  --trail-name clinic-audit-trail 
  --advanced-event-selectors '[
    {
      "Name": "PHI bucket object access",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Data"]},
        {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
        {"Field": "resources.ARN", "StartsWith":
          ["arn:aws:s3:::clinic-docs/patients/"]}
      ]
    }
  ]'</code></pre>



<p class="wp-block-paragraph">S3 server access logs are the other option and they are not equivalent. They record HTTP-level request detail and are delivered on a best-effort basis with no delivery guarantee, which makes them useful for traffic analysis and weak as evidence. CloudTrail data events carry caller identity, which is the field you actually need when the question is &#8220;who.&#8221;</p>



<p class="wp-block-paragraph">Two things to sort out while you&#8217;re in there. Send the trail to a bucket in a different account, or at minimum a bucket the portal&#8217;s roles cannot write to, so that compromising the application doesn&#8217;t compromise its own log. And put a lifecycle rule on the log bucket, because data event volume grows with usage and nobody budgets for it. If you already run centralized dashboards on something like Grafana Cloud or CloudWatch, wire an alert to unusual read volume per identity now rather than after the incident.</p>



<h2 class="wp-block-heading">Cutting off access when someone leaves</h2>



<p class="wp-block-paragraph">Disabling a Cognito user does not immediately end their session, and this surprises people. Tokens are self-contained. An access token already in a browser stays cryptographically valid until it expires, typically about an hour.</p>



<p class="wp-block-paragraph">Refresh tokens are the longer exposure. The default is thirty days, and app clients can be configured anywhere from sixty minutes to ten years. Ten-year refresh tokens on a portal holding medical records is not a hypothetical misconfiguration; it&#8217;s a checkbox someone ticked to stop users complaining about re-login.</p>



<p class="wp-block-paragraph">Cognito gives you revocation. Token revocation is enabled by default on newly created app clients. <code>AdminUserGlobalSignOut</code> revokes all of a user&#8217;s refresh, ID, and access tokens from the server side, and the <code>/oauth2/revoke</code> endpoint kills one refresh token and everything descended from it.</p>



<pre class="wp-block-code"><code># Server-side, authorized with IAM credentials. Revokes the user's refresh,
# ID and access tokens.
aws cognito-idp admin-user-global-sign-out 
  --user-pool-id us-east-1_EXAMPLE 
  --username patient-example

# Confirm revocation is switched on for the app client before you rely on it.
aws cognito-idp describe-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --query 'UserPoolClient.EnableTokenRevocation'</code></pre>



<p class="wp-block-paragraph">Now the part that catches teams out. AWS documents that a revoked token still passes verification in a generic JWT library that only checks signature and expiry. Revocation is enforced by Cognito, not baked into the token. If your API validates tokens by fetching the JWKS and calling a verify function, a revoked token sails straight through until it expires on its own.</p>



<p class="wp-block-paragraph">Which means short access token lifetimes are load-bearing. Revocation shortens the tail; it doesn&#8217;t close the window. Refresh token rotation, available on the Essentials and Plus feature tiers, replaces the refresh token on each exchange and narrows the window a stolen one is useful for. If you&#8217;re paying for Plus for threat protection features, you already have it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the usual suspects</h2>



<h3 class="wp-block-heading">Every request returns access denied after login</h3>



<p class="wp-block-paragraph">Nine times out of ten the prefix in your policy variable doesn&#8217;t match the prefix in your bucket. Print the identity ID your session actually resolved to and compare it character by character against a real key. Remember the identity pool ID and the user pool sub are different values.</p>



<h3 class="wp-block-heading">Signature does not match on a presigned URL</h3>



<p class="wp-block-paragraph">Usually something modified the request after signing. A client library adding a header, a proxy rewriting the path, or an email gateway URL-encoding the query string. Compare the <code>X-Amz-SignedHeaders</code> list in the URL against what the client actually sent.</p>



<h3 class="wp-block-heading">Uploads succeed but downloads fail</h3>



<p class="wp-block-paragraph">Check the KMS key policy separately from the IAM policy. A role can hold <code>kms:GenerateDataKey</code> (enough to write) without <code>kms:Decrypt</code> (needed to read). The S3 error will not say the word KMS prominently, which is why this one burns an afternoon.</p>



<h3 class="wp-block-heading">Browser uploads fail before any request reaches S3</h3>



<p class="wp-block-paragraph">That&#8217;s CORS. The bucket needs a CORS configuration allowing your portal&#8217;s origin, the methods you use, and any headers the SDK sends. Browser preflight failures never appear in S3 logs, so check the browser console first.</p>



<h3 class="wp-block-heading">Nothing appears in CloudTrail</h3>



<p class="wp-block-paragraph">Data events are not shown in CloudTrail Event History. Query the destination bucket directly, with Athena or CloudTrail Lake, and allow a few minutes for delivery.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Granting <code>s3:ListBucket</code> without an <code>s3:prefix</code> condition, so patients can enumerate every key even though downloads are blocked.</li>

<li>Putting identifying information in object keys or file names. Assume the key will leak eventually and use opaque identifiers.</li>

<li>Emailing presigned URLs. The link is the credential, and inboxes get forwarded, backed up, and breached.</li>

<li>Extending refresh token lifetime to silence complaints about re-login, instead of implementing rotation.</li>

<li>Assuming default encryption means access control. It means the disk is encrypted, nothing more.</li>

<li>Enabling versioning without a lifecycle rule, so deleted documents linger indefinitely as noncurrent versions you forgot exist.</li>

<li>Skipping CloudTrail data events to save money, then discovering the gap during an access review.</li>
</ul>



<h2 class="wp-block-heading">Best practices for a secure patient document portal</h2>



<ul class="wp-block-list">
<li>Keep one bucket for patient documents and nothing else. Mixed-purpose buckets make policies ambiguous and audits slow.</li>

<li>Turn on S3 Block Public Access at the account level, not just the bucket. Account level survives someone creating a new bucket in a hurry.</li>

<li>Enable versioning and set a noncurrent version expiration rule, so an overwritten document is recoverable but not permanent.</li>

<li>Use a KMS customer managed key so decrypt authority is separable from S3 authority.</li>

<li>Require MFA for staff accounts. TOTP costs nothing to enable and staff accounts are the ones with breadth.</li>

<li>Sign download URLs on demand with short expiry, and always from a backend that logged why it signed.</li>

<li>Put the portal behind a WAF and rate limiting. Cloudflare in front of the application, or AWS WAF on the distribution, both work; the point is that credential stuffing against a login page is the most likely thing to actually happen to you.</li>

<li>Restrict administrative access to the AWS console and to any bastion by network as well as identity, whether that&#8217;s IAM Identity Center with condition keys or a VPN such as NordVPN or Surfshark for a small team without central identity.</li>

<li>Write down the retention period before launch, then encode it as a lifecycle rule. Retention decided later is retention never decided.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Are S3 and Cognito enough for HIPAA compliance?</h3>



<p class="wp-block-paragraph">Both are covered under AWS&#8217;s HIPAA eligibility program, and Cognito&#8217;s documentation states it is HIPAA-BAA eligible. That makes them usable building blocks, not compliance. You need a signed Business Associate Agreement with AWS, and the configuration, access reviews, training, and documented policies remain yours. Service eligibility is a prerequisite, not an outcome.</p>



<h3 class="wp-block-heading">Should I use a Cognito identity pool or a backend API?</h3>



<p class="wp-block-paragraph">Use an identity pool when access is a pure function of identity and the rules will not change. Use a backend when documents have states, when a clinician gates release, or when you need to log intent alongside access. Most clinic portals become the second within a year.</p>



<h3 class="wp-block-heading">How long should a presigned URL last?</h3>



<p class="wp-block-paragraph">Minutes, not hours. Sign at click time. The maximum with long-lived credentials is seven days, but that ceiling is irrelevant here because the link is a bearer credential and a longer window is a longer leak.</p>



<h3 class="wp-block-heading">Can patients upload documents back to the portal?</h3>



<p class="wp-block-paragraph">Yes, using presigned PUT or presigned POST. Presigned POST is worth knowing about because its policy document can constrain content type and size before the object lands, which presigned PUT cannot. Either way, write uploads into a quarantine prefix, scan them, then move them.</p>



<h3 class="wp-block-heading">Does enabling CloudTrail data events cost much?</h3>



<p class="wp-block-paragraph">Data events are billed per event recorded with no free tier, unlike management events. For a portal serving a few thousand downloads a month the amount is small; for a bucket also serving application assets it is not. Scope your event selector to the patient prefix rather than the whole account and the question mostly goes away.</p>



<h3 class="wp-block-heading">What happens to sessions when I disable a user?</h3>



<p class="wp-block-paragraph">Nothing immediate. Existing access tokens remain valid until they expire. Call <code>AdminUserGlobalSignOut</code> to revoke tokens, and keep access token lifetime short so the residual window is measured in minutes.</p>



<h3 class="wp-block-heading">Do I need CloudFront in front of the bucket?</h3>



<p class="wp-block-paragraph">Not for correctness. It helps when patients are geographically spread, when you want signed cookies for a set of objects rather than one URL per object, or when you want a single edge to attach a WAF to. For a single-clinic portal, direct presigned URLs are simpler and one fewer thing to misconfigure.</p>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">A signed-in token proves who someone is. It does not decide what they may reach. In a <strong>secure patient document portal</strong> the boundary is the policy condition, the KMS grant, and the code that decides whether to sign a URL at all, and every one of those is something you have to write on purpose.</p>



<p class="wp-block-paragraph">The demo passes either way. The difference only shows up when someone reads a key name they shouldn&#8217;t have seen, and nothing in your system considers that an error.</p>



<p class="wp-block-paragraph">If you build one of these, spend an hour before launch on exactly two things: sign in as a test patient and try to list the bucket, then ask your logs who downloaded a specific file last week. If either answer is uncomfortable, you&#8217;ve found your next task.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your portal?</h2>



<p class="wp-block-paragraph">This is the kind of build where the gap between working and safe is invisible from the outside. I help small teams close it.</p>



<ul class="wp-block-list">
<li>Reviewing S3 bucket policies and Cognito identity pool roles for prefix isolation gaps, including the list-versus-get split most policies miss.</li>

<li>Designing the boundary: identity pool credentials in the browser versus a backend that brokers presigned URLs, with the trade-offs written down for your stakeholders.</li>

<li>Setting up KMS customer managed keys, key policies, and grant separation so decrypt authority is not implied by storage authority.</li>

<li>Turning on scoped CloudTrail data events, building the Athena tables to query them, and alerting on abnormal per-identity read volume.</li>

<li>Hardening the Cognito side: MFA, token lifetimes, refresh token rotation, and a revocation path that actually ends sessions.</li>

<li>Writing the whole thing as Terraform or OpenTofu so the next change is a reviewed pull request rather than a console click.</li>
</ul>



<p class="wp-block-paragraph">Send me your bucket policy and your identity pool role, with the account IDs redacted, and I&#8217;ll tell you what a signed-in patient can actually see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/secure-patient-document-portal-s3-cognito/">Building a Secure Patient Document Portal on S3 and Cognito</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Amazon Macie PII Detection: The Buckets It Never Opened</title>
		<link>https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 06 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Data Firehose]]></category>
		<category><![CDATA[Amazon Macie]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Classification]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[Security Hub]]></category>
		<category><![CDATA[Sensitive Data Discovery]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=405</guid>

					<description><![CDATA[<p>A Macie bucket labeled "Not sensitive" often just means Macie never read it. Extensionless objects, unsupported storage classes, unreachable KMS keys and quota truncation all produce silence that looks identical to a clean result. Here's how to measure coverage, fix the four gaps, tune identifiers, and keep the bill honest.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email came from legal, not from security. A customer had exercised a data access request, someone pulled the export by hand, and it contained email addresses and phone numbers sitting in a bucket Macie had labeled <em>Not sensitive</em>. Nobody had done anything wrong. The console was telling the truth as it understood it. It just didn&#8217;t understand very much about that bucket.</p>



<p class="wp-block-paragraph">That is the failure mode worth internalizing before anything else: Amazon Macie PII detection reports on what it managed to read, not on what is there. Objects it never opened produce no findings, and no findings looks identical to a clean result. The gap between &#8220;we scanned this&#8221; and &#8220;we found nothing&#8221; is where most Macie deployments quietly fail their first audit.</p>



<p class="wp-block-paragraph">This post covers where that gap comes from and how to close it: the eligibility chain every object goes through, the four distinct reasons an object gets skipped, how to tune identifiers so the findings are worth reading, what actually drives the bill, and how to route results somewhere a human will act on them.</p>



<h2 class="wp-block-heading">The eligibility chain behind Amazon Macie PII detection</h2>



<p class="wp-block-paragraph">Every object passes the same gates before a byte gets inspected. The order matters, because each gate fails differently and needs a different fix.</p>



<ol class="wp-block-list">
<li>Is it in a general purpose S3 bucket? Directory buckets are out of scope entirely.</li>



<li>Is the storage class supported? Standard, Standard-IA, One Zone-IA, Intelligent-Tiering, Glacier Instant Retrieval and Reduced Redundancy are in. Glacier Deep Archive and S3 Express One Zone are not.</li>



<li>Does the object key carry a recognized file extension? This is a string check on the name, not a content sniff.</li>



<li>Can Macie retrieve and decrypt it, given the bucket policy, object ACL and encryption key?</li>



<li>Does the content parse, and stay inside the per-file quotas?</li>



<li>Do the active data identifiers match anything in it?</li>
</ol>



<p class="wp-block-paragraph">Only the last gate produces a finding. Everything before it produces silence. Macie does record why each object was skipped, but that lives in coverage data and object samples, not on the findings page most people look at.</p>



<p class="wp-block-paragraph">One detail explains a lot of confusion: when you first enable automated sensitive data discovery, every bucket gets a sensitivity score of 50 and the label <em>Not yet analyzed</em>. A bucket whose permissions block Macie stays there permanently. It never turns red. It sits mid-list, looking unremarkable next to buckets that genuinely were analyzed.</p>



<h2 class="wp-block-heading">Gap one: objects Macie never opened</h2>



<p class="wp-block-paragraph">Macie calls these <em>unclassifiable</em>. They fail the storage class or extension check, so no retrieval is attempted. This is the largest source of false confidence I&#8217;ve seen, and the cheapest to fix.</p>



<h3 class="wp-block-heading">Missing file extensions</h3>



<p class="wp-block-paragraph">Classifiability comes from the file name extension. A file full of customer records named <code>part-00003-a4f9</code> is invisible. The same bytes named <code>part-00003-a4f9.json</code> get inspected and produce findings.</p>



<p class="wp-block-paragraph">This bites hardest on streaming ingestion. Amazon Data Firehose writes to S3 without appending an extension unless compression, format conversion, or the explicit file extension setting adds one. An uncompressed JSON stream lands as a tree of extensionless objects. Athena and Glue read them fine, because they infer format from the table definition. Macie skips every one.</p>



<p class="wp-block-paragraph">Fix it upstream: set the S3 file extension on the delivery configuration, or enable GZIP compression or Parquet conversion, both of which append a recognized extension. If you can&#8217;t change the producer, the fallback is a copy-and-rename step into a scan prefix, which costs you storage plus a pipeline to maintain. Fixing the producer is almost always cheaper.</p>



<h3 class="wp-block-heading">Unsupported storage classes</h3>



<p class="wp-block-paragraph">A lifecycle rule transitioning old exports to Glacier Deep Archive quietly removes them from scope. That may be the right storage decision, but it needs to be a conscious one, because archived customer data is still customer data when a regulator asks.</p>



<p class="wp-block-paragraph">Macie exposes these counts per bucket, split by cause. This is the first query I run against a new account:</p>



<pre class="wp-block-code"><code># Buckets where Macie can't classify objects, broken down by cause
aws macie2 describe-buckets 
  --query 'buckets[?unclassifiableObjectCount.total &gt; `0`].{
      bucket: bucketName,
      classifiable: classifiableObjectCount,
      skipped_total: unclassifiableObjectCount.total,
      skipped_extension: unclassifiableObjectCount.fileType,
      skipped_storage_class: unclassifiableObjectCount.storageClass
    }' 
  --output table</code></pre>



<p class="wp-block-paragraph">Read the ratio, not the raw number. Forty thousand classifiable objects and twelve skipped is fine. Three classifiable and four hundred thousand skipped is a reporting artifact pretending to be a scan result, and that bucket&#8217;s sensitivity score means nothing.</p>



<h2 class="wp-block-heading">Gap two: objects it opened and couldn&#8217;t read</h2>



<p class="wp-block-paragraph">These are classification errors rather than eligibility failures. Macie selected the object, tried to fetch it, and was refused. Three causes, and they need three different people to fix them.</p>



<ul class="wp-block-list">
<li><strong>Customer-provided keys (SSE-C).</strong> Macie cannot supply the key material, so it cannot retrieve the object. No permissions fix exists. Re-encrypt with S3 managed or KMS keys if you want coverage.</li>



<li><strong>KMS key policy.</strong> For customer managed keys, the key policy must allow the Macie service-linked role, <code>AWSServiceRoleForAmazonMacie</code>, to decrypt. Cross-account buckets need the key owner to grant it, not the bucket owner. AWS publishes a permission analyzer script in the <code>aws-samples/amazon-macie-scripts</code> repository that enumerates every key Macie needs and generates the CLI commands to fix them.</li>



<li><strong>Restrictive bucket policies.</strong> An explicit <code>Deny</code> conditioned on source IP or VPC endpoint blocks Macie along with everything else. The working pattern excludes the service-linked role ARN from the deny using the <code>aws:PrincipalArn</code> condition key.</li>
</ul>



<p class="wp-block-paragraph">To see which objects Macie actually touched in a bucket, pull the object samples. It&#8217;s the closest thing to a scan log you get:</p>



<pre class="wp-block-code"><code># Objects automated discovery selected, with per-object status
aws macie2 list-resource-profile-artifacts 
  --resource-arn arn:aws:s3:::your-bucket-name</code></pre>



<p class="wp-block-paragraph">Anything with a status of <code>SKIPPED</code> is a lead. Take the key, check its encryption settings in S3, and you usually have your answer inside a minute.</p>



<h2 class="wp-block-heading">Gap three: it read the file and nothing matched</h2>



<p class="wp-block-paragraph">Macie ships managed data identifiers covering common PII, financial data and credentials across many countries. Automated discovery uses a recommended subset by default rather than all of them. Sensible, but it is a default, and it will miss things that matter to you.</p>



<p class="wp-block-paragraph"><strong>Add the managed identifiers for your actual jurisdictions.</strong> If you hold records for customers in a country whose national ID identifier isn&#8217;t in the recommended set, you&#8217;re scanning for the wrong things with perfect efficiency. Pull the current list with <code>aws macie2 list-managed-data-identifiers</code> and compare it against where your customers live.</p>



<p class="wp-block-paragraph"><strong>Write custom identifiers for what only you know is sensitive.</strong> Internal account numbers, case references, employee IDs. A custom identifier is a regex plus optional keywords and a proximity rule, and the keywords are what save you. A bare <code>[0-9]{8}</code> matches timestamps, order totals and row counts, and you drown. The same regex with keywords and a match distance only fires when the number sits near a word that gives it meaning.</p>



<p class="wp-block-paragraph">Test the pattern before it goes near a job. This runs the criteria against sample text and returns match counts without creating anything:</p>



<pre class="wp-block-code"><code># Dry-run a custom identifier before creating it
aws macie2 test-custom-data-identifier 
  --regex 'ACC-[0-9]{4}-[0-9]{4}' 
  --keywords account customer acct 
  --maximum-match-distance 30 
  --sample-text 'customer account ACC-4821-9930 updated; total 1234-5678'</code></pre>



<p class="wp-block-paragraph">Run it against real samples, including the awkward ones, then against a file you know is clean and confirm zero. A pattern you only tested on positives is a pattern you haven&#8217;t tested.</p>



<p class="wp-block-paragraph">The mirror image is allow lists, which tell Macie to ignore specific text or patterns: your support inbox address, your published press contact, the seeded test records in everyone&#8217;s staging fixtures. Without them the same non-issues appear in every scan until the team stops reading findings. Allow lists accept predefined text or regex, and the per-account and per-job quotas are tight enough that you should curate rather than dump.</p>



<h2 class="wp-block-heading">Gap four: quota truncation inside large files</h2>



<p class="wp-block-paragraph">Subtle, because the object <em>is</em> analyzed. Just not completely, and a partial result reports like a full one.</p>



<ul class="wp-block-list">
<li>Per-file size quotas differ by format. Exceed the quota for a type and Macie analyzes none of that file, not part of it.</li>



<li>Archives have separate ceilings for nesting depth, extracted bytes and file count. If the metadata shows a breach up front, nothing is extracted. Cross a limit partway through and Macie stops, reporting only what it processed.</li>



<li>JSON and JSON Lines have a nested depth limit. Past it, the file is not analyzed at all.</li>



<li>Full names and mailing addresses cap out per file. After the cap Macie stops counting and stops reporting locations for that type, so the occurrence count is a floor, never a total.</li>
</ul>



<p class="wp-block-paragraph">Findings carry a status reason naming the limit that was hit, with values like <code>FILE_EXCEEDS_SIZE_LIMIT</code>, <code>ARCHIVE_NESTING_LEVEL_OVER_LIMIT</code> and <code>JSON_NESTING_LEVEL_OVER_LIMIT</code>. Filter on those before declaring a data lake of large compressed exports clean. Check current quota values in the Macie documentation rather than memorizing them, since they differ per format.</p>



<h2 class="wp-block-heading">Automated discovery or a discovery job?</h2>



<p class="wp-block-paragraph"><strong>Automated sensitive data discovery</strong> evaluates your inventory daily and samples representative objects across as many buckets as it can, grouping by metadata like prefix, extension and last-modified date. It is breadth-first and it does not re-analyze unchanged objects. Its job is to tell you which buckets deserve attention. It is not designed to prove a bucket is clean, and reading it that way is the root of most Macie misunderstandings.</p>



<p class="wp-block-paragraph"><strong>Sensitive data discovery jobs</strong> are the depth tool. You choose the buckets, scope by prefix, extension, size or object tag, choose the identifiers, and run once or on a schedule. This is what you point at the bucket automated discovery just flagged.</p>



<pre class="wp-block-code"><code># One-time job over a single prefix, recommended identifiers, 30% sampling
aws macie2 create-classification-job 
  --job-type ONE_TIME 
  --name "exports-prefix-review" 
  --managed-data-identifier-selector RECOMMENDED 
  --sampling-percentage 30 
  --s3-job-definition '{
    "bucketDefinitions": [
      {"accountId": "111122223333", "buckets": ["your-bucket-name"]}
    ],
    "scoping": {
      "includes": {
        "and": [
          {"simpleScopeTerm": {
             "comparator": "STARTS_WITH",
             "key": "OBJECT_KEY",
             "values": ["exports/"]
          }}
        ]
      }
    }
  }'</code></pre>



<p class="wp-block-paragraph">One detail about <code>--sampling-percentage</code> that trips people up: it selects a random percentage of eligible <em>objects</em> and then analyzes each selected object completely. It does not read a fraction of each file. For a bucket where every object comes from one pipeline, 20 or 30 percent tells you what you need. For heterogeneous uploads, sampling is a coin flip and you want the full pass.</p>



<h2 class="wp-block-heading">What actually drives the bill</h2>



<p class="wp-block-paragraph">Macie bills on three dimensions, and the one that surprises people is not the one they budget for.</p>



<ul class="wp-block-list">
<li><strong>Bucket evaluation.</strong> Per general purpose bucket monitored, prorated daily, up to the account ceiling. Predictable, driven by bucket count rather than data volume.</li>



<li><strong>Object monitoring.</strong> Per object tracked while automated discovery is enabled. This is inventory bookkeeping, not content reading, and it runs whether or not anything gets scanned that day. On an account with a log archive holding hundreds of millions of tiny objects, it can dwarf the other two.</li>



<li><strong>Data inspected.</strong> Per GB actually analyzed, covering both automated discovery and jobs.</li>
</ul>



<p class="wp-block-paragraph">So excluding buckets from automated discovery is a genuine cost lever, not just noise reduction. Log archives, CloudTrail destinations, build artifact stores and backup targets are usually safe exclusions. Exclude for the right reason though: a backup bucket holding database dumps of your customer table is exactly what you want scanned, however boring its name sounds.</p>



<p class="wp-block-paragraph">The three dimensions appear as distinct usage types in the Cost and Usage Report, so they split cleanly. A FinOps platform like Vantage or CloudZero, or just a Cost Explorer view grouped by usage type, tells you within a day whether object monitoring or inspection is driving the number. That changes the fix entirely: object monitoring is solved by scoping, inspection by sampling and scheduling.</p>



<h2 class="wp-block-heading">Getting findings somewhere a human sees them</h2>



<p class="wp-block-paragraph">Findings that only exist in the Macie console get read during onboarding week and never again. Two things are worth wiring up on day one.</p>



<p class="wp-block-paragraph"><strong>Publish findings outward.</strong> Macie emits findings to EventBridge and integrates with Security Hub. EventBridge earns its keep because you can filter on severity and finding type and route only what matters, whether that&#8217;s a Slack channel, a ticket, or a Lambda that tags the bucket. If you already centralize alerts in Grafana Cloud or similar, that path keeps sensitive data findings next to everything else on call sees.</p>



<p class="wp-block-paragraph"><strong>Configure the discovery results repository.</strong> Findings tell you what was found and where. The detailed discovery results, including analysis logs for objects where nothing was found, need an S3 bucket and a KMS key configured to persist. Nothing is retained long term until you set this up, and &#8220;we scanned it and found nothing&#8221; is exactly the record an auditor asks for.</p>



<p class="wp-block-paragraph">One operational note on the reveal feature, which retrieves sample occurrences so a human can confirm a finding is real. It needs a customer managed KMS key and it&#8217;s genuinely useful for triage. It also means someone is now looking at live customer PII, which belongs in your access model and audit trail. Same goes if an engineer downloads an object to investigate locally: that copy is unmanaged PII on a laptop and needs a secure deletion step, whether that&#8217;s a wipe tool from a vendor like O&amp;O Software or a documented and enforced process. Findings triage has a habit of creating the exposure it was meant to prevent.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting a bucket that reports nothing</h2>



<p class="wp-block-paragraph">When Amazon Macie PII detection reports nothing for a bucket you have doubts about, work down this in order. Each step rules out one gate, and stopping at the first hit saves you the rest.</p>



<ol class="wp-block-list">
<li><strong>Check the sensitivity label.</strong> <em>Not yet analyzed</em> at score 50 means nothing was analyzed. Do not read that as clean.</li>



<li><strong>Check the coverage page.</strong> Macie names the issue directly: access denied, invalid encryption, invalid KMS key, permission denied, or unclassifiable.</li>



<li><strong>Compare classifiable and unclassifiable counts</strong> with the <code>describe-buckets</code> query above.</li>



<li><strong>List a few object keys.</strong> Run <code>aws s3 ls</code> against a prefix and look for extensions. Extensionless keys explain the whole thing.</li>



<li><strong>Check the storage class.</strong> A lifecycle rule may have moved everything somewhere Macie doesn&#8217;t read.</li>



<li><strong>Check encryption.</strong> SSE-C is unfixable without re-encryption. A customer managed KMS key is fixable with a key policy change.</li>



<li><strong>Check object samples</strong> with <code>list-resource-profile-artifacts</code> to see exactly what was selected and skipped.</li>



<li><strong>Only now question the identifiers.</strong> Run a targeted job with a custom identifier for the data you expect and see whether it fires.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a low sensitivity score as evidence of no PII. It reflects what was found <em>and</em> how much was analyzed, so low coverage produces a low score.</li>



<li>Enabling Macie in one region and assuming account-wide coverage. Settings and results are per region, every time.</li>



<li>Running default identifiers in a business operating outside the jurisdictions the recommended set covers.</li>



<li>Building custom identifiers from regex alone, then abandoning Macie over the false positive volume.</li>



<li>Skipping the discovery results repository, then having no evidence trail when someone asks what was scanned and when.</li>



<li>Excluding buckets by name pattern without checking contents. Backup and export buckets are frequently the highest-risk ones you own.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Measure coverage before findings. A coverage report is the first deliverable of a Macie rollout, not an afterthought.</li>



<li>Fix extensions at the producer rather than building a rename pipeline downstream.</li>



<li>Run the KMS permission analyzer script once per account, then again whenever a new customer managed key appears.</li>



<li>Use automated discovery for breadth and targeted jobs for depth, letting the first choose targets for the second.</li>



<li>Test every custom identifier against both a positive and a negative sample before it reaches a job.</li>



<li>Curate allow lists early. The cost of a noisy first month is a team that stops reading findings.</li>



<li>Re-check coverage after any change to bucket policies, KMS keys, lifecycle rules or ingestion pipelines. All four silently remove data from scope.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Macie scan anything other than S3?</h3>



<p class="wp-block-paragraph">No. Macie analyzes objects in S3 general purpose buckets only. The documented workaround is bringing data to it: export RDS or Aurora snapshots to S3 in Parquet, or export a DynamoDB table to S3, then run a discovery job against the export. That works, but the export is now a second copy of your sensitive data with its own encryption and deletion requirements.</p>



<h3 class="wp-block-heading">Does Macie scan new objects as they land?</h3>



<p class="wp-block-paragraph">Not per object. Automated discovery evaluates inventory on a daily cycle and prioritizes objects that are new or recently changed, so fresh data moves up the queue rather than triggering an immediate scan. If you need scanning tied to arrival, drive a discovery job from S3 event notifications through EventBridge and accept that you own that orchestration.</p>



<h3 class="wp-block-heading">Why does a bucket I know has PII show a low sensitivity score?</h3>



<p class="wp-block-paragraph">Almost always coverage rather than detection. Check whether the objects are classifiable, whether Macie can decrypt them, and whether the label reads <em>Not yet analyzed</em>. If Macie genuinely read them and found nothing, the next suspect is the identifier set, and a targeted job with a custom identifier will tell you in one run.</p>



<h3 class="wp-block-heading">Do I need both automated discovery and discovery jobs?</h3>



<p class="wp-block-paragraph">For most teams, yes. Automated discovery gives you the map at predictable cost. Jobs give you proof for a specific bucket at a specific time, which is what compliance evidence actually requires. Running only jobs means you never discover the bucket nobody told you about.</p>



<h3 class="wp-block-heading">How do I cut false positives without missing real PII?</h3>



<p class="wp-block-paragraph">In order of preference: add keywords and a proximity rule so matches need context; use allow lists for specific known-benign values like your published contact details and test fixtures; and only then remove managed identifiers that are consistently wrong for your data. Removing identifiers is the bluntest option and the easiest to regret, so leave it last.</p>



<h3 class="wp-block-heading">Does Macie work across accounts in an organization?</h3>



<p class="wp-block-paragraph">Yes, through AWS Organizations with a delegated administrator. Design around this: the administrator&#8217;s automated discovery settings apply to member accounts, members can&#8217;t change them, and members see coverage and sensitivity data for their own buckets but not the sensitive data findings themselves. That shapes who can actually remediate what.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Amazon Macie PII detection is only as good as its coverage, and coverage fails silently by design. An object with the wrong extension, the wrong storage class, or an unreachable encryption key produces no finding, and no finding renders identically to a clean result. Before you hand a Macie report to anyone who will make a decision from it, put the coverage numbers next to it: how many objects were classifiable, how many were analyzed, how many were skipped and why.</p>



<p class="wp-block-paragraph">Do that once and the rest of the service becomes straightforward. Skip it and you&#8217;re maintaining an expensive dashboard reporting on whichever subset of your data happened to be readable.</p>



<h2 class="wp-block-heading">Need help getting Macie to actually see your data?</h2>



<p class="wp-block-paragraph">Most of the work in a Macie rollout isn&#8217;t turning it on. It&#8217;s the unglamorous part: proving what got scanned, fixing the reasons things didn&#8217;t, and making the output land somewhere a human acts on it. That&#8217;s the part I help with.</p>



<ul class="wp-block-list">
<li>Coverage audits across your S3 estate, with per-bucket classifiable, analyzed and skipped counts and the specific cause of each gap</li>



<li>Fixing those causes: KMS key policies, bucket policies blocking the service-linked role, lifecycle rules archiving data out of scope, pipelines writing extensionless objects</li>



<li>Custom data identifiers and allow lists tuned against your real data and tested on positive and negative samples first</li>



<li>Cost shaping: working out whether object monitoring or data inspection drives your bill, and scoping discovery so the number is defensible</li>



<li>Findings pipelines through EventBridge or Security Hub into the alerting and ticketing you already run, with severity filtering that keeps signal above noise</li>



<li>Multi-account setup under AWS Organizations, including the delegated administrator model and who can see and remediate what</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me the output of the <code>describe-buckets</code> query above, or a screenshot of your coverage page, and I&#8217;ll tell you what&#8217;s actually being scanned.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</title>
		<link>https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Comprehend]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=272</guid>

					<description><![CDATA[<p>A confident answer with a citation from a matter the reader was walled off from. Nothing crashed, nothing alerted. Building legal document intelligence on AWS with Textract, Bedrock and OpenSearch is the easy half; the hard half is tenant isolation, verified deletion, S3 Object Lock holds, what leaves the account, and an audit trail that names the actual user.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo was going well until someone asked where the third citation came from.</p>



<p class="wp-block-paragraph">The answer on screen was good. Fluent, specific, correctly hedged, three sources listed underneath it. Then a partner in the room asked a simple question: which matter is source three from? Nobody could answer it in the room, and when we went and looked, it was from a matter that half the people watching the demo were formally walled off from.</p>



<p class="wp-block-paragraph">Nothing had crashed. No alarm fired. The retrieval layer had done exactly what a similarity search does, which is return the nearest vectors, and the nearest vectors did not care about the ethical wall. That is the thing about building a legal document intelligence platform on AWS: the failures that matter almost never look like failures. They look like a confident answer with a citation attached.</p>



<p class="wp-block-paragraph">This post is about the parts of that build that are hard, organised around the five boundaries that actually have to hold. The extraction pipeline is the easy half. I will cover it, but quickly, because the AWS documentation is good and the failure modes are visible. The other four boundaries fail quietly, and that is where the engineering goes.</p>



<h2 class="wp-block-heading">The shape of the thing</h2>



<p class="wp-block-paragraph">Before the boundaries, the skeleton. A legal document intelligence platform on AWS almost always ends up looking like this, whether you plan it or arrive at it:</p>



<ul class="wp-block-list">
<li>Documents land in Amazon S3, one prefix per matter, versioning on.</li>



<li>An S3 event triggers AWS Lambda, which starts an asynchronous Amazon Textract job.</li>



<li>Textract output lands back in S3 as JSON. Something normalises it into text plus page and bounding-box coordinates.</li>



<li>Amazon Comprehend or a foundation model classifies the document and pulls entities: parties, dates, governing law, clause types.</li>



<li>Chunks get embedded and written to a vector store, usually Amazon OpenSearch Serverless behind Amazon Bedrock Knowledge Bases.</li>



<li>An application layer, typically API Gateway plus Lambda, takes a question, retrieves, and calls a model on Amazon Bedrock.</li>
</ul>



<p class="wp-block-paragraph">AWS Step Functions is worth reaching for once you have more than three stages, because Textract jobs are long-running and Lambda timeouts are not a retry strategy. That is the whole architecture. You can stand it up in a fortnight. Then you spend six months on everything below.</p>



<h2 class="wp-block-heading">Boundary one: isolation, and why metadata filtering is not optional</h2>



<p class="wp-block-paragraph">This is the one from the demo, and it is the one that ends engagements.</p>



<p class="wp-block-paragraph">A vector index has no native concept of a matter, a client, or an ethical wall. If you pool every document into one index and rely on the prompt to keep things separate, you have built a system where a well-phrased question can pull privileged material across a wall. Prompt instructions are not an access control. They are a suggestion that usually works.</p>



<p class="wp-block-paragraph">There are two shapes that do work, and the choice between them is a real trade-off rather than a best practice.</p>



<h3 class="wp-block-heading">Pooled index with server-side filtering</h3>



<p class="wp-block-paragraph">Every chunk carries a <code>matter_id</code> and a <code>client_id</code> as metadata. Every retrieval call attaches a filter. With an S3 data source in Bedrock Knowledge Bases, the metadata comes from a sidecar JSON file that sits next to the document and carries a <code>metadataAttributes</code> object.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "matter_id": "M-4417",
    "client_id": "C-108",
    "doc_type": "engagement_letter",
    "privileged": true
  }
}</code></pre>



<p class="wp-block-paragraph">The retrieval call then narrows the search before the model ever sees a chunk:</p>



<pre class="wp-block-code"><code>"retrievalConfiguration": {
  "vectorSearchConfiguration": {
    "filter": {
      "equals": { "key": "matter_id", "value": "M-4417" }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two rules make this safe, and both are the kind of thing that gets skipped under deadline. First, the filter value is derived on the server from the authenticated caller&#8217;s claims, never accepted from the client. If the browser can send you a <code>matter_id</code>, the browser can send you a different one. Second, the filter is applied by code that no feature request can bypass. A helper that builds every retrieval request, and a code review rule that no other path may call the retrieve API directly.</p>



<p class="wp-block-paragraph">The trap here is ordering. You cannot filter on an attribute the index does not have. Add <code>matter_id</code> after ingestion and every chunk already in the index is invisible to that filter, which means it either returns for everyone or for no one, depending on how your filter is written. Design the metadata schema before the first ingestion run, not after the first demo.</p>



<h3 class="wp-block-heading">Separate collection per client</h3>



<p class="wp-block-paragraph">The heavier option. A separate OpenSearch Serverless collection per client, which buys you a separate AWS KMS key per client, separate index settings, and a failure mode where a bug in the filtering logic cannot reach across clients at all.</p>



<p class="wp-block-paragraph">It costs you sprawl. Every collection is a resource to provision, monitor, patch policy on, and eventually delete. Ingestion jobs run per data source per knowledge base, so freshness guarantees fragment. For a firm with twelve institutional clients this is fine. For a platform onboarding a hundred small clients it becomes the main operational burden of the product.</p>



<p class="wp-block-paragraph">My default is pooled with strict server-side filtering, and per-client collections only where the client&#8217;s own contract demands a dedicated encryption key. If you cannot say out loud which of those two you are running, you are running neither properly.</p>



<h2 class="wp-block-heading">Boundary two: extraction, and the confidence score everyone ignores</h2>



<p class="wp-block-paragraph">Textract returns a confidence score between 0 and 100 with each prediction. AWS is explicit in its own best-practice guidance that applications sensitive to detection errors should enforce a minimum threshold and route anything below it for human scrutiny, and that the right threshold depends entirely on how the output gets used.</p>



<p class="wp-block-paragraph">Legal documents sit at the harsh end of that. A misread date on an archival scan is a curiosity. A misread date on a limitation period is a problem with a name on it.</p>



<p class="wp-block-paragraph">Practical notes from this layer:</p>



<ul class="wp-block-list">
<li>Use the asynchronous operations for anything multipage. <code>StartDocumentAnalysis</code> and <code>GetDocumentAnalysis</code> handle long PDFs and TIFFs; the synchronous <code>AnalyzeDocument</code> path is for single pages and will fight you on a 400-page bundle.</li>



<li>Textract&#8217;s Queries feature earns its place on structured instruments. Instead of extracting everything and grepping, you ask the document a direct question and get a scoped answer with its own confidence.</li>



<li>Keep the page number and bounding box for every chunk. When a lawyer asks where a sentence came from, &#8220;page 14, second column&#8221; is an answer. &#8220;It&#8217;s in the corpus somewhere&#8221; is not, and the platform loses credibility the first time you say it.</li>



<li>Amazon Augmented AI wires low-confidence predictions into a human review queue with a private workforce, so the reviewers are your people rather than an anonymous pool. For privileged material that distinction is the entire point.</li>
</ul>



<p class="wp-block-paragraph">What I would skip early: building a custom entity recognition model. Comprehend supports custom entity recognition and it is genuinely useful, but you need labelled data you do not have yet, and a foundation model with a decent prompt gets you far enough to find out which entities the users actually care about. Train the custom model once the requirement has stopped moving.</p>



<h2 class="wp-block-heading">Boundary three: retention, where deletion is a distributed problem</h2>



<p class="wp-block-paragraph">Here is the failure that will not show up in any test suite you write.</p>



<p class="wp-block-paragraph">A client asks for a document to be removed. Someone deletes the S3 object. The ticket closes. The document is still fully searchable, because its chunks are sitting in the vector index and will stay there until the data source is re-synced. Meanwhile the Textract JSON output is in a second bucket, the extracted text may be in a database, and if model invocation logging is on, whole passages of the document are sitting in CloudWatch Logs or another S3 bucket entirely.</p>



<p class="wp-block-paragraph">One delete, at least four places holding a copy. Write the deletion path as a real workflow with an assertion at the end, not as a single API call:</p>



<ol class="wp-block-list">
<li>Delete the source object and, if versioning is on, its versions.</li>



<li>Delete the derived artefacts: Textract JSON, normalised text, any thumbnails or page images.</li>



<li>Trigger an ingestion job so the knowledge base drops the orphaned chunks.</li>



<li>Confirm removal by querying the index for the document identifier and expecting nothing back.</li>



<li>Deal with the logs, which means either a retention policy short enough to make the problem expire or a deliberate decision, written down, that logs are out of scope.</li>
</ol>



<p class="wp-block-paragraph">Step four is the one people leave out, and it is the only step that actually proves anything.</p>



<h3 class="wp-block-heading">The opposite problem: things that must not be deleted</h3>



<p class="wp-block-paragraph">Legal work has the reverse requirement too. When a matter goes into litigation hold, the documents need to survive an administrator with delete permissions and a bad afternoon.</p>



<p class="wp-block-paragraph">S3 Object Lock is the mechanism, and it has two independent controls that people routinely conflate. A retention period protects an object version until a fixed date, in either governance mode, which privileged users can override, or compliance mode, which nobody can override and where the period cannot be shortened. A legal hold has no date at all. It stays until someone with <code>s3:PutObjectLegalHold</code> explicitly removes it. The two can be active at once, and while either is active the object version cannot be deleted or overwritten.</p>



<pre class="wp-block-code"><code># Place an indefinite hold on a specific object version
aws s3api put-object-legal-hold 
  --bucket matters-archive 
  --key M-4417/exhibit-c.pdf 
  --version-id 3sL7f2Qz9pXvB1kR 
  --legal-hold Status=ON</code></pre>



<p class="wp-block-paragraph">Two things bite here. Object Lock requires versioning and turns it on automatically, and once enabled on a bucket you cannot turn it off or suspend versioning again. And compliance mode is genuinely permanent: if you set a seven-year retention when you meant seven days, that is the answer, for everyone, including the account root. Test in governance mode first. The bypass path exists and is deliberately awkward, requiring both the <code>s3:BypassGovernanceRetention</code> permission and an explicit <code>x-amz-bypass-governance-retention:true</code> header on the request, which is exactly the level of friction you want on that operation.</p>



<p class="wp-block-paragraph">Design the hold model before you design the deletion model, because holds win. A deletion request that collides with an active hold is a legal question, not an engineering one, and the platform&#8217;s job is to surface the collision clearly rather than resolve it silently.</p>



<h2 class="wp-block-heading">Boundary four: disclosure, or what leaves the account</h2>



<p class="wp-block-paragraph">This is the boundary the client&#8217;s general counsel will ask about, usually in writing, usually before signature. Three separate controls, and they are not interchangeable.</p>



<h3 class="wp-block-heading">The AI services opt-out policy</h3>



<p class="wp-block-paragraph">AWS Organizations has a governance policy type that opts your accounts out of having content processed by certain AI services stored and used for service improvement. The list includes Amazon Textract and Amazon Comprehend, which are precisely the two doing the reading in this architecture. The policy type has to be enabled at the organisation root before you can attach anything:</p>



<pre class="wp-block-code"><code>aws organizations enable-policy-type 
  --root-id r-example 
  --policy-type AISERVICES_OPT_OUT_POLICY</code></pre>



<p class="wp-block-paragraph">Read the AWS documentation on this one carefully rather than taking my summary as gospel, because there is a caveat in it that matters: the services may still need to store your data operationally even when you have opted out of it being used for improvement. Opting out is not the same as the data never existing outside your account. Say that plainly in the client conversation. It is a much better position than being asked about it later.</p>



<h3 class="wp-block-heading">Bedrock&#8217;s data position, and the log that undoes it</h3>



<p class="wp-block-paragraph">Amazon Bedrock&#8217;s published position is that inputs and outputs are not shared with third-party model providers and are not used to train the base models, and that fine-tuning operates on a private copy. That is a strong starting point for privileged content and it is the main reason Bedrock rather than a direct provider API shows up in these builds.</p>



<p class="wp-block-paragraph">Then there is model invocation logging. It captures full prompt and response payloads to CloudWatch Logs or S3, and you want it on, because without it you cannot reconstruct what the system told someone. But understand what you have just built: a log that contains privileged document text, at a per-region setting, in a destination that probably has looser access controls than the document store it came from. Encrypt it with the same KMS key discipline, restrict it harder than you think you need to, and set a retention period on purpose.</p>



<p class="wp-block-paragraph">Worth separating in your head: CloudTrail records that an API call happened and who made it. Model invocation logging records what was in it. You need both, for different questions, and only one of them contains client confidences.</p>



<h3 class="wp-block-heading">Network path</h3>



<p class="wp-block-paragraph">Interface VPC endpoints via AWS PrivateLink keep traffic to Bedrock, Textract and the rest on the AWS network rather than out through an internet gateway or NAT. Add a gateway endpoint for S3 while you are there.</p>



<p class="wp-block-paragraph">Endpoint policies are the part that gets forgotten. An endpoint without a policy is a private path to the whole service, including into accounts that are not yours. Scope it to the operations and resources you actually use, and pair it with an S3 bucket policy that rejects requests not arriving through your endpoint. The first protects what leaves; the second protects what can be reached.</p>



<h3 class="wp-block-heading">The parts that are not on the architecture diagram</h3>



<p class="wp-block-paragraph">Two of these have caught people out badly, and neither is an AWS control.</p>



<p class="wp-block-paragraph">The first is the development environment. You do not want real client documents on a laptop or in a scratch account, so build the parsing and chunking code against synthetic documents on a small separate box. A cheap VPS from a provider like Contabo or InterServer is fine for this, and the separation is worth more than the convenience of iterating in the production account.</p>



<p class="wp-block-paragraph">The second is the reviewers. If your human review loop involves contractors working remotely, their network path and their disks are part of your boundary whether or not you drew them. A managed VPN such as NordVPN or Surfshark handles the network side, and when a matter closes and local copies have to go, a dedicated erasure tool from something like O&amp;O Software does what dragging a folder to the bin does not. Unglamorous, and it is the layer that gets audited.</p>



<h2 class="wp-block-heading">Boundary five: evidence, because someone will ask</h2>



<p class="wp-block-paragraph">At some point the question stops being &#8220;does it work&#8221; and becomes &#8220;who saw what, and when&#8221;. If you cannot answer that from stored records, the platform is not defensible regardless of how good the answers are.</p>



<ul class="wp-block-list">
<li>Turn on CloudTrail data events for the document buckets. They are off by default, they are billed separately from management events, and they are the only way to see individual object-level reads.</li>



<li>Log the retrieval, not just the generation. Store which chunks came back and which filter was applied. When somebody asks whether the wall held on a specific query, this record is the answer and there is no reconstructing it later.</li>



<li>Carry the end user&#8217;s identity through to the audit record. A Lambda execution role in the logs tells you the platform did something. It does not tell you who asked.</li>



<li>Alarm on the filter, not just on errors. A retrieval that ran without a matter filter should page someone, even though it returned HTTP 200 and a perfectly good answer.</li>
</ul>



<p class="wp-block-paragraph">That last one is the closest thing to a single takeaway in this post. The dangerous events in this architecture are successful ones.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the things that will go wrong</h2>



<h3 class="wp-block-heading">Retrieval returns nothing after you add a filter</h3>



<p class="wp-block-paragraph">Nearly always a metadata problem rather than a query problem. Either the sidecar metadata file was missing at ingestion, so the chunks carry no attribute to match, or the attribute was added after the chunks were indexed. Check whether the attribute exists on a known chunk before you touch the query. If it does not, you need a re-sync, not a different filter expression.</p>



<h3 class="wp-block-heading">Textract returns text but the layout is scrambled</h3>



<p class="wp-block-paragraph">Common on two-column pleadings and on documents with headers and footers on every page. Raw text detection reads in a reading order that is not always yours. Use the layout and table features rather than plain detection, and hold onto the geometry so you can reassemble columns yourself if the default order is wrong for that document class.</p>



<h3 class="wp-block-heading">The model cites a document that does not exist</h3>



<p class="wp-block-paragraph">Usually the citation is being generated rather than passed through. If document names are being written by the model instead of read from the retrieval response, it will invent plausible ones. Build citations from the retrieval result metadata in your application code and never from the generated text. Bedrock Guardrails helps with contextual grounding, but the structural fix is not asking the model to produce the reference in the first place.</p>



<h3 class="wp-block-heading">Ingestion is slow and nobody knows where</h3>



<p class="wp-block-paragraph">Instrument per stage before you optimise anything. In most of these pipelines the time is in Textract for large scanned bundles, and the fix is parallelism at the document level rather than tuning the extraction itself. Step Functions with a distributed map over documents gets you there without a queue you have to babysit.</p>



<h3 class="wp-block-heading">Costs climb faster than volume</h3>



<p class="wp-block-paragraph">Look at re-processing first. A pipeline that re-extracts a document every time anything downstream changes will quietly multiply your Textract bill against a static corpus. Key the extraction cache on the object&#8217;s content hash, not its path, and make re-ingestion an explicit action. The billing mechanisms differ per service, so read the current pricing pages rather than trusting a number from a blog post.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating the system prompt as an access control.</strong> It is a formatting instruction that happens to look like a rule.</li>



<li><strong>Taking the tenant identifier from the request body.</strong> If the client can name the matter, the client can name someone else&#8217;s.</li>



<li><strong>Designing the metadata schema after the first ingestion run.</strong> Retrofitting a filterable attribute means re-indexing the whole corpus.</li>



<li><strong>Assuming a deleted S3 object is gone from the platform.</strong> The chunks, the extraction output and the invocation logs all outlive it.</li>



<li><strong>Setting compliance-mode retention without testing in governance mode.</strong> There is no support ticket that fixes a seven-year mistake.</li>



<li><strong>Enabling model invocation logging without treating the log as privileged.</strong> You have just made a second copy of the documents in a less protected place.</li>



<li><strong>Building VPC endpoints and leaving the default policy on them.</strong> A private path with no policy is still a path.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>One retrieval helper, server-side filter injection, and a review rule that no other code path calls the retrieve API directly.</li>



<li>Metadata schema agreed and frozen before the first production ingestion. Include the fields you might filter on later, even if unused today.</li>



<li>Page-level provenance on every chunk, surfaced in the interface. It builds trust faster than any accuracy improvement.</li>



<li>A confidence threshold chosen per document class, with a human review path for everything under it.</li>



<li>Deletion implemented as a verified workflow across every store, ending in an assertion that the content is actually unretrievable.</li>



<li>Object Lock legal holds for matters under litigation hold, tested in governance mode before anything runs in compliance mode.</li>



<li>Customer-managed KMS keys across S3, the vector store and the logs, so key access is a control you can actually revoke.</li>



<li>An alarm on unfiltered retrievals, because the failure mode is a success response.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a legal document intelligence platform on AWS safe for privileged material?</h3>



<p class="wp-block-paragraph">The services support it. Bedrock&#8217;s stated position is that inputs and outputs are not shared with model providers or used to train base models, PrivateLink keeps traffic off the public internet, and KMS covers encryption at rest. What determines safety is your own configuration: tenant isolation, log handling and deletion behaviour. The platform is exactly as confidential as its weakest copy of the text.</p>



<h3 class="wp-block-heading">Should I use Bedrock Knowledge Bases or build retrieval myself?</h3>



<p class="wp-block-paragraph">Start with Knowledge Bases. It handles chunking, embedding, sync and metadata filtering, and those are weeks of work with no differentiation in them. Build your own when you need retrieval behaviour it does not expose, such as unusual re-ranking or per-tenant chunking strategies. Going custom on day one usually means reimplementing the managed service badly.</p>



<h3 class="wp-block-heading">How do I stop one client&#8217;s documents surfacing in another client&#8217;s answers?</h3>



<p class="wp-block-paragraph">Tag every chunk with a tenant identifier at ingestion, and inject the matching filter server-side on every retrieval from the authenticated caller&#8217;s identity. Never accept the identifier from the client. If the contract requires per-client encryption keys, use a separate collection per client instead, accepting the extra operational load.</p>



<h3 class="wp-block-heading">Does deleting a document from S3 remove it from the search index?</h3>



<p class="wp-block-paragraph">No. The chunks stay in the vector store until an ingestion job runs and reconciles the data source. Until then the document is deleted and still fully searchable, which is the worst combination available. Always finish a deletion by querying the index and confirming nothing comes back.</p>



<h3 class="wp-block-heading">What is the difference between an S3 legal hold and a retention period?</h3>



<p class="wp-block-paragraph">A retention period runs until a fixed date and comes in governance mode, which privileged users can override, or compliance mode, which nobody can. A legal hold has no end date and stays until someone explicitly removes it. They are independent, they can both be active on the same object version, and while either is active the object cannot be deleted or overwritten.</p>



<h3 class="wp-block-heading">Do I need human review, or is the model accurate enough?</h3>



<p class="wp-block-paragraph">Accuracy is the wrong frame. The question is what happens to a wrong answer downstream. Where output feeds a decision with consequences, you want a confidence threshold and a review queue, and Amazon Augmented AI with a private workforce gives you that without building the review tooling yourself. Where output is a search aid a person will verify anyway, review is friction you do not need.</p>



<h3 class="wp-block-heading">Can I run this in a single AWS account?</h3>



<p class="wp-block-paragraph">Technically yes, and for a pilot it is reasonable. It gets uncomfortable once you have production documents and a development environment in the same place, because an IAM mistake has nowhere to stop. Separate accounts under Organizations also gives you the policy layer, including the AI services opt-out policy, which only exists at the organisation level.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A legal document intelligence platform on AWS is not hard to build. Textract reads, Bedrock reasons, OpenSearch retrieves, and the tutorial version works on the first afternoon.</p>



<p class="wp-block-paragraph">What is hard is that its failures are silent and well-formed. The cross-matter citation, the deleted document that still answers questions, the privileged passage sitting in a log bucket, the retrieval that ran without a filter and returned a beautiful paragraph. None of these throw an error. All of them are the kind of thing that ends a client relationship.</p>



<p class="wp-block-paragraph">So build the boundaries first and the features second, and make sure every one of them is something you can prove from a stored record rather than something you believe about the code. If you can only take one habit from this: alarm on the successful requests that should not have been possible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your document pipeline?</h2>



<p class="wp-block-paragraph">I work on AWS document processing and retrieval systems where the confidentiality requirements are real. Typical things I get called in for:</p>



<ul class="wp-block-list">
<li>Reviewing tenant isolation in an existing RAG setup and finding the paths where the filter can be bypassed.</li>



<li>Designing the metadata and chunking schema before ingestion, so filtering works without a re-index later.</li>



<li>Building Textract pipelines with confidence thresholds, human review routing and page-level provenance.</li>



<li>Implementing verified deletion across S3, derived artefacts, the vector index and logs, with a proof step at the end.</li>



<li>Setting up retention and legal hold with S3 Object Lock, including the governance-mode rehearsal before compliance mode.</li>



<li>Locking down the network and logging path: VPC endpoints with real policies, KMS key separation, and audit records that name the actual user.</li>
</ul>



<p class="wp-block-paragraph">If you have an architecture diagram, a Step Functions definition or a retrieval request you are unsure about, send it over and I will tell you what I would change and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</title>
		<link>https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 25 Jul 2026 12:04:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Config]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[Healthcare Cloud]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[Restore Testing]]></category>
		<category><![CDATA[VPC]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=184</guid>

					<description><![CDATA[<p>A working engineer's guide to HIPAA compliance on AWS, organised by the gap between the control you configured and the obligation you actually carry. Covers BAA account scope, the eligible services list as a contract boundary, KMS key policy versus the encryption checkbox, what "six years" really applies to, backup and restore scope, and the subprocessor chain nobody inventories.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/">HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket usually reads something like: &#8220;Legal wants to know if the analytics account is in scope.&#8221; So you open Security Hub. Green. You check the Config rules. Passing. Every bucket is encrypted, every volume is encrypted, MFA is on, CloudTrail is running in all Regions. You reply that the account is fine.</p>



<p class="wp-block-paragraph">Then someone points out that a nightly job has been copying a de-identified extract into that account for eight months, the de-identification script never removed the admission dates, and the account was spun up before anyone thought about the Business Associate Addendum. Nothing was misconfigured. Every control you built worked exactly as designed. And you have been out of compliance the entire time.</p>



<p class="wp-block-paragraph">That is the shape of most real failures here. Not a breach, not a misconfiguration, but a mismatch between the boundary your tooling checks and the boundary your obligation actually follows. This post covers HIPAA compliance on AWS organised by those gaps: where the contract stops, where encryption stops being a control, what &#8220;six years&#8221; genuinely applies to, and which parts of the estate people forget are in scope at all.</p>



<h2 class="wp-block-heading">Eligible is not compliant, and the difference is the whole job</h2>



<p class="wp-block-paragraph">AWS does not sell HIPAA compliance. It sells HIPAA <em>eligible</em> services, which is a genuinely different thing. Eligible means AWS has built the service so it can lawfully handle electronic protected health information and has agreed to cover it under a Business Associate Addendum. Compliant describes an entire system: your architecture, your key management, your access reviews, your policies, your staff, your vendors.</p>



<p class="wp-block-paragraph">Under the shared responsibility model, AWS secures the infrastructure. You secure everything you build on it. Nothing about signing the BAA transfers a single obligation off your side of the line. An unencrypted RDS instance, an overly broad IAM policy or an application that logs a patient identifier into stdout is your problem in exactly the same way it would be in a rack you own.</p>



<p class="wp-block-paragraph">People know this in the abstract. Where it bites is in the specifics below.</p>



<h2 class="wp-block-heading">Gap one: the BAA is a contract boundary, and nothing enforces it</h2>



<p class="wp-block-paragraph">This is the one I would fix first, because it is invisible to every security tool you own.</p>



<p class="wp-block-paragraph">The AWS BAA is self-service through AWS Artifact, at no extra cost. You can accept it for a single account, or, if you are in the management account of an AWS Organization, accept it once so that existing and future member accounts are covered. That organization-level option is the one worth using, because the per-account version quietly rots: someone creates a new account for a proof of concept, nobody repeats the Artifact step, and six months later that account is running something real.</p>



<p class="wp-block-paragraph">The second half of the boundary is the HIPAA Eligible Services Reference that AWS publishes. Only services on that list may create, receive, process, maintain or transmit ePHI under the BAA. The list is long, it changes, and some entries carry carve-outs where the service is eligible but a specific feature is not. Reading a service name on the list and assuming every feature inside it is covered is the kind of mistake that only surfaces during an audit.</p>



<p class="wp-block-paragraph">Here is the part worth internalising: <strong>there is no AWS control that stops you putting PHI into a non-eligible service.</strong> No API error, no Config rule out of the box, no GuardDuty finding. The eligible services list is a contractual construct. Your infrastructure has no idea it exists.</p>



<h3 class="wp-block-heading">Turning a contract boundary into a technical one</h3>



<p class="wp-block-paragraph">The mechanism that actually helps is Service Control Policies on the organizational unit that holds your PHI accounts. SCPs set the ceiling on what any principal in those accounts can do, including the root user, so they work as a guardrail rather than a suggestion.</p>



<p class="wp-block-paragraph">Start with the easy one. Pin the accounts to the Regions you have actually assessed, because data residency assumptions fall apart the moment someone launches something in a Region you never reviewed:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnapprovedRegions",
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "organizations:*",
        "route53:*",
        "cloudfront:*",
        "support:*",
        "sts:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The <code>NotAction</code> list matters. Global services are backed by endpoints in specific Regions, so denying them wholesale by Region locks you out of IAM and breaks Route 53 and CloudFront. Those entries are exemptions, not an allow-list.</p>



<p class="wp-block-paragraph">The harder one is restricting which services can be used at all. The same <code>NotAction</code> pattern works, with the services you have approved for PHI listed as the exemptions and everything else denied. It is effective and it is blunt: every new service anyone wants becomes a change request against the policy, and if you forget a dependency you find out through a failure in production. I would only reach for it on a dedicated PHI OU where the workload is well understood, not across a general-purpose organization.</p>



<p class="wp-block-paragraph">Whichever route you take, write down the approved service list somewhere a human reviews on a schedule, and diff it against the AWS reference periodically. That review is itself a compliance artefact.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap two: encryption is a checkbox, the key policy is the control</h2>



<p class="wp-block-paragraph">Almost every guide to HIPAA compliance on AWS tells you to encrypt at rest and in transit. Almost none of them explain why it is worth doing properly rather than minimally, so teams enable default encryption with an AWS-managed key, watch the Config rule turn green, and move on.</p>



<p class="wp-block-paragraph">The reason to care is the Breach Notification Rule. It applies to <em>unsecured</em> PHI, meaning PHI that has not been rendered unusable, unreadable or indecipherable through a method HHS has specified. HHS guidance points at NIST-validated encryption. If PHI is encrypted to that standard and the decryption keys were not compromised alongside it, an incident involving that data generally does not trigger the notification machinery at all. No individual letters, no HHS portal submission, no press release for a large incident.</p>



<p class="wp-block-paragraph">Read that second condition again, because it is where the architecture decision lives. The safe harbour depends on the keys not being compromised with the data. If your encryption key is one an attacker inherits automatically the moment they compromise a role in the account, you have encryption but you may not have the argument.</p>



<h3 class="wp-block-heading">What that means in practice</h3>



<ul class="wp-block-list">
<li>Use customer managed KMS keys for anything holding PHI, not AWS-managed keys. Only a customer managed key gives you a key policy you can write, and only a key policy lets you deny decryption independently of the resource policy.</li>

<li>Separate the key administrators from the key users. The people who can schedule deletion of a key should not be the people whose application role uses it every second.</li>

<li>Use a distinct key per data domain rather than one key for the whole account. Blast radius and audit trail both improve, and you get the ability to revoke access to one dataset without touching another.</li>

<li>Constrain key usage with the <code>kms:ViaService</code> condition so a key that exists to encrypt RDS storage cannot be used to decrypt something a role dragged into Lambda.</li>

<li>Turn on key rotation and leave it on. It costs nothing operationally and it is the kind of thing an assessor asks about by reflex.</li>
</ul>



<p class="wp-block-paragraph">Pull the current key policy before you assume it says what you think:</p>



<pre class="wp-block-code"><code>aws kms get-key-policy 
  --key-id alias/phi-rds 
  --policy-name default 
  --output text

# Find storage that slipped through unencrypted
aws ec2 describe-volumes 
  --filters Name=encrypted,Values=false 
  --query 'Volumes[].{Id:VolumeId,AZ:AvailabilityZone}' 
  --output table

aws rds describe-db-instances 
  --query 'DBInstances[?StorageEncrypted==`false`].DBInstanceIdentifier' 
  --output text</code></pre>



<p class="wp-block-paragraph">The RDS query is the important one, because RDS encryption cannot be enabled in place. If that command returns anything, the fix is a snapshot, an encrypted copy of the snapshot, a restore, and a cutover. Plan for downtime or a replication strategy. This is the single most common &#8220;we will fix it later&#8221; item I see, and later gets expensive.</p>



<p class="wp-block-paragraph">Also switch on EBS encryption by default in every Region you use, so the next instance somebody launches from a console wizard is not a new exception:</p>



<pre class="wp-block-code"><code>aws ec2 enable-ebs-encryption-by-default --region us-east-1
aws ec2 get-ebs-encryption-by-default --region us-east-1</code></pre>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap three: you have logs, but you may not have evidence</h2>



<p class="wp-block-paragraph">The Security Rule requires audit controls: mechanisms that record and examine activity in systems containing ePHI. It also requires you to regularly review records of information system activity. Both of those are about having and using the records.</p>



<p class="wp-block-paragraph">Now the correction, because this one is repeated everywhere and it is wrong in a way that costs money. You will read that HIPAA requires six years of audit logs. It does not. The six-year requirement sits in the documentation standard, and it applies to the policies, procedures and records of actions, activities and assessments that the Security Rule requires you to keep, retained for six years from creation or from when the document was last in effect, whichever is later. There is no clause anywhere in the Security Rule that names a retention period for CloudTrail events.</p>



<p class="wp-block-paragraph">What this actually means is more demanding, not less. You have to <em>decide</em> your log retention period, write it into a policy, justify it against your risk analysis, and then keep that policy for six years. And an assessor will hold you to the number you wrote. Setting a CloudWatch Logs retention of thirty days while your policy claims one year is a finding. Storing seven years of everything because a blog told you to, when your policy says two, is not compliance, it is just a bill.</p>



<p class="wp-block-paragraph">So: pick a period you can defend, make the infrastructure match it exactly, and treat any gap between policy and configuration as a defect.</p>



<h3 class="wp-block-heading">Making logs into evidence</h3>



<p class="wp-block-paragraph">Retention is only half of it. The other half is being able to show that the records were not altered. CloudTrail has log file validation for exactly this, and it is off unless you turn it on:</p>



<pre class="wp-block-code"><code>aws cloudtrail update-trail 
  --name org-phi-trail 
  --enable-log-file-validation

# Later, prove a window of logs is intact
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:us-east-1:111122223333:trail/org-phi-trail 
  --start-time "$(date -u -d '90 days ago' +%Y-%m-%dT%H:%M:%SZ)"</code></pre>



<p class="wp-block-paragraph">With validation enabled, CloudTrail writes signed digest files alongside the log files, and <code>validate-logs</code> checks them. The difference between &#8220;here are our logs&#8221; and &#8220;here are our logs, and here is a cryptographic check that nothing was modified or deleted&#8221; is the difference between an assertion and evidence.</p>



<p class="wp-block-paragraph">Put the archive bucket in a separate account that the workload accounts cannot write to or delete from, and apply S3 Object Lock in compliance mode for the retention window you committed to. Object Lock in compliance mode cannot be shortened or bypassed by anyone, including the root user, which is exactly the property you want and exactly the property that will hurt if you set the period carelessly. Test it in governance mode first.</p>



<p class="wp-block-paragraph">For the review obligation, a query interface matters more than raw storage. Athena over the CloudTrail bucket is the cheap default. If you want alerting and dashboards on top of access patterns, this is a natural place for a platform such as Grafana, Datadog or Splunk, and any of them will hold access records for you. Just remember that if those records contain PHI, that vendor needs a BAA too. See the subprocessor section below.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap four: backups, snapshots and the parts of scope people forget</h2>



<p class="wp-block-paragraph">The Security Rule&#8217;s contingency plan standard is not optional decoration. It requires a data backup plan, a disaster recovery plan and an emergency mode operation plan, plus testing and revision procedures. Most teams have the backups. Far fewer have the tested restore, and the tested restore is the part that gets asked about.</p>



<p class="wp-block-paragraph">Three things routinely go wrong here.</p>



<ol class="wp-block-list">
<li><strong>Copies leave the boundary.</strong> A cross-Region snapshot copy lands in a Region you did not assess. A cross-account copy for the DR account lands somewhere outside the OU your SCPs protect. The data is still PHI. The controls did not travel with it.</li>

<li><strong>Re-encryption changes the key, not just the copy.</strong> Copying an encrypted snapshot to another account requires a key the destination can use. It is easy to end up with a shared or less restrictive key protecting your backups than protects production, which inverts the risk model.</li>

<li><strong>The restore is never rehearsed.</strong> A backup you have never restored is a hypothesis. Schedule a restore into an isolated account, record the elapsed time, and file the result. That record is your evidence for the testing requirement, and it is the single easiest compliance artefact to produce for free.</li>
</ol>



<p class="wp-block-paragraph">While you are inventorying, remember the places PHI ends up without anyone deciding it should: application logs that include request bodies, database slow query logs capturing parameter values, support tickets with screenshots attached, CSV extracts in an analyst&#8217;s bucket, and non-production environments seeded from a production dump. That last one is the classic. If your staging database is a copy of production, staging is in scope, and staging is almost never built to the same standard.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Gap five: the business associate chain does not stop at AWS</h2>



<p class="wp-block-paragraph">Your BAA with AWS covers AWS. It covers nothing else in your stack.</p>



<p class="wp-block-paragraph">Every vendor that can create, receive, maintain or transmit PHI on your behalf is a business associate and needs an agreement. In a typical AWS estate that means the error tracker holding stack traces, the log aggregation platform, the APM tool, the transactional email provider, the customer support desk, the CI system if it ever touches a production dataset, and any AI or analytics service you have wired in.</p>



<p class="wp-block-paragraph">Build the inventory as a table with three columns: vendor, what PHI it can see, and whether a signed agreement exists. The third column is usually where the surprises are. Some vendors sign readily, some only on higher-priced tiers, and some decline entirely, at which point you have an architecture decision rather than a procurement one.</p>



<p class="wp-block-paragraph">One structural move that reduces this surface considerably: keep everything that does not need PHI out of the PHI accounts entirely. Your marketing site, your docs, your status page and your public API gateway for non-clinical traffic do not belong in a regulated account. Running them on ordinary infrastructure, whether that is a separate AWS account, a straightforward VPS from a host like InterServer, or a static site behind Cloudflare, shrinks the estate you have to assess, evidence and defend. Fewer things in scope is the cheapest compliance win available.</p>



<p class="wp-block-paragraph">For tracking the paperwork side, compliance automation platforms such as Vanta, Drata or Secureframe pull evidence from AWS on a schedule and keep the vendor register current. They are genuinely useful for the collection and reminder burden. They do not design your architecture, and I have seen teams treat a green dashboard in one of those tools as though it were an assessment. It is not. It is a checklist that knows what you told it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What is changing, and why &#8220;addressable&#8221; is a bad thing to build on</h2>



<p class="wp-block-paragraph">Since it was adopted, the Security Rule has split implementation specifications into <em>required</em> and <em>addressable</em>. Addressable never meant optional. It meant you assess whether the specification is reasonable and appropriate, and if not, you implement an equivalent alternative or document why neither is necessary. In practice, a lot of organisations turned the documented justification into the deliverable and skipped the control.</p>



<p class="wp-block-paragraph">HHS published a Notice of Proposed Rulemaking in the Federal Register in January 2025 that would remove that distinction, making implementation specifications required with limited exceptions, and would explicitly require encryption of ePHI at rest and in transit and multi-factor authentication, again with limited exceptions. The comment period closed in March 2025.</p>



<p class="wp-block-paragraph">Be precise about the status, because a lot of vendor content is not: <strong>this is a proposed rule and it is not final.</strong> The expected timeline for final action has slipped more than once, and the requirements could still change or be withdrawn. Nobody should be telling you a compliance deadline as though it were settled.</p>



<p class="wp-block-paragraph">What is worth taking from it is the direction of travel. If your current position depends on having documented that encryption or MFA was not reasonable and appropriate, that position is fragile regardless of what the final rule says. On AWS specifically, encryption at rest and MFA are both cheap and both already best practice. Building the architecture on an addressable deferral is an unforced risk.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the findings you will actually hit</h2>



<h3 class="wp-block-heading">&#8220;An assessor asked which accounts are in BAA scope and nobody could answer&#8221;</h3>



<p class="wp-block-paragraph">Sign in to AWS Artifact from the management account and check the organization agreements tab to see whether the BAA was accepted at the organization level or per account. If it is per account, list your accounts, work out which hold PHI, and confirm each one individually. Then move to the organization-level agreement so this question has one answer forever.</p>



<h3 class="wp-block-heading">&#8220;Config says the bucket is encrypted but we cannot prove who read the objects&#8221;</h3>



<p class="wp-block-paragraph">Bucket encryption and object-level access logging are unrelated. CloudTrail management events do not record S3 object reads by default. You need CloudTrail data events for that bucket, or S3 server access logging, and both cost money proportional to request volume. Enable data events selectively on the buckets that hold PHI rather than account-wide.</p>



<h3 class="wp-block-heading">&#8220;We enabled an SCP and production broke&#8221;</h3>



<p class="wp-block-paragraph">Almost always a Region deny catching a global service endpoint, or a service allow-list missing a dependency the workload calls indirectly. Check CloudTrail for <code>AccessDenied</code> events with an explicit deny from an SCP, and look at the service name in the event rather than the one you expected. Attach new SCPs to a test OU with a representative workload before the PHI OU.</p>



<h3 class="wp-block-heading">&#8220;Snapshot copy to the DR account fails with a KMS error&#8221;</h3>



<p class="wp-block-paragraph">The destination account cannot use the source key. The source key policy has to grant the destination principal permission to use it, and the copy has to specify a key the destination can decrypt with. Fix it by granting explicitly on a key you control, not by falling back to an AWS-managed key, which is the tempting shortcut and gives up the key policy control you needed.</p>



<h3 class="wp-block-heading">&#8220;CloudWatch Logs retention was never set&#8221;</h3>



<p class="wp-block-paragraph">New log groups default to never expiring, which is both a cost problem and a policy mismatch. Audit them with <code>aws logs describe-log-groups</code> and look for groups with no <code>retentionInDays</code> value, then set the period your policy specifies with <code>aws logs put-retention-policy</code>.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the signed BAA as the finish line rather than the prerequisite. It is the thing you need before the first byte of PHI arrives, not evidence that anything is configured correctly.</li>

<li>Assuming a service is fully eligible because its name appears on the list, without reading the feature-level carve-outs next to it.</li>

<li>Quoting &#8220;six years&#8221; as a log retention requirement, then either overspending on storage or writing a policy that contradicts the actual configuration.</li>

<li>Using AWS-managed KMS keys for PHI, which leaves no key policy to write and no independent revocation path.</li>

<li>Seeding staging or test environments from production data and then holding those environments to a lower standard.</li>

<li>Forgetting that the risk analysis is a required, recurring, documented activity, not a one-off spreadsheet from the year you launched.</li>

<li>Signing a BAA with AWS and none of the ten other vendors that can see the same data.</li>

<li>Letting a compliance automation dashboard stand in for an architecture review.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li><strong>Isolate PHI in its own accounts and its own OU.</strong> Account boundaries are the strongest isolation AWS offers, and they make the scope question answerable in one sentence.</li>

<li><strong>Accept the BAA at the organization level.</strong> It removes an ongoing manual step that fails silently.</li>

<li><strong>Customer managed keys, one per data domain, with split admin and usage roles.</strong> This is what makes the breach safe harbour argument defensible rather than theoretical.</li>

<li><strong>Write the retention period down first, configure second.</strong> Policy and infrastructure should agree exactly, in both directions.</li>

<li><strong>Ship audit logs to a separate account with Object Lock and CloudTrail validation enabled.</strong> Immutability and integrity are what turn logs into evidence.</li>

<li><strong>Keep PHI out of everything that does not need it.</strong> De-identify early, tokenise where you can, and route non-clinical traffic through infrastructure that is not in scope.</li>

<li><strong>Define everything in Terraform or OpenTofu.</strong> A reviewable, version-controlled definition of your controls is worth more to an assessor than any screenshot, and it stops drift being invisible.</li>

<li><strong>Rehearse the restore and the breach response.</strong> Both are required, both are tested by asking for the record, and both are cheap to evidence if you actually do them.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is AWS HIPAA compliant?</h3>



<p class="wp-block-paragraph">Not on its own, and the phrasing is the problem. AWS offers HIPAA-eligible services and will sign a Business Associate Addendum, which means you can build a compliant system on it. Compliance is a property of your whole environment, including configuration, policies, vendors and staff. No provider can sell it to you as a finished product.</p>



<h3 class="wp-block-heading">How do I sign a BAA with AWS?</h3>



<p class="wp-block-paragraph">Through AWS Artifact in the console. It is self-service and there is no additional charge. Accept it for an individual account under account agreements, or from the management account of an AWS Organization under organization agreements so all current and future member accounts are covered. It should be accepted by someone with authority to bind your organisation, and it must be in place before any PHI reaches AWS.</p>



<h3 class="wp-block-heading">Does HIPAA require six years of CloudTrail logs?</h3>



<p class="wp-block-paragraph">No. The six-year requirement is a documentation retention rule covering the policies, procedures and records the Security Rule requires you to maintain, kept for six years from creation or from when they were last in effect. The audit controls standard requires the mechanism to record and examine activity but does not name a retention period for the logs themselves. You set that period in your own policy, justify it, and make the configuration match.</p>



<h3 class="wp-block-heading">Which AWS services can I use with PHI?</h3>



<p class="wp-block-paragraph">Only those on the AWS HIPAA Eligible Services Reference, and only in accounts covered by your BAA. Check the list before adopting anything new, read the feature-level exclusions noted against individual services, and re-check periodically because entries are added over time. Nothing in AWS will stop you using a non-eligible service with PHI, so this has to be an explicit process on your side.</p>



<h3 class="wp-block-heading">If encrypted PHI is exposed, do I still have to report a breach?</h3>



<p class="wp-block-paragraph">Generally no, provided the encryption meets the standard in HHS guidance and the decryption keys were not compromised along with the data. The Breach Notification Rule applies to unsecured PHI, and properly encrypted data does not meet that definition. This is why key management, not just enabling encryption, is the part that determines whether the protection is real. You still document the incident and the assessment.</p>



<h3 class="wp-block-heading">Are the new HIPAA Security Rule requirements in force?</h3>



<p class="wp-block-paragraph">Not at the time of writing. The proposals to make all implementation specifications required and to mandate encryption and multi-factor authentication came from a Notice of Proposed Rulemaking published in January 2025. The comment period has closed, but no final rule has been issued and the timeline has moved. Treat any specific compliance deadline you see quoted with suspicion and check the current status directly.</p>



<h3 class="wp-block-heading">Is a HIPAA-compliant AWS environment expensive to run?</h3>



<p class="wp-block-paragraph">The controls themselves are mostly cheap. KMS keys, CloudTrail validation, Config rules and account separation cost very little. The real costs are log storage volume, CloudTrail data events on busy buckets, running non-production environments to production standard, and staff time on risk analysis and evidence collection. Reducing what is in scope is the most effective cost lever, which is another reason to keep non-clinical workloads out of the regulated accounts.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">HIPAA compliance on AWS fails at the seams, not at the controls. Your encryption works. Your IAM policies are tight. What goes wrong is that the obligation follows the data into an account nobody added to the BAA, a Region nobody assessed, a staging database seeded from production, a vendor nobody signed an agreement with, or a retention period nobody wrote down.</p>



<p class="wp-block-paragraph">So build the boundary technically rather than trusting it contractually. Isolate PHI into its own accounts, wrap those accounts in guardrails that make the contract boundary enforceable, own your keys so the encryption means something legally, and make your logs provable rather than merely present. Then write the whole thing down, because in this domain the documentation genuinely is part of the control.</p>



<p class="wp-block-paragraph">None of that is exotic engineering. It is ordinary AWS work applied to a boundary that no dashboard draws for you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Working on a healthcare workload on AWS?</h2>



<p class="wp-block-paragraph">This is the kind of work I do. If you are building or inheriting a PHI environment on AWS, I can help with:</p>



<ul class="wp-block-list">
<li><strong>Scope and boundary review:</strong> mapping which accounts, Regions, services and vendors actually touch PHI, and finding the ones nobody knew about.</li>

<li><strong>Account and OU design with enforceable guardrails:</strong> SCPs, organization-level BAA coverage, and Region and service restrictions that hold without breaking your workloads.</li>

<li><strong>KMS key architecture:</strong> customer managed keys per data domain, split administration and usage, and key policies written so the breach safe harbour argument stands up.</li>

<li><strong>Audit logging that produces evidence:</strong> centralised CloudTrail with log file validation, an isolated archive account with Object Lock, and retention that matches your written policy exactly.</li>

<li><strong>Backup, restore and contingency testing:</strong> encrypted cross-account copies that stay inside your boundary, plus rehearsed restores documented as compliance artefacts.</li>

<li><strong>Terraform or OpenTofu modules for the whole control set,</strong> so your posture is reviewable, repeatable and does not drift between audits.</li>
</ul>



<p class="wp-block-paragraph">If you would rather start with something concrete than a discovery call, send me a redacted account structure, an SCP that is causing trouble, or the output of a Config or Security Hub run, and I will tell you what I would look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/hipaa-compliance-aws/">HIPAA Compliance on AWS: The Gaps That Pass Every Security Check</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
