<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Audit Logging | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/audit-logging/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/audit-logging/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 20 Oct 2026 06:00:04 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Audit Logging | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/audit-logging/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</title>
		<link>https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 20 Oct 2026 06:00:04 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Append-Only Storage]]></category>
		<category><![CDATA[Audit Evidence]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Backup]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Document Retention]]></category>
		<category><![CDATA[Immutable Storage]]></category>
		<category><![CDATA[Legal Hold]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Service Control Policies]]></category>
		<category><![CDATA[WORM Storage]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=910</guid>

					<description><![CDATA[<p>Enabling S3 Object Lock on a bucket does not protect the objects already in it. This is how to build an audit evidence repository on AWS that survives the question auditors actually ask, covering retention modes you cannot undo, delete markers that hide protected versions, chain of custody, and the storage bill compliance mode makes permanent.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/">Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that ends a smooth audit call is never &#8220;do you have the logs&#8221;. It&#8217;s &#8220;show me why nobody could have edited them.&#8221;</p>



<p class="wp-block-paragraph">That is a different question, and most evidence buckets fail it quietly. Versioning is on, the bucket is encrypted, access logging is enabled, and there is still an admin role in the account that can call <code>DeleteObjectVersion</code> on every object in it. Nothing is wrong until someone asks you to prove a negative.</p>



<p class="wp-block-paragraph">An audit evidence repository on AWS is not a bucket with a good name. It is a write path, a retention decision you cannot take back, and a set of guardrails that survive the person who built them leaving the company. This post covers how to build one with S3 Object Lock and AWS Backup Vault Lock, and more usefully, the five ways I have seen the design go wrong in ways nobody notices until the evidence is needed or the bill arrives.</p>



<h2 class="wp-block-heading">What &#8220;immutable&#8221; actually buys you on AWS</h2>



<p class="wp-block-paragraph">Three separate mechanisms get called immutability, and they protect different things.</p>



<ul class="wp-block-list">
<li><strong>S3 Object Lock</strong> stops an object version from being overwritten or deleted for a defined period. It works only on versioned buckets, and once you enable it on a bucket you cannot disable it or suspend versioning again.</li>



<li><strong>AWS Backup Vault Lock</strong> does the same job for recovery points in a backup vault, with its own retention floor and ceiling enforced at the vault level.</li>



<li><strong>Integrity validation</strong>, such as CloudTrail&#8217;s digest files, proves a file has not changed since delivery. That is a separate property from being undeletable, and auditors increasingly ask for both.</li>
</ul>



<p class="wp-block-paragraph">Object Lock is the core of an audit evidence repository on AWS because evidence usually arrives as files: exported reports, signed approvals, config snapshots, log archives. Vault Lock matters when the evidence is a backup of a database or an EBS volume. Integrity validation is what turns &#8220;we kept it&#8221; into &#8220;we kept it and here is the hash chain&#8221;.</p>



<h2 class="wp-block-heading">Failure one: picking the retention mode you cannot walk back</h2>



<p class="wp-block-paragraph">Object Lock has two retention modes and the difference is not cosmetic. In governance mode, a caller who holds the <code>s3:BypassGovernanceRetention</code> permission can shorten or remove retention and delete the version. In compliance mode, <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html">nobody can</a>, including the account root user, and the retention period cannot be shortened or downgraded to governance.</p>



<p class="wp-block-paragraph">Read that second sentence again with a typo in mind. Set seven years where you meant seven days, in compliance mode, and the only exit AWS documents is closing the account. That is not a bug. It is the entire point of the mode. But it means the mode is a business decision wearing a technical costume.</p>



<p class="wp-block-paragraph">How I decide: compliance mode only where an external rule names a retention period and an external party will inspect it. Everything else starts in governance mode, with the bypass permission granted to nobody by default and denied outright in a service control policy. Governance mode with no principal holding the bypass right is functionally immutable for day-to-day purposes, and it leaves you a documented, auditable escape hatch that requires an explicit policy change to use.</p>



<p class="wp-block-paragraph">The bypass is not silent, either. Overriding governance retention requires the <code>x-amz-bypass-governance-retention:true</code> header on the request, which means the action shows up in CloudTrail as a distinctly shaped call. That is a good alarm to build.</p>



<pre class="wp-block-code"><code># Create the bucket with Object Lock on. Versioning is enabled automatically
# and cannot be suspended afterwards.
aws s3api create-bucket 
  --bucket my-evidence-archive 
  --object-lock-enabled-for-bucket 
  --region eu-west-1 
  --create-bucket-configuration LocationConstraint=eu-west-1

# Default retention for NEW objects written to the bucket.
aws s3api put-object-lock-configuration 
  --bucket my-evidence-archive 
  --object-lock-configuration 
  '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}}'</code></pre>



<p class="wp-block-paragraph">Start there. Run it for a quarter. Watch what actually lands in the bucket and how big it gets. Then decide whether compliance mode is worth its permanence.</p>



<h2 class="wp-block-heading">Failure two: retention that was never applied to the objects that matter</h2>



<p class="wp-block-paragraph">This is the invisible one, and it is the reason to read the rest of this post.</p>



<p class="wp-block-paragraph">Object Lock is enabled at the bucket level, but retention lives on the individual object version. A default retention rule applies to objects written after the rule exists. It does not reach backwards. Enable Object Lock on an existing bucket holding three years of evidence and you get a bucket that reports Object Lock as enabled while every object already in it carries no retention at all.</p>



<p class="wp-block-paragraph">The bucket console looks correct. The compliance checklist gets ticked. The old evidence is still deletable by anyone with delete permissions.</p>



<p class="wp-block-paragraph">Two things to do about it. First, backfill retention on existing versions, either object by object with <code>put-object-retention</code> or, for anything above a few thousand keys, with S3 Batch Operations, which exists precisely for applying retention at scale. Second, verify per object rather than per bucket.</p>



<pre class="wp-block-code"><code># Check what protection a specific version actually carries.
# Look for ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus.
aws s3api head-object 
  --bucket my-evidence-archive 
  --key 2024/q3/access-review-export.csv 
  --version-id "SOME_VERSION_ID"</code></pre>



<p class="wp-block-paragraph">The stronger fix is to stop trusting the writer to set retention correctly and enforce a floor in the bucket policy. S3 exposes the <code>s3:object-lock-remaining-retention-days</code> condition key for exactly this, so you can reject writes that ask for too short a lock.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RejectShortRetention",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-evidence-archive/*",
      "Condition": {
        "NumericLessThan": {
          "s3:object-lock-remaining-retention-days": "365"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The write path now fails loudly when someone gets it wrong, which is what you want. A rejected upload gets noticed. A silently unprotected object does not.</p>



<h2 class="wp-block-heading">Failure three: the delete that looks like a delete</h2>



<p class="wp-block-paragraph">Object Lock protects object versions. It does not stop new versions being created, and it does not stop delete markers being placed on top of a protected object.</p>



<p class="wp-block-paragraph">So a user with ordinary delete permissions can issue a delete against a locked object, get a success response, and make it vanish from every normal listing. The protected version is still there. Nothing was destroyed. But an auditor browsing the bucket sees nothing, and the person who ran the command believes the file is gone.</p>



<p class="wp-block-paragraph">Three practical consequences:</p>



<ul class="wp-block-list">
<li>Deny <code>s3:DeleteObject</code> on the evidence prefix for everyone except a break-glass role. Immutability is not a substitute for permissions, it is a backstop under them.</li>



<li>Any retrieval tooling you hand to an auditor must list object <em>versions</em>, not objects. <code>list-object-versions</code>, not <code>list-objects-v2</code>.</li>



<li>Lifecycle rules interact with this in a way that surprises people. An expiration rule on a versioned bucket adds a delete marker rather than removing data, and a rule that tries to permanently expire a noncurrent version will not remove a version still under retention. Your cleanup policy and your retention policy are two different clocks, and the lock always wins.</li>
</ul>



<h2 class="wp-block-heading">Failure four: evidence that arrives already unprovable</h2>



<p class="wp-block-paragraph">Immutability starts when the object lands. Everything before that is chain of custody, and it is where most repositories are weakest.</p>



<p class="wp-block-paragraph">If the evidence is CloudTrail, use the integrity validation CloudTrail already ships. With log file validation enabled, CloudTrail writes an hourly digest file containing hashes of the log files delivered in that hour, signs each digest with a per-region private key, and puts the digests in a separate folder in the same bucket. You can then verify the chain later.</p>



<pre class="wp-block-code"><code># Turn on digest files for an existing trail.
aws cloudtrail update-trail 
  --name org-audit-trail 
  --enable-log-file-validation

# Verify a time range later. Files must still be in the location
# CloudTrail delivered them to, or validation cannot run.
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-audit-trail 
  --start-time 2020-01-01T00:00:00Z</code></pre>



<p class="wp-block-paragraph">Two details worth knowing before you design around it. Validation only covers log files referenced by digest files, so a gap in digest coverage is a gap in your proof. And the files must stay where CloudTrail put them, which means copying logs into a tidier prefix breaks the built-in verification and hands you the job of writing your own checker.</p>



<p class="wp-block-paragraph">For evidence that is not CloudTrail, the equivalent is doing the work yourself: hash the artifact at the point of generation, record the hash somewhere separate from the artifact, and write both. A manifest per collection run, checksummed and locked alongside the files, is unglamorous and holds up.</p>



<p class="wp-block-paragraph">One constraint that catches people mid-build: a bucket with Object Lock enabled cannot be used as the destination for S3 server access logs. If your design had one bucket receiving both the evidence and its own access logs, it needs splitting.</p>



<h2 class="wp-block-heading">Failure five: the bill you cannot switch off</h2>



<p class="wp-block-paragraph">Compliance mode plus a long retention period plus archival storage classes is where cost surprises live, because every lever you would normally pull to reduce spend has been deliberately disabled.</p>



<p class="wp-block-paragraph">The mechanics that drive it, none of which are secret but all of which get discovered late:</p>



<ul class="wp-block-list">
<li><strong>Minimum storage durations.</strong> S3 Glacier Flexible Retrieval carries a 90 day minimum and S3 Glacier Deep Archive carries 180 days. Transition earlier than that and you are billed for the remainder of the minimum anyway.</li>



<li><strong>Per-object overhead in Glacier classes.</strong> AWS documents 40 KB of metadata overhead per object for Glacier Flexible Retrieval and Deep Archive, of which 8 KB is billed at S3 Standard rates and 32 KB at the destination rate. For a repository full of 2 KB JSON approvals, the overhead is the bill.</li>



<li><strong>Billing starts when the rule is satisfied.</strong> Lifecycle transitions to Glacier classes happen asynchronously, but you are charged at the destination rate, and the minimum duration clock starts, from the date the rule matched rather than the date the bytes physically moved.</li>



<li><strong>You cannot delete your way out.</strong> Under compliance mode, oversized or mistakenly retained objects stay until their retain-until date passes, whatever the finance conversation says.</li>
</ul>



<p class="wp-block-paragraph">The design response is to aggregate before you lock. Batch small artifacts into per-day or per-collection archives, checksum the archive, and lock the archive rather than thousands of tiny files. It costs you a little retrieval convenience and saves you a per-object overhead multiplier you cannot renegotiate later.</p>



<p class="wp-block-paragraph">Also pick the retrieval tier against the real deadline. Deep Archive is the cheapest place to keep evidence and one of the slowest to get it back. If a regulator or a legal team may need files at short notice, that trade is a bad one regardless of the storage line item.</p>



<h2 class="wp-block-heading">Putting the repository together</h2>



<p class="wp-block-paragraph">With the failure families in view, the build order is fairly short.</p>



<ol class="wp-block-list">
<li><strong>Separate the account.</strong> Evidence lives in its own AWS account, not a prefix in the production log bucket. The blast radius of a compromised production admin should not include the record of what they did.</li>



<li><strong>Write the service control policies first.</strong> At the organisation level, deny <code>s3:BypassGovernanceRetention</code>, deny <code>s3:PutBucketPolicy</code> and <code>s3:DeleteBucketPolicy</code> on the evidence bucket, and deny disabling the trail. SCPs apply to the account root too, which is the point.</li>



<li><strong>Create the bucket with Object Lock and a default governance retention.</strong> Block public access, enforce TLS with an <code>aws:SecureTransport</code> condition, and encrypt with a customer managed KMS key whose policy is written for this bucket alone.</li>



<li><strong>Enforce the retention floor in the bucket policy</strong> using the remaining-retention-days condition, so a misconfigured collector fails rather than writes something unprotected.</li>



<li><strong>Replicate to a second account or region.</strong> Object Lock works with S3 Replication, and the destination bucket must also have Object Lock enabled. This is your answer to region-level loss and to an account-level mistake.</li>



<li><strong>Add Vault Lock for backup-shaped evidence.</strong> Snapshots and database backups belong in a locked backup vault rather than being exported to S3 by hand.</li>



<li><strong>Alarm on the write path.</strong> A day with no new evidence objects is the failure that never pages anyone. Alert on absence, not just on errors.</li>
</ol>



<p class="wp-block-paragraph">Vault Lock deserves its own note because its compliance mode has a grace period that Object Lock does not. You choose a cooling-off window, which AWS requires to be at least three days, during which the lock can still be removed. After that the vault and its lock are immutable and cannot be changed or deleted by anyone, including AWS.</p>



<pre class="wp-block-code"><code># Compliance mode: the presence of --changeable-for-days is what makes it
# compliance rather than governance. Minimum value is 3 (72 hours).
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name evidence-vault 
  --min-retention-days 365 
  --max-retention-days 2555 
  --changeable-for-days 3</code></pre>



<p class="wp-block-paragraph">Use that window. Run a backup job through the vault, confirm the retention arithmetic against what the policy document actually requires, and only then let the clock run out.</p>



<p class="wp-block-paragraph">One more design question worth asking out loud: does every copy need to be inside AWS? A second, offline-ish copy on independent infrastructure answers a different threat than cross-region replication does, because it survives an account-level compromise. Providers like InterServer, Contabo and Hetzner sell cheap storage that can hold an encrypted archive, and several S3-compatible services implement Object Lock semantics of their own. Confirm the provider actually supports Object Lock rather than assuming an S3-compatible API implies it, because that varies.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">AccessDenied when deleting a version you are allowed to delete</h3>



<p class="wp-block-paragraph">Usually governance retention plus a missing bypass. The IAM permission <code>s3:BypassGovernanceRetention</code> and the explicit bypass on the request are both required. If the object is in compliance mode, there is no combination of permissions that will work, and the error is telling you the truth.</p>



<pre class="wp-block-code"><code># Only valid for governance mode, and only with the matching IAM permission.
aws s3api delete-object 
  --bucket my-evidence-archive 
  --key path/to/object 
  --version-id "SOME_VERSION_ID" 
  --bypass-governance-retention</code></pre>



<h3 class="wp-block-heading">Replication fails on an Object Lock bucket</h3>



<p class="wp-block-paragraph">The destination needs Object Lock enabled too. Replicated objects take the lock configuration of the source, so a destination without Object Lock cannot accept them. Check the destination bucket first, then the replication role&#8217;s permissions.</p>



<h3 class="wp-block-heading">Lifecycle rule appears to do nothing</h3>



<p class="wp-block-paragraph">Two likely causes. The versions it targets are still under retention, so expiration cannot remove them. Or the rule targets current versions on a versioned bucket, in which case it is adding delete markers and the data is still there as noncurrent versions, still being billed.</p>



<h3 class="wp-block-heading">validate-logs reports invalid or missing files</h3>



<p class="wp-block-paragraph">Confirm nothing moved the objects. Validation requires the log and digest files to be in the location CloudTrail delivered them to, and the command runs per region. A copy job that reorganised the prefix layout is the usual culprit.</p>



<h3 class="wp-block-heading">Object Lock cannot be enabled on the bucket</h3>



<p class="wp-block-paragraph">Versioning has to be enabled first, and Object Lock cannot be turned off later. If the bucket is currently a server access log destination, that has to move before Object Lock will work for you at all.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Enabling Object Lock on an existing bucket and assuming the objects already in it are now protected. They are not, until retention is applied to each version.</li>



<li>Choosing compliance mode on day one to look thorough, then discovering the retention period was wrong.</li>



<li>Treating Object Lock as an access control. It is a floor under your permissions, not a replacement for them.</li>



<li>Leaving the bypass permission attached to a general administrator role because it was needed once during testing.</li>



<li>Locking millions of tiny objects into Glacier classes and paying per-object overhead on all of them.</li>



<li>Storing evidence in the same account as the systems it describes, which makes the evidence only as trustworthy as that account&#8217;s admins.</li>



<li>Never testing retrieval, so the first restore attempt happens under audit pressure with an unfamiliar Glacier restore workflow.</li>
</ul>



<h2 class="wp-block-heading">Best practices for an audit evidence repository on AWS</h2>



<ul class="wp-block-list">
<li>Governance mode by default, compliance mode only where an external requirement names the period. Write down which rule drove the choice, in the same repository.</li>



<li>Enforce the retention floor in the bucket policy so the write path fails loudly instead of writing unprotected objects.</li>



<li>Use legal hold, not extended retention, for investigations. Legal holds have no expiry and are removed explicitly, which is the right shape for something that ends on an unknown date.</li>



<li>Aggregate small artifacts into archives with a checksummed manifest before locking them.</li>



<li>Replicate to a second account with Object Lock enabled, and check replication metrics rather than assuming.</li>



<li>Rehearse retrieval quarterly, including the Glacier restore step, and keep the output as evidence of the rehearsal.</li>



<li>Monitor for absence of writes. Tools like Amazon CloudWatch, Grafana or Datadog are all fine here. The alert that matters is &#8220;no evidence arrived today&#8221;.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I enable S3 Object Lock on a bucket that already exists?</h3>



<p class="wp-block-paragraph">Yes. S3 supports enabling Object Lock on existing buckets, and versioning must be enabled. The important caveat is that enabling it does not retroactively protect the objects already stored. You apply retention to those separately, per object or with S3 Batch Operations.</p>



<h3 class="wp-block-heading">What is the difference between governance mode and compliance mode?</h3>



<p class="wp-block-paragraph">Governance mode can be overridden by a principal holding <code>s3:BypassGovernanceRetention</code> who sends the bypass header. Compliance mode cannot be overridden by anyone, including the root user, and the retention period cannot be shortened or the mode downgraded.</p>



<h3 class="wp-block-heading">Does S3 Object Lock stop someone deleting the file?</h3>



<p class="wp-block-paragraph">It stops the protected version being destroyed. It does not stop a delete marker being placed on top, which hides the object from ordinary listings while the version remains. Keep delete permissions tight and list versions when you audit.</p>



<h3 class="wp-block-heading">Do I need Object Lock if I already have versioning and MFA delete?</h3>



<p class="wp-block-paragraph">They solve different problems. Versioning preserves history but a sufficiently privileged caller can still remove versions. Object Lock removes that possibility for the duration of the retention period, which is the property an auditor is asking about.</p>



<h3 class="wp-block-heading">How do I prove the evidence has not been modified?</h3>



<p class="wp-block-paragraph">For CloudTrail, enable log file integrity validation and keep the signed digest files with the logs, then run <code>aws cloudtrail validate-logs</code> for the period in question. For other artifacts, hash at generation time, store the manifest separately, and lock both.</p>



<h3 class="wp-block-heading">Can a lifecycle rule delete objects that are still locked?</h3>



<p class="wp-block-paragraph">No. Retention wins over lifecycle expiration. Plan the two together, because a cleanup rule that quietly does nothing for years is a cost problem you will not see until you look at storage growth.</p>



<h3 class="wp-block-heading">Is Vault Lock compliance mode reversible?</h3>



<p class="wp-block-paragraph">Only during the cooling-off window you set at creation, which must be at least three days. After the lock date, the vault and its lock cannot be changed or deleted by anyone, including AWS.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An audit evidence repository on AWS fails in a specific, quiet way: the bucket says Object Lock is enabled, and the objects inside carry no retention. Everything else in this post is downstream of that gap. Check protection at the object version, not at the bucket, and enforce the floor in the write path so the system tells you when it is wrong.</p>



<p class="wp-block-paragraph">Then pick the mode like the one-way door it is. Governance mode with a locked-down bypass gets you almost all of the protection and keeps a documented way back. Compliance mode is worth its permanence when a real external rule demands it, and expensive theatre when it does not.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or reviewing one of these?</h2>



<p class="wp-block-paragraph">Immutable retention is easy to enable and easy to get subtly wrong, and the mistakes are the kind you find out about at the worst moment. This is work I take on directly:</p>



<ul class="wp-block-list">
<li>Designing an evidence account with Object Lock, KMS, service control policies and a locked-down write path</li>



<li>Auditing an existing bucket to find object versions that carry no retention despite Object Lock being enabled</li>



<li>Backfilling retention across existing objects with S3 Batch Operations without locking in the wrong period</li>



<li>Modelling the long-run cost of a retention decision before compliance mode makes it permanent</li>



<li>Setting up cross-account replication, AWS Backup Vault Lock and CloudTrail integrity validation as one coherent design</li>



<li>Writing the retrieval runbook and rehearsing it, so the first restore is not during an audit</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion on what you already have, send me the bucket policy, the Object Lock configuration and the output of a <code>head-object</code> call on one old evidence file. That is usually enough to tell you where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/">See how I work on projects</a></div>



<div class="wp-block-button is-style-outline is-style-outline--1"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/audit-evidence-repository-aws/">Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Secure Patient Document Portal on S3 and Cognito</title>
		<link>https://john-nessime.com/blog/cloud-computing/secure-patient-document-portal-s3-cognito/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 19 Oct 2026 06:00:12 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[ePHI]]></category>
		<category><![CDATA[Healthcare Cloud]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Identity Pools]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[MFA]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[Session Management]]></category>
		<category><![CDATA[Token Revocation]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=457</guid>

					<description><![CDATA[<p>Cognito for sign-in, S3 for the files, and a demo that passes on the first try. The problems start after that. Prefix isolation, presigned URL lifetimes, KMS as a second gate, CloudTrail data events, and token revocation that actually ends a session.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/secure-patient-document-portal-s3-cognito/">Building a Secure Patient Document Portal on S3 and Cognito</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The request usually arrives sideways. A clinic manager says something like: &#8220;we&#8217;re emailing scans as PDF attachments and our insurer flagged it, can you build us something?&#8221; Nobody says the words &#8220;patient portal.&#8221; They say &#8220;somewhere for patients to pick up their files.&#8221;</p>



<p class="wp-block-paragraph">So you sketch the obvious thing. Amazon Cognito for sign-in, Amazon S3 for the documents, a small front end in between. It&#8217;s three services and a weekend, and it works on the first try, which is exactly the problem. A <strong>secure patient document portal</strong> built this way passes every test you think to run, because the tests you think to run are all &#8220;can Alice download Alice&#8217;s file?&#8221; The one that matters is &#8220;what can Alice see about Bob?&#8221; and that one has an answer you won&#8217;t like.</p>



<p class="wp-block-paragraph">This post covers the parts that go wrong after the demo. Where to put the authorization boundary, how to scope the S3 policy so one patient can&#8217;t enumerate another, why your download links expire earlier than you configured, what encryption actually gates, and what your audit trail looks like on the day someone asks for it. Everything here is mechanism, not a compliance checklist. Getting a Business Associate Agreement in place is a separate conversation with your lawyer and your AWS account team.</p>



<h2 class="wp-block-heading">The failure that never pages you</h2>



<p class="wp-block-paragraph">Here&#8217;s the shape of it. You write an IAM policy that lets a signed-in user read objects under their own prefix. Something like <code>arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*</code>. You test it. Alice gets her file. Alice cannot get Bob&#8217;s file. Ship it.</p>



<p class="wp-block-paragraph">Then, because the front end needs to render a file list, someone adds <code>s3:ListBucket</code>. And <code>s3:ListBucket</code> is not an object-level action. It applies to the bucket ARN, not the object ARN, so the <code>${cognito-identity.amazonaws.com:sub}</code> substitution in your resource string does nothing for it. Unless you add a separate condition on the request, a signed-in patient can list every key in the bucket.</p>



<p class="wp-block-paragraph">They still can&#8217;t download anything. Every <code>GetObject</code> outside their prefix returns a denial. But they can read the key names. And key names in a clinic bucket are rarely neutral. <code>patients/8f2a.../oncology-referral-2024.pdf</code> tells you something about a person even if the bytes stay locked.</p>



<p class="wp-block-paragraph">Nothing alerts on this. No error appears in any log you&#8217;re watching, because nothing errored. The listing succeeded. That&#8217;s the failure mode worth building around: in a document portal, the metadata leak is the one that doesn&#8217;t announce itself.</p>



<p class="wp-block-paragraph">The fix is a condition key on the list action, not a resource string:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOwnPrefixOnly",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::clinic-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "patients/${cognito-identity.amazonaws.com:sub}/*"
          ]
        }
      }
    },
    {
      "Sid": "ReadOwnObjectsOnly",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*"
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Two statements, two different jobs. The first constrains what the caller may ask the bucket to enumerate. The second constrains what bytes the caller may pull. Write only the second one and you&#8217;ve secured the download path while leaving the discovery path wide open.</p>



<p class="wp-block-paragraph">One trap in that variable. <code>${cognito-identity.amazonaws.com:sub}</code> resolves to the identity ID from the Cognito <em>identity pool</em>, not the <code>sub</code> claim from the user pool. They are different values for the same human. If you name your prefixes with the user pool sub because that&#8217;s what your application code has handy, the policy will match nothing and every request will be denied, which at least fails loudly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Decide where the authorization boundary lives</h2>



<p class="wp-block-paragraph">There are two honest architectures here and the choice shapes everything downstream. Pick deliberately.</p>



<h3 class="wp-block-heading">Pattern A: browser holds AWS credentials</h3>



<p class="wp-block-paragraph">The user signs in to a Cognito user pool. The application exchanges that token with a Cognito identity pool for temporary AWS credentials, and the browser calls S3 directly using the AWS SDK.</p>



<p class="wp-block-paragraph">It&#8217;s less code and there&#8217;s no server to run. The authorization boundary is entirely the IAM role policy attached to the identity pool. That policy is the whole security model, so it had better be right, and it&#8217;s harder to reason about than application code because the enforcement happens in a service you can&#8217;t step through with a debugger.</p>



<p class="wp-block-paragraph">The real limitation is expressiveness. IAM policy conditions do not know that Bob&#8217;s referral was superseded, that Alice revoked consent last Tuesday, or that a document should only be visible after a clinician releases it. Anything with business logic in it has to be encoded as prefix structure or object tags, and that gets brittle fast.</p>



<h3 class="wp-block-heading">Pattern B: backend brokers every object</h3>



<p class="wp-block-paragraph">The browser never touches AWS credentials. It sends the Cognito access token to your own API. The API validates the token, checks its own records for who may see what, and returns a presigned URL for that one object. S3 is reachable only by the API&#8217;s execution role.</p>



<p class="wp-block-paragraph">You now have a place to put logic, a place to log intent alongside access, and a single choke point to rate limit. The cost is that you own a service, its uptime, and its dependencies.</p>



<p class="wp-block-paragraph">For a small clinic build, this is the one I reach for first. Not because Pattern A is insecure, but because the questions clients ask in month three (&#8220;can we hide results until the doctor reviews them?&#8221;) are answerable in Pattern B and require a re-architecture in Pattern A. The identity-pool approach earns its keep when access really is a pure function of who you are, which in healthcare it rarely stays.</p>



<p class="wp-block-paragraph">If you do want attribute-driven access without a backend, Cognito identity pools can map claims from the token onto principal tags on the temporary session, which you then reference in policy conditions. It works, and it is the right tool for coarse distinctions like patient versus staff. It is not the right tool for per-document consent state.</p>



<h2 class="wp-block-heading">Your download links expire before you think they do</h2>



<p class="wp-block-paragraph">This one generates support tickets on week two. You set a presigned URL to last a few hours. Patients report links dying in under one. Nothing in your code changed.</p>



<p class="wp-block-paragraph">A presigned URL carries a signature made with whatever credentials signed it, and it cannot outlive those credentials. AWS documents the ceilings clearly: with long-lived IAM user keys and Signature Version 4 the maximum is seven days; from the S3 console the range is one minute to twelve hours; with temporary credentials from an assumed role, the URL dies when the role session does, regardless of the expiry you asked for. A default <code>AssumeRole</code> session is one hour.</p>



<p class="wp-block-paragraph">Your Lambda or container signing those URLs is using temporary credentials. So the effective lifetime is the shorter of your requested expiry and the remaining time on that role session, and the remaining time depends on when the runtime last refreshed. That&#8217;s the source of the flapping. You can watch it yourself:</p>



<pre class="wp-block-code"><code># Sign a URL and inspect what was actually encoded into it.
aws s3 presign s3://clinic-docs/patients/example/scan.pdf --expires-in 3600

# The returned URL contains X-Amz-Date, X-Amz-Expires and X-Amz-Credential.
# X-Amz-Expires is the window you asked for. If X-Amz-Credential shows an
# ASIA... key id, you signed with temporary credentials and the real ceiling
# is the session, not the window.</code></pre>



<p class="wp-block-paragraph">The honest resolution is to stop fighting it. Sign short, on demand, when the user clicks. Fifteen minutes is generous for a download the patient is standing in front of. Long links are not a convenience feature in this context anyway, because a presigned URL is a bearer credential: whoever holds the string can fetch the object, signed in or not. Forwarding the email forwards the access. If your portal mails links, you have moved the authorization boundary into someone&#8217;s inbox.</p>



<p class="wp-block-paragraph">Mail a notification that says a document is ready and link to the portal login. Never mail the signed URL itself.</p>



<h2 class="wp-block-heading">Encryption that actually gates something</h2>



<p class="wp-block-paragraph">New S3 objects are encrypted at rest by default with S3-managed keys. That satisfies the sentence &#8220;data is encrypted at rest&#8221; and gates precisely nothing, because anyone with <code>s3:GetObject</code> gets plaintext back. It is a storage property, not an access control.</p>



<p class="wp-block-paragraph">Server-side encryption with an AWS KMS customer managed key is different in kind. Reading an object now requires two independent grants: the S3 permission and <code>kms:Decrypt</code> on the key. Those live in different policies and can be owned by different people. A bucket policy mistake no longer hands over plaintext on its own, and revoking the key grant cuts off reads without touching a single S3 permission.</p>



<p class="wp-block-paragraph">You also get a second audit stream. Every decrypt is a KMS API call, recorded separately from the S3 call, which is useful precisely because an attacker who tampers with one trail probably didn&#8217;t think about the other.</p>



<p class="wp-block-paragraph">The costs are real and worth stating. KMS charges per request, so a portal that fetches many small objects will notice. S3 Bucket Keys reduce that request volume substantially by caching a bucket-level key, at the price of coarser detail in the KMS audit records. For a document portal with modest traffic and high sensitivity, I&#8217;d take the per-request cost and keep the granularity. For a thumbnail-heavy application, the opposite.</p>



<h2 class="wp-block-heading">The audit trail you don&#8217;t have yet</h2>



<p class="wp-block-paragraph">Assume someone will eventually ask: who downloaded this specific file, and when? If you have not explicitly turned something on, the answer is that you don&#8217;t know.</p>



<p class="wp-block-paragraph">CloudTrail is on for management events by default, and management events are control-plane operations: creating the bucket, changing its policy, attaching a role. Object-level reads and writes are <em>data events</em>, and data events are off by default because of volume and cost. <code>GetObject</code> on a patient scan is a data event. Unless you configured a trail with data events for that bucket, no record of the download exists.</p>



<pre class="wp-block-code"><code># Log S3 object-level activity for one bucket on an existing trail.
# Advanced event selectors let you scope to a single bucket prefix rather
# than every object in the account.
aws cloudtrail put-event-selectors 
  --trail-name clinic-audit-trail 
  --advanced-event-selectors '[
    {
      "Name": "PHI bucket object access",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Data"]},
        {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
        {"Field": "resources.ARN", "StartsWith":
          ["arn:aws:s3:::clinic-docs/patients/"]}
      ]
    }
  ]'</code></pre>



<p class="wp-block-paragraph">S3 server access logs are the other option and they are not equivalent. They record HTTP-level request detail and are delivered on a best-effort basis with no delivery guarantee, which makes them useful for traffic analysis and weak as evidence. CloudTrail data events carry caller identity, which is the field you actually need when the question is &#8220;who.&#8221;</p>



<p class="wp-block-paragraph">Two things to sort out while you&#8217;re in there. Send the trail to a bucket in a different account, or at minimum a bucket the portal&#8217;s roles cannot write to, so that compromising the application doesn&#8217;t compromise its own log. And put a lifecycle rule on the log bucket, because data event volume grows with usage and nobody budgets for it. If you already run centralized dashboards on something like Grafana Cloud or CloudWatch, wire an alert to unusual read volume per identity now rather than after the incident.</p>



<h2 class="wp-block-heading">Cutting off access when someone leaves</h2>



<p class="wp-block-paragraph">Disabling a Cognito user does not immediately end their session, and this surprises people. Tokens are self-contained. An access token already in a browser stays cryptographically valid until it expires, typically about an hour.</p>



<p class="wp-block-paragraph">Refresh tokens are the longer exposure. The default is thirty days, and app clients can be configured anywhere from sixty minutes to ten years. Ten-year refresh tokens on a portal holding medical records is not a hypothetical misconfiguration; it&#8217;s a checkbox someone ticked to stop users complaining about re-login.</p>



<p class="wp-block-paragraph">Cognito gives you revocation. Token revocation is enabled by default on newly created app clients. <code>AdminUserGlobalSignOut</code> revokes all of a user&#8217;s refresh, ID, and access tokens from the server side, and the <code>/oauth2/revoke</code> endpoint kills one refresh token and everything descended from it.</p>



<pre class="wp-block-code"><code># Server-side, authorized with IAM credentials. Revokes the user's refresh,
# ID and access tokens.
aws cognito-idp admin-user-global-sign-out 
  --user-pool-id us-east-1_EXAMPLE 
  --username patient-example

# Confirm revocation is switched on for the app client before you rely on it.
aws cognito-idp describe-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --query 'UserPoolClient.EnableTokenRevocation'</code></pre>



<p class="wp-block-paragraph">Now the part that catches teams out. AWS documents that a revoked token still passes verification in a generic JWT library that only checks signature and expiry. Revocation is enforced by Cognito, not baked into the token. If your API validates tokens by fetching the JWKS and calling a verify function, a revoked token sails straight through until it expires on its own.</p>



<p class="wp-block-paragraph">Which means short access token lifetimes are load-bearing. Revocation shortens the tail; it doesn&#8217;t close the window. Refresh token rotation, available on the Essentials and Plus feature tiers, replaces the refresh token on each exchange and narrows the window a stolen one is useful for. If you&#8217;re paying for Plus for threat protection features, you already have it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the usual suspects</h2>



<h3 class="wp-block-heading">Every request returns access denied after login</h3>



<p class="wp-block-paragraph">Nine times out of ten the prefix in your policy variable doesn&#8217;t match the prefix in your bucket. Print the identity ID your session actually resolved to and compare it character by character against a real key. Remember the identity pool ID and the user pool sub are different values.</p>



<h3 class="wp-block-heading">Signature does not match on a presigned URL</h3>



<p class="wp-block-paragraph">Usually something modified the request after signing. A client library adding a header, a proxy rewriting the path, or an email gateway URL-encoding the query string. Compare the <code>X-Amz-SignedHeaders</code> list in the URL against what the client actually sent.</p>



<h3 class="wp-block-heading">Uploads succeed but downloads fail</h3>



<p class="wp-block-paragraph">Check the KMS key policy separately from the IAM policy. A role can hold <code>kms:GenerateDataKey</code> (enough to write) without <code>kms:Decrypt</code> (needed to read). The S3 error will not say the word KMS prominently, which is why this one burns an afternoon.</p>



<h3 class="wp-block-heading">Browser uploads fail before any request reaches S3</h3>



<p class="wp-block-paragraph">That&#8217;s CORS. The bucket needs a CORS configuration allowing your portal&#8217;s origin, the methods you use, and any headers the SDK sends. Browser preflight failures never appear in S3 logs, so check the browser console first.</p>



<h3 class="wp-block-heading">Nothing appears in CloudTrail</h3>



<p class="wp-block-paragraph">Data events are not shown in CloudTrail Event History. Query the destination bucket directly, with Athena or CloudTrail Lake, and allow a few minutes for delivery.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Granting <code>s3:ListBucket</code> without an <code>s3:prefix</code> condition, so patients can enumerate every key even though downloads are blocked.</li>

<li>Putting identifying information in object keys or file names. Assume the key will leak eventually and use opaque identifiers.</li>

<li>Emailing presigned URLs. The link is the credential, and inboxes get forwarded, backed up, and breached.</li>

<li>Extending refresh token lifetime to silence complaints about re-login, instead of implementing rotation.</li>

<li>Assuming default encryption means access control. It means the disk is encrypted, nothing more.</li>

<li>Enabling versioning without a lifecycle rule, so deleted documents linger indefinitely as noncurrent versions you forgot exist.</li>

<li>Skipping CloudTrail data events to save money, then discovering the gap during an access review.</li>
</ul>



<h2 class="wp-block-heading">Best practices for a secure patient document portal</h2>



<ul class="wp-block-list">
<li>Keep one bucket for patient documents and nothing else. Mixed-purpose buckets make policies ambiguous and audits slow.</li>

<li>Turn on S3 Block Public Access at the account level, not just the bucket. Account level survives someone creating a new bucket in a hurry.</li>

<li>Enable versioning and set a noncurrent version expiration rule, so an overwritten document is recoverable but not permanent.</li>

<li>Use a KMS customer managed key so decrypt authority is separable from S3 authority.</li>

<li>Require MFA for staff accounts. TOTP costs nothing to enable and staff accounts are the ones with breadth.</li>

<li>Sign download URLs on demand with short expiry, and always from a backend that logged why it signed.</li>

<li>Put the portal behind a WAF and rate limiting. Cloudflare in front of the application, or AWS WAF on the distribution, both work; the point is that credential stuffing against a login page is the most likely thing to actually happen to you.</li>

<li>Restrict administrative access to the AWS console and to any bastion by network as well as identity, whether that&#8217;s IAM Identity Center with condition keys or a VPN such as NordVPN or Surfshark for a small team without central identity.</li>

<li>Write down the retention period before launch, then encode it as a lifecycle rule. Retention decided later is retention never decided.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Are S3 and Cognito enough for HIPAA compliance?</h3>



<p class="wp-block-paragraph">Both are covered under AWS&#8217;s HIPAA eligibility program, and Cognito&#8217;s documentation states it is HIPAA-BAA eligible. That makes them usable building blocks, not compliance. You need a signed Business Associate Agreement with AWS, and the configuration, access reviews, training, and documented policies remain yours. Service eligibility is a prerequisite, not an outcome.</p>



<h3 class="wp-block-heading">Should I use a Cognito identity pool or a backend API?</h3>



<p class="wp-block-paragraph">Use an identity pool when access is a pure function of identity and the rules will not change. Use a backend when documents have states, when a clinician gates release, or when you need to log intent alongside access. Most clinic portals become the second within a year.</p>



<h3 class="wp-block-heading">How long should a presigned URL last?</h3>



<p class="wp-block-paragraph">Minutes, not hours. Sign at click time. The maximum with long-lived credentials is seven days, but that ceiling is irrelevant here because the link is a bearer credential and a longer window is a longer leak.</p>



<h3 class="wp-block-heading">Can patients upload documents back to the portal?</h3>



<p class="wp-block-paragraph">Yes, using presigned PUT or presigned POST. Presigned POST is worth knowing about because its policy document can constrain content type and size before the object lands, which presigned PUT cannot. Either way, write uploads into a quarantine prefix, scan them, then move them.</p>



<h3 class="wp-block-heading">Does enabling CloudTrail data events cost much?</h3>



<p class="wp-block-paragraph">Data events are billed per event recorded with no free tier, unlike management events. For a portal serving a few thousand downloads a month the amount is small; for a bucket also serving application assets it is not. Scope your event selector to the patient prefix rather than the whole account and the question mostly goes away.</p>



<h3 class="wp-block-heading">What happens to sessions when I disable a user?</h3>



<p class="wp-block-paragraph">Nothing immediate. Existing access tokens remain valid until they expire. Call <code>AdminUserGlobalSignOut</code> to revoke tokens, and keep access token lifetime short so the residual window is measured in minutes.</p>



<h3 class="wp-block-heading">Do I need CloudFront in front of the bucket?</h3>



<p class="wp-block-paragraph">Not for correctness. It helps when patients are geographically spread, when you want signed cookies for a set of objects rather than one URL per object, or when you want a single edge to attach a WAF to. For a single-clinic portal, direct presigned URLs are simpler and one fewer thing to misconfigure.</p>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">A signed-in token proves who someone is. It does not decide what they may reach. In a <strong>secure patient document portal</strong> the boundary is the policy condition, the KMS grant, and the code that decides whether to sign a URL at all, and every one of those is something you have to write on purpose.</p>



<p class="wp-block-paragraph">The demo passes either way. The difference only shows up when someone reads a key name they shouldn&#8217;t have seen, and nothing in your system considers that an error.</p>



<p class="wp-block-paragraph">If you build one of these, spend an hour before launch on exactly two things: sign in as a test patient and try to list the bucket, then ask your logs who downloaded a specific file last week. If either answer is uncomfortable, you&#8217;ve found your next task.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your portal?</h2>



<p class="wp-block-paragraph">This is the kind of build where the gap between working and safe is invisible from the outside. I help small teams close it.</p>



<ul class="wp-block-list">
<li>Reviewing S3 bucket policies and Cognito identity pool roles for prefix isolation gaps, including the list-versus-get split most policies miss.</li>

<li>Designing the boundary: identity pool credentials in the browser versus a backend that brokers presigned URLs, with the trade-offs written down for your stakeholders.</li>

<li>Setting up KMS customer managed keys, key policies, and grant separation so decrypt authority is not implied by storage authority.</li>

<li>Turning on scoped CloudTrail data events, building the Athena tables to query them, and alerting on abnormal per-identity read volume.</li>

<li>Hardening the Cognito side: MFA, token lifetimes, refresh token rotation, and a revocation path that actually ends sessions.</li>

<li>Writing the whole thing as Terraform or OpenTofu so the next change is a reviewed pull request rather than a console click.</li>
</ul>



<p class="wp-block-paragraph">Send me your bucket policy and your identity pool role, with the account IDs redacted, and I&#8217;ll tell you what a signed-in patient can actually see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/secure-patient-document-portal-s3-cognito/">Building a Secure Patient Document Portal on S3 and Cognito</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Your WordPress 403 Isn&#8217;t the Rule You Think: Tuning ModSecurity False Positives</title>
		<link>https://john-nessime.com/blog/web-security/modsecurity-false-positives-wordpress/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 13 Oct 2026 06:06:40 +0000</pubDate>
				<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[WordPress]]></category>
		<category><![CDATA[Anomaly Scoring]]></category>
		<category><![CDATA[Apache]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[Coraza]]></category>
		<category><![CDATA[False Positives]]></category>
		<category><![CDATA[Firewall]]></category>
		<category><![CDATA[ModSecurity]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[OWASP CRS]]></category>
		<category><![CDATA[Rule Exclusions]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[Web Application Firewall]]></category>
		<category><![CDATA[WordPress Security]]></category>
		<category><![CDATA[wp-admin]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=434</guid>

					<description><![CDATA[<p>A 403 from ModSecurity almost never names the rule you need to change. Under anomaly scoring, detection rules only add to a score and a separate blocking rule returns the error. This guide covers reading the audit log backwards, choosing between the four exclusion mechanisms, the WordPress failure families that cause most false positives, and why raising the anomaly threshold quietly destroys your coverage.</p>
<p>The post <a href="https://john-nessime.com/blog/web-security/modsecurity-false-positives-wordpress/">Your WordPress 403 Isn&#8217;t the Rule You Think: Tuning ModSecurity False Positives</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket almost always reads the same way. Someone spent twenty minutes writing a post, hit Update, and got a bare <code>403 Forbidden</code>. Nothing in the WordPress debug log. Nothing in the PHP error log. The plugin they blame was not updated. The post they wrote yesterday saved fine.</p>



<p class="wp-block-paragraph">Then you open the ModSecurity audit log and find eleven rule matches on a single request, none of them obviously the culprit.</p>



<p class="wp-block-paragraph">Dealing with ModSecurity false positives on WordPress is not hard once you understand how the OWASP Core Rule Set decides to block. It is very hard if you go looking for &#8220;the rule that blocked me,&#8221; because under the default configuration that rule does not exist. This post covers reading the log, choosing the right exclusion mechanism, the WordPress failure families that generate most of the noise, and the tuning knobs that quietly gut your coverage.</p>



<h2 class="wp-block-heading">The invisible part: no single rule blocked the request</h2>



<p class="wp-block-paragraph">The Core Rule Set runs in anomaly scoring mode by default. Individual detection rules do not deny anything. They match, they log, and they add to a running score with <code>setvar</code>. At the end of the request phase, a separate blocking evaluation rule compares the accumulated score against a threshold and returns the 403.</p>



<p class="wp-block-paragraph">Scores come from rule severity. A CRITICAL match adds 5, ERROR adds 4, WARNING adds 3, NOTICE adds 2. The inbound threshold ships at 5 and the outbound threshold at 4. So one critical match is enough on its own, and so are two notices plus a warning.</p>



<p class="wp-block-paragraph">That is the failure mode nobody warns you about. The log line that produced your 403 lives in <code>REQUEST-949-BLOCKING-EVALUATION.conf</code> and it says the inbound anomaly score was exceeded. It does not name what matched. Take that ID and disable it and you have not fixed a false positive, you have turned off blocking for the whole site while leaving every detection rule running and logging. The site will look fixed. It is not protected.</p>



<p class="wp-block-paragraph">First rule of tuning, then: the blocking rule is a symptom. Read backwards from it.</p>



<h2 class="wp-block-heading">Read the log before you touch a rule</h2>



<p class="wp-block-paragraph">Every alert carries a <code>unique_id</code>. That value ties the error log line to the full audit log record for the same transaction, and it is the only reliable way to see every rule that fired on one request instead of a scattering of lines from different visitors.</p>



<pre class="wp-block-code"><code># Paths vary by build. Check SecAuditLog in modsecurity.conf for yours.
tail -n 200 /var/log/httpd/error_log | grep -i modsecurity

# Then pull the whole transaction using the unique_id from that output
grep -F 'Zm9vYmFyCg' /var/log/httpd/modsec_audit.log</code></pre>



<p class="wp-block-paragraph">From each match, four fields matter and the rest is noise:</p>



<ul class="wp-block-list">
<li><strong>The rule ID</strong>, as <code>[id "941320"]</code>. This is what you exclude.</li>



<li><strong>The matched variable</strong>, at the end of the pattern-match message as something like <code>at ARGS:content</code>. This is what you exclude it <em>from</em>.</li>



<li><strong>The URI</strong>, which tells you how narrowly you can scope the exclusion.</li>



<li><strong>The paranoia level tag</strong>, as <code>[tag "paranoia-level/2"]</code>. If every alert is tagged level 2 or higher, the real fix may be a paranoia level change rather than an exclusion.</li>
</ul>



<p class="wp-block-paragraph">One warning most guides skip: audit logs of false positives contain the payload that triggered them. If a user&#8217;s password happens to look like a shell command, it lands in your log in plaintext. Treat these files as sensitive, rotate them, and redact before pasting anything into a public bug report.</p>



<h2 class="wp-block-heading">Pick the right exclusion mechanism</h2>



<p class="wp-block-paragraph">ModSecurity gives you four ways to write an exclusion, differing on two axes: applied once at startup or evaluated per request, and removing the whole rule or just one variable from it.</p>



<ul class="wp-block-list">
<li><code>SecRuleRemoveById</code> and <code>SecRuleRemoveByTag</code> delete a rule or a whole tagged category at configure time. Blunt, cheap, global.</li>



<li><code>SecRuleUpdateTargetById</code> and <code>SecRuleUpdateTargetByTag</code> strip one variable from a rule at configure time, leaving the rule live for everything else.</li>



<li><code>ctl:ruleRemoveById</code> and <code>ctl:ruleRemoveByTag</code> do the same removal, but only for transactions matching a condition you write.</li>



<li><code>ctl:ruleRemoveTargetById</code> and <code>ctl:ruleRemoveTargetByTag</code> strip one variable, conditionally. This is the surgical option and the one to reach for first.</li>
</ul>



<p class="wp-block-paragraph">Runtime exclusions cost a little, since each is an extra rule evaluated on every request. That overhead is small next to the regex work the rule set already does, and the precision is worth it.</p>



<h3 class="wp-block-heading">Placement decides whether an exclusion works at all</h3>



<p class="wp-block-paragraph">This is where most people lose an afternoon. The two types go in opposite places, for mechanical reasons rather than arbitrary ones.</p>



<ul class="wp-block-list">
<li><strong>Configure-time exclusions go after the CRS include.</strong> They remove rules, and a rule has to exist before you can remove it.</li>



<li><strong>Runtime exclusions go before the CRS include.</strong> They change how rules behave for this transaction, and changing a rule after it has already run does nothing.</li>
</ul>



<p class="wp-block-paragraph">The Core Rule Set ships two files for exactly this, both with an <code>.example</code> suffix you drop before they load. <code>REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf</code> holds runtime exclusions, and <code>RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf</code> holds configure-time ones. Use them. Local files that survive a rule set update are the entire point.</p>



<h2 class="wp-block-heading">Five failure families behind most ModSecurity false positives</h2>



<p class="wp-block-paragraph">Almost everything you will hit on WordPress falls into one of these. Recognizing the family tells you the shape of the fix before you read a single log line.</p>



<h3 class="wp-block-heading">1. Post content that legitimately contains markup</h3>



<p class="wp-block-paragraph">WordPress lets editors put HTML in posts. It maintains its own allowed-tag list, sanitizes server side, and that mechanism has been reviewed hard over the years. The Core Rule Set knows none of this. It sees an <code>&lt;h1&gt;</code> or an inline <code>style</code> attribute in a POST parameter and matches an XSS rule.</p>



<p class="wp-block-paragraph">This produces the &#8220;it saved yesterday&#8221; ticket, because it is content-dependent. Plain paragraphs pass. A pasted table with attributes does not. The fix is a target exclusion scoped to the save endpoint and the specific parameter, never a global rule removal:</p>



<pre class="wp-block-code"><code># Runtime exclusion. Goes in REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
# Replace 941320 and the parameter with what your audit log actually shows.
SecRule REQUEST_URI "@beginsWith /wp-admin/post.php" 
    "id:1000,
    phase:1,
    pass,
    t:none,
    nolog,
    ctl:ruleRemoveTargetById=941320;ARGS:content"</code></pre>



<p class="wp-block-paragraph">Pick your own IDs from a range CRS will never use. Reusing an existing ID is a startup error rather than a silent one, which is the good kind of mistake.</p>



<h3 class="wp-block-heading">2. REST and admin-ajax calls that fail with no visible error</h3>



<p class="wp-block-paragraph">This family is nastier, because there is no error page. The browser hits <code>/wp-json/</code> or <code>/wp-admin/admin-ajax.php</code>, gets a 403, and the JavaScript either swallows it or shows a generic &#8220;updating failed&#8221; toast. Autosave stops. An editor panel stays empty. A form plugin silently drops submissions. Users report this as flakiness rather than a security block, so it sits unfixed for months. When someone says the editor is &#8220;just slow sometimes,&#8221; open the browser network tab and look for 403s before going anywhere near PHP.</p>



<p class="wp-block-paragraph">Scoping differs here. REST routes are structured, so use <code>@rx</code> against <code>REQUEST_FILENAME</code> to hit a route family rather than one path. For <code>admin-ajax.php</code>, the useful discriminator is usually the <code>action</code> parameter, since that one endpoint multiplexes dozens of unrelated plugin handlers.</p>



<h3 class="wp-block-heading">3. Media uploads that break at a size boundary</h3>



<p class="wp-block-paragraph">Multipart uploads have two independent failure paths that look identical from the browser. The first is a genuine rule match on the multipart structure, or on binary content containing a byte sequence some rule dislikes. That shows up with a rule ID and you tune it normally.</p>



<p class="wp-block-paragraph">The second is an engine limit, not a rule. <code>SecRequestBodyLimit</code> caps the total body ModSecurity will buffer, <code>SecRequestBodyNoFilesLimit</code> caps the non-file portion, and <code>SecRequestBodyLimitAction</code> decides whether an oversized body is rejected outright or inspected partially. If uploads fail above a consistent size and no CRS rule ID appears anywhere in the log, stop tuning rules. That is a limit, and the fix is in engine config.</p>



<p class="wp-block-paragraph">Raising them costs memory per concurrent request, and partial inspection leaves part of the body unscanned. Decide which trade you are making rather than bumping the number until the ticket closes.</p>



<h3 class="wp-block-heading">4. Cookies with dynamic names</h3>



<p class="wp-block-paragraph">WordPress authentication cookies carry a hash in the name, so a literal exclusion is impossible. Configure-time directives accept a regular expression in the target if you wrap it in forward slashes:</p>



<pre class="wp-block-code"><code># Configure-time exclusion. Goes in RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
SecRuleUpdateTargetById 942440 "!REQUEST_COOKIES:/^wordpress_logged_in_.*/"</code></pre>



<p class="wp-block-paragraph">The catch is significant: the <code>ctl</code> action supports no regular expressions at all. If you need a pattern match on a variable name, you are forced into the configure-time form, which applies site-wide rather than to one path. That single constraint drives more design decisions than people expect.</p>



<h3 class="wp-block-heading">5. Chained rules that refuse to be tuned</h3>



<p class="wp-block-paragraph">Some rules are written as a chain of conditions. Target exclusions only apply to the first rule in the chain, so if the variable you want to strip is referenced further down, nothing you write has any effect and you will burn an hour convincing yourself the config file is not loading.</p>



<p class="wp-block-paragraph">When a target exclusion visibly does nothing on a rule you can watch matching, check whether it is chained. If it is, your only option is removing the whole rule under a narrow condition. Accept the wider blast radius, or change the request instead.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Install the WordPress plugin before writing anything custom</h2>



<p class="wp-block-paragraph">The Core Rule Set maintains a WordPress rule exclusions package. In CRS 3 it was built in and switched on with a variable in <code>crs-setup.conf</code>. In CRS 4 it moved out into a separate plugin published in the CRS plugin registry, and it is not installed by default.</p>



<p class="wp-block-paragraph">That catches people upgrading. Exclusions that used to work vanish, WordPress starts throwing 403s again, and the setup variable they remember is gone from the config file. The reasoning behind the split is sound: fewer rules loaded means less code that can carry a bug, and one of the old exclusion packages did carry a bypass.</p>



<p class="wp-block-paragraph">Plugins load through wildcard includes in a specific order. Get the order wrong and the plugin silently does nothing:</p>



<pre class="wp-block-code"><code>Include /path/to/crs/crs-setup.conf

Include /path/to/crs/plugins/*-config.conf
Include /path/to/crs/plugins/*-before.conf

Include /path/to/crs/rules/*.conf

Include /path/to/crs/plugins/*-after.conf</code></pre>



<p class="wp-block-paragraph">Two limits before you lean on it. The plugin covers vanilla WordPress only, so false positives from third-party plugins, page builders and form plugins are still yours to tune. And the comment field is deliberately left out of the exclusions, because comment forms have a long history as a real attack surface. If your comment form throws false positives, that is a decision the maintainers made on purpose, and overriding it is a genuine risk trade rather than a gap in their package.</p>



<p class="wp-block-paragraph">On a server hosting more than one application, scope the plugin to the WordPress vhost using <code>WebAppID</code> or a Host header condition. WordPress exclusions applied to your unrelated API are a hole you did not mean to open.</p>



<h2 class="wp-block-heading">The knobs that look like a fix and are not</h2>



<p class="wp-block-paragraph">When exclusions get tedious, two shortcuts start looking attractive. Both cost more than they appear to.</p>



<p class="wp-block-paragraph"><strong>Raising the anomaly threshold.</strong> Going from 5 to 20 makes most false positives vanish, and it also means any attack category that scores low can never trigger a block. Detections that only ever produce a single critical match become unblockable. The CRS documentation is direct here: keep the threshold at the default and change the paranoia level instead. Raising it is a temporary tuning aid, not a configuration.</p>



<p class="wp-block-paragraph"><strong>Dropping the paranoia level.</strong> Legitimate, but often applied backwards. Level 1 is designed to produce few false positives. Levels 2 and up add progressively more aggressive rules and progressively more tuning work. If your alerts are all tagged level 2, dropping to level 1 is the correct fix and it is honest about what you gave up.</p>



<p class="wp-block-paragraph">The better tool for moving up is the split between blocking and detection paranoia levels. Set <code>tx.blocking_paranoia_level</code> to what you actually enforce and <code>tx.detection_paranoia_level</code> higher. Rules from the higher level execute and log, but only rules up to the blocking level count toward the score. You get to see what level 2 would have caught, in production, on real traffic, without anyone getting blocked. Then you tune, then you promote. The performance cost lands as soon as you raise the detection level, though, since those rules run either way.</p>



<p class="wp-block-paragraph">Two smaller settings help here. <code>tx.sampling_percentage</code> sends only a fraction of requests into the rule set, a gentler introduction for a busy site than blocking everything on day one. And <code>tx.reporting_level</code> can log requests that <em>would</em> have been blocked at a higher paranoia level, which is exactly the signal you want while climbing.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>The exclusion has no effect.</strong> Check placement first. A runtime exclusion loaded after the rules, or a configure-time exclusion loaded before them, fails silently. Then check for a chained rule. Then confirm the file is being included at all by breaking it deliberately with a syntax error and seeing whether the server refuses to start.</p>



<p class="wp-block-paragraph"><strong>It works on one site and not another.</strong> Different vhosts load different config. On control panels, per-domain overrides usually sit in a separate file from the global rule set.</p>



<p class="wp-block-paragraph"><strong>It worked and then stopped.</strong> Something updated the rule set. If you edited CRS rule files directly, your edits are gone, which is the whole argument against editing them.</p>



<p class="wp-block-paragraph"><strong>Nothing appears in the log at all.</strong> Confirm the rule engine is on, and that auditing is enabled with enough log parts to include the request body. An engine in detection only mode logs without blocking, which is confusing if you did not know you were in it.</p>



<p class="wp-block-paragraph"><strong>You are on shared hosting and cannot edit any of this.</strong> Some hosts expose a per-domain exclusion box, some let you disable rule IDs, some give you nothing. That is a real ceiling. Providers that hand you root, such as InterServer or Contabo, let you own the rule set and its update cadence, which on a site with genuine tuning needs beats the panel convenience you trade away.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Disabling the blocking evaluation rule because it showed up in the log. It reports the score, it detects nothing, and removing it disables blocking site-wide.</li>



<li>Editing CRS rule files directly. Every update reverts you, and you now maintain a fork by hand.</li>



<li>Excluding by message rather than by ID. Messages carry dynamic data and special characters, so the match is fragile.</li>



<li>Removing an entire attack tag because two rules misfired. <code>SecRuleRemoveByTag attack-sqli</code> is defensible when nothing behind the WAF touches SQL. On WordPress it is not.</li>



<li>Excluding <code>/wp-admin/</code> wholesale, or stacking multiple rule set vendors. Admin is where a compromised session does the most damage, and overlapping rule sets make attribution guesswork.</li>



<li>Leaving known false positives in place. Real attacks get lost in the noise, and the logs accumulate user data you never intended to collect.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Start in detection only. Collect real traffic, tune what you find, then enable blocking. Enforcing first is how you learn about broken checkout at two in the morning.</li>



<li>Install the WordPress exclusions plugin before writing custom rules. Do not hand-write what upstream already maintains.</li>



<li>Write the narrowest exclusion that resolves the alert: one rule, one variable, one path. Comment it with the rule ID, the endpoint, and why you judged it safe. Six months later that comment is the only thing between you and a blind revert.</li>



<li>Keep exclusion files in version control and deploy them with the rest of your config. Test rule set updates on staging, since new signatures land regularly.</li>



<li>Ship audit logs somewhere queryable. Grepping a growing text file stops scaling fast, and a platform like Grafana Cloud turns &#8220;which rule fires most&#8221; into a query instead of an afternoon.</li>



<li>If an edge WAF such as Cloudflare sits in front of the origin, decide deliberately which layer owns which rule. Two WAFs scanning the same body doubles your false positive surface for no extra coverage.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How do I find which ModSecurity rule blocked my WordPress request?</h3>



<p class="wp-block-paragraph">Take the <code>unique_id</code> from the 403 entry in the error log and grep the audit log for it. That gives you every rule that fired on the transaction. The blocking evaluation rule is what returned the 403, but the detection rules listed alongside it are what you tune.</p>



<h3 class="wp-block-heading">Should I disable ModSecurity for WordPress instead of tuning it?</h3>



<p class="wp-block-paragraph">It is a defensible choice if the alternative is blocking mode with unresolved false positives, since a WAF everyone has learned to route around is worse than none. But detection only mode gives you the logs with none of the breakage, and that is almost always the better middle ground while you work through the tuning.</p>



<h3 class="wp-block-heading">Is the WordPress rule exclusions plugin enough on its own?</h3>



<p class="wp-block-paragraph">For a stock install on a standard theme, usually. It covers vanilla WordPress and both block-based and classic themes. It does not cover third-party plugins, page builders, or theme frameworks, and those generate most of the false positives on a real site.</p>



<h3 class="wp-block-heading">Why do false positives come back after a rule set update?</h3>



<p class="wp-block-paragraph">Either you edited rule files directly and the update overwrote them, or the update shipped new signatures your existing exclusions do not cover. The first is avoidable by keeping every change in the exclusion files. The second is the cost of running a maintained rule set, and it is why updates belong on staging first.</p>



<h3 class="wp-block-heading">Does ModSecurity slow WordPress down?</h3>



<p class="wp-block-paragraph">It adds per-request work, mostly regular expression evaluation against request data, and the cost scales with paranoia level and body size. The settings that matter are the paranoia level, the request body limits, and whether response body inspection is on. Measure on your own traffic rather than trusting someone else&#8217;s benchmark.</p>



<h3 class="wp-block-heading">Is ModSecurity still maintained?</h3>



<p class="wp-block-paragraph">The engine moved to the OWASP Foundation after Trustwave stepped back from commercial support, and it still receives fixes, though feature work is slower than it once was. The Core Rule Set is a separate project on its own cadence and is actively developed. Coraza is a compatible engine running the same rule language, worth evaluating for new deployments.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">The thing to hold on to about ModSecurity false positives on WordPress is that the rule which returned your 403 is almost never the rule you need to change. Anomaly scoring separates detection from blocking on purpose, and reading the log backwards from the blocking decision to the detection rules that fed it is the whole skill.</p>



<p class="wp-block-paragraph">Everything else follows. Install the maintained WordPress exclusions instead of reinventing them. Write the narrowest exclusion you can, in the file that survives updates. Move the paranoia level, not the threshold.</p>



<h2 class="wp-block-heading">Need help tuning your WAF?</h2>



<p class="wp-block-paragraph">Most of the WAF work I take on is untangling exactly this: a rule set that is technically installed and practically ignored, because nobody had time to work through the false positives. I can help with:</p>



<ul class="wp-block-list">
<li>Triaging your audit logs and mapping every recurring false positive to the rule, variable and endpoint that produced it</li>



<li>Writing scoped exclusion sets for WooCommerce, form plugins, page builders and custom REST routes that upstream packages do not cover</li>



<li>Moving a site off direct rule file edits and onto exclusion files that survive rule set updates</li>



<li>Planning a CRS 3 to CRS 4 migration, including the exclusion package to plugin transition and the paranoia level variable renames</li>



<li>Running a staged rollout from detection only to blocking, with the reporting and sampling settings that make it safe</li>



<li>Shipping ModSecurity audit logs into a queryable log platform so rule tuning becomes a dashboard instead of a grep</li>
</ul>



<p class="wp-block-paragraph">If you have a blocked request you cannot explain, send me the audit log entry and your exclusion file and I will tell you what is firing and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/web-security/modsecurity-false-positives-wordpress/">Your WordPress 403 Isn&#8217;t the Rule You Think: Tuning ModSecurity False Positives</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks</title>
		<link>https://john-nessime.com/blog/devops/eks-version-upgrades-without-downtime/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 04 Oct 2026 06:06:28 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Kubernetes]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Amazon EKS]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Blue/Green Deployments]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cluster Autoscaler]]></category>
		<category><![CDATA[Cluster Insights]]></category>
		<category><![CDATA[CoreDNS]]></category>
		<category><![CDATA[Deprecated APIs]]></category>
		<category><![CDATA[Drift Detection]]></category>
		<category><![CDATA[EKS Auto Mode]]></category>
		<category><![CDATA[Graceful Shutdown]]></category>
		<category><![CDATA[Karpenter]]></category>
		<category><![CDATA[Node Autoscaling]]></category>
		<category><![CDATA[Node Lifecycle]]></category>
		<category><![CDATA[Pod Disruption Budgets]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[Rollback Plan]]></category>
		<category><![CDATA[Rolling Updates]]></category>
		<category><![CDATA[Runbooks]]></category>
		<category><![CDATA[VPC CNI]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=839</guid>

					<description><![CDATA[<p>The control plane bump is the part AWS handles and almost never breaks. The damage happens in the data plane rollout afterwards. A practical walk through the three failure families that actually stall EKS version upgrades: removed APIs you did not know you were calling, add-on updates that silently discard your configuration, and pod disruption budgets that deadlock the node drain, plus what each escape hatch really costs.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/eks-version-upgrades-without-downtime/">EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The control plane upgrade finished in about twenty minutes and nobody noticed. Then the managed node group update went to <code>Updating</code>, sat there, and came back with <code>PodEvictionFailure</code>. Half the group was on the new AMI and half on the old, and the thing blocking the drain was a PodDisruptionBudget somebody wrote eighteen months earlier for a two-replica deployment.</p>



<p class="wp-block-paragraph">That is the shape of most EKS upgrade incidents. The part everyone books a maintenance window for, the control plane bump, is the part AWS handles and almost never goes wrong. The damage happens in the data plane rollout, because of decisions made months earlier that nobody connected to upgrades.</p>



<p class="wp-block-paragraph">This post covers the three failure families that actually break EKS version upgrades: removed APIs you did not know you were calling, add-on updates that silently discard your configuration, and pod disruption budgets that deadlock the drain. For each one: what it looks like, how to catch it beforehand, and what the escape hatch costs.</p>



<h2 class="wp-block-heading">Why the control plane is the easy half</h2>



<p class="wp-block-paragraph">An EKS cluster has two versions that must stay compatible. AWS manages the control plane; you manage the data plane, which in practice means the kubelet version on your nodes.</p>



<p class="wp-block-paragraph">Two constraints shape everything else. In-place upgrades move one minor version at a time, so being three versions behind means three sequential upgrades and three data plane rollouts. And the kubelet cannot run ahead of the API server, so once the control plane moves, the clock starts on getting nodes to follow.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure one: the API calls you did not know you were making</h2>



<p class="wp-block-paragraph">Deprecation and removal are different events, and the gap between them is where people get comfortable. An API version gets deprecated, everything keeps working, the warning scrolls past in CI, and several releases later the version is gone and whatever depended on it starts getting 404s.</p>



<p class="wp-block-paragraph">Your own manifests are usually fine; someone fixed those when the warning appeared. What breaks is the third-party agent nobody owns: a metrics collector, an ingress controller, a Datadog or New Relic agent installed by a Helm chart pinned two years ago. These call the API on a watch loop, so they hit removed APIs immediately, and they tend to fail quietly rather than crash-loop.</p>



<h3 class="wp-block-heading">Cluster insights, and what it is actually reading</h3>



<pre class="wp-block-code"><code>aws eks list-insights --cluster-name my-cluster

aws eks describe-insight 
    --cluster-name my-cluster 
    --id a1b2c3d4-5678-90ab-cdef-EXAMPLE11111</code></pre>



<p class="wp-block-paragraph">EKS runs upgrade readiness checks and exposes them through the insights API. The deprecated API check reads your control plane audit logs on a thirty-day rolling window, scanned about once a day. That has a consequence people trip over constantly: after you fix a deprecated call, the insight stays in <code>ERROR</code> until the last matching log entry ages out. The work is done, the check still fails, and if you gate automation on insight status your pipeline is blocked for weeks on a solved problem. Force a fresh scan rather than waiting for the daily one:</p>



<pre class="wp-block-code"><code>aws eks start-insights-refresh --cluster-name my-cluster
aws eks describe-insights-refresh --cluster-name my-cluster</code></pre>



<p class="wp-block-paragraph">The refresh re-reads the window; it does not shorten it. Audit-log detection also only sees APIs that were actually called, so a monthly CronJob or a chart sitting unapplied in a repo never appears. Insights tells you what your cluster did, not what it could do.</p>



<h3 class="wp-block-heading">Ask the API server directly</h3>



<p class="wp-block-paragraph">The API server keeps its own counter, and it names the release that removes each API. Fastest check available:</p>



<pre class="wp-block-code"><code>kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis</code></pre>



<p class="wp-block-paragraph">Read the <code>removed_release</code> label first: it tells you whether this is urgent for the next upgrade or a warning for the one after. Scrape the metric into Prometheus, Grafana Cloud or Amazon Managed Grafana and you get a standing signal instead of a scramble.</p>



<p class="wp-block-paragraph">For the callers, go to the audit logs. With control plane audit logging on, CloudWatch Logs Insights gives you the identity behind each deprecated call:</p>



<pre class="wp-block-code"><code>fields @message
| filter `annotations.k8s.io/deprecated` = "true"</code></pre>



<p class="wp-block-paragraph">The <code>user.username</code> field is what you need. It usually points at a service account in a namespace you did not install, which is your cue to find the chart&#8217;s owner. Static scanners like <code>kubent</code> and <code>pluto</code> cover the other half, the manifests that have not run recently. Use both.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure two: add-ons and the conflict resolution you picked once</h2>



<p class="wp-block-paragraph">Managed add-ons cover what a cluster cannot run without: the VPC CNI, CoreDNS, kube-proxy, the EBS CSI driver. They are version-pinned separately from the cluster, and their compatible ranges move with the Kubernetes version. The failure here is rarely the version, though. It is the conflict resolution mode.</p>



<p class="wp-block-paragraph">When you update an add-on, EKS wants to apply its own field values. If you edited those fields in-cluster, that is a conflict, resolved by what you pass:</p>



<ul class="wp-block-list">
<li><code>NONE</code> leaves your value alone and the update may simply fail. Safe, loud, and it stops your upgrade.</li>



<li><code>OVERWRITE</code> reverts your customization to the EKS default. The update succeeds. Your custom Corefile, your CNI tuning, your tolerations, gone, with no error.</li>



<li><code>PRESERVE</code> keeps your value through the update.</li>
</ul>



<p class="wp-block-paragraph"><code>OVERWRITE</code> is the one that hurts, because it is also the one people reach for when an update fails and they want it to stop failing. It is the standard fix when migrating a self-managed add-on to a managed one, so it gets learned there and carried into the update path, where it means something different. A CoreDNS update with <code>OVERWRITE</code> will replace a Corefile carrying your stub zones and forwarders, and resolution for those zones stops cluster-wide while every health check stays green.</p>



<pre class="wp-block-code"><code>aws eks update-addon 
    --cluster-name my-cluster 
    --addon-name coredns 
    --addon-version &lt;compatible-version&gt; 
    --resolve-conflicts PRESERVE</code></pre>



<p class="wp-block-paragraph">One more trap: the same value means different things on create and on update. On create, <code>PRESERVE</code> behaves close to <code>NONE</code>, leaving an existing self-managed add-on alone rather than adopting your settings. Check the operation, not the word.</p>



<p class="wp-block-paragraph">The durable fix is to stop editing add-on resources with <code>kubectl</code> at all. Put customizations in the add-on&#8217;s configuration values so EKS owns them and there is nothing left to conflict over. An afternoon of work, and this failure family goes away permanently.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure three: PDB deadlocks and the fifteen-minute drain clock</h2>



<p class="wp-block-paragraph">This is the one that stalls upgrades halfway, and the mechanics matter because the fix depends on where you are stuck. A managed node group update runs in four phases: setup, scale up, upgrade, scale down. On the default strategy, scale up launches replacements before anything is terminated, which is what preserves capacity. Then the upgrade phase does this, per node:</p>



<ol class="wp-block-list">
<li>Pick a node to replace, up to the configured <code>maxUnavailable</code>.</li>



<li>Cordon it once a new node reaches <code>Ready</code>.</li>



<li>Drain the pods off it.</li>



<li>Wait sixty seconds after eviction, then ask the Auto Scaling group to terminate it.</li>



<li>Repeat until nothing is left on the old launch template version.</li>
</ol>



<p class="wp-block-paragraph">Step three has a fifteen minute budget. If pods have not left in that time and you did not pass the force flag, the whole update fails with <code>PodEvictionFailure</code> and you are left with a half-migrated node group.</p>



<h3 class="wp-block-heading">The deadlock nobody sees coming</h3>



<p class="wp-block-paragraph">A PDB says how many pods must stay available during voluntary disruption, and eviction is voluntary disruption. So the arithmetic is simple and, once you see it, obvious: a deployment with two replicas and a budget requiring two available pods can never be evicted. Not slowly. Never. The drain retries until the clock runs out.</p>



<p class="wp-block-paragraph">One replica with a one-pod minimum is the same deadlock in friendlier clothing, as is a percentage budget that rounds up to the full replica count. And if two PDBs select the same pod, the more restrictive one wins, producing deadlocks invisible in either manifest read alone. The pre-flight check is cheap, and I run it before every upgrade:</p>



<pre class="wp-block-code"><code>kubectl get pdb --all-namespaces</code></pre>



<p class="wp-block-paragraph">Read the <code>ALLOWED DISRUPTIONS</code> column, not the configured minimum. Any budget sitting at zero is a drain that will hang. Fix those before touching the node group, by raising replicas or loosening the budget so at least one disruption is permitted.</p>



<p class="wp-block-paragraph">There is a second cause of <code>PodEvictionFailure</code> with nothing to do with PDBs. The upgrade taints nodes as unschedulable, so after eviction the node should be empty. Anything tolerating every taint keeps it non-empty, and the drain never satisfies its exit condition. If your budgets look fine and eviction still stalls, look at what tolerates <code>NoSchedule</code> unconditionally.</p>



<h3 class="wp-block-heading">What the force flag actually costs</h3>



<pre class="wp-block-code"><code>aws eks update-nodegroup-version 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup 
    --force</code></pre>



<p class="wp-block-paragraph">Force does not negotiate with budgets. It deletes the pods and terminates the node regardless. For a stateless service behind a load balancer with several replicas, that is fine and the upgrade completes. For a quorum-based system it is a bad idea: if the budget in front of your etcd, Consul, Kafka or ZooKeeper pods is doing its job, forcing past it can take the cluster below quorum. The budget was the protection; force is the decision to ignore it.</p>



<p class="wp-block-paragraph">My rule is that force is reasonable on a node group I know holds only stateless workloads, and never a first response on a mixed one. Reaching for it because you do not know what is on the node is the actual problem.</p>



<h3 class="wp-block-heading">If you run Karpenter, the levers are different</h3>



<p class="wp-block-paragraph">Karpenter has no update phases. It detects drift when a NodePool or EC2NodeClass stops matching a node&#8217;s configuration, then replaces the drifted nodes. Disruption budgets control the pace and can be scheduled, which is how you build a maintenance window without booking one:</p>



<pre class="wp-block-code"><code>spec:
  disruption:
    budgets:
      - nodes: "0"
        reasons: ["Drifted"]
        schedule: "0 6 * * mon-fri"
        duration: 12h</code></pre>



<p class="wp-block-paragraph">Two things bite people. Schedules are evaluated in UTC, so a window that looks like business hours to you may not be. And any period not covered by a budget is unrestricted, so gaps are not &#8220;no disruption&#8221;, they are &#8220;unlimited disruption&#8221;.</p>



<p class="wp-block-paragraph">The other lever, <code>terminationGracePeriod</code> on the NodePool, overrides everything. Once it elapses, remaining pods are force-deleted regardless of PDBs or the <code>karpenter.sh/do-not-disrupt</code> annotation. It is a deadlock breaker, not a safety setting, and if it is shorter than your budget schedule you can get forced terminations inside a window you thought was protected.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The rollback window changes the calculation</h2>



<p class="wp-block-paragraph">Upstream Kubernetes has no control plane rollback. That single fact is why so many teams built heavyweight approval processes around upgrades. EKS now supports reverting an in-place control plane upgrade to the previous minor version within a seven day window, preserving etcd data and running workloads. It covers in-place upgrades only, which makes sense: with blue/green the old cluster is still there and rollback is a DNS change.</p>



<p class="wp-block-paragraph">What changes is the shape of the risk. An upgrade used to be a one-way door, so the rational response was exhaustive testing beforehand. Now the first week is the test, under real traffic, with a way back. That argues for upgrading more often on smaller jumps, and treating the week after as an active watch period rather than a closed ticket.</p>



<p class="wp-block-paragraph">It is not a general undo. Once the window closes, or you upgrade again, the previous version is gone, and it does not roll back add-ons or application state. Blue/green still has a place when you are several versions behind, since sequential in-place hops multiply exposure. The cost is running two clusters during migration, worth modelling first in Cost Explorer with both tagged, or in a tool like Vantage or CloudZero if you already run one.</p>



<h2 class="wp-block-heading">Troubleshooting a stalled or failed upgrade</h2>



<p class="wp-block-paragraph">When a node group update fails, the reason comes back as an error code. Start there rather than guessing:</p>



<pre class="wp-block-code"><code>aws eks describe-nodegroup 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup</code></pre>



<ul class="wp-block-list">
<li><strong>PodEvictionFailure.</strong> A budget blocked the drain, or something tolerates all taints. Check <code>ALLOWED DISRUPTIONS</code> first, then tolerations.</li>



<li><strong>NodeCreationFailure in scale up.</strong> New nodes never reached <code>Ready</code>. Usually no capacity for your instance type in an Availability Zone, an EC2 instance limit in the account, or custom user data breaking the bootstrap.</li>



<li><strong>Slow bootstrap.</strong> A node has fifteen minutes to join, measured from when the Auto Scaling group size increases. Heavy user data, large images pulled at boot or a slow init path push past it.</li>



<li><strong>Pods stuck terminating with no budget in sight.</strong> Look for blocked I/O. A pod with an NFS mount whose server is unreachable will not terminate, and eviction pressure changes nothing.</li>



<li><strong>Insights still in ERROR after the fix.</strong> The thirty day audit window. Refresh, confirm another way, move on.</li>
</ul>



<p class="wp-block-paragraph">One structural note: if the Cluster Autoscaler scales the node group during the scale down phase, the upgrade workflow exits without restoring the group to its original size. Check your desired and max counts afterwards.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating insights as the only pre-flight check. It reads audit logs, so it cannot see code paths that have not run.</li>



<li>Reaching for <code>OVERWRITE</code> to make a failing add-on update succeed. It succeeds by discarding your configuration.</li>



<li>Upgrading add-ons and the control plane in one unbroken pipeline run, so you cannot tell which change broke things.</li>



<li>Leaving <code>maxUnavailable</code> at its default of one on a large node group, then wondering why the rollout takes hours.</li>



<li>Forgetting self-managed components. Cluster Autoscaler, the AWS Load Balancer Controller and metrics-server are versioned against Kubernetes too, and nothing reminds you.</li>
</ul>



<h2 class="wp-block-heading">How I sequence EKS version upgrades</h2>



<ol class="wp-block-list">
<li>Read the release notes for the target version. Not the summary post, the removals section.</li>



<li>Run all three deprecation checks: insights for what the cluster did, the API server metric for what it does now, a static scanner for what sits in the repo.</li>



<li>Audit PDBs and fix every zero. Highest-value ten minutes in the process.</li>



<li>Update add-ons with <code>PRESERVE</code> to versions compatible with both the current and target Kubernetes version, so components talking to the API server are ready before it moves.</li>



<li>Upgrade the control plane and stop. Watch error rates and API server latency long enough to mean something.</li>



<li>Roll the least critical node group. Confirm pods reschedule cleanly and nothing depended on a node label or an AMI-specific detail.</li>



<li>Roll the rest, confirm every kubelet matches the control plane, then treat the following week as the real test. That is what the rollback window is for.</li>
</ol>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Can you do EKS version upgrades with zero downtime?</h3>



<p class="wp-block-paragraph">For stateless workloads with sensible replica counts, spread across zones, with budgets allowing at least one disruption, yes. The default node group strategy launches replacements before terminating anything, so capacity holds throughout. Stateful workloads can too, but only if the application handles rolling leadership changes and the budget permits exactly one disruption at a time.</p>



<h3 class="wp-block-heading">Why does the update fail with PodEvictionFailure when my PDBs look fine?</h3>



<p class="wp-block-paragraph">Two likely causes. Multiple budgets may select the same pod, in which case the most restrictive applies and neither manifest reveals it alone. Or something on the node tolerates every taint, so the node never becomes empty and the drain never completes. Check <code>ALLOWED DISRUPTIONS</code> across all namespaces, then check tolerations.</p>



<h3 class="wp-block-heading">Is the force update option safe?</h3>



<p class="wp-block-paragraph">Safe for stateless workloads with multiple replicas. Not safe for quorum-based systems, because it deletes pods regardless of the budget protecting quorum. Use it when you know what is on the node group and the answer is nothing that minds being killed.</p>



<h3 class="wp-block-heading">Does Karpenter make upgrades easier than managed node groups?</h3>



<p class="wp-block-paragraph">More automatic, which is not the same thing. Drift detection replaces outdated nodes without you running a command, but you inherit a different set of controls to get wrong: budgets evaluated in UTC, uncovered schedule gaps that mean unlimited disruption, and a grace period that overrides PDBs entirely. Neither is easier. They fail differently.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">EKS version upgrades do not fail at the moment you run the command. They fail on decisions already sitting in the cluster: a budget written for a two-replica deployment, an add-on customized with <code>kubectl</code> rather than through its configuration values, a chart pinned before an API was removed.</p>



<p class="wp-block-paragraph">Which means the pre-flight checks are the upgrade. Ten minutes reading <code>ALLOWED DISRUPTIONS</code> and the deprecated API metric tells you more about how tomorrow goes than any amount of care during the rollout. The rollback window gives you a second chance on the control plane. It will not un-delete a pod that force-eviction pulled out of a quorum.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your upgrade path?</h2>



<p class="wp-block-paragraph">Most of this work is unglamorous and specific. What I would actually do on a cluster facing a version bump:</p>



<ul class="wp-block-list">
<li>Audit every PDB against its replica count and flag the ones that will deadlock a drain.</li>



<li>Trace deprecated API calls back to the chart, controller or service account making them, from audit logs rather than guesswork.</li>



<li>Move add-on customizations into managed configuration values, so conflict resolution stops being a decision made under pressure.</li>



<li>Write a repeatable upgrade runbook: pre-flight checks, sequencing, and the rollback path for your specific workloads.</li>



<li>Alert on the deprecated API metric and node group update events, so the next version is a scheduled task rather than an incident.</li>



<li>Review Karpenter disruption budgets and grace periods for the gaps that turn into unplanned rollouts.</li>
</ul>



<p class="wp-block-paragraph">If you are staring at a failed node group update right now, send me the output of <code>kubectl get pdb --all-namespaces</code> and the error code from <code>describe-nodegroup</code>. That is usually enough to say what is holding the drain.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/eks-version-upgrades-without-downtime/">EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Property SaaS Document Storage: The Five Failures That Show Up in an Audit</title>
		<link>https://john-nessime.com/blog/cloud-security/property-saas-document-storage/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 28 Sep 2026 06:04:50 +0000</pubDate>
				<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Real Estate Technology]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[ABAC]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS STS]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[Cross-Tenant Leakage]]></category>
		<category><![CDATA[Crypto Shredding]]></category>
		<category><![CDATA[Data Minimization]]></category>
		<category><![CDATA[Document Retention]]></category>
		<category><![CDATA[Encryption Context]]></category>
		<category><![CDATA[File Upload Security]]></category>
		<category><![CDATA[GDPR]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[Object Storage]]></category>
		<category><![CDATA[Presigned POST]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[Property Management Software]]></category>
		<category><![CDATA[Real Estate Data]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[SaaS Architecture]]></category>
		<category><![CDATA[Secure Disposal]]></category>
		<category><![CDATA[Stored XSS]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=898</guid>

					<description><![CDATA[<p>Storing tenant IDs, leases and referencing packs is easy until someone asks who actually read them. Five failure families in property SaaS document storage, with the IAM policies, presigned URL mechanics, upload controls and deletion patterns that close each one.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/property-saas-document-storage/">Property SaaS Document Storage: The Five Failures That Show Up in an Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket that starts this is almost never &#8220;our document storage is insecure.&#8221; It&#8217;s a security questionnaire from a prospect, or a data subject access request, or an insurer asking one specific question: list everyone who has viewed this tenant&#8217;s ID document in the last twelve months.</p>



<p class="wp-block-paragraph">So you open the audit table. And you find it records who <em>requested</em> a link to the document. Not who opened it. The actual downloads never touched your application at all, because you handed out a signed storage URL and the browser went straight to the bucket. Your log is a record of intentions.</p>



<p class="wp-block-paragraph">That gap is typical of how property SaaS document storage goes wrong. The failures are rarely dramatic. Nobody dumps a bucket publicly. The guarantees you thought you had just turn out to be enforced in application code that ran once, long before the file was actually read. What follows is five failure families worth designing against when a platform stores tenant ID scans, signed leases, deposit certificates, right-to-rent evidence and referencing packs.</p>



<h2 class="wp-block-heading">What&#8217;s actually in the pile</h2>



<p class="wp-block-paragraph">Before architecture, be honest about the contents. A property platform&#8217;s document store is at least four risk classes wearing the same file extension:</p>



<ul class="wp-block-list">
<li><strong>Identity evidence.</strong> Passport and driving licence scans, visa documents, sometimes a selfie held next to the document. Highest blast radius, lowest genuine need for long retention.</li>



<li><strong>Contracts.</strong> Tenancy agreements, addenda, guarantor deeds, signed inventories. Legally driven retention, plus an integrity requirement: proving the file hasn&#8217;t changed matters more than hiding it.</li>



<li><strong>Financial and referencing material.</strong> Bank statements, payslips, credit reference reports. Sensitive, third-party sourced, and often covered by contractual restrictions from the referencing provider.</li>



<li><strong>Operational clutter.</strong> Inspection photos, safety certificates, maintenance invoices. Low sensitivity, high volume, and the thing that dominates your storage bill.</li>
</ul>



<p class="wp-block-paragraph">These want different retention clocks, different access rules and arguably different buckets. Treating them as one &#8220;documents&#8221; table is where most of the later pain originates.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure one: the access check and the access happen at different times</h2>



<p class="wp-block-paragraph">Presigned URLs are the right pattern. They keep the bucket private, take the transfer off your servers, and mean you never hand storage credentials to a browser. I reach for them first.</p>



<p class="wp-block-paragraph">The problem is what a presigned URL <em>is</em>. It&#8217;s a request you already authorised, signed, and then gave away. Anyone holding it can replay it until it expires. Your permission logic ran once and never runs again. Which means all of these keep working after the fact:</p>



<ul class="wp-block-list">
<li>A letting agent who left the company yesterday and whose account you disabled this morning.</li>



<li>A link forwarded into a WhatsApp group, pasted into a support ticket, or sitting in browser history on a shared office machine.</li>



<li>A document the tenant asked you to delete, if the object is still there behind a delete marker.</li>
</ul>



<h3 class="wp-block-heading">Make the URL die when the session does</h3>



<p class="wp-block-paragraph">Amazon&#8217;s own guidance is clearer on this than most people realise. A presigned URL created with temporary credentials expires when those credentials expire, <em>even if you asked for a longer expiry</em>. It also stops working if the signing credential is revoked, deleted or deactivated.</p>



<p class="wp-block-paragraph">That&#8217;s a lever, not a footnote. Sign with a long-lived IAM user key and the seven day maximum the SDK allows is a real seven days. Sign inside a short-lived assumed-role session and the URL cannot outlive that session, whatever number you passed in. So: never sign document URLs with static credentials. Assume a role per request, tag the session with the tenant, sign inside it.</p>



<pre class="wp-block-code"><code>import boto3

sts = boto3.client("sts")

# The tag key here must match the principal tag key used in the role's policy.
session = sts.assume_role(
    RoleArn="arn:aws:iam::111122223333:role/TenantDocumentAccess",
    RoleSessionName="doc-download",
    Tags=[{"Key": "TenantID", "Value": tenant_id}],
    DurationSeconds=900,
)["Credentials"]

s3 = boto3.client(
    "s3",
    aws_access_key_id=session["AccessKeyId"],
    aws_secret_access_key=session["SecretAccessKey"],
    aws_session_token=session["SessionToken"],
)

url = s3.generate_presigned_url(
    "get_object",
    Params={
        "Bucket": "prop-docs",
        "Key": f"tenants/{tenant_id}/identity/{doc_id}.pdf",
        "ResponseContentDisposition": 'attachment; filename="id-document.pdf"',
        "ResponseContentType": "application/octet-stream",
    },
    ExpiresIn=120,
)</code></pre>



<p class="wp-block-paragraph">Two things are doing work. <code>DurationSeconds</code> caps the real lifetime of the link regardless of <code>ExpiresIn</code>. And <code>ResponseContentDisposition</code> overrides how the object is served on this request, which matters for the serving problem further down.</p>



<p class="wp-block-paragraph">The trade-off is real: an STS call per download adds latency, adds a dependency, and STS has request limits you can hit at volume. AWS&#8217;s prescriptive guidance says as much. For identity documents I&#8217;d pay that cost. For inspection photos I wouldn&#8217;t.</p>



<h3 class="wp-block-heading">Log where the read actually happens</h3>



<p class="wp-block-paragraph">Your application log records issuance. To record retrieval you need logging at the storage layer: CloudTrail data events for object-level operations, or S3 server access logging. Data events are billed per event, so scope them to the identity and contract prefixes rather than the bucket holding thousands of inspection photos.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure two: isolation that lives in your ORM instead of your IAM policy</h2>



<p class="wp-block-paragraph">Ask most teams how tenant isolation works and you get a description of a WHERE clause. The key is built as <code>tenants/{tenant_id}/...</code> and the tenant ID comes from the session. Fine until one code path takes it from a request parameter, or a background job iterates without scoping, or someone writes an admin endpoint in a hurry.</p>



<p class="wp-block-paragraph">Move the boundary below your code. With attribute-based access control the tenant becomes a session tag and the IAM policy interpolates it into the resource ARN. One role serves every tenant, and the policy count doesn&#8217;t grow as you onboard.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadWriteOwnTenantPrefix",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::prop-docs/tenants/${aws:PrincipalTag/TenantID}/*"
    },
    {
      "Sid": "ListOnlyOwnTenantPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::prop-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": "tenants/${aws:PrincipalTag/TenantID}/*"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Two statements because S3 splits the problem. The first governs the object, where the tenant prefix is part of the resource ARN. The second governs listing, a bucket-level action, so the prefix has to be constrained through a condition instead. Skip it and a tenant can enumerate every key in the bucket even though they can&#8217;t read any of them, and key names in this domain leak plenty on their own.</p>



<p class="wp-block-paragraph">The failure mode worth internalising: if a code path forgets the session tag, <code>${aws:PrincipalTag/TenantID}</code> resolves to nothing and the statement doesn&#8217;t match. That&#8217;s a deny, which is what you want. But it only holds if the role has no broader fallback statement. One well-meaning <code>s3:GetObject</code> on the whole bucket, added to unblock a migration script, quietly turns the scheme into decoration.</p>



<h3 class="wp-block-heading">When to use a bucket per tenant instead</h3>



<p class="wp-block-paragraph">A bucket or an account per tenant is a genuinely harder boundary and I wouldn&#8217;t dismiss it. With a few dozen large agency customers it&#8217;s defensible and makes offboarding trivially provable. It stops being defensible once you onboard self-serve landlords, because bucket and account counts run into service quotas and the operational surface grows with every signup. Pick on tenant count, not on which sounds more secure. And note that ABAC is only as trustworthy as the tenant claim: if the caller can influence the value reaching the tagging call, you&#8217;ve built an elegant no-op.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure three: the upload and serve path</h2>



<p class="wp-block-paragraph">Uploads here come from people you have no relationship with yet. Prospective tenants, guarantors, contractors. That&#8217;s an untrusted channel by definition, and it produces two separate problems.</p>



<h3 class="wp-block-heading">Constrain the upload itself</h3>



<p class="wp-block-paragraph">A presigned PUT URL lets the holder write whatever they like to that key, at whatever size. Presigned POST is the better tool for public-facing uploads: the signature covers a policy document, and the policy carries conditions the browser cannot alter.</p>



<pre class="wp-block-code"><code>post = s3.generate_presigned_post(
    Bucket="prop-docs",
    Key=f"tenants/{tenant_id}/inbox/{upload_id}",
    Fields={"Content-Type": "application/pdf"},
    Conditions=[
        {"Content-Type": "application/pdf"},
        ["content-length-range", 1, 15728640],
    ],
    ExpiresIn=600,
)</code></pre>



<p class="wp-block-paragraph">The <code>content-length-range</code> condition is the one people skip, and it&#8217;s the one that stops a stranger using your referencing portal as free storage for a 40 GB file. A declared <code>Content-Type</code> is a claim, not a fact, so land uploads in a quarantine prefix, verify the actual bytes, and only then move them to the tenant prefix.</p>



<p class="wp-block-paragraph">Scan on that path too. ClamAV in a container is the zero-cost option and covers known malware; GuardDuty Malware Protection for S3 and hosted APIs such as Cloudmersive trade money for not maintaining it. What you&#8217;re protecting is mostly the letting agent who opens the attachment on a Windows machine, so this is a control for your customers rather than your servers.</p>



<h3 class="wp-block-heading">Never serve tenant documents from your application origin</h3>



<p class="wp-block-paragraph">This is the one that turns a document store into an account takeover. SVG and HTML are formats browsers execute. If a file uploaded by a prospective tenant is served inline from the same origin as your app, any script inside it runs with your application&#8217;s cookies and session context. A long stream of real advisories follows exactly this shape, particularly in self-hosted deployments where the object store sits on the app&#8217;s own domain.</p>



<p class="wp-block-paragraph">Three controls, and they stack rather than substitute:</p>



<ol class="wp-block-list">
<li><strong>Force download.</strong> Serve with <code>Content-Disposition: attachment</code> and <code>X-Content-Type-Options: nosniff</code>. Set these as object metadata at upload time <em>and</em> override them per request. The override protects links you mint; the stored metadata protects anything that bypasses your minting path.</li>



<li><strong>Use a separate origin.</strong> Put user content on a distinct domain, not a subdomain of the app. GitHub&#8217;s separate apex for user content is the reference implementation. Same-origin policy then contains the blast radius if control one is missed somewhere.</li>



<li><strong>Normalise where you can.</strong> Rendering an uploaded ID to a flattened image server side removes the executable content entirely. It costs CPU and fidelity. For identity documents, where you rarely need the original bytes back, it&#8217;s the right trade.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure four: deleting the document doesn&#8217;t delete the document</h2>



<p class="wp-block-paragraph">A tenant asks you to erase their passport scan. Your app sets a flag, the file disappears from the UI, and you reply confirming deletion. Then count the copies that still exist.</p>



<ul class="wp-block-list">
<li>The previous object versions, if versioning is on. A delete without a version ID just writes a delete marker.</li>



<li>Cross-region replicas with their own versioning and lifecycle rules, plus any backup or snapshot taken before the deletion.</li>



<li>Derivatives you generated and forgot: thumbnails, page previews, OCR text in a search index. Run a passport through Amazon Textract and the extracted fields are a copy of the sensitive data in a different shape.</li>



<li>Copies that left the platform. A document attached to a CRM record in something like GoHighLevel, forwarded by email, or downloaded to a letting agent&#8217;s laptop.</li>
</ul>



<p class="wp-block-paragraph">The ICO&#8217;s position is that a valid erasure request extends to backups as well as live systems, and that where backup data cannot be immediately overwritten it must be put beyond use: isolated, unused for anything else, deleted when that backup cycles. Workable, but only if you know which backup sets contain documents and how long they cycle. If you can&#8217;t answer that, you can&#8217;t answer the request.</p>



<h3 class="wp-block-heading">Crypto-shredding, and what it actually costs</h3>



<p class="wp-block-paragraph">The one technique that makes deletion provable across copies you can&#8217;t reach is a per-tenant KMS key. Encrypt that tenant&#8217;s objects under their own customer managed key; when they offboard, schedule the key for deletion. Every ciphertext copy anywhere, backups included, becomes undecryptable. You delete the data by deleting the ability to read it. The honest costs:</p>



<ul class="wp-block-list">
<li>KMS bills per key per month plus per request, so per-tenant keys are a real line item. Check current rates and quotas first: there&#8217;s an account limit on customer managed keys that a self-serve signup flow will eventually meet.</li>



<li>Key deletion has a mandatory waiting period. Crypto-shredding is not instant, and your erasure response has to say so.</li>



<li>It&#8217;s all-or-nothing per key. If a tenant asks you to erase one document rather than offboard, the key doesn&#8217;t help.</li>
</ul>



<h3 class="wp-block-heading">Object Lock fights erasure, on purpose</h3>



<p class="wp-block-paragraph">S3 Object Lock is tempting for signed tenancy agreements, because write-once storage is a strong integrity story. But compliance mode means nobody, root included, can delete the object until retention lapses. That is the feature. It is also a direct collision with an erasure request, and you cannot engineer your way out of it afterwards.</p>



<p class="wp-block-paragraph">My line: Object Lock on executed contracts, where a statutory retention basis overrides erasure, and never on identity documents or referencing material. Different buckets, different rules, decided before anything is written. For per-document erasure elsewhere, lifecycle rules that expire noncurrent versions on the sensitive prefixes are worth setting up before you need them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure five: storing what you never needed to store</h2>



<p class="wp-block-paragraph">The cheapest control on this list is not storing the file. The ICO is clear that UK GDPR sets no fixed retention periods and puts the burden on you to justify how long you keep personal data in identifiable form. That&#8217;s a design instruction, not a legal footnote.</p>



<p class="wp-block-paragraph">For identity documents, ask what your platform needs once the check is done. Usually it&#8217;s the result, not the evidence: document type, issuing country, expiry date, a check reference, who performed it and when. If your customer&#8217;s regulator requires the image itself, that&#8217;s a retention rule with a stated basis, which is a different conversation from keeping it because deleting felt risky.</p>



<p class="wp-block-paragraph">Practical version: give every document class a retention clock wired to a business event rather than upload date, and let lifecycle rules do the deleting. Tenancy end plus a defined period. Check completion plus a defined period. Not &#8220;whenever someone remembers.&#8221;</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the breakages you&#8217;ll actually hit</h2>



<h3 class="wp-block-heading">Everything broke after someone enabled S3 Bucket Keys</h3>



<p class="wp-block-paragraph">This one catches teams who did the sophisticated thing. If your IAM or KMS key policies use the object ARN as the encryption context to scope access, enabling an S3 Bucket Key changes that context to the <em>bucket</em> ARN and those policies stop matching. AWS documents it explicitly as something to fix before flipping the setting.</p>



<p class="wp-block-paragraph">The quieter side effect: KMS CloudTrail events then log the bucket ARN instead of the object ARN, and there are far fewer of them because S3 caches the bucket-level key. If you were treating KMS CloudTrail volume as a proxy for document access, that signal degrades without anything appearing to break. Bucket Keys are still worth enabling for the request cost reduction. Move the conditions first, then re-check what your dashboards count.</p>



<h3 class="wp-block-heading">The presigned URL expires far sooner than requested</h3>



<p class="wp-block-paragraph">Almost always the signing credential is a role session shorter than the expiry you asked for. This is documented behaviour, not a bug. Either raise the session duration within the role&#8217;s maximum, or accept the shorter window as the security property it is.</p>



<h3 class="wp-block-heading">Listing works, reading returns Access Denied</h3>



<p class="wp-block-paragraph">Classic split between the bucket-level and object-level statements. Check both exist and reference the same tag key spelling. Session tag keys are case sensitive, and a <code>tenantId</code> in the code against a <code>TenantID</code> in the policy fails silently as a non-match.</p>



<h3 class="wp-block-heading">Access denied only for encrypted objects</h3>



<p class="wp-block-paragraph">An SSE-KMS object needs two grants, not one. The S3 action on the object, and <code>kms:Decrypt</code> on the key it was encrypted under. In a per-tenant key setup that second grant is usually also tag-conditioned, so a missing or mismatched tag on the key itself produces a denial that looks like an S3 problem and isn&#8217;t.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Signing document URLs with a long-lived IAM user access key, which makes every link independently long-lived.</li>



<li>Granting the tenant role a broad bucket-wide read statement alongside the tag-scoped one, usually to unblock a one-off script that then stayed.</li>



<li>Serving user uploads from a subdomain of the application and assuming that counts as origin separation. It doesn&#8217;t for cookie scope.</li>



<li>Enabling versioning for safety and never adding a noncurrent version expiry rule, so every deletion is reversible forever.</li>



<li>Building deletion that handles the object but not the OCR output, the thumbnail, or the search index entry.</li>



<li>Putting Object Lock on the bucket that also holds identity documents.</li>
</ul>



<h2 class="wp-block-heading">Design rules worth holding to</h2>



<ol class="wp-block-list">
<li><strong>Separate buckets by risk class,</strong> not by feature. Identity, contracts, financial, operational. Different encryption, retention, logging and lock settings follow naturally from that split.</li>



<li><strong>Enforce isolation in IAM,</strong> with application checks as defence in depth rather than the boundary itself.</li>



<li><strong>Sign short, sign from a session.</strong> Minutes, not days, and always from assumed-role credentials so revocation actually revokes.</li>



<li><strong>Deny unencrypted writes at the bucket policy,</strong> using the server-side encryption request headers as conditions, so a misconfigured client fails loudly instead of writing plaintext.</li>



<li><strong>Give every document class an expiry clock</strong> tied to a business event, implemented in lifecycle rules rather than a cron job someone maintains.</li>



<li><strong>Write the deletion runbook before launch,</strong> listing every place a copy lands. If you can&#8217;t enumerate them, that&#8217;s the finding.</li>



<li><strong>Test cross-tenant access in CI.</strong> A negative test that asserts tenant A&#8217;s session cannot read tenant B&#8217;s key is worth more than any amount of policy review.</li>
</ol>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How long should a presigned URL for a tenant ID document last?</h3>



<p class="wp-block-paragraph">Short enough that the user clicks it immediately. A couple of minutes covers a click-to-download flow. If the UI needs longer, re-mint on demand rather than extend. And remember the ceiling is set by the signing credential, so a short assumed-role session enforces the limit even if a later code change passes a bigger number.</p>



<h3 class="wp-block-heading">Do I need AWS, or will Backblaze B2 or Cloudflare R2 do?</h3>



<p class="wp-block-paragraph">They all speak the S3 API and presigned URLs work on all of them, so the core patterns transfer. Wasabi, Backblaze B2, Cloudflare R2 and DigitalOcean Spaces are reasonable for the operational tier, and R2&#8217;s egress model is attractive if agents download a lot. They differ in the depth of the IAM story: session-tag ABAC and per-tenant KMS keys are AWS-specific. If isolation and crypto-shredding are hard requirements, keep the sensitive prefixes on S3 even when the bulk photos live somewhere cheaper.</p>



<h3 class="wp-block-heading">Can I store scanned passports on a normal VPS instead?</h3>



<p class="wp-block-paragraph">You can, and plenty of platforms run the app tier on a VPS from Contabo or InterServer while keeping documents in object storage. What you lose on local disk is the deletion story and the access log. Proving an image is gone from a filesystem, its backups and its snapshots is far harder than scheduling a KMS key for deletion. If documents stay local, budget real effort for secure disposal, including on staff machines where tools like O&amp;O SafeErase have a place.</p>



<h3 class="wp-block-heading">What about e-signed tenancy agreements?</h3>



<p class="wp-block-paragraph">With a provider like DocuSign, the executed document and its audit trail live there as well as in your platform. That&#8217;s a second copy under a second retention regime, and it belongs in your deletion runbook and data map. It also strengthens your integrity story, which is a fair reason to skip Object Lock on contracts you retrieve from the provider.</p>



<h3 class="wp-block-heading">Should the CDN cache tenant documents?</h3>



<p class="wp-block-paragraph">Not the sensitive classes. A CDN in front of private documents adds a cache that can serve content after your access decision changed, and header handling at the edge is a common source of the inline-rendering bug. CloudFront with origin access control and signed URLs is legitimate if you need the performance, but for identity documents I&#8217;d go straight to the bucket. For inspection photos, cache away.</p>



<h3 class="wp-block-heading">How do I prove any of this to a prospect&#8217;s security team?</h3>



<p class="wp-block-paragraph">Artefacts, not assertions. The bucket policy, the ABAC policy, CI output showing cross-tenant denial, a sample of storage-layer access logs, and a retention schedule with a clock per document class. That set answers most questionnaires without a call.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Good property SaaS document storage is mostly about closing the gap between when you check access and when access happens. Every failure family above is a version of that gap: the presigned link that outlives the permission, the isolation enforced once in code and never at the storage layer, the deletion that updates a flag while the bytes sit behind a delete marker.</p>



<p class="wp-block-paragraph">Close it by pushing the decision down: into IAM policies that evaluate on every request, into credentials short enough that revocation is real, into lifecycle rules that delete without anyone remembering to. Then the answer to &#8220;show us who read this tenant&#8217;s ID document&#8221; is a query, not a project.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your document storage?</h2>



<p class="wp-block-paragraph">Most of what I do on property and lettings platforms is this exact work. Things I&#8217;d help with here:</p>



<ul class="wp-block-list">
<li>Reviewing bucket and IAM policies for cross-tenant read paths, including the ones added to unblock a migration and never removed.</li>



<li>Moving tenant isolation from application code into session-tag ABAC, with a CI test that fails the build on cross-tenant access.</li>



<li>Rebuilding the upload path with presigned POST conditions, quarantine, content verification and scanning before anything lands in a tenant prefix.</li>



<li>Fixing the serving path so uploaded files can&#8217;t execute against your origin, including header handling through a CDN.</li>



<li>Designing retention clocks and lifecycle rules per document class, plus a deletion runbook that names every copy including derivatives and third-party systems.</li>



<li>Setting up storage-layer access logging scoped to the sensitive prefixes so you can actually answer an access request.</li>
</ul>



<p class="wp-block-paragraph">If it&#8217;s useful, send me your bucket policy and the IAM policy attached to your document role, with the account IDs redacted, and I&#8217;ll tell you what a determined tenant could reach.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/property-saas-document-storage/">Property SaaS Document Storage: The Five Failures That Show Up in an Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Cognito MFA for Healthcare Apps: The Settings That Look Compliant but Aren&#8217;t</title>
		<link>https://john-nessime.com/blog/cloud-computing/cognito-mfa-healthcare-apps/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 30 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Account Recovery]]></category>
		<category><![CDATA[Adaptive Authentication]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Artifact]]></category>
		<category><![CDATA[Business Associate Agreement]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[ePHI]]></category>
		<category><![CDATA[Healthcare Cloud]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[Identity Propagation]]></category>
		<category><![CDATA[MFA]]></category>
		<category><![CDATA[OIDC]]></category>
		<category><![CDATA[Passkeys]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[Session Management]]></category>
		<category><![CDATA[TOTP]]></category>
		<category><![CDATA[WebAuthn]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=348</guid>

					<description><![CDATA[<p>A user pool set to "MFA: Required" can still let clinicians in on a password alone. This is a practical guide to Cognito MFA for healthcare apps: why federated users bypass enforcement, how the account recovery collision locks people out, why required MFA and adaptive authentication cannot coexist, and how to make session lifetimes match what your documentation claims.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cognito-mfa-healthcare-apps/">Cognito MFA for Healthcare Apps: The Settings That Look Compliant but Aren&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The finding came back from a penetration test on a patient portal: the tester signed in as a clinical user with nothing but a username and password. No second factor. No prompt. Straight into a screen with appointment history on it.</p>



<p class="wp-block-paragraph">The user pool was set to <code>MFA: Required</code>. Somebody had screenshotted that console page for the security questionnaire. The screenshot was accurate. It just didn&#8217;t mean what everyone assumed it meant.</p>



<p class="wp-block-paragraph">That gap is the whole problem with Cognito MFA for healthcare apps. The service gives you a setting that reads like a guarantee, and then quietly carves out entire classes of users, sessions, and sign-in paths that the setting never touches. Nothing errors. Nothing logs a warning. Your compliance evidence looks fine right up until someone actually tests it.</p>



<p class="wp-block-paragraph">This post covers what the MFA switch in Amazon Cognito actually controls, the five failure families that show up in real healthcare deployments, how to wire a user pool so the enforcement matches the claim, and what to check when a clinician calls the help desk because they can&#8217;t get in.</p>



<h2 class="wp-block-heading">What HIPAA actually requires from MFA right now</h2>



<p class="wp-block-paragraph">Worth getting this straight before touching any configuration, because a lot of vendor content is confidently wrong about it.</p>



<p class="wp-block-paragraph">The HIPAA Security Rule as it stands today does not name multi-factor authentication. It requires person or entity authentication, and it leaves the mechanism to your risk analysis. MFA is how most people satisfy that in practice, but the rule does not spell it out.</p>



<p class="wp-block-paragraph">What changes the picture is the Notice of Proposed Rulemaking that HHS published in the Federal Register on January 6, 2025. It would make MFA an explicit requirement for access to systems handling ePHI, with narrow exceptions, and it would remove the &#8220;addressable&#8221; escape hatch that currently lets organizations document a control as infeasible. This one matters enough to name the date, because timelines have moved: the comment period closed in March 2025, the rule is still proposed, and the Unified Agenda has pushed anticipated final action out to 2027. That estimate is not binding and has already slipped once.</p>



<p class="wp-block-paragraph">Practical read: build for the proposed rule now, but don&#8217;t tell a client the mandate is already law. It isn&#8217;t, and auditors know the difference.</p>



<p class="wp-block-paragraph">On the AWS side, Amazon Cognito is on the HIPAA Eligible Services list and is covered by the AWS Business Associate Agreement. Eligibility is a starting line, not a finish line. You still have to accept the BAA in each AWS account that touches PHI, and you still have to configure the service correctly. Nothing about creating a user pool makes it compliant.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: half your users never see the MFA prompt</h2>



<p class="wp-block-paragraph">This is the one that produced the pen test finding, and it&#8217;s the most common serious gap I run into.</p>



<p class="wp-block-paragraph">Cognito user pool MFA applies to <em>local</em> users. Those are the accounts that live in the user pool directory and authenticate against it. For <em>federated</em> users, the ones arriving through SAML or OIDC from a hospital identity provider, Okta, Entra ID, or an in-house ADFS, Cognito hands the entire authentication decision to the upstream IdP. It does not add a second factor. It cannot. It never sees the password.</p>



<p class="wp-block-paragraph">So a portal with patients as local users and staff federated from a hospital directory has exactly the split you&#8217;d fear: patients get MFA, clinicians don&#8217;t, unless the hospital IdP is enforcing it independently. And the hospital IdP is usually somebody else&#8217;s system, run by a team you don&#8217;t have a Slack channel with.</p>



<p class="wp-block-paragraph">There&#8217;s a second version of this. The app client, not the user pool, controls which authentication flows are permitted. If a legacy mobile client is still allowed to use a flow you thought you&#8217;d retired, that path stays open regardless of what the pool-level MFA setting says.</p>



<p class="wp-block-paragraph">What to do about it:</p>



<ul class="wp-block-list"><li>Inventory every app client attached to the pool. Not the ones you remember creating. Run <code>list-user-pool-clients</code> and account for all of them.</li><li>For each federated IdP, get written confirmation of what the upstream enforces. A trust relationship is not a control. Put the answer in your risk analysis where an auditor will find it.</li><li>Treat federation as a boundary you inherit risk across, and say so out loud in design reviews. This is the single most useful sentence you can contribute to a healthcare identity project.</li></ul>



<h2 class="wp-block-heading">Failure family two: the recovery attribute collision</h2>



<p class="wp-block-paragraph">This one is genuinely subtle, and it doesn&#8217;t bite during development. It bites during rollout, at scale, to the people least able to tolerate it.</p>



<p class="wp-block-paragraph">Cognito refuses to let a user&#8217;s MFA delivery method be the same channel as their account recovery method. The logic is sound: if a password reset code and a second factor both land in the same inbox, an attacker with that inbox has everything, and the second factor is decoration.</p>



<p class="wp-block-paragraph">The consequence is where it gets sharp. If MFA is required, a user has only an email address and no phone number, email is your account recovery channel, and they have not registered a TOTP authenticator, then that user <strong>cannot complete sign-in at all</strong>. Cognito returns an error. They are not locked out temporarily. There is no path forward from the login screen.</p>



<p class="wp-block-paragraph">Now think about who that describes in a healthcare deployment. Part-time nursing staff enrolled from an HR export that had work email but no mobile number. Older patients who gave an email at registration and nothing else. Contractors. Locums. It is never a small number, and they all arrive at the help desk in the same week.</p>



<p class="wp-block-paragraph">The same rule cuts the other way on password reset. A user whose preferred MFA is email cannot receive a reset code by email. If they have no phone number, self-service recovery is gone, and someone with administrative credentials has to intervene for every single one.</p>



<p class="wp-block-paragraph">Three ways out, in the order I&#8217;d reach for them:</p>



<ul class="wp-block-list"><li><strong>Make TOTP the default factor.</strong> An authenticator app doesn&#8217;t consume either the email or phone attribute, so the collision never occurs. This is the one I reach for first in staff-facing apps.</li><li><strong>Require both email and phone_number as sign-up attributes.</strong> When a user has both, Cognito automatically routes reset codes to whichever channel isn&#8217;t their MFA factor. Clean, but it forces a phone number on people who may not want to give one.</li><li><strong>Configure a recovery fallback.</strong> Set account recovery to prefer email but fall back to SMS, so users who aren&#8217;t eligible for email recovery still have a route.</li></ul>



<p class="wp-block-paragraph">Whatever you choose, run a query against your existing user records for accounts missing a phone number before you flip MFA to required. Finding those users in a report is cheap. Finding them through a help desk queue is not.</p>



<h2 class="wp-block-heading">Failure family three: required MFA and adaptive authentication are mutually exclusive</h2>



<p class="wp-block-paragraph">This trips up almost every security-conscious design I review, because both features are things a healthcare client will ask for by name.</p>



<p class="wp-block-paragraph">Cognito&#8217;s threat protection includes adaptive authentication: risk scoring on each sign-in, with automated responses like requiring MFA or blocking outright. To use those automated responses in full-function mode, <strong>MFA must be set to optional in the user pool</strong>. That is a documented constraint, not a bug. The mechanism only makes sense if there&#8217;s a state where MFA is not already being demanded.</p>



<p class="wp-block-paragraph">So you cannot check both boxes. Either MFA is required for everyone and adaptive responses have nothing to escalate to, or MFA is optional and risk scoring can escalate, which means low-risk sign-ins may proceed on a password alone.</p>



<p class="wp-block-paragraph">For anything touching ePHI, I&#8217;d take required MFA and give up the adaptive escalation. A control that always applies beats a control that usually applies, and &#8220;usually&#8221; is a hard thing to defend in an audit.</p>



<p class="wp-block-paragraph">You don&#8217;t have to give up the visibility, though. Threat protection runs in audit-only mode alongside required MFA. You still get risk signals, device fingerprinting, and compromised credential detection surfaced in CloudWatch and in the user event logs, without the enforcement conflict. That&#8217;s a genuinely good trade: enforcement from the required setting, detection from threat protection.</p>



<p class="wp-block-paragraph">Two things to know before you budget for it. Threat protection lives in the Plus feature tier, and audit-only mode is billed the same as full-function. There&#8217;s no free observation period. Export those events somewhere you&#8217;ll actually look at them, whether that&#8217;s a CloudWatch Logs destination feeding your SIEM, or a platform like Datadog or Grafana Cloud if that&#8217;s where your on-call already lives.</p>



<h2 class="wp-block-heading">Failure family four: the session outlives the shift</h2>



<p class="wp-block-paragraph">MFA protects the moment of sign-in. It does nothing about the eight hours after it. In healthcare that gap is unusually expensive, because the workstation is shared, the browser stays open, and the next person to touch the keyboard is a different human being.</p>



<p class="wp-block-paragraph">Cognito&#8217;s defaults are built for consumer apps. Access and ID tokens last one hour. Refresh tokens last thirty days. Neither default is wrong in general and both are wrong for a clinical workstation.</p>



<p class="wp-block-paragraph">The ranges you have to work with:</p>



<ul class="wp-block-list"><li>Access token and ID token: five minutes to one day, set per app client.</li><li>Refresh token: sixty minutes to ten years, set per app client. It cannot be shorter than the access token lifetime.</li></ul>



<p class="wp-block-paragraph">Now the part that catches people, and the reason I put this failure family on the list. <strong>If you use managed login, Cognito sets a browser session cookie valid for one hour.</strong> Setting your access token to fifteen minutes does not shorten the re-authentication window. During that hour the user&#8217;s session refreshes without presenting credentials again, because the cookie is still good. Teams configure short tokens, write &#8220;15-minute session timeout&#8221; in the security documentation, and the observable behavior is an hour.</p>



<p class="wp-block-paragraph">The second trap is revocation. Revoking a token, or a global sign-out, invalidates refresh tokens. It does not invalidate an access token that has already been issued. That token stays valid until it expires, and any API accepting it will keep accepting it. So the practical blast radius of a compromised session is your access token TTL, not the moment you clicked &#8220;sign out this user.&#8221;</p>



<p class="wp-block-paragraph">That&#8217;s the argument for short access tokens: it is the only lever that shortens the window between &#8220;we revoked access&#8221; and &#8220;access actually stopped.&#8221;</p>



<pre class="wp-block-code"><code># Tighten an app client used by clinical staff.
# Access tokens live 15 minutes; refresh tokens expire at the end of a shift.
aws cognito-idp update-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --access-token-validity 15 
  --id-token-validity 15 
  --refresh-token-validity 9 
  --token-validity-units AccessToken=minutes,IdToken=minutes,RefreshToken=hours 
  --enable-token-revocation</code></pre>



<p class="wp-block-paragraph">Refresh token rotation is worth turning on where your client can support it. Each refresh returns a new refresh token and invalidates the old one, so a stolen refresh token is usable once at most. Two caveats that bite in practice: rotation does not extend the original lifetime, so a nine-hour refresh token still dies nine hours after sign-in no matter how often it rotated, and app clients with rotation enabled cannot use the older <code>REFRESH_TOKEN_AUTH</code> flow. That second one has broken more than one mobile release. Rotation requires the Essentials tier or higher.</p>



<p class="wp-block-paragraph">None of this replaces workstation-level controls. Screen lock policy, OS session timeouts, and endpoint hardening tooling like the utilities from O&amp;O Software do work your identity layer cannot do from the outside. If staff reach the admin console remotely, a managed VPN such as NordLayer or Surfshark reduces network exposure, but be honest in your documentation about what that is. It is not an authentication control, and treating it as one is exactly the reasoning auditors push back on.</p>



<h2 class="wp-block-heading">Failure family five: the factor you picked is phishable</h2>



<p class="wp-block-paragraph">Cognito offers SMS, email, and TOTP as second factors, plus passkeys as a first factor under specific conditions. They are not equivalent, and the console presents them as though they were.</p>



<p class="wp-block-paragraph"><strong>SMS</strong> is the weakest. SIM swap and SS7 interception are real, and an SMS code is trivially relayed through a phishing proxy. It has one genuine advantage worth naming: enrollment friction is near zero, and for elderly or low-technology-comfort patient populations that is not a small thing. An SMS factor that gets used beats a TOTP factor that gets abandoned. Just don&#8217;t put clinical staff behind it. There&#8217;s also an operational trap: if your account is in the Amazon SNS SMS sandbox in the relevant region, you must verify each destination number before delivery works, which turns a smooth rollout into a stalled one.</p>



<p class="wp-block-paragraph"><strong>Email OTP</strong> is roughly SMS-equivalent in phishing resistance and worse if the mailbox is on the same laptop as the session you&#8217;re protecting. It needs the Essentials tier or higher and a configured Amazon SES sending identity. Useful as a fallback, weak as a primary.</p>



<p class="wp-block-paragraph"><strong>TOTP</strong> is the sensible default for staff. It&#8217;s phishable, the code can be relayed in real time by an attacker-in-the-middle, but it&#8217;s immune to SIM swap and it doesn&#8217;t collide with your account recovery channel. Any standard authenticator works. If your organization already uses a password manager with built-in TOTP, such as 1Password or Bitwarden, enrollment gets meaningfully easier, at the cost of both factors living in one vault. That&#8217;s a trade-off to make deliberately rather than by accident.</p>



<p class="wp-block-paragraph"><strong>Passkeys</strong> are the only phishing-resistant option here, and their integration has a shape you need to understand. A passkey cannot be a second factor on top of a password in Cognito. It can satisfy the MFA requirement on its own, as a first factor, when you set <code>FactorConfiguration</code> to <code>MULTI_FACTOR_WITH_USER_VERIFICATION</code> in the pool&#8217;s WebAuthn configuration. The user verification step, biometric or device PIN, is what makes it count as multi-factor.</p>



<p class="wp-block-paragraph">That&#8217;s the path I&#8217;d push for privileged accounts: administrators, anyone with bulk record export, anyone who can change the pool configuration itself. Platform authenticators cover most staff; hardware keys like YubiKey are worth the cost for the handful of accounts whose compromise ends the conversation.</p>



<p class="wp-block-paragraph">One hard constraint to plan around: you cannot set MFA to required in a pool that allows email or SMS one-time passwords as a <em>first</em> factor. Passwordless OTP sign-in and required MFA are incompatible. Passkeys with user verification are the documented exception.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring up Cognito MFA for healthcare apps</h2>



<p class="wp-block-paragraph">An order of operations that avoids locking people out. Do this in a non-production pool first, with synthetic users only.</p>



<ol class="wp-block-list"><li><strong>Accept the BAA</strong> in the AWS account through AWS Artifact, before any real user data lands in the pool. The BAA is per-account and is not inherited by new accounts.</li><li><strong>Audit your user records</strong> for accounts missing <code>phone_number</code> or <code>email</code>. This is the population that will fail after you enable required MFA.</li><li><strong>Decide the recovery configuration</strong> and set it explicitly, with a fallback channel, so the MFA factor and the recovery channel can never be the same for any user.</li><li><strong>Enable MFA methods with the pool set to optional first.</strong> Let users enroll voluntarily. Track enrollment coverage as a number you report weekly.</li><li><strong>Flip to required only when coverage is high</strong> and the stragglers have a support path. Managed login will prompt unenrolled users to set up a factor at next sign-in, but only when MFA is required.</li><li><strong>Tighten token lifetimes per app client</strong>, then verify the observable behavior rather than the configuration. If you use managed login, remember the one-hour cookie.</li><li><strong>Turn on threat protection in audit-only mode</strong> and stream the events out to wherever you review security signal.</li></ol>



<p class="wp-block-paragraph">The core enforcement setting, once you&#8217;re ready:</p>



<pre class="wp-block-code"><code># Require MFA pool-wide, TOTP as the available method.
aws cognito-idp set-user-pool-mfa-config 
  --user-pool-id us-east-1_EXAMPLE 
  --mfa-configuration ON 
  --software-token-mfa-configuration Enabled=true

# Set a specific user's preferred factor administratively.
aws cognito-idp admin-set-user-mfa-preference 
  --user-pool-id us-east-1_EXAMPLE 
  --username clinician@example.org 
  --software-token-mfa-settings Enabled=true,PreferredMfa=true</code></pre>



<p class="wp-block-paragraph">Note that <code>admin-set-user-mfa-preference</code> does not reset an existing TOTP registration. To issue a new authenticator secret for a user who lost their phone, you go through <code>AssociateSoftwareToken</code> and <code>VerifySoftwareToken</code>. Build that runbook before you need it, because you will need it, and improvising an identity-proofing procedure at 2pm on a Tuesday is how bad help desk habits start.</p>



<p class="wp-block-paragraph">A staging pool holding only synthetic patients doesn&#8217;t need to live inside the BAA account, and there&#8217;s no reason a test environment should carry production&#8217;s cost structure. A plain VPS from InterServer or Contabo is fine for the app tier in front of a throwaway pool, provided you are certain no real PHI can reach it. That certainty is the whole condition, and it&#8217;s worth writing down rather than assuming.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">A user gets an error at sign-in and never reaches an MFA prompt</h3>



<p class="wp-block-paragraph">Almost always the recovery collision. Check whether the account has both <code>email</code> and <code>phone_number</code>, and compare the populated attribute against the pool&#8217;s account recovery setting. If the only available attribute is also the recovery channel, and there&#8217;s no TOTP registration, the user has no eligible factor. Register TOTP for them or add the missing attribute.</p>



<h3 class="wp-block-heading">Sign-in succeeds without MFA even though the pool says required</h3>



<p class="wp-block-paragraph">Work backward from the user, not from the pool. Is this a federated user? Then Cognito is not in the authentication path and pool MFA does not apply. Is it their first sign-in after account creation? Cognito treats the initial verification message as the second factor on that first sign-in and prompts for factor registration afterward. Is a different app client involved with different permitted flows?</p>



<h3 class="wp-block-heading">Password reset stopped working after enabling MFA</h3>



<p class="wp-block-paragraph">Expected behavior, not a regression. The reset code cannot go to the same channel as the user&#8217;s MFA factor. Users with only one contact attribute lose self-service recovery. Add the second attribute or configure a fallback recovery mechanism.</p>



<h3 class="wp-block-heading">SMS codes never arrive</h3>



<p class="wp-block-paragraph">Check the Amazon SNS SMS sandbox status in the region holding your SNS resources. In the sandbox, destination numbers must be verified in SNS before delivery. Also confirm the IAM role Cognito assumes for SNS is present and correctly scoped.</p>



<h3 class="wp-block-heading">Repeated MFA failures lock a user out</h3>



<p class="wp-block-paragraph">After five unsuccessful MFA code attempts, Cognito starts an exponential timeout lockout. It resolves on its own. Most real cases are clock drift on the user&#8217;s phone breaking TOTP validation, so check that before assuming an attack.</p>



<h3 class="wp-block-heading">Tokens keep working after you revoked them</h3>



<p class="wp-block-paragraph">Correct and by design. Revocation kills refresh tokens; already-issued access tokens survive to their expiry. If that window is unacceptable for your risk profile, shorten the access token TTL. There is no other lever short of checking revocation on every API call.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Treating the pool-level MFA setting as coverage for federated users. It is not, and this is the gap most likely to survive to production.</li><li>Storing PHI in Cognito custom attributes. Diagnosis codes, MRNs, and care team assignments do not belong in a directory record that gets copied into ID token claims, which then land in browser storage, proxy logs, and error reports. Keep the identity layer to identity.</li><li>Enabling required MFA in one deploy without an enrollment ramp. You will manufacture a support incident.</li><li>Assuming a short access token gives a short session when managed login is in play. The one-hour cookie governs re-authentication.</li><li>Believing revocation is immediate.</li><li>Leaving threat protection off because required MFA is on. Audit mode and required MFA coexist, and the detection signal is separately valuable.</li><li>Enabling refresh token rotation without checking whether your client uses <code>REFRESH_TOKEN_AUTH</code>. That combination is not permitted and it breaks at runtime.</li></ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list"><li>Separate app clients for patient-facing and staff-facing applications, with independently tuned token lifetimes. Staff sessions should not inherit consumer defaults.</li><li>TOTP as the baseline factor for anyone with access to more than their own record.</li><li>Passkeys with user verification for administrators and bulk-export roles.</li><li>Both <code>email</code> and <code>phone_number</code> collected at enrollment where the population allows it, purely to keep recovery working.</li><li>Access token TTL short enough that revocation means something. Fifteen minutes is a reasonable starting point for clinical apps.</li><li>Threat protection in audit mode, with events exported and actually reviewed.</li><li>Manage the pool in Terraform or OpenTofu so MFA settings, recovery configuration, and token lifetimes are reviewable in a diff instead of discovered in the console during an audit.</li><li>Test the MFA path in CI against a non-production pool. Configuration drift in an identity provider is silent by nature.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is Amazon Cognito HIPAA compliant?</h3>



<p class="wp-block-paragraph">No service is HIPAA compliant by itself. Cognito is HIPAA <em>eligible</em> and covered under the AWS Business Associate Agreement, which means you may process PHI with it once you have accepted the BAA in that account and configured it appropriately. Compliance is a property of your whole system and your documentation, not of the service.</p>



<h3 class="wp-block-heading">Does HIPAA require MFA today?</h3>



<p class="wp-block-paragraph">Not explicitly. The current Security Rule requires person or entity authentication and leaves the method to your risk analysis. The proposed 2025 update would make MFA an express requirement, but it remains proposed and the timeline for final action has moved more than once.</p>



<h3 class="wp-block-heading">Can I use required MFA and adaptive authentication together?</h3>



<p class="wp-block-paragraph">Not in full-function mode. Adaptive authentication&#8217;s automated responses require MFA to be optional in the user pool. You can run threat protection in audit-only mode alongside required MFA, which keeps the detection signal without the conflict.</p>



<h3 class="wp-block-heading">Which MFA factor should healthcare apps use?</h3>



<p class="wp-block-paragraph">TOTP as the baseline for staff, passkeys with user verification for privileged accounts. SMS is defensible for patient populations where enrollment friction genuinely determines whether MFA gets used at all, but it should not protect clinical access.</p>



<h3 class="wp-block-heading">Does Cognito MFA apply to users signing in through our hospital SSO?</h3>



<p class="wp-block-paragraph">No. For federated users arriving via SAML or OIDC, Cognito delegates authentication entirely to the upstream identity provider and does not add a second factor. Enforcement has to happen at the IdP, and you should get that in writing.</p>



<h3 class="wp-block-heading">How short should session timeouts be for a clinical application?</h3>



<p class="wp-block-paragraph">Set access tokens short, commonly around fifteen minutes, and refresh tokens to roughly a shift length rather than the thirty-day default. Then verify what users actually experience, because managed login&#8217;s one-hour browser cookie can keep a session alive past your token settings.</p>



<h3 class="wp-block-heading">Can I store patient identifiers in Cognito user attributes?</h3>



<p class="wp-block-paragraph">You can, and I would avoid it. Custom attributes flow into ID token claims, which end up in browser storage, application logs, and error tracking. Keep an opaque internal identifier in the token and resolve clinical data from your application database behind its own authorization check.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you remember one thing about Cognito MFA for healthcare apps, make it this: the MFA setting is a claim about local users at the moment of sign-in, and nothing else. It says nothing about your federated clinicians, nothing about the session that continues for the next eight hours, and nothing about whether the factor you chose survives a phishing proxy.</p>



<p class="wp-block-paragraph">Verify enforcement by observing behavior, per user population, per app client. Not by screenshotting the console page. That screenshot is what got flagged in the pen test at the top of this post, and it was completely accurate.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with Cognito and healthcare authentication?</h2>



<p class="wp-block-paragraph">I work with teams building and hardening authentication on AWS, particularly where a compliance obligation makes &#8220;it works&#8221; insufficient. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing an existing user pool for the gaps in this post: federated coverage, recovery collisions, token lifetimes, app client sprawl</li><li>Designing an MFA rollout that won&#8217;t generate a help desk queue, including the pre-flight query against your user records and the enrollment ramp</li><li>Implementing passkeys with user verification for privileged accounts, alongside TOTP for general staff</li><li>Moving user pool configuration into Terraform or OpenTofu so identity settings are reviewable and drift is visible</li><li>Wiring threat protection events into your existing monitoring and building the alerts that are worth waking someone for</li><li>Writing the technical evidence that maps your configuration to your risk analysis, in language an auditor accepts</li></ul>



<p class="wp-block-paragraph">If you have a user pool that isn&#8217;t behaving the way the console suggests it should, send me the output of <code>describe-user-pool</code> and <code>describe-user-pool-client</code> with the identifiers stripped, and I&#8217;ll tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cognito-mfa-healthcare-apps/">Cognito MFA for Healthcare Apps: The Settings That Look Compliant but Aren&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Secure Client Document Vault Your Accounting Firm Can Defend</title>
		<link>https://john-nessime.com/blog/cloud-security/secure-client-document-vault-accounting-firm/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 28 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Accounting Technology]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Accounting Firms]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[Client Portal]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Crypto Shredding]]></category>
		<category><![CDATA[Data Classification]]></category>
		<category><![CDATA[Envelope Encryption]]></category>
		<category><![CDATA[GLBA Safeguards Rule]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[MFA]]></category>
		<category><![CDATA[Offboarding]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Secure Disposal]]></category>
		<category><![CDATA[WISP]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=544</guid>

					<description><![CDATA[<p>Most accounting-firm document vaults do not fail at the bucket. They fail at the edges: a download link that outlives the engagement, a KMS key policy as wide as the bucket policy, a retention lock that cannot be undone, and an audit trail nobody switched on. Six failure families, and the control that closes each.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/secure-client-document-vault-accounting-firm/">Building a Secure Client Document Vault Your Accounting Firm Can Defend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket usually reads like this: <em>client says the download link expired, can you send a new one.</em> Thirty seconds of work. Generate a fresh link, paste it into a reply, close the ticket.</p>



<p class="wp-block-paragraph">Nobody looks at the old link. If it was signed with a long expiry and mailed out, it is still live: in the client&#8217;s inbox, in the inbox of the bookkeeper on the CC line, and in whatever mail archive that firm runs. It keeps working for anyone holding the URL until the clock runs out, because a presigned S3 URL is a bearer token. AWS says exactly that in its own documentation. The URL does not know who you are. It knows that you have it.</p>



<p class="wp-block-paragraph">That is the shape of most document vault failures at a small accounting firm. Not a wide-open bucket. Not a stolen partner password. An ordinary delivery mechanism that leaks quietly and writes no log line that looks wrong six months later.</p>



<p class="wp-block-paragraph">This post covers how to build a <strong>secure client document vault</strong> for an accounting practice: what the regulation actually demands, the six failure families that decide whether it holds, and the control that closes each one. The examples are AWS because that is where most of these land, but the failure families are platform independent.</p>



<h2 class="wp-block-heading">Let the regulation write your requirements</h2>



<p class="wp-block-paragraph">Most firms treat compliance as paperwork done after the build. That is backwards. For a US accounting or tax practice, the FTC Safeguards Rule at 16 CFR Part 314 is the closest thing you will get to a free requirements document.</p>



<p class="wp-block-paragraph">It treats tax preparers, bookkeepers and accounting firms as financial institutions under the Gramm-Leach-Bliley Act. Size does not get you out of it. The parts that translate directly into architecture:</p>



<ul class="wp-block-list">
<li><strong>Access controls</strong> limiting each user to the customer information they need, and limiting customers to their own information.</li>



<li><strong>Encryption</strong> in transit over external networks and at rest, with compensating controls only where encryption is infeasible and your Qualified Individual signs off in writing.</li>



<li><strong>Multi-factor authentication</strong> for any individual accessing any information system, subject to the same written-approval exception.</li>



<li><strong>Secure disposal</strong>, with an outer bound of two years after last use unless retention is required by law or necessary for legitimate business purposes.</li>



<li><strong>Logging of authorized user activity</strong>, plus a named Qualified Individual who owns the program.</li>
</ul>



<p class="wp-block-paragraph">One more clause should shape your design more than any other. The Rule requires notifying the FTC within thirty days of discovering a security event involving the unencrypted information of at least five hundred consumers, and it counts encrypted information as unencrypted if the key was accessed by an unauthorized person.</p>



<p class="wp-block-paragraph">Read that twice. Encryption only helps at the breach-analysis stage if the key was genuinely out of reach. That is the entire argument for treating key custody as a control separate from storage.</p>



<p class="wp-block-paragraph">Separately, IRS Publication 4557 expects paid preparers to keep a Written Information Security Plan, and PTIN renewal asks you to confirm you have one. Treat the WISP as the document describing the vault you built, not a substitute for building it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure one: the link that outlives the engagement</h2>



<p class="wp-block-paragraph">Presigned URLs are the sensible answer to &#8220;let the client download this without an AWS account&#8221;. The problem is their expiry semantics, which almost nobody has read.</p>



<p class="wp-block-paragraph">Two clocks apply and the shorter wins: the expiry you asked for, and the lifetime of the credential you signed with. Long-lived IAM user credentials under Signature Version 4 reach seven days. A role session from <code>AssumeRole</code> dies when the session ends, one hour by default, regardless of what you requested. An EC2 instance profile is bound by metadata credential rotation. The console caps at twelve hours.</p>



<p class="wp-block-paragraph">So the two symptoms are opposites and both are common. Links dying early is a credential problem. Links living too long is a policy problem, and that is the one that hurts.</p>



<p class="wp-block-paragraph">The fix is not a smaller number. It is changing what goes in the email.</p>



<ol class="wp-block-list">
<li>Email a notification, not the file link. The client logs in with MFA and the portal mints a URL at click time.</li>



<li>Keep that expiry in minutes. The URL only has to survive a redirect and a download.</li>



<li>Scope every URL to one object and one verb.</li>



<li>If you truly need a long-lived share, use CloudFront signed URLs or signed cookies, where you can revoke by rotating the key group without touching the object.</li>
</ol>



<pre class="wp-block-code"><code># Sign a download URL valid for five minutes.
# --expires-in takes seconds; the default is 3600.
aws s3 presign s3://firm-client-vault/clients/acme-llc/workpapers.pdf 
    --expires-in 300</code></pre>



<p class="wp-block-paragraph">Run that under an assumed role, wait past the session expiry, and watch it fail well before five minutes. That is behaviour worth seeing once in a test rather than discovering during an incident review.</p>



<h2 class="wp-block-heading">Failure two: everyone inside the firm sees everything</h2>



<p class="wp-block-paragraph">This shows up in nearly every small-firm build, and the reasoning is always the same. Nine people, everyone helps with everything during filing season, and compartmentalising access is friction exactly when friction is expensive.</p>



<p class="wp-block-paragraph">The case for flat access is real, so say it out loud: a scoped model people fight gets worked around, and a workaround is worse than the flat model you started with. If your access control makes a senior accountant email a PDF to themselves at 11pm, you have made things less safe.</p>



<p class="wp-block-paragraph">So scope by engagement rather than by person, and automate the grant so nobody files a request. Make the key prefix the boundary, then grant the prefix instead of the bucket. Two statements, two jobs: the first controls what someone can enumerate, the second what they can fetch.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOnlyThisClient",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::firm-client-vault",
      "Condition": {
        "StringLike": { "s3:prefix": "clients/acme-llc/*" }
      }
    },
    {
      "Sid": "ReadWriteThisClient",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::firm-client-vault/clients/acme-llc/*"
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Without the prefix condition on <code>ListBucket</code>, a scoped user still gets a full inventory of your client list. Client names are confidential in this business. A directory listing that reads like a book of business is a disclosure even when no file opens.</p>



<p class="wp-block-paragraph">The other half of this family is standing administrative access. Somebody has to fix things, so somebody ends up with a permanent <code>s3:*</code> on the whole bucket. That principal is the highest-value target in the firm and usually has the weakest MFA, because it is a service account nobody wants to touch.</p>



<p class="wp-block-paragraph">Replace it with an assumable break-glass role: MFA required, short session, and an alert on every assumption. The alert is the point. Emergency access used twice a year is fine. Used weekly, it means your normal path is broken.</p>



<h2 class="wp-block-heading">Failure three: encrypted at rest, key sitting right next to it</h2>



<p class="wp-block-paragraph">Default bucket encryption gets ticked in week one and forgotten. It satisfies the letter of &#8220;encrypted at rest&#8221; and protects against exactly one threat: somebody walking out of a data centre with a disk. That is not your threat.</p>



<p class="wp-block-paragraph">Go back to the notification clause. If an attacker reaches a principal that can both read the object and use the key, your data is unencrypted for reporting purposes. The question is not &#8220;was the bucket encrypted&#8221;. It is &#8220;could whoever got in also use the key&#8221;.</p>



<ul class="wp-block-list">
<li>Use a customer managed KMS key. Only that gives you a key policy you control.</li>



<li>Name the principals allowed to call <code>Decrypt</code>, and keep that list shorter than the list allowed to call <code>GetObject</code>. Two different lists means an attacker needs two wins.</li>



<li>Use encryption context so grants can be conditioned per client, and so CloudTrail entries are readable later.</li>



<li>Enable S3 Bucket Keys to cut KMS request volume. KMS bills per API call, so a vault serving many small reads generates a surprising line item.</li>
</ul>



<p class="wp-block-paragraph">Encryption in transit needs its own control, because IAM policies say nothing about the connection. A bucket policy does:</p>



<pre class="wp-block-code"><code>{
  "Sid": "DenyPlaintextTransport",
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": [
    "arn:aws:s3:::firm-client-vault",
    "arn:aws:s3:::firm-client-vault/*"
  ],
  "Condition": {
    "Bool": { "aws:SecureTransport": "false" }
  }
}</code></pre>



<p class="wp-block-paragraph">An explicit deny beats every allow, including one you add by mistake later. Five lines that permanently remove a category of misconfiguration, which makes it the best value item on this page.</p>



<h2 class="wp-block-heading">Failure four: retention rules that fight your disposal duty</h2>



<p class="wp-block-paragraph">Here is the trap, and it is why I would not let an accounting firm switch on Object Lock without a conversation first.</p>



<p class="wp-block-paragraph">Object Lock has two retention modes. In governance mode, a principal holding <code>s3:BypassGovernanceRetention</code> can remove or shorten the lock. In compliance mode, nobody can. Not an administrator, not the account root user. AWS is explicit that the only way to delete a compliance-locked object before its retention date is to close the AWS account.</p>



<p class="wp-block-paragraph">Set that against the Safeguards Rule&#8217;s two-year outer bound on disposal of information you no longer need. Apply a blanket seven-year compliance-mode retention to everything landing in the vault and you have written a promise you cannot revoke, on documents you may later be obliged to destroy.</p>



<ol class="wp-block-list">
<li>Classify by document type before client. Filed returns, engagement letters and supporting workpapers have real statutory retention arguments. Raw intake uploads and scratch scans usually do not.</li>



<li>Default to governance mode. It stops accidental and casual deletion, which is what you need, while keeping a documented and audited escape hatch.</li>



<li>Give <code>s3:BypassGovernanceRetention</code> to almost nobody and alert on every use through CloudTrail.</li>



<li>Reserve compliance mode for records where an external mandate genuinely requires immutability and you chose the period deliberately.</li>



<li>Use legal hold, not a retention period, for litigation. A legal hold has no end date and stays on until explicitly removed, which is right when you cannot know how long a matter runs.</li>
</ol>



<p class="wp-block-paragraph">Two details catch people out. Object Lock requires versioning and normally has to be enabled at bucket creation, so it is a design decision rather than a retrofit. And lifecycle expiration will not delete a version still under retention, so lifecycle and Object Lock are two layers that must agree, not alternatives.</p>



<pre class="wp-block-code"><code># What lock configuration is this bucket actually running?
aws s3api get-object-lock-configuration --bucket firm-client-vault

# Freeze one object version for a matter with no known end date.
aws s3api put-object-legal-hold 
    --bucket firm-client-vault 
    --key clients/acme-llc/2023/return.pdf 
    --legal-hold Status=ON</code></pre>



<p class="wp-block-paragraph">For deletion itself, crypto-shredding is worth knowing. If a client&#8217;s documents sit under a per-client KMS key, scheduling that key for deletion renders every object unreadable without touching the objects. Clean, fast, provable, and irreversible once the waiting period ends, so it belongs behind two-person approval rather than in a script.</p>



<h2 class="wp-block-heading">Failure five: no evidence when somebody asks</h2>



<p class="wp-block-paragraph">Every control above is worth nothing if you cannot show it held. The Rule expects logging of authorized user activity and an annual written report from the Qualified Individual.</p>



<p class="wp-block-paragraph">The trap is that CloudTrail does not record S3 object-level reads and writes by default. Management events cover things like changing a bucket policy. Data events cover the actual <code>GetObject</code> calls, and they are opt-in and billed per event. Firms enable CloudTrail, see a trail in the console, assume they have a record of file access, then find during an incident that they have nothing at object level.</p>



<p class="wp-block-paragraph">Turn on data events for the vault bucket. At firm scale the volume is manageable, and it is the difference between &#8220;we believe two people opened that file&#8221; and being able to show it.</p>



<p class="wp-block-paragraph">Then decide where logs live. Access records for client financial data are themselves sensitive, so keeping them in the same account under the same administrators defeats the purpose. A separate logging account, or at minimum a separate bucket with its own key and retention, is the shape you want. If you already run Grafana Cloud or similar for infrastructure monitoring, pointing vault access logs there gives you dashboards and alerting without standing anything new up.</p>



<p class="wp-block-paragraph">The alerts worth having are boring and few: any use of the break-glass role, any change to the bucket or key policy, any bulk download outside normal working patterns, and any failed decrypt. Four alerts that fire almost never beat forty that everyone mutes.</p>



<h2 class="wp-block-heading">Failure six: the two ends of the pipeline</h2>



<p class="wp-block-paragraph">A vault only protects documents that are in it. The two ends are where firms quietly undo everything.</p>



<h3 class="wp-block-heading">Intake</h3>



<p class="wp-block-paragraph">If clients can still email a W-2 as an attachment, they will, because it is easier. Each attachment becomes customer information sitting in a mailbox, a sent-items folder and a mail archive, each with a different retention policy and access model than your vault.</p>



<p class="wp-block-paragraph">A policy memo will not fix this. Make the vault path faster than the email path: an upload link that works on a phone, no account creation for a one-off drop, and a mail rule that returns certain attachment types with the upload link instead. Then sweep the mailboxes you already have, because that backlog is real customer information and it counts.</p>



<h3 class="wp-block-heading">Offboarding</h3>



<p class="wp-block-paragraph">Seasonal staff are normal here, so offboarding happens in batches and gets rushed. Disabling the SSO login is the part everyone does. The parts that get missed:</p>



<ul class="wp-block-list">
<li>Active role sessions and refresh tokens, which survive the account being disabled until they expire on their own.</li>



<li>Presigned URLs that person generated, which stay valid because they are bound to the signing credential, not the login session.</li>



<li>Named principals still sitting in the KMS key policy.</li>



<li>Local copies on laptops and external drives. When hardware is retired, a certified erase tool such as O&amp;O SafeErase is the difference between disposal and hoping.</li>
</ul>



<p class="wp-block-paragraph">The same applies to service providers. The Rule makes you responsible for selecting providers capable of maintaining appropriate safeguards and holding them to it contractually. If an outsourced bookkeeping team touches your document workflow, that relationship is in scope, and direct network access means they need MFA too. Cloudflare Zero Trust or a business VPN such as NordLayer in front of the portal gives you somewhere to enforce that without rebuilding the application.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting a secure client document vault</h2>



<ul class="wp-block-list">
<li><strong>Links expire far earlier than the expiry you set.</strong> You signed with temporary credentials that expired first. Sign from a principal whose credential outlives the link, or shorten the link and mint on demand.</li>



<li><strong>Uploads fail with access denied despite an allow on PutObject.</strong> With a customer managed key, both the IAM policy and the key policy must permit the caller. Check <code>kms:GenerateDataKey</code>, not just S3 actions.</li>



<li><strong>A file will not delete and no policy seems to block it.</strong> Look for Object Lock retention or a legal hold on that version. They are independent and either one alone blocks the delete.</li>



<li><strong>CloudTrail shows the bucket being created but no file access.</strong> Data events are not enabled. Management events will never show <code>GetObject</code>.</li>



<li><strong>A scoped user can see every client folder name.</strong> The prefix condition is missing from <code>ListBucket</code>, or was applied to the object ARN instead of the bucket ARN.</li>



<li><strong>Lifecycle rules appear to do nothing.</strong> Versions under retention are not expired by lifecycle. Confirm lock state before assuming the rule is wrong.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes, and what to do instead</h2>



<ul class="wp-block-list">
<li><strong>Emailing the download link.</strong> Email a notification that a document is waiting, and mint the link after the client authenticates.</li>



<li><strong>Treating default bucket encryption as finished.</strong> Use a customer managed key and keep the decrypt list shorter than the read list.</li>



<li><strong>Granting bucket-wide list permissions.</strong> Condition <code>ListBucket</code> on the client prefix, or the roster leaks even when no document opens.</li>



<li><strong>Applying compliance-mode Object Lock broadly because it sounds safer.</strong> Write the retention schedule per document class first, get partners to sign it, then configure to match.</li>



<li><strong>Keeping a standing full-access principal.</strong> Replace it with an MFA-gated break-glass role that alerts on every assumption.</li>



<li><strong>Building a portal and leaving email intake open.</strong> Both channels then carry client data and only one is governed. Close the mail path and sweep the backlog.</li>



<li><strong>Storing access logs beside the data they describe.</strong> Ship S3 data events somewhere with different administrators and different keys.</li>



<li><strong>Never testing any of it.</strong> Verify a restore and a scoped-access denial quarterly. An unverified control is an assumption.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does a small accounting firm really fall under the FTC Safeguards Rule?</h3>



<p class="wp-block-paragraph">Almost certainly, if it prepares returns or handles client financial data in the United States. The Rule&#8217;s definition of a financial institution covers activities financial in nature, and the FTC&#8217;s own examples include tax preparation firms. A narrow exception at 16 CFR 314.6 covers institutions holding information on fewer than five thousand consumers, but it exempts only four specific provisions. Encryption, MFA, access controls and disposal still apply. Confirm your own position with counsel, not a blog post.</p>



<h3 class="wp-block-heading">Should I build this or buy a client portal product?</h3>



<p class="wp-block-paragraph">Buy, if the product genuinely does what you need. A firm with no engineering capacity is safer on a maintained product than a bespoke stack nobody patches. Build when you need controls the product will not give you: per-client key isolation, your own retention logic, or integration with a pipeline you already run. The honest test is whether someone will still own the system in a year.</p>



<h3 class="wp-block-heading">Can I self-host a secure client document vault instead of using AWS?</h3>



<p class="wp-block-paragraph">Yes. Nextcloud or similar on a well-run VPS from a provider such as InterServer or Contabo can meet the same requirements, and it is often simpler to reason about. What you take on is patching, backups, TLS renewal and log retention. The regulation does not care where the data sits. It cares that the controls exist and that you can show they held.</p>



<h3 class="wp-block-heading">How long should client documents be retained?</h3>



<p class="wp-block-paragraph">There is no single number, which is why a schedule beats a default. Tax and professional standards drive minimums for filed returns and engagement documentation, and those vary by jurisdiction and engagement type. The Safeguards Rule pushes the other way. Write both directions down per document class, then configure to match.</p>



<h3 class="wp-block-heading">Is a presigned URL safe to send to a client?</h3>



<p class="wp-block-paragraph">It is safe to use, not safe to send. Treat it as a bearer token with a short life, minted after the client authenticates and consumed immediately. What travels over email should be a notification, not the credential.</p>



<h3 class="wp-block-heading">Does encrypting the bucket mean I avoid breach notification?</h3>



<p class="wp-block-paragraph">Only if the key stayed out of the attacker&#8217;s hands. The Rule treats encrypted customer information as unencrypted where the key was accessed by an unauthorized person. That is the whole argument for keeping key permissions narrower than storage permissions.</p>



<h3 class="wp-block-heading">What is the minimum viable version for a two-person practice?</h3>



<p class="wp-block-paragraph">One bucket with per-client prefixes, a customer managed key, an explicit deny on non-TLS transport, MFA on every human account, short mint-on-click links, S3 data events enabled, and a written retention schedule. That is a weekend of work and it closes five of the six failure families above.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A secure client document vault is not defined by the storage you picked. It is defined by how narrow the path is between an attacker and a readable file, and by your ability to prove afterwards how many people walked it.</p>



<p class="wp-block-paragraph">Most firms get the storage right and lose at the edges: a link that outlives the engagement, a key policy as wide as the bucket policy, a retention setting that cannot be undone, an audit trail nobody turned on. Fix those four and the rest is ordinary engineering.</p>



<h2 class="wp-block-heading">Where I can help</h2>



<p class="wp-block-paragraph">I work with small firms and the MSPs that support them on exactly this kind of build. Things I take on regularly:</p>



<ul class="wp-block-list">
<li>Designing and building the vault itself: bucket layout, per-client prefix isolation, KMS key policy, and the portal glue that mints short-lived links.</li>



<li>Reviewing an existing setup against the Safeguards Rule&#8217;s technical elements and producing a gap list your Qualified Individual can work from.</li>



<li>Untangling retention: turning a vague policy into a per-document-class schedule, implemented with lifecycle rules, Object Lock and legal holds that do not conflict.</li>



<li>Standing up the evidence layer: S3 data events, a separate logging account, dashboards, and the small set of alerts worth waking up for.</li>



<li>Closing the email intake path without making life harder for clients, including mobile upload flows and a sweep of the historic mailbox backlog.</li>



<li>Offboarding and access reviews: break-glass roles, session revocation, key policy hygiene, and a repeatable quarterly check.</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send the thing itself: a bucket policy, an IAM policy, a CloudTrail event selector, or a description of how documents reach your clients today. It is much easier to say something useful about a real configuration than about a general question.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/secure-client-document-vault-accounting-firm/">Building a Secure Client Document Vault Your Accounting Firm Can Defend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</title>
		<link>https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Comprehend]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=272</guid>

					<description><![CDATA[<p>A confident answer with a citation from a matter the reader was walled off from. Nothing crashed, nothing alerted. Building legal document intelligence on AWS with Textract, Bedrock and OpenSearch is the easy half; the hard half is tenant isolation, verified deletion, S3 Object Lock holds, what leaves the account, and an audit trail that names the actual user.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo was going well until someone asked where the third citation came from.</p>



<p class="wp-block-paragraph">The answer on screen was good. Fluent, specific, correctly hedged, three sources listed underneath it. Then a partner in the room asked a simple question: which matter is source three from? Nobody could answer it in the room, and when we went and looked, it was from a matter that half the people watching the demo were formally walled off from.</p>



<p class="wp-block-paragraph">Nothing had crashed. No alarm fired. The retrieval layer had done exactly what a similarity search does, which is return the nearest vectors, and the nearest vectors did not care about the ethical wall. That is the thing about building a legal document intelligence platform on AWS: the failures that matter almost never look like failures. They look like a confident answer with a citation attached.</p>



<p class="wp-block-paragraph">This post is about the parts of that build that are hard, organised around the five boundaries that actually have to hold. The extraction pipeline is the easy half. I will cover it, but quickly, because the AWS documentation is good and the failure modes are visible. The other four boundaries fail quietly, and that is where the engineering goes.</p>



<h2 class="wp-block-heading">The shape of the thing</h2>



<p class="wp-block-paragraph">Before the boundaries, the skeleton. A legal document intelligence platform on AWS almost always ends up looking like this, whether you plan it or arrive at it:</p>



<ul class="wp-block-list">
<li>Documents land in Amazon S3, one prefix per matter, versioning on.</li>



<li>An S3 event triggers AWS Lambda, which starts an asynchronous Amazon Textract job.</li>



<li>Textract output lands back in S3 as JSON. Something normalises it into text plus page and bounding-box coordinates.</li>



<li>Amazon Comprehend or a foundation model classifies the document and pulls entities: parties, dates, governing law, clause types.</li>



<li>Chunks get embedded and written to a vector store, usually Amazon OpenSearch Serverless behind Amazon Bedrock Knowledge Bases.</li>



<li>An application layer, typically API Gateway plus Lambda, takes a question, retrieves, and calls a model on Amazon Bedrock.</li>
</ul>



<p class="wp-block-paragraph">AWS Step Functions is worth reaching for once you have more than three stages, because Textract jobs are long-running and Lambda timeouts are not a retry strategy. That is the whole architecture. You can stand it up in a fortnight. Then you spend six months on everything below.</p>



<h2 class="wp-block-heading">Boundary one: isolation, and why metadata filtering is not optional</h2>



<p class="wp-block-paragraph">This is the one from the demo, and it is the one that ends engagements.</p>



<p class="wp-block-paragraph">A vector index has no native concept of a matter, a client, or an ethical wall. If you pool every document into one index and rely on the prompt to keep things separate, you have built a system where a well-phrased question can pull privileged material across a wall. Prompt instructions are not an access control. They are a suggestion that usually works.</p>



<p class="wp-block-paragraph">There are two shapes that do work, and the choice between them is a real trade-off rather than a best practice.</p>



<h3 class="wp-block-heading">Pooled index with server-side filtering</h3>



<p class="wp-block-paragraph">Every chunk carries a <code>matter_id</code> and a <code>client_id</code> as metadata. Every retrieval call attaches a filter. With an S3 data source in Bedrock Knowledge Bases, the metadata comes from a sidecar JSON file that sits next to the document and carries a <code>metadataAttributes</code> object.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "matter_id": "M-4417",
    "client_id": "C-108",
    "doc_type": "engagement_letter",
    "privileged": true
  }
}</code></pre>



<p class="wp-block-paragraph">The retrieval call then narrows the search before the model ever sees a chunk:</p>



<pre class="wp-block-code"><code>"retrievalConfiguration": {
  "vectorSearchConfiguration": {
    "filter": {
      "equals": { "key": "matter_id", "value": "M-4417" }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two rules make this safe, and both are the kind of thing that gets skipped under deadline. First, the filter value is derived on the server from the authenticated caller&#8217;s claims, never accepted from the client. If the browser can send you a <code>matter_id</code>, the browser can send you a different one. Second, the filter is applied by code that no feature request can bypass. A helper that builds every retrieval request, and a code review rule that no other path may call the retrieve API directly.</p>



<p class="wp-block-paragraph">The trap here is ordering. You cannot filter on an attribute the index does not have. Add <code>matter_id</code> after ingestion and every chunk already in the index is invisible to that filter, which means it either returns for everyone or for no one, depending on how your filter is written. Design the metadata schema before the first ingestion run, not after the first demo.</p>



<h3 class="wp-block-heading">Separate collection per client</h3>



<p class="wp-block-paragraph">The heavier option. A separate OpenSearch Serverless collection per client, which buys you a separate AWS KMS key per client, separate index settings, and a failure mode where a bug in the filtering logic cannot reach across clients at all.</p>



<p class="wp-block-paragraph">It costs you sprawl. Every collection is a resource to provision, monitor, patch policy on, and eventually delete. Ingestion jobs run per data source per knowledge base, so freshness guarantees fragment. For a firm with twelve institutional clients this is fine. For a platform onboarding a hundred small clients it becomes the main operational burden of the product.</p>



<p class="wp-block-paragraph">My default is pooled with strict server-side filtering, and per-client collections only where the client&#8217;s own contract demands a dedicated encryption key. If you cannot say out loud which of those two you are running, you are running neither properly.</p>



<h2 class="wp-block-heading">Boundary two: extraction, and the confidence score everyone ignores</h2>



<p class="wp-block-paragraph">Textract returns a confidence score between 0 and 100 with each prediction. AWS is explicit in its own best-practice guidance that applications sensitive to detection errors should enforce a minimum threshold and route anything below it for human scrutiny, and that the right threshold depends entirely on how the output gets used.</p>



<p class="wp-block-paragraph">Legal documents sit at the harsh end of that. A misread date on an archival scan is a curiosity. A misread date on a limitation period is a problem with a name on it.</p>



<p class="wp-block-paragraph">Practical notes from this layer:</p>



<ul class="wp-block-list">
<li>Use the asynchronous operations for anything multipage. <code>StartDocumentAnalysis</code> and <code>GetDocumentAnalysis</code> handle long PDFs and TIFFs; the synchronous <code>AnalyzeDocument</code> path is for single pages and will fight you on a 400-page bundle.</li>



<li>Textract&#8217;s Queries feature earns its place on structured instruments. Instead of extracting everything and grepping, you ask the document a direct question and get a scoped answer with its own confidence.</li>



<li>Keep the page number and bounding box for every chunk. When a lawyer asks where a sentence came from, &#8220;page 14, second column&#8221; is an answer. &#8220;It&#8217;s in the corpus somewhere&#8221; is not, and the platform loses credibility the first time you say it.</li>



<li>Amazon Augmented AI wires low-confidence predictions into a human review queue with a private workforce, so the reviewers are your people rather than an anonymous pool. For privileged material that distinction is the entire point.</li>
</ul>



<p class="wp-block-paragraph">What I would skip early: building a custom entity recognition model. Comprehend supports custom entity recognition and it is genuinely useful, but you need labelled data you do not have yet, and a foundation model with a decent prompt gets you far enough to find out which entities the users actually care about. Train the custom model once the requirement has stopped moving.</p>



<h2 class="wp-block-heading">Boundary three: retention, where deletion is a distributed problem</h2>



<p class="wp-block-paragraph">Here is the failure that will not show up in any test suite you write.</p>



<p class="wp-block-paragraph">A client asks for a document to be removed. Someone deletes the S3 object. The ticket closes. The document is still fully searchable, because its chunks are sitting in the vector index and will stay there until the data source is re-synced. Meanwhile the Textract JSON output is in a second bucket, the extracted text may be in a database, and if model invocation logging is on, whole passages of the document are sitting in CloudWatch Logs or another S3 bucket entirely.</p>



<p class="wp-block-paragraph">One delete, at least four places holding a copy. Write the deletion path as a real workflow with an assertion at the end, not as a single API call:</p>



<ol class="wp-block-list">
<li>Delete the source object and, if versioning is on, its versions.</li>



<li>Delete the derived artefacts: Textract JSON, normalised text, any thumbnails or page images.</li>



<li>Trigger an ingestion job so the knowledge base drops the orphaned chunks.</li>



<li>Confirm removal by querying the index for the document identifier and expecting nothing back.</li>



<li>Deal with the logs, which means either a retention policy short enough to make the problem expire or a deliberate decision, written down, that logs are out of scope.</li>
</ol>



<p class="wp-block-paragraph">Step four is the one people leave out, and it is the only step that actually proves anything.</p>



<h3 class="wp-block-heading">The opposite problem: things that must not be deleted</h3>



<p class="wp-block-paragraph">Legal work has the reverse requirement too. When a matter goes into litigation hold, the documents need to survive an administrator with delete permissions and a bad afternoon.</p>



<p class="wp-block-paragraph">S3 Object Lock is the mechanism, and it has two independent controls that people routinely conflate. A retention period protects an object version until a fixed date, in either governance mode, which privileged users can override, or compliance mode, which nobody can override and where the period cannot be shortened. A legal hold has no date at all. It stays until someone with <code>s3:PutObjectLegalHold</code> explicitly removes it. The two can be active at once, and while either is active the object version cannot be deleted or overwritten.</p>



<pre class="wp-block-code"><code># Place an indefinite hold on a specific object version
aws s3api put-object-legal-hold 
  --bucket matters-archive 
  --key M-4417/exhibit-c.pdf 
  --version-id 3sL7f2Qz9pXvB1kR 
  --legal-hold Status=ON</code></pre>



<p class="wp-block-paragraph">Two things bite here. Object Lock requires versioning and turns it on automatically, and once enabled on a bucket you cannot turn it off or suspend versioning again. And compliance mode is genuinely permanent: if you set a seven-year retention when you meant seven days, that is the answer, for everyone, including the account root. Test in governance mode first. The bypass path exists and is deliberately awkward, requiring both the <code>s3:BypassGovernanceRetention</code> permission and an explicit <code>x-amz-bypass-governance-retention:true</code> header on the request, which is exactly the level of friction you want on that operation.</p>



<p class="wp-block-paragraph">Design the hold model before you design the deletion model, because holds win. A deletion request that collides with an active hold is a legal question, not an engineering one, and the platform&#8217;s job is to surface the collision clearly rather than resolve it silently.</p>



<h2 class="wp-block-heading">Boundary four: disclosure, or what leaves the account</h2>



<p class="wp-block-paragraph">This is the boundary the client&#8217;s general counsel will ask about, usually in writing, usually before signature. Three separate controls, and they are not interchangeable.</p>



<h3 class="wp-block-heading">The AI services opt-out policy</h3>



<p class="wp-block-paragraph">AWS Organizations has a governance policy type that opts your accounts out of having content processed by certain AI services stored and used for service improvement. The list includes Amazon Textract and Amazon Comprehend, which are precisely the two doing the reading in this architecture. The policy type has to be enabled at the organisation root before you can attach anything:</p>



<pre class="wp-block-code"><code>aws organizations enable-policy-type 
  --root-id r-example 
  --policy-type AISERVICES_OPT_OUT_POLICY</code></pre>



<p class="wp-block-paragraph">Read the AWS documentation on this one carefully rather than taking my summary as gospel, because there is a caveat in it that matters: the services may still need to store your data operationally even when you have opted out of it being used for improvement. Opting out is not the same as the data never existing outside your account. Say that plainly in the client conversation. It is a much better position than being asked about it later.</p>



<h3 class="wp-block-heading">Bedrock&#8217;s data position, and the log that undoes it</h3>



<p class="wp-block-paragraph">Amazon Bedrock&#8217;s published position is that inputs and outputs are not shared with third-party model providers and are not used to train the base models, and that fine-tuning operates on a private copy. That is a strong starting point for privileged content and it is the main reason Bedrock rather than a direct provider API shows up in these builds.</p>



<p class="wp-block-paragraph">Then there is model invocation logging. It captures full prompt and response payloads to CloudWatch Logs or S3, and you want it on, because without it you cannot reconstruct what the system told someone. But understand what you have just built: a log that contains privileged document text, at a per-region setting, in a destination that probably has looser access controls than the document store it came from. Encrypt it with the same KMS key discipline, restrict it harder than you think you need to, and set a retention period on purpose.</p>



<p class="wp-block-paragraph">Worth separating in your head: CloudTrail records that an API call happened and who made it. Model invocation logging records what was in it. You need both, for different questions, and only one of them contains client confidences.</p>



<h3 class="wp-block-heading">Network path</h3>



<p class="wp-block-paragraph">Interface VPC endpoints via AWS PrivateLink keep traffic to Bedrock, Textract and the rest on the AWS network rather than out through an internet gateway or NAT. Add a gateway endpoint for S3 while you are there.</p>



<p class="wp-block-paragraph">Endpoint policies are the part that gets forgotten. An endpoint without a policy is a private path to the whole service, including into accounts that are not yours. Scope it to the operations and resources you actually use, and pair it with an S3 bucket policy that rejects requests not arriving through your endpoint. The first protects what leaves; the second protects what can be reached.</p>



<h3 class="wp-block-heading">The parts that are not on the architecture diagram</h3>



<p class="wp-block-paragraph">Two of these have caught people out badly, and neither is an AWS control.</p>



<p class="wp-block-paragraph">The first is the development environment. You do not want real client documents on a laptop or in a scratch account, so build the parsing and chunking code against synthetic documents on a small separate box. A cheap VPS from a provider like Contabo or InterServer is fine for this, and the separation is worth more than the convenience of iterating in the production account.</p>



<p class="wp-block-paragraph">The second is the reviewers. If your human review loop involves contractors working remotely, their network path and their disks are part of your boundary whether or not you drew them. A managed VPN such as NordVPN or Surfshark handles the network side, and when a matter closes and local copies have to go, a dedicated erasure tool from something like O&amp;O Software does what dragging a folder to the bin does not. Unglamorous, and it is the layer that gets audited.</p>



<h2 class="wp-block-heading">Boundary five: evidence, because someone will ask</h2>



<p class="wp-block-paragraph">At some point the question stops being &#8220;does it work&#8221; and becomes &#8220;who saw what, and when&#8221;. If you cannot answer that from stored records, the platform is not defensible regardless of how good the answers are.</p>



<ul class="wp-block-list">
<li>Turn on CloudTrail data events for the document buckets. They are off by default, they are billed separately from management events, and they are the only way to see individual object-level reads.</li>



<li>Log the retrieval, not just the generation. Store which chunks came back and which filter was applied. When somebody asks whether the wall held on a specific query, this record is the answer and there is no reconstructing it later.</li>



<li>Carry the end user&#8217;s identity through to the audit record. A Lambda execution role in the logs tells you the platform did something. It does not tell you who asked.</li>



<li>Alarm on the filter, not just on errors. A retrieval that ran without a matter filter should page someone, even though it returned HTTP 200 and a perfectly good answer.</li>
</ul>



<p class="wp-block-paragraph">That last one is the closest thing to a single takeaway in this post. The dangerous events in this architecture are successful ones.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the things that will go wrong</h2>



<h3 class="wp-block-heading">Retrieval returns nothing after you add a filter</h3>



<p class="wp-block-paragraph">Nearly always a metadata problem rather than a query problem. Either the sidecar metadata file was missing at ingestion, so the chunks carry no attribute to match, or the attribute was added after the chunks were indexed. Check whether the attribute exists on a known chunk before you touch the query. If it does not, you need a re-sync, not a different filter expression.</p>



<h3 class="wp-block-heading">Textract returns text but the layout is scrambled</h3>



<p class="wp-block-paragraph">Common on two-column pleadings and on documents with headers and footers on every page. Raw text detection reads in a reading order that is not always yours. Use the layout and table features rather than plain detection, and hold onto the geometry so you can reassemble columns yourself if the default order is wrong for that document class.</p>



<h3 class="wp-block-heading">The model cites a document that does not exist</h3>



<p class="wp-block-paragraph">Usually the citation is being generated rather than passed through. If document names are being written by the model instead of read from the retrieval response, it will invent plausible ones. Build citations from the retrieval result metadata in your application code and never from the generated text. Bedrock Guardrails helps with contextual grounding, but the structural fix is not asking the model to produce the reference in the first place.</p>



<h3 class="wp-block-heading">Ingestion is slow and nobody knows where</h3>



<p class="wp-block-paragraph">Instrument per stage before you optimise anything. In most of these pipelines the time is in Textract for large scanned bundles, and the fix is parallelism at the document level rather than tuning the extraction itself. Step Functions with a distributed map over documents gets you there without a queue you have to babysit.</p>



<h3 class="wp-block-heading">Costs climb faster than volume</h3>



<p class="wp-block-paragraph">Look at re-processing first. A pipeline that re-extracts a document every time anything downstream changes will quietly multiply your Textract bill against a static corpus. Key the extraction cache on the object&#8217;s content hash, not its path, and make re-ingestion an explicit action. The billing mechanisms differ per service, so read the current pricing pages rather than trusting a number from a blog post.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating the system prompt as an access control.</strong> It is a formatting instruction that happens to look like a rule.</li>



<li><strong>Taking the tenant identifier from the request body.</strong> If the client can name the matter, the client can name someone else&#8217;s.</li>



<li><strong>Designing the metadata schema after the first ingestion run.</strong> Retrofitting a filterable attribute means re-indexing the whole corpus.</li>



<li><strong>Assuming a deleted S3 object is gone from the platform.</strong> The chunks, the extraction output and the invocation logs all outlive it.</li>



<li><strong>Setting compliance-mode retention without testing in governance mode.</strong> There is no support ticket that fixes a seven-year mistake.</li>



<li><strong>Enabling model invocation logging without treating the log as privileged.</strong> You have just made a second copy of the documents in a less protected place.</li>



<li><strong>Building VPC endpoints and leaving the default policy on them.</strong> A private path with no policy is still a path.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>One retrieval helper, server-side filter injection, and a review rule that no other code path calls the retrieve API directly.</li>



<li>Metadata schema agreed and frozen before the first production ingestion. Include the fields you might filter on later, even if unused today.</li>



<li>Page-level provenance on every chunk, surfaced in the interface. It builds trust faster than any accuracy improvement.</li>



<li>A confidence threshold chosen per document class, with a human review path for everything under it.</li>



<li>Deletion implemented as a verified workflow across every store, ending in an assertion that the content is actually unretrievable.</li>



<li>Object Lock legal holds for matters under litigation hold, tested in governance mode before anything runs in compliance mode.</li>



<li>Customer-managed KMS keys across S3, the vector store and the logs, so key access is a control you can actually revoke.</li>



<li>An alarm on unfiltered retrievals, because the failure mode is a success response.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a legal document intelligence platform on AWS safe for privileged material?</h3>



<p class="wp-block-paragraph">The services support it. Bedrock&#8217;s stated position is that inputs and outputs are not shared with model providers or used to train base models, PrivateLink keeps traffic off the public internet, and KMS covers encryption at rest. What determines safety is your own configuration: tenant isolation, log handling and deletion behaviour. The platform is exactly as confidential as its weakest copy of the text.</p>



<h3 class="wp-block-heading">Should I use Bedrock Knowledge Bases or build retrieval myself?</h3>



<p class="wp-block-paragraph">Start with Knowledge Bases. It handles chunking, embedding, sync and metadata filtering, and those are weeks of work with no differentiation in them. Build your own when you need retrieval behaviour it does not expose, such as unusual re-ranking or per-tenant chunking strategies. Going custom on day one usually means reimplementing the managed service badly.</p>



<h3 class="wp-block-heading">How do I stop one client&#8217;s documents surfacing in another client&#8217;s answers?</h3>



<p class="wp-block-paragraph">Tag every chunk with a tenant identifier at ingestion, and inject the matching filter server-side on every retrieval from the authenticated caller&#8217;s identity. Never accept the identifier from the client. If the contract requires per-client encryption keys, use a separate collection per client instead, accepting the extra operational load.</p>



<h3 class="wp-block-heading">Does deleting a document from S3 remove it from the search index?</h3>



<p class="wp-block-paragraph">No. The chunks stay in the vector store until an ingestion job runs and reconciles the data source. Until then the document is deleted and still fully searchable, which is the worst combination available. Always finish a deletion by querying the index and confirming nothing comes back.</p>



<h3 class="wp-block-heading">What is the difference between an S3 legal hold and a retention period?</h3>



<p class="wp-block-paragraph">A retention period runs until a fixed date and comes in governance mode, which privileged users can override, or compliance mode, which nobody can. A legal hold has no end date and stays until someone explicitly removes it. They are independent, they can both be active on the same object version, and while either is active the object cannot be deleted or overwritten.</p>



<h3 class="wp-block-heading">Do I need human review, or is the model accurate enough?</h3>



<p class="wp-block-paragraph">Accuracy is the wrong frame. The question is what happens to a wrong answer downstream. Where output feeds a decision with consequences, you want a confidence threshold and a review queue, and Amazon Augmented AI with a private workforce gives you that without building the review tooling yourself. Where output is a search aid a person will verify anyway, review is friction you do not need.</p>



<h3 class="wp-block-heading">Can I run this in a single AWS account?</h3>



<p class="wp-block-paragraph">Technically yes, and for a pilot it is reasonable. It gets uncomfortable once you have production documents and a development environment in the same place, because an IAM mistake has nowhere to stop. Separate accounts under Organizations also gives you the policy layer, including the AI services opt-out policy, which only exists at the organisation level.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A legal document intelligence platform on AWS is not hard to build. Textract reads, Bedrock reasons, OpenSearch retrieves, and the tutorial version works on the first afternoon.</p>



<p class="wp-block-paragraph">What is hard is that its failures are silent and well-formed. The cross-matter citation, the deleted document that still answers questions, the privileged passage sitting in a log bucket, the retrieval that ran without a filter and returned a beautiful paragraph. None of these throw an error. All of them are the kind of thing that ends a client relationship.</p>



<p class="wp-block-paragraph">So build the boundaries first and the features second, and make sure every one of them is something you can prove from a stored record rather than something you believe about the code. If you can only take one habit from this: alarm on the successful requests that should not have been possible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your document pipeline?</h2>



<p class="wp-block-paragraph">I work on AWS document processing and retrieval systems where the confidentiality requirements are real. Typical things I get called in for:</p>



<ul class="wp-block-list">
<li>Reviewing tenant isolation in an existing RAG setup and finding the paths where the filter can be bypassed.</li>



<li>Designing the metadata and chunking schema before ingestion, so filtering works without a re-index later.</li>



<li>Building Textract pipelines with confidence thresholds, human review routing and page-level provenance.</li>



<li>Implementing verified deletion across S3, derived artefacts, the vector index and logs, with a proof step at the end.</li>



<li>Setting up retention and legal hold with S3 Object Lock, including the governance-mode rehearsal before compliance mode.</li>



<li>Locking down the network and logging path: VPC endpoints with real policies, KMS key separation, and audit records that name the actual user.</li>
</ul>



<p class="wp-block-paragraph">If you have an architecture diagram, a Step Functions definition or a retrieval request you are unsure about, send it over and I will tell you what I would change and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>As-Planned vs As-Built Analysis: Building a Platform That Survives Cross-Examination</title>
		<link>https://john-nessime.com/blog/technical-guides/as-planned-vs-as-built-analysis-platform/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 24 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[As-Built Schedule]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[Capital Projects]]></category>
		<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Delay Analysis]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Extension of Time]]></category>
		<category><![CDATA[Forensic Schedule Analysis]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[Project Controls]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[XER Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=274</guid>

					<description><![CDATA[<p>Most as-planned vs as-built analysis compares the baseline to the last P6 update and calls the result an as-built. It isn't one. This is how to build the data layer underneath a delay analysis: versioned XER ingestion, activity identity across renumbering, a first-appearance table that proves when every actual date entered the record, calendar-safe float, and record linking that proposes candidates instead of asserting cause.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/as-planned-vs-as-built-analysis-platform/">As-Planned vs As-Built Analysis: Building a Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that kills a delay analysis is never about methodology. It is five words from the other side&#8217;s expert: &#8220;which update did that come from?&#8221;</p>



<p class="wp-block-paragraph">Picture the experts&#8217; meeting. Your exhibit shows activity C-2140, structural steel erection to level four, finishing well behind where the baseline put it. He has the same XER files you do. He opens update 19 and points out that C-2140 carries no actual finish. Same in update 20. In update 21 an actual finish appears, and the date on it sits three weeks in the past, inside the window you already closed out.</p>



<p class="wp-block-paragraph">So which file is the record? And if one activity was back-filled a month after the fact, what does that say about the other four thousand?</p>



<p class="wp-block-paragraph">That is the failure an <strong>as-planned vs as-built analysis</strong> has to survive, and no amount of care inside the Excel workbook fixes it. It is a data problem before it is a scheduling problem. This post covers the platform underneath the analysis: how to ingest a long series of Primavera P6 updates, how to prove where every actual date came from, and how to produce output a tribunal can audit instead of having to trust.</p>



<h2 class="wp-block-heading">The as-built is not a file, and treating it as one is the invisible failure</h2>



<p class="wp-block-paragraph">Almost every quick as-planned vs as-built comparison does the same thing: take the approved baseline, take the last update in the sequence, join on activity ID, subtract the dates, sort by variance. It produces a clean-looking table in an afternoon.</p>



<p class="wp-block-paragraph">It is also the weakest possible construction of an as-built, for three reasons that never show up in the output.</p>



<ul class="wp-block-list">
<li><strong>A P6 update is a forecast document.</strong> It was produced to tell the employer when the job would finish, not to record what happened. Actual dates are a by-product of that exercise, entered by a planner working to a monthly deadline.</li>

<li><strong>The last update has been edited the most.</strong> By the end of a long job it has usually been re-baselined, partially renumbered, had calendars swapped, had activities merged or deleted. Every one of those edits is invisible in a two-file comparison.</li>

<li><strong>Actual dates get back-filled.</strong> A date typed in month 21 describing month 19 is not contemporaneous evidence in the same sense as a date typed in month 19. It might still be right. It is a different quality of record, and the difference is exactly what gets probed in cross-examination.</li>
</ul>



<p class="wp-block-paragraph">AACE International&#8217;s Recommended Practice 29R-03 places the as-planned versus as-built family in its observational, static group: you compare a planned network against an as-built without inserting or removing delay events. The SCL Delay and Disruption Protocol, second edition, is blunter about what makes any of it work, pushing hard on agreeing a record-keeping regime up front and never overwriting a programme version. Both point at the same requirement. You need the whole sequence of updates, intact, and you need to know which one told you what.</p>



<p class="wp-block-paragraph">The platform&#8217;s whole job is to make that sequence queryable.</p>



<h2 class="wp-block-heading">A worked example: what thirty-four updates actually look like</h2>



<p class="wp-block-paragraph">Take a job shaped like this. Three years of monthly updates, call it thirty-four XER exports. Around four thousand activities in the current programme. Three baseline revisions, two agreed and one submitted but never accepted. Somewhere near month fourteen the contractor moved most trades from a five-day to a six-day calendar to recover. The numbers are illustrative rather than from any particular project, but the shape is ordinary.</p>



<p class="wp-block-paragraph">Run the naive comparison and you get a variance table. Run the sequence properly and four separate questions fall out, none of which the variance table can answer:</p>



<ol class="wp-block-list">
<li>Which activities existed in the baseline, disappeared mid-job, and came back under a different activity ID?</li>

<li>For each actual date in the final update, in which update did that value first appear, and how far behind the event was it?</li>

<li>When float moved, was it because work moved, or because someone changed a calendar or a duration?</li>

<li>What contemporaneous record sits underneath each actual date, and does it agree?</li>
</ol>



<p class="wp-block-paragraph">Those four questions are the platform. Everything below is how to answer each one without hand-checking four thousand rows thirty-four times.</p>



<h2 class="wp-block-heading">Failure family one: activity identity across versions</h2>



<p class="wp-block-paragraph">An XER file is tab-delimited text. It opens in a text editor. The structure is a header line beginning <code>ERMHDR</code>, then a run of tables, each introduced by <code>%T</code> with the table name, a <code>%F</code> line carrying the field names, and <code>%R</code> lines carrying rows, closed by <code>%E</code>. The tables you care about first are <code>PROJECT</code>, <code>TASK</code>, <code>TASKPRED</code> and <code>CALENDAR</code>.</p>



<pre class="wp-block-code"><code>ERMHDR	19.12	2026-01-15	Project	admin	Admin User	dbxDatabaseNoName	Project Management	USD
%T	TASK
%F	task_id	proj_id	wbs_id	clndr_id	task_code	task_name	status_code	act_start_date	act_end_date	target_start_date	target_end_date	total_float_hr_cnt
%R	1001	100	200	1	C-2140	Structural Steel Erection L4	TK_Complete	...
%E</code></pre>



<p class="wp-block-paragraph">Here is the trap. <code>task_id</code> is the primary key of the P6 database the file was exported from. It is not a stable identifier for an activity across time. If the project was ever copied between databases, restored, or exported from a different P6 instance, the same physical activity can carry a different <code>task_id</code>. Worse, two XERs from different sources can reuse the same <code>task_id</code> for completely unrelated activities.</p>



<p class="wp-block-paragraph"><code>task_code</code> is the Activity ID a human sees, and it is stable right up until someone renumbers. Neither field alone gives you identity. So do not pick one. Build an identity resolution layer and keep the evidence for every decision it makes:</p>



<ul class="wp-block-list">
<li>Use <code>(proj_id, task_code)</code> as the working join key, scoped to a single source database.</li>

<li>Store <code>task_id</code>, <code>task_name</code> and <code>wbs_id</code> on every snapshot row so a rename or a WBS move is detectable after the fact.</li>

<li>Emit an explicit <em>unresolved</em> record when an activity vanishes or appears mid-sequence. Do not guess.</li>
</ul>



<p class="wp-block-paragraph">That last point is the one people get wrong. When C-2140 disappears in update 22 and an activity with the same name but the ID C-2140A appears in update 23, there are three candidate explanations: deletion and replacement, renumbering, or a split. The platform cannot know which. A tool that silently picks one is worse than a tool that flags it, because the analyst never learns there was a question. Surface it, hand it to a human, record the decision and the reason.</p>



<h2 class="wp-block-heading">Failure family two: date provenance, and the table that makes the whole thing worth building</h2>



<p class="wp-block-paragraph">If you build one thing, build this. For every activity and every date field, record the first update in which that value appeared, and every subsequent change to it.</p>



<p class="wp-block-paragraph">Assume a snapshot table with one row per activity per update, appended and never modified. A window function does the work:</p>



<pre class="wp-block-code"><code>-- First appearance of each distinct actual finish value,
-- and how far the reported date lags the update that reported it.
WITH changes AS (
  SELECT
    task_code,
    version_no,
    data_date,
    act_end_date,
    LAG(act_end_date) OVER (
      PARTITION BY task_code ORDER BY version_no
    ) AS prev_act_end
  FROM activity_snapshot
)
SELECT
  task_code,
  version_no          AS first_reported_in_update,
  act_end_date        AS reported_actual_finish,
  data_date           AS update_data_date,
  data_date::date - act_end_date::date AS reporting_lag_days
FROM changes
WHERE act_end_date IS NOT NULL
  AND (prev_act_end IS NULL OR prev_act_end &lt;&gt; act_end_date)
ORDER BY reporting_lag_days DESC;</code></pre>



<p class="wp-block-paragraph">Three things arrive at once. You can answer the question that opened this post, for any activity, in a second: C-2140&#8217;s actual finish first appeared in update 21, carrying a date twenty-three days behind that update&#8217;s data date.</p>



<p class="wp-block-paragraph">The distribution of that lag column is then a finding in its own right. If most activities report within days and one subcontractor&#8217;s work consistently reports a month late, that is not a rounding artefact, it is a records issue you can evidence. And any actual date that changes after it was first stated deserves a hard look. Actuals are supposed to lock. A value that moves has either been corrected, which should have a paper trail, or overwritten, which should not have happened.</p>



<p class="wp-block-paragraph">P6 ships a Schedule Comparison tool, the one that used to be called Claim Digger and now lives inside Visualizer. It does a field-by-field diff and dumps a large HTML table, which is genuinely useful for checking one submitted update against the last. That is what it was built for. It was not built to reason across thirty-four files at once, and pushing its output into Excel does not change that. Deltek&#8217;s Acumen Fuse and Steelray&#8217;s Delay Analyzer go further, with what the vendors describe as half-step analysis, separating progress from scope revisions across successive versions. If your instruction allows the licence and the timetable allows the learning curve, look at them seriously before writing any code. Neither is priced publicly, both target enterprise project controls teams, and neither does the records-linking work in the last section for you.</p>



<h2 class="wp-block-heading">Failure family three: calendars, and float that quietly means nothing</h2>



<p class="wp-block-paragraph">Durations and float in an XER are stored in hours. <code>total_float_hr_cnt</code> holds hours, not days. A value of 40 is five days only if the activity runs an eight-hour calendar. Divide everything by eight and a job with mixed shift patterns produces float numbers that are confidently wrong.</p>



<p class="wp-block-paragraph">The calendar itself lives in the <code>CALENDAR</code> table, in a <code>clndr_data</code> field holding a proprietary parenthesised blob describing work weeks, exceptions and holidays. It is the hardest part of the format to parse correctly, and getting it wrong produces durations and dates that look plausible and are not.</p>



<p class="wp-block-paragraph">Two rules follow, and both are about the read path, not the maths.</p>



<ul class="wp-block-list">
<li>Store float and durations in hours exactly as exported. Convert at presentation time, using that activity&#8217;s own calendar, and print the calendar name next to the number. If you cannot resolve the calendar, print hours and say so.</li>

<li>Track calendar assignment as a versioned attribute, the same as any date. A change from a five-day to a six-day calendar changes every float figure downstream of it, with no change to logic and no change to progress. An activity can go from critical to non-critical because of an administrative edit, and a variance table will attribute that shift to the works.</li>
</ul>



<p class="wp-block-paragraph">This is also why an as-built critical path is not simply the chain with zero float in the last update. Retrospective longest path and as-planned versus as-built windows analysis exist as separate methods precisely because the float figures in a progressed programme are a function of how the programme was maintained.</p>



<h2 class="wp-block-heading">Failure family four: separating movement from editing</h2>



<p class="wp-block-paragraph">Every delta between two consecutive updates falls into one of a small number of categories, and collapsing them into a single &#8220;variance&#8221; number is where analyses lose their defensibility. Classify each change explicitly:</p>



<ul class="wp-block-list">
<li><strong>Progress:</strong> an actual start or finish appeared, or remaining duration reduced consistent with work done.</li>

<li><strong>Duration change:</strong> original or remaining duration edited without corresponding progress.</li>

<li><strong>Logic change:</strong> a relationship added, removed, retyped, or its lag altered.</li>

<li><strong>Calendar change:</strong> activity reassigned, or the calendar definition itself edited.</li>

<li><strong>Constraint change:</strong> a date constraint applied, moved or removed.</li>

<li><strong>Scope change:</strong> activity added or deleted.</li>
</ul>



<p class="wp-block-paragraph">Six buckets, one row per change, with the before value, the after value and the version pair. Once that table exists, a lot of the argument stops being an argument. &#8220;The completion date moved out eighteen days in window 12&#8221; becomes &#8220;of which fourteen days sit with progress on the critical chain and four with two logic edits applied in the same submission.&#8221; The second version is the one that gets tested and holds.</p>



<p class="wp-block-paragraph">Logic changes deserve their own view. Pull <code>TASKPRED</code> for every version, key each relationship on the predecessor and successor activity codes plus the relationship type, and diff the sets. Relationships that appear late in a job, on activities that are already in progress, are worth reading one by one.</p>



<h2 class="wp-block-heading">Failure family five: joining the schedule to the records</h2>



<p class="wp-block-paragraph">The schedule tells you when. It never tells you why. Cause comes from daily reports, site instructions, RFIs, correspondence, minutes, inspection records, weather data, and delivery notes. That material arrives as scanned PDFs and email exports, and it does not carry activity IDs. Nobody has ever written &#8220;C-2140&#8221; on a site diary.</p>



<p class="wp-block-paragraph">So the join key is not the activity ID. It is a fuzzy composite of date, location or area, and trade or discipline. The practical approach:</p>



<ol class="wp-block-list">
<li>Extract structured fields from each record: document date, report date, area, trade, headcount if present, and the narrative text. Text extraction plus a layout-aware OCR pass handles most of it. Handwriting on older site diaries will not fully automate, and you should budget for that.</li>

<li>Normalise area and trade against the schedule&#8217;s own coding. Activity codes and WBS paths are usually the best available crosswalk, since they encode area and discipline already.</li>

<li>Generate <em>candidate</em> links: for each activity&#8217;s as-built window, every record whose date falls inside it and whose area or trade matches.</li>

<li>Have a human confirm, reject or annotate each candidate. Store who decided and when.</li>
</ol>



<p class="wp-block-paragraph">Step four is not optional and it is not a nicety. A machine-proposed link is a search result. Causation is an opinion, and the expert has to own it. Build the tool so it accelerates the search and refuses to state the conclusion, and you get the productivity without handing opposing counsel an argument about black boxes.</p>



<h2 class="wp-block-heading">The data model, in about six tables</h2>



<p class="wp-block-paragraph">The grain that makes everything else easy: one immutable row per activity per schedule version. Nothing is ever updated in place.</p>



<pre class="wp-block-code"><code>CREATE TABLE schedule_version (
  version_id     bigserial PRIMARY KEY,
  version_no     integer     NOT NULL,   -- ordinal in the sequence
  source_file    text        NOT NULL,
  file_sha256    char(64)    NOT NULL,   -- proves the file was not altered
  p6_export_ts   timestamptz,            -- from the ERMHDR line
  data_date      timestamptz NOT NULL,
  ingested_at    timestamptz NOT NULL DEFAULT now(),
  notes          text
);

CREATE TABLE activity_snapshot (
  version_id           bigint  NOT NULL REFERENCES schedule_version,
  task_code            text    NOT NULL,
  task_id              bigint  NOT NULL,
  task_name            text,
  wbs_path             text,
  clndr_id             bigint,
  status_code          text,
  target_start_date    timestamptz,
  target_end_date      timestamptz,
  act_start_date       timestamptz,
  act_end_date         timestamptz,
  total_float_hr_cnt   numeric,
  PRIMARY KEY (version_id, task_code)
);</code></pre>



<p class="wp-block-paragraph">Add <code>relationship_snapshot</code> keyed on <code>(version_id, pred_task_code, succ_task_code, rel_type)</code>, a <code>calendar_snapshot</code> holding the raw <code>clndr_data</code> alongside your parsed working pattern, a <code>record_document</code> table for the contemporaneous material, and a <code>record_link</code> table carrying the human decision on each candidate link.</p>



<p class="wp-block-paragraph">Thirty-four updates at four thousand activities is under a hundred and fifty thousand rows. This is not a big-data problem and it should not be built like one. PostgreSQL on a single machine, with the parsing done in Python, will run every query in this post fast enough that you stop thinking about it.</p>



<p class="wp-block-paragraph">On the parsing side, an XER is simple enough to read with the standard library, and there are open-source parsers such as PyP6Xer if you would rather not write the tokeniser yourself. Whichever route you take, keep the raw file. Hash it on ingest, store the hash in <code>schedule_version</code>, and never write to the original. Chain of custody over the source files is the cheapest credibility you will ever buy.</p>



<h2 class="wp-block-heading">Where to run it, and the confidentiality problem nobody budgets for</h2>



<p class="wp-block-paragraph">A modest VPS is plenty. Contabo and InterServer both sell machines with more than enough memory and disk for a job this size, and the specification matters far less than the question you should ask first: does your confidentiality undertaking actually permit the disclosure material to sit on that machine, in that jurisdiction? Often the answer is no, and the analysis has to run on a workstation inside the client&#8217;s own environment. Find that out before you build anything.</p>



<p class="wp-block-paragraph">Three practical points that follow from the material being privileged rather than ordinary project data:</p>



<ul class="wp-block-list">
<li><strong>Full-disk encryption on anything that holds the files</strong>, including the laptop you take to the hearing. This is table stakes and it is still routinely skipped.</li>

<li><strong>A VPN for untrusted networks.</strong> You will open the data room from a hotel or an airport at some point. NordVPN and Surfshark both handle that case. It protects the transport; it does nothing about where the data comes to rest, so do not let it substitute for the jurisdiction question above.</li>

<li><strong>A disposal plan.</strong> Undertakings usually require destruction or return at the close of the engagement, and deleting a file does not destroy it. Overwrite-based tools such as O&amp;O SafeErase cover the workstation case. If the volume was encrypted from the start, destroying the key is faster and cleaner than overwriting the data, and easier to certify.</li>
</ul>



<p class="wp-block-paragraph">If cloud storage is permitted, write-once object storage with a retention lock suits the ingested originals well. It gives you an immutability story that does not rest on your own discipline.</p>



<h2 class="wp-block-heading">Troubleshooting the ingest</h2>



<ul class="wp-block-list">
<li><strong>The parser dies on a decode error.</strong> XER files are not reliably UTF-8, particularly when they came from a database with a non-English locale. Try a single-byte fallback encoding before assuming the file is corrupt, and log which encoding succeeded against the version record.</li>

<li><strong>Row and field counts do not match.</strong> Activity names can contain characters that confuse a naive split, and some exports wrap fields inconsistently. Parse defensively on field count per <code>%F</code> line rather than assuming.</li>

<li><strong>Duplicate <code>task_code</code> in one file.</strong> One XER can carry several projects plus the shared global data they depend on. Always scope by <code>proj_id</code> and confirm you have the right project before anything else.</li>

<li><strong>Activity counts drop between updates for no reason.</strong> Check whether the export was filtered or taken at a WBS level rather than the full project. A partial export looks exactly like a deletion event in your diff.</li>

<li><strong>Dates shift by a few hours.</strong> P6 stores times, not just dates, and calendar start hours vary. Normalise to the activity calendar&#8217;s working day before comparing, or every finish looks a shift early.</li>

<li><strong>Float figures look wrong across the board.</strong> Check the hours-versus-days conversion and the calendar assignment before checking anything else. It is almost always one of those two.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the final update as the as-built without stating that assumption anywhere in the report.</li>

<li>Joining on <code>task_id</code> because it is the primary key, and inheriting whatever database history came with the file.</li>

<li>Letting the platform resolve an ambiguous activity match silently, so the analyst never sees the ambiguity existed.</li>

<li>Converting float to days with a fixed divisor, then reporting a single variance figure that mixes progress with programme edits.</li>

<li>Automating the record-to-activity link all the way to a stated cause, which turns an evidence tool into a target.</li>

<li>Modifying source XERs in place, then having no way to demonstrate they are as received.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Hash every source file on ingest and record the hash next to the version. It takes one line and it answers a question you will otherwise answer badly.</li>

<li>Make every table append-only. Corrections become new rows with a reason, never overwrites.</li>

<li>Build the first-appearance view early. It reframes the analysis from &#8220;what do the files say&#8221; to &#8220;when did the files start saying it&#8221;, which is the more defensible question.</li>

<li>Make every exhibit reproducible from one query, so a challenged figure traces back to a file and a row rather than a spreadsheet cell.</li>

<li>Keep a written record of every judgement call the platform surfaced and a human resolved. That log is often more useful in the hearing than the analysis itself.</li>

<li>Validate against the commercial tools where you can. If Acumen Fuse or Schedule Comparison disagrees with your diff on a given version pair, one of you is wrong and you want to know which before the report goes out.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is as-planned vs as-built analysis still accepted in construction disputes?</h3>



<p class="wp-block-paragraph">Yes, as one of a recognised family of methods. AACE Recommended Practice 29R-03 includes it in its taxonomy as an observational method, and the SCL Protocol&#8217;s second edition deliberately stopped naming a single preferred technique for retrospective analysis, setting out selection factors instead. Acceptance in a given dispute turns on the records available, the contract and the forum, not on the method&#8217;s reputation in the abstract.</p>



<h3 class="wp-block-heading">How many schedule updates do I need before building a platform is worth it?</h3>



<p class="wp-block-paragraph">Roughly: below about eight updates, a careful analyst with a workbook is faster. Above twenty, manual comparison stops being reliable, not because people are careless but because the number of pairwise comparisons grows past what anyone can hold. The crossover in practice sits somewhere in the low teens, and it moves earlier if there are multiple baselines or a renumbering event.</p>



<h3 class="wp-block-heading">Can I do as-planned vs as-built analysis without a Primavera P6 licence?</h3>



<p class="wp-block-paragraph">For the data work, yes. An XER is plain text and parses without P6 installed, and browser-based and open-source viewers exist. You will still want licensed access somewhere in the engagement for anything that requires recalculating the network, because reproducing P6&#8217;s scheduling engine faithfully is not a project you want inside a disputes timetable.</p>



<h3 class="wp-block-heading">What is the difference between as-planned vs as-built and a windows analysis?</h3>



<p class="wp-block-paragraph">The simple form of as-planned versus as-built compares planned dates against actual dates observationally, without necessarily using network logic. A windows analysis divides the project into periods, usually monthly, and examines the contemporaneous critical path and the critical delay in each period before investigating the cause. Windows demands more of the records and generally carries more weight where the programme was maintained properly. This platform supports both, because both need the same version history underneath.</p>



<h3 class="wp-block-heading">What do I do when the contractor&#8217;s updates were not maintained properly?</h3>



<p class="wp-block-paragraph">State it, quantify it, and let it drive method selection. That is exactly what the first-appearance table is for: reporting lag, actuals that moved after being stated, out-of-sequence progress and unexplained logic edits are all measurable. Recommended practice treats recreating a schedule from records as a last resort, used when contemporaneous data is unavailable, and recreated schedules carry less evidential weight. Showing the deficiency with numbers is stronger than asserting it.</p>



<h3 class="wp-block-heading">Should I build this or buy commercial delay analysis software?</h3>



<p class="wp-block-paragraph">Buy for schedule diagnostics and version comparison. That problem is well solved and the established tools have years of edge cases behind them. Build for the parts specific to your instruction: the record linking, the exhibit generation in your house format, and the audit trail. The hybrid is usually right, and the deciding factor is more often the licence and the timescale than the technical fit.</p>



<h3 class="wp-block-heading">Does the platform need to reproduce P6&#8217;s critical path calculation?</h3>



<p class="wp-block-paragraph">No, and it should not try. Read the float and date values P6 already calculated and stored in each export, and treat them as evidence of what the programme said at that moment. Anywhere you genuinely need a recalculated network, do it in the scheduling tool so the result is reproducible by anyone with the same file.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A credible as-planned vs as-built analysis is not a better comparison between two files. It is a complete, immutable, queryable history of what every schedule update said and when it started saying it. Build the version history and the first-appearance table first, and the exhibits fall out of it. Build the exhibits first and you will spend the hearing defending a spreadsheet whose provenance you cannot reconstruct.</p>



<p class="wp-block-paragraph">The scheduling judgement stays with the expert. The platform&#8217;s job is narrower and duller: make sure that when someone asks where a date came from, the answer takes one query and not one week.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need the data side of this built?</h2>



<p class="wp-block-paragraph">I build the engineering layer underneath delay and quantum work, so the expert spends their time on opinion rather than on data wrangling. That usually looks like:</p>



<ul class="wp-block-list">
<li>Ingesting a full sequence of P6 XER or XML updates into a versioned, append-only PostgreSQL model with hashing and chain of custody on every source file.</li>

<li>Building the first-appearance and reporting-lag views that show when each actual date entered the record, and flagging actuals that moved after being stated.</li>

<li>Classifying every between-update change into progress, duration, logic, calendar, constraint or scope, with before and after values on each row.</li>

<li>Extracting structured fields from daily reports, RFIs and correspondence with OCR and document processing, then generating candidate activity links for an analyst to confirm or reject.</li>

<li>Producing exhibit-grade output in Power BI or as static deliverables, where every figure traces back to a query, a file and a row.</li>

<li>Standing it up on infrastructure that fits the confidentiality regime you work under, whether that is a locked-down VPS or a machine inside the client&#8217;s own environment.</li>
</ul>



<p class="wp-block-paragraph">If you have a stack of XERs and a deadline, send me two consecutive updates and whatever the record-keeping looked like, and I will tell you what the sequence can and cannot support.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/as-planned-vs-as-built-analysis-platform/">As-Planned vs As-Built Analysis: Building a Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</title>
		<link>https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 20:19:52 +0000</pubDate>
				<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[As-Built Schedule]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[Bates Numbering]]></category>
		<category><![CDATA[Capital Projects]]></category>
		<category><![CDATA[Chain of Custody]]></category>
		<category><![CDATA[Common Data Environment]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Delay Analysis]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Extension of Time]]></category>
		<category><![CDATA[Forensic Schedule Analysis]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Hybrid Search]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[Load Files]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Oracle Aconex]]></category>
		<category><![CDATA[pgvector]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[Project Controls]]></category>
		<category><![CDATA[Provenance]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Reciprocal Rank Fusion]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[System of Record]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<category><![CDATA[XER Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=336</guid>

					<description><![CDATA[<p>A layer-by-layer architecture for a construction dispute intelligence platform: hashed intake and immutable originals, OCR with page coordinates, a versioned Primavera P6 schedule plane, an event model that joins documents to activities, hybrid retrieval on pgvector, and a generation layer bounded by human confirmation. Written around what each layer prevents.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/">Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that kills these systems is always the same one, and it always comes from the expert, not the engineer: &#8220;Where did this date come from?&#8221;</p>



<p class="wp-block-paragraph">You have a screen showing a beautiful chronology. Two hundred events, colour-coded by discipline, cross-referenced against the programme. The delay expert points at row 47, the one that says the piling rig arrived late, and asks which document that came from, who produced it, what page, and whether the other side has seen it. If the answer involves anyone opening a folder and searching, the platform has already failed. Not technically. Evidentially. A chronology nobody can trace back to a produced document is a nice picture, and a nice picture is worth nothing in an adjudication.</p>



<p class="wp-block-paragraph">That is the constraint that shapes everything else. A construction dispute intelligence platform is not a search engine with a construction theme on it. It is a provenance system that happens to do search. Get that ordering wrong and you build something impressive that no expert will sign their name to.</p>



<p class="wp-block-paragraph">This post walks through the architecture layer by layer: intake and chain of custody, normalization, the schedule data plane, the chronology model that joins the two, retrieval, and the tightly bounded generation layer on top. For each one I&#8217;ll say what it prevents, because that is the only reason any of these layers exist.</p>



<h2 class="wp-block-heading">What a construction dispute intelligence platform actually has to do</h2>



<p class="wp-block-paragraph">Strip away the vocabulary and the job is narrow. Take a pile of heterogeneous records from a capital project, some of them structured and some of them scanned faxes, and let a small number of expensive humans answer three questions quickly and defensibly:</p>



<ul class="wp-block-list">
<li>What happened, in what order, and who said so at the time?</li>



<li>What did the programme say before and after each of those things happened?</li>



<li>Which contemporaneous record supports every single assertion in the report?</li>
</ul>



<p class="wp-block-paragraph">The third one is not a nice-to-have. Both of the industry references people actually argue from lean on it. The Society of Construction Law Delay and Disruption Protocol raised record keeping to a core principle in its second edition and stopped expressing a preference between retrospective delay analysis methods, on the reasoning that the records you have largely determine the method available to you. AACE International&#8217;s Recommended Practice 29R-03 codifies forensic schedule analysis into a taxonomy of Method Implementation Protocols, and its minimum implementation protocols are essentially statements about which source schedules you must be able to produce. Both documents are, underneath, about evidence handling. So is your architecture.</p>



<h2 class="wp-block-heading">Layer one: intake that a forensic expert can defend</h2>



<p class="wp-block-paragraph">Everything upstream of this layer is somebody else&#8217;s mess. Site photos on a phone, a shared drive nobody pruned since mobilization, an Aconex export, a hard drive that arrived in an envelope. Your job at intake is to make an irreversible statement about what you received and when, before anyone touches it.</p>



<p class="wp-block-paragraph">Hash every file on arrival and store the hash next to the object, not in a spreadsheet. SHA-256 over the raw bytes, computed before any conversion, deduplication or renaming. The hash becomes the document&#8217;s identity for the rest of its life.</p>



<pre class="wp-block-code"><code># Build an arrival manifest before anything else runs
find ./ACME-INTAKE-001 -type f -print0 
  | xargs -0 sha256sum &gt; manifest-ACME-INTAKE-001.txt

# Re-verify later, after the files have moved through processing
sha256sum -c manifest-ACME-INTAKE-001.txt</code></pre>



<p class="wp-block-paragraph">Then put the originals somewhere they cannot be quietly edited. On AWS, S3 Object Lock in compliance mode with a retention period is the blunt instrument: once set, the object version cannot be overwritten or deleted by anyone, including the root account, until retention expires. Governance mode is the softer variant, where a specifically permissioned principal can lift the lock, which is fine for internal working copies and wrong for originals. If you are self-hosting on a Contabo or InterServer box because the client will not put project data in a hyperscaler, the equivalent discipline is a write-once staging volume plus an append-only manifest log, and you accept that you are relying on process rather than on the storage layer refusing the write.</p>



<h3 class="wp-block-heading">The manifest is the product</h3>



<p class="wp-block-paragraph">If the matter is already in formal disclosure, you will be handed productions rather than raw folders, and those arrive as load files. A Concordance DAT carries one row per document with fields like BegBates, EndBates, Custodian, BegAttach, EndAttach and an MD5 hash. An Opticon OPT carries one row per page, mapping Bates numbers to image paths. The delimiters are deliberately weird characters that real text rarely contains, and different tools disagree about the exact code points, so parse defensively and log what you detected rather than assuming.</p>



<p class="wp-block-paragraph">The failure mode here is quiet and expensive. Variable-width Bates numbers sort lexically instead of numerically, so ABC10 lands between ABC1 and ABC2, attachment ranges resolve to the wrong parents, and a family of documents silently splits. Nobody notices until an expert cites a covering email whose attachment, in your system, belongs to a different letter. Pad to fixed width on ingest and reconcile every BegAttach range against the documents you actually loaded.</p>



<h2 class="wp-block-heading">Layer two: normalization, and the OCR tax nobody budgets for</h2>



<p class="wp-block-paragraph">Construction corpora are the worst kind of mixed. Native Office files with clean text, born-digital PDFs, scanned site instructions with a signature and a coffee ring, photographs of whiteboards, and drawings where the only text that matters is inside a title block in the bottom right corner.</p>



<p class="wp-block-paragraph">Route by what the file actually is, not by extension. For a PDF, try text extraction first; if the extracted layer is empty or nearly empty, it is a scan and needs OCR. Tesseract via OCRmyPDF is the cheap path and it is genuinely good on clean typed pages. It degrades badly on handwriting, on stamps, and on anything skewed, which describes most site paperwork. A managed service such as Amazon Textract handles tables and forms better and returns geometry for every detected block, which matters more than accuracy alone.</p>



<p class="wp-block-paragraph">Geometry is the part teams skip and then rebuild six months later. If you store only the extracted string, you can prove a phrase exists somewhere in a document. If you store the page number and the bounding box, you can put a highlight on the exact line in the viewer, and the expert can confirm it in two seconds instead of reading eleven pages. That difference is the difference between a tool people use and a tool people abandon.</p>



<p class="wp-block-paragraph">Keep the OCR text separate from the original in storage, and record which engine and which settings produced it. When a disputed reading of a handwritten date turns up, and it will, you want to be able to say exactly how that character was produced.</p>



<h2 class="wp-block-heading">Layer three: the schedule plane is a separate data model</h2>



<p class="wp-block-paragraph">This is where most teams underestimate the work. Programme data is not documents. It is a graph, versioned, with a calendar attached, and it needs its own tables, its own ingest path and its own validation.</p>



<p class="wp-block-paragraph">Primavera P6 exports as XER, a tab-delimited text format with a table-per-section layout, or as P6 XML. Either way you are pulling out projects, the WBS tree, activities, logic links and calendars. In XER these live in sections including PROJECT, PROJWBS, TASK, TASKPRED and CALENDAR. Parsing it by hand is a bad use of a week; PyP6Xer is an open source Python parser that will get you to objects quickly.</p>



<pre class="wp-block-code"><code>from xerparser.reader import Reader

xer = Reader("update-14.xer")

for project in xer.projects:
    print(project.proj_short_name)
    for activity in project.activities:
        print(activity.task_code, activity.task_name)</code></pre>



<p class="wp-block-paragraph">Two things bite immediately. First, several duration and float fields in XER are stored in hours rather than days, which you can see in the column names themselves. If you load them straight into a &#8220;days&#8221; column because the numbers looked plausible on a five-day calendar, every float figure downstream is wrong by a factor that varies with the calendar. Check your own export before you trust any mapping, including this one.</p>



<p class="wp-block-paragraph">Second, activity IDs are not stable across a project&#8217;s life. Activities get renumbered, split, deleted and re-added between updates. If your schema keys on the activity ID alone, two updates will silently merge activities that were never the same work. Key on a composite of schedule version plus activity ID, and maintain a separate mapping table for identity across versions that a human can correct. That table is boring, it is not automatable, and it is the difference between a windows analysis that holds together and one that quietly compares apples to a different orchard.</p>



<p class="wp-block-paragraph">Store every update as an immutable snapshot with its data date. Never mutate a loaded schedule. The whole point of the layer is that you can reconstruct exactly what the programme said on a given day, which is precisely what a time slice or windows method needs.</p>



<h2 class="wp-block-heading">Layer four: the chronology is the join, and the join is the hard part</h2>



<p class="wp-block-paragraph">Documents on one side, schedule snapshots on the other. The product is the thing in the middle: an event, with a date, a description, a set of supporting document references, and a set of affected activities.</p>



<p class="wp-block-paragraph">Model it explicitly. An event is not a document and it is not an activity. One letter can create three events. One event can be evidenced by nine documents and a photograph. The relationship is many-to-many in both directions and every attempt I have seen to flatten it ends with someone maintaining a spreadsheet on the side.</p>



<pre class="wp-block-code"><code>-- Every assertion resolves to a page and a box on that page
CREATE TABLE evidence_ref (
  id            BIGSERIAL PRIMARY KEY,
  event_id      BIGINT NOT NULL REFERENCES event(id),
  document_id   UUID   NOT NULL REFERENCES document(id),
  page_number   INT    NOT NULL,
  bbox          JSONB,             -- normalized coordinates from OCR
  quoted_text   TEXT   NOT NULL,
  asserted_by   TEXT   NOT NULL,   -- 'analyst' or 'model'
  confirmed_by  TEXT,              -- NULL until a human confirms
  confirmed_at  TIMESTAMPTZ
);</code></pre>



<p class="wp-block-paragraph">Those last three columns are the ones that matter. They encode the difference between something the machine proposed and something a person stands behind. Report generation reads only rows where <code>confirmed_by</code> is set. Everything else is a suggestion queue. This is not a compliance box; it is the mechanism by which an expert can put their name on the output, and it needs to exist in the schema rather than in a workflow convention that erodes under deadline.</p>



<h2 class="wp-block-heading">Layer five: retrieval that finds document numbers, not just meaning</h2>



<p class="wp-block-paragraph">Pure vector search is the wrong default for this corpus, and it fails in a way that looks like success. Embeddings are good at meaning and bad at tokens. Ask for &#8220;RFI 0442&#8221; and a dense retriever will cheerfully hand you eleven chunks about requests for information, none of which is RFI 0442, because the identifier got smeared into its semantic neighbourhood. Construction disputes run on identifiers: RFI numbers, variation order references, drawing revisions, activity IDs, Bates ranges.</p>



<p class="wp-block-paragraph">Run lexical and semantic retrieval in parallel and fuse the rankings. PostgreSQL with the pgvector extension does both without adding infrastructure: an HNSW index over the embedding column, a GIN index over a generated tsvector column, and reciprocal rank fusion to merge the two candidate lists.</p>



<pre class="wp-block-code"><code>CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunk (
  id          BIGSERIAL PRIMARY KEY,
  document_id UUID NOT NULL,
  page_number INT  NOT NULL,
  body        TEXT NOT NULL,
  body_tsv    tsvector GENERATED ALWAYS AS
                (to_tsvector('english', body)) STORED,
  embedding   vector(1536)
);

CREATE INDEX chunk_tsv_idx  ON chunk USING gin (body_tsv);
CREATE INDEX chunk_hnsw_idx ON chunk USING hnsw (embedding vector_cosine_ops);</code></pre>



<p class="wp-block-paragraph">Over-fetch from each side and trim after fusion rather than pulling the final count from each; more candidates give the fusion step something to work with. Add a trigram index if your corpus has OCR noise, because &#8220;RFI 0442&#8221; and &#8220;RFl 0442&#8221; are the same document to a human and different tokens to Postgres.</p>



<p class="wp-block-paragraph">Filter hard on metadata before ranking. Matter, custodian, date range, privilege status. A retrieval layer that can return a privileged document into a shared workspace is a liability, not a feature.</p>



<h2 class="wp-block-heading">Layer six: generation, bounded hard</h2>



<p class="wp-block-paragraph">Language models are useful here and the useful jobs are smaller than the demos suggest: proposing candidate events from a batch of correspondence, drafting neutral one-line summaries, clustering near-duplicate records, suggesting which activities an event might touch. Every one of those is a proposal that lands in a review queue.</p>



<p class="wp-block-paragraph">The reason to be strict is not squeamishness. Stanford&#8217;s RegLab and HAI ran a preregistered evaluation of commercial legal research tools built on retrieval-augmented generation, with expert-scored answers, and found hallucination rates in the range of roughly 17 to 33 percent depending on the product, against vendor marketing that had claimed hallucination-free citations. The vendors disputed the characterization. The finding that should worry you more than the headline number is the failure mode underneath it: a citation that exists, is quoted accurately, and does not stand for what the answer says it stands for. Retrieval reduces fabrication. It does not produce correct reasoning about what a document means, and misgrounding is far harder to spot than an invented reference.</p>



<p class="wp-block-paragraph">So bound it. Generation writes to the proposal table, never to the report. Every generated assertion carries the chunk IDs it was built from, and the UI shows the source page beside the claim by default rather than behind a click. If a proposed event has no evidence reference, it does not enter the queue at all.</p>



<h2 class="wp-block-heading">Tenancy, privilege, and the thing that ends the engagement</h2>



<p class="wp-block-paragraph">Matters do not share data. Ever. Not &#8220;logically separated by a WHERE clause that we always remember to add&#8221;. The cheapest defensible pattern is a database per matter, or at minimum a schema per matter with connection-level scoping, so that a missing predicate produces an empty result rather than another client&#8217;s correspondence.</p>



<p class="wp-block-paragraph">Two more that get discovered late. Privilege state is per document per matter and it changes; model it as a versioned attribute with an audit trail, not a boolean someone flipped. And plan the exit at the start: matters end, retention clocks expire, and someone has to certify destruction. That means knowing every copy you made, including OCR text, thumbnails, embeddings and backups. Tools like the O&amp;O Software utilities have a place at the endpoint end of that job; the platform side is a deletion manifest that enumerates derived artifacts per document, written at ingest rather than reconstructed under pressure.</p>



<p class="wp-block-paragraph">If experts are working from home, from site cabins, or from three countries, the access path is part of the design. A managed VPN such as NordVPN or Surfshark on the client side plus network-level allowlisting on the platform side is a pragmatic combination, and it is worth writing into the engagement documentation so nobody improvises with a personal Dropbox at eleven at night.</p>



<h2 class="wp-block-heading">Where these platforms break in practice</h2>



<ul class="wp-block-list">
<li><strong>Chronology dates drift from document dates.</strong> Usually a timezone or a date-format collision: a scanned form read as DD/MM, an email header parsed as UTC, a schedule data date stored as a naive timestamp. Store the raw string you extracted alongside the parsed value and show both in the review UI.</li>



<li><strong>Float figures look wrong by a constant factor.</strong> Hours versus days in the schedule import. Check against the same activity opened in P6 before you debug anything else.</li>



<li><strong>Retrieval misses a document everyone knows exists.</strong> Either it was never OCR&#8217;d because the PDF had a thin text layer that passed your emptiness check, or the identifier is only in the title block of a drawing. Log extraction character counts per page and alert on suspiciously low ones.</li>



<li><strong>Two activities merged across updates.</strong> Composite keying was not applied, or an activity was renumbered. Diff activity code sets between consecutive snapshots and surface additions and deletions as a report the planner reviews.</li>



<li><strong>Document families split.</strong> Bates padding or an unreconciled attachment range from the load file. Re-run the family reconciliation and count orphaned attachments.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the schedule as documents. It is a graph with versions, and shoehorning it into the document store costs you every analysis you actually wanted.</li>



<li>Storing extracted text without page and coordinate provenance. Cheap now, a full reprocessing run later.</li>



<li>Vector-only retrieval. Looks great in a demo built from prose, falls over on the first RFI number.</li>



<li>Letting model output reach a deliverable without a recorded human confirmation.</li>



<li>Mutating loaded schedules to &#8220;fix&#8221; them. Fix on the way in, or hold the correction as an annotation. Never edit the snapshot.</li>



<li>Deferring the deletion story to the end of the matter, when the derived artifacts are already scattered.</li>
</ul>



<h2 class="wp-block-heading">Best practices for building a construction dispute intelligence platform</h2>



<ul class="wp-block-list">
<li>Hash at the door, lock the originals, and never process in place.</li>



<li>Make every derived artifact point back to a document ID, a page and a region.</li>



<li>Give schedules their own immutable, version-keyed store with the data date as a first-class field.</li>



<li>Model events separately from both documents and activities, with many-to-many links in each direction.</li>



<li>Run hybrid retrieval and fuse the rankings; filter on matter and privilege before you rank.</li>



<li>Keep generation in a proposal queue with confirmation recorded in the schema.</li>



<li>Isolate matters at the database or schema boundary, not in application code.</li>



<li>Instrument ingest as seriously as you would a production pipeline. Grafana over your extraction counts and OCR queue depth will tell you a corpus is going wrong days before an analyst does.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a dedicated vector database for this?</h3>



<p class="wp-block-paragraph">Almost certainly not. Matter corpora are large in gigabytes and small in chunk count compared to consumer-scale search. PostgreSQL with pgvector keeps the embeddings in the same transaction as the provenance rows, which means you cannot end up with a chunk whose document record was rolled back. That consistency is worth more here than the last few milliseconds of query latency.</p>



<h3 class="wp-block-heading">Should the platform pick a delay analysis method?</h3>



<p class="wp-block-paragraph">No. That choice belongs to the expert and depends on the contract, the forum, the records available and the timing of the analysis. What the platform should do is make it obvious which methods the record set can support, by reporting what you actually have: how many schedule updates, at what intervals, with what data dates, and how complete the contemporaneous correspondence is around each window.</p>



<h3 class="wp-block-heading">XER or P6 XML for schedule ingest?</h3>



<p class="wp-block-paragraph">Accept both, because you will be sent both. XER is what planners export by habit and it is easy to parse with existing libraries. P6 XML is more explicit about structure and less prone to encoding surprises. Whichever arrives, keep the original file, hashed, alongside the parsed rows.</p>



<h3 class="wp-block-heading">How much of the review can realistically be automated?</h3>



<p class="wp-block-paragraph">The ranking and the grouping, most of it. The judgement, none of it. Automation that reduces a hundred thousand documents to a few thousand worth reading is enormously valuable and uncontroversial. Automation that decides what a letter means is where you inherit risk you cannot price.</p>



<h3 class="wp-block-heading">Can I build this on top of a common data environment we already have?</h3>



<p class="wp-block-paragraph">Partly. A CDE such as Aconex is a good source of records and a poor system of record for a dispute, because access changes, exports are point-in-time, and the retention rules belong to the project rather than to the matter. Pull from it, hash what you pulled, and hold your own copy under your own lock.</p>



<h3 class="wp-block-heading">What does this cost to run?</h3>



<p class="wp-block-paragraph">The shape of the bill is more useful than a number. OCR and embedding are one-time per document and scale with page count, so a scan-heavy corpus costs multiples of a born-digital one. Storage is small and constant. Retrieval is negligible. Generation scales with how much you push through it, which is another argument for narrow, queued use rather than a chat box over the whole corpus.</p>



<h3 class="wp-block-heading">Is a general eDiscovery platform not enough?</h3>



<p class="wp-block-paragraph">For document review, often yes, and they are considerably better at productions and privilege logging than anything you will build in a quarter. What they do not have is the schedule plane. The value of a construction-specific build is the join between correspondence and programme versions, so a sensible architecture frequently uses an established review platform for documents and adds the schedule and chronology layers alongside it.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A construction dispute intelligence platform earns its place by making provenance cheap. Search, clustering and summarization are commodity capabilities now; anyone can bolt them onto a corpus in a fortnight. What is hard, and what decides whether an expert will rely on the output, is that every date, every event and every sentence in the eventual report resolves in one click to a specific page of a specific produced document, with a record of who confirmed it.</p>



<p class="wp-block-paragraph">Build the provenance spine first. Hash at intake, keep coordinates, version the schedules, model events as their own thing, and keep the model on the proposal side of a human confirmation. The clever features are easy to add afterwards. Retrofitting evidence handling into a system that was designed as a search tool is a rebuild, and it always arrives at the worst possible moment, which is about three weeks before a hearing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or fixing this?</h2>



<p class="wp-block-paragraph">I work on the infrastructure side of exactly this problem: getting messy project records into a system where every claim is traceable. Things I can help with:</p>



<ul class="wp-block-list">
<li>Designing the intake and chain-of-custody layer, including hashing, immutable storage, and load file parsing that survives odd delimiters and inconsistent Bates padding.</li>



<li>Building the schedule ingest path for XER and P6 XML, with version-keyed snapshots and an activity identity map that holds across updates.</li>



<li>Standing up hybrid retrieval on PostgreSQL and pgvector, including page-level and coordinate-level provenance so highlights land on the right line.</li>



<li>Setting up the OCR and normalization pipeline, with routing rules, quality checks, and alerting on pages that extracted suspiciously little text.</li>



<li>Matter isolation, privilege state modelling, audit logging, and a deletion manifest you can act on at the end of the engagement.</li>



<li>Reviewing an existing platform and telling you plainly where the provenance chain breaks.</li>
</ul>



<p class="wp-block-paragraph">If you have something concrete, send it over: an XER export, a sample DAT and OPT pair, a page count breakdown of your corpus, or the schema you are currently working from. It is much easier to say something useful about a real file than about a description of one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/">Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
