<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Tenant Isolation | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/tenant-isolation/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/tenant-isolation/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 17 Oct 2026 06:00:06 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Tenant Isolation | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/tenant-isolation/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Building a Personalized Learning Assistant on AWS Without Leaking Student Data</title>
		<link>https://john-nessime.com/blog/technical-guides/personalized-learning-assistant-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 17 Oct 2026 06:00:06 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Education Technology]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3 Vectors]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[EdTech]]></category>
		<category><![CDATA[FERPA]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Identity Propagation]]></category>
		<category><![CDATA[Learning Management System]]></category>
		<category><![CDATA[LTI 1.3]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[pgvector]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Student Data Privacy]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=466</guid>

					<description><![CDATA[<p>A personalized learning assistant on AWS rarely fails loudly. It fails by being helpful to the wrong student. Five failure families that decide whether an AI tutor built on Amazon Bedrock is safe to put in front of a class: LTI identity, retrieval isolation, guardrail limits, the cost shape of an academic term, and being able to prove what the assistant saw.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/personalized-learning-assistant-aws/">Building a Personalized Learning Assistant on AWS Without Leaking Student Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket that kicks this project into its second phase usually reads something like: &#8220;The assistant is giving my students the answers.&#8221; Attached is a screenshot where the tutor bot cheerfully walks a student through the marking scheme for a coursework brief that has not been released yet. Nothing crashed. No 5xx, no alarm, no failed health check. The rubric PDF had been dropped into the same bucket prefix as the lecture notes weeks earlier, and retrieval did exactly what retrieval does.</p>



<p class="wp-block-paragraph">That is the shape of every serious problem in this space. A personalized learning assistant on AWS almost never fails loudly. It fails by being helpful to the wrong person, or by being confidently wrong in a way that reads as authoritative because it came from the institution&#8217;s own system. Both failures are silent until a human reports them, and by then the assistant has been answering that way for a term.</p>



<p class="wp-block-paragraph">This post covers the parts that are genuinely hard: getting identity and scope out of the learning platform instead of the browser, making retrieval respect the boundary, knowing what Bedrock Guardrails will and will not do for you, sizing a system whose traffic collapses every summer, and proving after the fact what the assistant saw. I assume you can already stand up a RAG pipeline. The pipeline is the easy part.</p>



<h2 class="wp-block-heading">What &#8220;personalized&#8221; has to mean before you build anything</h2>



<p class="wp-block-paragraph">Personalization in an education setting is three separate questions that people tend to collapse into one:</p>



<ul class="wp-block-list">
<li><strong>What is this learner allowed to see?</strong> Course enrollment, section, release dates, role. This is an authorization question and it belongs nowhere near the model.</li>



<li><strong>What has this learner already done?</strong> Submissions, attempts, viewed material. This is state, and it belongs in a database you control.</li>



<li><strong>How is this learner doing?</strong> Mastery, misconceptions, pace. This is inference, and it is the only part where a model earns its keep.</li>
</ul>



<p class="wp-block-paragraph">Teams that skip straight to the third question end up asking the model to enforce the first one through prompt instructions. That does not hold. A system prompt saying &#8220;do not reveal solutions before the due date&#8221; is a suggestion; a retrieval filter that never returns the solution chunk is a control. Keep the boundary in the retrieval layer and the personalization in the generation layer, and most of the scary failure modes stop being possible rather than becoming unlikely.</p>



<h2 class="wp-block-heading">Where a personalized learning assistant on AWS actually breaks</h2>



<p class="wp-block-paragraph">Organized by failure family rather than by service, because the services are documented and the failures are not. Five of them, roughly in the order they bite.</p>



<h2 class="wp-block-heading">Failure family one: identity that stops at the front door</h2>



<p class="wp-block-paragraph">The assistant lives inside Canvas, Moodle, Blackboard, Brightspace or Open edX. It gets there through LTI 1.3, which replaced the old shared-secret model with an OpenID Connect launch and an RSA-signed JWT. The platform initiates a login against your tool&#8217;s login endpoint, your tool responds with an auth request, the platform posts back a signed token, and you validate it against the platform&#8217;s published keys.</p>



<p class="wp-block-paragraph">Almost everyone gets that far. The failure is what happens next.</p>



<p class="wp-block-paragraph">The launch token carries a stable subject identifier, a roles claim, a context claim describing the course, and a deployment identifier, all namespaced under the 1EdTech spec URIs. Teams validate the signature, pull the user ID out, then build the rest of the session from whatever the front end sends afterwards. Course ID as a query parameter. Role in local storage. It works in testing because the front end sends the truth, and it stops working the moment a curious student opens developer tools. If your filter values arrive from the client, you have built an access control system that asks the attacker what they should be allowed to see.</p>



<h3 class="wp-block-heading">Take the scope from the launch, then keep it server-side</h3>



<p class="wp-block-paragraph">The pattern that holds up:</p>



<ol class="wp-block-list">
<li>Validate the launch JWT against the platform&#8217;s key set. Check the issuer, the audience, the nonce and the deployment identifier, not just the signature.</li>



<li>Extract subject, roles and context from the token and write them into a server-side session. Hand the browser an opaque session cookie and nothing else.</li>



<li>Derive every retrieval filter from that session record. The client never names a course, a section or a role.</li>



<li>If you need the full roster rather than just the launching user, call Names and Role Provisioning Services with an OAuth 2.0 client credentials token. That gives you learners who have never launched the tool, which matters for cohort-level features.</li>



<li>If you write grades back, do it through Assignment and Grade Services against a line item you created. A launch that works tells you nothing about whether passback works.</li>
</ol>



<p class="wp-block-paragraph">Test launch, roster fetch and grade passback as three independent contracts. They fail independently and they fail quietly. A term can start with a tool that launches beautifully and drops every grade on the floor because AGS was never wired to a line item.</p>



<p class="wp-block-paragraph">Worth knowing before you commit: LTI 1.3 is meaningfully more setup than 1.1. Key registration, the OIDC handshake, per-deployment configuration on every platform instance. That is the price of not signing student traffic with a shared secret, and it is worth paying, but budget for it honestly. If you need to test against several platforms, a self-managed Moodle or Open edX instance on a cheap VPS from somewhere like Contabo or InterServer beats waiting on sandbox tenants, and you get to break it freely.</p>



<h2 class="wp-block-heading">Failure family two: retrieval that ignores the boundary</h2>



<p class="wp-block-paragraph">This is the one from the opening. Vector search does not know about courses. It knows about similarity. If the answer key is semantically close to the question, it comes back, and the model uses it, and the response is fluent and correct and completely inappropriate.</p>



<p class="wp-block-paragraph">Amazon Bedrock Knowledge Bases handles this through metadata filtering: you attach attributes to each document at ingestion, then pass a filter at query time that pre-filters the vector store before the similarity search runs. The filter is the control. Everything else is decoration.</p>



<h3 class="wp-block-heading">The sidecar file is the only metadata that counts</h3>



<p class="wp-block-paragraph">For an S3 data source, attributes come from a companion file sitting next to the document with <code>.metadata.json</code> appended to the object key. This trips people up constantly, because S3 objects already have user-defined metadata and object tags, and neither of those is read for retrieval filtering. Keep lifecycle and cost allocation in S3 tags where they belong, and put filterable attributes in the sidecar.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "course_id": "BIO201",
    "section_id": "BIO201-B",
    "institution_id": "uni-main",
    "material_type": "lecture_notes",
    "visibility": "student"
  }
}</code></pre>



<p class="wp-block-paragraph">Note <code>visibility</code> as a first-class attribute rather than an inference from folder structure. Rubrics, model answers and marking schemes get <code>visibility: "instructor"</code>, and the student-facing filter excludes them by value, not by path convention. Path conventions get violated the first time a busy tutor uploads a file to the wrong folder.</p>



<p class="wp-block-paragraph">At query time you compose the filter server-side from the session record. Bedrock&#8217;s retrieval configuration supports boolean composition, so the tenant condition and the visibility condition are separate clauses that must both hold:</p>



<pre class="wp-block-code"><code>{
  "vectorSearchConfiguration": {
    "filter": {
      "andAll": [
        { "equals": { "key": "institution_id", "value": "uni-main" } },
        { "equals": { "key": "course_id",      "value": "BIO201"   } },
        { "equals": { "key": "visibility",     "value": "student"  } }
      ]
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two habits make this survive production. Build the filter through a single function that always prepends institution and course, so no future endpoint can forget it. Then write a test that queries a known instructor-only chunk&#8217;s exact wording from a student session and asserts zero results. Manual smoke testing never catches this, because human testers ask reasonable questions and the failure only appears when someone asks an unreasonable one.</p>



<h3 class="wp-block-heading">Silo, pool or bridge</h3>



<p class="wp-block-paragraph">AWS frames multi-tenant isolation as three patterns, and the trade-off is the same one you always get between blast radius and cost:</p>



<ul class="wp-block-list">
<li><strong>Silo</strong> gives each institution its own knowledge base and its own vector index. Strongest isolation, easiest compliance conversation, and you multiply your idle infrastructure cost by the number of tenants. There is also a per-account service quota on knowledge bases, so check Service Quotas before you promise a hundred districts their own.</li>



<li><strong>Pool</strong> puts everyone in one index and relies entirely on metadata filtering. Cheapest by a wide margin, and one filter bug is a cross-institution disclosure rather than an embarrassment.</li>



<li><strong>Bridge</strong> splits the difference: separate knowledge bases sharing underlying vector infrastructure, which keeps the isolation story cleaner without paying for a full stack per tenant.</li>
</ul>



<p class="wp-block-paragraph">For a single university running many courses, pool with rigorous filtering is usually right, because the institution is the tenant and the courses are not. For an edtech vendor selling to multiple districts, I would start at bridge and only pool if the economics force it, because the conversation with a district&#8217;s data protection officer goes very differently when your answer is &#8220;we filter&#8221; rather than &#8220;they are separate indexes.&#8221;</p>



<h2 class="wp-block-heading">Failure family three: guardrails that do less than you assume</h2>



<p class="wp-block-paragraph">Bedrock Guardrails is a useful layer and the first thing I reach for on an education workload: content filters, denied topics, word filters, PII detection and contextual grounding checks, applied at inference time and usable across models. For a system that teenagers will deliberately try to provoke, enforcement inside the invocation path beats anything bolted on afterwards. Three limits are documented and routinely missed.</p>



<p class="wp-block-paragraph"><strong>Contextual grounding does not cover chatbots.</strong> AWS documents the supported use cases as summarization, paraphrasing and question answering, and states plainly that conversational QA and chatbot use cases are not supported. A tutoring assistant is a multi-turn conversation. If your hallucination story is &#8220;we turned on grounding checks,&#8221; read that page before it reaches a procurement document. The check still earns its place on single-shot retrieve-and-answer paths, which is where I would apply it, but a multi-turn tutor needs citation display and human review too.</p>



<p class="wp-block-paragraph"><strong>PII detection is probabilistic and text-only.</strong> The sensitive information filter is a machine learning model that uses surrounding context to decide whether something is an identifier. It works better on prose than on bare fragments. AWS also documents that it operates on text output and does not inspect tool use parameters, so if your assistant calls a function with a student identifier in the arguments, that path is not covered. Do not treat it as a substitute for not putting identifiers in the prompt in the first place.</p>



<p class="wp-block-paragraph"><strong>Blocked content lands in your logs.</strong> AWS documents that content blocked by guardrail policies appears as plain text in Bedrock model invocation logs when those logs are enabled. That is defensible for debugging and awkward when the blocked content is a distressed message from a fourteen year old. Decide deliberately whether invocation logging is on, where it lands, who can read it, and how long it lives. In education this is a governance decision, not a logging default.</p>



<p class="wp-block-paragraph">None of this argues against Guardrails. It argues for knowing its edges, because the gap between what a feature name suggests and what the documentation says is where compliance commitments go to die.</p>



<h2 class="wp-block-heading">Failure family four: the cost shape of an academic term</h2>



<p class="wp-block-paragraph">Education traffic is not a smooth curve. It is near zero over summer, spikes brutally in the week before every assessment deadline, and dies again. That shape punishes the default architecture in a specific way.</p>



<p class="wp-block-paragraph">The console nudges you toward Amazon OpenSearch Serverless as the vector store. It bills per OpenSearch Compute Unit hour with a minimum baseline footprint that exists whether or not anyone queries it, so an empty index over a quiet August still bills. AWS has adjusted those minimums over time, so pull current figures from the OpenSearch Serverless capacity documentation rather than from any blog post, this one included. The mechanism is the part that matters: your floor is set by capacity that exists, not by traffic that happens.</p>



<p class="wp-block-paragraph">Bedrock Knowledge Bases supports other backends, and the question that decides between them is idle cost:</p>



<ul class="wp-block-list">
<li><strong>Aurora PostgreSQL with pgvector.</strong> A first-class documented option. Aurora Serverless v2 bills per Aurora Capacity Unit hour and can scale down to zero when idle, which fits the academic calendar almost too neatly. You take on index lifecycle work in exchange: choosing between HNSW and IVFFlat, keeping statistics fresh, watching bloat. If your institution already runs Postgres, that is work your DBAs recognize.</li>



<li><strong>Amazon S3 Vectors.</strong> Bills on storage and requests rather than provisioned capacity, so genuine idle costs approach nothing. The trade is retrieval latency, which is higher than a warm search cluster. For a tutor where a response already takes a second or two of model time, that may not be the bottleneck you think it is.</li>



<li><strong>OpenSearch Serverless.</strong> The right answer when retrieval latency is genuinely your constraint, when you need hybrid search and k-NN tuning, or when you already run collections for other workloads and the baseline is already paid for.</li>
</ul>



<p class="wp-block-paragraph">On the model side, the cost driver is retrieved context, not the question. Every turn ships the retrieved chunks as input tokens, so chunk size and the number of results retrieved multiply directly into your bill on every single message. Halving retrieved results is usually a bigger lever than switching to a cheaper model, and it often improves answer quality too, because the model has less irrelevant material to get distracted by.</p>



<p class="wp-block-paragraph">Tag by course and faculty from day one. The first budget question you will be asked is which department is driving the spend, and retrofitting cost allocation tags after a pilot goes institution-wide is miserable. Cost Explorer answers it if the tags exist; tools like Vantage or CloudZero make the per-course view easier to hand to a finance team that does not want an AWS login.</p>



<h2 class="wp-block-heading">Failure family five: no evidence when someone asks</h2>



<p class="wp-block-paragraph">FERPA governs education records in the United States, and the mechanism that lets a vendor touch them at all is the school official exception: the institution must maintain direct control over the vendor&#8217;s use and maintenance of the records. Similar logic applies under other regimes, and many districts and state contracts add data residency requirements that FERPA itself does not impose.</p>



<p class="wp-block-paragraph">What that means architecturally is less exotic than it sounds, and it is mostly about being able to answer questions after the fact:</p>



<ul class="wp-block-list">
<li>Log the retrieval, not just the answer. Store which chunk identifiers were returned for each turn. When an instructor claims the assistant leaked a rubric, the retrieval log is the only thing that settles it.</li>



<li>Keep student identifiers out of prompts. Pass an opaque session-scoped pseudonym and resolve it in your own system. The model does not need a real name to be helpful.</li>



<li>Encrypt with customer managed KMS keys and use encryption context that names the tenant, so a key policy can actually constrain who decrypts what.</li>



<li>Pin the region and enforce it with service control policies if residency is contractually required. An SCP is checkable evidence; a runbook is not.</li>



<li>Write down retention for conversation history and enforce it with lifecycle rules. &#8220;We keep it forever because storage is cheap&#8221; is a defensible engineering position and an indefensible one in an education procurement review.</li>



<li>Pull the current compliance reports and agreements from AWS Artifact rather than quoting a blog about what AWS certifies.</li>
</ul>



<p class="wp-block-paragraph">None of this is glamorous and all of it is cheaper to build in during the pilot than to retrofit when the institution&#8217;s legal team gets involved.</p>



<h2 class="wp-block-heading">Troubleshooting the things that actually page you</h2>



<h3 class="wp-block-heading">&#8220;The new material isn&#8217;t showing up&#8221;</h3>



<p class="wp-block-paragraph">Almost always ingestion, not retrieval. Walk it in order rather than guessing. Did a sync actually run after the change? For an S3 data source, an edit is invisible until an ingestion job processes it.</p>



<pre class="wp-block-code"><code>aws bedrock-agent list-ingestion-jobs 
  --knowledge-base-id KB_ID 
  --data-source-id DS_ID</code></pre>



<p class="wp-block-paragraph">Look for a completed job starting after the upload. If the most recent job predates the change, your scheduler or event handler is the bug and retrieval is innocent. If a job did run, read its statistics: a modified-documents-indexed count of zero when you expected a change means change detection did not fire, which happens when an object is re-uploaded with identical bytes. A non-zero failed count means your file is probably the one that failed, and the failure reasons say why.</p>



<h3 class="wp-block-heading">&#8220;The filter returns nothing at all&#8221;</h3>



<p class="wp-block-paragraph">Query once with no filter to confirm the content is indexed. If it comes back unfiltered and vanishes when filtered, it is a metadata problem: the sidecar was not ingested, the attribute name does not match exactly, or the value differs in case. These comparisons are literal. <code>BIO201</code> and <code>bio201</code> are two different courses as far as the filter is concerned, which is an argument for normalizing identifiers at both ingestion and query time in one shared helper.</p>



<h3 class="wp-block-heading">&#8220;The launch shows a blank iframe&#8221;</h3>



<p class="wp-block-paragraph">Two suspects dominate. Cookies: your tool is in a cross-site iframe, so session cookies need the right SameSite and Secure attributes or the browser drops them and every launch looks like a fresh unauthenticated user. Registration mismatch: issuer, client ID or deployment ID differs between what the platform sends and what you stored. Log the received issuer and deployment ID on validation failure. Guessing at OIDC failures is slow; the values are right there.</p>



<h3 class="wp-block-heading">&#8220;Answers are right but the citations are wrong&#8221;</h3>



<p class="wp-block-paragraph">Usually chunking. Slides and worksheets fragment badly under naive fixed-size chunking, and a chunk that starts mid-table cites a heading that belongs to something else. Before you change models, look at what the retriever actually returned for the query that misbehaved. Nine times out of ten the model was faithful to a bad chunk.</p>



<h3 class="wp-block-heading">&#8220;Grades aren&#8217;t reaching the gradebook&#8221;</h3>



<p class="wp-block-paragraph">Check that a line item exists and that your service token carries the right AGS scopes. Passback failures are asynchronous, so they surface as an empty gradebook column weeks later rather than as an error the student sees. Alert on passback failure rate explicitly; nobody will report it in time.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Enforcing access control in the system prompt.</strong> Instructions are not permissions. If a chunk can be retrieved, assume it can be surfaced.</li>



<li><strong>Trusting client-supplied course or role values.</strong> Derive scope from the validated launch token and store it server-side, always.</li>



<li><strong>Ingesting the whole course folder.</strong> Instructor material and student material in one prefix is how rubrics leak. Separate at ingestion, tag at ingestion.</li>



<li><strong>Putting real names and student numbers in prompts.</strong> Pseudonymize before the model call and resolve afterwards.</li>



<li><strong>Sizing the vector store from peak week.</strong> The bill is set by the idle floor, and in education idle is most of the year.</li>



<li><strong>Assuming a guardrail feature does what its name implies.</strong> Read the supported use cases. Contextual grounding on a chatbot is the classic example.</li>



<li><strong>Piloting with one enthusiastic lecturer.</strong> That lecturer&#8217;s material is unusually well organized. The second department will not be.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Build the retrieval filter in exactly one function, and make the tenant and visibility clauses impossible to omit.</li>



<li>Keep a small regression set of adversarial queries, including students asking for solutions and asking about classmates, and run it on every deployment.</li>



<li>Show citations in the interface. Instructors trust the system faster when they can see the source, and it converts silent hallucinations into reportable ones.</li>



<li>Give instructors a preview mode that answers as a student would see it. They will find scoping errors far faster than any test suite.</li>



<li>Route the tool endpoint through a CDN and WAF such as Cloudflare, with rate limiting per session. Assessment week is a load test you did not schedule.</li>



<li>Dashboard retrieval count, token spend per course and passback failure rate together. CloudWatch metrics fronted by Amazon Managed Grafana or Grafana Cloud gives faculty a view without handing out console access.</li>



<li>Have an escalation path for distressing content that reaches a human being, and make sure the people on it know they are on it. This is a policy question you must answer before launch, not after.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do we need to fine-tune a model for our curriculum?</h3>



<p class="wp-block-paragraph">Almost certainly not. Curriculum content changes every term, and retrieval updates in minutes where fine-tuning is a rebuild. Fine-tuning earns its place for style and format consistency, not for facts. Start with retrieval, and revisit only if you have a specific behavior that prompting genuinely cannot produce.</p>



<h3 class="wp-block-heading">How do we stop it from just doing the homework?</h3>



<p class="wp-block-paragraph">Partly retrieval scope, partly prompting, partly accepting you cannot fully win. Exclude solutions and rubrics from student-visible retrieval, prompt for Socratic responses on assessed material, flag the assessment context on the way in so the assistant knows which mode it is in. A determined student can paste the question into a consumer chatbot regardless, which is a course design question rather than an infrastructure one.</p>



<h3 class="wp-block-heading">Is Bedrock the right choice versus calling a model API directly?</h3>



<p class="wp-block-paragraph">If the institution is already on AWS, Bedrock keeps the traffic inside your account boundary, gives you IAM and CloudTrail on the same footing as everything else, and can be reached over VPC endpoints. A direct provider API is often simpler and sometimes gets new models sooner. The deciding factor is usually procurement: one existing AWS agreement is a much shorter conversation than a new vendor review.</p>



<h3 class="wp-block-heading">Does student data get used to train the model?</h3>



<p class="wp-block-paragraph">AWS states that Bedrock does not use customer inputs or outputs to train foundation models. Verify it against current AWS documentation and your agreement rather than any third-party summary, because this is exactly the clause a district will ask you to evidence in writing.</p>



<h3 class="wp-block-heading">What does it cost to run a personalized learning assistant on AWS?</h3>



<p class="wp-block-paragraph">Model your bill as three parts: a vector store floor that bills whether or not anyone asks anything, a one-off embedding cost when you ingest the corpus, and per-turn inference dominated by retrieved context rather than by the question. For a small corpus with a term-shaped traffic curve, the floor usually dominates, which is why the vector store choice matters more than the model choice.</p>



<h3 class="wp-block-heading">How do we know whether it actually helps anyone learn?</h3>



<p class="wp-block-paragraph">Decide the measure before launch and instrument for it. Usage is not learning. Pair the assistant&#8217;s logs with assessment outcomes on comparable cohorts and accept that a clean answer needs a study design, not a dashboard. Engineering can supply the data; it cannot supply the conclusion.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A personalized learning assistant on AWS will never tell you when it is wrong about who someone is. The model has no concept of enrollment, and retrieval has no concept of a due date. Both live entirely in the metadata you attach at ingestion and the filter you build server-side from a validated launch token. Get that right and what remains is ordinary engineering: chunking, cost curves, dashboards. Get it wrong and you ship something that works flawlessly right up until the week it embarrasses the institution.</p>



<p class="wp-block-paragraph">Build the boundary first. Make it a control, not an instruction. Then go be clever with the tutoring.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand building or auditing one of these?</h2>



<p class="wp-block-paragraph">I work on the infrastructure side of AI assistants for education, which mostly means the parts that decide whether the thing is safe to put in front of students:</p>



<ul class="wp-block-list">
<li>Designing and reviewing the LTI 1.3 launch path, roster access and grade passback, including the cross-site cookie and registration issues that cause blank-iframe launches.</li>



<li>Building the retrieval isolation layer: sidecar metadata schemas, server-side filter composition, and an adversarial regression suite that proves instructor material stays invisible to students.</li>



<li>Choosing and migrating the vector store when the OpenSearch Serverless floor is eating a pilot budget, including Aurora pgvector and S3 Vectors options.</li>



<li>Guardrail configuration that matches the documented behavior rather than the feature name, plus a logging and retention design you can defend in a procurement review.</li>



<li>Cost allocation tagging by course and faculty, with dashboards that answer &#8220;which department is spending this&#8221; without handing out AWS console access.</li>



<li>Infrastructure as code for the whole stack so a second institution is a variable file rather than a fortnight.</li>
</ul>



<p class="wp-block-paragraph">If you have something concrete, send it over: a retrieval filter you are not sure is airtight, an ingestion job that keeps skipping documents, a failing LTI launch log, or a Cost Explorer breakdown that stopped making sense. Easier to look at the real thing than to talk in the abstract.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/personalized-learning-assistant-aws/">Building a Personalized Learning Assistant on AWS Without Leaking Student Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Property SaaS Document Storage: The Five Failures That Show Up in an Audit</title>
		<link>https://john-nessime.com/blog/cloud-security/property-saas-document-storage/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 28 Sep 2026 06:04:50 +0000</pubDate>
				<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Real Estate Technology]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[ABAC]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS STS]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[Cross-Tenant Leakage]]></category>
		<category><![CDATA[Crypto Shredding]]></category>
		<category><![CDATA[Data Minimization]]></category>
		<category><![CDATA[Document Retention]]></category>
		<category><![CDATA[Encryption Context]]></category>
		<category><![CDATA[File Upload Security]]></category>
		<category><![CDATA[GDPR]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[Object Storage]]></category>
		<category><![CDATA[Presigned POST]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[Property Management Software]]></category>
		<category><![CDATA[Real Estate Data]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[SaaS Architecture]]></category>
		<category><![CDATA[Secure Disposal]]></category>
		<category><![CDATA[Stored XSS]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=898</guid>

					<description><![CDATA[<p>Storing tenant IDs, leases and referencing packs is easy until someone asks who actually read them. Five failure families in property SaaS document storage, with the IAM policies, presigned URL mechanics, upload controls and deletion patterns that close each one.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/property-saas-document-storage/">Property SaaS Document Storage: The Five Failures That Show Up in an Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket that starts this is almost never &#8220;our document storage is insecure.&#8221; It&#8217;s a security questionnaire from a prospect, or a data subject access request, or an insurer asking one specific question: list everyone who has viewed this tenant&#8217;s ID document in the last twelve months.</p>



<p class="wp-block-paragraph">So you open the audit table. And you find it records who <em>requested</em> a link to the document. Not who opened it. The actual downloads never touched your application at all, because you handed out a signed storage URL and the browser went straight to the bucket. Your log is a record of intentions.</p>



<p class="wp-block-paragraph">That gap is typical of how property SaaS document storage goes wrong. The failures are rarely dramatic. Nobody dumps a bucket publicly. The guarantees you thought you had just turn out to be enforced in application code that ran once, long before the file was actually read. What follows is five failure families worth designing against when a platform stores tenant ID scans, signed leases, deposit certificates, right-to-rent evidence and referencing packs.</p>



<h2 class="wp-block-heading">What&#8217;s actually in the pile</h2>



<p class="wp-block-paragraph">Before architecture, be honest about the contents. A property platform&#8217;s document store is at least four risk classes wearing the same file extension:</p>



<ul class="wp-block-list">
<li><strong>Identity evidence.</strong> Passport and driving licence scans, visa documents, sometimes a selfie held next to the document. Highest blast radius, lowest genuine need for long retention.</li>



<li><strong>Contracts.</strong> Tenancy agreements, addenda, guarantor deeds, signed inventories. Legally driven retention, plus an integrity requirement: proving the file hasn&#8217;t changed matters more than hiding it.</li>



<li><strong>Financial and referencing material.</strong> Bank statements, payslips, credit reference reports. Sensitive, third-party sourced, and often covered by contractual restrictions from the referencing provider.</li>



<li><strong>Operational clutter.</strong> Inspection photos, safety certificates, maintenance invoices. Low sensitivity, high volume, and the thing that dominates your storage bill.</li>
</ul>



<p class="wp-block-paragraph">These want different retention clocks, different access rules and arguably different buckets. Treating them as one &#8220;documents&#8221; table is where most of the later pain originates.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure one: the access check and the access happen at different times</h2>



<p class="wp-block-paragraph">Presigned URLs are the right pattern. They keep the bucket private, take the transfer off your servers, and mean you never hand storage credentials to a browser. I reach for them first.</p>



<p class="wp-block-paragraph">The problem is what a presigned URL <em>is</em>. It&#8217;s a request you already authorised, signed, and then gave away. Anyone holding it can replay it until it expires. Your permission logic ran once and never runs again. Which means all of these keep working after the fact:</p>



<ul class="wp-block-list">
<li>A letting agent who left the company yesterday and whose account you disabled this morning.</li>



<li>A link forwarded into a WhatsApp group, pasted into a support ticket, or sitting in browser history on a shared office machine.</li>



<li>A document the tenant asked you to delete, if the object is still there behind a delete marker.</li>
</ul>



<h3 class="wp-block-heading">Make the URL die when the session does</h3>



<p class="wp-block-paragraph">Amazon&#8217;s own guidance is clearer on this than most people realise. A presigned URL created with temporary credentials expires when those credentials expire, <em>even if you asked for a longer expiry</em>. It also stops working if the signing credential is revoked, deleted or deactivated.</p>



<p class="wp-block-paragraph">That&#8217;s a lever, not a footnote. Sign with a long-lived IAM user key and the seven day maximum the SDK allows is a real seven days. Sign inside a short-lived assumed-role session and the URL cannot outlive that session, whatever number you passed in. So: never sign document URLs with static credentials. Assume a role per request, tag the session with the tenant, sign inside it.</p>



<pre class="wp-block-code"><code>import boto3

sts = boto3.client("sts")

# The tag key here must match the principal tag key used in the role's policy.
session = sts.assume_role(
    RoleArn="arn:aws:iam::111122223333:role/TenantDocumentAccess",
    RoleSessionName="doc-download",
    Tags=[{"Key": "TenantID", "Value": tenant_id}],
    DurationSeconds=900,
)["Credentials"]

s3 = boto3.client(
    "s3",
    aws_access_key_id=session["AccessKeyId"],
    aws_secret_access_key=session["SecretAccessKey"],
    aws_session_token=session["SessionToken"],
)

url = s3.generate_presigned_url(
    "get_object",
    Params={
        "Bucket": "prop-docs",
        "Key": f"tenants/{tenant_id}/identity/{doc_id}.pdf",
        "ResponseContentDisposition": 'attachment; filename="id-document.pdf"',
        "ResponseContentType": "application/octet-stream",
    },
    ExpiresIn=120,
)</code></pre>



<p class="wp-block-paragraph">Two things are doing work. <code>DurationSeconds</code> caps the real lifetime of the link regardless of <code>ExpiresIn</code>. And <code>ResponseContentDisposition</code> overrides how the object is served on this request, which matters for the serving problem further down.</p>



<p class="wp-block-paragraph">The trade-off is real: an STS call per download adds latency, adds a dependency, and STS has request limits you can hit at volume. AWS&#8217;s prescriptive guidance says as much. For identity documents I&#8217;d pay that cost. For inspection photos I wouldn&#8217;t.</p>



<h3 class="wp-block-heading">Log where the read actually happens</h3>



<p class="wp-block-paragraph">Your application log records issuance. To record retrieval you need logging at the storage layer: CloudTrail data events for object-level operations, or S3 server access logging. Data events are billed per event, so scope them to the identity and contract prefixes rather than the bucket holding thousands of inspection photos.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure two: isolation that lives in your ORM instead of your IAM policy</h2>



<p class="wp-block-paragraph">Ask most teams how tenant isolation works and you get a description of a WHERE clause. The key is built as <code>tenants/{tenant_id}/...</code> and the tenant ID comes from the session. Fine until one code path takes it from a request parameter, or a background job iterates without scoping, or someone writes an admin endpoint in a hurry.</p>



<p class="wp-block-paragraph">Move the boundary below your code. With attribute-based access control the tenant becomes a session tag and the IAM policy interpolates it into the resource ARN. One role serves every tenant, and the policy count doesn&#8217;t grow as you onboard.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadWriteOwnTenantPrefix",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::prop-docs/tenants/${aws:PrincipalTag/TenantID}/*"
    },
    {
      "Sid": "ListOnlyOwnTenantPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::prop-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": "tenants/${aws:PrincipalTag/TenantID}/*"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Two statements because S3 splits the problem. The first governs the object, where the tenant prefix is part of the resource ARN. The second governs listing, a bucket-level action, so the prefix has to be constrained through a condition instead. Skip it and a tenant can enumerate every key in the bucket even though they can&#8217;t read any of them, and key names in this domain leak plenty on their own.</p>



<p class="wp-block-paragraph">The failure mode worth internalising: if a code path forgets the session tag, <code>${aws:PrincipalTag/TenantID}</code> resolves to nothing and the statement doesn&#8217;t match. That&#8217;s a deny, which is what you want. But it only holds if the role has no broader fallback statement. One well-meaning <code>s3:GetObject</code> on the whole bucket, added to unblock a migration script, quietly turns the scheme into decoration.</p>



<h3 class="wp-block-heading">When to use a bucket per tenant instead</h3>



<p class="wp-block-paragraph">A bucket or an account per tenant is a genuinely harder boundary and I wouldn&#8217;t dismiss it. With a few dozen large agency customers it&#8217;s defensible and makes offboarding trivially provable. It stops being defensible once you onboard self-serve landlords, because bucket and account counts run into service quotas and the operational surface grows with every signup. Pick on tenant count, not on which sounds more secure. And note that ABAC is only as trustworthy as the tenant claim: if the caller can influence the value reaching the tagging call, you&#8217;ve built an elegant no-op.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure three: the upload and serve path</h2>



<p class="wp-block-paragraph">Uploads here come from people you have no relationship with yet. Prospective tenants, guarantors, contractors. That&#8217;s an untrusted channel by definition, and it produces two separate problems.</p>



<h3 class="wp-block-heading">Constrain the upload itself</h3>



<p class="wp-block-paragraph">A presigned PUT URL lets the holder write whatever they like to that key, at whatever size. Presigned POST is the better tool for public-facing uploads: the signature covers a policy document, and the policy carries conditions the browser cannot alter.</p>



<pre class="wp-block-code"><code>post = s3.generate_presigned_post(
    Bucket="prop-docs",
    Key=f"tenants/{tenant_id}/inbox/{upload_id}",
    Fields={"Content-Type": "application/pdf"},
    Conditions=[
        {"Content-Type": "application/pdf"},
        ["content-length-range", 1, 15728640],
    ],
    ExpiresIn=600,
)</code></pre>



<p class="wp-block-paragraph">The <code>content-length-range</code> condition is the one people skip, and it&#8217;s the one that stops a stranger using your referencing portal as free storage for a 40 GB file. A declared <code>Content-Type</code> is a claim, not a fact, so land uploads in a quarantine prefix, verify the actual bytes, and only then move them to the tenant prefix.</p>



<p class="wp-block-paragraph">Scan on that path too. ClamAV in a container is the zero-cost option and covers known malware; GuardDuty Malware Protection for S3 and hosted APIs such as Cloudmersive trade money for not maintaining it. What you&#8217;re protecting is mostly the letting agent who opens the attachment on a Windows machine, so this is a control for your customers rather than your servers.</p>



<h3 class="wp-block-heading">Never serve tenant documents from your application origin</h3>



<p class="wp-block-paragraph">This is the one that turns a document store into an account takeover. SVG and HTML are formats browsers execute. If a file uploaded by a prospective tenant is served inline from the same origin as your app, any script inside it runs with your application&#8217;s cookies and session context. A long stream of real advisories follows exactly this shape, particularly in self-hosted deployments where the object store sits on the app&#8217;s own domain.</p>



<p class="wp-block-paragraph">Three controls, and they stack rather than substitute:</p>



<ol class="wp-block-list">
<li><strong>Force download.</strong> Serve with <code>Content-Disposition: attachment</code> and <code>X-Content-Type-Options: nosniff</code>. Set these as object metadata at upload time <em>and</em> override them per request. The override protects links you mint; the stored metadata protects anything that bypasses your minting path.</li>



<li><strong>Use a separate origin.</strong> Put user content on a distinct domain, not a subdomain of the app. GitHub&#8217;s separate apex for user content is the reference implementation. Same-origin policy then contains the blast radius if control one is missed somewhere.</li>



<li><strong>Normalise where you can.</strong> Rendering an uploaded ID to a flattened image server side removes the executable content entirely. It costs CPU and fidelity. For identity documents, where you rarely need the original bytes back, it&#8217;s the right trade.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure four: deleting the document doesn&#8217;t delete the document</h2>



<p class="wp-block-paragraph">A tenant asks you to erase their passport scan. Your app sets a flag, the file disappears from the UI, and you reply confirming deletion. Then count the copies that still exist.</p>



<ul class="wp-block-list">
<li>The previous object versions, if versioning is on. A delete without a version ID just writes a delete marker.</li>



<li>Cross-region replicas with their own versioning and lifecycle rules, plus any backup or snapshot taken before the deletion.</li>



<li>Derivatives you generated and forgot: thumbnails, page previews, OCR text in a search index. Run a passport through Amazon Textract and the extracted fields are a copy of the sensitive data in a different shape.</li>



<li>Copies that left the platform. A document attached to a CRM record in something like GoHighLevel, forwarded by email, or downloaded to a letting agent&#8217;s laptop.</li>
</ul>



<p class="wp-block-paragraph">The ICO&#8217;s position is that a valid erasure request extends to backups as well as live systems, and that where backup data cannot be immediately overwritten it must be put beyond use: isolated, unused for anything else, deleted when that backup cycles. Workable, but only if you know which backup sets contain documents and how long they cycle. If you can&#8217;t answer that, you can&#8217;t answer the request.</p>



<h3 class="wp-block-heading">Crypto-shredding, and what it actually costs</h3>



<p class="wp-block-paragraph">The one technique that makes deletion provable across copies you can&#8217;t reach is a per-tenant KMS key. Encrypt that tenant&#8217;s objects under their own customer managed key; when they offboard, schedule the key for deletion. Every ciphertext copy anywhere, backups included, becomes undecryptable. You delete the data by deleting the ability to read it. The honest costs:</p>



<ul class="wp-block-list">
<li>KMS bills per key per month plus per request, so per-tenant keys are a real line item. Check current rates and quotas first: there&#8217;s an account limit on customer managed keys that a self-serve signup flow will eventually meet.</li>



<li>Key deletion has a mandatory waiting period. Crypto-shredding is not instant, and your erasure response has to say so.</li>



<li>It&#8217;s all-or-nothing per key. If a tenant asks you to erase one document rather than offboard, the key doesn&#8217;t help.</li>
</ul>



<h3 class="wp-block-heading">Object Lock fights erasure, on purpose</h3>



<p class="wp-block-paragraph">S3 Object Lock is tempting for signed tenancy agreements, because write-once storage is a strong integrity story. But compliance mode means nobody, root included, can delete the object until retention lapses. That is the feature. It is also a direct collision with an erasure request, and you cannot engineer your way out of it afterwards.</p>



<p class="wp-block-paragraph">My line: Object Lock on executed contracts, where a statutory retention basis overrides erasure, and never on identity documents or referencing material. Different buckets, different rules, decided before anything is written. For per-document erasure elsewhere, lifecycle rules that expire noncurrent versions on the sensitive prefixes are worth setting up before you need them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure five: storing what you never needed to store</h2>



<p class="wp-block-paragraph">The cheapest control on this list is not storing the file. The ICO is clear that UK GDPR sets no fixed retention periods and puts the burden on you to justify how long you keep personal data in identifiable form. That&#8217;s a design instruction, not a legal footnote.</p>



<p class="wp-block-paragraph">For identity documents, ask what your platform needs once the check is done. Usually it&#8217;s the result, not the evidence: document type, issuing country, expiry date, a check reference, who performed it and when. If your customer&#8217;s regulator requires the image itself, that&#8217;s a retention rule with a stated basis, which is a different conversation from keeping it because deleting felt risky.</p>



<p class="wp-block-paragraph">Practical version: give every document class a retention clock wired to a business event rather than upload date, and let lifecycle rules do the deleting. Tenancy end plus a defined period. Check completion plus a defined period. Not &#8220;whenever someone remembers.&#8221;</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the breakages you&#8217;ll actually hit</h2>



<h3 class="wp-block-heading">Everything broke after someone enabled S3 Bucket Keys</h3>



<p class="wp-block-paragraph">This one catches teams who did the sophisticated thing. If your IAM or KMS key policies use the object ARN as the encryption context to scope access, enabling an S3 Bucket Key changes that context to the <em>bucket</em> ARN and those policies stop matching. AWS documents it explicitly as something to fix before flipping the setting.</p>



<p class="wp-block-paragraph">The quieter side effect: KMS CloudTrail events then log the bucket ARN instead of the object ARN, and there are far fewer of them because S3 caches the bucket-level key. If you were treating KMS CloudTrail volume as a proxy for document access, that signal degrades without anything appearing to break. Bucket Keys are still worth enabling for the request cost reduction. Move the conditions first, then re-check what your dashboards count.</p>



<h3 class="wp-block-heading">The presigned URL expires far sooner than requested</h3>



<p class="wp-block-paragraph">Almost always the signing credential is a role session shorter than the expiry you asked for. This is documented behaviour, not a bug. Either raise the session duration within the role&#8217;s maximum, or accept the shorter window as the security property it is.</p>



<h3 class="wp-block-heading">Listing works, reading returns Access Denied</h3>



<p class="wp-block-paragraph">Classic split between the bucket-level and object-level statements. Check both exist and reference the same tag key spelling. Session tag keys are case sensitive, and a <code>tenantId</code> in the code against a <code>TenantID</code> in the policy fails silently as a non-match.</p>



<h3 class="wp-block-heading">Access denied only for encrypted objects</h3>



<p class="wp-block-paragraph">An SSE-KMS object needs two grants, not one. The S3 action on the object, and <code>kms:Decrypt</code> on the key it was encrypted under. In a per-tenant key setup that second grant is usually also tag-conditioned, so a missing or mismatched tag on the key itself produces a denial that looks like an S3 problem and isn&#8217;t.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Signing document URLs with a long-lived IAM user access key, which makes every link independently long-lived.</li>



<li>Granting the tenant role a broad bucket-wide read statement alongside the tag-scoped one, usually to unblock a one-off script that then stayed.</li>



<li>Serving user uploads from a subdomain of the application and assuming that counts as origin separation. It doesn&#8217;t for cookie scope.</li>



<li>Enabling versioning for safety and never adding a noncurrent version expiry rule, so every deletion is reversible forever.</li>



<li>Building deletion that handles the object but not the OCR output, the thumbnail, or the search index entry.</li>



<li>Putting Object Lock on the bucket that also holds identity documents.</li>
</ul>



<h2 class="wp-block-heading">Design rules worth holding to</h2>



<ol class="wp-block-list">
<li><strong>Separate buckets by risk class,</strong> not by feature. Identity, contracts, financial, operational. Different encryption, retention, logging and lock settings follow naturally from that split.</li>



<li><strong>Enforce isolation in IAM,</strong> with application checks as defence in depth rather than the boundary itself.</li>



<li><strong>Sign short, sign from a session.</strong> Minutes, not days, and always from assumed-role credentials so revocation actually revokes.</li>



<li><strong>Deny unencrypted writes at the bucket policy,</strong> using the server-side encryption request headers as conditions, so a misconfigured client fails loudly instead of writing plaintext.</li>



<li><strong>Give every document class an expiry clock</strong> tied to a business event, implemented in lifecycle rules rather than a cron job someone maintains.</li>



<li><strong>Write the deletion runbook before launch,</strong> listing every place a copy lands. If you can&#8217;t enumerate them, that&#8217;s the finding.</li>



<li><strong>Test cross-tenant access in CI.</strong> A negative test that asserts tenant A&#8217;s session cannot read tenant B&#8217;s key is worth more than any amount of policy review.</li>
</ol>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How long should a presigned URL for a tenant ID document last?</h3>



<p class="wp-block-paragraph">Short enough that the user clicks it immediately. A couple of minutes covers a click-to-download flow. If the UI needs longer, re-mint on demand rather than extend. And remember the ceiling is set by the signing credential, so a short assumed-role session enforces the limit even if a later code change passes a bigger number.</p>



<h3 class="wp-block-heading">Do I need AWS, or will Backblaze B2 or Cloudflare R2 do?</h3>



<p class="wp-block-paragraph">They all speak the S3 API and presigned URLs work on all of them, so the core patterns transfer. Wasabi, Backblaze B2, Cloudflare R2 and DigitalOcean Spaces are reasonable for the operational tier, and R2&#8217;s egress model is attractive if agents download a lot. They differ in the depth of the IAM story: session-tag ABAC and per-tenant KMS keys are AWS-specific. If isolation and crypto-shredding are hard requirements, keep the sensitive prefixes on S3 even when the bulk photos live somewhere cheaper.</p>



<h3 class="wp-block-heading">Can I store scanned passports on a normal VPS instead?</h3>



<p class="wp-block-paragraph">You can, and plenty of platforms run the app tier on a VPS from Contabo or InterServer while keeping documents in object storage. What you lose on local disk is the deletion story and the access log. Proving an image is gone from a filesystem, its backups and its snapshots is far harder than scheduling a KMS key for deletion. If documents stay local, budget real effort for secure disposal, including on staff machines where tools like O&amp;O SafeErase have a place.</p>



<h3 class="wp-block-heading">What about e-signed tenancy agreements?</h3>



<p class="wp-block-paragraph">With a provider like DocuSign, the executed document and its audit trail live there as well as in your platform. That&#8217;s a second copy under a second retention regime, and it belongs in your deletion runbook and data map. It also strengthens your integrity story, which is a fair reason to skip Object Lock on contracts you retrieve from the provider.</p>



<h3 class="wp-block-heading">Should the CDN cache tenant documents?</h3>



<p class="wp-block-paragraph">Not the sensitive classes. A CDN in front of private documents adds a cache that can serve content after your access decision changed, and header handling at the edge is a common source of the inline-rendering bug. CloudFront with origin access control and signed URLs is legitimate if you need the performance, but for identity documents I&#8217;d go straight to the bucket. For inspection photos, cache away.</p>



<h3 class="wp-block-heading">How do I prove any of this to a prospect&#8217;s security team?</h3>



<p class="wp-block-paragraph">Artefacts, not assertions. The bucket policy, the ABAC policy, CI output showing cross-tenant denial, a sample of storage-layer access logs, and a retention schedule with a clock per document class. That set answers most questionnaires without a call.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Good property SaaS document storage is mostly about closing the gap between when you check access and when access happens. Every failure family above is a version of that gap: the presigned link that outlives the permission, the isolation enforced once in code and never at the storage layer, the deletion that updates a flag while the bytes sit behind a delete marker.</p>



<p class="wp-block-paragraph">Close it by pushing the decision down: into IAM policies that evaluate on every request, into credentials short enough that revocation is real, into lifecycle rules that delete without anyone remembering to. Then the answer to &#8220;show us who read this tenant&#8217;s ID document&#8221; is a query, not a project.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your document storage?</h2>



<p class="wp-block-paragraph">Most of what I do on property and lettings platforms is this exact work. Things I&#8217;d help with here:</p>



<ul class="wp-block-list">
<li>Reviewing bucket and IAM policies for cross-tenant read paths, including the ones added to unblock a migration and never removed.</li>



<li>Moving tenant isolation from application code into session-tag ABAC, with a CI test that fails the build on cross-tenant access.</li>



<li>Rebuilding the upload path with presigned POST conditions, quarantine, content verification and scanning before anything lands in a tenant prefix.</li>



<li>Fixing the serving path so uploaded files can&#8217;t execute against your origin, including header handling through a CDN.</li>



<li>Designing retention clocks and lifecycle rules per document class, plus a deletion runbook that names every copy including derivatives and third-party systems.</li>



<li>Setting up storage-layer access logging scoped to the sensitive prefixes so you can actually answer an access request.</li>
</ul>



<p class="wp-block-paragraph">If it&#8217;s useful, send me your bucket policy and the IAM policy attached to your document role, with the account IDs redacted, and I&#8217;ll tell you what a determined tenant could reach.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/property-saas-document-storage/">Property SaaS Document Storage: The Five Failures That Show Up in an Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Tenant Isolation on AWS: Building a Multi-Tenant Workshop Platform That Doesn&#8217;t Leak</title>
		<link>https://john-nessime.com/blog/cloud-computing/tenant-isolation-aws-multi-tenant-saas/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 31 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[ABAC]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS STS]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[Cost Allocation Tags]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[Noisy Neighbor]]></category>
		<category><![CDATA[Pool Model]]></category>
		<category><![CDATA[Row-Level Security]]></category>
		<category><![CDATA[SaaS Architecture]]></category>
		<category><![CDATA[Session Policies]]></category>
		<category><![CDATA[Silo Model]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Tenant Onboarding]]></category>
		<category><![CDATA[Token Vending Machine]]></category>
		<category><![CDATA[Workshop Management]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=538</guid>

					<description><![CDATA[<p>A missing tenant filter doesn't throw an error, it returns a 200 with too many rows. This is how to build a multi-tenant workshop management platform on AWS where the isolation boundary sits below your application code: STS session tags feeding IAM conditions, DynamoDB leading keys, scoped S3 prefixes, forced PostgreSQL row-level security, and a control plane that verifies each new tenant is fenced before anyone logs in.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/tenant-isolation-aws-multi-tenant-saas/">Tenant Isolation on AWS: Building a Multi-Tenant Workshop Platform That Doesn&#8217;t Leak</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A ticket comes in from one of the workshops on the platform. A service advisor pulled the weekly job report and there&#8217;s a vehicle on it that never came through their door. Wrong registration, wrong customer, wrong shop.</p>



<p class="wp-block-paragraph">By the time that ticket lands, the leak already happened. You can patch the query in an hour. What you cannot do is tell the affected garage how many other reports were wrong, how long it had been wrong, or whether anyone downloaded a CSV. There&#8217;s no log that answers those questions, because nothing ever denied anything. The database happily returned the rows. The API happily serialised them.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of the problem. Tenant isolation on AWS is not really about writing careful code. It&#8217;s about arranging things so that careless code fails loudly instead of quietly returning somebody else&#8217;s data.</p>



<p class="wp-block-paragraph">This post walks through building a multi-tenant workshop management platform on AWS with isolation baked in from the first commit: where tenant context comes from, how the boundary changes shape per storage service, what the control plane has to own, and how you prove any of it works. The example is a shop management system, with job cards, vehicle histories, parts inventory and technician timesheets, but the patterns apply to any vertical SaaS product where one customer&#8217;s records must never touch another&#8217;s.</p>



<h2 class="wp-block-heading">The failure mode that stays invisible</h2>



<p class="wp-block-paragraph">Most multi-tenant systems start with a <code>tenant_id</code> column and a convention: every query filters on it. That works right up until it doesn&#8217;t.</p>



<p class="wp-block-paragraph">The convention breaks in ordinary ways. Someone adds a reporting endpoint and copies a query from a script that ran as an admin. A new join pulls in a table nobody remembered to filter. An ORM lazy-loads a relationship and the filter lives on the parent, not the child. A background job that recalculates parts margins runs without any tenant in scope at all, because it processes everything.</p>



<p class="wp-block-paragraph">None of these throw. That&#8217;s the whole issue. A missing authorisation check produces a 403 you&#8217;ll notice in staging. A missing tenant filter produces a 200 with too many rows, and 200s don&#8217;t page anyone.</p>



<p class="wp-block-paragraph">The fix is not more discipline. It&#8217;s moving the filter somewhere the application cannot forget it: into IAM, into the database engine, or both. AWS makes this point directly in its own SaaS guidance, and it&#8217;s the right one. If your only defence against cross-tenant reads is that developers remember, you don&#8217;t have a defence, you have a habit.</p>



<h2 class="wp-block-heading">Pick the isolation model before you write code</h2>



<p class="wp-block-paragraph">There are three shapes, and they&#8217;re usually described as pool, silo and bridge.</p>



<ul class="wp-block-list">
<li><strong>Pool.</strong> Every tenant shares the same tables, buckets and compute. Cheapest to run, cheapest to deploy, and the model where isolation has to be enforced explicitly because nothing physical separates anyone.</li>



<li><strong>Silo.</strong> Each tenant gets dedicated resources: its own database, its own bucket, sometimes its own account. Isolation is close to free, operations are not. Migrations, deploys and monitoring all multiply by tenant count.</li>



<li><strong>Bridge.</strong> Mixed. Shared compute, dedicated storage, or pooled for the standard tier and siloed for the enterprise tier that asked hard questions in procurement.</li>
</ul>



<p class="wp-block-paragraph">For a workshop platform, most independent garages will be small, and pooling is the only sane starting point. The trap is treating that as permanent. Sooner or later a dealer group with forty sites will ask for a dedicated database, and if you have not left room for a per-tenant routing decision, retrofitting it means rewriting your data access layer.</p>



<p class="wp-block-paragraph">What I&#8217;d actually do: build pooled, but put the storage target behind a resolver from day one. A function that takes a tenant ID and returns a connection, a table name or a bucket prefix. When the first silo tenant arrives, you change the resolver, not four hundred call sites.</p>



<h2 class="wp-block-heading">Where tenant context comes from</h2>



<p class="wp-block-paragraph">This is the part people get subtly wrong, and it undermines everything downstream. The tenant identifier must come from the authenticated identity, never from the request. Not a header, not a query parameter, not a field in the JSON body. If a client can influence the tenant ID, your isolation model is decoration.</p>



<p class="wp-block-paragraph">In practice that means a custom claim in the token your identity provider issues. Amazon Cognito can carry a custom attribute for this, and so can any external IdP you federate with. The API layer reads the claim, and from that point the tenant is a fact about the caller rather than an input to the call.</p>



<p class="wp-block-paragraph">Then you push that fact down into AWS itself using a session tag. When your service assumes a role, it attaches the tenant as a tag on the session. Every subsequent AWS API call made with those credentials carries the tag in the request context, where IAM policies can reference it as <code>aws:PrincipalTag</code>.</p>



<pre class="wp-block-code"><code>import boto3

def scoped_session(tenant_id, role_arn):
    sts = boto3.client("sts")
    resp = sts.assume_role(
        RoleArn=role_arn,
        RoleSessionName=f"workshop-{tenant_id}",
        Tags=[{"Key": "TenantID", "Value": tenant_id}],
    )
    c = resp["Credentials"]
    return boto3.Session(
        aws_access_key_id=c["AccessKeyId"],
        aws_secret_access_key=c["SecretAccessKey"],
        aws_session_token=c["SessionToken"],
    )</code></pre>



<p class="wp-block-paragraph">Two things make this work, and both are easy to miss. The role&#8217;s trust policy has to allow <code>sts:TagSession</code> alongside the assume-role action, or the call fails. And the calling principal needs permission to pass that tag. Get either wrong and you&#8217;ll spend an afternoon reading an error that sounds like it&#8217;s about the role rather than the tag.</p>



<p class="wp-block-paragraph">One session tag, one role, one policy. That&#8217;s attribute-based access control, and it&#8217;s the reason ABAC scales where a role per tenant does not. Roles are a finite resource in an AWS account. Tags are not.</p>



<h2 class="wp-block-heading">Tenant isolation on AWS changes shape per service</h2>



<p class="wp-block-paragraph">There is no single isolation control. Each service exposes a different lever, and you have to learn every one your architecture touches.</p>



<h3 class="wp-block-heading">DynamoDB: the partition key is the boundary</h3>



<p class="wp-block-paragraph">If job cards live in DynamoDB with the tenant ID as the partition key, IAM can pin every read and write to that key. The condition key is <code>dynamodb:LeadingKeys</code>.</p>



<pre class="wp-block-code"><code>{
  "Effect": "Allow",
  "Action": [
    "dynamodb:GetItem",
    "dynamodb:PutItem",
    "dynamodb:UpdateItem",
    "dynamodb:Query"
  ],
  "Resource": "arn:aws:dynamodb:REGION:ACCOUNT:table/JobCards",
  "Condition": {
    "ForAllValues:StringEquals": {
      "dynamodb:LeadingKeys": ["${aws:PrincipalTag/TenantID}"]
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Now a Query that omits the tenant partition key doesn&#8217;t return other tenants&#8217; job cards. It gets denied. That&#8217;s the behaviour you want: loud, logged, and impossible to miss in CloudTrail.</p>



<p class="wp-block-paragraph">Two caveats worth knowing before you commit. Scan operations don&#8217;t have a leading key to constrain, so granting <code>dynamodb:Scan</code> alongside this condition undermines the whole arrangement. And global secondary indexes have their own key structure, so an index whose partition key isn&#8217;t the tenant needs separate thought. Design the access patterns so that no query ever needs to look across tenants, and this stops being a problem.</p>



<h3 class="wp-block-heading">S3: two surfaces, not one</h3>



<p class="wp-block-paragraph">Vehicle photos, inspection PDFs and signed job sheets go to S3 under a per-tenant prefix. The mistake is scoping only the object actions and leaving <code>ListBucket</code> open, which lets a tenant enumerate every other garage&#8217;s filenames even without reading them. Filenames leak plenty: customer names, registration plates, invoice numbers.</p>



<p class="wp-block-paragraph">Object actions are scoped through the resource ARN. Listing is scoped through the <code>s3:prefix</code> request condition. You need both statements, because they protect different operations.</p>



<pre class="wp-block-code"><code>[
  {
    "Effect": "Allow",
    "Action": "s3:ListBucket",
    "Resource": "arn:aws:s3:::workshop-tenant-files",
    "Condition": {
      "StringLike": {
        "s3:prefix": ["${aws:PrincipalTag/TenantID}/*"]
      }
    }
  },
  {
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:PutObject"],
    "Resource": "arn:aws:s3:::workshop-tenant-files/${aws:PrincipalTag/TenantID}/*"
  }
]</code></pre>



<p class="wp-block-paragraph">The policy variable in the resource ARN is doing real work there. One policy, every tenant, no template rendering at request time.</p>



<h3 class="wp-block-heading">Aurora PostgreSQL: row-level security, and the trap in it</h3>



<p class="wp-block-paragraph">Relational data is where most workshop platforms actually live, because job cards, parts lines and labour rates are relational. IAM cannot see inside a table, so the boundary moves into PostgreSQL itself via row-level security.</p>



<pre class="wp-block-code"><code>ALTER TABLE job_cards ENABLE ROW LEVEL SECURITY;
ALTER TABLE job_cards FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON job_cards
  USING (tenant_id = current_setting('app.tenant_id', true))
  WITH CHECK (tenant_id = current_setting('app.tenant_id', true));</code></pre>



<p class="wp-block-paragraph"><code>USING</code> controls which rows are visible to reads, updates and deletes. <code>WITH CHECK</code> controls what can be written. Without the second clause, a tenant can read only its own rows but insert a row stamped with someone else&#8217;s tenant ID. The first protects the read path, the second protects the write path, and you want both.</p>



<p class="wp-block-paragraph">Now the trap, and it&#8217;s the one that gives teams false confidence. PostgreSQL superusers and roles carrying <code>BYPASSRLS</code> ignore row security entirely, and by default so does the table owner. If your application connects as the same role that ran the migrations, your policies are not in effect and your tests pass anyway. That&#8217;s why <code>FORCE ROW LEVEL SECURITY</code> is in the snippet above, and why the application should connect as a dedicated non-owner, non-superuser role.</p>



<p class="wp-block-paragraph">The second trap is connection reuse. Set the tenant with <code>SET LOCAL</code> inside a transaction, or with <code>set_config</code> using the transaction-local flag. Plain <code>SET</code> persists for the life of the connection, and a pooled connection outlives the request. That&#8217;s how one garage&#8217;s context ends up serving the next garage&#8217;s query.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Test row-level security as the application role, against a pooled connection, with at least two tenants in the table. Testing as the owner tells you nothing.</p>
</blockquote>



<h2 class="wp-block-heading">Compute isolation is a different question</h2>



<p class="wp-block-paragraph">Data isolation stops one tenant reading another&#8217;s records. It does nothing about one tenant consuming everyone&#8217;s capacity.</p>



<p class="wp-block-paragraph">Workshop platforms have a specific version of this. End of month, every garage runs its invoicing and MOT reminder batch at roughly the same time. A dealer group importing three years of service history will happily saturate a shared worker pool while forty independents wait for their job cards to load.</p>



<p class="wp-block-paragraph">Levers worth knowing, roughly in order of how much they cost you:</p>



<ul class="wp-block-list">
<li>API Gateway usage plans, keyed per tenant, to cap request rates at the edge before anything expensive runs.</li>



<li>Separate queues, or at minimum separate consumer concurrency, for bulk imports versus interactive requests. Bulk work should never share a lane with a screen someone is waiting on.</li>



<li>Reserved or provisioned concurrency on the Lambda functions serving interactive paths, so a batch surge cannot starve them.</li>



<li>Dedicated compute for premium tenants. This is silo by another name, and it&#8217;s the honest answer when a customer&#8217;s load profile genuinely doesn&#8217;t fit the pool.</li>
</ul>



<p class="wp-block-paragraph">Be honest with yourself about which problem you&#8217;re solving. Throttling is not isolation. It limits blast radius, it doesn&#8217;t create a boundary.</p>



<h2 class="wp-block-heading">What the control plane has to own</h2>



<p class="wp-block-paragraph">Separate the control plane from the application plane early. The control plane manages tenants; the application plane serves them. Mixing the two is how a bug in the onboarding flow ends up with production credentials.</p>



<p class="wp-block-paragraph">Onboarding a new workshop is a sequence, and it should be a single idempotent workflow rather than a checklist someone follows:</p>



<ol class="wp-block-list">
<li>Generate a non-guessable tenant identifier. Lowercase alphanumeric, no customer name in it, because it will end up inside resource ARNs and key prefixes.</li>



<li>Write the tenant record: tier, isolation model, storage target, status.</li>



<li>Provision identity. User pool group or IdP mapping, with the tenant claim wired in.</li>



<li>Provision storage. For a pooled tenant that&#8217;s a prefix and a seeded row. For a siloed one it&#8217;s real infrastructure, which is why this step must be asynchronous.</li>



<li>Apply tags used for cost allocation and reporting.</li>



<li>Run a verification step that proves the new tenant can reach its own data and cannot reach a canary tenant&#8217;s data.</li>
</ol>



<p class="wp-block-paragraph">Step six is the one teams skip. It&#8217;s also the only step that tells you the previous five worked.</p>



<p class="wp-block-paragraph">On per-tenant cost: activated cost allocation tags attribute anything that is a distinct tagged resource, which covers siloed tenants nicely. Pooled resources will not split by themselves, because a shared table doesn&#8217;t know which garage caused which read. If you need per-tenant margin, emit consumption as a metric dimension from the application: request counts, storage bytes, document pages processed. Tools like Vantage or CloudZero can allocate shared spend afterwards, but only from the signal you produce. Nothing recovers attribution you never recorded.</p>



<p class="wp-block-paragraph">The AWS SaaS Builder Toolkit is worth a look here. It codifies control plane concepts as CDK constructs and will save you real time on onboarding plumbing. Read its own guidance first: the project describes itself as sample code, and expects you to review security fit before production. That&#8217;s a fair description rather than a warning label, but treat it as a starting point, not a finished platform.</p>



<h2 class="wp-block-heading">Proving isolation actually holds</h2>



<p class="wp-block-paragraph">An isolation model you haven&#8217;t tried to break is a design document, not a control.</p>



<p class="wp-block-paragraph">The tests that earn their keep are negative ones. Seed two tenants. Authenticate as the first. Then deliberately do the wrong thing: request the second tenant&#8217;s job card by ID, list the second tenant&#8217;s S3 prefix, run a query with the session variable set to the wrong value. Every one of those should fail, and the test should assert on the failure.</p>



<ul class="wp-block-list">
<li>Run the cross-tenant suite on every pull request, not nightly. It&#8217;s the regression that matters most and the one most likely to be introduced by an innocent refactor.</li>



<li>Use the IAM policy simulator to check a policy change before it ships, particularly when someone widens a resource ARN.</li>



<li>Turn on IAM Access Analyzer so external access grants surface without anyone having to notice them.</li>



<li>Audit the database on a schedule: tables with RLS enabled but no policy attached, tables missing FORCE, application roles that have quietly acquired ownership or BYPASSRLS during an incident.</li>



<li>Alert on AccessDenied volume per tenant. A spike is either a bug you introduced or someone probing, and both are worth knowing about.</li>
</ul>



<p class="wp-block-paragraph">Keep a permanent canary tenant in every environment, including production, holding nothing but synthetic data. Every negative test targets it. It costs almost nothing and it means your isolation tests never need real customer records.</p>



<h2 class="wp-block-heading">Troubleshooting the failures you&#8217;ll actually hit</h2>



<h3 class="wp-block-heading">AccessDenied on a policy that looks correct</h3>



<p class="wp-block-paragraph">Almost always the session tag isn&#8217;t present. If the tag is missing from the request context, the condition can&#8217;t match and the statement doesn&#8217;t apply. Call <code>sts:GetCallerIdentity</code> with the scoped credentials and confirm you&#8217;re on the assumed role you think you are, then check CloudTrail for the AssumeRole event and look at whether the tag was actually passed. Tag keys are case sensitive, and <code>TenantId</code> is not <code>TenantID</code>.</p>



<h3 class="wp-block-heading">Queries return zero rows instead of the right rows</h3>



<p class="wp-block-paragraph">Classic RLS symptom. The session variable is unset, so the policy predicate compares against null and nothing matches. Check <code>current_setting</code> inside the same transaction as the query, not in a separate connection from your SQL client. Empty results are the safe failure here, which is exactly why they&#8217;re easy to misread as a data problem.</p>



<h3 class="wp-block-heading">One tenant intermittently sees another&#8217;s data</h3>



<p class="wp-block-paragraph">Intermittent means state reuse. Look at connection pooling first, then at any per-request context stored in thread-local or async-local storage that isn&#8217;t reset when the request finishes. If you&#8217;re using a proxy in front of the database, understand how it handles session state, because some proxies pin a connection to a client once session-level settings are detected, which changes the behaviour you tested against.</p>



<h3 class="wp-block-heading">Isolation works in the API but not in background jobs</h3>



<p class="wp-block-paragraph">Because the job has no request, so it has no token, so it has no tenant. Whatever runs asynchronously needs the tenant carried on the message and a session assumed per tenant when the work is processed. A worker that loops over all tenants with admin credentials is the single most common place isolation quietly stops applying.</p>



<h3 class="wp-block-heading">Session tags don&#8217;t survive a second AssumeRole</h3>



<p class="wp-block-paragraph">Session tags are not automatically carried forward when you chain roles unless they were marked transitive. If your architecture hops through more than one role, this is where the tenant context evaporates. Flatten the chain if you can; if you can&#8217;t, mark the tag transitive deliberately and document why.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Taking the tenant ID from a request header or body instead of the authenticated token.</li>



<li>Using the customer&#8217;s name or a sequential integer as the tenant identifier, then putting it in bucket prefixes where it becomes both guessable and enumerable.</li>



<li>Scoping S3 object actions but leaving bucket listing wide open.</li>



<li>Running the application as the PostgreSQL table owner, so RLS is enabled and silently inert.</li>



<li>Writing a USING clause with no WITH CHECK, leaving the write path open.</li>



<li>Creating one IAM role per tenant and discovering the account ceiling somewhere around the point the business gets interesting.</li>



<li>Assuming cost allocation tags will attribute pooled spend. They won&#8217;t, and by the time you need the numbers the history is gone.</li>



<li>Testing isolation only through the UI, where the frontend is already sending the right tenant every time.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Derive tenant context from the token, propagate it as a session tag, and never let application code choose it.</li>



<li>Enforce the boundary at the layer below your code: IAM conditions for AWS resources, RLS for relational rows.</li>



<li>Put storage targets behind a resolver so moving a tenant from pool to silo is a configuration change.</li>



<li>Make onboarding one idempotent workflow that ends in a verification step.</li>



<li>Emit tenant as a dimension on logs and metrics from the start, so cost and performance questions stay answerable.</li>



<li>Keep a canary tenant and run cross-tenant negative tests in CI on every change.</li>



<li>Put a WAF or edge layer such as Cloudflare in front of tenant subdomains, and keep tenant routing decisions out of the origin application where you can.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a shared database with a tenant_id column ever acceptable?</h3>



<p class="wp-block-paragraph">Yes, provided the column is enforced by the engine rather than by convention. A shared table with row-level security, forced, queried by a non-owner role, is a real boundary. A shared table where every query is expected to include the filter is not. The schema is the same; the guarantee is completely different.</p>



<h3 class="wp-block-heading">ABAC or dynamically generated IAM policies?</h3>



<p class="wp-block-paragraph">ABAC for most cases. One role, one policy, tenant supplied per session, and nothing grows as you add customers. Dynamic policy generation, sometimes called a token vending machine, earns its place when a single policy genuinely cannot express the rule, such as when per-tenant resource names have to be injected rather than a key prefix. The cost is that you now own the correctness of a policy generator, plus per-request latency, and session policies have a size ceiling you can hit.</p>



<h3 class="wp-block-heading">Should each tenant get its own AWS account?</h3>



<p class="wp-block-paragraph">It&#8217;s the strongest boundary available and the most expensive to operate. For a workshop platform serving independent garages it&#8217;s overkill. It becomes reasonable when a customer&#8217;s contract, regulator or data residency requirement makes shared infrastructure a non-starter, and at that point you&#8217;re pricing it as a premium tier rather than absorbing it.</p>



<h3 class="wp-block-heading">How do I handle a user who works at two workshops?</h3>



<p class="wp-block-paragraph">Model it as one identity with multiple tenant memberships and an explicit active tenant per session, rather than a token carrying a list. The active tenant becomes the session tag. Switching workshops means a new session, which is exactly the behaviour you want because it makes the switch visible in your audit trail.</p>



<h3 class="wp-block-heading">Does row-level security hurt query performance?</h3>



<p class="wp-block-paragraph">It adds a predicate the planner has to satisfy, so the answer depends on your indexes. Index the tenant column, and index it as the leading column of composite indexes that support your common filters. Compare plans as the application role before and after enabling policies, because a plan captured as the owner may not reflect what the application actually runs.</p>



<h3 class="wp-block-heading">Where does application-level authorisation fit?</h3>



<p class="wp-block-paragraph">Alongside, not instead. Tenant isolation answers &#8220;which organisation&#8217;s data is this&#8221;. Authorisation answers &#8220;may this technician void an invoice&#8221;. Different questions, different layers. A policy engine such as Amazon Verified Permissions handles the second cleanly, and keeping them separate stops role logic creeping into your isolation boundary.</p>



<h3 class="wp-block-heading">Can I retrofit isolation onto a platform that already has tenants?</h3>



<p class="wp-block-paragraph">You can, and it&#8217;s tedious rather than impossible. Enforce at the database first, because that&#8217;s where the leak actually happens: table by table, FORCE enabled, application moved to a non-owner role. Then move the tenant ID out of request payloads and into the token. Add IAM conditions last, since they&#8217;re the least likely source of a live leak. Expect to find at least one background job with no tenant scope at all.</p>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Tenant isolation on AWS works when the boundary sits below your application code, in a layer that denies rather than trusts. IAM conditions on session tags for AWS resources, forced row-level security for relational data, and a control plane that verifies a new tenant is properly fenced before anyone logs in.</p>



<p class="wp-block-paragraph">Retrofitting that onto a running multi-tenant platform is possible but grim, because you&#8217;re doing it while real workshops have real data in the system. Doing it on day one costs a week. That&#8217;s the entire trade, and it&#8217;s not a close call.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your multi-tenant architecture?</h2>



<p class="wp-block-paragraph">Most of the isolation problems I see are not exotic. They&#8217;re a missing WITH CHECK clause, an application connecting as the table owner, or a background job nobody scoped. Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing an existing multi-tenant design and finding where the boundary is enforced by convention rather than by the platform.</li>



<li>Implementing ABAC with STS session tags across DynamoDB, S3 and Aurora, including the trust policy wiring that trips people up.</li>



<li>Setting up PostgreSQL row-level security correctly, with forced policies, a dedicated application role and pooling that doesn&#8217;t leak session state.</li>



<li>Building a tenant onboarding workflow that provisions identity, storage and tagging idempotently and verifies itself.</li>



<li>Writing the cross-tenant negative test suite and wiring it into CI so isolation regressions fail the build.</li>



<li>Adding per-tenant usage metering so cost, performance and tier decisions rest on data instead of guesses.</li>
</ul>



<p class="wp-block-paragraph">Send me a policy document, an RLS definition or a CloudTrail AccessDenied event and I&#8217;ll tell you what it&#8217;s actually enforcing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/tenant-isolation-aws-multi-tenant-saas/">Tenant Isolation on AWS: Building a Multi-Tenant Workshop Platform That Doesn&#8217;t Leak</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</title>
		<link>https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Comprehend]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=272</guid>

					<description><![CDATA[<p>A confident answer with a citation from a matter the reader was walled off from. Nothing crashed, nothing alerted. Building legal document intelligence on AWS with Textract, Bedrock and OpenSearch is the easy half; the hard half is tenant isolation, verified deletion, S3 Object Lock holds, what leaves the account, and an audit trail that names the actual user.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo was going well until someone asked where the third citation came from.</p>



<p class="wp-block-paragraph">The answer on screen was good. Fluent, specific, correctly hedged, three sources listed underneath it. Then a partner in the room asked a simple question: which matter is source three from? Nobody could answer it in the room, and when we went and looked, it was from a matter that half the people watching the demo were formally walled off from.</p>



<p class="wp-block-paragraph">Nothing had crashed. No alarm fired. The retrieval layer had done exactly what a similarity search does, which is return the nearest vectors, and the nearest vectors did not care about the ethical wall. That is the thing about building a legal document intelligence platform on AWS: the failures that matter almost never look like failures. They look like a confident answer with a citation attached.</p>



<p class="wp-block-paragraph">This post is about the parts of that build that are hard, organised around the five boundaries that actually have to hold. The extraction pipeline is the easy half. I will cover it, but quickly, because the AWS documentation is good and the failure modes are visible. The other four boundaries fail quietly, and that is where the engineering goes.</p>



<h2 class="wp-block-heading">The shape of the thing</h2>



<p class="wp-block-paragraph">Before the boundaries, the skeleton. A legal document intelligence platform on AWS almost always ends up looking like this, whether you plan it or arrive at it:</p>



<ul class="wp-block-list">
<li>Documents land in Amazon S3, one prefix per matter, versioning on.</li>



<li>An S3 event triggers AWS Lambda, which starts an asynchronous Amazon Textract job.</li>



<li>Textract output lands back in S3 as JSON. Something normalises it into text plus page and bounding-box coordinates.</li>



<li>Amazon Comprehend or a foundation model classifies the document and pulls entities: parties, dates, governing law, clause types.</li>



<li>Chunks get embedded and written to a vector store, usually Amazon OpenSearch Serverless behind Amazon Bedrock Knowledge Bases.</li>



<li>An application layer, typically API Gateway plus Lambda, takes a question, retrieves, and calls a model on Amazon Bedrock.</li>
</ul>



<p class="wp-block-paragraph">AWS Step Functions is worth reaching for once you have more than three stages, because Textract jobs are long-running and Lambda timeouts are not a retry strategy. That is the whole architecture. You can stand it up in a fortnight. Then you spend six months on everything below.</p>



<h2 class="wp-block-heading">Boundary one: isolation, and why metadata filtering is not optional</h2>



<p class="wp-block-paragraph">This is the one from the demo, and it is the one that ends engagements.</p>



<p class="wp-block-paragraph">A vector index has no native concept of a matter, a client, or an ethical wall. If you pool every document into one index and rely on the prompt to keep things separate, you have built a system where a well-phrased question can pull privileged material across a wall. Prompt instructions are not an access control. They are a suggestion that usually works.</p>



<p class="wp-block-paragraph">There are two shapes that do work, and the choice between them is a real trade-off rather than a best practice.</p>



<h3 class="wp-block-heading">Pooled index with server-side filtering</h3>



<p class="wp-block-paragraph">Every chunk carries a <code>matter_id</code> and a <code>client_id</code> as metadata. Every retrieval call attaches a filter. With an S3 data source in Bedrock Knowledge Bases, the metadata comes from a sidecar JSON file that sits next to the document and carries a <code>metadataAttributes</code> object.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "matter_id": "M-4417",
    "client_id": "C-108",
    "doc_type": "engagement_letter",
    "privileged": true
  }
}</code></pre>



<p class="wp-block-paragraph">The retrieval call then narrows the search before the model ever sees a chunk:</p>



<pre class="wp-block-code"><code>"retrievalConfiguration": {
  "vectorSearchConfiguration": {
    "filter": {
      "equals": { "key": "matter_id", "value": "M-4417" }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two rules make this safe, and both are the kind of thing that gets skipped under deadline. First, the filter value is derived on the server from the authenticated caller&#8217;s claims, never accepted from the client. If the browser can send you a <code>matter_id</code>, the browser can send you a different one. Second, the filter is applied by code that no feature request can bypass. A helper that builds every retrieval request, and a code review rule that no other path may call the retrieve API directly.</p>



<p class="wp-block-paragraph">The trap here is ordering. You cannot filter on an attribute the index does not have. Add <code>matter_id</code> after ingestion and every chunk already in the index is invisible to that filter, which means it either returns for everyone or for no one, depending on how your filter is written. Design the metadata schema before the first ingestion run, not after the first demo.</p>



<h3 class="wp-block-heading">Separate collection per client</h3>



<p class="wp-block-paragraph">The heavier option. A separate OpenSearch Serverless collection per client, which buys you a separate AWS KMS key per client, separate index settings, and a failure mode where a bug in the filtering logic cannot reach across clients at all.</p>



<p class="wp-block-paragraph">It costs you sprawl. Every collection is a resource to provision, monitor, patch policy on, and eventually delete. Ingestion jobs run per data source per knowledge base, so freshness guarantees fragment. For a firm with twelve institutional clients this is fine. For a platform onboarding a hundred small clients it becomes the main operational burden of the product.</p>



<p class="wp-block-paragraph">My default is pooled with strict server-side filtering, and per-client collections only where the client&#8217;s own contract demands a dedicated encryption key. If you cannot say out loud which of those two you are running, you are running neither properly.</p>



<h2 class="wp-block-heading">Boundary two: extraction, and the confidence score everyone ignores</h2>



<p class="wp-block-paragraph">Textract returns a confidence score between 0 and 100 with each prediction. AWS is explicit in its own best-practice guidance that applications sensitive to detection errors should enforce a minimum threshold and route anything below it for human scrutiny, and that the right threshold depends entirely on how the output gets used.</p>



<p class="wp-block-paragraph">Legal documents sit at the harsh end of that. A misread date on an archival scan is a curiosity. A misread date on a limitation period is a problem with a name on it.</p>



<p class="wp-block-paragraph">Practical notes from this layer:</p>



<ul class="wp-block-list">
<li>Use the asynchronous operations for anything multipage. <code>StartDocumentAnalysis</code> and <code>GetDocumentAnalysis</code> handle long PDFs and TIFFs; the synchronous <code>AnalyzeDocument</code> path is for single pages and will fight you on a 400-page bundle.</li>



<li>Textract&#8217;s Queries feature earns its place on structured instruments. Instead of extracting everything and grepping, you ask the document a direct question and get a scoped answer with its own confidence.</li>



<li>Keep the page number and bounding box for every chunk. When a lawyer asks where a sentence came from, &#8220;page 14, second column&#8221; is an answer. &#8220;It&#8217;s in the corpus somewhere&#8221; is not, and the platform loses credibility the first time you say it.</li>



<li>Amazon Augmented AI wires low-confidence predictions into a human review queue with a private workforce, so the reviewers are your people rather than an anonymous pool. For privileged material that distinction is the entire point.</li>
</ul>



<p class="wp-block-paragraph">What I would skip early: building a custom entity recognition model. Comprehend supports custom entity recognition and it is genuinely useful, but you need labelled data you do not have yet, and a foundation model with a decent prompt gets you far enough to find out which entities the users actually care about. Train the custom model once the requirement has stopped moving.</p>



<h2 class="wp-block-heading">Boundary three: retention, where deletion is a distributed problem</h2>



<p class="wp-block-paragraph">Here is the failure that will not show up in any test suite you write.</p>



<p class="wp-block-paragraph">A client asks for a document to be removed. Someone deletes the S3 object. The ticket closes. The document is still fully searchable, because its chunks are sitting in the vector index and will stay there until the data source is re-synced. Meanwhile the Textract JSON output is in a second bucket, the extracted text may be in a database, and if model invocation logging is on, whole passages of the document are sitting in CloudWatch Logs or another S3 bucket entirely.</p>



<p class="wp-block-paragraph">One delete, at least four places holding a copy. Write the deletion path as a real workflow with an assertion at the end, not as a single API call:</p>



<ol class="wp-block-list">
<li>Delete the source object and, if versioning is on, its versions.</li>



<li>Delete the derived artefacts: Textract JSON, normalised text, any thumbnails or page images.</li>



<li>Trigger an ingestion job so the knowledge base drops the orphaned chunks.</li>



<li>Confirm removal by querying the index for the document identifier and expecting nothing back.</li>



<li>Deal with the logs, which means either a retention policy short enough to make the problem expire or a deliberate decision, written down, that logs are out of scope.</li>
</ol>



<p class="wp-block-paragraph">Step four is the one people leave out, and it is the only step that actually proves anything.</p>



<h3 class="wp-block-heading">The opposite problem: things that must not be deleted</h3>



<p class="wp-block-paragraph">Legal work has the reverse requirement too. When a matter goes into litigation hold, the documents need to survive an administrator with delete permissions and a bad afternoon.</p>



<p class="wp-block-paragraph">S3 Object Lock is the mechanism, and it has two independent controls that people routinely conflate. A retention period protects an object version until a fixed date, in either governance mode, which privileged users can override, or compliance mode, which nobody can override and where the period cannot be shortened. A legal hold has no date at all. It stays until someone with <code>s3:PutObjectLegalHold</code> explicitly removes it. The two can be active at once, and while either is active the object version cannot be deleted or overwritten.</p>



<pre class="wp-block-code"><code># Place an indefinite hold on a specific object version
aws s3api put-object-legal-hold 
  --bucket matters-archive 
  --key M-4417/exhibit-c.pdf 
  --version-id 3sL7f2Qz9pXvB1kR 
  --legal-hold Status=ON</code></pre>



<p class="wp-block-paragraph">Two things bite here. Object Lock requires versioning and turns it on automatically, and once enabled on a bucket you cannot turn it off or suspend versioning again. And compliance mode is genuinely permanent: if you set a seven-year retention when you meant seven days, that is the answer, for everyone, including the account root. Test in governance mode first. The bypass path exists and is deliberately awkward, requiring both the <code>s3:BypassGovernanceRetention</code> permission and an explicit <code>x-amz-bypass-governance-retention:true</code> header on the request, which is exactly the level of friction you want on that operation.</p>



<p class="wp-block-paragraph">Design the hold model before you design the deletion model, because holds win. A deletion request that collides with an active hold is a legal question, not an engineering one, and the platform&#8217;s job is to surface the collision clearly rather than resolve it silently.</p>



<h2 class="wp-block-heading">Boundary four: disclosure, or what leaves the account</h2>



<p class="wp-block-paragraph">This is the boundary the client&#8217;s general counsel will ask about, usually in writing, usually before signature. Three separate controls, and they are not interchangeable.</p>



<h3 class="wp-block-heading">The AI services opt-out policy</h3>



<p class="wp-block-paragraph">AWS Organizations has a governance policy type that opts your accounts out of having content processed by certain AI services stored and used for service improvement. The list includes Amazon Textract and Amazon Comprehend, which are precisely the two doing the reading in this architecture. The policy type has to be enabled at the organisation root before you can attach anything:</p>



<pre class="wp-block-code"><code>aws organizations enable-policy-type 
  --root-id r-example 
  --policy-type AISERVICES_OPT_OUT_POLICY</code></pre>



<p class="wp-block-paragraph">Read the AWS documentation on this one carefully rather than taking my summary as gospel, because there is a caveat in it that matters: the services may still need to store your data operationally even when you have opted out of it being used for improvement. Opting out is not the same as the data never existing outside your account. Say that plainly in the client conversation. It is a much better position than being asked about it later.</p>



<h3 class="wp-block-heading">Bedrock&#8217;s data position, and the log that undoes it</h3>



<p class="wp-block-paragraph">Amazon Bedrock&#8217;s published position is that inputs and outputs are not shared with third-party model providers and are not used to train the base models, and that fine-tuning operates on a private copy. That is a strong starting point for privileged content and it is the main reason Bedrock rather than a direct provider API shows up in these builds.</p>



<p class="wp-block-paragraph">Then there is model invocation logging. It captures full prompt and response payloads to CloudWatch Logs or S3, and you want it on, because without it you cannot reconstruct what the system told someone. But understand what you have just built: a log that contains privileged document text, at a per-region setting, in a destination that probably has looser access controls than the document store it came from. Encrypt it with the same KMS key discipline, restrict it harder than you think you need to, and set a retention period on purpose.</p>



<p class="wp-block-paragraph">Worth separating in your head: CloudTrail records that an API call happened and who made it. Model invocation logging records what was in it. You need both, for different questions, and only one of them contains client confidences.</p>



<h3 class="wp-block-heading">Network path</h3>



<p class="wp-block-paragraph">Interface VPC endpoints via AWS PrivateLink keep traffic to Bedrock, Textract and the rest on the AWS network rather than out through an internet gateway or NAT. Add a gateway endpoint for S3 while you are there.</p>



<p class="wp-block-paragraph">Endpoint policies are the part that gets forgotten. An endpoint without a policy is a private path to the whole service, including into accounts that are not yours. Scope it to the operations and resources you actually use, and pair it with an S3 bucket policy that rejects requests not arriving through your endpoint. The first protects what leaves; the second protects what can be reached.</p>



<h3 class="wp-block-heading">The parts that are not on the architecture diagram</h3>



<p class="wp-block-paragraph">Two of these have caught people out badly, and neither is an AWS control.</p>



<p class="wp-block-paragraph">The first is the development environment. You do not want real client documents on a laptop or in a scratch account, so build the parsing and chunking code against synthetic documents on a small separate box. A cheap VPS from a provider like Contabo or InterServer is fine for this, and the separation is worth more than the convenience of iterating in the production account.</p>



<p class="wp-block-paragraph">The second is the reviewers. If your human review loop involves contractors working remotely, their network path and their disks are part of your boundary whether or not you drew them. A managed VPN such as NordVPN or Surfshark handles the network side, and when a matter closes and local copies have to go, a dedicated erasure tool from something like O&amp;O Software does what dragging a folder to the bin does not. Unglamorous, and it is the layer that gets audited.</p>



<h2 class="wp-block-heading">Boundary five: evidence, because someone will ask</h2>



<p class="wp-block-paragraph">At some point the question stops being &#8220;does it work&#8221; and becomes &#8220;who saw what, and when&#8221;. If you cannot answer that from stored records, the platform is not defensible regardless of how good the answers are.</p>



<ul class="wp-block-list">
<li>Turn on CloudTrail data events for the document buckets. They are off by default, they are billed separately from management events, and they are the only way to see individual object-level reads.</li>



<li>Log the retrieval, not just the generation. Store which chunks came back and which filter was applied. When somebody asks whether the wall held on a specific query, this record is the answer and there is no reconstructing it later.</li>



<li>Carry the end user&#8217;s identity through to the audit record. A Lambda execution role in the logs tells you the platform did something. It does not tell you who asked.</li>



<li>Alarm on the filter, not just on errors. A retrieval that ran without a matter filter should page someone, even though it returned HTTP 200 and a perfectly good answer.</li>
</ul>



<p class="wp-block-paragraph">That last one is the closest thing to a single takeaway in this post. The dangerous events in this architecture are successful ones.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the things that will go wrong</h2>



<h3 class="wp-block-heading">Retrieval returns nothing after you add a filter</h3>



<p class="wp-block-paragraph">Nearly always a metadata problem rather than a query problem. Either the sidecar metadata file was missing at ingestion, so the chunks carry no attribute to match, or the attribute was added after the chunks were indexed. Check whether the attribute exists on a known chunk before you touch the query. If it does not, you need a re-sync, not a different filter expression.</p>



<h3 class="wp-block-heading">Textract returns text but the layout is scrambled</h3>



<p class="wp-block-paragraph">Common on two-column pleadings and on documents with headers and footers on every page. Raw text detection reads in a reading order that is not always yours. Use the layout and table features rather than plain detection, and hold onto the geometry so you can reassemble columns yourself if the default order is wrong for that document class.</p>



<h3 class="wp-block-heading">The model cites a document that does not exist</h3>



<p class="wp-block-paragraph">Usually the citation is being generated rather than passed through. If document names are being written by the model instead of read from the retrieval response, it will invent plausible ones. Build citations from the retrieval result metadata in your application code and never from the generated text. Bedrock Guardrails helps with contextual grounding, but the structural fix is not asking the model to produce the reference in the first place.</p>



<h3 class="wp-block-heading">Ingestion is slow and nobody knows where</h3>



<p class="wp-block-paragraph">Instrument per stage before you optimise anything. In most of these pipelines the time is in Textract for large scanned bundles, and the fix is parallelism at the document level rather than tuning the extraction itself. Step Functions with a distributed map over documents gets you there without a queue you have to babysit.</p>



<h3 class="wp-block-heading">Costs climb faster than volume</h3>



<p class="wp-block-paragraph">Look at re-processing first. A pipeline that re-extracts a document every time anything downstream changes will quietly multiply your Textract bill against a static corpus. Key the extraction cache on the object&#8217;s content hash, not its path, and make re-ingestion an explicit action. The billing mechanisms differ per service, so read the current pricing pages rather than trusting a number from a blog post.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating the system prompt as an access control.</strong> It is a formatting instruction that happens to look like a rule.</li>



<li><strong>Taking the tenant identifier from the request body.</strong> If the client can name the matter, the client can name someone else&#8217;s.</li>



<li><strong>Designing the metadata schema after the first ingestion run.</strong> Retrofitting a filterable attribute means re-indexing the whole corpus.</li>



<li><strong>Assuming a deleted S3 object is gone from the platform.</strong> The chunks, the extraction output and the invocation logs all outlive it.</li>



<li><strong>Setting compliance-mode retention without testing in governance mode.</strong> There is no support ticket that fixes a seven-year mistake.</li>



<li><strong>Enabling model invocation logging without treating the log as privileged.</strong> You have just made a second copy of the documents in a less protected place.</li>



<li><strong>Building VPC endpoints and leaving the default policy on them.</strong> A private path with no policy is still a path.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>One retrieval helper, server-side filter injection, and a review rule that no other code path calls the retrieve API directly.</li>



<li>Metadata schema agreed and frozen before the first production ingestion. Include the fields you might filter on later, even if unused today.</li>



<li>Page-level provenance on every chunk, surfaced in the interface. It builds trust faster than any accuracy improvement.</li>



<li>A confidence threshold chosen per document class, with a human review path for everything under it.</li>



<li>Deletion implemented as a verified workflow across every store, ending in an assertion that the content is actually unretrievable.</li>



<li>Object Lock legal holds for matters under litigation hold, tested in governance mode before anything runs in compliance mode.</li>



<li>Customer-managed KMS keys across S3, the vector store and the logs, so key access is a control you can actually revoke.</li>



<li>An alarm on unfiltered retrievals, because the failure mode is a success response.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a legal document intelligence platform on AWS safe for privileged material?</h3>



<p class="wp-block-paragraph">The services support it. Bedrock&#8217;s stated position is that inputs and outputs are not shared with model providers or used to train base models, PrivateLink keeps traffic off the public internet, and KMS covers encryption at rest. What determines safety is your own configuration: tenant isolation, log handling and deletion behaviour. The platform is exactly as confidential as its weakest copy of the text.</p>



<h3 class="wp-block-heading">Should I use Bedrock Knowledge Bases or build retrieval myself?</h3>



<p class="wp-block-paragraph">Start with Knowledge Bases. It handles chunking, embedding, sync and metadata filtering, and those are weeks of work with no differentiation in them. Build your own when you need retrieval behaviour it does not expose, such as unusual re-ranking or per-tenant chunking strategies. Going custom on day one usually means reimplementing the managed service badly.</p>



<h3 class="wp-block-heading">How do I stop one client&#8217;s documents surfacing in another client&#8217;s answers?</h3>



<p class="wp-block-paragraph">Tag every chunk with a tenant identifier at ingestion, and inject the matching filter server-side on every retrieval from the authenticated caller&#8217;s identity. Never accept the identifier from the client. If the contract requires per-client encryption keys, use a separate collection per client instead, accepting the extra operational load.</p>



<h3 class="wp-block-heading">Does deleting a document from S3 remove it from the search index?</h3>



<p class="wp-block-paragraph">No. The chunks stay in the vector store until an ingestion job runs and reconciles the data source. Until then the document is deleted and still fully searchable, which is the worst combination available. Always finish a deletion by querying the index and confirming nothing comes back.</p>



<h3 class="wp-block-heading">What is the difference between an S3 legal hold and a retention period?</h3>



<p class="wp-block-paragraph">A retention period runs until a fixed date and comes in governance mode, which privileged users can override, or compliance mode, which nobody can. A legal hold has no end date and stays until someone explicitly removes it. They are independent, they can both be active on the same object version, and while either is active the object cannot be deleted or overwritten.</p>



<h3 class="wp-block-heading">Do I need human review, or is the model accurate enough?</h3>



<p class="wp-block-paragraph">Accuracy is the wrong frame. The question is what happens to a wrong answer downstream. Where output feeds a decision with consequences, you want a confidence threshold and a review queue, and Amazon Augmented AI with a private workforce gives you that without building the review tooling yourself. Where output is a search aid a person will verify anyway, review is friction you do not need.</p>



<h3 class="wp-block-heading">Can I run this in a single AWS account?</h3>



<p class="wp-block-paragraph">Technically yes, and for a pilot it is reasonable. It gets uncomfortable once you have production documents and a development environment in the same place, because an IAM mistake has nowhere to stop. Separate accounts under Organizations also gives you the policy layer, including the AI services opt-out policy, which only exists at the organisation level.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A legal document intelligence platform on AWS is not hard to build. Textract reads, Bedrock reasons, OpenSearch retrieves, and the tutorial version works on the first afternoon.</p>



<p class="wp-block-paragraph">What is hard is that its failures are silent and well-formed. The cross-matter citation, the deleted document that still answers questions, the privileged passage sitting in a log bucket, the retrieval that ran without a filter and returned a beautiful paragraph. None of these throw an error. All of them are the kind of thing that ends a client relationship.</p>



<p class="wp-block-paragraph">So build the boundaries first and the features second, and make sure every one of them is something you can prove from a stored record rather than something you believe about the code. If you can only take one habit from this: alarm on the successful requests that should not have been possible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your document pipeline?</h2>



<p class="wp-block-paragraph">I work on AWS document processing and retrieval systems where the confidentiality requirements are real. Typical things I get called in for:</p>



<ul class="wp-block-list">
<li>Reviewing tenant isolation in an existing RAG setup and finding the paths where the filter can be bypassed.</li>



<li>Designing the metadata and chunking schema before ingestion, so filtering works without a re-index later.</li>



<li>Building Textract pipelines with confidence thresholds, human review routing and page-level provenance.</li>



<li>Implementing verified deletion across S3, derived artefacts, the vector index and logs, with a proof step at the end.</li>



<li>Setting up retention and legal hold with S3 Object Lock, including the governance-mode rehearsal before compliance mode.</li>



<li>Locking down the network and logging path: VPC endpoints with real policies, KMS key separation, and audit records that name the actual user.</li>
</ul>



<p class="wp-block-paragraph">If you have an architecture diagram, a Step Functions definition or a retrieval request you are unsure about, send it over and I will tell you what I would change and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</title>
		<link>https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 20:19:52 +0000</pubDate>
				<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[As-Built Schedule]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[Bates Numbering]]></category>
		<category><![CDATA[Capital Projects]]></category>
		<category><![CDATA[Chain of Custody]]></category>
		<category><![CDATA[Common Data Environment]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Delay Analysis]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Extension of Time]]></category>
		<category><![CDATA[Forensic Schedule Analysis]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Hybrid Search]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[Load Files]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Oracle Aconex]]></category>
		<category><![CDATA[pgvector]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[Project Controls]]></category>
		<category><![CDATA[Provenance]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Reciprocal Rank Fusion]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[System of Record]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<category><![CDATA[XER Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=336</guid>

					<description><![CDATA[<p>A layer-by-layer architecture for a construction dispute intelligence platform: hashed intake and immutable originals, OCR with page coordinates, a versioned Primavera P6 schedule plane, an event model that joins documents to activities, hybrid retrieval on pgvector, and a generation layer bounded by human confirmation. Written around what each layer prevents.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/">Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that kills these systems is always the same one, and it always comes from the expert, not the engineer: &#8220;Where did this date come from?&#8221;</p>



<p class="wp-block-paragraph">You have a screen showing a beautiful chronology. Two hundred events, colour-coded by discipline, cross-referenced against the programme. The delay expert points at row 47, the one that says the piling rig arrived late, and asks which document that came from, who produced it, what page, and whether the other side has seen it. If the answer involves anyone opening a folder and searching, the platform has already failed. Not technically. Evidentially. A chronology nobody can trace back to a produced document is a nice picture, and a nice picture is worth nothing in an adjudication.</p>



<p class="wp-block-paragraph">That is the constraint that shapes everything else. A construction dispute intelligence platform is not a search engine with a construction theme on it. It is a provenance system that happens to do search. Get that ordering wrong and you build something impressive that no expert will sign their name to.</p>



<p class="wp-block-paragraph">This post walks through the architecture layer by layer: intake and chain of custody, normalization, the schedule data plane, the chronology model that joins the two, retrieval, and the tightly bounded generation layer on top. For each one I&#8217;ll say what it prevents, because that is the only reason any of these layers exist.</p>



<h2 class="wp-block-heading">What a construction dispute intelligence platform actually has to do</h2>



<p class="wp-block-paragraph">Strip away the vocabulary and the job is narrow. Take a pile of heterogeneous records from a capital project, some of them structured and some of them scanned faxes, and let a small number of expensive humans answer three questions quickly and defensibly:</p>



<ul class="wp-block-list">
<li>What happened, in what order, and who said so at the time?</li>



<li>What did the programme say before and after each of those things happened?</li>



<li>Which contemporaneous record supports every single assertion in the report?</li>
</ul>



<p class="wp-block-paragraph">The third one is not a nice-to-have. Both of the industry references people actually argue from lean on it. The Society of Construction Law Delay and Disruption Protocol raised record keeping to a core principle in its second edition and stopped expressing a preference between retrospective delay analysis methods, on the reasoning that the records you have largely determine the method available to you. AACE International&#8217;s Recommended Practice 29R-03 codifies forensic schedule analysis into a taxonomy of Method Implementation Protocols, and its minimum implementation protocols are essentially statements about which source schedules you must be able to produce. Both documents are, underneath, about evidence handling. So is your architecture.</p>



<h2 class="wp-block-heading">Layer one: intake that a forensic expert can defend</h2>



<p class="wp-block-paragraph">Everything upstream of this layer is somebody else&#8217;s mess. Site photos on a phone, a shared drive nobody pruned since mobilization, an Aconex export, a hard drive that arrived in an envelope. Your job at intake is to make an irreversible statement about what you received and when, before anyone touches it.</p>



<p class="wp-block-paragraph">Hash every file on arrival and store the hash next to the object, not in a spreadsheet. SHA-256 over the raw bytes, computed before any conversion, deduplication or renaming. The hash becomes the document&#8217;s identity for the rest of its life.</p>



<pre class="wp-block-code"><code># Build an arrival manifest before anything else runs
find ./ACME-INTAKE-001 -type f -print0 
  | xargs -0 sha256sum &gt; manifest-ACME-INTAKE-001.txt

# Re-verify later, after the files have moved through processing
sha256sum -c manifest-ACME-INTAKE-001.txt</code></pre>



<p class="wp-block-paragraph">Then put the originals somewhere they cannot be quietly edited. On AWS, S3 Object Lock in compliance mode with a retention period is the blunt instrument: once set, the object version cannot be overwritten or deleted by anyone, including the root account, until retention expires. Governance mode is the softer variant, where a specifically permissioned principal can lift the lock, which is fine for internal working copies and wrong for originals. If you are self-hosting on a Contabo or InterServer box because the client will not put project data in a hyperscaler, the equivalent discipline is a write-once staging volume plus an append-only manifest log, and you accept that you are relying on process rather than on the storage layer refusing the write.</p>



<h3 class="wp-block-heading">The manifest is the product</h3>



<p class="wp-block-paragraph">If the matter is already in formal disclosure, you will be handed productions rather than raw folders, and those arrive as load files. A Concordance DAT carries one row per document with fields like BegBates, EndBates, Custodian, BegAttach, EndAttach and an MD5 hash. An Opticon OPT carries one row per page, mapping Bates numbers to image paths. The delimiters are deliberately weird characters that real text rarely contains, and different tools disagree about the exact code points, so parse defensively and log what you detected rather than assuming.</p>



<p class="wp-block-paragraph">The failure mode here is quiet and expensive. Variable-width Bates numbers sort lexically instead of numerically, so ABC10 lands between ABC1 and ABC2, attachment ranges resolve to the wrong parents, and a family of documents silently splits. Nobody notices until an expert cites a covering email whose attachment, in your system, belongs to a different letter. Pad to fixed width on ingest and reconcile every BegAttach range against the documents you actually loaded.</p>



<h2 class="wp-block-heading">Layer two: normalization, and the OCR tax nobody budgets for</h2>



<p class="wp-block-paragraph">Construction corpora are the worst kind of mixed. Native Office files with clean text, born-digital PDFs, scanned site instructions with a signature and a coffee ring, photographs of whiteboards, and drawings where the only text that matters is inside a title block in the bottom right corner.</p>



<p class="wp-block-paragraph">Route by what the file actually is, not by extension. For a PDF, try text extraction first; if the extracted layer is empty or nearly empty, it is a scan and needs OCR. Tesseract via OCRmyPDF is the cheap path and it is genuinely good on clean typed pages. It degrades badly on handwriting, on stamps, and on anything skewed, which describes most site paperwork. A managed service such as Amazon Textract handles tables and forms better and returns geometry for every detected block, which matters more than accuracy alone.</p>



<p class="wp-block-paragraph">Geometry is the part teams skip and then rebuild six months later. If you store only the extracted string, you can prove a phrase exists somewhere in a document. If you store the page number and the bounding box, you can put a highlight on the exact line in the viewer, and the expert can confirm it in two seconds instead of reading eleven pages. That difference is the difference between a tool people use and a tool people abandon.</p>



<p class="wp-block-paragraph">Keep the OCR text separate from the original in storage, and record which engine and which settings produced it. When a disputed reading of a handwritten date turns up, and it will, you want to be able to say exactly how that character was produced.</p>



<h2 class="wp-block-heading">Layer three: the schedule plane is a separate data model</h2>



<p class="wp-block-paragraph">This is where most teams underestimate the work. Programme data is not documents. It is a graph, versioned, with a calendar attached, and it needs its own tables, its own ingest path and its own validation.</p>



<p class="wp-block-paragraph">Primavera P6 exports as XER, a tab-delimited text format with a table-per-section layout, or as P6 XML. Either way you are pulling out projects, the WBS tree, activities, logic links and calendars. In XER these live in sections including PROJECT, PROJWBS, TASK, TASKPRED and CALENDAR. Parsing it by hand is a bad use of a week; PyP6Xer is an open source Python parser that will get you to objects quickly.</p>



<pre class="wp-block-code"><code>from xerparser.reader import Reader

xer = Reader("update-14.xer")

for project in xer.projects:
    print(project.proj_short_name)
    for activity in project.activities:
        print(activity.task_code, activity.task_name)</code></pre>



<p class="wp-block-paragraph">Two things bite immediately. First, several duration and float fields in XER are stored in hours rather than days, which you can see in the column names themselves. If you load them straight into a &#8220;days&#8221; column because the numbers looked plausible on a five-day calendar, every float figure downstream is wrong by a factor that varies with the calendar. Check your own export before you trust any mapping, including this one.</p>



<p class="wp-block-paragraph">Second, activity IDs are not stable across a project&#8217;s life. Activities get renumbered, split, deleted and re-added between updates. If your schema keys on the activity ID alone, two updates will silently merge activities that were never the same work. Key on a composite of schedule version plus activity ID, and maintain a separate mapping table for identity across versions that a human can correct. That table is boring, it is not automatable, and it is the difference between a windows analysis that holds together and one that quietly compares apples to a different orchard.</p>



<p class="wp-block-paragraph">Store every update as an immutable snapshot with its data date. Never mutate a loaded schedule. The whole point of the layer is that you can reconstruct exactly what the programme said on a given day, which is precisely what a time slice or windows method needs.</p>



<h2 class="wp-block-heading">Layer four: the chronology is the join, and the join is the hard part</h2>



<p class="wp-block-paragraph">Documents on one side, schedule snapshots on the other. The product is the thing in the middle: an event, with a date, a description, a set of supporting document references, and a set of affected activities.</p>



<p class="wp-block-paragraph">Model it explicitly. An event is not a document and it is not an activity. One letter can create three events. One event can be evidenced by nine documents and a photograph. The relationship is many-to-many in both directions and every attempt I have seen to flatten it ends with someone maintaining a spreadsheet on the side.</p>



<pre class="wp-block-code"><code>-- Every assertion resolves to a page and a box on that page
CREATE TABLE evidence_ref (
  id            BIGSERIAL PRIMARY KEY,
  event_id      BIGINT NOT NULL REFERENCES event(id),
  document_id   UUID   NOT NULL REFERENCES document(id),
  page_number   INT    NOT NULL,
  bbox          JSONB,             -- normalized coordinates from OCR
  quoted_text   TEXT   NOT NULL,
  asserted_by   TEXT   NOT NULL,   -- 'analyst' or 'model'
  confirmed_by  TEXT,              -- NULL until a human confirms
  confirmed_at  TIMESTAMPTZ
);</code></pre>



<p class="wp-block-paragraph">Those last three columns are the ones that matter. They encode the difference between something the machine proposed and something a person stands behind. Report generation reads only rows where <code>confirmed_by</code> is set. Everything else is a suggestion queue. This is not a compliance box; it is the mechanism by which an expert can put their name on the output, and it needs to exist in the schema rather than in a workflow convention that erodes under deadline.</p>



<h2 class="wp-block-heading">Layer five: retrieval that finds document numbers, not just meaning</h2>



<p class="wp-block-paragraph">Pure vector search is the wrong default for this corpus, and it fails in a way that looks like success. Embeddings are good at meaning and bad at tokens. Ask for &#8220;RFI 0442&#8221; and a dense retriever will cheerfully hand you eleven chunks about requests for information, none of which is RFI 0442, because the identifier got smeared into its semantic neighbourhood. Construction disputes run on identifiers: RFI numbers, variation order references, drawing revisions, activity IDs, Bates ranges.</p>



<p class="wp-block-paragraph">Run lexical and semantic retrieval in parallel and fuse the rankings. PostgreSQL with the pgvector extension does both without adding infrastructure: an HNSW index over the embedding column, a GIN index over a generated tsvector column, and reciprocal rank fusion to merge the two candidate lists.</p>



<pre class="wp-block-code"><code>CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunk (
  id          BIGSERIAL PRIMARY KEY,
  document_id UUID NOT NULL,
  page_number INT  NOT NULL,
  body        TEXT NOT NULL,
  body_tsv    tsvector GENERATED ALWAYS AS
                (to_tsvector('english', body)) STORED,
  embedding   vector(1536)
);

CREATE INDEX chunk_tsv_idx  ON chunk USING gin (body_tsv);
CREATE INDEX chunk_hnsw_idx ON chunk USING hnsw (embedding vector_cosine_ops);</code></pre>



<p class="wp-block-paragraph">Over-fetch from each side and trim after fusion rather than pulling the final count from each; more candidates give the fusion step something to work with. Add a trigram index if your corpus has OCR noise, because &#8220;RFI 0442&#8221; and &#8220;RFl 0442&#8221; are the same document to a human and different tokens to Postgres.</p>



<p class="wp-block-paragraph">Filter hard on metadata before ranking. Matter, custodian, date range, privilege status. A retrieval layer that can return a privileged document into a shared workspace is a liability, not a feature.</p>



<h2 class="wp-block-heading">Layer six: generation, bounded hard</h2>



<p class="wp-block-paragraph">Language models are useful here and the useful jobs are smaller than the demos suggest: proposing candidate events from a batch of correspondence, drafting neutral one-line summaries, clustering near-duplicate records, suggesting which activities an event might touch. Every one of those is a proposal that lands in a review queue.</p>



<p class="wp-block-paragraph">The reason to be strict is not squeamishness. Stanford&#8217;s RegLab and HAI ran a preregistered evaluation of commercial legal research tools built on retrieval-augmented generation, with expert-scored answers, and found hallucination rates in the range of roughly 17 to 33 percent depending on the product, against vendor marketing that had claimed hallucination-free citations. The vendors disputed the characterization. The finding that should worry you more than the headline number is the failure mode underneath it: a citation that exists, is quoted accurately, and does not stand for what the answer says it stands for. Retrieval reduces fabrication. It does not produce correct reasoning about what a document means, and misgrounding is far harder to spot than an invented reference.</p>



<p class="wp-block-paragraph">So bound it. Generation writes to the proposal table, never to the report. Every generated assertion carries the chunk IDs it was built from, and the UI shows the source page beside the claim by default rather than behind a click. If a proposed event has no evidence reference, it does not enter the queue at all.</p>



<h2 class="wp-block-heading">Tenancy, privilege, and the thing that ends the engagement</h2>



<p class="wp-block-paragraph">Matters do not share data. Ever. Not &#8220;logically separated by a WHERE clause that we always remember to add&#8221;. The cheapest defensible pattern is a database per matter, or at minimum a schema per matter with connection-level scoping, so that a missing predicate produces an empty result rather than another client&#8217;s correspondence.</p>



<p class="wp-block-paragraph">Two more that get discovered late. Privilege state is per document per matter and it changes; model it as a versioned attribute with an audit trail, not a boolean someone flipped. And plan the exit at the start: matters end, retention clocks expire, and someone has to certify destruction. That means knowing every copy you made, including OCR text, thumbnails, embeddings and backups. Tools like the O&amp;O Software utilities have a place at the endpoint end of that job; the platform side is a deletion manifest that enumerates derived artifacts per document, written at ingest rather than reconstructed under pressure.</p>



<p class="wp-block-paragraph">If experts are working from home, from site cabins, or from three countries, the access path is part of the design. A managed VPN such as NordVPN or Surfshark on the client side plus network-level allowlisting on the platform side is a pragmatic combination, and it is worth writing into the engagement documentation so nobody improvises with a personal Dropbox at eleven at night.</p>



<h2 class="wp-block-heading">Where these platforms break in practice</h2>



<ul class="wp-block-list">
<li><strong>Chronology dates drift from document dates.</strong> Usually a timezone or a date-format collision: a scanned form read as DD/MM, an email header parsed as UTC, a schedule data date stored as a naive timestamp. Store the raw string you extracted alongside the parsed value and show both in the review UI.</li>



<li><strong>Float figures look wrong by a constant factor.</strong> Hours versus days in the schedule import. Check against the same activity opened in P6 before you debug anything else.</li>



<li><strong>Retrieval misses a document everyone knows exists.</strong> Either it was never OCR&#8217;d because the PDF had a thin text layer that passed your emptiness check, or the identifier is only in the title block of a drawing. Log extraction character counts per page and alert on suspiciously low ones.</li>



<li><strong>Two activities merged across updates.</strong> Composite keying was not applied, or an activity was renumbered. Diff activity code sets between consecutive snapshots and surface additions and deletions as a report the planner reviews.</li>



<li><strong>Document families split.</strong> Bates padding or an unreconciled attachment range from the load file. Re-run the family reconciliation and count orphaned attachments.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the schedule as documents. It is a graph with versions, and shoehorning it into the document store costs you every analysis you actually wanted.</li>



<li>Storing extracted text without page and coordinate provenance. Cheap now, a full reprocessing run later.</li>



<li>Vector-only retrieval. Looks great in a demo built from prose, falls over on the first RFI number.</li>



<li>Letting model output reach a deliverable without a recorded human confirmation.</li>



<li>Mutating loaded schedules to &#8220;fix&#8221; them. Fix on the way in, or hold the correction as an annotation. Never edit the snapshot.</li>



<li>Deferring the deletion story to the end of the matter, when the derived artifacts are already scattered.</li>
</ul>



<h2 class="wp-block-heading">Best practices for building a construction dispute intelligence platform</h2>



<ul class="wp-block-list">
<li>Hash at the door, lock the originals, and never process in place.</li>



<li>Make every derived artifact point back to a document ID, a page and a region.</li>



<li>Give schedules their own immutable, version-keyed store with the data date as a first-class field.</li>



<li>Model events separately from both documents and activities, with many-to-many links in each direction.</li>



<li>Run hybrid retrieval and fuse the rankings; filter on matter and privilege before you rank.</li>



<li>Keep generation in a proposal queue with confirmation recorded in the schema.</li>



<li>Isolate matters at the database or schema boundary, not in application code.</li>



<li>Instrument ingest as seriously as you would a production pipeline. Grafana over your extraction counts and OCR queue depth will tell you a corpus is going wrong days before an analyst does.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a dedicated vector database for this?</h3>



<p class="wp-block-paragraph">Almost certainly not. Matter corpora are large in gigabytes and small in chunk count compared to consumer-scale search. PostgreSQL with pgvector keeps the embeddings in the same transaction as the provenance rows, which means you cannot end up with a chunk whose document record was rolled back. That consistency is worth more here than the last few milliseconds of query latency.</p>



<h3 class="wp-block-heading">Should the platform pick a delay analysis method?</h3>



<p class="wp-block-paragraph">No. That choice belongs to the expert and depends on the contract, the forum, the records available and the timing of the analysis. What the platform should do is make it obvious which methods the record set can support, by reporting what you actually have: how many schedule updates, at what intervals, with what data dates, and how complete the contemporaneous correspondence is around each window.</p>



<h3 class="wp-block-heading">XER or P6 XML for schedule ingest?</h3>



<p class="wp-block-paragraph">Accept both, because you will be sent both. XER is what planners export by habit and it is easy to parse with existing libraries. P6 XML is more explicit about structure and less prone to encoding surprises. Whichever arrives, keep the original file, hashed, alongside the parsed rows.</p>



<h3 class="wp-block-heading">How much of the review can realistically be automated?</h3>



<p class="wp-block-paragraph">The ranking and the grouping, most of it. The judgement, none of it. Automation that reduces a hundred thousand documents to a few thousand worth reading is enormously valuable and uncontroversial. Automation that decides what a letter means is where you inherit risk you cannot price.</p>



<h3 class="wp-block-heading">Can I build this on top of a common data environment we already have?</h3>



<p class="wp-block-paragraph">Partly. A CDE such as Aconex is a good source of records and a poor system of record for a dispute, because access changes, exports are point-in-time, and the retention rules belong to the project rather than to the matter. Pull from it, hash what you pulled, and hold your own copy under your own lock.</p>



<h3 class="wp-block-heading">What does this cost to run?</h3>



<p class="wp-block-paragraph">The shape of the bill is more useful than a number. OCR and embedding are one-time per document and scale with page count, so a scan-heavy corpus costs multiples of a born-digital one. Storage is small and constant. Retrieval is negligible. Generation scales with how much you push through it, which is another argument for narrow, queued use rather than a chat box over the whole corpus.</p>



<h3 class="wp-block-heading">Is a general eDiscovery platform not enough?</h3>



<p class="wp-block-paragraph">For document review, often yes, and they are considerably better at productions and privilege logging than anything you will build in a quarter. What they do not have is the schedule plane. The value of a construction-specific build is the join between correspondence and programme versions, so a sensible architecture frequently uses an established review platform for documents and adds the schedule and chronology layers alongside it.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A construction dispute intelligence platform earns its place by making provenance cheap. Search, clustering and summarization are commodity capabilities now; anyone can bolt them onto a corpus in a fortnight. What is hard, and what decides whether an expert will rely on the output, is that every date, every event and every sentence in the eventual report resolves in one click to a specific page of a specific produced document, with a record of who confirmed it.</p>



<p class="wp-block-paragraph">Build the provenance spine first. Hash at intake, keep coordinates, version the schedules, model events as their own thing, and keep the model on the proposal side of a human confirmation. The clever features are easy to add afterwards. Retrofitting evidence handling into a system that was designed as a search tool is a rebuild, and it always arrives at the worst possible moment, which is about three weeks before a hearing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or fixing this?</h2>



<p class="wp-block-paragraph">I work on the infrastructure side of exactly this problem: getting messy project records into a system where every claim is traceable. Things I can help with:</p>



<ul class="wp-block-list">
<li>Designing the intake and chain-of-custody layer, including hashing, immutable storage, and load file parsing that survives odd delimiters and inconsistent Bates padding.</li>



<li>Building the schedule ingest path for XER and P6 XML, with version-keyed snapshots and an activity identity map that holds across updates.</li>



<li>Standing up hybrid retrieval on PostgreSQL and pgvector, including page-level and coordinate-level provenance so highlights land on the right line.</li>



<li>Setting up the OCR and normalization pipeline, with routing rules, quality checks, and alerting on pages that extracted suspiciously little text.</li>



<li>Matter isolation, privilege state modelling, audit logging, and a deletion manifest you can act on at the end of the engagement.</li>



<li>Reviewing an existing platform and telling you plainly where the provenance chain breaks.</li>
</ul>



<p class="wp-block-paragraph">If you have something concrete, send it over: an XER export, a sample DAT and OPT pair, a page count breakdown of your corpus, or the schema you are currently working from. It is much easier to say something useful about a real file than about a description of one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/construction-dispute-intelligence-platform/">Designing a Construction Dispute Intelligence Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
