<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Bedrock Guardrails | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/bedrock-guardrails/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/bedrock-guardrails/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 17 Oct 2026 06:00:06 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Bedrock Guardrails | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/bedrock-guardrails/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Building a Personalized Learning Assistant on AWS Without Leaking Student Data</title>
		<link>https://john-nessime.com/blog/technical-guides/personalized-learning-assistant-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 17 Oct 2026 06:00:06 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Education Technology]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3 Vectors]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[EdTech]]></category>
		<category><![CDATA[FERPA]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Identity Propagation]]></category>
		<category><![CDATA[Learning Management System]]></category>
		<category><![CDATA[LTI 1.3]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[pgvector]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Student Data Privacy]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=466</guid>

					<description><![CDATA[<p>A personalized learning assistant on AWS rarely fails loudly. It fails by being helpful to the wrong student. Five failure families that decide whether an AI tutor built on Amazon Bedrock is safe to put in front of a class: LTI identity, retrieval isolation, guardrail limits, the cost shape of an academic term, and being able to prove what the assistant saw.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/personalized-learning-assistant-aws/">Building a Personalized Learning Assistant on AWS Without Leaking Student Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket that kicks this project into its second phase usually reads something like: &#8220;The assistant is giving my students the answers.&#8221; Attached is a screenshot where the tutor bot cheerfully walks a student through the marking scheme for a coursework brief that has not been released yet. Nothing crashed. No 5xx, no alarm, no failed health check. The rubric PDF had been dropped into the same bucket prefix as the lecture notes weeks earlier, and retrieval did exactly what retrieval does.</p>



<p class="wp-block-paragraph">That is the shape of every serious problem in this space. A personalized learning assistant on AWS almost never fails loudly. It fails by being helpful to the wrong person, or by being confidently wrong in a way that reads as authoritative because it came from the institution&#8217;s own system. Both failures are silent until a human reports them, and by then the assistant has been answering that way for a term.</p>



<p class="wp-block-paragraph">This post covers the parts that are genuinely hard: getting identity and scope out of the learning platform instead of the browser, making retrieval respect the boundary, knowing what Bedrock Guardrails will and will not do for you, sizing a system whose traffic collapses every summer, and proving after the fact what the assistant saw. I assume you can already stand up a RAG pipeline. The pipeline is the easy part.</p>



<h2 class="wp-block-heading">What &#8220;personalized&#8221; has to mean before you build anything</h2>



<p class="wp-block-paragraph">Personalization in an education setting is three separate questions that people tend to collapse into one:</p>



<ul class="wp-block-list">
<li><strong>What is this learner allowed to see?</strong> Course enrollment, section, release dates, role. This is an authorization question and it belongs nowhere near the model.</li>



<li><strong>What has this learner already done?</strong> Submissions, attempts, viewed material. This is state, and it belongs in a database you control.</li>



<li><strong>How is this learner doing?</strong> Mastery, misconceptions, pace. This is inference, and it is the only part where a model earns its keep.</li>
</ul>



<p class="wp-block-paragraph">Teams that skip straight to the third question end up asking the model to enforce the first one through prompt instructions. That does not hold. A system prompt saying &#8220;do not reveal solutions before the due date&#8221; is a suggestion; a retrieval filter that never returns the solution chunk is a control. Keep the boundary in the retrieval layer and the personalization in the generation layer, and most of the scary failure modes stop being possible rather than becoming unlikely.</p>



<h2 class="wp-block-heading">Where a personalized learning assistant on AWS actually breaks</h2>



<p class="wp-block-paragraph">Organized by failure family rather than by service, because the services are documented and the failures are not. Five of them, roughly in the order they bite.</p>



<h2 class="wp-block-heading">Failure family one: identity that stops at the front door</h2>



<p class="wp-block-paragraph">The assistant lives inside Canvas, Moodle, Blackboard, Brightspace or Open edX. It gets there through LTI 1.3, which replaced the old shared-secret model with an OpenID Connect launch and an RSA-signed JWT. The platform initiates a login against your tool&#8217;s login endpoint, your tool responds with an auth request, the platform posts back a signed token, and you validate it against the platform&#8217;s published keys.</p>



<p class="wp-block-paragraph">Almost everyone gets that far. The failure is what happens next.</p>



<p class="wp-block-paragraph">The launch token carries a stable subject identifier, a roles claim, a context claim describing the course, and a deployment identifier, all namespaced under the 1EdTech spec URIs. Teams validate the signature, pull the user ID out, then build the rest of the session from whatever the front end sends afterwards. Course ID as a query parameter. Role in local storage. It works in testing because the front end sends the truth, and it stops working the moment a curious student opens developer tools. If your filter values arrive from the client, you have built an access control system that asks the attacker what they should be allowed to see.</p>



<h3 class="wp-block-heading">Take the scope from the launch, then keep it server-side</h3>



<p class="wp-block-paragraph">The pattern that holds up:</p>



<ol class="wp-block-list">
<li>Validate the launch JWT against the platform&#8217;s key set. Check the issuer, the audience, the nonce and the deployment identifier, not just the signature.</li>



<li>Extract subject, roles and context from the token and write them into a server-side session. Hand the browser an opaque session cookie and nothing else.</li>



<li>Derive every retrieval filter from that session record. The client never names a course, a section or a role.</li>



<li>If you need the full roster rather than just the launching user, call Names and Role Provisioning Services with an OAuth 2.0 client credentials token. That gives you learners who have never launched the tool, which matters for cohort-level features.</li>



<li>If you write grades back, do it through Assignment and Grade Services against a line item you created. A launch that works tells you nothing about whether passback works.</li>
</ol>



<p class="wp-block-paragraph">Test launch, roster fetch and grade passback as three independent contracts. They fail independently and they fail quietly. A term can start with a tool that launches beautifully and drops every grade on the floor because AGS was never wired to a line item.</p>



<p class="wp-block-paragraph">Worth knowing before you commit: LTI 1.3 is meaningfully more setup than 1.1. Key registration, the OIDC handshake, per-deployment configuration on every platform instance. That is the price of not signing student traffic with a shared secret, and it is worth paying, but budget for it honestly. If you need to test against several platforms, a self-managed Moodle or Open edX instance on a cheap VPS from somewhere like Contabo or InterServer beats waiting on sandbox tenants, and you get to break it freely.</p>



<h2 class="wp-block-heading">Failure family two: retrieval that ignores the boundary</h2>



<p class="wp-block-paragraph">This is the one from the opening. Vector search does not know about courses. It knows about similarity. If the answer key is semantically close to the question, it comes back, and the model uses it, and the response is fluent and correct and completely inappropriate.</p>



<p class="wp-block-paragraph">Amazon Bedrock Knowledge Bases handles this through metadata filtering: you attach attributes to each document at ingestion, then pass a filter at query time that pre-filters the vector store before the similarity search runs. The filter is the control. Everything else is decoration.</p>



<h3 class="wp-block-heading">The sidecar file is the only metadata that counts</h3>



<p class="wp-block-paragraph">For an S3 data source, attributes come from a companion file sitting next to the document with <code>.metadata.json</code> appended to the object key. This trips people up constantly, because S3 objects already have user-defined metadata and object tags, and neither of those is read for retrieval filtering. Keep lifecycle and cost allocation in S3 tags where they belong, and put filterable attributes in the sidecar.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "course_id": "BIO201",
    "section_id": "BIO201-B",
    "institution_id": "uni-main",
    "material_type": "lecture_notes",
    "visibility": "student"
  }
}</code></pre>



<p class="wp-block-paragraph">Note <code>visibility</code> as a first-class attribute rather than an inference from folder structure. Rubrics, model answers and marking schemes get <code>visibility: "instructor"</code>, and the student-facing filter excludes them by value, not by path convention. Path conventions get violated the first time a busy tutor uploads a file to the wrong folder.</p>



<p class="wp-block-paragraph">At query time you compose the filter server-side from the session record. Bedrock&#8217;s retrieval configuration supports boolean composition, so the tenant condition and the visibility condition are separate clauses that must both hold:</p>



<pre class="wp-block-code"><code>{
  "vectorSearchConfiguration": {
    "filter": {
      "andAll": [
        { "equals": { "key": "institution_id", "value": "uni-main" } },
        { "equals": { "key": "course_id",      "value": "BIO201"   } },
        { "equals": { "key": "visibility",     "value": "student"  } }
      ]
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two habits make this survive production. Build the filter through a single function that always prepends institution and course, so no future endpoint can forget it. Then write a test that queries a known instructor-only chunk&#8217;s exact wording from a student session and asserts zero results. Manual smoke testing never catches this, because human testers ask reasonable questions and the failure only appears when someone asks an unreasonable one.</p>



<h3 class="wp-block-heading">Silo, pool or bridge</h3>



<p class="wp-block-paragraph">AWS frames multi-tenant isolation as three patterns, and the trade-off is the same one you always get between blast radius and cost:</p>



<ul class="wp-block-list">
<li><strong>Silo</strong> gives each institution its own knowledge base and its own vector index. Strongest isolation, easiest compliance conversation, and you multiply your idle infrastructure cost by the number of tenants. There is also a per-account service quota on knowledge bases, so check Service Quotas before you promise a hundred districts their own.</li>



<li><strong>Pool</strong> puts everyone in one index and relies entirely on metadata filtering. Cheapest by a wide margin, and one filter bug is a cross-institution disclosure rather than an embarrassment.</li>



<li><strong>Bridge</strong> splits the difference: separate knowledge bases sharing underlying vector infrastructure, which keeps the isolation story cleaner without paying for a full stack per tenant.</li>
</ul>



<p class="wp-block-paragraph">For a single university running many courses, pool with rigorous filtering is usually right, because the institution is the tenant and the courses are not. For an edtech vendor selling to multiple districts, I would start at bridge and only pool if the economics force it, because the conversation with a district&#8217;s data protection officer goes very differently when your answer is &#8220;we filter&#8221; rather than &#8220;they are separate indexes.&#8221;</p>



<h2 class="wp-block-heading">Failure family three: guardrails that do less than you assume</h2>



<p class="wp-block-paragraph">Bedrock Guardrails is a useful layer and the first thing I reach for on an education workload: content filters, denied topics, word filters, PII detection and contextual grounding checks, applied at inference time and usable across models. For a system that teenagers will deliberately try to provoke, enforcement inside the invocation path beats anything bolted on afterwards. Three limits are documented and routinely missed.</p>



<p class="wp-block-paragraph"><strong>Contextual grounding does not cover chatbots.</strong> AWS documents the supported use cases as summarization, paraphrasing and question answering, and states plainly that conversational QA and chatbot use cases are not supported. A tutoring assistant is a multi-turn conversation. If your hallucination story is &#8220;we turned on grounding checks,&#8221; read that page before it reaches a procurement document. The check still earns its place on single-shot retrieve-and-answer paths, which is where I would apply it, but a multi-turn tutor needs citation display and human review too.</p>



<p class="wp-block-paragraph"><strong>PII detection is probabilistic and text-only.</strong> The sensitive information filter is a machine learning model that uses surrounding context to decide whether something is an identifier. It works better on prose than on bare fragments. AWS also documents that it operates on text output and does not inspect tool use parameters, so if your assistant calls a function with a student identifier in the arguments, that path is not covered. Do not treat it as a substitute for not putting identifiers in the prompt in the first place.</p>



<p class="wp-block-paragraph"><strong>Blocked content lands in your logs.</strong> AWS documents that content blocked by guardrail policies appears as plain text in Bedrock model invocation logs when those logs are enabled. That is defensible for debugging and awkward when the blocked content is a distressed message from a fourteen year old. Decide deliberately whether invocation logging is on, where it lands, who can read it, and how long it lives. In education this is a governance decision, not a logging default.</p>



<p class="wp-block-paragraph">None of this argues against Guardrails. It argues for knowing its edges, because the gap between what a feature name suggests and what the documentation says is where compliance commitments go to die.</p>



<h2 class="wp-block-heading">Failure family four: the cost shape of an academic term</h2>



<p class="wp-block-paragraph">Education traffic is not a smooth curve. It is near zero over summer, spikes brutally in the week before every assessment deadline, and dies again. That shape punishes the default architecture in a specific way.</p>



<p class="wp-block-paragraph">The console nudges you toward Amazon OpenSearch Serverless as the vector store. It bills per OpenSearch Compute Unit hour with a minimum baseline footprint that exists whether or not anyone queries it, so an empty index over a quiet August still bills. AWS has adjusted those minimums over time, so pull current figures from the OpenSearch Serverless capacity documentation rather than from any blog post, this one included. The mechanism is the part that matters: your floor is set by capacity that exists, not by traffic that happens.</p>



<p class="wp-block-paragraph">Bedrock Knowledge Bases supports other backends, and the question that decides between them is idle cost:</p>



<ul class="wp-block-list">
<li><strong>Aurora PostgreSQL with pgvector.</strong> A first-class documented option. Aurora Serverless v2 bills per Aurora Capacity Unit hour and can scale down to zero when idle, which fits the academic calendar almost too neatly. You take on index lifecycle work in exchange: choosing between HNSW and IVFFlat, keeping statistics fresh, watching bloat. If your institution already runs Postgres, that is work your DBAs recognize.</li>



<li><strong>Amazon S3 Vectors.</strong> Bills on storage and requests rather than provisioned capacity, so genuine idle costs approach nothing. The trade is retrieval latency, which is higher than a warm search cluster. For a tutor where a response already takes a second or two of model time, that may not be the bottleneck you think it is.</li>



<li><strong>OpenSearch Serverless.</strong> The right answer when retrieval latency is genuinely your constraint, when you need hybrid search and k-NN tuning, or when you already run collections for other workloads and the baseline is already paid for.</li>
</ul>



<p class="wp-block-paragraph">On the model side, the cost driver is retrieved context, not the question. Every turn ships the retrieved chunks as input tokens, so chunk size and the number of results retrieved multiply directly into your bill on every single message. Halving retrieved results is usually a bigger lever than switching to a cheaper model, and it often improves answer quality too, because the model has less irrelevant material to get distracted by.</p>



<p class="wp-block-paragraph">Tag by course and faculty from day one. The first budget question you will be asked is which department is driving the spend, and retrofitting cost allocation tags after a pilot goes institution-wide is miserable. Cost Explorer answers it if the tags exist; tools like Vantage or CloudZero make the per-course view easier to hand to a finance team that does not want an AWS login.</p>



<h2 class="wp-block-heading">Failure family five: no evidence when someone asks</h2>



<p class="wp-block-paragraph">FERPA governs education records in the United States, and the mechanism that lets a vendor touch them at all is the school official exception: the institution must maintain direct control over the vendor&#8217;s use and maintenance of the records. Similar logic applies under other regimes, and many districts and state contracts add data residency requirements that FERPA itself does not impose.</p>



<p class="wp-block-paragraph">What that means architecturally is less exotic than it sounds, and it is mostly about being able to answer questions after the fact:</p>



<ul class="wp-block-list">
<li>Log the retrieval, not just the answer. Store which chunk identifiers were returned for each turn. When an instructor claims the assistant leaked a rubric, the retrieval log is the only thing that settles it.</li>



<li>Keep student identifiers out of prompts. Pass an opaque session-scoped pseudonym and resolve it in your own system. The model does not need a real name to be helpful.</li>



<li>Encrypt with customer managed KMS keys and use encryption context that names the tenant, so a key policy can actually constrain who decrypts what.</li>



<li>Pin the region and enforce it with service control policies if residency is contractually required. An SCP is checkable evidence; a runbook is not.</li>



<li>Write down retention for conversation history and enforce it with lifecycle rules. &#8220;We keep it forever because storage is cheap&#8221; is a defensible engineering position and an indefensible one in an education procurement review.</li>



<li>Pull the current compliance reports and agreements from AWS Artifact rather than quoting a blog about what AWS certifies.</li>
</ul>



<p class="wp-block-paragraph">None of this is glamorous and all of it is cheaper to build in during the pilot than to retrofit when the institution&#8217;s legal team gets involved.</p>



<h2 class="wp-block-heading">Troubleshooting the things that actually page you</h2>



<h3 class="wp-block-heading">&#8220;The new material isn&#8217;t showing up&#8221;</h3>



<p class="wp-block-paragraph">Almost always ingestion, not retrieval. Walk it in order rather than guessing. Did a sync actually run after the change? For an S3 data source, an edit is invisible until an ingestion job processes it.</p>



<pre class="wp-block-code"><code>aws bedrock-agent list-ingestion-jobs 
  --knowledge-base-id KB_ID 
  --data-source-id DS_ID</code></pre>



<p class="wp-block-paragraph">Look for a completed job starting after the upload. If the most recent job predates the change, your scheduler or event handler is the bug and retrieval is innocent. If a job did run, read its statistics: a modified-documents-indexed count of zero when you expected a change means change detection did not fire, which happens when an object is re-uploaded with identical bytes. A non-zero failed count means your file is probably the one that failed, and the failure reasons say why.</p>



<h3 class="wp-block-heading">&#8220;The filter returns nothing at all&#8221;</h3>



<p class="wp-block-paragraph">Query once with no filter to confirm the content is indexed. If it comes back unfiltered and vanishes when filtered, it is a metadata problem: the sidecar was not ingested, the attribute name does not match exactly, or the value differs in case. These comparisons are literal. <code>BIO201</code> and <code>bio201</code> are two different courses as far as the filter is concerned, which is an argument for normalizing identifiers at both ingestion and query time in one shared helper.</p>



<h3 class="wp-block-heading">&#8220;The launch shows a blank iframe&#8221;</h3>



<p class="wp-block-paragraph">Two suspects dominate. Cookies: your tool is in a cross-site iframe, so session cookies need the right SameSite and Secure attributes or the browser drops them and every launch looks like a fresh unauthenticated user. Registration mismatch: issuer, client ID or deployment ID differs between what the platform sends and what you stored. Log the received issuer and deployment ID on validation failure. Guessing at OIDC failures is slow; the values are right there.</p>



<h3 class="wp-block-heading">&#8220;Answers are right but the citations are wrong&#8221;</h3>



<p class="wp-block-paragraph">Usually chunking. Slides and worksheets fragment badly under naive fixed-size chunking, and a chunk that starts mid-table cites a heading that belongs to something else. Before you change models, look at what the retriever actually returned for the query that misbehaved. Nine times out of ten the model was faithful to a bad chunk.</p>



<h3 class="wp-block-heading">&#8220;Grades aren&#8217;t reaching the gradebook&#8221;</h3>



<p class="wp-block-paragraph">Check that a line item exists and that your service token carries the right AGS scopes. Passback failures are asynchronous, so they surface as an empty gradebook column weeks later rather than as an error the student sees. Alert on passback failure rate explicitly; nobody will report it in time.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Enforcing access control in the system prompt.</strong> Instructions are not permissions. If a chunk can be retrieved, assume it can be surfaced.</li>



<li><strong>Trusting client-supplied course or role values.</strong> Derive scope from the validated launch token and store it server-side, always.</li>



<li><strong>Ingesting the whole course folder.</strong> Instructor material and student material in one prefix is how rubrics leak. Separate at ingestion, tag at ingestion.</li>



<li><strong>Putting real names and student numbers in prompts.</strong> Pseudonymize before the model call and resolve afterwards.</li>



<li><strong>Sizing the vector store from peak week.</strong> The bill is set by the idle floor, and in education idle is most of the year.</li>



<li><strong>Assuming a guardrail feature does what its name implies.</strong> Read the supported use cases. Contextual grounding on a chatbot is the classic example.</li>



<li><strong>Piloting with one enthusiastic lecturer.</strong> That lecturer&#8217;s material is unusually well organized. The second department will not be.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Build the retrieval filter in exactly one function, and make the tenant and visibility clauses impossible to omit.</li>



<li>Keep a small regression set of adversarial queries, including students asking for solutions and asking about classmates, and run it on every deployment.</li>



<li>Show citations in the interface. Instructors trust the system faster when they can see the source, and it converts silent hallucinations into reportable ones.</li>



<li>Give instructors a preview mode that answers as a student would see it. They will find scoping errors far faster than any test suite.</li>



<li>Route the tool endpoint through a CDN and WAF such as Cloudflare, with rate limiting per session. Assessment week is a load test you did not schedule.</li>



<li>Dashboard retrieval count, token spend per course and passback failure rate together. CloudWatch metrics fronted by Amazon Managed Grafana or Grafana Cloud gives faculty a view without handing out console access.</li>



<li>Have an escalation path for distressing content that reaches a human being, and make sure the people on it know they are on it. This is a policy question you must answer before launch, not after.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do we need to fine-tune a model for our curriculum?</h3>



<p class="wp-block-paragraph">Almost certainly not. Curriculum content changes every term, and retrieval updates in minutes where fine-tuning is a rebuild. Fine-tuning earns its place for style and format consistency, not for facts. Start with retrieval, and revisit only if you have a specific behavior that prompting genuinely cannot produce.</p>



<h3 class="wp-block-heading">How do we stop it from just doing the homework?</h3>



<p class="wp-block-paragraph">Partly retrieval scope, partly prompting, partly accepting you cannot fully win. Exclude solutions and rubrics from student-visible retrieval, prompt for Socratic responses on assessed material, flag the assessment context on the way in so the assistant knows which mode it is in. A determined student can paste the question into a consumer chatbot regardless, which is a course design question rather than an infrastructure one.</p>



<h3 class="wp-block-heading">Is Bedrock the right choice versus calling a model API directly?</h3>



<p class="wp-block-paragraph">If the institution is already on AWS, Bedrock keeps the traffic inside your account boundary, gives you IAM and CloudTrail on the same footing as everything else, and can be reached over VPC endpoints. A direct provider API is often simpler and sometimes gets new models sooner. The deciding factor is usually procurement: one existing AWS agreement is a much shorter conversation than a new vendor review.</p>



<h3 class="wp-block-heading">Does student data get used to train the model?</h3>



<p class="wp-block-paragraph">AWS states that Bedrock does not use customer inputs or outputs to train foundation models. Verify it against current AWS documentation and your agreement rather than any third-party summary, because this is exactly the clause a district will ask you to evidence in writing.</p>



<h3 class="wp-block-heading">What does it cost to run a personalized learning assistant on AWS?</h3>



<p class="wp-block-paragraph">Model your bill as three parts: a vector store floor that bills whether or not anyone asks anything, a one-off embedding cost when you ingest the corpus, and per-turn inference dominated by retrieved context rather than by the question. For a small corpus with a term-shaped traffic curve, the floor usually dominates, which is why the vector store choice matters more than the model choice.</p>



<h3 class="wp-block-heading">How do we know whether it actually helps anyone learn?</h3>



<p class="wp-block-paragraph">Decide the measure before launch and instrument for it. Usage is not learning. Pair the assistant&#8217;s logs with assessment outcomes on comparable cohorts and accept that a clean answer needs a study design, not a dashboard. Engineering can supply the data; it cannot supply the conclusion.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A personalized learning assistant on AWS will never tell you when it is wrong about who someone is. The model has no concept of enrollment, and retrieval has no concept of a due date. Both live entirely in the metadata you attach at ingestion and the filter you build server-side from a validated launch token. Get that right and what remains is ordinary engineering: chunking, cost curves, dashboards. Get it wrong and you ship something that works flawlessly right up until the week it embarrasses the institution.</p>



<p class="wp-block-paragraph">Build the boundary first. Make it a control, not an instruction. Then go be clever with the tutoring.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand building or auditing one of these?</h2>



<p class="wp-block-paragraph">I work on the infrastructure side of AI assistants for education, which mostly means the parts that decide whether the thing is safe to put in front of students:</p>



<ul class="wp-block-list">
<li>Designing and reviewing the LTI 1.3 launch path, roster access and grade passback, including the cross-site cookie and registration issues that cause blank-iframe launches.</li>



<li>Building the retrieval isolation layer: sidecar metadata schemas, server-side filter composition, and an adversarial regression suite that proves instructor material stays invisible to students.</li>



<li>Choosing and migrating the vector store when the OpenSearch Serverless floor is eating a pilot budget, including Aurora pgvector and S3 Vectors options.</li>



<li>Guardrail configuration that matches the documented behavior rather than the feature name, plus a logging and retention design you can defend in a procurement review.</li>



<li>Cost allocation tagging by course and faculty, with dashboards that answer &#8220;which department is spending this&#8221; without handing out AWS console access.</li>



<li>Infrastructure as code for the whole stack so a second institution is a variable file rather than a fortnight.</li>
</ul>



<p class="wp-block-paragraph">If you have something concrete, send it over: a retrieval filter you are not sure is airtight, an ingestion job that keeps skipping documents, a failing LTI launch log, or a Cost Explorer breakdown that stopped making sense. Easier to look at the real thing than to talk in the abstract.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/personalized-learning-assistant-aws/">Building a Personalized Learning Assistant on AWS Without Leaking Student Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>CV Parsing at Scale on AWS: Textract, Bedrock, and the Bias You Have to Document</title>
		<link>https://john-nessime.com/blog/cloud-computing/cv-parsing-aws-textract-bedrock-bias/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 30 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Adverse Impact]]></category>
		<category><![CDATA[Algorithmic Bias]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[EU AI Act]]></category>
		<category><![CDATA[HR Technology]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Local Law 144]]></category>
		<category><![CDATA[Model Invocation Logging]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Resume Parsing]]></category>
		<category><![CDATA[Structured Outputs]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=540</guid>

					<description><![CDATA[<p>A CV parsing pipeline stops being a parser the moment a score sorts a list. Where Textract and Bedrock actually fit, why redacting names does nothing for bias, and what to document before anyone asks you to explain a ranking.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cv-parsing-aws-textract-bedrock-bias/">CV Parsing at Scale on AWS: Textract, Bedrock, and the Bias You Have to Document</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;parser is working fine.&#8221; It was. Extraction confidence was high, the dead letter queue was empty, and recruiting liked the shortlists. Then someone in legal asked why candidate 4,412 ranked below candidate 118.</p>



<p class="wp-block-paragraph">Nobody could answer. Not because anything was broken, but because the system had quietly stopped being a parser somewhere between the S3 upload and the sorted list on a recruiter&#8217;s screen. It had become a selection procedure, and nothing in it was designed to explain itself.</p>



<p class="wp-block-paragraph">That is the failure mode worth leading with, because it never pages you. Your alarms stay green while the thing you built lands in a category of regulated system you never intended to build.</p>



<p class="wp-block-paragraph">This post covers CV parsing on AWS as it actually goes: what Amazon Textract gives you and what it doesn&#8217;t, where Bedrock belongs and where it becomes a liability, the bias problem you have to document rather than solve, and the failures that only surface at volume.</p>



<h2 class="wp-block-heading">When CV parsing on AWS becomes a hiring decision</h2>



<p class="wp-block-paragraph">Most of these pipelines start honestly. Someone wants structured JSON out of a PDF pile so the ATS populates fields instead of a coordinator retyping them. Extraction is a data problem with clean success criteria.</p>



<p class="wp-block-paragraph">Then a product manager asks for a match score. It seems small. You already have the skills list and the job description, so you ask a model how well they line up and emit a number. The API contract barely changes.</p>



<p class="wp-block-paragraph">What changed is everything. The moment a recruiter sorts by that number and works down from the top, the pipeline substantially assists a hiring decision. Under New York City&#8217;s Local Law 144 that is an automated employment decision tool, and the obligations attach to the employer using it, not the vendor who wrote the code. Under the EU AI Act, Annex III point 4 lists systems intended to analyse and filter job applications and evaluate candidates as high-risk by classification, not by how confident you are in your prompt.</p>



<p class="wp-block-paragraph">The EU date is worth stating plainly because it moved. Annex III obligations were originally due to apply from 2 August 2026; the Digital Omnibus package pushed stand-alone Annex III systems out to 2 December 2027. That is a reprieve on paperwork, not on design. Systems procured now will still be running then, and the separate prohibition on workplace emotion-recognition AI, plus existing GDPR duties around automated decisions, were never on that timetable.</p>



<p class="wp-block-paragraph">So the first architectural decision isn&#8217;t about services. It&#8217;s about where extraction stops and judgement begins, and whether that boundary is visible in the code or buried in a prompt.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Getting text out: what Textract actually does</h2>



<p class="wp-block-paragraph">Textract is good at CVs in a narrow way. It reads printed and handwritten text, detects tables and form-style key-value pairs, and with the Layout feature type it identifies structural elements such as titles, paragraphs, lists, headers and footers. For a two-column CV with a skills sidebar, layout awareness is the difference between usable text and word salad.</p>



<h3 class="wp-block-heading">Sync versus async is not a performance choice</h3>



<p class="wp-block-paragraph"><code>AnalyzeDocument</code> is synchronous and takes a single image or a document passed as bytes or an S3 object. <code>StartDocumentAnalysis</code> is the asynchronous entry point for multipage documents; it publishes completion status to an SNS topic you drain from SQS before calling <code>GetDocumentAnalysis</code> with the returned job ID.</p>



<p class="wp-block-paragraph">People pick sync because it&#8217;s easier to reason about, then discover half of real CVs are three pages. Go async from the start. The SNS-to-SQS hop isn&#8217;t overhead, it&#8217;s the only thing that gives you a retry surface when a job fails partway through a batch.</p>



<h3 class="wp-block-heading">Queries beat walking the block graph by hand</h3>



<p class="wp-block-paragraph">The Queries feature lets you ask natural-language questions of a document without knowing where the data sits on the page. Pass <code>QUERIES</code> in <code>FeatureTypes</code>, supply a <code>QueriesConfig</code>, then read the <code>QUERY_RESULT</code> blocks out of the response.</p>



<pre class="wp-block-code"><code>aws textract start-document-analysis 
  --document '{"S3Object":{"Bucket":"bucket","Name":"document"}}' 
  --feature-types '["QUERIES"]' 
  --queries-config '{"Queries":[{"Text":"Question"}]}'</code></pre>



<p class="wp-block-paragraph">Each query can carry an alias so answers map back to your schema without positional guessing. In Python, walk the returned blocks and pick the ones whose <code>BlockType</code> is <code>QUERY_RESULT</code>, each carrying answer text and a confidence score.</p>



<pre class="wp-block-code"><code>response = client.analyze_document(
    Document={'S3Object': {'Bucket': bucket, 'Name': document}},
    FeatureTypes=["TABLES", "FORMS", "QUERIES"],
    QueriesConfig={'Queries': [{'Text': question}]}
)

for block in response['Blocks']:
    if block["BlockType"] == "QUERY_RESULT":
        print(block["Text"])</code></pre>



<p class="wp-block-paragraph">Use Queries for facts that exist verbatim on the page: employer name, job title, dates, certification number. Not for anything requiring inference. &#8220;What is this person&#8217;s seniority level?&#8221; is a judgement, and pushing it through an extraction API hides that judgement inside something that looks like OCR.</p>



<h2 class="wp-block-heading">Where Bedrock belongs, and where it stops</h2>



<p class="wp-block-paragraph">Bedrock earns its place in one job: turning messy extracted text into a consistent schema. CVs are the worst kind of semi-structured document because the structure is a fashion trend. One person lists dates as &#8220;Mar 2019 to present,&#8221; another renders them as a bar chart. Normalising that is exactly the fuzzy transformation a language model handles well, and one you can verify.</p>



<ol class="wp-block-list">
<li>Constrain output to a schema and reject anything that fails validation. A job title you didn&#8217;t ask for, or an employment date absent from the source text, is a failed record rather than a creative flourish.</li>



<li>Turn on model invocation logging before you process a single real CV. It&#8217;s disabled by default, and it&#8217;s the only native record of what you sent and what came back.</li>
</ol>



<p class="wp-block-paragraph">That second point isn&#8217;t a nice-to-have. When someone asks a year from now why a candidate scored the way they did, the invocation log is your answer or you don&#8217;t have one.</p>



<pre class="wp-block-code"><code>aws bedrock put-model-invocation-logging-configuration 
  --logging-config '{
    "s3Config": {"bucketName": "my-bedrock-logs", "keyPrefix": "cv-pipeline/"},
    "textDataDeliveryEnabled": true
  }'</code></pre>



<p class="wp-block-paragraph">Logs go to CloudWatch Logs, S3, or both, and only destinations in the same account and Region are supported. Apply lifecycle rules and encryption up front, because these logs contain full CVs and they outlive the hiring round that produced them.</p>



<p class="wp-block-paragraph">Bedrock Guardrails fits here through sensitive information filters, which detect PII in prompts and responses and either block the payload or mask individual entities with identifier tags. There&#8217;s a detect mode that reports what a guardrail would have caught without acting, which is the right way to tune against real documents first. Don&#8217;t oversell it to yourself: masking is not anonymisation, and a filter firing on <code>NAME</code> and <code>EMAIL</code> does very little about bias, for reasons that are the whole point of the next section.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The bias problem you must document, not solve</h2>



<p class="wp-block-paragraph">Every engineer&#8217;s instinct is to strip the protected attributes and declare it handled. Remove the name, the photo, the address, and the model can&#8217;t discriminate on what it can&#8217;t see. It doesn&#8217;t work, and understanding why separates a pipeline you can defend from one you can&#8217;t.</p>



<h3 class="wp-block-heading">Proxies survive redaction</h3>



<p class="wp-block-paragraph">A CV is dense with correlated signal. University name tracks geography and socioeconomics. Graduation year is an age proxy that survives every name redaction you can write. Employment gaps correlate with caregiving. Postcode correlates with almost everything. Add professional association membership, the language a qualification is written in, whether military service appears, the phrasing conventions of a non-native English speaker.</p>



<p class="wp-block-paragraph">None of those are protected attributes. All of them carry information about protected attributes, and a model optimising for similarity to your existing successful hires will use them. Redaction removes the label, not the signal, and it makes disparity harder to detect because you deleted the field you would have measured against.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">You can&#8217;t prove a screening pipeline is unbiased. You can measure its outcomes, record what you measured, and act when the numbers move.</p>
</blockquote>



<h3 class="wp-block-heading">What the four-fifths rule actually asks for</h3>



<p class="wp-block-paragraph">The Uniform Guidelines on Employee Selection Procedures set out the rule of thumb federal enforcement agencies apply: a selection rate for any race, sex or ethnic group falling below four-fifths of the rate for the highest group is generally treated as evidence of adverse impact. It sits at 29 CFR 1607.4(D).</p>



<ul class="wp-block-list">
<li><strong>It&#8217;s a rule of thumb, not a safe harbour.</strong> The same guidelines note smaller differences can still constitute adverse impact where they&#8217;re significant in statistical and practical terms, and courts have declined to treat clearing 0.8 as proof nothing is wrong.</li>



<li><strong>It applies to the selection procedure.</strong> That means your parser, your scorer and the recruiter&#8217;s sort order taken together. Auditing the model in isolation measures the wrong thing.</li>
</ul>



<p class="wp-block-paragraph">The computation is trivial, which is why there&#8217;s no excuse for skipping it.</p>



<pre class="wp-block-code"><code>SELECT
  demographic_group,
  COUNT(*) AS applicants,
  SUM(CASE WHEN advanced THEN 1 ELSE 0 END) AS selected,
  1.0 * SUM(CASE WHEN advanced THEN 1 ELSE 0 END) / COUNT(*) AS selection_rate
FROM screening_outcomes
WHERE requisition_id = :req
GROUP BY demographic_group;</code></pre>



<p class="wp-block-paragraph">Divide each selection rate by the largest and you have the impact ratio. Anything under 0.8 is a signal to stop and look, not a number to explain away.</p>



<h3 class="wp-block-heading">Where the demographic data comes from</h3>



<p class="wp-block-paragraph">This is the genuinely hard part and where most teams stall. You need demographic data to measure disparity, and you must not feed it into the screening decision. The pattern that works is a strict split: self-reported responses collected separately at application time, stored in a different bucket under a different KMS key, joined to outcomes only in an analysis account the scoring pipeline has no path to.</p>



<p class="wp-block-paragraph">NYC&#8217;s regime allows historical or test data, and requires the audit be performed by someone not involved in using or developing the tool. That independence requirement is easy to miss when you&#8217;re the consultant who built the thing. You can build the measurement harness. You can&#8217;t be the independent auditor of your own pipeline.</p>



<h3 class="wp-block-heading">Document the pipeline, not the model</h3>



<p class="wp-block-paragraph">The artifact that saves you is boring and written down. At minimum: which model and version produced scores in each date range, the scoring logic in force, every field that reached the scorer, every field withheld and why, where human review sits and what a reviewer can override, and the impact ratios from each measurement run.</p>



<p class="wp-block-paragraph">Keep it in version control beside the infrastructure code. A change to the prompt is a change to the selection procedure and should be reviewable as one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Problems that only appear at volume</h2>



<ul class="wp-block-list">
<li><strong>Throttling arrives as data loss.</strong> Without retries using exponential backoff and jitter, a throttled call becomes a missing candidate rather than an error anyone sees. Assert on record counts at every stage boundary.</li>



<li><strong>Duplicates are the norm.</strong> The same person applies to three roles with two slightly different files. Deduplicate on content hash before you spend on extraction, and keep an idempotency key so a retried job doesn&#8217;t create a second candidate record.</li>



<li><strong>Cost is dominated by re-runs.</strong> Extraction is the expensive step and it&#8217;s deterministic on the same input. Store the raw Textract response in S3 keyed by document hash. Reprocessing after a schema change should cost storage, not inference.</li>



<li><strong>Retention outlives the requisition.</strong> Extracted text, invocation logs and intermediate JSON all contain personal data and all quietly accumulate. Set lifecycle rules at build time, not after the first data subject request.</li>



<li><strong>Scanned CVs are a different pipeline.</strong> Photographed documents behave nothing like exported PDFs. Route them separately with lower confidence thresholds and higher review rates.</li>
</ul>



<p class="wp-block-paragraph">On cost visibility, native tooling shows the shape of spend but is slow to attribute it. If this runs alongside other AI workloads, a dedicated platform such as Vantage or CloudZero makes per-pipeline attribution less painful than slicing Cost Explorer by tag every month. And if the upload front door is public, putting Cloudflare in front of it deals with bot traffic before it becomes Textract spend.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>High confidence, wrong fields.</strong> Almost always multi-column layouts read in the wrong order. Add the Layout feature type and check reading order in the block graph against what a human sees.</li>



<li><strong>Async jobs complete but results never arrive.</strong> Check the SNS topic policy allows publishing to your queue and that the role passed to Textract can publish. This fails silently and looks like a stuck pipeline.</li>



<li><strong>The model invents employment history.</strong> Your prompt is asking it to summarise rather than extract. Constrain the schema, require every value trace to source text, and fail validation instead of accepting a plausible answer.</li>



<li><strong>Scores shift after a model update.</strong> Pin the model identifier rather than tracking a moving alias, and re-run your measurement set against a held-out corpus before promoting any change.</li>



<li><strong>Guardrails block legitimate CVs.</strong> Sensitive information filters are probabilistic and context-dependent. Run detect mode against a real corpus and read the traces before switching to block.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a match score as a data field rather than a decision, and skipping the governance that follows.</li>



<li>Redacting names and considering the bias question closed.</li>



<li>Enabling invocation logging after the pilot, leaving the first thousand decisions with no record.</li>



<li>Building human review that only sees the shortlist, which reviews the output of the bias rather than the bias.</li>



<li>Assuming a US-headquartered employer is out of scope. Both regimes reach based on where the candidate is, not where the company is.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Draw a hard line between extraction and scoring: separate services, separate IAM roles, separate logs.</li>



<li>Enable model invocation logging before the first real document, with encryption and lifecycle rules already applied.</li>



<li>Cache raw extraction output keyed by document hash so reprocessing never re-bills inference.</li>



<li>Collect demographic data separately, key it differently, and join it to outcomes only in an isolated analysis account.</li>



<li>Run impact ratios on a schedule rather than on request, and alert on movement instead of waiting for an audit.</li>



<li>Version prompts and scoring logic in the same repository as the infrastructure, reviewed as changes to a selection procedure.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Textract or Bedrock the right tool for CV parsing?</h3>



<p class="wp-block-paragraph">Both, for different halves. Textract turns pixels into text with layout awareness and confidence scores. Bedrock turns that text into a consistent schema. Using a language model for the OCR step gives up per-field confidence, which is exactly the signal you need to decide what goes to human review.</p>



<h3 class="wp-block-heading">Does removing names and photos make a resume screening pipeline compliant?</h3>



<p class="wp-block-paragraph">No. Redaction handles direct identifiers while leaving proxies such as graduation year, institution, postcode and employment gaps intact. It also removes the fields you&#8217;d need to measure disparity, so it can make a pipeline harder to audit while feeling safer.</p>



<h3 class="wp-block-heading">Do these rules apply if my company isn&#8217;t in New York or the EU?</h3>



<p class="wp-block-paragraph">Possibly. Local Law 144 turns on whether the candidate resides in New York City, including for remote roles, rather than where the employer sits. The EU AI Act has similar extraterritorial reach. Check with counsel, not with an architecture diagram.</p>



<h3 class="wp-block-heading">Can I run the bias audit myself?</h3>



<p class="wp-block-paragraph">You can build the measurement pipeline and run it continuously, and you should. The formal NYC bias audit has to be performed by an independent party not involved in using or developing the tool, which rules out the team that built it.</p>



<h3 class="wp-block-heading">How do I keep extraction costs predictable at scale?</h3>



<p class="wp-block-paragraph">Deduplicate before extraction, cache raw responses by content hash, and choose feature types deliberately. Requesting tables, forms and queries on every document when you only need queries multiplies cost for nothing. Both services bill per unit of work, so the lever is doing less work rather than finding a cheaper tier.</p>



<h3 class="wp-block-heading">Should the pipeline auto-reject candidates?</h3>



<p class="wp-block-paragraph">Treat that as a question for counsel, not engineering. GDPR restricts solely automated decisions with significant effects on individuals, and the practical answer in most designs is a meaningful human review before any rejection. Build the human step in early; retrofitting it is painful.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">CV parsing on AWS is two systems wearing one name. The extraction half is an engineering problem with clean success criteria, and Textract plus Bedrock will carry you a long way given retries, caching and schema validation. The scoring half is a selection procedure, governed by rules that don&#8217;t care what you called the service.</p>



<p class="wp-block-paragraph">You won&#8217;t eliminate bias from a system trained on hiring history. What you can do is know where the boundary sits, log what crossed it, measure outcomes on a schedule, and be able to answer the question about candidate 4,412 when it arrives. Build the documentation while you build the pipeline. Reconstructing it later from CloudWatch and memory is a much worse job.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Work with me on document pipelines and AI governance</h2>



<p class="wp-block-paragraph">Most of the work here isn&#8217;t the extraction. It&#8217;s the boundary drawing, the evidence trail and the parts that only break at volume. Things I help with:</p>



<ul class="wp-block-list">
<li>Designing Textract and Bedrock pipelines with an auditable split between extraction and any scoring or ranking step.</li>



<li>Building the measurement harness that computes selection rates and impact ratios on a schedule, with demographic data isolated from the scoring path.</li>



<li>Setting up model invocation logging, retention, encryption and access controls so the evidence trail exists before you need it.</li>



<li>Fixing throughput and cost problems: throttling, deduplication, extraction caching and reprocessing that doesn&#8217;t re-bill inference.</li>



<li>Adding human-in-the-loop review that routes on confidence and score bands, including sampled rejections rather than shortlists only.</li>



<li>Writing the technical documentation an independent auditor or your legal team will actually ask for.</li>
</ul>



<p class="wp-block-paragraph">Send me a sample Textract response, a scoring prompt or an architecture sketch and I&#8217;ll tell you where the boundary is currently blurred.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cv-parsing-aws-textract-bedrock-bias/">CV Parsing at Scale on AWS: Textract, Bedrock, and the Bias You Have to Document</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</title>
		<link>https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Comprehend]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=272</guid>

					<description><![CDATA[<p>A confident answer with a citation from a matter the reader was walled off from. Nothing crashed, nothing alerted. Building legal document intelligence on AWS with Textract, Bedrock and OpenSearch is the easy half; the hard half is tenant isolation, verified deletion, S3 Object Lock holds, what leaves the account, and an audit trail that names the actual user.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo was going well until someone asked where the third citation came from.</p>



<p class="wp-block-paragraph">The answer on screen was good. Fluent, specific, correctly hedged, three sources listed underneath it. Then a partner in the room asked a simple question: which matter is source three from? Nobody could answer it in the room, and when we went and looked, it was from a matter that half the people watching the demo were formally walled off from.</p>



<p class="wp-block-paragraph">Nothing had crashed. No alarm fired. The retrieval layer had done exactly what a similarity search does, which is return the nearest vectors, and the nearest vectors did not care about the ethical wall. That is the thing about building a legal document intelligence platform on AWS: the failures that matter almost never look like failures. They look like a confident answer with a citation attached.</p>



<p class="wp-block-paragraph">This post is about the parts of that build that are hard, organised around the five boundaries that actually have to hold. The extraction pipeline is the easy half. I will cover it, but quickly, because the AWS documentation is good and the failure modes are visible. The other four boundaries fail quietly, and that is where the engineering goes.</p>



<h2 class="wp-block-heading">The shape of the thing</h2>



<p class="wp-block-paragraph">Before the boundaries, the skeleton. A legal document intelligence platform on AWS almost always ends up looking like this, whether you plan it or arrive at it:</p>



<ul class="wp-block-list">
<li>Documents land in Amazon S3, one prefix per matter, versioning on.</li>



<li>An S3 event triggers AWS Lambda, which starts an asynchronous Amazon Textract job.</li>



<li>Textract output lands back in S3 as JSON. Something normalises it into text plus page and bounding-box coordinates.</li>



<li>Amazon Comprehend or a foundation model classifies the document and pulls entities: parties, dates, governing law, clause types.</li>



<li>Chunks get embedded and written to a vector store, usually Amazon OpenSearch Serverless behind Amazon Bedrock Knowledge Bases.</li>



<li>An application layer, typically API Gateway plus Lambda, takes a question, retrieves, and calls a model on Amazon Bedrock.</li>
</ul>



<p class="wp-block-paragraph">AWS Step Functions is worth reaching for once you have more than three stages, because Textract jobs are long-running and Lambda timeouts are not a retry strategy. That is the whole architecture. You can stand it up in a fortnight. Then you spend six months on everything below.</p>



<h2 class="wp-block-heading">Boundary one: isolation, and why metadata filtering is not optional</h2>



<p class="wp-block-paragraph">This is the one from the demo, and it is the one that ends engagements.</p>



<p class="wp-block-paragraph">A vector index has no native concept of a matter, a client, or an ethical wall. If you pool every document into one index and rely on the prompt to keep things separate, you have built a system where a well-phrased question can pull privileged material across a wall. Prompt instructions are not an access control. They are a suggestion that usually works.</p>



<p class="wp-block-paragraph">There are two shapes that do work, and the choice between them is a real trade-off rather than a best practice.</p>



<h3 class="wp-block-heading">Pooled index with server-side filtering</h3>



<p class="wp-block-paragraph">Every chunk carries a <code>matter_id</code> and a <code>client_id</code> as metadata. Every retrieval call attaches a filter. With an S3 data source in Bedrock Knowledge Bases, the metadata comes from a sidecar JSON file that sits next to the document and carries a <code>metadataAttributes</code> object.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "matter_id": "M-4417",
    "client_id": "C-108",
    "doc_type": "engagement_letter",
    "privileged": true
  }
}</code></pre>



<p class="wp-block-paragraph">The retrieval call then narrows the search before the model ever sees a chunk:</p>



<pre class="wp-block-code"><code>"retrievalConfiguration": {
  "vectorSearchConfiguration": {
    "filter": {
      "equals": { "key": "matter_id", "value": "M-4417" }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two rules make this safe, and both are the kind of thing that gets skipped under deadline. First, the filter value is derived on the server from the authenticated caller&#8217;s claims, never accepted from the client. If the browser can send you a <code>matter_id</code>, the browser can send you a different one. Second, the filter is applied by code that no feature request can bypass. A helper that builds every retrieval request, and a code review rule that no other path may call the retrieve API directly.</p>



<p class="wp-block-paragraph">The trap here is ordering. You cannot filter on an attribute the index does not have. Add <code>matter_id</code> after ingestion and every chunk already in the index is invisible to that filter, which means it either returns for everyone or for no one, depending on how your filter is written. Design the metadata schema before the first ingestion run, not after the first demo.</p>



<h3 class="wp-block-heading">Separate collection per client</h3>



<p class="wp-block-paragraph">The heavier option. A separate OpenSearch Serverless collection per client, which buys you a separate AWS KMS key per client, separate index settings, and a failure mode where a bug in the filtering logic cannot reach across clients at all.</p>



<p class="wp-block-paragraph">It costs you sprawl. Every collection is a resource to provision, monitor, patch policy on, and eventually delete. Ingestion jobs run per data source per knowledge base, so freshness guarantees fragment. For a firm with twelve institutional clients this is fine. For a platform onboarding a hundred small clients it becomes the main operational burden of the product.</p>



<p class="wp-block-paragraph">My default is pooled with strict server-side filtering, and per-client collections only where the client&#8217;s own contract demands a dedicated encryption key. If you cannot say out loud which of those two you are running, you are running neither properly.</p>



<h2 class="wp-block-heading">Boundary two: extraction, and the confidence score everyone ignores</h2>



<p class="wp-block-paragraph">Textract returns a confidence score between 0 and 100 with each prediction. AWS is explicit in its own best-practice guidance that applications sensitive to detection errors should enforce a minimum threshold and route anything below it for human scrutiny, and that the right threshold depends entirely on how the output gets used.</p>



<p class="wp-block-paragraph">Legal documents sit at the harsh end of that. A misread date on an archival scan is a curiosity. A misread date on a limitation period is a problem with a name on it.</p>



<p class="wp-block-paragraph">Practical notes from this layer:</p>



<ul class="wp-block-list">
<li>Use the asynchronous operations for anything multipage. <code>StartDocumentAnalysis</code> and <code>GetDocumentAnalysis</code> handle long PDFs and TIFFs; the synchronous <code>AnalyzeDocument</code> path is for single pages and will fight you on a 400-page bundle.</li>



<li>Textract&#8217;s Queries feature earns its place on structured instruments. Instead of extracting everything and grepping, you ask the document a direct question and get a scoped answer with its own confidence.</li>



<li>Keep the page number and bounding box for every chunk. When a lawyer asks where a sentence came from, &#8220;page 14, second column&#8221; is an answer. &#8220;It&#8217;s in the corpus somewhere&#8221; is not, and the platform loses credibility the first time you say it.</li>



<li>Amazon Augmented AI wires low-confidence predictions into a human review queue with a private workforce, so the reviewers are your people rather than an anonymous pool. For privileged material that distinction is the entire point.</li>
</ul>



<p class="wp-block-paragraph">What I would skip early: building a custom entity recognition model. Comprehend supports custom entity recognition and it is genuinely useful, but you need labelled data you do not have yet, and a foundation model with a decent prompt gets you far enough to find out which entities the users actually care about. Train the custom model once the requirement has stopped moving.</p>



<h2 class="wp-block-heading">Boundary three: retention, where deletion is a distributed problem</h2>



<p class="wp-block-paragraph">Here is the failure that will not show up in any test suite you write.</p>



<p class="wp-block-paragraph">A client asks for a document to be removed. Someone deletes the S3 object. The ticket closes. The document is still fully searchable, because its chunks are sitting in the vector index and will stay there until the data source is re-synced. Meanwhile the Textract JSON output is in a second bucket, the extracted text may be in a database, and if model invocation logging is on, whole passages of the document are sitting in CloudWatch Logs or another S3 bucket entirely.</p>



<p class="wp-block-paragraph">One delete, at least four places holding a copy. Write the deletion path as a real workflow with an assertion at the end, not as a single API call:</p>



<ol class="wp-block-list">
<li>Delete the source object and, if versioning is on, its versions.</li>



<li>Delete the derived artefacts: Textract JSON, normalised text, any thumbnails or page images.</li>



<li>Trigger an ingestion job so the knowledge base drops the orphaned chunks.</li>



<li>Confirm removal by querying the index for the document identifier and expecting nothing back.</li>



<li>Deal with the logs, which means either a retention policy short enough to make the problem expire or a deliberate decision, written down, that logs are out of scope.</li>
</ol>



<p class="wp-block-paragraph">Step four is the one people leave out, and it is the only step that actually proves anything.</p>



<h3 class="wp-block-heading">The opposite problem: things that must not be deleted</h3>



<p class="wp-block-paragraph">Legal work has the reverse requirement too. When a matter goes into litigation hold, the documents need to survive an administrator with delete permissions and a bad afternoon.</p>



<p class="wp-block-paragraph">S3 Object Lock is the mechanism, and it has two independent controls that people routinely conflate. A retention period protects an object version until a fixed date, in either governance mode, which privileged users can override, or compliance mode, which nobody can override and where the period cannot be shortened. A legal hold has no date at all. It stays until someone with <code>s3:PutObjectLegalHold</code> explicitly removes it. The two can be active at once, and while either is active the object version cannot be deleted or overwritten.</p>



<pre class="wp-block-code"><code># Place an indefinite hold on a specific object version
aws s3api put-object-legal-hold 
  --bucket matters-archive 
  --key M-4417/exhibit-c.pdf 
  --version-id 3sL7f2Qz9pXvB1kR 
  --legal-hold Status=ON</code></pre>



<p class="wp-block-paragraph">Two things bite here. Object Lock requires versioning and turns it on automatically, and once enabled on a bucket you cannot turn it off or suspend versioning again. And compliance mode is genuinely permanent: if you set a seven-year retention when you meant seven days, that is the answer, for everyone, including the account root. Test in governance mode first. The bypass path exists and is deliberately awkward, requiring both the <code>s3:BypassGovernanceRetention</code> permission and an explicit <code>x-amz-bypass-governance-retention:true</code> header on the request, which is exactly the level of friction you want on that operation.</p>



<p class="wp-block-paragraph">Design the hold model before you design the deletion model, because holds win. A deletion request that collides with an active hold is a legal question, not an engineering one, and the platform&#8217;s job is to surface the collision clearly rather than resolve it silently.</p>



<h2 class="wp-block-heading">Boundary four: disclosure, or what leaves the account</h2>



<p class="wp-block-paragraph">This is the boundary the client&#8217;s general counsel will ask about, usually in writing, usually before signature. Three separate controls, and they are not interchangeable.</p>



<h3 class="wp-block-heading">The AI services opt-out policy</h3>



<p class="wp-block-paragraph">AWS Organizations has a governance policy type that opts your accounts out of having content processed by certain AI services stored and used for service improvement. The list includes Amazon Textract and Amazon Comprehend, which are precisely the two doing the reading in this architecture. The policy type has to be enabled at the organisation root before you can attach anything:</p>



<pre class="wp-block-code"><code>aws organizations enable-policy-type 
  --root-id r-example 
  --policy-type AISERVICES_OPT_OUT_POLICY</code></pre>



<p class="wp-block-paragraph">Read the AWS documentation on this one carefully rather than taking my summary as gospel, because there is a caveat in it that matters: the services may still need to store your data operationally even when you have opted out of it being used for improvement. Opting out is not the same as the data never existing outside your account. Say that plainly in the client conversation. It is a much better position than being asked about it later.</p>



<h3 class="wp-block-heading">Bedrock&#8217;s data position, and the log that undoes it</h3>



<p class="wp-block-paragraph">Amazon Bedrock&#8217;s published position is that inputs and outputs are not shared with third-party model providers and are not used to train the base models, and that fine-tuning operates on a private copy. That is a strong starting point for privileged content and it is the main reason Bedrock rather than a direct provider API shows up in these builds.</p>



<p class="wp-block-paragraph">Then there is model invocation logging. It captures full prompt and response payloads to CloudWatch Logs or S3, and you want it on, because without it you cannot reconstruct what the system told someone. But understand what you have just built: a log that contains privileged document text, at a per-region setting, in a destination that probably has looser access controls than the document store it came from. Encrypt it with the same KMS key discipline, restrict it harder than you think you need to, and set a retention period on purpose.</p>



<p class="wp-block-paragraph">Worth separating in your head: CloudTrail records that an API call happened and who made it. Model invocation logging records what was in it. You need both, for different questions, and only one of them contains client confidences.</p>



<h3 class="wp-block-heading">Network path</h3>



<p class="wp-block-paragraph">Interface VPC endpoints via AWS PrivateLink keep traffic to Bedrock, Textract and the rest on the AWS network rather than out through an internet gateway or NAT. Add a gateway endpoint for S3 while you are there.</p>



<p class="wp-block-paragraph">Endpoint policies are the part that gets forgotten. An endpoint without a policy is a private path to the whole service, including into accounts that are not yours. Scope it to the operations and resources you actually use, and pair it with an S3 bucket policy that rejects requests not arriving through your endpoint. The first protects what leaves; the second protects what can be reached.</p>



<h3 class="wp-block-heading">The parts that are not on the architecture diagram</h3>



<p class="wp-block-paragraph">Two of these have caught people out badly, and neither is an AWS control.</p>



<p class="wp-block-paragraph">The first is the development environment. You do not want real client documents on a laptop or in a scratch account, so build the parsing and chunking code against synthetic documents on a small separate box. A cheap VPS from a provider like Contabo or InterServer is fine for this, and the separation is worth more than the convenience of iterating in the production account.</p>



<p class="wp-block-paragraph">The second is the reviewers. If your human review loop involves contractors working remotely, their network path and their disks are part of your boundary whether or not you drew them. A managed VPN such as NordVPN or Surfshark handles the network side, and when a matter closes and local copies have to go, a dedicated erasure tool from something like O&amp;O Software does what dragging a folder to the bin does not. Unglamorous, and it is the layer that gets audited.</p>



<h2 class="wp-block-heading">Boundary five: evidence, because someone will ask</h2>



<p class="wp-block-paragraph">At some point the question stops being &#8220;does it work&#8221; and becomes &#8220;who saw what, and when&#8221;. If you cannot answer that from stored records, the platform is not defensible regardless of how good the answers are.</p>



<ul class="wp-block-list">
<li>Turn on CloudTrail data events for the document buckets. They are off by default, they are billed separately from management events, and they are the only way to see individual object-level reads.</li>



<li>Log the retrieval, not just the generation. Store which chunks came back and which filter was applied. When somebody asks whether the wall held on a specific query, this record is the answer and there is no reconstructing it later.</li>



<li>Carry the end user&#8217;s identity through to the audit record. A Lambda execution role in the logs tells you the platform did something. It does not tell you who asked.</li>



<li>Alarm on the filter, not just on errors. A retrieval that ran without a matter filter should page someone, even though it returned HTTP 200 and a perfectly good answer.</li>
</ul>



<p class="wp-block-paragraph">That last one is the closest thing to a single takeaway in this post. The dangerous events in this architecture are successful ones.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the things that will go wrong</h2>



<h3 class="wp-block-heading">Retrieval returns nothing after you add a filter</h3>



<p class="wp-block-paragraph">Nearly always a metadata problem rather than a query problem. Either the sidecar metadata file was missing at ingestion, so the chunks carry no attribute to match, or the attribute was added after the chunks were indexed. Check whether the attribute exists on a known chunk before you touch the query. If it does not, you need a re-sync, not a different filter expression.</p>



<h3 class="wp-block-heading">Textract returns text but the layout is scrambled</h3>



<p class="wp-block-paragraph">Common on two-column pleadings and on documents with headers and footers on every page. Raw text detection reads in a reading order that is not always yours. Use the layout and table features rather than plain detection, and hold onto the geometry so you can reassemble columns yourself if the default order is wrong for that document class.</p>



<h3 class="wp-block-heading">The model cites a document that does not exist</h3>



<p class="wp-block-paragraph">Usually the citation is being generated rather than passed through. If document names are being written by the model instead of read from the retrieval response, it will invent plausible ones. Build citations from the retrieval result metadata in your application code and never from the generated text. Bedrock Guardrails helps with contextual grounding, but the structural fix is not asking the model to produce the reference in the first place.</p>



<h3 class="wp-block-heading">Ingestion is slow and nobody knows where</h3>



<p class="wp-block-paragraph">Instrument per stage before you optimise anything. In most of these pipelines the time is in Textract for large scanned bundles, and the fix is parallelism at the document level rather than tuning the extraction itself. Step Functions with a distributed map over documents gets you there without a queue you have to babysit.</p>



<h3 class="wp-block-heading">Costs climb faster than volume</h3>



<p class="wp-block-paragraph">Look at re-processing first. A pipeline that re-extracts a document every time anything downstream changes will quietly multiply your Textract bill against a static corpus. Key the extraction cache on the object&#8217;s content hash, not its path, and make re-ingestion an explicit action. The billing mechanisms differ per service, so read the current pricing pages rather than trusting a number from a blog post.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating the system prompt as an access control.</strong> It is a formatting instruction that happens to look like a rule.</li>



<li><strong>Taking the tenant identifier from the request body.</strong> If the client can name the matter, the client can name someone else&#8217;s.</li>



<li><strong>Designing the metadata schema after the first ingestion run.</strong> Retrofitting a filterable attribute means re-indexing the whole corpus.</li>



<li><strong>Assuming a deleted S3 object is gone from the platform.</strong> The chunks, the extraction output and the invocation logs all outlive it.</li>



<li><strong>Setting compliance-mode retention without testing in governance mode.</strong> There is no support ticket that fixes a seven-year mistake.</li>



<li><strong>Enabling model invocation logging without treating the log as privileged.</strong> You have just made a second copy of the documents in a less protected place.</li>



<li><strong>Building VPC endpoints and leaving the default policy on them.</strong> A private path with no policy is still a path.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>One retrieval helper, server-side filter injection, and a review rule that no other code path calls the retrieve API directly.</li>



<li>Metadata schema agreed and frozen before the first production ingestion. Include the fields you might filter on later, even if unused today.</li>



<li>Page-level provenance on every chunk, surfaced in the interface. It builds trust faster than any accuracy improvement.</li>



<li>A confidence threshold chosen per document class, with a human review path for everything under it.</li>



<li>Deletion implemented as a verified workflow across every store, ending in an assertion that the content is actually unretrievable.</li>



<li>Object Lock legal holds for matters under litigation hold, tested in governance mode before anything runs in compliance mode.</li>



<li>Customer-managed KMS keys across S3, the vector store and the logs, so key access is a control you can actually revoke.</li>



<li>An alarm on unfiltered retrievals, because the failure mode is a success response.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a legal document intelligence platform on AWS safe for privileged material?</h3>



<p class="wp-block-paragraph">The services support it. Bedrock&#8217;s stated position is that inputs and outputs are not shared with model providers or used to train base models, PrivateLink keeps traffic off the public internet, and KMS covers encryption at rest. What determines safety is your own configuration: tenant isolation, log handling and deletion behaviour. The platform is exactly as confidential as its weakest copy of the text.</p>



<h3 class="wp-block-heading">Should I use Bedrock Knowledge Bases or build retrieval myself?</h3>



<p class="wp-block-paragraph">Start with Knowledge Bases. It handles chunking, embedding, sync and metadata filtering, and those are weeks of work with no differentiation in them. Build your own when you need retrieval behaviour it does not expose, such as unusual re-ranking or per-tenant chunking strategies. Going custom on day one usually means reimplementing the managed service badly.</p>



<h3 class="wp-block-heading">How do I stop one client&#8217;s documents surfacing in another client&#8217;s answers?</h3>



<p class="wp-block-paragraph">Tag every chunk with a tenant identifier at ingestion, and inject the matching filter server-side on every retrieval from the authenticated caller&#8217;s identity. Never accept the identifier from the client. If the contract requires per-client encryption keys, use a separate collection per client instead, accepting the extra operational load.</p>



<h3 class="wp-block-heading">Does deleting a document from S3 remove it from the search index?</h3>



<p class="wp-block-paragraph">No. The chunks stay in the vector store until an ingestion job runs and reconciles the data source. Until then the document is deleted and still fully searchable, which is the worst combination available. Always finish a deletion by querying the index and confirming nothing comes back.</p>



<h3 class="wp-block-heading">What is the difference between an S3 legal hold and a retention period?</h3>



<p class="wp-block-paragraph">A retention period runs until a fixed date and comes in governance mode, which privileged users can override, or compliance mode, which nobody can. A legal hold has no end date and stays until someone explicitly removes it. They are independent, they can both be active on the same object version, and while either is active the object cannot be deleted or overwritten.</p>



<h3 class="wp-block-heading">Do I need human review, or is the model accurate enough?</h3>



<p class="wp-block-paragraph">Accuracy is the wrong frame. The question is what happens to a wrong answer downstream. Where output feeds a decision with consequences, you want a confidence threshold and a review queue, and Amazon Augmented AI with a private workforce gives you that without building the review tooling yourself. Where output is a search aid a person will verify anyway, review is friction you do not need.</p>



<h3 class="wp-block-heading">Can I run this in a single AWS account?</h3>



<p class="wp-block-paragraph">Technically yes, and for a pilot it is reasonable. It gets uncomfortable once you have production documents and a development environment in the same place, because an IAM mistake has nowhere to stop. Separate accounts under Organizations also gives you the policy layer, including the AI services opt-out policy, which only exists at the organisation level.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A legal document intelligence platform on AWS is not hard to build. Textract reads, Bedrock reasons, OpenSearch retrieves, and the tutorial version works on the first afternoon.</p>



<p class="wp-block-paragraph">What is hard is that its failures are silent and well-formed. The cross-matter citation, the deleted document that still answers questions, the privileged passage sitting in a log bucket, the retrieval that ran without a filter and returned a beautiful paragraph. None of these throw an error. All of them are the kind of thing that ends a client relationship.</p>



<p class="wp-block-paragraph">So build the boundaries first and the features second, and make sure every one of them is something you can prove from a stored record rather than something you believe about the code. If you can only take one habit from this: alarm on the successful requests that should not have been possible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your document pipeline?</h2>



<p class="wp-block-paragraph">I work on AWS document processing and retrieval systems where the confidentiality requirements are real. Typical things I get called in for:</p>



<ul class="wp-block-list">
<li>Reviewing tenant isolation in an existing RAG setup and finding the paths where the filter can be bypassed.</li>



<li>Designing the metadata and chunking schema before ingestion, so filtering works without a re-index later.</li>



<li>Building Textract pipelines with confidence thresholds, human review routing and page-level provenance.</li>



<li>Implementing verified deletion across S3, derived artefacts, the vector index and logs, with a proof step at the end.</li>



<li>Setting up retention and legal hold with S3 Object Lock, including the governance-mode rehearsal before compliance mode.</li>



<li>Locking down the network and logging path: VPC endpoints with real policies, KMS key separation, and audit records that name the actual user.</li>
</ul>



<p class="wp-block-paragraph">If you have an architecture diagram, a Step Functions definition or a retrieval request you are unsure about, send it over and I will tell you what I would change and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/legal-document-intelligence-aws/">Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</title>
		<link>https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[AgentCore]]></category>
		<category><![CDATA[Agentforce]]></category>
		<category><![CDATA[AI Agents]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[External Client Apps]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Identity Propagation]]></category>
		<category><![CDATA[MCP]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=211</guid>

					<description><![CDATA[<p>The demo works, then someone sees records they shouldn't. Nothing errors. Here's how to build a Salesforce AI assistant with Amazon Bedrock that respects your sharing model: which doors AWS and Salesforce just closed, why identity propagation is the failure mode that bites, and how to shape the tool surface so the agent can't wander.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/">Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo went fine. Someone from the support team asked the assistant which accounts looked at risk this quarter, and it answered. Good answer, too. Right opportunities, right open cases, sensible summary.</p>



<p class="wp-block-paragraph">Then somebody noticed that the person who asked couldn&#8217;t see half those opportunities in the Salesforce UI. Different role, different branch of the hierarchy. They saw them through the assistant.</p>



<p class="wp-block-paragraph">Nothing errored. No 403, no exception in the logs, no alert. The integration user had access, so the API call succeeded, so the model got the records, so it summarised them. Every layer did exactly what it was configured to do. That is the failure mode that should worry you most when you build a Salesforce AI assistant with Amazon Bedrock: it is silent, it is correct-looking, and you will not find it in a load test.</p>



<p class="wp-block-paragraph">This post covers what actually goes wrong when you wire a large language model into a CRM: which door AWS closed on the entry path, whose permissions your agent is really running under, how to shape the tool surface so the model can&#8217;t wander, and what happens to your org&#8217;s API budget when an agent starts looping. There&#8217;s a build walkthrough, but the build is the easy part.</p>



<h2 class="wp-block-heading">Two doors closed while you were reading the old tutorial</h2>



<p class="wp-block-paragraph">Most of what ranks for this topic was written against an architecture you can no longer build in a fresh account. Both vendors changed the front door within a few months of each other, and the tutorials haven&#8217;t caught up. Check this before you follow anything, including this post.</p>



<h3 class="wp-block-heading">On the AWS side: Bedrock Agents is now Bedrock Agents Classic</h3>



<p class="wp-block-paragraph">AWS renamed the original Bedrock Agents service to Bedrock Agents Classic and put it into maintenance mode. It stopped accepting new customers on 30 July 2026. This one is worth stating with the date attached, because the date is the whole point.</p>



<p class="wp-block-paragraph">The mechanics matter more than the branding:</p>



<ul class="wp-block-list">
<li>Only <code>CreateAgent</code> and <code>InvokeInlineAgent</code> are restricted. Every other API stays available to everyone.</li>

<li>Allowlisting is per AWS account, based on whether that account used Bedrock Agents in the previous twelve months. Multi-account estates get a split result: your old sandbox works, your new production account doesn&#8217;t.</li>

<li>A non-allowlisted account calling <code>CreateAgent</code> gets an <code>AccessDeniedException</code> with HTTP 403, not a quota error. If you&#8217;re debugging IAM and seeing 403s, this is a plausible cause that no policy change will fix.</li>

<li>There is no exception process. AWS documents this plainly.</li>

<li>The model catalogue inside Classic is frozen. Bedrock itself keeps getting new models; the Classic orchestration layer does not.</li>
</ul>



<p class="wp-block-paragraph">The replacement is Amazon Bedrock AgentCore. It splits into a managed harness (config-driven, closest to the old experience) and code-defined agents on the AgentCore runtime (bring Strands, LangGraph, CrewAI, or your own loop). For a Salesforce assistant, the harness is usually enough, and you can move to code later without re-plumbing the tools.</p>



<p class="wp-block-paragraph">Existing Classic agents keep running. There&#8217;s no announced end-of-life. But a frozen model catalogue sets a ceiling on the thing, and if your team can&#8217;t recreate the agent in a new account, your disaster recovery story has a hole in it.</p>



<h3 class="wp-block-heading">On the Salesforce side: Connected Apps gave way to External Client Apps</h3>



<p class="wp-block-paragraph">Salesforce blocked creation of new Connected Apps through both UI and API, with package installation as the exception. External Client Apps are the replacement. Existing Connected Apps keep working and keep their OAuth flows, so this is a wall on new creation rather than a sunset.</p>



<p class="wp-block-paragraph">The practical symptom is that the <em>New Connected App</em> button is greyed out with no explanation. Every AWS-to-Salesforce tutorial that opens with &#8220;first, create a Connected App&#8221; stops working right there.</p>



<p class="wp-block-paragraph">One difference matters for agent work specifically: External Client Apps do not support the username-password OAuth flow at all. If your integration design assumed you&#8217;d stash a service account username and password somewhere and call it a day, that design is gone. Which is a good thing, and it forces the question the next section is about.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: whose permissions is the assistant using?</h2>



<p class="wp-block-paragraph">This is the one from the opening, and it&#8217;s the reason most of these projects should slow down before they ship.</p>



<p class="wp-block-paragraph">Salesforce&#8217;s security model is genuinely good. An OAuth token runs in a user context, and object permissions, field-level security, and sharing rules all still apply to API calls. A token with the <code>api</code> scope does not bypass CRUD, FLS, or sharing. People hear that and relax.</p>



<p class="wp-block-paragraph">The catch is the phrase &#8220;a user context.&#8221; Which user?</p>



<p class="wp-block-paragraph">If you authenticate the gateway to Salesforce with client credentials, every request from every person runs as one integration user. Salesforce enforces that user&#8217;s permissions perfectly. It has no idea a different human asked the question. You&#8217;ve built what&#8217;s sometimes called a multiplexor: one licence fronting many people, with all the real access control now living in your application layer instead of the platform&#8217;s.</p>



<p class="wp-block-paragraph">At that point the sharing model is decoration. A prompt that talks the model into calling a query tool with a wider filter returns data the asker was never entitled to, and Salesforce logs it as a legitimate read by a legitimate user. There&#8217;s nothing to alert on, because nothing went wrong.</p>



<h3 class="wp-block-heading">Two-legged versus three-legged, and when each is honest</h3>



<p class="wp-block-paragraph">AgentCore Gateway supports both client credentials (two-legged) and authorization code (three-legged) flows for reaching downstream targets, and handles ingress and egress authorisation separately. That distinction is the control you need.</p>



<ul class="wp-block-list">
<li><strong>Two-legged is honest when the data is genuinely org-wide.</strong> Product catalogues, published knowledge articles, case deflection content. Nobody&#8217;s entitlement varies, so a single service identity represents the truth.</li>

<li><strong>Three-legged is the only correct answer for record data.</strong> Opportunities, accounts, contacts, custom objects with sharing rules on them. The token is minted for the actual human, Salesforce applies their sharing, and the answer is bounded by what they could have looked up themselves.</li>
</ul>



<p class="wp-block-paragraph">Three-legged costs you something real, and I&#8217;d rather say so than pretend it&#8217;s free. You need per-user consent, token storage, refresh handling, and a way to deal with a user whose token has expired mid-conversation. AgentCore Identity provides a token vault and handles the consent flow, including binding the authorisation URL to the session that started it so a shared link can&#8217;t be completed by someone else. It&#8217;s still more moving parts than a single secret in Secrets Manager.</p>



<p class="wp-block-paragraph">Pay the cost anyway for record data. The alternative is a system whose blast radius is &#8220;everything the integration user can see,&#8221; discovered by a customer or an auditor rather than by you.</p>



<p class="wp-block-paragraph">If you already run Okta, Auth0, or Ping as your identity provider, Gateway can sit in front of them for inbound auth rather than you standing up a parallel Cognito user pool. Reusing the IdP you already audit is worth more here than saving a config file.</p>



<h2 class="wp-block-heading">Failure family two: the shape of the tool surface</h2>



<p class="wp-block-paragraph">The second mistake is giving the model a tool called <code>run_soql</code> that accepts an arbitrary query string. It demos beautifully. Ask anything, get anything.</p>



<p class="wp-block-paragraph">You&#8217;ve now made the model your query planner and your authorisation layer at once. Every safety property of the system depends on prompt text. Guardrails help with content, not with whether a generated <code>WHERE</code> clause happened to be too generous.</p>



<p class="wp-block-paragraph">Narrow tools are the fix, and they&#8217;re boring on purpose. <code>get_open_cases_for_account(account_id)</code> beats <code>run_soql(query)</code> because the query shape is fixed in your code, the model only supplies an identifier, and you can log and rate-limit each tool independently. When something goes wrong you&#8217;re reading a parameter, not parsing SOQL to work out what the model asked for.</p>



<p class="wp-block-paragraph">AgentCore Gateway takes OpenAPI specs, Smithy models, and Lambda functions as tool input types, and exposes them to the agent as MCP tools. It also has one-click integrations for several SaaS products including Salesforce. Practically that gives you three routes:</p>



<ol class="wp-block-list">
<li><strong>The built-in Salesforce integration</strong> for the fastest path to something working. Good for a proof of concept. Look closely at which operations it exposes before you take it further, because breadth is exactly what you don&#8217;t want.</li>

<li><strong>An OpenAPI target</strong> describing a small purpose-built API in front of Salesforce. This is the one I reach for first. You control the surface completely, and the spec doubles as the tool description the model reads.</li>

<li><strong>Lambda targets</strong> when a tool needs to combine several Salesforce calls, or talk to something else at the same time.</li>
</ol>



<p class="wp-block-paragraph">MCP server targets are also supported, with the gateway aggregating tools from multiple targets into a single <code>tools/list</code> response. If you&#8217;re running your own MCP server as the Salesforce facade, it needs a stable HTTPS endpoint the gateway can reach; a small VPS from a provider like InterServer or Hetzner is plenty for the development and staging copies, with the production one behind whatever your standard ingress is.</p>



<h2 class="wp-block-heading">Failure family three: the write path</h2>



<p class="wp-block-paragraph">Reads leak. Writes destroy. They deserve different treatment and usually get the same treatment, which is how you end up with three hundred opportunities silently restaged.</p>



<p class="wp-block-paragraph">Start read-only and stay there longer than feels necessary. When you do add writes, the constraints that actually help:</p>



<ul class="wp-block-list">
<li><strong>One record per call.</strong> No bulk update tool. If the model wants to change twenty records it makes twenty calls, each individually logged and individually rate-limited. Slow is a feature here.</li>

<li><strong>Separate the write identity.</strong> A distinct External Client App and permission set for writes means you can revoke the write path at 2am without taking the assistant offline.</li>

<li><strong>Human confirmation on anything irreversible.</strong> The AgentCore harness supports inline function tools that pause the agent and hand control back to your client code, equivalent to return-of-control. That pause is where a person says yes.</li>

<li><strong>Stamp the source.</strong> A custom field recording that a change came from the assistant, with the session and actor ID, is the difference between a ten-minute rollback and an afternoon of forensics.</li>
</ul>



<p class="wp-block-paragraph">Salesforce field history tracking is on a limited number of fields per object. If the field your assistant writes to isn&#8217;t tracked, you have no before-value. Check that before go-live, not after.</p>



<h2 class="wp-block-heading">Failure family four: what happens under load</h2>



<p class="wp-block-paragraph">Salesforce meters API requests against a daily org-wide allocation that depends on your edition and licence count. Your assistant does not consume that budget the way a nightly sync does.</p>



<p class="wp-block-paragraph">A single conversational turn can be several tool calls. A model that doesn&#8217;t get a clean answer will often retry with a different filter. Ten support agents having ten conversations each becomes a request pattern nobody modelled, and the org-wide limit means the integration that breaks first might be your ETL job, your marketing platform, or your billing sync. Not the assistant. That&#8217;s what makes it hard to diagnose.</p>



<p class="wp-block-paragraph">Three things that help, in order of how much they buy you:</p>



<ul class="wp-block-list">
<li>Cap tool calls per session in your agent config, and return a clear message when the cap is hit rather than letting the loop run.</li>

<li>Cache anything org-wide and slow-moving. Picklist values, record type metadata, product catalogue. These are a surprising share of calls and almost never change mid-day.</li>

<li>Watch API usage as a leading indicator, not a post-mortem. Salesforce exposes remaining daily requests via the REST limits resource. Scrape it on a schedule into CloudWatch, Grafana Cloud, or whatever you already page from, and alert on the trend rather than the wall.</li>
</ul>



<h2 class="wp-block-heading">Wiring it up</h2>



<p class="wp-block-paragraph">With the design decisions settled, the build is short. The AgentCore CLI scaffolds and deploys the harness.</p>



<ol class="wp-block-list">
<li>Create an External Client App in Salesforce with the narrowest OAuth scopes the tools need. Assign it via permission set, not profile.</li>

<li>Build the tool layer: an OpenAPI spec for your purpose-built endpoints, or Lambda functions for composite operations.</li>

<li>Create an AgentCore Gateway, attach the target, and configure the credential provider in AgentCore Identity for the egress side.</li>

<li>Create the harness, point it at the gateway, and deploy.</li>
</ol>



<p class="wp-block-paragraph">Scaffolding a harness and attaching a gateway looks like this. The <code>--gateway-arn</code> value is the gateway you created in step three, and the <code>--type</code> tells the CLI this tool is a gateway rather than one of the built-in tools such as the browser or code interpreter:</p>



<pre class="wp-block-code"><code>agentcore create --name sf-assistant

agentcore add tool --harness sf-assistant 
  --type agentcore_gateway --name sf-gateway 
  --gateway-arn arn:aws:bedrock-agentcore:REGION:ACCOUNT_ID:gateway/GATEWAY_ID</code></pre>



<p class="wp-block-paragraph">Then declare the model and the system prompt, and deploy. Check the current model identifiers in the Bedrock console rather than copying one from any blog post, this one included, because they change:</p>



<pre class="wp-block-code"><code>agentcore add harness 
  --name sf-assistant 
  --model-id YOUR_MODEL_ID 
  --system-prompt "You answer questions about CRM records using only the provided tools. Never guess a record value." 
  --tools sf-gateway

agentcore deploy</code></pre>



<p class="wp-block-paragraph">Invoking it takes a session ID, and an actor ID if you want memory scoped per user. Skipping <code>--actor-id</code> is the memory equivalent of the integration user problem: one shared memory store, everyone&#8217;s context in it.</p>



<pre class="wp-block-code"><code>agentcore invoke --harness sf-assistant 
  --session-id "$(uuidgen)" --actor-id alice 
  "Which of my open opportunities have had no activity in three weeks?"</code></pre>



<p class="wp-block-paragraph">Note &#8220;my&#8221; in that question. With three-legged auth it means something. With an integration user it means nothing, and the model will cheerfully invent a definition.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong><code>AccessDeniedException</code> when creating an agent.</strong> If you&#8217;re calling <code>CreateAgent</code> or <code>InvokeInlineAgent</code> on the Classic API, check whether the account has prior Bedrock Agents usage before you touch IAM. No policy fixes this one. Build on AgentCore instead.</p>



<p class="wp-block-paragraph"><strong>The New Connected App button is greyed out.</strong> Expected. Create an External Client App. If you have a documented technical requirement that only a Connected App satisfies, that&#8217;s a Salesforce Support request, not a setting you can toggle.</p>



<p class="wp-block-paragraph"><strong>Tools appear in <code>tools/list</code> but calls fail.</strong> Gateway can cache tool schemas from a target without holding a working credential for calls. Discovery succeeding proves the schema synced, not that egress auth works. Look at the credential provider.</p>



<p class="wp-block-paragraph"><strong>The assistant returns fewer records than the user expects.</strong> Under three-legged auth this is often correct behaviour that looks like a bug. Have the user run the same query in the Salesforce UI. If the UI shows more, check FLS on the specific fields; if it shows the same, the assistant is right and the user&#8217;s mental model is wrong.</p>



<p class="wp-block-paragraph"><strong>Answers are confidently wrong about record values.</strong> Usually the model answering from training data instead of calling a tool. Tighten the system prompt to forbid answering record questions without a tool result, and check the traces to confirm which turns actually called out.</p>



<p class="wp-block-paragraph"><strong>Everything works, then breaks at the same time daily.</strong> Look at org-wide API consumption before you look at your own code. Something else in the org may be eating the allocation on a schedule.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Using a System Administrator profile for the integration identity because it makes the errors go away.</li>

<li>Exposing a generic SOQL tool and calling the prompt a security control.</li>

<li>Testing only in a sandbox with one user, where the sharing model is invisible because there&#8217;s nothing to share.</li>

<li>Treating Guardrails as an access control layer. They filter content; they don&#8217;t know who&#8217;s asking.</li>

<li>Shipping writes in the first release because the read-only version felt underwhelming in the demo.</li>

<li>Building on Bedrock Agents Classic in a new account without checking allowlist status first.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Use three-legged OAuth for anything governed by sharing rules. Reserve client credentials for genuinely org-wide data.</li>

<li>Give each tool one job and a fixed query shape. The model supplies parameters, never structure.</li>

<li>Separate read and write identities so you can kill one without the other.</li>

<li>Scope memory per actor, not per deployment.</li>

<li>Test with at least three users at different levels of the role hierarchy, and diff the answers. Identical answers to &#8220;show me my pipeline&#8221; is the signal that identity isn&#8217;t propagating.</li>

<li>Keep end-to-end traces and ship them somewhere you already look, so agent behaviour lands in the same place as the rest of your telemetry.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I still use Bedrock Agents for a new Salesforce project?</h3>



<p class="wp-block-paragraph">Only if the AWS account already used Bedrock Agents in the previous twelve months. Accounts without that history get an <code>AccessDeniedException</code> on <code>CreateAgent</code>, and AWS has said there&#8217;s no exception process. For anything new, build on AgentCore.</p>



<h3 class="wp-block-heading">Do I need Agentforce to build a Salesforce AI assistant with Amazon Bedrock?</h3>



<p class="wp-block-paragraph">No. Agentforce is Salesforce&#8217;s own agent layer, and it can call out to AWS, which is useful when you want the assistant to live inside the Salesforce UI and inherit the Einstein Trust Layer. But an AgentCore agent calling Salesforce APIs through a gateway is a complete architecture on its own, and it&#8217;s the better fit when the assistant also needs to reach systems outside Salesforce.</p>



<h3 class="wp-block-heading">Does the model see all my CRM data?</h3>



<p class="wp-block-paragraph">It sees whatever your tools return. That&#8217;s determined by the OAuth identity and the tool definitions, not by the model. This is why the identity decision matters more than the model choice.</p>



<h3 class="wp-block-heading">Will Bedrock Guardrails stop the assistant leaking records?</h3>



<p class="wp-block-paragraph">No, and this is worth being blunt about. Guardrails filter content categories and can mask sensitive data patterns. They have no concept of which user is asking or which records that user is entitled to. A record correctly retrieved under an over-permissioned token passes every guardrail check, because there&#8217;s nothing objectionable about it. Guardrails are a good layer. They are not this layer.</p>



<h3 class="wp-block-heading">How do I keep the assistant from burning the org&#8217;s API limit?</h3>



<p class="wp-block-paragraph">Cap tool calls per session, cache org-wide reference data, and monitor remaining daily requests via the REST limits resource on a schedule. The allocation is org-wide, so the symptom of overuse usually shows up in some other integration first.</p>



<h3 class="wp-block-heading">Should I migrate an existing Bedrock Agents Classic setup?</h3>



<p class="wp-block-paragraph">There&#8217;s no deadline and no announced end-of-life, so there&#8217;s no emergency. The reasons to move are the frozen model catalogue and the fact that new accounts can&#8217;t recreate your agent. If you&#8217;re still actively developing the thing, move. If it&#8217;s stable and finished, it can wait. AWS publishes a capability mapping and a CLI import path for existing configurations.</p>



<h3 class="wp-block-heading">Is an External Client App harder to set up than a Connected App?</h3>



<p class="wp-block-paragraph">Slightly different rather than harder. The main structural change is that developer settings and admin policies are separated, so the app definition and the per-org access controls are configured independently. The one real removal is the username-password flow, which no longer exists in the ECA model.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AI assistant with Amazon Bedrock is not really a model problem. Model quality is the part that will take care of itself, and the part you can change with one config line later.</p>



<p class="wp-block-paragraph">The part that won&#8217;t take care of itself is identity. If the assistant answers as one integration user, you have quietly moved your entire access control model from a platform that enforces it well into a prompt that enforces it not at all, and nothing in your monitoring will tell you. Get that right first. Then argue about models.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or auditing this?</h2>



<p class="wp-block-paragraph">Most of the work on these projects is unglamorous: deciding what the agent is allowed to touch, and proving it can&#8217;t touch anything else. That&#8217;s the part I&#8217;m useful for.</p>



<ul class="wp-block-list">
<li>Auditing an existing Salesforce AI integration for identity propagation gaps, including testing across role hierarchy levels to find silent over-permissioning.</li>

<li>Migrating a Bedrock Agents Classic deployment to AgentCore, including mapping action groups to gateway targets.</li>

<li>Designing the tool surface: replacing generic query tools with narrow, logged, individually rate-limited operations.</li>

<li>Setting up three-legged OAuth between AgentCore Gateway and Salesforce, including token vault and consent handling.</li>

<li>Moving integrations from Connected Apps to External Client Apps without breaking existing OAuth flows.</li>

<li>Instrumenting API consumption and agent traces so you see problems before an unrelated integration starts failing.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got something concrete, send it over: a gateway target config, an agent trace, a tool schema, or just the error you&#8217;re stuck on. Easier to say something useful about a real setup than a hypothetical one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/">Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny</title>
		<link>https://john-nessime.com/blog/case-studies/ai-construction-claims-platform-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 03 Aug 2026 14:05:46 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon S3 Vectors]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=130</guid>

					<description><![CDATA[<p>Semantic search finds the most persuasive document, not the earliest one. Here is how to architect an AI construction claims and dispute intelligence platform on AWS so retrieval respects the contractual clock, schedule data stays out of the vector index, every answer resolves to a page, and privileged material never shares a retrieval path with project records.</p>
<p>The post <a href="https://john-nessime.com/blog/case-studies/ai-construction-claims-platform-aws/">Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone hands you a shared drive and asks a question that sounds trivial: &#8220;Did we give notice of the delay event inside the contractual period, or didn&#8217;t we?&#8221;</p>



<p class="wp-block-paragraph">The answer is in there. It is one email, or one line in a site diary, sitting among forty thousand other files. Nobody can read forty thousand files, so the instinct is to point a language model at the pile and ask it. That instinct is right. The naive implementation of it is where the money goes.</p>



<p class="wp-block-paragraph">Here is the failure mode that bites hardest, and it is invisible until an expert challenges you on it. You build retrieval over the document set, ask about notice of delay, and the system confidently returns a letter that discusses the delay event in great detail. It is a good letter. It is also dated eleven months after the event, written by the claims consultant during preparation of the claim itself. It scored highest precisely because it was written to argue the point. The contemporaneous notice, the thing you actually needed, was four badly typed lines in a routine progress email that mentioned the word &#8220;delay&#8221; once.</p>



<p class="wp-block-paragraph">Semantic similarity has no concept of a deadline. That single gap is the difference between an <strong>AI construction claims platform</strong> that shortens a disclosure exercise and one that quietly manufactures a wrong answer with a citation attached to it.</p>



<p class="wp-block-paragraph">This post covers how to build that platform on AWS: how to lay out ingestion, how to make retrieval respect the contractual clock, why schedule data must never go anywhere near your vector index, how to keep privileged material out of the same retrieval path as project records, and which AWS building blocks are actually the current ones now that several of the obvious candidates have been moved to maintenance mode.</p>



<h2 class="wp-block-heading">What a claims platform actually has to answer</h2>



<p class="wp-block-paragraph">Before any architecture, be honest about the question shapes. They are not all the same problem and they do not all get solved by retrieval.</p>



<ol class="wp-block-list"><li><strong>Chronology.</strong> What happened, in what order, and on what date was it recorded? This is a retrieval and metadata problem.</li><li><strong>Entitlement.</strong> Which clause applies, and what did it require the parties to do? This is retrieval over the contract plus careful prompting.</li><li><strong>Causation.</strong> Which event moved the critical path, and by how much? This is schedule data and date arithmetic. It is not a language problem at all.</li><li><strong>Quantum.</strong> What did the disruption cost? This is cost and resource data, joined to the events above.</li></ol>



<p class="wp-block-paragraph">Treat all four as &#8220;ask the documents&#8221; and you will get fluent nonsense on two of them. The architecture below splits them deliberately.</p>



<h2 class="wp-block-heading">Failure one: retrieval that finds the best match instead of the first one</h2>



<p class="wp-block-paragraph">Two corpora live in every dispute bundle and they look identical to an embedding model.</p>



<ul class="wp-block-list"><li><strong>Contemporaneous records.</strong> Site diaries, progress emails, minutes, early warnings, RFIs, instructions. Written while the project was running, by people with no idea a dispute was coming.</li><li><strong>Claim-era material.</strong> Narratives, expert reports, without-prejudice correspondence, internal analysis. Written afterwards, specifically to be persuasive about the same events.</li></ul>



<p class="wp-block-paragraph">Claim-era material wins on cosine similarity almost every time, because it is denser in exactly the terms you searched for. If your retriever cannot distinguish them, every answer is contaminated by the argument you were trying to test.</p>



<p class="wp-block-paragraph">The fix is metadata, applied at ingestion, and it is cheap to get right and expensive to retrofit. Amazon Bedrock Knowledge Bases reads a sidecar file that sits next to each document in S3, named with the full original filename plus <code>.metadata.json</code>. So <code>letter-0421.pdf</code> gets <code>letter-0421.pdf.metadata.json</code>. The naming convention is the only link between them; there is no separate registration step.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "doc_date": 20240314,
    "corpus": "contemporaneous",
    "doc_type": "site_correspondence",
    "matter_id": "matter-0007",
    "date_source": "email_header",
    "privileged": false
  }
}</code></pre>



<p class="wp-block-paragraph">Look closely at <code>doc_date</code>. It is an integer, not a string, and that is not a style choice. Bedrock Knowledge Bases metadata attributes support STRING, NUMBER, BOOLEAN and STRING_LIST. The range comparison operators, the ones you need to express &#8220;on or before the notice deadline&#8221;, only apply to NUMBER. Store the date as <code>"2024-03-14"</code> and your filter will not throw an error. It will just quietly match nothing, or match everything, depending on how you wrote it. You will find out weeks later when someone asks why a document they can see in the bundle never appears in results.</p>



<p class="wp-block-paragraph">With the date as a sortable integer, a query filter can express the contractual window directly.</p>



<pre class="wp-block-code"><code>{
  "andAll": [
    { "equals":              { "key": "corpus",   "value": "contemporaneous" } },
    { "equals":              { "key": "matter_id","value": "matter-0007" } },
    { "greaterThanOrEquals": { "key": "doc_date", "value": 20240301 } },
    { "lessThanOrEquals":    { "key": "doc_date", "value": 20240329 } }
  ]
}</code></pre>



<p class="wp-block-paragraph">That is the whole trick. You are no longer asking &#8220;what is the most relevant document about this delay&#8221;. You are asking &#8220;what did the parties actually write during the window in which the contract required them to write it&#8221;. Those are different questions and only one of them is worth anything in a dispute.</p>



<h3 class="wp-block-heading">Where the date comes from matters more than the date</h3>



<p class="wp-block-paragraph">Do not use the S3 object timestamp. It records when someone copied a folder, usually years after the fact and identical across ten thousand files. Derive the date from the document itself: the <code>Date:</code> header on an email, the printed date on a letter, the period covered by a diary entry.</p>



<p class="wp-block-paragraph">Sometimes you cannot, because the scanned undated fax exists in every project archive. Record that honestly with a <code>date_source</code> attribute rather than guessing, and treat unknown-date documents as a separate review pile. An extension of time argument built on an inferred date is an argument you will lose.</p>



<h2 class="wp-block-heading">Failure two: treating the programme like a document</h2>



<p class="wp-block-paragraph">This one is worse, because the output looks right.</p>



<p class="wp-block-paragraph">Oracle Primavera P6 exports XER and PMXML files. Asta Powerproject and Microsoft Project have their own formats. XER in particular is a plain text dump of relational tables, so it goes through a text pipeline without complaint. Chunk it, embed it, and you now have vectors representing fragments of a table of activity codes with no relationships attached.</p>



<p class="wp-block-paragraph">Ask that index how much float activity A1200 had at the March data date and you will get a number. It will be well formatted and it will be invented. Total float is the product of a forward and backward pass across the whole logic network under a specific calendar. It cannot be recovered from a retrieved fragment, and a language model asked to produce it will produce something plausible instead of admitting that.</p>



<p class="wp-block-paragraph">Schedule data goes into a structured store, and the model queries it rather than reasoning about it.</p>



<ol class="wp-block-list"><li>Parse each programme file into tables. <code>PyP6Xer</code> handles XER from Python; MPXJ is a Java library that reads XER, PMXML, Asta Powerproject and MSPDI among others, which matters when the bundle contains four scheduling tools.</li><li>Load activities, logic links, calendars, resource assignments and WBS into Amazon Aurora PostgreSQL for interactive work, or into S3 with AWS Glue and Amazon Athena when you have hundreds of updates and want columnar scans.</li><li>Stamp every row with the <em>data date</em> of the update it came from. This is the single most important column in the whole platform. Without it you have a pile of schedules; with it you have a time series of the project&#8217;s own view of itself.</li><li>Run windows analysis, as-planned versus as-built comparison and float erosion in SQL or Python, deterministically, so the same inputs always give the same numbers.</li><li>Expose the results to the model as a tool it can call, or as generated SQL against a defined schema. The model turns a question into a query and narrates the result. It does not do the arithmetic.</li></ol>



<p class="wp-block-paragraph">A rough shape of the query that makes float erosion visible:</p>



<pre class="wp-block-code"><code>SELECT
    a.activity_id,
    a.data_date,
    a.total_float_days,
    a.total_float_days - LAG(a.total_float_days)
        OVER (PARTITION BY a.activity_id ORDER BY a.data_date)
      AS float_change
FROM   schedule_activities a
WHERE  a.project_id = 'PRJ-01'
  AND  a.data_date BETWEEN DATE '2024-01-01' AND DATE '2024-06-30'
ORDER BY a.activity_id, a.data_date;</code></pre>



<p class="wp-block-paragraph">Nothing clever there, and that is the point. Every number is traceable to a row that came from a named XER file. When an opposing expert asks where a figure came from, the answer is a file name and a query, not &#8220;the model said so&#8221;.</p>



<p class="wp-block-paragraph">Be realistic about effort here. Programme parsing and normalisation across inconsistent updates is the hardest part of the build and the part clients always underestimate. Activity IDs get reused, calendars change mid-project, and someone will have re-baselined without telling anyone. Budget for it.</p>



<h2 class="wp-block-heading">Failure three: an answer with no paper trail</h2>



<p class="wp-block-paragraph">In most RAG applications a citation is a nice touch. In dispute work it <em>is</em> the product. An answer that cannot be traced to a page of a disclosed document is not evidence, it is a rumour with good grammar.</p>



<p class="wp-block-paragraph">Design for that from the ingestion layer, not the presentation layer.</p>



<ul class="wp-block-list"><li><strong>Keep page and position.</strong> Amazon Bedrock Data Automation returns confidence scores and bounding box data alongside extracted fields, and Amazon Textract returns geometry per block. Carry both through the pipeline so a citation resolves to a page and a region, not just a file.</li><li><strong>Route low confidence to humans.</strong> Handwritten site diaries and faxed variation orders will produce low-confidence extractions. Those should land in a review queue by default rather than silently entering the index.</li><li><strong>Reject ungrounded answers.</strong> Amazon Bedrock Guardrails includes contextual grounding checks that score whether a response is supported by the retrieved passages. It reduces confident invention. It does not eliminate it, and anyone who tells you otherwise is selling something.</li><li><strong>Keep an immutable evidential copy.</strong> S3 Versioning plus S3 Object Lock on the landing bucket means the file the platform indexed is provably the file that was disclosed.</li></ul>



<p class="wp-block-paragraph">One design rule underpins all of it: the platform shortlists evidence, it does not decide entitlement. Recognised frameworks for this work, the Society of Construction Law Delay and Disruption Protocol and AACE International&#8217;s Recommended Practice 29R-03 on forensic schedule analysis, both assume a named analyst applying a stated method and exercising judgement. A system that outputs &#8220;the contractor is entitled to 42 days&#8221; is not helping. A system that outputs &#8220;here are the eleven contemporaneous documents inside the notice window, here is the float movement across those updates, here is what is missing&#8221; is doing real work.</p>



<h2 class="wp-block-heading">Failure four: one index for privileged and non-privileged material</h2>



<p class="wp-block-paragraph">Dispute bundles contain legal advice, counsel&#8217;s opinions, without-prejudice correspondence and internal settlement analysis. Those must not be retrievable through the same path as project records.</p>



<p class="wp-block-paragraph">The tempting shortcut is a <code>privileged: false</code> metadata filter on every query. Do not rely on that as your boundary. A metadata filter is a query parameter. One missing filter in one code path, one debug endpoint, one caching layer that drops it, and privileged material surfaces in a general search. The blast radius of that mistake is not a bug report.</p>



<p class="wp-block-paragraph">Separate the indexes physically and separate the IAM roles that can reach them. Amazon S3 Vectors makes this practical: you can set a dedicated customer-managed KMS key per vector index, and you get a large number of indexes per vector bucket, so per-matter and per-sensitivity separation does not become an operational burden. Keep the metadata flag as well, because defence in depth is free, but make the identity boundary the one you actually trust.</p>



<p class="wp-block-paragraph">Amazon Macie is worth pointing at the landing bucket to find personal data you did not expect, particularly in HR records and accident reports that get swept into project archives.</p>



<h2 class="wp-block-heading">Choosing the AWS building blocks, including what not to build on</h2>



<p class="wp-block-paragraph">A lot of published architectures for this kind of platform are now pointing at services AWS has stopped developing. Two matter here, and the dates are the point.</p>



<ul class="wp-block-list"><li><strong>Amazon Kendra</strong> entered maintenance mode on 30 June 2026 and stops accepting new customers on 30 July 2026. Existing customers keep support and security fixes but no new capability. AWS directs new enterprise search and RAG work to Amazon Bedrock Knowledge Bases. If a tutorial or a proposal you are reading starts with a Kendra index, it predates that change.</li><li><strong>Amazon Bedrock Agents</strong> moved to maintenance mode in the same round of service availability changes, with Amazon Bedrock AgentCore as the successor for agentic orchestration. Check the current AWS service availability page before you commit an orchestration layer.</li></ul>



<p class="wp-block-paragraph">For the retrieval layer itself, Bedrock Knowledge Bases now comes in two shapes and the choice is a real trade-off rather than a marketing tier.</p>



<h3 class="wp-block-heading">Managed Knowledge Base</h3>



<p class="wp-block-paragraph">AWS manages the vector store, embeddings model, re-ranker and retrieval orchestration as a single primitive, with native connectors for Amazon S3, SharePoint, Confluence, Google Drive, OneDrive and a web crawler, plus automatic parsing strategy selection and a retriever that decomposes multi-step queries. The connectors pull source permissions along with content, which matters when the document set lives in the client&#8217;s SharePoint rather than a bucket you control.</p>



<p class="wp-block-paragraph">Where it wins: you get a working retrieval layer in an afternoon instead of a fortnight, and the parsing tuning that normally eats the first weeks of a build is done for you. For a first matter, or a proof of value before a client commits budget, this is the one I would reach for.</p>



<h3 class="wp-block-heading">Custom Knowledge Base</h3>



<p class="wp-block-paragraph">You bring your own vector store and control chunking, embedding model and index layout.</p>



<p class="wp-block-paragraph">Where it wins: claims work has awkward chunking requirements. A two-page letter split mid-sentence at a page boundary produces a chunk where the notice sentence has lost its date and its addressee. Controlling chunk boundaries around document structure, and controlling which index a document lands in, are both easier when you own the store. Where it doesn&#8217;t: you now own embedding model upgrades, re-indexing, sync failures and capacity, which is real ongoing work for a small team.</p>



<p class="wp-block-paragraph">Start managed, build a retrieval evaluation set of real questions with known correct documents, and only move to custom when that set demonstrates the problem is chunking. Most teams migrate on a hunch and discover the problem was metadata all along.</p>



<h3 class="wp-block-heading">Where the vector storage bill actually comes from</h3>



<p class="wp-block-paragraph">Rates change, so learn the billing mechanism rather than a number. Amazon S3 Vectors charges on three axes: upload volume by logical gigabyte, storage by logical gigabyte, and queries by data processed, where data processed scales with the size of the index being searched. Note that filtering does not reduce the data processed by a query.</p>



<p class="wp-block-paragraph">That shape suits claims work unusually well. A dispute archive is enormous and cold: millions of chunks, queried by a handful of analysts a few hundred times a day, so you pay mostly for storage, which is the cheap axis. Compare that against Amazon OpenSearch Serverless, which prices on provisioned compute units and therefore rewards high query volume against a smaller index, or Aurora PostgreSQL with pgvector when you already need Aurora for the schedule tables and would rather run one system than two.</p>



<p class="wp-block-paragraph">The practical lever is to split indexes per matter. Query cost scales with index size, so one giant index across every dispute you have ever run makes every query more expensive than it needs to be, on top of being a bad idea for confidentiality.</p>



<h2 class="wp-block-heading">A reference pipeline</h2>



<ol class="wp-block-list"><li>Everything lands in S3 under a per-matter prefix, with Versioning and Object Lock enabled on the evidential copy.</li><li>S3 event notifications trigger AWS Step Functions. Use Step Functions rather than a chain of Lambdas so that a failed extraction on page 300 of a 400-page bundle is visible and resumable.</li><li>Classify and split. Scanned bundles arrive as one PDF containing forty separate documents. Splitting them correctly is a prerequisite for dating them correctly.</li><li>Extract text with Amazon Bedrock Data Automation or Amazon Textract, keeping confidence scores and geometry.</li><li>Derive the document date and write the <code>.metadata.json</code> sidecar. Anything undated goes to the review queue.</li><li>Route by type: correspondence to the knowledge base, programme files to the XER parser and the relational store, cost data to its own tables.</li><li>Sync the knowledge base, then run your retrieval evaluation set before anyone uses it. A sync that succeeds is not the same as an index that answers correctly.</li><li>Serve through an API that refuses to return an answer without citations, and log every query with the filters that were applied.</li></ol>



<p class="wp-block-paragraph">Define the whole thing in Terraform or OpenTofu from the start. Matters are per-client and short-lived, and standing one up should be a variable file, not an afternoon in the console. Point Amazon CloudWatch, or Grafana Cloud if you already run Grafana elsewhere, at the Step Functions execution metrics so a silently failing extraction stage does not go unnoticed for a week.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Date filters return nothing, and no error</h3>



<p class="wp-block-paragraph">Almost always the date was stored as a string. Range operators need NUMBER. Convert to an integer in <code>YYYYMMDD</code> form and re-sync the affected documents.</p>



<h3 class="wp-block-heading">A document is in the bucket but never appears in results</h3>



<p class="wp-block-paragraph">Check the sidecar filename first. It must be the complete original filename with <code>.metadata.json</code> appended, extension included. <code>report.pdf.metadata.json</code> works; <code>report.metadata.json</code> is a file the ingestion job will happily ignore. After that, check whether a filter in the query path is excluding it.</p>



<h3 class="wp-block-heading">Answers cite the right document but the wrong passage</h3>



<p class="wp-block-paragraph">Chunking split the document somewhere structurally meaningful. Look at the raw chunks for that file. If the notice sentence and its date are in different chunks, no amount of prompt tuning fixes it. That is the signal to take control of chunking.</p>



<h3 class="wp-block-heading">Float figures do not match the client&#8217;s own analysis</h3>



<p class="wp-block-paragraph">Check calendars before you check logic. Different activity calendars, a changed default calendar, or an update where someone applied a progress override will move float without any logic change. Reconcile activity counts between your parsed tables and the source file before trusting anything downstream.</p>



<h3 class="wp-block-heading">Query costs jumped without more usage</h3>



<p class="wp-block-paragraph">An index grew. With storage-side vector search, query cost tracks the size of the index being scanned, so ingesting a large new bundle raises the price of every subsequent query against that index. Split by matter.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Using the file&#8217;s storage timestamp as the document date. It records the migration, not the event.</li><li>Indexing claim narratives and contemporaneous records into the same corpus with no way to tell them apart.</li><li>Embedding programme exports because they happen to be text files.</li><li>Treating a metadata filter as a privilege boundary instead of an optimisation.</li><li>Letting the model state entitlement conclusions rather than assembling and citing evidence.</li><li>Building on services that have moved to maintenance mode because the tutorial you followed predates the change.</li><li>Shipping without a retrieval evaluation set, so you have no way to know whether a change made things better or worse.</li><li>One index for every matter, which is both a cost problem and a confidentiality problem.</li></ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list"><li>Make the document date a first-class, numeric, filterable attribute, and record where it came from.</li><li>Keep an immutable evidential copy separate from the working copy the pipeline mutates.</li><li>Separate structured schedule and cost data from unstructured documents, and let the model query the former rather than reason about it.</li><li>Build a retrieval evaluation set from real questions with known correct documents before you tune anything.</li><li>Enforce citations at the API layer, so an uncited answer is impossible rather than discouraged.</li><li>Isolate privileged material by index and by IAM role, with metadata as a second layer.</li><li>Log every query with its filters, so you can reconstruct how any given answer was reached.</li><li>Define infrastructure as code so a new matter is a deployment, not a project.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Can an AI construction claims platform replace a delay expert?</h3>



<p class="wp-block-paragraph">No, and building toward that goal produces something unusable. Established forensic frameworks assume a named analyst applying a stated method whose reasoning can be tested. The platform&#8217;s value is compressing weeks of document review into hours and making the schedule data queryable, so the expert spends their time on judgement rather than searching.</p>



<h3 class="wp-block-heading">Should I use Amazon Kendra for the search layer?</h3>



<p class="wp-block-paragraph">Not for a new build. Kendra entered maintenance mode on 30 June 2026 and closed to new customers on 30 July 2026, with AWS pointing to Bedrock Knowledge Bases for equivalent and more current capability. Existing Kendra deployments continue to be supported, so this is a migration assessment rather than an emergency, but starting there now means starting on a service with no roadmap.</p>



<h3 class="wp-block-heading">How do I stop the model inventing float and delay figures?</h3>



<p class="wp-block-paragraph">Do not give it the chance. Keep schedule data in a relational or columnar store and have the model generate queries against a defined schema, or call a tool that runs a fixed calculation. The arithmetic happens in SQL or Python where it is deterministic and reproducible; the model only turns questions into queries and results into sentences.</p>



<h3 class="wp-block-heading">Which vector store should I choose for a claims archive?</h3>



<p class="wp-block-paragraph">Match the store to your query pattern. Large, cold archives queried by a few analysts favour storage-priced options like Amazon S3 Vectors, where you mostly pay to keep the data. Smaller indexes hit constantly favour compute-priced options like Amazon OpenSearch Serverless. If you already run Aurora PostgreSQL for schedule data, pgvector alongside it is a legitimate way to avoid operating a second system.</p>



<h3 class="wp-block-heading">How do I handle scanned and handwritten site records?</h3>



<p class="wp-block-paragraph">Extract them with confidence scores retained, set a threshold, and route everything below it to human review before indexing. Handwritten diaries are frequently the most probative documents in a delay claim and also the least reliable to read automatically, so the review queue is not an edge case. Plan capacity for it.</p>



<h3 class="wp-block-heading">Where do documents come from if they are not already in S3?</h3>



<p class="wp-block-paragraph">Most project records live in a common data environment such as Procore, Autodesk Construction Cloud, Aconex or a client SharePoint tenancy. Bedrock Managed Knowledge Base has native connectors for SharePoint, Confluence, Google Drive and OneDrive that ingest permissions alongside content. For platforms without a native connector, export to S3 and keep the export manifest as part of the disclosure record.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An <strong>AI construction claims platform</strong> lives or dies on whether it understands time. Every hard requirement in this build traces back to that: numeric dates so you can filter to a contractual window, a data date on every schedule row so float movement is measurable, a corpus flag so contemporaneous records are not drowned out by material written to argue about them, and citations that resolve to a page so any answer can be checked.</p>



<p class="wp-block-paragraph">Get the temporal metadata right at ingestion and the rest of the architecture is ordinary AWS work. Get it wrong and you have built a very expensive way to retrieve the most persuasive document instead of the true one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building this on AWS?</h2>



<p class="wp-block-paragraph">I design and build document and data platforms on AWS, and this kind of system sits squarely in that work. Things I can help with:</p>



<ul class="wp-block-list"><li>Designing the ingestion pipeline: S3 landing zones with Object Lock, Step Functions orchestration, splitting and classifying scanned bundles, and confidence-based routing to human review.</li><li>Getting the temporal metadata model right, including date derivation, sidecar generation and filter design against Amazon Bedrock Knowledge Bases.</li><li>Parsing Primavera P6 XER and PMXML exports into queryable tables in Aurora PostgreSQL or S3 with Glue and Athena, with a data date on every row.</li><li>Choosing and sizing the vector layer across Amazon S3 Vectors, OpenSearch Serverless and pgvector, based on your actual query pattern rather than a benchmark.</li><li>Building index and IAM separation for privileged material, plus KMS key strategy and Macie scanning of landing buckets.</li><li>Setting up retrieval evaluation, citation enforcement, query audit logging and CloudWatch or Grafana dashboards over the pipeline so failures surface early.</li></ul>



<p class="wp-block-paragraph">If you are partway into something like this already, send me a sample metadata sidecar, a Step Functions execution history, or a query that returns the wrong document, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/case-studies/ai-construction-claims-platform-aws/">Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Build a Secure AI Medical Assistant on AWS: The Boundaries That Actually Leak</title>
		<link>https://john-nessime.com/blog/case-studies/secure-ai-medical-assistant-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 03 Aug 2026 13:45:12 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Comprehend Medical]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Healthcare AI]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Vector Database]]></category>
		<category><![CDATA[VPC]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=127</guid>

					<description><![CDATA[<p>A practical architecture for a secure AI medical assistant on AWS, organised by the boundary the data crosses: the input box, your own invocation logs, cross-Region inference routing, the retrieval index, and clinical accuracy. Includes real commands, the failure modes that stay invisible until an audit, and the trade-offs worth knowing before you build.</p>
<p>The post <a href="https://john-nessime.com/blog/case-studies/secure-ai-medical-assistant-aws/">Build a Secure AI Medical Assistant on AWS: The Boundaries That Actually Leak</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on a Friday afternoon: &#8220;One of the residents pasted a real discharge summary into the demo.&#8221; Nobody meant anything by it. The thing was a study aid, a chat box over a pile of reference material, running in a sandbox account with no Business Associate Addendum in place and invocation logging switched on because logging is a good habit. And now there is protected health information sitting in plaintext in a CloudWatch log group in an account that was never in scope for it.</p>



<p class="wp-block-paragraph">That is the failure mode worth planning for. Not a jailbreak, not a model saying something clinically wrong on stage. A user typing something perfectly reasonable into a box you built, and the data ending up somewhere you never drew on the diagram. This post walks through how to build a secure AI medical assistant on AWS, organised by the boundary the data actually crosses: the input box, your own logs, the Region the inference runs in, the retrieval index, and finally the answer itself. Commands are included where they explain something. Where a value depends on your account or your counsel, I say so instead of making one up.</p>



<h2 class="wp-block-heading">Two different products hiding behind one request</h2>



<p class="wp-block-paragraph">&#8220;An AI assistant for medical students and doctors&#8221; is two builds with two risk profiles, and conflating them is the root of most of the trouble.</p>



<ul class="wp-block-list">
<li><strong>The study tool.</strong> Question banks, guideline summaries, differential drills, spaced repetition. In theory it never touches patient data. Its real risk is confident wrongness and unattributed answers, not privacy.</li>

<li><strong>The clinical assistant.</strong> Note summarisation, coding support, chart question answering. It handles PHI by design, so the whole thing has to sit inside a HIPAA-designated account from day one.</li>
</ul>



<p class="wp-block-paragraph">The trap is that the study tool becomes the clinical assistant without anyone shipping a release. A student rehearses a case they saw on the ward. A doctor tries the study tool on a real chart because it is the one that is already open. The moment your input box accepts free text from someone with clinical access, you should assume PHI will arrive in it. Build accordingly, or put the study tool on infrastructure where PHI arriving is survivable.</p>



<p class="wp-block-paragraph">My default is to run both in the same HIPAA-designated account with the same controls, and keep only the marketing site, the docs and the waitlist form somewhere ordinary and cheap like InterServer or any commodity host, entirely outside the AWS organisation. Small blast radius beats clever separation you have to explain to an auditor.</p>



<h2 class="wp-block-heading">Boundary one: PHI arrives before you decide to accept it</h2>



<p class="wp-block-paragraph">Start with the thing that trips up almost every first build: HIPAA-eligible and HIPAA-compliant are not the same word. AWS designating a service as HIPAA-eligible means you are permitted to process PHI with it under an executed Business Associate Addendum. It says nothing about whether your deployment is compliant. That part is entirely yours.</p>



<p class="wp-block-paragraph">Three conditions have to hold together before any PHI touches a service: the service is on the AWS HIPAA Eligible Services Reference, you have an executed BAA, and the account is designated for HIPAA use. Amazon Bedrock and Amazon Comprehend Medical both appear on that list, and the BAA is accepted through AWS Artifact rather than negotiated by email. Check the reference page yourself before you commit to an architecture, because the list changes and every component in your diagram has to be on it, not just the model.</p>



<h3 class="wp-block-heading">Detecting PHI at the door</h3>



<p class="wp-block-paragraph">Amazon Comprehend Medical has an operation specifically for finding protected health information in unstructured clinical text. You hand it free text, it returns detected entities with a type, a confidence score and character offsets.</p>



<pre class="wp-block-code"><code>aws comprehendmedical detect-phi 
  --region us-east-1 
  --text "Patient seen for chest pain, MRN 004512, discharged Tuesday."</code></pre>



<p class="wp-block-paragraph">The offsets are the useful part. They let you redact or tokenise in place before the text goes anywhere else, rather than throwing the whole message away and telling the user to try again.</p>



<p class="wp-block-paragraph">Now the caveat that matters more than the feature. AWS states plainly that Comprehend Medical may not identify PHI in all circumstances and that it does not, on its own, meet HIPAA&#8217;s requirements for de-identification. Read that as: it is a good filter and a terrible guarantee. If your compliance story is &#8220;we strip PHI before the model sees it, so we are outside HIPAA scope,&#8221; that story does not hold. Treat detection as defence in depth inside a compliant account, not as an escape hatch from needing one.</p>



<h2 class="wp-block-heading">Boundary two: your own logs are the most likely leak</h2>



<p class="wp-block-paragraph">This is the invisible one. Bedrock model invocation logging is disabled by default and captures the full request data, response data and metadata for every call in the account, in that Region. You turn it on for a good reason, usually because someone in security asked who prompted what and when. Then it quietly becomes the largest concentration of raw clinical text you own, and it is nowhere on the architecture diagram because it is a checkbox rather than a component.</p>



<p class="wp-block-paragraph">Check whether it is on before you assume anything:</p>



<pre class="wp-block-code"><code>aws bedrock get-model-invocation-logging-configuration --region us-east-1</code></pre>



<p class="wp-block-paragraph">The configuration is per account per Region, so run it in every Region where anyone has ever opened the Bedrock console. Setting it deliberately looks like this:</p>



<pre class="wp-block-code"><code>aws bedrock put-model-invocation-logging-configuration 
  --region us-east-1 
  --logging-config '{
    "s3Config": {
      "bucketName": "med-assistant-invocation-logs",
      "keyPrefix": "bedrock/"
    },
    "textDataDeliveryEnabled": true,
    "imageDataDeliveryEnabled": false,
    "embeddingDataDeliveryEnabled": false
  }'</code></pre>



<p class="wp-block-paragraph">Each delivery flag is a separate decision about a separate category of PHI. Text is the obvious one. Image delivery matters the moment anyone uploads a photographed chart or a scan, because burned-in identifiers travel with the pixels and no text filter will ever see them. Embedding delivery is the one people leave on without thinking; vectors derived from clinical text are not a safe artefact, and they are large.</p>



<p class="wp-block-paragraph">The sharpest detail is buried in the Guardrails documentation: AWS notes you can disable invocation logs if you do not want blocked content appearing as plaintext in them. Read the implication. A guardrail can refuse a prompt, mask the identifiers, and stop the model ever seeing them, and the original text still lands in your log destination. The guardrail protects the model call. It does not protect the log.</p>



<ul class="wp-block-list">
<li>Encrypt the log destination with a customer-managed KMS key, and keep the key policy tight enough that &#8220;everyone with S3 read&#8221; is not also &#8220;everyone with chart access&#8221;.</li>

<li>Set a retention period that reflects a legal decision someone actually made, not the CloudWatch default of never expiring.</li>

<li>Send logs to a separate, tightly scoped account if your organisation is large enough to have people who need dashboards but not records.</li>

<li>Remember your application logs too. A framework that logs request bodies on error will do this to you long before Bedrock does.</li>
</ul>



<h2 class="wp-block-heading">Boundary three: where the inference actually runs</h2>



<p class="wp-block-paragraph">Cross-Region inference in Bedrock exists because on-demand capacity is uneven and bursts happen. It routes your request to another Region to get it served. There are two flavours and the difference is not cosmetic.</p>



<ul class="wp-block-list">
<li><strong>Geographic profiles</strong> keep routing inside a defined geography such as the US or the EU. A request that starts in the EU stays in EU Regions. This is the one built for residency requirements.</li>

<li><strong>Global profiles</strong> route to supported commercial Regions worldwide for maximum throughput. AWS documents that a request can be routed to a destination Region even if you never opted that Region into your account.</li>
</ul>



<p class="wp-block-paragraph">To be fair to global profiles: data is not stored in the destination Region, transfer happens encrypted across the AWS network, and your invocation logs, knowledge bases and configuration all stay in the source Region. For a workload with no geographic constraint it is a genuinely good default, and it typically carries a lower per-token rate than staying in-geography. For a clinical workload with a residency commitment in a contract, it is the wrong tool, and &#8220;the prompt left the geography but was not stored there&#8221; is a sentence you do not want to be constructing during an audit.</p>



<p class="wp-block-paragraph">Pin it in policy rather than trusting a config value in a repo. A Service Control Policy denying the Bedrock API outside your approved Regions closes both doors at once, because invoking a cross-Region profile requires model access in the destination Regions as well as the source:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyBedrockOutsideApprovedRegions",
      "Effect": "Deny",
      "Action": "bedrock:*",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The trade-off is real and you should know it before you apply this. If you later adopt a geographic profile whose destination list includes a Region you denied, invocations fail with an access error that looks nothing like a routing problem. Keep the approved Region list and the profile&#8217;s destination list in the same review, not in two different tickets.</p>



<h2 class="wp-block-heading">Boundary four: retrieval, and the grounding check that will not save you</h2>



<p class="wp-block-paragraph">A useful medical assistant is retrieval-augmented. The model alone is a fluent generalist; the value comes from grounding answers in a curated corpus, whether that is institutional guidelines, formulary rules or a licensed reference set. Bedrock Knowledge Bases will manage the ingestion and the vector store for you, or you can run your own index in Amazon OpenSearch Service.</p>



<p class="wp-block-paragraph">Two things bite here.</p>



<p class="wp-block-paragraph"><strong>Tenant isolation in the index.</strong> If you serve more than one hospital, department or study cohort, a shared index with a filter applied in application code is one refactor away from cross-tenant retrieval. Filters are easy to forget and impossible to notice, because a wrong answer that cites a real document looks exactly like a right one. Separate indexes per tenant, or fine-grained access control enforced below your application, cost more and fail safe.</p>



<p class="wp-block-paragraph"><strong>The grounding check has a scope limit.</strong> Guardrails contextual grounding checks detect responses that are not supported by the source material or not relevant to the question. Excellent feature. The documentation states the supported use cases are summarisation, paraphrasing and question answering, and that conversational chatbot use cases are not supported. If your product is a chat interface with turn history, do not assume this check is covering you. There is also a streaming wrinkle: relevance is assessed per chunk, so an irrelevant response can reach the user before it is marked irrelevant at the end of the stream.</p>



<p class="wp-block-paragraph">What you can rely on regardless of interface is applying a guardrail to arbitrary text directly, which is how you check an input before it enters your own pipeline:</p>



<pre class="wp-block-code"><code>import boto3

client = boto3.client("bedrock-runtime", region_name="us-east-1")

response = client.apply_guardrail(
    guardrailIdentifier=GUARDRAIL_ID,
    guardrailVersion="DRAFT",
    source="INPUT",
    content=[{"text": {"text": user_message}}],
)

if response["action"] == "GUARDRAIL_INTERVENED":
    # Stop here. Do not forward, and do not write the raw text anywhere.
    handle_blocked(response["assessments"])</code></pre>



<p class="wp-block-paragraph">Note what the comment is doing. The most common bug in this pattern is catching the intervention and then logging the offending input &#8220;for debugging&#8221;, which reintroduces exactly the leak the guardrail just prevented. Log the assessment, log a request identifier, never the text.</p>



<p class="wp-block-paragraph">Sensitive information filters give you two handling modes: block, which rejects the request outright, and mask, which replaces detected entities with placeholder tags. For a study tool, mask is usually right, because a student who typed a name by reflex gets a useful answer instead of a wall. For a clinical assistant working over charts, blocking on unexpected identifiers in an input that should have arrived structured is a better signal that something upstream is wrong.</p>



<h2 class="wp-block-heading">Boundary five: being confidently wrong</h2>



<p class="wp-block-paragraph">Everything above is about data leaving. This one is about a bad answer arriving, and for a medical audience it is the reputational failure that actually ends products.</p>



<ul class="wp-block-list">
<li><strong>Cite or refuse.</strong> Return the retrieved passages alongside the answer. If retrieval returned nothing above your relevance threshold, say so rather than letting the model answer from parametric memory. A student cannot verify what they cannot see.</li>

<li><strong>Version the corpus, not just the model.</strong> When a guideline changes, you need to know which answers were generated against the old text. Store a corpus revision identifier with every logged response.</li>

<li><strong>Use denied topics deliberately.</strong> Dosing for a named patient, and anything that reads as a treatment directive rather than reference information, are reasonable things to route to a refusal with a clear explanation.</li>

<li><strong>Get regulatory advice early.</strong> Whether clinical decision support software is regulated as a medical device depends on your jurisdiction and, critically, on the claims you make about it. This is a legal question with engineering consequences, and it is much cheaper to answer before the interface exists.</li>
</ul>



<h2 class="wp-block-heading">A build order for a secure AI medical assistant on AWS</h2>



<p class="wp-block-paragraph">Sequence matters here more than in most builds, because several of these are painful to retrofit.</p>



<ol class="wp-block-list">
<li>Accept the BAA through AWS Artifact and designate the account for HIPAA use. Do this before the first prototype, not before the first customer.</li>

<li>Pin Regions with a Service Control Policy, and decide the geographic-versus-global inference profile question in writing.</li>

<li>Configure invocation logging deliberately, with a customer-managed KMS key, an explicit retention period, and each data-type delivery flag chosen rather than defaulted.</li>

<li>Put the application in private subnets and reach Bedrock over VPC endpoints so PHI-bearing traffic does not traverse the public internet.</li>

<li>Build the guardrail before the prompt. Sensitive information filters, denied topics, and grounding checks where they apply.</li>

<li>Add Comprehend Medical detection in the ingestion path for anything you are storing, and at the input boundary for anything a user types.</li>

<li>Build retrieval with tenant isolation from the first index, not the second.</li>

<li>Only now write the assistant&#8217;s prompt and interface, and put a WAF such as Cloudflare or AWS WAF in front of the public endpoint.</li>
</ol>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>AccessDenied on a model that clearly works elsewhere.</strong> Usually one of three things: model access not requested in this Region, a cross-Region profile whose destination Regions your SCP denies, or an IAM policy that grants the model in the source Region only.</li>

<li><strong>No invocation logs appearing.</strong> The configuration is per Region and disabled by default. Confirm you queried the same Region the application calls, and that the delivery flag for the data type you expect is enabled.</li>

<li><strong>Guardrail passes text you expected it to catch.</strong> Sensitive information detection is probabilistic and context-dependent. Very short inputs give it little to work with. Test with realistic clinical phrasing, not single tokens, and add regex patterns for structured identifiers like MRNs that follow a house format.</li>

<li><strong>Grounding check appears to do nothing.</strong> Check your interface shape against the supported use cases before assuming it is misconfigured, and check whether streaming is masking the result until the response completes.</li>

<li><strong>Retrieval returns plausible but wrong documents.</strong> Look at chunking before you look at the model. Clinical guidelines chunked mid-table or mid-criteria retrieve badly no matter what embedding you use.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Prototyping in a personal or sandbox account and promising to migrate later. The prototype is where the first real chart gets pasted.</li>

<li>Assuming the eligibility of Bedrock covers the whole stack. Every component that touches PHI needs to be eligible, including the vector store, the queue and the cache.</li>

<li>Treating PHI detection as de-identification. AWS says explicitly that it is not.</li>

<li>Leaving image and embedding log delivery enabled by copy-paste.</li>

<li>Logging blocked prompts to debug the guardrail.</li>

<li>Shipping a chat interface and citing contextual grounding as the hallucination control.</li>

<li>Filtering tenants in application code over a shared index.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Write the data-flow diagram with logs, backups and the vector index drawn as first-class destinations. If PHI can land there, it is on the diagram.</li>

<li>Define everything in Terraform or OpenTofu so the guardrail, the logging configuration and the SCP are reviewable artefacts rather than console state.</li>

<li>Keep a small evaluation set of realistic clinical questions with known-good answers, and run it on every prompt or model change.</li>

<li>Alarm on guardrail intervention rate. A sudden rise usually means a change upstream, not a change in users.</li>

<li>Dashboard invocation counts, latency and intervention rates somewhere your on-call actually looks, whether that is CloudWatch, Grafana or Datadog.</li>

<li>Scope IAM to specific model ARNs and specific guardrail identifiers. A wildcard on <code>bedrock:InvokeModel</code> means any model, including ones you never evaluated.</li>

<li>Rehearse the breach path once. Knowing which bucket, which log group and which key you would need to reason about is worth an afternoon.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Amazon Bedrock HIPAA compliant?</h3>



<p class="wp-block-paragraph">Bedrock is HIPAA-eligible, which is a different claim. Eligibility means AWS permits you to process PHI with the service under an executed BAA. Compliance is a property of your deployment: your account designation, encryption, access control, network isolation, logging and retention. A Bedrock workload with a signed BAA and public endpoints and unbounded plaintext logs is not compliant.</p>



<h3 class="wp-block-heading">Do I need a BAA if the tool is only for medical students?</h3>



<p class="wp-block-paragraph">If the tool genuinely never receives PHI, HIPAA is not engaged. The practical question is whether you can guarantee that, given that your users have clinical access and a free-text box. If you cannot, get the BAA. It costs you a self-service acceptance in AWS Artifact and removes an entire category of incident.</p>



<h3 class="wp-block-heading">Does AWS use my prompts to train the models?</h3>



<p class="wp-block-paragraph">AWS states that customer content submitted to Bedrock is not used to train the underlying foundation models or shared with model providers. That statement is the sort of thing a hospital security review will want quoted verbatim from the current AWS data protection documentation rather than from a blog, so pull the live wording when you write your assessment.</p>



<h3 class="wp-block-heading">Is Comprehend Medical enough to de-identify clinical text?</h3>



<p class="wp-block-paragraph">No. AWS documents that it may not identify PHI in all circumstances and does not by itself meet HIPAA&#8217;s de-identification requirements. Use it as a detection layer and a redaction aid inside a compliant environment. Formal de-identification, whether by the Safe Harbor method or expert determination, is a separate exercise with its own review.</p>



<h3 class="wp-block-heading">Where should the vector store live?</h3>



<p class="wp-block-paragraph">In the same account and Region as the rest of the workload, on a HIPAA-eligible service, encrypted with a customer-managed key, reachable only from private subnets. Embeddings derived from clinical text are not sanitised data and should not be treated as a lower-sensitivity artefact than the source.</p>



<h3 class="wp-block-heading">Should I use a global or geographic inference profile?</h3>



<p class="wp-block-paragraph">Geographic if you have any residency commitment, contractual or regulatory. Global if you have none and want the throughput and the lower per-token rate. Decide once, document the reasoning, and enforce it with a Service Control Policy rather than a configuration constant.</p>



<h3 class="wp-block-heading">Does an AI medical assistant count as a medical device?</h3>



<p class="wp-block-paragraph">It depends on your jurisdiction and on what you claim the software does. Software that surfaces reference information a clinician independently reviews has generally been treated differently from software that directs a clinical decision, but the boundary is fact-specific and moves. This is a question for regulatory counsel before launch, not a question for your architecture diagram.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A secure AI medical assistant on AWS is not mainly a model problem. Bedrock, Guardrails and Comprehend Medical are the easy part, and the documentation for them is good. The hard part is that PHI leaves through the paths you did not draw: an invocation log you enabled for good reasons, a global inference profile that routes wherever capacity exists, an index shared between tenants, a debug line added at two in the morning.</p>



<p class="wp-block-paragraph">So build the boundary first and the assistant second. Get the BAA accepted, pin the Regions in policy, decide consciously what your logs are allowed to hold, and isolate retrieval per tenant before there is a second tenant. Every one of those is cheap on day one and expensive in month six.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on your build?</h2>



<p class="wp-block-paragraph">I work with teams building AI on AWS where the data is sensitive and the failure modes are quiet. Things I can help with on a project like this:</p>



<ul class="wp-block-list">
<li>Reviewing a Bedrock architecture against the boundaries above and telling you where PHI can actually land</li>

<li>Setting up account separation, BAA scope, Region pinning with Service Control Policies, and VPC endpoint access to Bedrock</li>

<li>Designing and tuning Guardrails policies, including custom regex for house identifier formats, and the block-versus-mask decision per surface</li>

<li>Building the retrieval layer with per-tenant isolation, sensible clinical chunking, and citation-or-refuse behaviour</li>

<li>Auditing invocation logging, KMS key policies, retention and application-level log hygiene for accidental PHI capture</li>

<li>Putting the whole thing in Terraform or OpenTofu so your controls are reviewable instead of remembered</li>
</ul>



<p class="wp-block-paragraph">If any of that is on your plate, send me the piece you are least sure about. A redacted architecture diagram, a guardrail configuration, a logging policy, an <code>AccessDenied</code> you cannot explain. I would rather look at the real thing than talk in generalities.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/case-studies/secure-ai-medical-assistant-aws/">Build a Secure AI Medical Assistant on AWS: The Boundaries That Actually Leak</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
