<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Generative AI | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/generative-ai/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/generative-ai/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 17 Oct 2026 06:00:06 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Generative AI | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/generative-ai/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Building a Personalized Learning Assistant on AWS Without Leaking Student Data</title>
		<link>https://john-nessime.com/blog/technical-guides/personalized-learning-assistant-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 17 Oct 2026 06:00:06 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Education Technology]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3 Vectors]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[EdTech]]></category>
		<category><![CDATA[FERPA]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Identity Propagation]]></category>
		<category><![CDATA[Learning Management System]]></category>
		<category><![CDATA[LTI 1.3]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[pgvector]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Student Data Privacy]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=466</guid>

					<description><![CDATA[<p>A personalized learning assistant on AWS rarely fails loudly. It fails by being helpful to the wrong student. Five failure families that decide whether an AI tutor built on Amazon Bedrock is safe to put in front of a class: LTI identity, retrieval isolation, guardrail limits, the cost shape of an academic term, and being able to prove what the assistant saw.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/personalized-learning-assistant-aws/">Building a Personalized Learning Assistant on AWS Without Leaking Student Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket that kicks this project into its second phase usually reads something like: &#8220;The assistant is giving my students the answers.&#8221; Attached is a screenshot where the tutor bot cheerfully walks a student through the marking scheme for a coursework brief that has not been released yet. Nothing crashed. No 5xx, no alarm, no failed health check. The rubric PDF had been dropped into the same bucket prefix as the lecture notes weeks earlier, and retrieval did exactly what retrieval does.</p>



<p class="wp-block-paragraph">That is the shape of every serious problem in this space. A personalized learning assistant on AWS almost never fails loudly. It fails by being helpful to the wrong person, or by being confidently wrong in a way that reads as authoritative because it came from the institution&#8217;s own system. Both failures are silent until a human reports them, and by then the assistant has been answering that way for a term.</p>



<p class="wp-block-paragraph">This post covers the parts that are genuinely hard: getting identity and scope out of the learning platform instead of the browser, making retrieval respect the boundary, knowing what Bedrock Guardrails will and will not do for you, sizing a system whose traffic collapses every summer, and proving after the fact what the assistant saw. I assume you can already stand up a RAG pipeline. The pipeline is the easy part.</p>



<h2 class="wp-block-heading">What &#8220;personalized&#8221; has to mean before you build anything</h2>



<p class="wp-block-paragraph">Personalization in an education setting is three separate questions that people tend to collapse into one:</p>



<ul class="wp-block-list">
<li><strong>What is this learner allowed to see?</strong> Course enrollment, section, release dates, role. This is an authorization question and it belongs nowhere near the model.</li>



<li><strong>What has this learner already done?</strong> Submissions, attempts, viewed material. This is state, and it belongs in a database you control.</li>



<li><strong>How is this learner doing?</strong> Mastery, misconceptions, pace. This is inference, and it is the only part where a model earns its keep.</li>
</ul>



<p class="wp-block-paragraph">Teams that skip straight to the third question end up asking the model to enforce the first one through prompt instructions. That does not hold. A system prompt saying &#8220;do not reveal solutions before the due date&#8221; is a suggestion; a retrieval filter that never returns the solution chunk is a control. Keep the boundary in the retrieval layer and the personalization in the generation layer, and most of the scary failure modes stop being possible rather than becoming unlikely.</p>



<h2 class="wp-block-heading">Where a personalized learning assistant on AWS actually breaks</h2>



<p class="wp-block-paragraph">Organized by failure family rather than by service, because the services are documented and the failures are not. Five of them, roughly in the order they bite.</p>



<h2 class="wp-block-heading">Failure family one: identity that stops at the front door</h2>



<p class="wp-block-paragraph">The assistant lives inside Canvas, Moodle, Blackboard, Brightspace or Open edX. It gets there through LTI 1.3, which replaced the old shared-secret model with an OpenID Connect launch and an RSA-signed JWT. The platform initiates a login against your tool&#8217;s login endpoint, your tool responds with an auth request, the platform posts back a signed token, and you validate it against the platform&#8217;s published keys.</p>



<p class="wp-block-paragraph">Almost everyone gets that far. The failure is what happens next.</p>



<p class="wp-block-paragraph">The launch token carries a stable subject identifier, a roles claim, a context claim describing the course, and a deployment identifier, all namespaced under the 1EdTech spec URIs. Teams validate the signature, pull the user ID out, then build the rest of the session from whatever the front end sends afterwards. Course ID as a query parameter. Role in local storage. It works in testing because the front end sends the truth, and it stops working the moment a curious student opens developer tools. If your filter values arrive from the client, you have built an access control system that asks the attacker what they should be allowed to see.</p>



<h3 class="wp-block-heading">Take the scope from the launch, then keep it server-side</h3>



<p class="wp-block-paragraph">The pattern that holds up:</p>



<ol class="wp-block-list">
<li>Validate the launch JWT against the platform&#8217;s key set. Check the issuer, the audience, the nonce and the deployment identifier, not just the signature.</li>



<li>Extract subject, roles and context from the token and write them into a server-side session. Hand the browser an opaque session cookie and nothing else.</li>



<li>Derive every retrieval filter from that session record. The client never names a course, a section or a role.</li>



<li>If you need the full roster rather than just the launching user, call Names and Role Provisioning Services with an OAuth 2.0 client credentials token. That gives you learners who have never launched the tool, which matters for cohort-level features.</li>



<li>If you write grades back, do it through Assignment and Grade Services against a line item you created. A launch that works tells you nothing about whether passback works.</li>
</ol>



<p class="wp-block-paragraph">Test launch, roster fetch and grade passback as three independent contracts. They fail independently and they fail quietly. A term can start with a tool that launches beautifully and drops every grade on the floor because AGS was never wired to a line item.</p>



<p class="wp-block-paragraph">Worth knowing before you commit: LTI 1.3 is meaningfully more setup than 1.1. Key registration, the OIDC handshake, per-deployment configuration on every platform instance. That is the price of not signing student traffic with a shared secret, and it is worth paying, but budget for it honestly. If you need to test against several platforms, a self-managed Moodle or Open edX instance on a cheap VPS from somewhere like Contabo or InterServer beats waiting on sandbox tenants, and you get to break it freely.</p>



<h2 class="wp-block-heading">Failure family two: retrieval that ignores the boundary</h2>



<p class="wp-block-paragraph">This is the one from the opening. Vector search does not know about courses. It knows about similarity. If the answer key is semantically close to the question, it comes back, and the model uses it, and the response is fluent and correct and completely inappropriate.</p>



<p class="wp-block-paragraph">Amazon Bedrock Knowledge Bases handles this through metadata filtering: you attach attributes to each document at ingestion, then pass a filter at query time that pre-filters the vector store before the similarity search runs. The filter is the control. Everything else is decoration.</p>



<h3 class="wp-block-heading">The sidecar file is the only metadata that counts</h3>



<p class="wp-block-paragraph">For an S3 data source, attributes come from a companion file sitting next to the document with <code>.metadata.json</code> appended to the object key. This trips people up constantly, because S3 objects already have user-defined metadata and object tags, and neither of those is read for retrieval filtering. Keep lifecycle and cost allocation in S3 tags where they belong, and put filterable attributes in the sidecar.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "course_id": "BIO201",
    "section_id": "BIO201-B",
    "institution_id": "uni-main",
    "material_type": "lecture_notes",
    "visibility": "student"
  }
}</code></pre>



<p class="wp-block-paragraph">Note <code>visibility</code> as a first-class attribute rather than an inference from folder structure. Rubrics, model answers and marking schemes get <code>visibility: "instructor"</code>, and the student-facing filter excludes them by value, not by path convention. Path conventions get violated the first time a busy tutor uploads a file to the wrong folder.</p>



<p class="wp-block-paragraph">At query time you compose the filter server-side from the session record. Bedrock&#8217;s retrieval configuration supports boolean composition, so the tenant condition and the visibility condition are separate clauses that must both hold:</p>



<pre class="wp-block-code"><code>{
  "vectorSearchConfiguration": {
    "filter": {
      "andAll": [
        { "equals": { "key": "institution_id", "value": "uni-main" } },
        { "equals": { "key": "course_id",      "value": "BIO201"   } },
        { "equals": { "key": "visibility",     "value": "student"  } }
      ]
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two habits make this survive production. Build the filter through a single function that always prepends institution and course, so no future endpoint can forget it. Then write a test that queries a known instructor-only chunk&#8217;s exact wording from a student session and asserts zero results. Manual smoke testing never catches this, because human testers ask reasonable questions and the failure only appears when someone asks an unreasonable one.</p>



<h3 class="wp-block-heading">Silo, pool or bridge</h3>



<p class="wp-block-paragraph">AWS frames multi-tenant isolation as three patterns, and the trade-off is the same one you always get between blast radius and cost:</p>



<ul class="wp-block-list">
<li><strong>Silo</strong> gives each institution its own knowledge base and its own vector index. Strongest isolation, easiest compliance conversation, and you multiply your idle infrastructure cost by the number of tenants. There is also a per-account service quota on knowledge bases, so check Service Quotas before you promise a hundred districts their own.</li>



<li><strong>Pool</strong> puts everyone in one index and relies entirely on metadata filtering. Cheapest by a wide margin, and one filter bug is a cross-institution disclosure rather than an embarrassment.</li>



<li><strong>Bridge</strong> splits the difference: separate knowledge bases sharing underlying vector infrastructure, which keeps the isolation story cleaner without paying for a full stack per tenant.</li>
</ul>



<p class="wp-block-paragraph">For a single university running many courses, pool with rigorous filtering is usually right, because the institution is the tenant and the courses are not. For an edtech vendor selling to multiple districts, I would start at bridge and only pool if the economics force it, because the conversation with a district&#8217;s data protection officer goes very differently when your answer is &#8220;we filter&#8221; rather than &#8220;they are separate indexes.&#8221;</p>



<h2 class="wp-block-heading">Failure family three: guardrails that do less than you assume</h2>



<p class="wp-block-paragraph">Bedrock Guardrails is a useful layer and the first thing I reach for on an education workload: content filters, denied topics, word filters, PII detection and contextual grounding checks, applied at inference time and usable across models. For a system that teenagers will deliberately try to provoke, enforcement inside the invocation path beats anything bolted on afterwards. Three limits are documented and routinely missed.</p>



<p class="wp-block-paragraph"><strong>Contextual grounding does not cover chatbots.</strong> AWS documents the supported use cases as summarization, paraphrasing and question answering, and states plainly that conversational QA and chatbot use cases are not supported. A tutoring assistant is a multi-turn conversation. If your hallucination story is &#8220;we turned on grounding checks,&#8221; read that page before it reaches a procurement document. The check still earns its place on single-shot retrieve-and-answer paths, which is where I would apply it, but a multi-turn tutor needs citation display and human review too.</p>



<p class="wp-block-paragraph"><strong>PII detection is probabilistic and text-only.</strong> The sensitive information filter is a machine learning model that uses surrounding context to decide whether something is an identifier. It works better on prose than on bare fragments. AWS also documents that it operates on text output and does not inspect tool use parameters, so if your assistant calls a function with a student identifier in the arguments, that path is not covered. Do not treat it as a substitute for not putting identifiers in the prompt in the first place.</p>



<p class="wp-block-paragraph"><strong>Blocked content lands in your logs.</strong> AWS documents that content blocked by guardrail policies appears as plain text in Bedrock model invocation logs when those logs are enabled. That is defensible for debugging and awkward when the blocked content is a distressed message from a fourteen year old. Decide deliberately whether invocation logging is on, where it lands, who can read it, and how long it lives. In education this is a governance decision, not a logging default.</p>



<p class="wp-block-paragraph">None of this argues against Guardrails. It argues for knowing its edges, because the gap between what a feature name suggests and what the documentation says is where compliance commitments go to die.</p>



<h2 class="wp-block-heading">Failure family four: the cost shape of an academic term</h2>



<p class="wp-block-paragraph">Education traffic is not a smooth curve. It is near zero over summer, spikes brutally in the week before every assessment deadline, and dies again. That shape punishes the default architecture in a specific way.</p>



<p class="wp-block-paragraph">The console nudges you toward Amazon OpenSearch Serverless as the vector store. It bills per OpenSearch Compute Unit hour with a minimum baseline footprint that exists whether or not anyone queries it, so an empty index over a quiet August still bills. AWS has adjusted those minimums over time, so pull current figures from the OpenSearch Serverless capacity documentation rather than from any blog post, this one included. The mechanism is the part that matters: your floor is set by capacity that exists, not by traffic that happens.</p>



<p class="wp-block-paragraph">Bedrock Knowledge Bases supports other backends, and the question that decides between them is idle cost:</p>



<ul class="wp-block-list">
<li><strong>Aurora PostgreSQL with pgvector.</strong> A first-class documented option. Aurora Serverless v2 bills per Aurora Capacity Unit hour and can scale down to zero when idle, which fits the academic calendar almost too neatly. You take on index lifecycle work in exchange: choosing between HNSW and IVFFlat, keeping statistics fresh, watching bloat. If your institution already runs Postgres, that is work your DBAs recognize.</li>



<li><strong>Amazon S3 Vectors.</strong> Bills on storage and requests rather than provisioned capacity, so genuine idle costs approach nothing. The trade is retrieval latency, which is higher than a warm search cluster. For a tutor where a response already takes a second or two of model time, that may not be the bottleneck you think it is.</li>



<li><strong>OpenSearch Serverless.</strong> The right answer when retrieval latency is genuinely your constraint, when you need hybrid search and k-NN tuning, or when you already run collections for other workloads and the baseline is already paid for.</li>
</ul>



<p class="wp-block-paragraph">On the model side, the cost driver is retrieved context, not the question. Every turn ships the retrieved chunks as input tokens, so chunk size and the number of results retrieved multiply directly into your bill on every single message. Halving retrieved results is usually a bigger lever than switching to a cheaper model, and it often improves answer quality too, because the model has less irrelevant material to get distracted by.</p>



<p class="wp-block-paragraph">Tag by course and faculty from day one. The first budget question you will be asked is which department is driving the spend, and retrofitting cost allocation tags after a pilot goes institution-wide is miserable. Cost Explorer answers it if the tags exist; tools like Vantage or CloudZero make the per-course view easier to hand to a finance team that does not want an AWS login.</p>



<h2 class="wp-block-heading">Failure family five: no evidence when someone asks</h2>



<p class="wp-block-paragraph">FERPA governs education records in the United States, and the mechanism that lets a vendor touch them at all is the school official exception: the institution must maintain direct control over the vendor&#8217;s use and maintenance of the records. Similar logic applies under other regimes, and many districts and state contracts add data residency requirements that FERPA itself does not impose.</p>



<p class="wp-block-paragraph">What that means architecturally is less exotic than it sounds, and it is mostly about being able to answer questions after the fact:</p>



<ul class="wp-block-list">
<li>Log the retrieval, not just the answer. Store which chunk identifiers were returned for each turn. When an instructor claims the assistant leaked a rubric, the retrieval log is the only thing that settles it.</li>



<li>Keep student identifiers out of prompts. Pass an opaque session-scoped pseudonym and resolve it in your own system. The model does not need a real name to be helpful.</li>



<li>Encrypt with customer managed KMS keys and use encryption context that names the tenant, so a key policy can actually constrain who decrypts what.</li>



<li>Pin the region and enforce it with service control policies if residency is contractually required. An SCP is checkable evidence; a runbook is not.</li>



<li>Write down retention for conversation history and enforce it with lifecycle rules. &#8220;We keep it forever because storage is cheap&#8221; is a defensible engineering position and an indefensible one in an education procurement review.</li>



<li>Pull the current compliance reports and agreements from AWS Artifact rather than quoting a blog about what AWS certifies.</li>
</ul>



<p class="wp-block-paragraph">None of this is glamorous and all of it is cheaper to build in during the pilot than to retrofit when the institution&#8217;s legal team gets involved.</p>



<h2 class="wp-block-heading">Troubleshooting the things that actually page you</h2>



<h3 class="wp-block-heading">&#8220;The new material isn&#8217;t showing up&#8221;</h3>



<p class="wp-block-paragraph">Almost always ingestion, not retrieval. Walk it in order rather than guessing. Did a sync actually run after the change? For an S3 data source, an edit is invisible until an ingestion job processes it.</p>



<pre class="wp-block-code"><code>aws bedrock-agent list-ingestion-jobs 
  --knowledge-base-id KB_ID 
  --data-source-id DS_ID</code></pre>



<p class="wp-block-paragraph">Look for a completed job starting after the upload. If the most recent job predates the change, your scheduler or event handler is the bug and retrieval is innocent. If a job did run, read its statistics: a modified-documents-indexed count of zero when you expected a change means change detection did not fire, which happens when an object is re-uploaded with identical bytes. A non-zero failed count means your file is probably the one that failed, and the failure reasons say why.</p>



<h3 class="wp-block-heading">&#8220;The filter returns nothing at all&#8221;</h3>



<p class="wp-block-paragraph">Query once with no filter to confirm the content is indexed. If it comes back unfiltered and vanishes when filtered, it is a metadata problem: the sidecar was not ingested, the attribute name does not match exactly, or the value differs in case. These comparisons are literal. <code>BIO201</code> and <code>bio201</code> are two different courses as far as the filter is concerned, which is an argument for normalizing identifiers at both ingestion and query time in one shared helper.</p>



<h3 class="wp-block-heading">&#8220;The launch shows a blank iframe&#8221;</h3>



<p class="wp-block-paragraph">Two suspects dominate. Cookies: your tool is in a cross-site iframe, so session cookies need the right SameSite and Secure attributes or the browser drops them and every launch looks like a fresh unauthenticated user. Registration mismatch: issuer, client ID or deployment ID differs between what the platform sends and what you stored. Log the received issuer and deployment ID on validation failure. Guessing at OIDC failures is slow; the values are right there.</p>



<h3 class="wp-block-heading">&#8220;Answers are right but the citations are wrong&#8221;</h3>



<p class="wp-block-paragraph">Usually chunking. Slides and worksheets fragment badly under naive fixed-size chunking, and a chunk that starts mid-table cites a heading that belongs to something else. Before you change models, look at what the retriever actually returned for the query that misbehaved. Nine times out of ten the model was faithful to a bad chunk.</p>



<h3 class="wp-block-heading">&#8220;Grades aren&#8217;t reaching the gradebook&#8221;</h3>



<p class="wp-block-paragraph">Check that a line item exists and that your service token carries the right AGS scopes. Passback failures are asynchronous, so they surface as an empty gradebook column weeks later rather than as an error the student sees. Alert on passback failure rate explicitly; nobody will report it in time.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Enforcing access control in the system prompt.</strong> Instructions are not permissions. If a chunk can be retrieved, assume it can be surfaced.</li>



<li><strong>Trusting client-supplied course or role values.</strong> Derive scope from the validated launch token and store it server-side, always.</li>



<li><strong>Ingesting the whole course folder.</strong> Instructor material and student material in one prefix is how rubrics leak. Separate at ingestion, tag at ingestion.</li>



<li><strong>Putting real names and student numbers in prompts.</strong> Pseudonymize before the model call and resolve afterwards.</li>



<li><strong>Sizing the vector store from peak week.</strong> The bill is set by the idle floor, and in education idle is most of the year.</li>



<li><strong>Assuming a guardrail feature does what its name implies.</strong> Read the supported use cases. Contextual grounding on a chatbot is the classic example.</li>



<li><strong>Piloting with one enthusiastic lecturer.</strong> That lecturer&#8217;s material is unusually well organized. The second department will not be.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Build the retrieval filter in exactly one function, and make the tenant and visibility clauses impossible to omit.</li>



<li>Keep a small regression set of adversarial queries, including students asking for solutions and asking about classmates, and run it on every deployment.</li>



<li>Show citations in the interface. Instructors trust the system faster when they can see the source, and it converts silent hallucinations into reportable ones.</li>



<li>Give instructors a preview mode that answers as a student would see it. They will find scoping errors far faster than any test suite.</li>



<li>Route the tool endpoint through a CDN and WAF such as Cloudflare, with rate limiting per session. Assessment week is a load test you did not schedule.</li>



<li>Dashboard retrieval count, token spend per course and passback failure rate together. CloudWatch metrics fronted by Amazon Managed Grafana or Grafana Cloud gives faculty a view without handing out console access.</li>



<li>Have an escalation path for distressing content that reaches a human being, and make sure the people on it know they are on it. This is a policy question you must answer before launch, not after.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do we need to fine-tune a model for our curriculum?</h3>



<p class="wp-block-paragraph">Almost certainly not. Curriculum content changes every term, and retrieval updates in minutes where fine-tuning is a rebuild. Fine-tuning earns its place for style and format consistency, not for facts. Start with retrieval, and revisit only if you have a specific behavior that prompting genuinely cannot produce.</p>



<h3 class="wp-block-heading">How do we stop it from just doing the homework?</h3>



<p class="wp-block-paragraph">Partly retrieval scope, partly prompting, partly accepting you cannot fully win. Exclude solutions and rubrics from student-visible retrieval, prompt for Socratic responses on assessed material, flag the assessment context on the way in so the assistant knows which mode it is in. A determined student can paste the question into a consumer chatbot regardless, which is a course design question rather than an infrastructure one.</p>



<h3 class="wp-block-heading">Is Bedrock the right choice versus calling a model API directly?</h3>



<p class="wp-block-paragraph">If the institution is already on AWS, Bedrock keeps the traffic inside your account boundary, gives you IAM and CloudTrail on the same footing as everything else, and can be reached over VPC endpoints. A direct provider API is often simpler and sometimes gets new models sooner. The deciding factor is usually procurement: one existing AWS agreement is a much shorter conversation than a new vendor review.</p>



<h3 class="wp-block-heading">Does student data get used to train the model?</h3>



<p class="wp-block-paragraph">AWS states that Bedrock does not use customer inputs or outputs to train foundation models. Verify it against current AWS documentation and your agreement rather than any third-party summary, because this is exactly the clause a district will ask you to evidence in writing.</p>



<h3 class="wp-block-heading">What does it cost to run a personalized learning assistant on AWS?</h3>



<p class="wp-block-paragraph">Model your bill as three parts: a vector store floor that bills whether or not anyone asks anything, a one-off embedding cost when you ingest the corpus, and per-turn inference dominated by retrieved context rather than by the question. For a small corpus with a term-shaped traffic curve, the floor usually dominates, which is why the vector store choice matters more than the model choice.</p>



<h3 class="wp-block-heading">How do we know whether it actually helps anyone learn?</h3>



<p class="wp-block-paragraph">Decide the measure before launch and instrument for it. Usage is not learning. Pair the assistant&#8217;s logs with assessment outcomes on comparable cohorts and accept that a clean answer needs a study design, not a dashboard. Engineering can supply the data; it cannot supply the conclusion.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A personalized learning assistant on AWS will never tell you when it is wrong about who someone is. The model has no concept of enrollment, and retrieval has no concept of a due date. Both live entirely in the metadata you attach at ingestion and the filter you build server-side from a validated launch token. Get that right and what remains is ordinary engineering: chunking, cost curves, dashboards. Get it wrong and you ship something that works flawlessly right up until the week it embarrasses the institution.</p>



<p class="wp-block-paragraph">Build the boundary first. Make it a control, not an instruction. Then go be clever with the tutoring.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand building or auditing one of these?</h2>



<p class="wp-block-paragraph">I work on the infrastructure side of AI assistants for education, which mostly means the parts that decide whether the thing is safe to put in front of students:</p>



<ul class="wp-block-list">
<li>Designing and reviewing the LTI 1.3 launch path, roster access and grade passback, including the cross-site cookie and registration issues that cause blank-iframe launches.</li>



<li>Building the retrieval isolation layer: sidecar metadata schemas, server-side filter composition, and an adversarial regression suite that proves instructor material stays invisible to students.</li>



<li>Choosing and migrating the vector store when the OpenSearch Serverless floor is eating a pilot budget, including Aurora pgvector and S3 Vectors options.</li>



<li>Guardrail configuration that matches the documented behavior rather than the feature name, plus a logging and retention design you can defend in a procurement review.</li>



<li>Cost allocation tagging by course and faculty, with dashboards that answer &#8220;which department is spending this&#8221; without handing out AWS console access.</li>



<li>Infrastructure as code for the whole stack so a second institution is a variable file rather than a fortnight.</li>
</ul>



<p class="wp-block-paragraph">If you have something concrete, send it over: a retrieval filter you are not sure is airtight, an ingestion job that keeps skipping documents, a failing LTI launch log, or a Cost Explorer breakdown that stopped making sense. Easier to look at the real thing than to talk in the abstract.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/personalized-learning-assistant-aws/">Building a Personalized Learning Assistant on AWS Without Leaking Student Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AI-Generated IAM Policies: Where the Hard Boundary Belongs</title>
		<link>https://john-nessime.com/blog/devops/ai-generated-iam-policies/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 13 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[AI-Assisted Coding]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cloudsplaining]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[IAM Access Analyzer]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Parliament]]></category>
		<category><![CDATA[Permissions Boundaries]]></category>
		<category><![CDATA[Policy Generation]]></category>
		<category><![CDATA[Policy Validation]]></category>
		<category><![CDATA[Privilege Escalation]]></category>
		<category><![CDATA[Service Control Policies]]></category>
		<category><![CDATA[Trust Policies]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=858</guid>

					<description><![CDATA[<p>A generated policy with no wildcards can still hand over your account. Here are the three failure shapes that survive human review, why the review step is the wrong place to put your defence, and how to build a ceiling that holds regardless of what the model writes.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/ai-generated-iam-policies/">AI-Generated IAM Policies: Where the Hard Boundary Belongs</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The policy looked fine. No action wildcard, no resource wildcard, a Sid on every statement, formatting tidier than anything I write by hand. It came out of a chat window, went into a pull request, and got two approvals in about ninety seconds.</p>



<p class="wp-block-paragraph">It granted <code>iam:PassRole</code> on one role ARN and <code>lambda:CreateFunction</code> in the same document. That is not a wildcard and it does not read as administrative access. It is administrative access anyway, to whatever that role can do, for anyone who can write a Lambda function.</p>



<p class="wp-block-paragraph">This is the shape of the problem with AI-generated IAM policies. The failure is almost never the obvious one. Models have absorbed enough style guidance to avoid <code>"Action": "*"</code>, so what comes back passes the eyeball test and the linter and still hands over more than you meant to give. The dangerous output is the one that looks careful.</p>



<p class="wp-block-paragraph">This post covers what policy generation genuinely does well, the failure shapes that survive human review, and where the hard boundary belongs: not in the review step, which is fallible, but in a mechanism that cannot be talked out of its answer.</p>



<h2 class="wp-block-heading">What AI-generated IAM policies get right</h2>



<p class="wp-block-paragraph">Start with the honest case, because it is a strong one. IAM has thousands of actions across hundreds of services and the naming is inconsistent in ways nobody holds in their head. Some services use <code>Describe</code>, some use <code>Get</code>, some use both for different things. ARN formats vary per service and per resource type inside a service. Condition keys apply to some actions and not others.</p>



<p class="wp-block-paragraph">A model is good at that recall problem, and at shape work: splitting a monolithic statement into per-resource statements, adding conditions, converting an inline policy into Terraform. Paste an <code>AccessDenied</code> message and you get a plausible read of which layer refused. So the argument is not that the tooling is useless. It is that recall and reasoning about consequences are different capabilities, and IAM is a system where the second one protects you.</p>



<h2 class="wp-block-heading">The failure modes that survive review</h2>



<h3 class="wp-block-heading">Permissions that compose into something bigger</h3>



<p class="wp-block-paragraph">Every action is individually defensible. The escalation lives in the combination.</p>



<p class="wp-block-paragraph">The canonical case is <code>iam:PassRole</code> beside any service that runs code: Lambda, EC2, ECS, Glue. Passing a role to a compute service means your code executes with that role&#8217;s permissions, so your ceiling is not your own policy, it is whatever the most powerful passable role can do. Rhino Security Labs catalogued a long list of these chains and every one of them is built from permissions that look unremarkable alone.</p>



<p class="wp-block-paragraph">Another that catches people: <code>iam:CreatePolicyVersion</code> on a policy you are attached to. A new version normally needs to be set as default to take effect, but the create call accepts a flag that makes it default immediately, and that flag does not require <code>iam:SetDefaultPolicyVersion</code>. A permission that reads as &#8220;can update policy documents&#8221; reads to an attacker as &#8220;can write myself an admin policy.&#8221;</p>



<p class="wp-block-paragraph">Ask a model whether a policy grants admin and it will check the statements and say no, correctly, one at a time. Composition is the part it does not reliably do, and it is the part that matters.</p>



<h3 class="wp-block-heading">Actions that do not exist</h3>



<p class="wp-block-paragraph">Plausible action names that no service publishes. <code>s3:ListBucketContents</code> instead of <code>s3:ListBucket</code>. IAM accepts a policy containing an unrecognized action string because the document is syntactically valid, so nothing fails at attach time.</p>



<p class="wp-block-paragraph">What happens instead is worse. The application throws <code>AccessDenied</code> in production, somebody debugging under pressure decides the granular list must be wrong, and the fix is a service wildcard. The invented action does not grant too much. It causes a human to grant too much a week later, and by then nobody connects the two events.</p>



<h3 class="wp-block-heading">Resource ARNs that look constrained and are not</h3>



<p class="wp-block-paragraph">An ARN with real characters in it feels safer than a star. Often it is not. The common versions are a trailing wildcard placed one segment too high, and an action whose resource element the service ignores. <code>s3:ListAllMyBuckets</code> is account-scoped no matter what you put in <code>Resource</code>. Several <code>ec2:Describe</code> actions behave the same way. A policy can read as resource-scoped throughout and be account-wide for the actions that count.</p>



<p class="wp-block-paragraph">The related trap is the missing condition: cross-account trust policies without an external ID, bucket policies without <code>aws:PrincipalOrgID</code>, integration roles without <code>aws:SourceIp</code> where the caller has a stable egress address. Generated policies include conditions when you ask and omit them when you do not, because an absent constraint never produces an error.</p>



<h2 class="wp-block-heading">Four things the model cannot know</h2>



<p class="wp-block-paragraph">Some of this is not a capability gap you can prompt past. It is missing input:</p>



<ul class="wp-block-list"><li><strong>Which resources exist.</strong> A bucket name is a guess unless you supplied it. Guessed names either fail closed or, if the wildcard is loose enough, match something you never intended.</li><li><strong>Which resources are sensitive.</strong> Your backup bucket and your public assets bucket are indistinguishable strings. Sensitivity lives nowhere in the document.</li><li><strong>What else is attached.</strong> Effective permissions come from identity policies, resource policies, boundaries, SCPs and session policies together. Reviewing one document tells you very little.</li><li><strong>What the blast radius is.</strong> &#8220;Can delete objects&#8221; means one thing for a scratch bucket and another for the only copy of a client&#8217;s data.</li></ul>



<p class="wp-block-paragraph">The training material is also skewed. Public IAM examples, including plenty of vendor documentation and countless forum answers, are permissive because permissive examples work on the first try. A model that has read the internet has read a lot of over-broad policies presented as correct.</p>



<h2 class="wp-block-heading">Where the hard boundary belongs</h2>



<p class="wp-block-paragraph">Review is not a boundary. Review is a filter with a pass rate that drops the more policies you look at in a week, and generated policies are cheap to produce, so volume rises exactly when attention falls.</p>



<p class="wp-block-paragraph">The boundary has to be something that says no without reading the document. AWS gives you two mechanisms and they are not interchangeable.</p>



<p class="wp-block-paragraph">A <strong>permissions boundary</strong> is a managed policy attached to a user or role that caps what identity-based policies can grant it. It grants nothing itself. Effective permissions become the intersection of the identity policy and the boundary, so an identity policy allowing <code>iam:*</code> against a boundary that omits IAM produces no IAM access at all. If no boundary is attached, none is evaluated, so this is opt-in per identity.</p>



<p class="wp-block-paragraph">A <strong>service control policy</strong> sets the maximum available permissions for every principal in an account or organizational unit, and also grants nothing. It is the right home for things nobody in that account should ever do, and it applies whether or not somebody remembered to attach a boundary.</p>



<p class="wp-block-paragraph">The division I use: SCPs carry account-wide invariants, such as denying escalation-relevant IAM write actions to everyone outside the break-glass role and denying regions you do not operate in. Boundaries carry the per-workload ceiling, and they are what makes it safe to let a pipeline role create other roles. You can permit that role to create roles while requiring, through a condition on the boundary key, that every role it creates carries your boundary. The pipeline can then generate whatever policy it likes, because the ceiling is enforced at evaluation time rather than at review time.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">If the only thing between a generated policy and production is a person reading JSON, you do not have a boundary. You have a habit.</p>
</blockquote>



<h2 class="wp-block-heading">The gate I would actually run</h2>



<p class="wp-block-paragraph">Mechanical checks in a pipeline, failing the build, not a checklist somebody is supposed to remember.</p>



<ol class="wp-block-list"><li><strong>Lint for nonsense.</strong> Parliament catches unknown action names, resource formats that cannot match the action, and type mismatches. Cheapest way to catch an invented action before it becomes a wildcard.</li><li><strong>Run AWS policy validation.</strong> Access Analyzer knows service-specific rules a generic linter does not.</li><li><strong>Assert the escalation actions are absent.</strong> Name the actions you never want granted and fail if any appear. This is the check that maps to the composition failure.</li><li><strong>Compare against the current policy.</strong> For edits, check whether the proposed document grants access the existing one does not. This one is billed per call, so run it on changed policies rather than the whole repository.</li><li><strong>Scan for known risky patterns.</strong> Cloudsplaining scores policies against categories including privilege escalation, data exfiltration and resource exposure, and produces something you can hand to a client.</li></ol>



<pre class="wp-block-code"><code># 1. Lint. Exit status is non-zero when there are findings.
parliament --file policy.json

# 2. AWS-side validation
aws accessanalyzer validate-policy 
  --policy-document file://policy.json 
  --policy-type IDENTITY_POLICY

# 3. Fail if any escalation-relevant action is granted
aws accessanalyzer check-access-not-granted 
  --policy-document file://policy.json 
  --access actions="iam:PassRole","iam:CreatePolicyVersion","iam:AttachRolePolicy" 
  --policy-type IDENTITY_POLICY

# 4. Does the proposed policy grant more than the current one?
aws accessanalyzer check-no-new-access 
  --existing-policy-document file://current.json 
  --new-policy-document file://proposed.json 
  --policy-type IDENTITY_POLICY

# 5. Risk-categorised report
cloudsplaining scan-policy-file --input-file policy.json</code></pre>



<p class="wp-block-paragraph">Step three is the one to add first if you only add one. It turns &#8220;we reviewed it&#8221; into &#8220;the build fails.&#8221;</p>



<p class="wp-block-paragraph">Attaching the boundary is a single call, and it is the part people skip because the role already works without it:</p>



<pre class="wp-block-code"><code>aws iam put-role-permissions-boundary 
  --role-name app-deploy-role 
  --permissions-boundary arn:aws:iam::123456789012:policy/WorkloadCeiling</code></pre>



<h2 class="wp-block-heading">Generate from observed behavior instead</h2>



<p class="wp-block-paragraph">There is a better source of truth than a model&#8217;s guess about what an application needs, and it is the application. Access Analyzer policy generation reads CloudTrail events for a role over a chosen window and writes a policy from what was actually called. Know its limits before relying on it:</p>



<ul class="wp-block-list"><li>It analyzes up to ninety days of history and needs a trail already logging for the account, so a role that only exercises certain paths quarterly produces an incomplete policy.</li><li>Coverage is per-service. For some services it identifies individual actions; for others it can only tell you the service was used and prompts you to fill in the actions yourself.</li><li>It does not produce action-level detail for data events such as S3 object-level operations.</li><li><code>iam:PassRole</code> is not included in generated policies. If your workload needs it, a human adds it back by hand, so the most escalation-relevant line in the document is the one the generator did not write.</li></ul>



<p class="wp-block-paragraph">Behavior-derived policies also fail in a useful direction. A missing permission produces a denial you can see in CloudTrail, which beats a permission you did not know you granted.</p>



<h2 class="wp-block-heading">Arguments that do not survive contact</h2>



<ul class="wp-block-list"><li><strong>&#8220;We review every policy before merge.&#8221;</strong> Review quality is a function of volume and attention, and generation raises volume. Composition failures are also the hardest to spot by reading, because spotting them means knowing what other roles exist and what they can do.</li><li><strong>&#8220;Access Analyzer came back clean.&#8221;</strong> Validation checks the document. It does not know that the role ARN you may pass happens to have <code>AdministratorAccess</code> attached. Clean findings on a policy that grants a dangerous chain is the normal case, not an anomaly.</li><li><strong>&#8220;A permissions boundary is overkill for a small project.&#8221;</strong> Small projects are where this bites hardest. One account, one deploy role, no organization, nothing between a bad policy and everything you own. The boundary is one managed policy and one CLI call.</li><li><strong>&#8220;We will tighten it after launch.&#8221;</strong> Nobody does. If you genuinely intend to, drive it from Access Analyzer&#8217;s unused access findings so you have a work queue instead of an intention.</li><li><strong>&#8220;Just prompt the model to follow least privilege.&#8221;</strong> That changes the output surface, not the reasoning. Fewer wildcards, better statements, no evaluation of what the combination permits.</li></ul>



<h2 class="wp-block-heading">How I would decide</h2>



<p class="wp-block-paragraph">My rule is about which actions are in the document, not about who or what wrote it.</p>



<p class="wp-block-paragraph"><strong>Ship generated policies freely</strong> when the document touches only data-plane actions on named resources, the identity has a boundary, and the pipeline checks pass. Read a bucket prefix, publish to a topic, write to a table. That is most policies, and hand-writing them wastes your afternoon.</p>



<p class="wp-block-paragraph"><strong>Treat generated output as a first draft</strong> when the policy includes any IAM write action, any <code>PassRole</code>, any trust policy edit, any resource policy readable from outside the account, or any KMS key policy. Use it to save typing, then verify every statement against the service authorization reference yourself.</p>



<p class="wp-block-paragraph"><strong>Do not generate at all</strong> for the boundary policies and SCPs. Those are the mechanism. Something that constrains everything else should be short, hand-written, understood by whoever owns the account, and changed rarely.</p>



<p class="wp-block-paragraph">Two notes for small engagements. If you run application infrastructure on a VPS from a provider like Contabo or InterServer and pull AWS in for storage or mail, that server holds long-lived credentials somewhere your AWS controls do not reach, so scope its policy to a bucket prefix and attach a boundary before worrying about anything else. And if the credentials go to a SaaS connector, a marketing platform such as GoHighLevel or a payments integration, you are handing keys to a system whose behavior you cannot audit. Those are the policies to write by hand.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Are AI-generated IAM policies safe to use in production?</h3>



<p class="wp-block-paragraph">For data-plane permissions on named resources, generally yes, provided the identity carries a permissions boundary and the policy passes automated validation. For anything involving IAM write actions, <code>PassRole</code>, trust policies or key policies, treat the output as a draft and verify each statement yourself. What matters is the blast radius of the actions, not the authorship.</p>



<h3 class="wp-block-heading">Why does a generated policy pass validation and still grant too much?</h3>



<p class="wp-block-paragraph">Validation examines one document in isolation. Privilege escalation usually comes from a combination of individually reasonable permissions, or from the interaction between the policy and something outside it, such as which roles are passable and what those roles can do. A document can be internally correct and still open a path.</p>



<h3 class="wp-block-heading">What is the difference between a permissions boundary and an SCP?</h3>



<p class="wp-block-paragraph">Both cap permissions and neither grants any. A boundary attaches to a single IAM user or role and limits what that identity&#8217;s policies can grant it. An SCP applies to every principal in an account or organizational unit and needs AWS Organizations. With one account and no organization, boundaries are what you have. With an organization, use both.</p>



<h3 class="wp-block-heading">Can IAM Access Analyzer replace writing policies by hand?</h3>



<p class="wp-block-paragraph">Partly. Generation from CloudTrail activity beats a guess because it reflects what the workload actually called. It is limited by the analysis window, by per-service coverage differences, by the absence of action-level detail for data events, and by <code>iam:PassRole</code> being excluded. Treat it as a strong draft that needs a human pass over anything the generator could not see.</p>



<h3 class="wp-block-heading">Which IAM actions should I block outright in a small account?</h3>



<p class="wp-block-paragraph">The ones that let a principal rewrite its own permissions or borrow another identity&#8217;s: creating and setting default policy versions, attaching and putting inline policies on users, roles and groups, creating access keys for other users, and passing roles to compute services. Deny them for everything except a break-glass identity, then grant them back deliberately where a workload genuinely needs one.</p>



<h3 class="wp-block-heading">How do I catch an invented action name before production?</h3>



<p class="wp-block-paragraph">Run a linter that carries the IAM action catalogue. Parliament flags unknown actions and exits non-zero when it finds anything, so it drops straight into a pipeline step. Catching it early matters less because the fake action is dangerous and more because the eventual fix for the denial tends to be a service-level wildcard.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">AI-generated IAM policies are not more dangerous because a model wrote them. They are more dangerous because they arrive faster than you can think about them, and because they arrive looking finished.</p>



<p class="wp-block-paragraph">So stop trying to make the review step better. Put the boundary somewhere that does not depend on anyone paying attention: a permissions boundary on every identity a pipeline can create or modify, an SCP denying the escalation actions to everyone who does not need them, and a build step that fails when a forbidden action shows up. Then let generation happen at whatever speed it wants.</p>



<p class="wp-block-paragraph">The policy approved in ninety seconds is not the failure. The absence of anything underneath it was.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help drawing that boundary in your account?</h2>



<p class="wp-block-paragraph">Most of the IAM work I take on is exactly this: an account that grew organically, permissions nobody wants to touch in case something breaks, and no ceiling underneath any of it. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing existing roles and policies for privilege escalation chains, including the passable-role paths that never show up in a single-document review</li><li>Designing and attaching permissions boundaries for pipeline, workload and human identities, including the delegation pattern that lets a deploy role create roles safely</li><li>Writing the SCP set for a small organization: escalation-relevant IAM actions, region restrictions, and protection for logging and billing resources</li><li>Adding Access Analyzer validation and custom policy checks to a CI pipeline so forbidden actions fail the build instead of relying on review</li><li>Replacing over-broad roles with policies generated from CloudTrail activity, then closing the gaps the generator cannot see</li><li>Scoping down credentials handed to third-party SaaS integrations and to servers outside AWS, where your account controls do not apply</li></ul>



<p class="wp-block-paragraph">If you want a second opinion on something specific, send me the policy JSON, the output of a failing <code>check-access-not-granted</code> run, or the role list from your account, and I will tell you what I would change and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/ai-generated-iam-policies/">AI-Generated IAM Policies: Where the Hard Boundary Belongs</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records</title>
		<link>https://john-nessime.com/blog/devops/salesforce-crm-knowledge-assistant-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 23 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[AgentCore]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Kendra]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Connected App]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Row-Level Security]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=243</guid>

					<description><![CDATA[<p>Connecting Salesforce to Amazon Bedrock Knowledge Bases takes an afternoon. Making sure the assistant does not read people records they were never allowed to see takes longer. A practical walkthrough of the connector, the permission model it quietly drops, and the two architectures that actually hold up.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-crm-knowledge-assistant-bedrock/">Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A support lead types a question into the assistant you demoed last week: &#8220;What&#8217;s the renewal value on the Northwind account?&#8221; It answers in about two seconds, correctly, with a citation back to the source record. Everyone in the room is pleased.</p>



<p class="wp-block-paragraph">Then someone points out that this particular support lead has never had access to Opportunity records. Not through her profile, not through the role hierarchy, not through a sharing rule. And the assistant just read one out to her.</p>



<p class="wp-block-paragraph">That is the failure that kills most CRM assistant projects, and it almost never shows up in testing, because you test as an admin. Everything you ask works. Everything you ask is supposed to work. The gap only opens when a real user with a real, narrow profile starts asking questions.</p>



<p class="wp-block-paragraph">This post walks through building a CRM knowledge assistant on Salesforce and Amazon Bedrock Knowledge Bases: which of the two knowledge base paths you are actually allowed to use, how the Salesforce connector authenticates, what it does and does not carry across from your CRM, and the two architectures that survive a real permission model. The wiring is the easy part. The scoping is the work.</p>



<h2 class="wp-block-heading">The three moving parts</h2>



<p class="wp-block-paragraph">Strip away the marketing and a CRM knowledge assistant is three things bolted together:</p>



<ul class="wp-block-list">
<li><strong>Ingestion</strong> — something pulls Cases, Knowledge articles, Opportunities and Accounts out of Salesforce on a schedule, chunks them, embeds them, and writes the vectors somewhere.</li>

<li><strong>Retrieval</strong> — a query comes in, gets embedded, and the closest chunks come back. This is where filtering lives, and it is the security surface.</li>

<li><strong>Generation</strong> — a foundation model reads those chunks and writes an answer with citations.</li>
</ul>



<p class="wp-block-paragraph">Generation is the part everyone stares at and the part you should worry about least. It is a model choice and a prompt. Retrieval is where the assistant becomes either trustworthy or a liability.</p>



<h2 class="wp-block-heading">Decision one: which knowledge base you are allowed to use</h2>



<p class="wp-block-paragraph">Bedrock now has two shapes of knowledge base, and this decision gets made for you the moment you say &#8220;Salesforce&#8221;.</p>



<p class="wp-block-paragraph"><strong>Bedrock Managed Knowledge Base</strong> is the newer, opinionated one. AWS manages the vector store, picks the embedding and reranking models by default, handles parsing, and exposes agentic multi-hop retrieval. It ships with a small set of first-party connectors: Amazon S3, SharePoint, Confluence, Google Drive, OneDrive, Web Crawler, and a custom ingestion path. Salesforce is not on that list.</p>



<p class="wp-block-paragraph"><strong>Customer-managed knowledge bases</strong> are the older shape. You bring the vector store, you pick the embedding model, you own more of the pipeline. This is where the Salesforce connector lives, and AWS still flags that connector as a preview release subject to change. It also has a hard constraint worth reading twice: with a Salesforce data source, Amazon OpenSearch Serverless is currently the only supported vector store. If your organisation has standardised on Aurora pgvector or something else, that standard does not apply here.</p>



<p class="wp-block-paragraph">So the fork is real. Either you use the preview Salesforce connector on the customer-managed path, or you land Salesforce data in S3 yourself and use the managed path. I will come back to why the second option is often the right call, because it is not just a workaround.</p>



<p class="wp-block-paragraph">One piece of context that matters if you are inheriting an older design: Amazon Kendra entered maintenance mode on 30 June 2026 and stopped accepting new customers on 30 July 2026. Kendra was the standard answer to &#8220;search my Salesforce org&#8221; for years, and AWS now points people at Bedrock Knowledge Bases instead. Existing indexes keep running and no end-of-service date has been set, but Kendra will not gain features. If a proposal on your desk starts with a Kendra index, that is worth raising before anyone writes code.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The permission model is the whole problem</h2>



<p class="wp-block-paragraph">Salesforce spends enormous effort on who can see what. Profiles, permission sets, the role hierarchy, org-wide defaults, criteria-based sharing rules, field-level security, restriction rules. A given Opportunity is visible to a specific, computed set of users, and that computation happens on every query.</p>



<p class="wp-block-paragraph">None of that survives ingestion.</p>



<p class="wp-block-paragraph">The AWS documentation is blunt about it, and this is the single most important sentence in the whole feature: everything you sync from your data source becomes available to anyone holding <code>bedrock:Retrieve</code> on that knowledge base, including content that had controlled permissions at the source. Three layers of Salesforce scoping collapse into one IAM action.</p>



<p class="wp-block-paragraph">Worse, the connector authenticates as a single identity. You configure a Connected App using the OAuth 2.0 client credentials flow, and that flow runs as one named user. Whatever that user can see, the crawler ingests. Pick a System Administrator to &#8220;make sure the sync works&#8221; and you have just copied the entire org into a flat index that your whole application tier can read.</p>



<h3 class="wp-block-heading">Why this is invisible until it isn&#8217;t</h3>



<p class="wp-block-paragraph">Nothing errors. There is no warning in the console, no failed sync, no denied API call. Retrieval returns chunks with good similarity scores and the model writes a fluent, cited answer. The only signal that something is wrong is a human recognising a record they should not be looking at, and that human is usually a customer-facing employee, not you.</p>



<p class="wp-block-paragraph">Treat the knowledge base as a published dataset, not as a view over Salesforce. Ask the question you would ask about an S3 bucket: if every row of this landed in front of every user of the app, who gets hurt?</p>



<h3 class="wp-block-heading">The two architectures that actually hold up</h3>



<p class="wp-block-paragraph"><strong>Option A: ingest only what everyone is allowed to see.</strong> Run the Connected App as a deliberately low-privilege integration user whose profile grants read access to Knowledge articles, published solutions, product data and price books, and nothing else. Exclude Contact, Lead, Case and Opportunity entirely. You get a genuinely useful support and enablement assistant, the permission question disappears, and you can ship it without a security review turning into a project.</p>



<p class="wp-block-paragraph">This is the one I reach for first. Most of the value people want from a CRM knowledge assistant is &#8220;what do we know about this product, this policy, this recurring problem&#8221;, and none of that needs record-level data.</p>



<p class="wp-block-paragraph"><strong>Option B: export to S3 with your own access metadata.</strong> If you genuinely need per-record answers, stop using the connector. Pull the objects you need through the Salesforce API or a scheduled extract, write each record to S3 as a document, and write a sidecar metadata file next to it carrying the access attribute you will filter on. Then point a knowledge base at the bucket and apply an explicit metadata filter on every retrieval, derived from the caller&#8217;s identity rather than from the query text.</p>



<p class="wp-block-paragraph">The cost is honest: you now own an extraction pipeline, a metadata schema, and the job of keeping the access attribute correct when a record changes owner. That is real work. The benefit is that you can point at exactly where the boundary is enforced and test it.</p>



<p class="wp-block-paragraph">What you cannot do is enforce the boundary in the prompt. &#8220;Only answer about accounts the user owns&#8221; is a suggestion to a language model, not an access control. Retrieval already happened by then.</p>



<h2 class="wp-block-heading">Wiring the Salesforce connector</h2>



<p class="wp-block-paragraph">Assuming you have chosen Option A and want the connector, here is the shape of it.</p>



<h3 class="wp-block-heading">On the Salesforce side</h3>



<ol class="wp-block-list">
<li>Create a Connected App in your org and enable the OAuth 2.0 client credentials flow.</li>

<li>Under the client credentials flow settings, set the <strong>Run As</strong> field to your integration user. This is the step people skip, and the sync fails without it. It is also the step that decides your entire blast radius, so pick that user before you pick anything else.</li>

<li>Copy the consumer key and consumer secret from the app&#8217;s OAuth settings.</li>

<li>Note your instance URL.</li>
</ol>



<h3 class="wp-block-heading">On the AWS side</h3>



<p class="wp-block-paragraph">Store the credentials in AWS Secrets Manager. The connector looks for three specific keys, and it will not fall back to alternatives if you name them differently:</p>



<pre class="wp-block-code"><code>{
  "consumerKey": "your Connected App client ID",
  "consumerSecret": "your Connected App client secret",
  "authenticationUrl": "https://yourcompany.my.salesforce.com/"
}</code></pre>



<p class="wp-block-paragraph">The secret has to live in the same AWS region as the knowledge base. Cross-region reads are not supported here, and the resulting error is generic enough to waste an hour. Do not reuse this secret for any other data source; rotate it on the same schedule as your other integration credentials.</p>



<h3 class="wp-block-heading">Creating the data source</h3>



<p class="wp-block-paragraph">The data source configuration is a JSON document. The interesting part is <code>patternObjectFilter</code>, which is how you keep the crawler away from objects you never wanted:</p>



<pre class="wp-block-code"><code>{
  "salesforceConfiguration": {
    "sourceConfiguration": {
      "hostUrl": "https://yourcompany.my.salesforce.com/",
      "authType": "OAUTH2_CLIENT_CREDENTIALS",
      "credentialsSecretArn": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:NAME"
    },
    "crawlerConfiguration": {
      "filterConfiguration": {
        "type": "PATTERN",
        "patternObjectFilter": {
          "filters": [
            {
              "objectType": "Knowledge__kav",
              "inclusionFilters": [".*"]
            },
            {
              "objectType": "Case",
              "exclusionFilters": [".*"]
            }
          ]
        }
      }
    }
  },
  "type": "SALESFORCE"
}</code></pre>



<p class="wp-block-paragraph">Two things to internalise about these filters. First, they are regular expressions matched against a per-object identifier, not against record contents: for <code>Knowledge__kav</code> that is the article title, for <code>Case</code> the case number, for <code>Attachment</code> the file name. Second, when an inclusion and an exclusion filter both match the same document, exclusion wins. That precedence is what makes an allow-list pattern safe to write.</p>



<p class="wp-block-paragraph">Then create the data source and attach it to an existing knowledge base:</p>



<pre class="wp-block-code"><code>aws bedrock create-data-source 
  --name "salesforce-knowledge" 
  --knowledge-base-id "YOUR_KB_ID" 
  --data-source-configuration file://salesforce-connector-config.json 
  --data-deletion-policy "DELETE" 
  --vector-ingestion-configuration '{"chunkingConfiguration":[{"chunkingStrategy":"FIXED_SIZE","fixedSizeChunkingConfiguration":[{"maxTokens":"300","overlapPercentage":"10"}]}]}'</code></pre>



<p class="wp-block-paragraph">A note on the command namespace: the agent-plane operations have historically lived under a separate CLI command group from the model-inference ones, and this has shifted between CLI versions. Run <code>aws bedrock help</code> against your installed version before assuming. The JSON payload is the stable part.</p>



<p class="wp-block-paragraph"><code>--data-deletion-policy DELETE</code> means the vectors get removed from the store when you delete the data source. The alternative retains them. Choose <code>DELETE</code> unless you have a specific reason not to, because orphaned vectors in an OpenSearch Serverless collection are both a cost line and a surprise waiting for whoever inherits this.</p>



<p class="wp-block-paragraph">After that, kick off a sync with <code>StartIngestionJob</code>, or hit <strong>Sync</strong> in the console. The connector does incremental syncs after the first run: added, updated and deleted content only. The first run crawls everything.</p>



<h2 class="wp-block-heading">Chunking is a one-way door</h2>



<p class="wp-block-paragraph">You cannot change the chunking strategy after connecting the data source. Not tune it, not swap it. You delete the data source and start over, which means re-crawling and re-embedding everything.</p>



<p class="wp-block-paragraph">So spend twenty minutes here rather than five. CRM content is not uniform. Knowledge articles are structured prose and behave like documentation. Case comment threads are short, conversational, and lose their meaning when split mid-exchange. Opportunity descriptions are often two sentences. A fixed-size chunk that suits one of these will mangle another.</p>



<p class="wp-block-paragraph">The practical move is to stop trying to find one strategy that fits everything. Create separate data sources for materially different content shapes, each with its own chunking configuration, all pointing at the same knowledge base. It costs you a bit of configuration and saves you a full re-index later.</p>



<p class="wp-block-paragraph">One more constraint: the Salesforce connector does not handle multimodal content. Tables, charts and diagrams embedded in your Knowledge articles are not going to be understood. If your best troubleshooting articles are mostly screenshots, this connector will quietly ingest the surrounding text and nothing else, and your retrieval quality will be worse than the article count suggests.</p>



<h2 class="wp-block-heading">Retrieval, filtering, and one trap</h2>



<p class="wp-block-paragraph">Bedrock gives you two kinds of metadata filter at query time, and confusing them is a security bug.</p>



<p class="wp-block-paragraph"><strong>Explicit filters</strong> are the ones you construct in code and pass in the retrieval configuration. Your application decides them from the authenticated caller. They are deterministic.</p>



<pre class="wp-block-code"><code>{
  "retrievalQuery": { "text": "renewal value on the Northwind account" },
  "retrievalConfiguration": {
    "vectorSearchConfiguration": {
      "numberOfResults": 8,
      "filter": {
        "andAll": [
          { "equals": { "key": "visibility_group", "value": "emea-enterprise" } },
          { "equals": { "key": "record_class", "value": "internal" } }
        ]
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph"><strong>Implicit filters</strong> are the other kind. You declare which metadata attributes exist and Bedrock uses a model to infer filters from the natural-language query. Someone asks about last quarter, it infers a date filter. This is a genuinely nice relevance feature and it is not an access control. A model inferring a filter from user-supplied text is a model that can be talked out of it. Use implicit filters to improve answers; use explicit filters to enforce boundaries.</p>



<p class="wp-block-paragraph">The trap: filters on the Salesforce connector operate on auto-detected fields from your CRM records. You cannot inject your own <code>visibility_group</code> attribute at ingestion. That capability comes from sidecar metadata files, which is an S3 data source feature. This is the concrete reason Option B routes through a bucket. If per-user scoping is a requirement, the connector cannot give it to you no matter how carefully you configure it.</p>



<h2 class="wp-block-heading">Where the money goes</h2>



<p class="wp-block-paragraph">Rates change, so learn the meters instead of memorising numbers.</p>



<ul class="wp-block-list">
<li><strong>Embedding at ingestion</strong> — billed per token processed. A full re-crawl re-embeds everything, which is exactly why the chunking decision is expensive to reverse.</li>

<li><strong>Vector storage</strong> — on the customer-managed path this is your OpenSearch Serverless collection, billed on provisioned compute units and storage, with a floor whether or not anyone queries it. This is the line item that surprises people on a proof of concept that nobody uses. Managed Knowledge Base bills on indexed data size and retrieval count instead, with no idle collection sitting underneath.</li>

<li><strong>Generation</strong> — input and output tokens on whichever foundation model you chose. Retrieved chunks are input tokens, so <code>numberOfResults</code> is a direct cost lever and worth tuning down until quality actually degrades.</li>

<li><strong>Everything around it</strong> — the API layer, the chat frontend, and the observability stack. A small VPS from a provider like InterServer or a Lambda behind API Gateway both work; the choice barely matters next to the Bedrock line. Push retrieval latency and citation counts into Grafana Cloud or Datadog early, because &#8220;the assistant feels worse this week&#8221; is otherwise unfalsifiable.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Sync fails immediately with an authentication error.</strong> Check the Run As user on the Connected App&#8217;s client credentials flow first. An empty Run As field is the most common cause. Then check that the secret keys are spelled exactly <code>consumerKey</code>, <code>consumerSecret</code>, <code>authenticationUrl</code>, and that the secret is in the knowledge base&#8217;s region.</li>

<li><strong>Sync succeeds but the document count is far lower than expected.</strong> Usually the integration user&#8217;s profile, not the filters. Query the object as that user in Salesforce and count. Also check the per-knowledge-base file count and file size quotas, which are separate limits and are silently enforced.</li>

<li><strong>Retrieval returns nothing for queries you know are covered.</strong> Test with no filter at all. If results appear, your filter keys do not match the auto-detected field names. If they do not, the content probably never made it past chunking; check whether those articles are mostly images.</li>

<li><strong>Deleted Salesforce records still turn up in answers.</strong> Incremental sync handles deletions, but only on the next sync. If you deleted something sensitive, run a sync immediately rather than waiting for the schedule.</li>

<li><strong>Answers are confidently wrong about numbers.</strong> Fixed-size chunking split a record away from its label. Retrieve the raw chunks for that query and read them. Nine times out of ten the problem is visible in the chunk, not in the model.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Running the Connected App as a System Administrator &#8220;just to get the sync working&#8221;. That decision is almost never revisited.</li>

<li>Assuming sharing rules and field-level security follow the data into the index. They do not.</li>

<li>Enforcing access in the system prompt. Retrieval has already happened by the time the model reads your instructions.</li>

<li>Ingesting Contact and Lead because they were there. Both carry personal data and both are usually irrelevant to the questions people actually ask.</li>

<li>Accepting default chunking, then discovering it is immutable.</li>

<li>Building on a preview connector without telling stakeholders it is preview. The configuration surface can change under you.</li>

<li>Testing only as an admin. Every acceptance test should run as the narrowest real profile you have.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Decide the integration user&#8217;s profile before you touch AWS. It defines the security boundary more than any IAM policy will.</li>

<li>Start with Knowledge articles only. Ship that, get real queries, then argue about whether Opportunity data is worth the architecture it requires.</li>

<li>Write filters as allow-lists. Exclusion beats inclusion, so an allow-list fails closed.</li>

<li>Split content shapes across separate data sources so chunking suits each one.</li>

<li>Keep a fixed evaluation set of about fifty real questions with known-good answers, and re-run it after every configuration change.</li>

<li>Log the retrieved chunk IDs alongside every generated answer. When someone reports a bad answer months from now, that log is the only thing that makes it debuggable.</li>

<li>Treat <code>bedrock:Retrieve</code> as a privileged permission and scope it to the specific knowledge base ARN, not to a wildcard.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does the Bedrock Salesforce connector respect sharing rules?</h3>



<p class="wp-block-paragraph">No. The crawler authenticates as one user and everything it can read becomes retrievable by anyone with <code>bedrock:Retrieve</code> on the knowledge base. Per-user scoping has to be built at the retrieval layer with explicit metadata filters, which in practice means routing through S3 rather than using the connector.</p>



<h3 class="wp-block-heading">Can I use Aurora or pgvector instead of OpenSearch Serverless?</h3>



<p class="wp-block-paragraph">Not with the Salesforce data source. AWS currently restricts that connector to Amazon OpenSearch Serverless. If your vector store is non-negotiable, extract to S3 and use a data source that supports your chosen store.</p>



<h3 class="wp-block-heading">Should I use Agentforce instead of building this on Bedrock?</h3>



<p class="wp-block-paragraph">If your questions are answerable entirely from Salesforce data and your users already live in Salesforce, Agentforce is the shorter path and it operates inside the platform&#8217;s existing permission model, which removes the hardest problem in this post. Bedrock earns its complexity when the assistant has to reason across CRM data plus sources that are not in Salesforce, or when you need a specific foundation model, or when the data must stay inside your AWS account. Cost profiles differ substantially and are worth modelling on your real query volume before committing.</p>



<h3 class="wp-block-heading">How current will the assistant&#8217;s answers be?</h3>



<p class="wp-block-paragraph">As current as your last sync. Ingestion is scheduled, not live. For rapidly changing fields such as case status or pipeline stage, do not answer from the index at all: have the assistant call the Salesforce API for the live value and use the knowledge base only for the surrounding context.</p>



<h3 class="wp-block-heading">What happens to the connector when it leaves preview?</h3>



<p class="wp-block-paragraph">Unknown, which is the point of the label. Preview means the configuration surface, supported objects and behaviour can change without the compatibility guarantees you would expect from a GA service. Keep your data source configuration in version control so a re-create is a rerun rather than an archaeology exercise.</p>



<h3 class="wp-block-heading">Is a CRM knowledge assistant worth building if we already have Salesforce search?</h3>



<p class="wp-block-paragraph">It depends on the question shape. Search is better when people know what record they want. A CRM knowledge assistant wins on synthesis: &#8220;what usually causes this error for enterprise customers&#8221; pulls from twenty articles and cases at once, which no search result page does well. If your users mostly navigate to known records, you will not get much return.</p>



<h3 class="wp-block-heading">Do I need Bedrock AgentCore for this?</h3>



<p class="wp-block-paragraph">Not for retrieval and generation. AgentCore matters when you want the assistant to take actions, chain tools, or be exposed to agent frameworks over MCP. A read-only question-answering assistant works fine against the knowledge base APIs directly, and skipping AgentCore removes a layer you would otherwise have to operate.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Connecting Salesforce to Amazon Bedrock takes an afternoon. The connector works, the syncs run, the answers are good. That speed is exactly what makes this dangerous, because nothing in the setup path forces you to think about who the answers are for.</p>



<p class="wp-block-paragraph">A CRM knowledge assistant is a publishing decision wearing an engineering costume. Every record you ingest, you are publishing to everyone who can call retrieve. Decide what that audience is allowed to see, encode it in the integration user&#8217;s profile and in your inclusion filters, and test every question as your narrowest real user. Do that first and the rest of the build is routine.</p>



<h2 class="wp-block-heading">Need help getting this into production?</h2>



<p class="wp-block-paragraph">I work with teams building retrieval systems on AWS, usually at the point where a working prototype has to survive a security review. Things I can help with on a CRM knowledge assistant:</p>



<ul class="wp-block-list">
<li>Reviewing an existing Bedrock knowledge base for records that should never have been ingested, and scoping the integration user&#8217;s Salesforce profile to match.</li>

<li>Designing the S3 extraction path with sidecar access metadata when per-user filtering is a hard requirement.</li>

<li>Choosing between the managed and customer-managed knowledge base paths, with the OpenSearch Serverless cost floor modelled against your actual query volume.</li>

<li>Building the chunking and data source split before it becomes an immutable mistake, plus a repeatable evaluation set to measure changes against.</li>

<li>Terraform or OpenTofu for the whole stack: knowledge base, data sources, Secrets Manager, IAM, and the vector store.</li>

<li>Retrieval observability so you can tell whether answer quality moved, and why.</li>

<li>Migration planning for teams currently on a Kendra index.</li>
</ul>



<p class="wp-block-paragraph">If you have a data source configuration, a failing sync, or a retrieval result that looks wrong, send it over and we can work out what it is doing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-crm-knowledge-assistant-bedrock/">Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</title>
		<link>https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[AgentCore]]></category>
		<category><![CDATA[Agentforce]]></category>
		<category><![CDATA[AI Agents]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[External Client Apps]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Identity Propagation]]></category>
		<category><![CDATA[MCP]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=211</guid>

					<description><![CDATA[<p>The demo works, then someone sees records they shouldn't. Nothing errors. Here's how to build a Salesforce AI assistant with Amazon Bedrock that respects your sharing model: which doors AWS and Salesforce just closed, why identity propagation is the failure mode that bites, and how to shape the tool surface so the agent can't wander.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/">Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo went fine. Someone from the support team asked the assistant which accounts looked at risk this quarter, and it answered. Good answer, too. Right opportunities, right open cases, sensible summary.</p>



<p class="wp-block-paragraph">Then somebody noticed that the person who asked couldn&#8217;t see half those opportunities in the Salesforce UI. Different role, different branch of the hierarchy. They saw them through the assistant.</p>



<p class="wp-block-paragraph">Nothing errored. No 403, no exception in the logs, no alert. The integration user had access, so the API call succeeded, so the model got the records, so it summarised them. Every layer did exactly what it was configured to do. That is the failure mode that should worry you most when you build a Salesforce AI assistant with Amazon Bedrock: it is silent, it is correct-looking, and you will not find it in a load test.</p>



<p class="wp-block-paragraph">This post covers what actually goes wrong when you wire a large language model into a CRM: which door AWS closed on the entry path, whose permissions your agent is really running under, how to shape the tool surface so the model can&#8217;t wander, and what happens to your org&#8217;s API budget when an agent starts looping. There&#8217;s a build walkthrough, but the build is the easy part.</p>



<h2 class="wp-block-heading">Two doors closed while you were reading the old tutorial</h2>



<p class="wp-block-paragraph">Most of what ranks for this topic was written against an architecture you can no longer build in a fresh account. Both vendors changed the front door within a few months of each other, and the tutorials haven&#8217;t caught up. Check this before you follow anything, including this post.</p>



<h3 class="wp-block-heading">On the AWS side: Bedrock Agents is now Bedrock Agents Classic</h3>



<p class="wp-block-paragraph">AWS renamed the original Bedrock Agents service to Bedrock Agents Classic and put it into maintenance mode. It stopped accepting new customers on 30 July 2026. This one is worth stating with the date attached, because the date is the whole point.</p>



<p class="wp-block-paragraph">The mechanics matter more than the branding:</p>



<ul class="wp-block-list">
<li>Only <code>CreateAgent</code> and <code>InvokeInlineAgent</code> are restricted. Every other API stays available to everyone.</li>

<li>Allowlisting is per AWS account, based on whether that account used Bedrock Agents in the previous twelve months. Multi-account estates get a split result: your old sandbox works, your new production account doesn&#8217;t.</li>

<li>A non-allowlisted account calling <code>CreateAgent</code> gets an <code>AccessDeniedException</code> with HTTP 403, not a quota error. If you&#8217;re debugging IAM and seeing 403s, this is a plausible cause that no policy change will fix.</li>

<li>There is no exception process. AWS documents this plainly.</li>

<li>The model catalogue inside Classic is frozen. Bedrock itself keeps getting new models; the Classic orchestration layer does not.</li>
</ul>



<p class="wp-block-paragraph">The replacement is Amazon Bedrock AgentCore. It splits into a managed harness (config-driven, closest to the old experience) and code-defined agents on the AgentCore runtime (bring Strands, LangGraph, CrewAI, or your own loop). For a Salesforce assistant, the harness is usually enough, and you can move to code later without re-plumbing the tools.</p>



<p class="wp-block-paragraph">Existing Classic agents keep running. There&#8217;s no announced end-of-life. But a frozen model catalogue sets a ceiling on the thing, and if your team can&#8217;t recreate the agent in a new account, your disaster recovery story has a hole in it.</p>



<h3 class="wp-block-heading">On the Salesforce side: Connected Apps gave way to External Client Apps</h3>



<p class="wp-block-paragraph">Salesforce blocked creation of new Connected Apps through both UI and API, with package installation as the exception. External Client Apps are the replacement. Existing Connected Apps keep working and keep their OAuth flows, so this is a wall on new creation rather than a sunset.</p>



<p class="wp-block-paragraph">The practical symptom is that the <em>New Connected App</em> button is greyed out with no explanation. Every AWS-to-Salesforce tutorial that opens with &#8220;first, create a Connected App&#8221; stops working right there.</p>



<p class="wp-block-paragraph">One difference matters for agent work specifically: External Client Apps do not support the username-password OAuth flow at all. If your integration design assumed you&#8217;d stash a service account username and password somewhere and call it a day, that design is gone. Which is a good thing, and it forces the question the next section is about.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: whose permissions is the assistant using?</h2>



<p class="wp-block-paragraph">This is the one from the opening, and it&#8217;s the reason most of these projects should slow down before they ship.</p>



<p class="wp-block-paragraph">Salesforce&#8217;s security model is genuinely good. An OAuth token runs in a user context, and object permissions, field-level security, and sharing rules all still apply to API calls. A token with the <code>api</code> scope does not bypass CRUD, FLS, or sharing. People hear that and relax.</p>



<p class="wp-block-paragraph">The catch is the phrase &#8220;a user context.&#8221; Which user?</p>



<p class="wp-block-paragraph">If you authenticate the gateway to Salesforce with client credentials, every request from every person runs as one integration user. Salesforce enforces that user&#8217;s permissions perfectly. It has no idea a different human asked the question. You&#8217;ve built what&#8217;s sometimes called a multiplexor: one licence fronting many people, with all the real access control now living in your application layer instead of the platform&#8217;s.</p>



<p class="wp-block-paragraph">At that point the sharing model is decoration. A prompt that talks the model into calling a query tool with a wider filter returns data the asker was never entitled to, and Salesforce logs it as a legitimate read by a legitimate user. There&#8217;s nothing to alert on, because nothing went wrong.</p>



<h3 class="wp-block-heading">Two-legged versus three-legged, and when each is honest</h3>



<p class="wp-block-paragraph">AgentCore Gateway supports both client credentials (two-legged) and authorization code (three-legged) flows for reaching downstream targets, and handles ingress and egress authorisation separately. That distinction is the control you need.</p>



<ul class="wp-block-list">
<li><strong>Two-legged is honest when the data is genuinely org-wide.</strong> Product catalogues, published knowledge articles, case deflection content. Nobody&#8217;s entitlement varies, so a single service identity represents the truth.</li>

<li><strong>Three-legged is the only correct answer for record data.</strong> Opportunities, accounts, contacts, custom objects with sharing rules on them. The token is minted for the actual human, Salesforce applies their sharing, and the answer is bounded by what they could have looked up themselves.</li>
</ul>



<p class="wp-block-paragraph">Three-legged costs you something real, and I&#8217;d rather say so than pretend it&#8217;s free. You need per-user consent, token storage, refresh handling, and a way to deal with a user whose token has expired mid-conversation. AgentCore Identity provides a token vault and handles the consent flow, including binding the authorisation URL to the session that started it so a shared link can&#8217;t be completed by someone else. It&#8217;s still more moving parts than a single secret in Secrets Manager.</p>



<p class="wp-block-paragraph">Pay the cost anyway for record data. The alternative is a system whose blast radius is &#8220;everything the integration user can see,&#8221; discovered by a customer or an auditor rather than by you.</p>



<p class="wp-block-paragraph">If you already run Okta, Auth0, or Ping as your identity provider, Gateway can sit in front of them for inbound auth rather than you standing up a parallel Cognito user pool. Reusing the IdP you already audit is worth more here than saving a config file.</p>



<h2 class="wp-block-heading">Failure family two: the shape of the tool surface</h2>



<p class="wp-block-paragraph">The second mistake is giving the model a tool called <code>run_soql</code> that accepts an arbitrary query string. It demos beautifully. Ask anything, get anything.</p>



<p class="wp-block-paragraph">You&#8217;ve now made the model your query planner and your authorisation layer at once. Every safety property of the system depends on prompt text. Guardrails help with content, not with whether a generated <code>WHERE</code> clause happened to be too generous.</p>



<p class="wp-block-paragraph">Narrow tools are the fix, and they&#8217;re boring on purpose. <code>get_open_cases_for_account(account_id)</code> beats <code>run_soql(query)</code> because the query shape is fixed in your code, the model only supplies an identifier, and you can log and rate-limit each tool independently. When something goes wrong you&#8217;re reading a parameter, not parsing SOQL to work out what the model asked for.</p>



<p class="wp-block-paragraph">AgentCore Gateway takes OpenAPI specs, Smithy models, and Lambda functions as tool input types, and exposes them to the agent as MCP tools. It also has one-click integrations for several SaaS products including Salesforce. Practically that gives you three routes:</p>



<ol class="wp-block-list">
<li><strong>The built-in Salesforce integration</strong> for the fastest path to something working. Good for a proof of concept. Look closely at which operations it exposes before you take it further, because breadth is exactly what you don&#8217;t want.</li>

<li><strong>An OpenAPI target</strong> describing a small purpose-built API in front of Salesforce. This is the one I reach for first. You control the surface completely, and the spec doubles as the tool description the model reads.</li>

<li><strong>Lambda targets</strong> when a tool needs to combine several Salesforce calls, or talk to something else at the same time.</li>
</ol>



<p class="wp-block-paragraph">MCP server targets are also supported, with the gateway aggregating tools from multiple targets into a single <code>tools/list</code> response. If you&#8217;re running your own MCP server as the Salesforce facade, it needs a stable HTTPS endpoint the gateway can reach; a small VPS from a provider like InterServer or Hetzner is plenty for the development and staging copies, with the production one behind whatever your standard ingress is.</p>



<h2 class="wp-block-heading">Failure family three: the write path</h2>



<p class="wp-block-paragraph">Reads leak. Writes destroy. They deserve different treatment and usually get the same treatment, which is how you end up with three hundred opportunities silently restaged.</p>



<p class="wp-block-paragraph">Start read-only and stay there longer than feels necessary. When you do add writes, the constraints that actually help:</p>



<ul class="wp-block-list">
<li><strong>One record per call.</strong> No bulk update tool. If the model wants to change twenty records it makes twenty calls, each individually logged and individually rate-limited. Slow is a feature here.</li>

<li><strong>Separate the write identity.</strong> A distinct External Client App and permission set for writes means you can revoke the write path at 2am without taking the assistant offline.</li>

<li><strong>Human confirmation on anything irreversible.</strong> The AgentCore harness supports inline function tools that pause the agent and hand control back to your client code, equivalent to return-of-control. That pause is where a person says yes.</li>

<li><strong>Stamp the source.</strong> A custom field recording that a change came from the assistant, with the session and actor ID, is the difference between a ten-minute rollback and an afternoon of forensics.</li>
</ul>



<p class="wp-block-paragraph">Salesforce field history tracking is on a limited number of fields per object. If the field your assistant writes to isn&#8217;t tracked, you have no before-value. Check that before go-live, not after.</p>



<h2 class="wp-block-heading">Failure family four: what happens under load</h2>



<p class="wp-block-paragraph">Salesforce meters API requests against a daily org-wide allocation that depends on your edition and licence count. Your assistant does not consume that budget the way a nightly sync does.</p>



<p class="wp-block-paragraph">A single conversational turn can be several tool calls. A model that doesn&#8217;t get a clean answer will often retry with a different filter. Ten support agents having ten conversations each becomes a request pattern nobody modelled, and the org-wide limit means the integration that breaks first might be your ETL job, your marketing platform, or your billing sync. Not the assistant. That&#8217;s what makes it hard to diagnose.</p>



<p class="wp-block-paragraph">Three things that help, in order of how much they buy you:</p>



<ul class="wp-block-list">
<li>Cap tool calls per session in your agent config, and return a clear message when the cap is hit rather than letting the loop run.</li>

<li>Cache anything org-wide and slow-moving. Picklist values, record type metadata, product catalogue. These are a surprising share of calls and almost never change mid-day.</li>

<li>Watch API usage as a leading indicator, not a post-mortem. Salesforce exposes remaining daily requests via the REST limits resource. Scrape it on a schedule into CloudWatch, Grafana Cloud, or whatever you already page from, and alert on the trend rather than the wall.</li>
</ul>



<h2 class="wp-block-heading">Wiring it up</h2>



<p class="wp-block-paragraph">With the design decisions settled, the build is short. The AgentCore CLI scaffolds and deploys the harness.</p>



<ol class="wp-block-list">
<li>Create an External Client App in Salesforce with the narrowest OAuth scopes the tools need. Assign it via permission set, not profile.</li>

<li>Build the tool layer: an OpenAPI spec for your purpose-built endpoints, or Lambda functions for composite operations.</li>

<li>Create an AgentCore Gateway, attach the target, and configure the credential provider in AgentCore Identity for the egress side.</li>

<li>Create the harness, point it at the gateway, and deploy.</li>
</ol>



<p class="wp-block-paragraph">Scaffolding a harness and attaching a gateway looks like this. The <code>--gateway-arn</code> value is the gateway you created in step three, and the <code>--type</code> tells the CLI this tool is a gateway rather than one of the built-in tools such as the browser or code interpreter:</p>



<pre class="wp-block-code"><code>agentcore create --name sf-assistant

agentcore add tool --harness sf-assistant 
  --type agentcore_gateway --name sf-gateway 
  --gateway-arn arn:aws:bedrock-agentcore:REGION:ACCOUNT_ID:gateway/GATEWAY_ID</code></pre>



<p class="wp-block-paragraph">Then declare the model and the system prompt, and deploy. Check the current model identifiers in the Bedrock console rather than copying one from any blog post, this one included, because they change:</p>



<pre class="wp-block-code"><code>agentcore add harness 
  --name sf-assistant 
  --model-id YOUR_MODEL_ID 
  --system-prompt "You answer questions about CRM records using only the provided tools. Never guess a record value." 
  --tools sf-gateway

agentcore deploy</code></pre>



<p class="wp-block-paragraph">Invoking it takes a session ID, and an actor ID if you want memory scoped per user. Skipping <code>--actor-id</code> is the memory equivalent of the integration user problem: one shared memory store, everyone&#8217;s context in it.</p>



<pre class="wp-block-code"><code>agentcore invoke --harness sf-assistant 
  --session-id "$(uuidgen)" --actor-id alice 
  "Which of my open opportunities have had no activity in three weeks?"</code></pre>



<p class="wp-block-paragraph">Note &#8220;my&#8221; in that question. With three-legged auth it means something. With an integration user it means nothing, and the model will cheerfully invent a definition.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong><code>AccessDeniedException</code> when creating an agent.</strong> If you&#8217;re calling <code>CreateAgent</code> or <code>InvokeInlineAgent</code> on the Classic API, check whether the account has prior Bedrock Agents usage before you touch IAM. No policy fixes this one. Build on AgentCore instead.</p>



<p class="wp-block-paragraph"><strong>The New Connected App button is greyed out.</strong> Expected. Create an External Client App. If you have a documented technical requirement that only a Connected App satisfies, that&#8217;s a Salesforce Support request, not a setting you can toggle.</p>



<p class="wp-block-paragraph"><strong>Tools appear in <code>tools/list</code> but calls fail.</strong> Gateway can cache tool schemas from a target without holding a working credential for calls. Discovery succeeding proves the schema synced, not that egress auth works. Look at the credential provider.</p>



<p class="wp-block-paragraph"><strong>The assistant returns fewer records than the user expects.</strong> Under three-legged auth this is often correct behaviour that looks like a bug. Have the user run the same query in the Salesforce UI. If the UI shows more, check FLS on the specific fields; if it shows the same, the assistant is right and the user&#8217;s mental model is wrong.</p>



<p class="wp-block-paragraph"><strong>Answers are confidently wrong about record values.</strong> Usually the model answering from training data instead of calling a tool. Tighten the system prompt to forbid answering record questions without a tool result, and check the traces to confirm which turns actually called out.</p>



<p class="wp-block-paragraph"><strong>Everything works, then breaks at the same time daily.</strong> Look at org-wide API consumption before you look at your own code. Something else in the org may be eating the allocation on a schedule.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Using a System Administrator profile for the integration identity because it makes the errors go away.</li>

<li>Exposing a generic SOQL tool and calling the prompt a security control.</li>

<li>Testing only in a sandbox with one user, where the sharing model is invisible because there&#8217;s nothing to share.</li>

<li>Treating Guardrails as an access control layer. They filter content; they don&#8217;t know who&#8217;s asking.</li>

<li>Shipping writes in the first release because the read-only version felt underwhelming in the demo.</li>

<li>Building on Bedrock Agents Classic in a new account without checking allowlist status first.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Use three-legged OAuth for anything governed by sharing rules. Reserve client credentials for genuinely org-wide data.</li>

<li>Give each tool one job and a fixed query shape. The model supplies parameters, never structure.</li>

<li>Separate read and write identities so you can kill one without the other.</li>

<li>Scope memory per actor, not per deployment.</li>

<li>Test with at least three users at different levels of the role hierarchy, and diff the answers. Identical answers to &#8220;show me my pipeline&#8221; is the signal that identity isn&#8217;t propagating.</li>

<li>Keep end-to-end traces and ship them somewhere you already look, so agent behaviour lands in the same place as the rest of your telemetry.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I still use Bedrock Agents for a new Salesforce project?</h3>



<p class="wp-block-paragraph">Only if the AWS account already used Bedrock Agents in the previous twelve months. Accounts without that history get an <code>AccessDeniedException</code> on <code>CreateAgent</code>, and AWS has said there&#8217;s no exception process. For anything new, build on AgentCore.</p>



<h3 class="wp-block-heading">Do I need Agentforce to build a Salesforce AI assistant with Amazon Bedrock?</h3>



<p class="wp-block-paragraph">No. Agentforce is Salesforce&#8217;s own agent layer, and it can call out to AWS, which is useful when you want the assistant to live inside the Salesforce UI and inherit the Einstein Trust Layer. But an AgentCore agent calling Salesforce APIs through a gateway is a complete architecture on its own, and it&#8217;s the better fit when the assistant also needs to reach systems outside Salesforce.</p>



<h3 class="wp-block-heading">Does the model see all my CRM data?</h3>



<p class="wp-block-paragraph">It sees whatever your tools return. That&#8217;s determined by the OAuth identity and the tool definitions, not by the model. This is why the identity decision matters more than the model choice.</p>



<h3 class="wp-block-heading">Will Bedrock Guardrails stop the assistant leaking records?</h3>



<p class="wp-block-paragraph">No, and this is worth being blunt about. Guardrails filter content categories and can mask sensitive data patterns. They have no concept of which user is asking or which records that user is entitled to. A record correctly retrieved under an over-permissioned token passes every guardrail check, because there&#8217;s nothing objectionable about it. Guardrails are a good layer. They are not this layer.</p>



<h3 class="wp-block-heading">How do I keep the assistant from burning the org&#8217;s API limit?</h3>



<p class="wp-block-paragraph">Cap tool calls per session, cache org-wide reference data, and monitor remaining daily requests via the REST limits resource on a schedule. The allocation is org-wide, so the symptom of overuse usually shows up in some other integration first.</p>



<h3 class="wp-block-heading">Should I migrate an existing Bedrock Agents Classic setup?</h3>



<p class="wp-block-paragraph">There&#8217;s no deadline and no announced end-of-life, so there&#8217;s no emergency. The reasons to move are the frozen model catalogue and the fact that new accounts can&#8217;t recreate your agent. If you&#8217;re still actively developing the thing, move. If it&#8217;s stable and finished, it can wait. AWS publishes a capability mapping and a CLI import path for existing configurations.</p>



<h3 class="wp-block-heading">Is an External Client App harder to set up than a Connected App?</h3>



<p class="wp-block-paragraph">Slightly different rather than harder. The main structural change is that developer settings and admin policies are separated, so the app definition and the per-org access controls are configured independently. The one real removal is the username-password flow, which no longer exists in the ECA model.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AI assistant with Amazon Bedrock is not really a model problem. Model quality is the part that will take care of itself, and the part you can change with one config line later.</p>



<p class="wp-block-paragraph">The part that won&#8217;t take care of itself is identity. If the assistant answers as one integration user, you have quietly moved your entire access control model from a platform that enforces it well into a prompt that enforces it not at all, and nothing in your monitoring will tell you. Get that right first. Then argue about models.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or auditing this?</h2>



<p class="wp-block-paragraph">Most of the work on these projects is unglamorous: deciding what the agent is allowed to touch, and proving it can&#8217;t touch anything else. That&#8217;s the part I&#8217;m useful for.</p>



<ul class="wp-block-list">
<li>Auditing an existing Salesforce AI integration for identity propagation gaps, including testing across role hierarchy levels to find silent over-permissioning.</li>

<li>Migrating a Bedrock Agents Classic deployment to AgentCore, including mapping action groups to gateway targets.</li>

<li>Designing the tool surface: replacing generic query tools with narrow, logged, individually rate-limited operations.</li>

<li>Setting up three-legged OAuth between AgentCore Gateway and Salesforce, including token vault and consent handling.</li>

<li>Moving integrations from Connected Apps to External Client Apps without breaking existing OAuth flows.</li>

<li>Instrumenting API consumption and agent traces so you see problems before an unrelated integration starts failing.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got something concrete, send it over: a gateway target config, an agent trace, a tool schema, or just the error you&#8217;re stuck on. Easier to say something useful about a real setup than a hypothetical one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/">Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin</title>
		<link>https://john-nessime.com/blog/devops/agentforce-aws-architecture-security/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 06 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Agentforce]]></category>
		<category><![CDATA[AI Agents]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Einstein Trust Layer]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[MCP]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Salesforce]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=155</guid>

					<description><![CDATA[<p>Agentforce and AWS wire together in four standard patterns, and every one of them has a point where Salesforce's guarantees stop and yours start. This traces a single request across each boundary it crosses, covers the Trust Layer default most write-ups get wrong (LLM data masking is disabled for agents), and sets out what changes the moment a callout lands in your own account: retention, audit trail, and user identity that does not travel.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/agentforce-aws-architecture-security/">Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The architecture review goes fine right up until someone from security asks the only question that matters: when the agent answers a customer&#8217;s question, where exactly does that customer&#8217;s data go, and who keeps a copy?</p>



<p class="wp-block-paragraph">&#8220;Salesforce handles it&#8221; is not an answer. Neither is &#8220;it&#8217;s in our VPC,&#8221; because it usually isn&#8217;t, and the parts that are in your VPC are the parts nobody has threat-modelled.</p>



<p class="wp-block-paragraph">This post traces a single Agentforce request end to end and stops at each boundary it crosses. Agentforce and AWS get wired together in three or four standard patterns, and every one of them has a specific point where Salesforce&#8217;s guarantees stop and yours start. I will cover where that line sits, what changes about the security model when you cross it, and one default that surprises almost everyone.</p>



<h2 class="wp-block-heading">The boundaries a single request crosses</h2>



<p class="wp-block-paragraph">One user turn touches more systems than the diagrams suggest. Roughly:</p>



<ol class="wp-block-list">
<li><strong>Retrieval.</strong> The agent grounds itself in CRM records, Data Cloud objects and knowledge content.</li>

<li><strong>Reasoning.</strong> The request gets classified to a topic, then an action gets chosen. Both steps involve a model call.</li>

<li><strong>Generation.</strong> A prompt is assembled and sent through the LLM gateway to a model provider.</li>

<li><strong>Action.</strong> If the agent decides to do something, it calls out. That callout is where AWS usually enters the picture.</li>
</ol>



<p class="wp-block-paragraph">Steps one to three happen inside Salesforce&#8217;s trust boundary, and the Einstein Trust Layer applies. Step four does not, and it doesn&#8217;t. That sentence is the whole post, but the detail matters.</p>



<h2 class="wp-block-heading">Retrieval: the part that&#8217;s better than you&#8217;d expect</h2>



<p class="wp-block-paragraph">Grounding is where Agentforce gets its context, and the architecturally interesting decision is that vector search lives inside Data Cloud rather than in a separate vector database. Structured records and unstructured content are retrieved from the same layer, which removes a whole class of sync problem you would otherwise be building.</p>



<p class="wp-block-paragraph">The security-relevant property is that retrieval is permission-aware. The agent retrieves as the running user, so sharing rules and field-level security apply. If a rep cannot see an opportunity, the agent acting on their behalf should not be able to summarise it either.</p>



<p class="wp-block-paragraph">Two caveats worth putting in your notes. First, this only holds while retrieval goes through Salesforce; the moment you ground from an external source, you are enforcing access control yourself. Second, if your CRM sharing model is loose, the agent inherits that looseness and makes it much easier to exploit, because a model will cheerfully summarise a thousand records a human would never have opened one by one. Agentforce does not create the over-permissioning problem, it just removes the friction that was hiding it.</p>



<p class="wp-block-paragraph">Where AWS shows up here: if your analytical data lives in Redshift, an Iceberg lake or Databricks, zero-copy federation lets Data Cloud register those tables and query them in place rather than replicating them. Good for cost and duplication, and it moves an access-control decision into the federation configuration, which is a place people forget to audit.</p>



<h2 class="wp-block-heading">The Trust Layer, and the default nobody expects</h2>



<p class="wp-block-paragraph">The Einstein Trust Layer sits between every Salesforce-originated prompt and the model. Its components are well documented: secure grounding, data masking, system policies against prompt injection, the LLM gateway, zero data retention agreements with model providers, toxicity checks on the way back, and an audit trail of the whole journey.</p>



<p class="wp-block-paragraph">Here is the part that gets misreported constantly, including in a lot of otherwise decent write-ups: <strong>LLM data masking is disabled for agents.</strong> Salesforce says so plainly in its own documentation on the Trust Layer architecture and repeats it in the Trailhead module on masking. Masking remains available and configurable for embedded generative features like service replies and work summaries. For Agentforce agents, it is off.</p>



<p class="wp-block-paragraph">The reason is not hard to work out once you think about what an agent does. Masking swaps real values for placeholders. An agent that has to actually perform an action needs the real record ID, the real amount, the real email address, because those become tool arguments. You cannot pass <code>&lt;Person_0&gt;</code> to a Flow and expect it to update a contact. Masking and acting are in genuine tension, and Salesforce resolved it in favour of acting.</p>



<p class="wp-block-paragraph">What this changes in practice:</p>



<ul class="wp-block-list">
<li>Real PII reaches the model provider. The zero data retention commitment is what protects it, not de-identification. Those are different controls with different failure modes: masking is technical and verifiable, a retention agreement is contractual.</li>

<li>If your compliance position was &#8220;no personal data leaves our boundary in identifiable form,&#8221; Agentforce does not satisfy it the way you may have assumed, and this is worth raising before your security review rather than during it.</li>

<li>Controlling exposure moves upstream. What the model sees is now decided by what you ground on and what your sharing model permits, not by a masking filter downstream.</li>
</ul>



<p class="wp-block-paragraph">None of this makes the Trust Layer weak. Zero retention, the gateway, system policies and the audit trail all still apply to agent traffic, and the Trust Layer typically runs several times in a single user turn, once for topic classification and again for generation. It is a real control plane. Just know which control is doing the work.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Connecting Agentforce and AWS: four patterns</h2>



<p class="wp-block-paragraph">There are four ways these two systems talk, and they have different latency, failure and security characteristics.</p>



<h3 class="wp-block-heading">Synchronous outbound: agent action calls AWS</h3>



<p class="wp-block-paragraph">A custom agent action, implemented via External Services with an OpenAPI schema, an Apex invocable method, or a Flow, calls an endpoint you own. Behind it sits API Gateway and Lambda, and often a Bedrock agent or knowledge base.</p>



<p class="wp-block-paragraph">Authentication uses named credentials, which keep the secret out of your Apex and centralise the auth config. Use them; hardcoding a key into an Apex class is the version of this that ends up in a git history.</p>



<p class="wp-block-paragraph">The trade-off is that you have put a network call and a second model invocation inside a conversational turn. Users notice. Budget for it, and set the timeout deliberately rather than inheriting a default.</p>



<h3 class="wp-block-heading">Asynchronous outbound: Event Relay to EventBridge</h3>



<p class="wp-block-paragraph">Salesforce Event Relay delivers platform events from the Salesforce event bus to Amazon EventBridge without code, and EventBridge routes onward to Lambda, S3, Step Functions or a Bedrock agent.</p>



<p class="wp-block-paragraph">This is the right pattern for anything that does not need to complete inside the conversation: enrichment, downstream processing, long-running work. It also fails better, because a retry is a queue concern rather than a user watching a spinner.</p>



<h3 class="wp-block-heading">Inbound: AWS calls Agentforce</h3>



<p class="wp-block-paragraph">The Agent API lets something on the AWS side invoke an Agentforce agent, typically through EventBridge API destinations. That gives you bidirectional multi-agent flows: a Bedrock agent that detects a problem can open a case through Agentforce rather than writing to the Salesforce API directly and bypassing the agent&#8217;s own logic.</p>



<p class="wp-block-paragraph">Worth being deliberate here. An inbound path means an AWS workload can trigger agent reasoning, which means anything that can inject content into that workload can influence a prompt. Treat the payload as untrusted input, because it is.</p>



<h3 class="wp-block-heading">MCP: tools rather than integrations</h3>



<p class="wp-block-paragraph">Agentforce has a native MCP client, so an agent can connect to any MCP-compliant server without a bespoke integration, alongside a server registry intended to let admins govern which agents connect to which tools under central policy.</p>



<p class="wp-block-paragraph">This is the pattern that will cause the most trouble over the next couple of years, and not because the protocol is bad. It is because MCP makes adding a tool trivial, and every tool is a new path by which text from somewhere else reaches your agent&#8217;s context and by which your agent&#8217;s decisions reach someone else&#8217;s system. The registry exists precisely so that &#8220;which tools can this agent reach&#8221; is an administered answer rather than an emergent one. Use it as a control, not a catalogue.</p>



<h3 class="wp-block-heading">Keeping the traffic private</h3>



<p class="wp-block-paragraph">By default these callouts traverse the public internet with TLS. Salesforce Private Connect uses AWS PrivateLink to give you a private path to your own AWS endpoints instead. Whether that is required is a policy question rather than a technical one, but it is much easier to set up at design time than to retrofit after an auditor asks.</p>



<h2 class="wp-block-heading">What changes the moment you cross into AWS</h2>



<p class="wp-block-paragraph">This is the seam, and it is where most real incidents will come from. Four things stop being true simultaneously.</p>



<ul class="wp-block-list">
<li><strong>Zero retention becomes your policy, not Salesforce&#8217;s.</strong> If your Lambda logs its input at INFO, you now have customer PII in CloudWatch Logs with whatever retention that log group happens to have, which by default is forever. Your own debug logging is the single most likely leak path in this entire architecture, and it will not show up in any Salesforce audit report.</li>

<li><strong>The audit trail splits.</strong> Salesforce logs the prompt journey. AWS logs the invocation. Nothing correlates them unless you pass a request identifier through the callout and log it on both sides. Do that from day one; reconstructing an incident across two disconnected audit systems is genuinely unpleasant.</li>

<li><strong>User identity does not travel.</strong> This is the important one. Inside Salesforce the agent acts as the running user and sharing rules apply. Your Lambda runs under an IAM execution role with fixed permissions and no idea who asked. If that role can read a whole DynamoDB table, then every user of the agent can effectively read the whole table through it, regardless of their CRM permissions. That is a classic confused deputy, and it is easy to build without noticing. If the action needs to be scoped per user, pass the identity explicitly and enforce it in your own code.</li>

<li><strong>Model governance forks.</strong> A Bedrock model you invoke from your own Lambda is outside the Trust Layer. Its guardrails, logging and retention are configured in your account and are yours to get right. If you want prompt filtering or PII detection on that leg, you are building or buying it.</li>
</ul>



<p class="wp-block-paragraph">Note the difference between two things people conflate. Salesforce running on Hyperforce, which is built on public cloud infrastructure including AWS, means Salesforce&#8217;s own workloads run there under Salesforce&#8217;s controls. It does not mean your Salesforce data sits in your AWS account. When an agent action calls your Lambda, that is the first moment data lands in infrastructure you control and are responsible for.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Assuming masking protects agent traffic to the model. It does not; check the current documentation yourself rather than taking a blog&#8217;s word for it, including this one.</li>

<li>Logging full request payloads in Lambda during development and never turning it off.</li>

<li>Giving the Lambda execution role broad data access because &#8220;the agent is trusted,&#8221; when the agent is a proxy for every user who can talk to it.</li>

<li>Putting a slow AWS call inside a synchronous agent action when the work did not need to be synchronous.</li>

<li>Not passing a correlation identifier across the boundary, so Salesforce and CloudWatch tell two unlinked stories.</li>

<li>Treating content returned from an external tool or MCP server as trusted context rather than as untrusted input that will be read by a model.</li>

<li>Deploying agents against a permissive sharing model and discovering the blast radius later.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does Agentforce data get used to train models?</h3>



<p class="wp-block-paragraph">Not on the Salesforce leg. Salesforce&#8217;s zero data retention arrangements with model providers mean prompts and responses are not stored or used for training. That commitment covers traffic going through the LLM gateway. It says nothing about a model you invoke yourself from your own AWS account, where the retention and logging configuration is entirely yours.</p>



<h3 class="wp-block-heading">Can I use my own Bedrock model inside Agentforce?</h3>



<p class="wp-block-paragraph">There are two distinct things here. Bedrock-hosted models can be brought into Salesforce through the Einstein and Bedrock integration, in which case they sit behind the Trust Layer. Separately, you can call a Bedrock model or agent from a custom agent action, in which case you are outside the Trust Layer and own the controls. Be clear which one you are building, because they have completely different compliance stories.</p>



<h3 class="wp-block-heading">Sync or async for AWS integrations?</h3>



<p class="wp-block-paragraph">Default to async through Event Relay and EventBridge unless the agent genuinely needs the result to continue the conversation. Synchronous callouts put a network round trip and often a second inference call inside a user&#8217;s turn, and they fail in front of the user rather than into a retry queue.</p>



<h3 class="wp-block-heading">How do I stop the agent seeing data a user shouldn&#8217;t?</h3>



<p class="wp-block-paragraph">Inside Salesforce, fix the sharing model, because retrieval respects it and inherits its flaws. Outside Salesforce, sharing rules do not apply at all: pass the user identity through the callout and enforce authorisation in your own code, or scope the integration to data that is safe for every user of that agent.</p>



<h3 class="wp-block-heading">Do I need PrivateLink for this?</h3>



<p class="wp-block-paragraph">Technically no; callouts work over TLS on the public internet. Practically it depends on your regulatory position and whether your AWS endpoints should be publicly reachable at all. If they should not be, Private Connect over PrivateLink is the supported path and is far cheaper to design in than to add later.</p>



<h3 class="wp-block-heading">What should I log, and where?</h3>



<p class="wp-block-paragraph">On the AWS side, log the correlation identifier, the action name, the outcome and the timing. Do not log the payload. If you need payload-level debugging, gate it behind a flag that is off in production and set an explicit short retention on the log group. Ship the operational metrics somewhere you already watch, whether that is CloudWatch alarms, Grafana or Datadog, because an agent action that silently starts failing looks like nothing at all from the Salesforce side.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Wiring up Agentforce and AWS is not hard. The patterns are documented, the connectors exist, and a working proof of concept is a day&#8217;s work.</p>



<p class="wp-block-paragraph">What is hard is knowing precisely where Salesforce&#8217;s guarantees end. They end at the callout. Everything before it is covered by the Trust Layer, with the significant exception that masking is off for agents, so what reaches the model is real data protected by a retention agreement rather than by de-identification. Everything after it is yours: your retention, your audit trail, your authorisation, your logs.</p>



<p class="wp-block-paragraph">Draw that line on your architecture diagram before the security review, not during it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help designing or reviewing this?</h2>



<p class="wp-block-paragraph">I work on the AWS side of Salesforce integrations, which is usually the side nobody owns. Things I can help with:</p>



<ul class="wp-block-list">
<li>Threat-modelling the seam between Agentforce and your AWS account, and writing down what each side actually guarantees.</li>

<li>Building agent actions on API Gateway and Lambda with named credentials, sane timeouts and no payload logging.</li>

<li>Event-driven integration via Event Relay and EventBridge, including retry, dead-letter and replay design.</li>

<li>Scoping IAM execution roles so an agent action cannot become a confused deputy for every user who can reach it.</li>

<li>Correlated observability across Salesforce and CloudWatch so an incident can be reconstructed in one timeline.</li>

<li>Private Connect and PrivateLink setup where callouts should not touch the public internet.</li>
</ul>



<p class="wp-block-paragraph">If you have an integration already built, send me the agent action definition and the Lambda&#8217;s IAM policy. Those two together usually tell the whole story.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/agentforce-aws-architecture-security/">Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>It Counted the Chunks: Preparing CRM Data for Amazon Bedrock</title>
		<link>https://john-nessime.com/blog/devops/crm-data-amazon-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 00:55:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=116</guid>

					<description><![CDATA[<p>Asked how many open opportunities an account had, the assistant said five. It had nineteen. Retrieval returned five chunks and the model counted them. CRM data is structured and RAG is built for text, so the preparation work is mostly deciding which questions are retrieval and which are queries.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/crm-data-amazon-bedrock/">It Counted the Chunks: Preparing CRM Data for Amazon Bedrock</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo goes well until someone from sales asks the assistant how many open opportunities Acme has. It answers immediately: five. Confident, well-phrased, cited.</p>



<p class="wp-block-paragraph">Acme has nineteen. The assistant retrieved five chunks that happened to mention Acme, because five is roughly how many chunks a retrieval returns, and then it counted them. There is no bug. Nothing failed. Similarity search returned the most similar things and the model described what it was given.</p>



<p class="wp-block-paragraph">That is the defining problem with <strong>preparing CRM data for Amazon Bedrock</strong>, and it is upstream of chunk sizes and embedding models. CRM data is structured. Retrieval-augmented generation is built for unstructured text. Feed a table of opportunities through an embedding pipeline and you get a system that is genuinely good at &#8220;what did we discuss with this account&#8221; and quietly terrible at &#8220;how many&#8221;, &#8220;how much&#8221; and &#8220;since when&#8221;.</p>



<p class="wp-block-paragraph">This covers the split that fixes it, what a good document actually looks like, the permissions problem nobody notices until it is a problem, and how to know whether any of it works.</p>



<h2 class="wp-block-heading">Route the question before you build anything</h2>



<p class="wp-block-paragraph">Your users will ask two kinds of question and they need two different mechanisms.</p>



<ul class="wp-block-list">
<li><strong>Narrative questions.</strong> &#8220;What were the objections on the Acme renewal?&#8221; &#8220;Summarise our history with this account.&#8221; &#8220;What did the customer say about pricing?&#8221; These live in notes, emails, call summaries and descriptions. Semantic retrieval is exactly right for them.</li>
<li><strong>Analytical questions.</strong> &#8220;How many open deals over fifty thousand?&#8221; &#8220;What is total pipeline this quarter?&#8221; &#8220;Which accounts have had no activity in ninety days?&#8221; These need aggregation, filtering and joins. Retrieval cannot do them and will not tell you it cannot.</li>
</ul>



<p class="wp-block-paragraph">The useful part is that you do not have to build the second half yourself. Bedrock Knowledge Bases supports structured data retrieval: you point it at Amazon Redshift or a Glue Data Catalog, and it generates and executes SQL from natural language against your data in place, with no copying or embedding. The counting question goes to SQL, the narrative question goes to the vector store, and both answers are right for the right reasons.</p>



<p class="wp-block-paragraph">So the first architectural decision is not which chunking strategy to use. It is: which of your fields are text worth embedding, and which are facts worth querying. Get that wrong and no amount of tuning downstream will save it.</p>



<p class="wp-block-paragraph">One practical note if you take the structured route. Bedrock&#8217;s execution role authenticates to Redshift as an IAM identity, and you grant it access the same way you would any other reader. Point it at curated views rather than base tables, exactly as you would a BI tool:</p>



<pre class="wp-block-code"><code>-- Give the knowledge base the same narrow access you'd give a
-- reporting user. A view is a contract; a schema grant is not.
GRANT USAGE ON SCHEMA analytics
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

GRANT SELECT ON analytics.v_opportunities
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";</code></pre>



<p class="wp-block-paragraph">Named columns and clear table names matter more here than anywhere else, because the model is reading your schema to write the SQL. A column called <code>flag_2</code> will produce exactly the query quality it deserves.</p>



<h2 class="wp-block-heading">What a good document looks like</h2>



<p class="wp-block-paragraph">For the narrative half, the instinct is to export records to CSV and let the chunker deal with it. That produces chunks that are half a row of one record and half a row of another, with no context about which is which.</p>



<p class="wp-block-paragraph">Build documents deliberately instead. Three rules that do most of the work:</p>



<p class="wp-block-paragraph"><strong>One record, one document.</strong> An opportunity and its notes become a single file. A chunk from it is then always about one thing.</p>



<p class="wp-block-paragraph"><strong>Denormalise the context in.</strong> A note that says &#8220;they want a two-year term&#8221; is useless in isolation. Embed the account name, the opportunity name, the stage and the owner into the same document, so a chunk carries enough context to be retrieved and to be understood once retrieved.</p>



<p class="wp-block-paragraph"><strong>Write it as prose, not as fields.</strong> Embedding models were trained on language. <code>Stage__c: Negotiation</code> embeds worse than &#8220;The opportunity is at the Negotiation stage.&#8221; This feels silly and it measurably improves retrieval.</p>



<p class="wp-block-paragraph">On chunking itself: start with the default strategy and change it only when evaluation tells you to. If your documents are one record each and reasonably short, semantic or hierarchical chunking buys you less than people expect. The gains that matter come from document construction and metadata, not from chunk size.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The permissions problem</h2>



<p class="wp-block-paragraph">This is the one that turns a nice project into an incident, and it is easy to miss because nothing about it looks broken.</p>



<p class="wp-block-paragraph">Salesforce has an elaborate sharing model: role hierarchies, sharing rules, territory management, field-level security. A rep sees their accounts and not everyone else&#8217;s. Your vector store has none of that. Once a record is embedded, it is a vector like any other, and anyone who can query the knowledge base can retrieve it.</p>



<p class="wp-block-paragraph">So a rep asks about a competitor deal they were never on the team for, and the assistant helpfully summarises it. Or someone asks about compensation-adjacent records and gets an answer. No alert fires, because from the system&#8217;s perspective retrieval worked perfectly.</p>



<p class="wp-block-paragraph">Three ways out, in increasing order of effort:</p>



<ol class="wp-block-list">
<li><strong>Only embed what everyone may see.</strong> Blunt, easy, and often correct for a first version. Exclude anything sensitive and be explicit that the assistant does not know about it.</li>
<li><strong>Filter at retrieval time using metadata.</strong> Attach owner, team or region to every document, and pass a filter with each query derived from the caller&#8217;s identity. This is the practical answer for most teams.</li>
<li><strong>Separate knowledge bases per audience.</strong> Heavier to operate, and the only option that gives you a hard boundary rather than a filter you have to remember to apply.</li>
</ol>



<p class="wp-block-paragraph">Whichever you pick, decide it before ingestion, because metadata has to be configured at ingest time. Retrofitting a filter you did not plan for means rebuilding the index.</p>



<h2 class="wp-block-heading">Metadata is most of the value</h2>



<p class="wp-block-paragraph">Metadata filtering narrows the candidate set before similarity search runs. That improves relevance, reduces tokens in the prompt, and gives you the access control lever above. For S3 sources it lives in a sidecar file named after the object it describes:</p>



<pre class="wp-block-code"><code>// acme-renewal.txt.metadata.json, alongside acme-renewal.txt in S3
{
  "metadataAttributes": {
    "object_type": "Opportunity",
    "owner_id": "0051t00000XYZ",
    "account_id": "0011t00000ABC",
    "region": "EMEA",
    "is_closed": false,
    "last_modified_epoch": 1750000000
  }
}</code></pre>



<p class="wp-block-paragraph">Include, at minimum: what kind of record it is, who owns it, which account it belongs to, whether it is still open, and when it last changed. Those five turn &#8220;search everything&#8221; into &#8220;search the open opportunities this person owns, changed recently&#8221;, which is a different product.</p>



<h2 class="wp-block-heading">Freshness, and the vectors that outlive the record</h2>



<p class="wp-block-paragraph">An embedding is a copy. When the source record changes, the copy is stale. When the source record is deleted, the copy is not.</p>



<p class="wp-block-paragraph">The consequences are worse in a CRM than in a document store, because CRM records change constantly and confidently answering with last quarter&#8217;s stage is indistinguishable from answering correctly. An assistant that cites a deal which closed-lost in March, or a contact who left the company, is not obviously wrong to the person reading it.</p>



<ul class="wp-block-list">
<li><strong>Sync on a schedule that matches how the data is used.</strong> Daily is fine for account summaries and useless for live pipeline questions, which should be going to SQL anyway.</li>
<li><strong>Handle deletes explicitly.</strong> Whatever removes a record from your lake must also remove its document from the source bucket, or the vector survives.</li>
<li><strong>Put the record&#8217;s last-modified timestamp in the metadata</strong> and surface it in the answer. &#8220;As of three days ago&#8221; is a small change that prevents a large class of misunderstanding.</li>
<li><strong>Reconcile document count against source record count</strong> periodically. Divergence means orphans.</li>
</ul>



<h2 class="wp-block-heading">What not to embed</h2>



<p class="wp-block-paragraph">CRM free-text fields are where people put things. Notes fields contain personal details, occasionally payment information, opinions about customers that were never meant to leave a private conversation, and years of accumulated noise.</p>



<p class="wp-block-paragraph">Once embedded, all of it is retrievable by anyone with query access, and a vector is not something you can easily grep for a mistake.</p>



<ul class="wp-block-list">
<li>Run detection over free-text fields before ingestion and redact rather than hope.</li>
<li>Exclude fields nobody asks questions about. Every extra field is noise competing for retrieval slots.</li>
<li>Consider excluding records closed long ago; they dilute results and rarely answer anything.</li>
<li>Add Bedrock Guardrails on the output side as a second layer, not as your only control.</li>
<li>Keep the whole thing on private networking with VPC endpoints if the data warrants it.</li>
</ul>



<p class="wp-block-paragraph">There is a governance question here too, and it is worth raising before someone else does: your customers&#8217; personal data is now in a vector store feeding a language model. Whether your privacy notice covers that is not an engineering decision, but it is an engineering responsibility to ask.</p>



<h2 class="wp-block-heading">Evaluate it before you show anyone</h2>



<p class="wp-block-paragraph">A demo proves the pipeline runs. It does not tell you whether the answers are right, and this is a system whose failure mode is being wrong fluently.</p>



<p class="wp-block-paragraph">Write thirty real questions from the people who will use it, with correct answers you have verified by hand. Include the analytical ones specifically, because those are where retrieval fails silently. Then run the set after every meaningful change to chunking, metadata or the document format, and compare.</p>



<p class="wp-block-paragraph">Use the Retrieve API rather than RetrieveAndGenerate while you are tuning. Seeing which chunks came back tells you whether a bad answer is a retrieval problem or a generation problem, and those have completely different fixes.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Embedding structured records and expecting aggregation to work.</li>
<li>Exporting to CSV and letting the chunker split rows arbitrarily.</li>
<li>Field-value dumps instead of prose, so embeddings have little language to work with.</li>
<li>No metadata, so every query searches everything and access control is impossible.</li>
<li>Assuming Salesforce sharing rules follow the data into the vector store.</li>
<li>Deciding on filtering after ingestion, then discovering it has to be configured at ingest.</li>
<li>Syncing changes but never removing documents for deleted records.</li>
<li>Embedding every field because it was easier than choosing.</li>
<li>No PII detection over free-text notes.</li>
<li>Tuning chunk size before fixing document construction.</li>
<li>Judging the system on a demo instead of a scored question set.</li>
<li>Granting the execution role a whole schema rather than specific views.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Route analytical questions to structured retrieval and narrative questions to the vector store.</li>
<li>One record per document, with context denormalised in, written as prose.</li>
<li>Metadata on every document: type, owner, account, open state, last modified.</li>
<li>Decide the access model before ingestion, and filter at retrieval time by the caller&#8217;s identity.</li>
<li>Redact free-text fields before embedding, and exclude fields nobody queries.</li>
<li>Delete documents when source records are deleted, and reconcile the counts.</li>
<li>Surface record age in answers so staleness is visible rather than implied.</li>
<li>Expose curated views to the execution role, never whole schemas.</li>
<li>Meaningful table and column names, because the model reads your schema to write SQL.</li>
<li>A scored evaluation set, run on every change, including questions you expect it to fail.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Why does it get counts wrong?</h3>



<p class="wp-block-paragraph">Because retrieval returns the most similar chunks, not all matching records, and the model counts what it was handed. Counting is a query, not a similarity search. Send those questions to structured retrieval over your warehouse instead.</p>



<h3 class="wp-block-heading">Should I use the Salesforce connector or export to S3 first?</h3>



<p class="wp-block-paragraph">The connector is quicker to stand up. Going via S3 gives you control over document construction, metadata and redaction, which is where most of the quality lives. Start with the connector to learn what people ask, then move to S3 once you know what the documents should look like.</p>



<h3 class="wp-block-heading">Which chunking strategy is best?</h3>



<p class="wp-block-paragraph">Usually the default, honestly. If each document is one record, chunking has less work to do. Change it in response to evaluation results rather than in advance, because chunking is the most over-discussed and least decisive variable in a CRM RAG setup.</p>



<h3 class="wp-block-heading">How do I stop people seeing records they shouldn&#8217;t?</h3>



<p class="wp-block-paragraph">Metadata filters applied per query based on the caller&#8217;s identity, separate knowledge bases per audience, or simply not embedding sensitive records. There is no automatic inheritance of CRM sharing rules, and assuming otherwise is the most consequential mistake in this post.</p>



<h3 class="wp-block-heading">How often should I re-sync?</h3>



<p class="wp-block-paragraph">Match the questions. Account histories and notes tolerate a daily sync comfortably. Anything needing current numbers should not be coming from embeddings at all. Whatever the interval, make sure deletions propagate.</p>



<h3 class="wp-block-heading">Do I need a vector database at all?</h3>



<p class="wp-block-paragraph">Only for the narrative half. If every question your users ask is analytical, structured retrieval over Redshift answers all of them with no embedding pipeline, no vector store and no staleness. Plenty of CRM assistants would be better products if they had noticed that.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">The model will answer either way. It will answer the narrative question well and the counting question badly, in the same tone, with the same citations, and nothing in the response distinguishes them.</p>



<p class="wp-block-paragraph">So the preparation work is mostly triage: decide which questions are retrieval and which are queries, build documents that carry their own context, attach metadata before you ingest rather than after, and test with real questions whose answers you already know. Chunk size is the last thing to worry about and the first thing everybody argues about.</p>



<h2 class="wp-block-heading">Building one of these?</h2>



<p class="wp-block-paragraph">The pipeline is usually the easy part; the decisions before it are where these projects succeed or quietly disappoint. Work I take on:</p>



<ul class="wp-block-list">
<li>Designing the split between semantic retrieval and structured querying, so counting questions get correct answers.</li>
<li>Building the document preparation pipeline from CRM data: record-level documents, denormalised context, metadata, redaction.</li>
<li>Access control design for knowledge bases, including per-caller metadata filtering and audience separation.</li>
<li>Freshness and deletion handling so the index does not diverge from the source.</li>
<li>Curated Redshift views and IAM grants for structured data retrieval.</li>
<li>Evaluation harnesses with scored question sets, run on every change rather than once before launch.</li>
</ul>



<p class="wp-block-paragraph">Send me ten questions your users would actually ask, and I will tell you which of them retrieval can answer.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/crm-data-amazon-bedrock/">It Counted the Chunks: Preparing CRM Data for Amazon Bedrock</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny</title>
		<link>https://john-nessime.com/blog/case-studies/ai-construction-claims-platform-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 03 Aug 2026 14:05:46 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon S3 Vectors]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=130</guid>

					<description><![CDATA[<p>Semantic search finds the most persuasive document, not the earliest one. Here is how to architect an AI construction claims and dispute intelligence platform on AWS so retrieval respects the contractual clock, schedule data stays out of the vector index, every answer resolves to a page, and privileged material never shares a retrieval path with project records.</p>
<p>The post <a href="https://john-nessime.com/blog/case-studies/ai-construction-claims-platform-aws/">Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone hands you a shared drive and asks a question that sounds trivial: &#8220;Did we give notice of the delay event inside the contractual period, or didn&#8217;t we?&#8221;</p>



<p class="wp-block-paragraph">The answer is in there. It is one email, or one line in a site diary, sitting among forty thousand other files. Nobody can read forty thousand files, so the instinct is to point a language model at the pile and ask it. That instinct is right. The naive implementation of it is where the money goes.</p>



<p class="wp-block-paragraph">Here is the failure mode that bites hardest, and it is invisible until an expert challenges you on it. You build retrieval over the document set, ask about notice of delay, and the system confidently returns a letter that discusses the delay event in great detail. It is a good letter. It is also dated eleven months after the event, written by the claims consultant during preparation of the claim itself. It scored highest precisely because it was written to argue the point. The contemporaneous notice, the thing you actually needed, was four badly typed lines in a routine progress email that mentioned the word &#8220;delay&#8221; once.</p>



<p class="wp-block-paragraph">Semantic similarity has no concept of a deadline. That single gap is the difference between an <strong>AI construction claims platform</strong> that shortens a disclosure exercise and one that quietly manufactures a wrong answer with a citation attached to it.</p>



<p class="wp-block-paragraph">This post covers how to build that platform on AWS: how to lay out ingestion, how to make retrieval respect the contractual clock, why schedule data must never go anywhere near your vector index, how to keep privileged material out of the same retrieval path as project records, and which AWS building blocks are actually the current ones now that several of the obvious candidates have been moved to maintenance mode.</p>



<h2 class="wp-block-heading">What a claims platform actually has to answer</h2>



<p class="wp-block-paragraph">Before any architecture, be honest about the question shapes. They are not all the same problem and they do not all get solved by retrieval.</p>



<ol class="wp-block-list"><li><strong>Chronology.</strong> What happened, in what order, and on what date was it recorded? This is a retrieval and metadata problem.</li><li><strong>Entitlement.</strong> Which clause applies, and what did it require the parties to do? This is retrieval over the contract plus careful prompting.</li><li><strong>Causation.</strong> Which event moved the critical path, and by how much? This is schedule data and date arithmetic. It is not a language problem at all.</li><li><strong>Quantum.</strong> What did the disruption cost? This is cost and resource data, joined to the events above.</li></ol>



<p class="wp-block-paragraph">Treat all four as &#8220;ask the documents&#8221; and you will get fluent nonsense on two of them. The architecture below splits them deliberately.</p>



<h2 class="wp-block-heading">Failure one: retrieval that finds the best match instead of the first one</h2>



<p class="wp-block-paragraph">Two corpora live in every dispute bundle and they look identical to an embedding model.</p>



<ul class="wp-block-list"><li><strong>Contemporaneous records.</strong> Site diaries, progress emails, minutes, early warnings, RFIs, instructions. Written while the project was running, by people with no idea a dispute was coming.</li><li><strong>Claim-era material.</strong> Narratives, expert reports, without-prejudice correspondence, internal analysis. Written afterwards, specifically to be persuasive about the same events.</li></ul>



<p class="wp-block-paragraph">Claim-era material wins on cosine similarity almost every time, because it is denser in exactly the terms you searched for. If your retriever cannot distinguish them, every answer is contaminated by the argument you were trying to test.</p>



<p class="wp-block-paragraph">The fix is metadata, applied at ingestion, and it is cheap to get right and expensive to retrofit. Amazon Bedrock Knowledge Bases reads a sidecar file that sits next to each document in S3, named with the full original filename plus <code>.metadata.json</code>. So <code>letter-0421.pdf</code> gets <code>letter-0421.pdf.metadata.json</code>. The naming convention is the only link between them; there is no separate registration step.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "doc_date": 20240314,
    "corpus": "contemporaneous",
    "doc_type": "site_correspondence",
    "matter_id": "matter-0007",
    "date_source": "email_header",
    "privileged": false
  }
}</code></pre>



<p class="wp-block-paragraph">Look closely at <code>doc_date</code>. It is an integer, not a string, and that is not a style choice. Bedrock Knowledge Bases metadata attributes support STRING, NUMBER, BOOLEAN and STRING_LIST. The range comparison operators, the ones you need to express &#8220;on or before the notice deadline&#8221;, only apply to NUMBER. Store the date as <code>"2024-03-14"</code> and your filter will not throw an error. It will just quietly match nothing, or match everything, depending on how you wrote it. You will find out weeks later when someone asks why a document they can see in the bundle never appears in results.</p>



<p class="wp-block-paragraph">With the date as a sortable integer, a query filter can express the contractual window directly.</p>



<pre class="wp-block-code"><code>{
  "andAll": [
    { "equals":              { "key": "corpus",   "value": "contemporaneous" } },
    { "equals":              { "key": "matter_id","value": "matter-0007" } },
    { "greaterThanOrEquals": { "key": "doc_date", "value": 20240301 } },
    { "lessThanOrEquals":    { "key": "doc_date", "value": 20240329 } }
  ]
}</code></pre>



<p class="wp-block-paragraph">That is the whole trick. You are no longer asking &#8220;what is the most relevant document about this delay&#8221;. You are asking &#8220;what did the parties actually write during the window in which the contract required them to write it&#8221;. Those are different questions and only one of them is worth anything in a dispute.</p>



<h3 class="wp-block-heading">Where the date comes from matters more than the date</h3>



<p class="wp-block-paragraph">Do not use the S3 object timestamp. It records when someone copied a folder, usually years after the fact and identical across ten thousand files. Derive the date from the document itself: the <code>Date:</code> header on an email, the printed date on a letter, the period covered by a diary entry.</p>



<p class="wp-block-paragraph">Sometimes you cannot, because the scanned undated fax exists in every project archive. Record that honestly with a <code>date_source</code> attribute rather than guessing, and treat unknown-date documents as a separate review pile. An extension of time argument built on an inferred date is an argument you will lose.</p>



<h2 class="wp-block-heading">Failure two: treating the programme like a document</h2>



<p class="wp-block-paragraph">This one is worse, because the output looks right.</p>



<p class="wp-block-paragraph">Oracle Primavera P6 exports XER and PMXML files. Asta Powerproject and Microsoft Project have their own formats. XER in particular is a plain text dump of relational tables, so it goes through a text pipeline without complaint. Chunk it, embed it, and you now have vectors representing fragments of a table of activity codes with no relationships attached.</p>



<p class="wp-block-paragraph">Ask that index how much float activity A1200 had at the March data date and you will get a number. It will be well formatted and it will be invented. Total float is the product of a forward and backward pass across the whole logic network under a specific calendar. It cannot be recovered from a retrieved fragment, and a language model asked to produce it will produce something plausible instead of admitting that.</p>



<p class="wp-block-paragraph">Schedule data goes into a structured store, and the model queries it rather than reasoning about it.</p>



<ol class="wp-block-list"><li>Parse each programme file into tables. <code>PyP6Xer</code> handles XER from Python; MPXJ is a Java library that reads XER, PMXML, Asta Powerproject and MSPDI among others, which matters when the bundle contains four scheduling tools.</li><li>Load activities, logic links, calendars, resource assignments and WBS into Amazon Aurora PostgreSQL for interactive work, or into S3 with AWS Glue and Amazon Athena when you have hundreds of updates and want columnar scans.</li><li>Stamp every row with the <em>data date</em> of the update it came from. This is the single most important column in the whole platform. Without it you have a pile of schedules; with it you have a time series of the project&#8217;s own view of itself.</li><li>Run windows analysis, as-planned versus as-built comparison and float erosion in SQL or Python, deterministically, so the same inputs always give the same numbers.</li><li>Expose the results to the model as a tool it can call, or as generated SQL against a defined schema. The model turns a question into a query and narrates the result. It does not do the arithmetic.</li></ol>



<p class="wp-block-paragraph">A rough shape of the query that makes float erosion visible:</p>



<pre class="wp-block-code"><code>SELECT
    a.activity_id,
    a.data_date,
    a.total_float_days,
    a.total_float_days - LAG(a.total_float_days)
        OVER (PARTITION BY a.activity_id ORDER BY a.data_date)
      AS float_change
FROM   schedule_activities a
WHERE  a.project_id = 'PRJ-01'
  AND  a.data_date BETWEEN DATE '2024-01-01' AND DATE '2024-06-30'
ORDER BY a.activity_id, a.data_date;</code></pre>



<p class="wp-block-paragraph">Nothing clever there, and that is the point. Every number is traceable to a row that came from a named XER file. When an opposing expert asks where a figure came from, the answer is a file name and a query, not &#8220;the model said so&#8221;.</p>



<p class="wp-block-paragraph">Be realistic about effort here. Programme parsing and normalisation across inconsistent updates is the hardest part of the build and the part clients always underestimate. Activity IDs get reused, calendars change mid-project, and someone will have re-baselined without telling anyone. Budget for it.</p>



<h2 class="wp-block-heading">Failure three: an answer with no paper trail</h2>



<p class="wp-block-paragraph">In most RAG applications a citation is a nice touch. In dispute work it <em>is</em> the product. An answer that cannot be traced to a page of a disclosed document is not evidence, it is a rumour with good grammar.</p>



<p class="wp-block-paragraph">Design for that from the ingestion layer, not the presentation layer.</p>



<ul class="wp-block-list"><li><strong>Keep page and position.</strong> Amazon Bedrock Data Automation returns confidence scores and bounding box data alongside extracted fields, and Amazon Textract returns geometry per block. Carry both through the pipeline so a citation resolves to a page and a region, not just a file.</li><li><strong>Route low confidence to humans.</strong> Handwritten site diaries and faxed variation orders will produce low-confidence extractions. Those should land in a review queue by default rather than silently entering the index.</li><li><strong>Reject ungrounded answers.</strong> Amazon Bedrock Guardrails includes contextual grounding checks that score whether a response is supported by the retrieved passages. It reduces confident invention. It does not eliminate it, and anyone who tells you otherwise is selling something.</li><li><strong>Keep an immutable evidential copy.</strong> S3 Versioning plus S3 Object Lock on the landing bucket means the file the platform indexed is provably the file that was disclosed.</li></ul>



<p class="wp-block-paragraph">One design rule underpins all of it: the platform shortlists evidence, it does not decide entitlement. Recognised frameworks for this work, the Society of Construction Law Delay and Disruption Protocol and AACE International&#8217;s Recommended Practice 29R-03 on forensic schedule analysis, both assume a named analyst applying a stated method and exercising judgement. A system that outputs &#8220;the contractor is entitled to 42 days&#8221; is not helping. A system that outputs &#8220;here are the eleven contemporaneous documents inside the notice window, here is the float movement across those updates, here is what is missing&#8221; is doing real work.</p>



<h2 class="wp-block-heading">Failure four: one index for privileged and non-privileged material</h2>



<p class="wp-block-paragraph">Dispute bundles contain legal advice, counsel&#8217;s opinions, without-prejudice correspondence and internal settlement analysis. Those must not be retrievable through the same path as project records.</p>



<p class="wp-block-paragraph">The tempting shortcut is a <code>privileged: false</code> metadata filter on every query. Do not rely on that as your boundary. A metadata filter is a query parameter. One missing filter in one code path, one debug endpoint, one caching layer that drops it, and privileged material surfaces in a general search. The blast radius of that mistake is not a bug report.</p>



<p class="wp-block-paragraph">Separate the indexes physically and separate the IAM roles that can reach them. Amazon S3 Vectors makes this practical: you can set a dedicated customer-managed KMS key per vector index, and you get a large number of indexes per vector bucket, so per-matter and per-sensitivity separation does not become an operational burden. Keep the metadata flag as well, because defence in depth is free, but make the identity boundary the one you actually trust.</p>



<p class="wp-block-paragraph">Amazon Macie is worth pointing at the landing bucket to find personal data you did not expect, particularly in HR records and accident reports that get swept into project archives.</p>



<h2 class="wp-block-heading">Choosing the AWS building blocks, including what not to build on</h2>



<p class="wp-block-paragraph">A lot of published architectures for this kind of platform are now pointing at services AWS has stopped developing. Two matter here, and the dates are the point.</p>



<ul class="wp-block-list"><li><strong>Amazon Kendra</strong> entered maintenance mode on 30 June 2026 and stops accepting new customers on 30 July 2026. Existing customers keep support and security fixes but no new capability. AWS directs new enterprise search and RAG work to Amazon Bedrock Knowledge Bases. If a tutorial or a proposal you are reading starts with a Kendra index, it predates that change.</li><li><strong>Amazon Bedrock Agents</strong> moved to maintenance mode in the same round of service availability changes, with Amazon Bedrock AgentCore as the successor for agentic orchestration. Check the current AWS service availability page before you commit an orchestration layer.</li></ul>



<p class="wp-block-paragraph">For the retrieval layer itself, Bedrock Knowledge Bases now comes in two shapes and the choice is a real trade-off rather than a marketing tier.</p>



<h3 class="wp-block-heading">Managed Knowledge Base</h3>



<p class="wp-block-paragraph">AWS manages the vector store, embeddings model, re-ranker and retrieval orchestration as a single primitive, with native connectors for Amazon S3, SharePoint, Confluence, Google Drive, OneDrive and a web crawler, plus automatic parsing strategy selection and a retriever that decomposes multi-step queries. The connectors pull source permissions along with content, which matters when the document set lives in the client&#8217;s SharePoint rather than a bucket you control.</p>



<p class="wp-block-paragraph">Where it wins: you get a working retrieval layer in an afternoon instead of a fortnight, and the parsing tuning that normally eats the first weeks of a build is done for you. For a first matter, or a proof of value before a client commits budget, this is the one I would reach for.</p>



<h3 class="wp-block-heading">Custom Knowledge Base</h3>



<p class="wp-block-paragraph">You bring your own vector store and control chunking, embedding model and index layout.</p>



<p class="wp-block-paragraph">Where it wins: claims work has awkward chunking requirements. A two-page letter split mid-sentence at a page boundary produces a chunk where the notice sentence has lost its date and its addressee. Controlling chunk boundaries around document structure, and controlling which index a document lands in, are both easier when you own the store. Where it doesn&#8217;t: you now own embedding model upgrades, re-indexing, sync failures and capacity, which is real ongoing work for a small team.</p>



<p class="wp-block-paragraph">Start managed, build a retrieval evaluation set of real questions with known correct documents, and only move to custom when that set demonstrates the problem is chunking. Most teams migrate on a hunch and discover the problem was metadata all along.</p>



<h3 class="wp-block-heading">Where the vector storage bill actually comes from</h3>



<p class="wp-block-paragraph">Rates change, so learn the billing mechanism rather than a number. Amazon S3 Vectors charges on three axes: upload volume by logical gigabyte, storage by logical gigabyte, and queries by data processed, where data processed scales with the size of the index being searched. Note that filtering does not reduce the data processed by a query.</p>



<p class="wp-block-paragraph">That shape suits claims work unusually well. A dispute archive is enormous and cold: millions of chunks, queried by a handful of analysts a few hundred times a day, so you pay mostly for storage, which is the cheap axis. Compare that against Amazon OpenSearch Serverless, which prices on provisioned compute units and therefore rewards high query volume against a smaller index, or Aurora PostgreSQL with pgvector when you already need Aurora for the schedule tables and would rather run one system than two.</p>



<p class="wp-block-paragraph">The practical lever is to split indexes per matter. Query cost scales with index size, so one giant index across every dispute you have ever run makes every query more expensive than it needs to be, on top of being a bad idea for confidentiality.</p>



<h2 class="wp-block-heading">A reference pipeline</h2>



<ol class="wp-block-list"><li>Everything lands in S3 under a per-matter prefix, with Versioning and Object Lock enabled on the evidential copy.</li><li>S3 event notifications trigger AWS Step Functions. Use Step Functions rather than a chain of Lambdas so that a failed extraction on page 300 of a 400-page bundle is visible and resumable.</li><li>Classify and split. Scanned bundles arrive as one PDF containing forty separate documents. Splitting them correctly is a prerequisite for dating them correctly.</li><li>Extract text with Amazon Bedrock Data Automation or Amazon Textract, keeping confidence scores and geometry.</li><li>Derive the document date and write the <code>.metadata.json</code> sidecar. Anything undated goes to the review queue.</li><li>Route by type: correspondence to the knowledge base, programme files to the XER parser and the relational store, cost data to its own tables.</li><li>Sync the knowledge base, then run your retrieval evaluation set before anyone uses it. A sync that succeeds is not the same as an index that answers correctly.</li><li>Serve through an API that refuses to return an answer without citations, and log every query with the filters that were applied.</li></ol>



<p class="wp-block-paragraph">Define the whole thing in Terraform or OpenTofu from the start. Matters are per-client and short-lived, and standing one up should be a variable file, not an afternoon in the console. Point Amazon CloudWatch, or Grafana Cloud if you already run Grafana elsewhere, at the Step Functions execution metrics so a silently failing extraction stage does not go unnoticed for a week.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Date filters return nothing, and no error</h3>



<p class="wp-block-paragraph">Almost always the date was stored as a string. Range operators need NUMBER. Convert to an integer in <code>YYYYMMDD</code> form and re-sync the affected documents.</p>



<h3 class="wp-block-heading">A document is in the bucket but never appears in results</h3>



<p class="wp-block-paragraph">Check the sidecar filename first. It must be the complete original filename with <code>.metadata.json</code> appended, extension included. <code>report.pdf.metadata.json</code> works; <code>report.metadata.json</code> is a file the ingestion job will happily ignore. After that, check whether a filter in the query path is excluding it.</p>



<h3 class="wp-block-heading">Answers cite the right document but the wrong passage</h3>



<p class="wp-block-paragraph">Chunking split the document somewhere structurally meaningful. Look at the raw chunks for that file. If the notice sentence and its date are in different chunks, no amount of prompt tuning fixes it. That is the signal to take control of chunking.</p>



<h3 class="wp-block-heading">Float figures do not match the client&#8217;s own analysis</h3>



<p class="wp-block-paragraph">Check calendars before you check logic. Different activity calendars, a changed default calendar, or an update where someone applied a progress override will move float without any logic change. Reconcile activity counts between your parsed tables and the source file before trusting anything downstream.</p>



<h3 class="wp-block-heading">Query costs jumped without more usage</h3>



<p class="wp-block-paragraph">An index grew. With storage-side vector search, query cost tracks the size of the index being scanned, so ingesting a large new bundle raises the price of every subsequent query against that index. Split by matter.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Using the file&#8217;s storage timestamp as the document date. It records the migration, not the event.</li><li>Indexing claim narratives and contemporaneous records into the same corpus with no way to tell them apart.</li><li>Embedding programme exports because they happen to be text files.</li><li>Treating a metadata filter as a privilege boundary instead of an optimisation.</li><li>Letting the model state entitlement conclusions rather than assembling and citing evidence.</li><li>Building on services that have moved to maintenance mode because the tutorial you followed predates the change.</li><li>Shipping without a retrieval evaluation set, so you have no way to know whether a change made things better or worse.</li><li>One index for every matter, which is both a cost problem and a confidentiality problem.</li></ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list"><li>Make the document date a first-class, numeric, filterable attribute, and record where it came from.</li><li>Keep an immutable evidential copy separate from the working copy the pipeline mutates.</li><li>Separate structured schedule and cost data from unstructured documents, and let the model query the former rather than reason about it.</li><li>Build a retrieval evaluation set from real questions with known correct documents before you tune anything.</li><li>Enforce citations at the API layer, so an uncited answer is impossible rather than discouraged.</li><li>Isolate privileged material by index and by IAM role, with metadata as a second layer.</li><li>Log every query with its filters, so you can reconstruct how any given answer was reached.</li><li>Define infrastructure as code so a new matter is a deployment, not a project.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Can an AI construction claims platform replace a delay expert?</h3>



<p class="wp-block-paragraph">No, and building toward that goal produces something unusable. Established forensic frameworks assume a named analyst applying a stated method whose reasoning can be tested. The platform&#8217;s value is compressing weeks of document review into hours and making the schedule data queryable, so the expert spends their time on judgement rather than searching.</p>



<h3 class="wp-block-heading">Should I use Amazon Kendra for the search layer?</h3>



<p class="wp-block-paragraph">Not for a new build. Kendra entered maintenance mode on 30 June 2026 and closed to new customers on 30 July 2026, with AWS pointing to Bedrock Knowledge Bases for equivalent and more current capability. Existing Kendra deployments continue to be supported, so this is a migration assessment rather than an emergency, but starting there now means starting on a service with no roadmap.</p>



<h3 class="wp-block-heading">How do I stop the model inventing float and delay figures?</h3>



<p class="wp-block-paragraph">Do not give it the chance. Keep schedule data in a relational or columnar store and have the model generate queries against a defined schema, or call a tool that runs a fixed calculation. The arithmetic happens in SQL or Python where it is deterministic and reproducible; the model only turns questions into queries and results into sentences.</p>



<h3 class="wp-block-heading">Which vector store should I choose for a claims archive?</h3>



<p class="wp-block-paragraph">Match the store to your query pattern. Large, cold archives queried by a few analysts favour storage-priced options like Amazon S3 Vectors, where you mostly pay to keep the data. Smaller indexes hit constantly favour compute-priced options like Amazon OpenSearch Serverless. If you already run Aurora PostgreSQL for schedule data, pgvector alongside it is a legitimate way to avoid operating a second system.</p>



<h3 class="wp-block-heading">How do I handle scanned and handwritten site records?</h3>



<p class="wp-block-paragraph">Extract them with confidence scores retained, set a threshold, and route everything below it to human review before indexing. Handwritten diaries are frequently the most probative documents in a delay claim and also the least reliable to read automatically, so the review queue is not an edge case. Plan capacity for it.</p>



<h3 class="wp-block-heading">Where do documents come from if they are not already in S3?</h3>



<p class="wp-block-paragraph">Most project records live in a common data environment such as Procore, Autodesk Construction Cloud, Aconex or a client SharePoint tenancy. Bedrock Managed Knowledge Base has native connectors for SharePoint, Confluence, Google Drive and OneDrive that ingest permissions alongside content. For platforms without a native connector, export to S3 and keep the export manifest as part of the disclosure record.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An <strong>AI construction claims platform</strong> lives or dies on whether it understands time. Every hard requirement in this build traces back to that: numeric dates so you can filter to a contractual window, a data date on every schedule row so float movement is measurable, a corpus flag so contemporaneous records are not drowned out by material written to argue about them, and citations that resolve to a page so any answer can be checked.</p>



<p class="wp-block-paragraph">Get the temporal metadata right at ingestion and the rest of the architecture is ordinary AWS work. Get it wrong and you have built a very expensive way to retrieve the most persuasive document instead of the true one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building this on AWS?</h2>



<p class="wp-block-paragraph">I design and build document and data platforms on AWS, and this kind of system sits squarely in that work. Things I can help with:</p>



<ul class="wp-block-list"><li>Designing the ingestion pipeline: S3 landing zones with Object Lock, Step Functions orchestration, splitting and classifying scanned bundles, and confidence-based routing to human review.</li><li>Getting the temporal metadata model right, including date derivation, sidecar generation and filter design against Amazon Bedrock Knowledge Bases.</li><li>Parsing Primavera P6 XER and PMXML exports into queryable tables in Aurora PostgreSQL or S3 with Glue and Athena, with a data date on every row.</li><li>Choosing and sizing the vector layer across Amazon S3 Vectors, OpenSearch Serverless and pgvector, based on your actual query pattern rather than a benchmark.</li><li>Building index and IAM separation for privileged material, plus KMS key strategy and Macie scanning of landing buckets.</li><li>Setting up retrieval evaluation, citation enforcement, query audit logging and CloudWatch or Grafana dashboards over the pipeline so failures surface early.</li></ul>



<p class="wp-block-paragraph">If you are partway into something like this already, send me a sample metadata sidecar, a Step Functions execution history, or a query that returns the wrong document, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/case-studies/ai-construction-claims-platform-aws/">Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Build a Secure AI Medical Assistant on AWS: The Boundaries That Actually Leak</title>
		<link>https://john-nessime.com/blog/case-studies/secure-ai-medical-assistant-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 03 Aug 2026 13:45:12 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Comprehend Medical]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Healthcare AI]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[PHI]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Vector Database]]></category>
		<category><![CDATA[VPC]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=127</guid>

					<description><![CDATA[<p>A practical architecture for a secure AI medical assistant on AWS, organised by the boundary the data crosses: the input box, your own invocation logs, cross-Region inference routing, the retrieval index, and clinical accuracy. Includes real commands, the failure modes that stay invisible until an audit, and the trade-offs worth knowing before you build.</p>
<p>The post <a href="https://john-nessime.com/blog/case-studies/secure-ai-medical-assistant-aws/">Build a Secure AI Medical Assistant on AWS: The Boundaries That Actually Leak</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on a Friday afternoon: &#8220;One of the residents pasted a real discharge summary into the demo.&#8221; Nobody meant anything by it. The thing was a study aid, a chat box over a pile of reference material, running in a sandbox account with no Business Associate Addendum in place and invocation logging switched on because logging is a good habit. And now there is protected health information sitting in plaintext in a CloudWatch log group in an account that was never in scope for it.</p>



<p class="wp-block-paragraph">That is the failure mode worth planning for. Not a jailbreak, not a model saying something clinically wrong on stage. A user typing something perfectly reasonable into a box you built, and the data ending up somewhere you never drew on the diagram. This post walks through how to build a secure AI medical assistant on AWS, organised by the boundary the data actually crosses: the input box, your own logs, the Region the inference runs in, the retrieval index, and finally the answer itself. Commands are included where they explain something. Where a value depends on your account or your counsel, I say so instead of making one up.</p>



<h2 class="wp-block-heading">Two different products hiding behind one request</h2>



<p class="wp-block-paragraph">&#8220;An AI assistant for medical students and doctors&#8221; is two builds with two risk profiles, and conflating them is the root of most of the trouble.</p>



<ul class="wp-block-list">
<li><strong>The study tool.</strong> Question banks, guideline summaries, differential drills, spaced repetition. In theory it never touches patient data. Its real risk is confident wrongness and unattributed answers, not privacy.</li>

<li><strong>The clinical assistant.</strong> Note summarisation, coding support, chart question answering. It handles PHI by design, so the whole thing has to sit inside a HIPAA-designated account from day one.</li>
</ul>



<p class="wp-block-paragraph">The trap is that the study tool becomes the clinical assistant without anyone shipping a release. A student rehearses a case they saw on the ward. A doctor tries the study tool on a real chart because it is the one that is already open. The moment your input box accepts free text from someone with clinical access, you should assume PHI will arrive in it. Build accordingly, or put the study tool on infrastructure where PHI arriving is survivable.</p>



<p class="wp-block-paragraph">My default is to run both in the same HIPAA-designated account with the same controls, and keep only the marketing site, the docs and the waitlist form somewhere ordinary and cheap like InterServer or any commodity host, entirely outside the AWS organisation. Small blast radius beats clever separation you have to explain to an auditor.</p>



<h2 class="wp-block-heading">Boundary one: PHI arrives before you decide to accept it</h2>



<p class="wp-block-paragraph">Start with the thing that trips up almost every first build: HIPAA-eligible and HIPAA-compliant are not the same word. AWS designating a service as HIPAA-eligible means you are permitted to process PHI with it under an executed Business Associate Addendum. It says nothing about whether your deployment is compliant. That part is entirely yours.</p>



<p class="wp-block-paragraph">Three conditions have to hold together before any PHI touches a service: the service is on the AWS HIPAA Eligible Services Reference, you have an executed BAA, and the account is designated for HIPAA use. Amazon Bedrock and Amazon Comprehend Medical both appear on that list, and the BAA is accepted through AWS Artifact rather than negotiated by email. Check the reference page yourself before you commit to an architecture, because the list changes and every component in your diagram has to be on it, not just the model.</p>



<h3 class="wp-block-heading">Detecting PHI at the door</h3>



<p class="wp-block-paragraph">Amazon Comprehend Medical has an operation specifically for finding protected health information in unstructured clinical text. You hand it free text, it returns detected entities with a type, a confidence score and character offsets.</p>



<pre class="wp-block-code"><code>aws comprehendmedical detect-phi 
  --region us-east-1 
  --text "Patient seen for chest pain, MRN 004512, discharged Tuesday."</code></pre>



<p class="wp-block-paragraph">The offsets are the useful part. They let you redact or tokenise in place before the text goes anywhere else, rather than throwing the whole message away and telling the user to try again.</p>



<p class="wp-block-paragraph">Now the caveat that matters more than the feature. AWS states plainly that Comprehend Medical may not identify PHI in all circumstances and that it does not, on its own, meet HIPAA&#8217;s requirements for de-identification. Read that as: it is a good filter and a terrible guarantee. If your compliance story is &#8220;we strip PHI before the model sees it, so we are outside HIPAA scope,&#8221; that story does not hold. Treat detection as defence in depth inside a compliant account, not as an escape hatch from needing one.</p>



<h2 class="wp-block-heading">Boundary two: your own logs are the most likely leak</h2>



<p class="wp-block-paragraph">This is the invisible one. Bedrock model invocation logging is disabled by default and captures the full request data, response data and metadata for every call in the account, in that Region. You turn it on for a good reason, usually because someone in security asked who prompted what and when. Then it quietly becomes the largest concentration of raw clinical text you own, and it is nowhere on the architecture diagram because it is a checkbox rather than a component.</p>



<p class="wp-block-paragraph">Check whether it is on before you assume anything:</p>



<pre class="wp-block-code"><code>aws bedrock get-model-invocation-logging-configuration --region us-east-1</code></pre>



<p class="wp-block-paragraph">The configuration is per account per Region, so run it in every Region where anyone has ever opened the Bedrock console. Setting it deliberately looks like this:</p>



<pre class="wp-block-code"><code>aws bedrock put-model-invocation-logging-configuration 
  --region us-east-1 
  --logging-config '{
    "s3Config": {
      "bucketName": "med-assistant-invocation-logs",
      "keyPrefix": "bedrock/"
    },
    "textDataDeliveryEnabled": true,
    "imageDataDeliveryEnabled": false,
    "embeddingDataDeliveryEnabled": false
  }'</code></pre>



<p class="wp-block-paragraph">Each delivery flag is a separate decision about a separate category of PHI. Text is the obvious one. Image delivery matters the moment anyone uploads a photographed chart or a scan, because burned-in identifiers travel with the pixels and no text filter will ever see them. Embedding delivery is the one people leave on without thinking; vectors derived from clinical text are not a safe artefact, and they are large.</p>



<p class="wp-block-paragraph">The sharpest detail is buried in the Guardrails documentation: AWS notes you can disable invocation logs if you do not want blocked content appearing as plaintext in them. Read the implication. A guardrail can refuse a prompt, mask the identifiers, and stop the model ever seeing them, and the original text still lands in your log destination. The guardrail protects the model call. It does not protect the log.</p>



<ul class="wp-block-list">
<li>Encrypt the log destination with a customer-managed KMS key, and keep the key policy tight enough that &#8220;everyone with S3 read&#8221; is not also &#8220;everyone with chart access&#8221;.</li>

<li>Set a retention period that reflects a legal decision someone actually made, not the CloudWatch default of never expiring.</li>

<li>Send logs to a separate, tightly scoped account if your organisation is large enough to have people who need dashboards but not records.</li>

<li>Remember your application logs too. A framework that logs request bodies on error will do this to you long before Bedrock does.</li>
</ul>



<h2 class="wp-block-heading">Boundary three: where the inference actually runs</h2>



<p class="wp-block-paragraph">Cross-Region inference in Bedrock exists because on-demand capacity is uneven and bursts happen. It routes your request to another Region to get it served. There are two flavours and the difference is not cosmetic.</p>



<ul class="wp-block-list">
<li><strong>Geographic profiles</strong> keep routing inside a defined geography such as the US or the EU. A request that starts in the EU stays in EU Regions. This is the one built for residency requirements.</li>

<li><strong>Global profiles</strong> route to supported commercial Regions worldwide for maximum throughput. AWS documents that a request can be routed to a destination Region even if you never opted that Region into your account.</li>
</ul>



<p class="wp-block-paragraph">To be fair to global profiles: data is not stored in the destination Region, transfer happens encrypted across the AWS network, and your invocation logs, knowledge bases and configuration all stay in the source Region. For a workload with no geographic constraint it is a genuinely good default, and it typically carries a lower per-token rate than staying in-geography. For a clinical workload with a residency commitment in a contract, it is the wrong tool, and &#8220;the prompt left the geography but was not stored there&#8221; is a sentence you do not want to be constructing during an audit.</p>



<p class="wp-block-paragraph">Pin it in policy rather than trusting a config value in a repo. A Service Control Policy denying the Bedrock API outside your approved Regions closes both doors at once, because invoking a cross-Region profile requires model access in the destination Regions as well as the source:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyBedrockOutsideApprovedRegions",
      "Effect": "Deny",
      "Action": "bedrock:*",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The trade-off is real and you should know it before you apply this. If you later adopt a geographic profile whose destination list includes a Region you denied, invocations fail with an access error that looks nothing like a routing problem. Keep the approved Region list and the profile&#8217;s destination list in the same review, not in two different tickets.</p>



<h2 class="wp-block-heading">Boundary four: retrieval, and the grounding check that will not save you</h2>



<p class="wp-block-paragraph">A useful medical assistant is retrieval-augmented. The model alone is a fluent generalist; the value comes from grounding answers in a curated corpus, whether that is institutional guidelines, formulary rules or a licensed reference set. Bedrock Knowledge Bases will manage the ingestion and the vector store for you, or you can run your own index in Amazon OpenSearch Service.</p>



<p class="wp-block-paragraph">Two things bite here.</p>



<p class="wp-block-paragraph"><strong>Tenant isolation in the index.</strong> If you serve more than one hospital, department or study cohort, a shared index with a filter applied in application code is one refactor away from cross-tenant retrieval. Filters are easy to forget and impossible to notice, because a wrong answer that cites a real document looks exactly like a right one. Separate indexes per tenant, or fine-grained access control enforced below your application, cost more and fail safe.</p>



<p class="wp-block-paragraph"><strong>The grounding check has a scope limit.</strong> Guardrails contextual grounding checks detect responses that are not supported by the source material or not relevant to the question. Excellent feature. The documentation states the supported use cases are summarisation, paraphrasing and question answering, and that conversational chatbot use cases are not supported. If your product is a chat interface with turn history, do not assume this check is covering you. There is also a streaming wrinkle: relevance is assessed per chunk, so an irrelevant response can reach the user before it is marked irrelevant at the end of the stream.</p>



<p class="wp-block-paragraph">What you can rely on regardless of interface is applying a guardrail to arbitrary text directly, which is how you check an input before it enters your own pipeline:</p>



<pre class="wp-block-code"><code>import boto3

client = boto3.client("bedrock-runtime", region_name="us-east-1")

response = client.apply_guardrail(
    guardrailIdentifier=GUARDRAIL_ID,
    guardrailVersion="DRAFT",
    source="INPUT",
    content=[{"text": {"text": user_message}}],
)

if response["action"] == "GUARDRAIL_INTERVENED":
    # Stop here. Do not forward, and do not write the raw text anywhere.
    handle_blocked(response["assessments"])</code></pre>



<p class="wp-block-paragraph">Note what the comment is doing. The most common bug in this pattern is catching the intervention and then logging the offending input &#8220;for debugging&#8221;, which reintroduces exactly the leak the guardrail just prevented. Log the assessment, log a request identifier, never the text.</p>



<p class="wp-block-paragraph">Sensitive information filters give you two handling modes: block, which rejects the request outright, and mask, which replaces detected entities with placeholder tags. For a study tool, mask is usually right, because a student who typed a name by reflex gets a useful answer instead of a wall. For a clinical assistant working over charts, blocking on unexpected identifiers in an input that should have arrived structured is a better signal that something upstream is wrong.</p>



<h2 class="wp-block-heading">Boundary five: being confidently wrong</h2>



<p class="wp-block-paragraph">Everything above is about data leaving. This one is about a bad answer arriving, and for a medical audience it is the reputational failure that actually ends products.</p>



<ul class="wp-block-list">
<li><strong>Cite or refuse.</strong> Return the retrieved passages alongside the answer. If retrieval returned nothing above your relevance threshold, say so rather than letting the model answer from parametric memory. A student cannot verify what they cannot see.</li>

<li><strong>Version the corpus, not just the model.</strong> When a guideline changes, you need to know which answers were generated against the old text. Store a corpus revision identifier with every logged response.</li>

<li><strong>Use denied topics deliberately.</strong> Dosing for a named patient, and anything that reads as a treatment directive rather than reference information, are reasonable things to route to a refusal with a clear explanation.</li>

<li><strong>Get regulatory advice early.</strong> Whether clinical decision support software is regulated as a medical device depends on your jurisdiction and, critically, on the claims you make about it. This is a legal question with engineering consequences, and it is much cheaper to answer before the interface exists.</li>
</ul>



<h2 class="wp-block-heading">A build order for a secure AI medical assistant on AWS</h2>



<p class="wp-block-paragraph">Sequence matters here more than in most builds, because several of these are painful to retrofit.</p>



<ol class="wp-block-list">
<li>Accept the BAA through AWS Artifact and designate the account for HIPAA use. Do this before the first prototype, not before the first customer.</li>

<li>Pin Regions with a Service Control Policy, and decide the geographic-versus-global inference profile question in writing.</li>

<li>Configure invocation logging deliberately, with a customer-managed KMS key, an explicit retention period, and each data-type delivery flag chosen rather than defaulted.</li>

<li>Put the application in private subnets and reach Bedrock over VPC endpoints so PHI-bearing traffic does not traverse the public internet.</li>

<li>Build the guardrail before the prompt. Sensitive information filters, denied topics, and grounding checks where they apply.</li>

<li>Add Comprehend Medical detection in the ingestion path for anything you are storing, and at the input boundary for anything a user types.</li>

<li>Build retrieval with tenant isolation from the first index, not the second.</li>

<li>Only now write the assistant&#8217;s prompt and interface, and put a WAF such as Cloudflare or AWS WAF in front of the public endpoint.</li>
</ol>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>AccessDenied on a model that clearly works elsewhere.</strong> Usually one of three things: model access not requested in this Region, a cross-Region profile whose destination Regions your SCP denies, or an IAM policy that grants the model in the source Region only.</li>

<li><strong>No invocation logs appearing.</strong> The configuration is per Region and disabled by default. Confirm you queried the same Region the application calls, and that the delivery flag for the data type you expect is enabled.</li>

<li><strong>Guardrail passes text you expected it to catch.</strong> Sensitive information detection is probabilistic and context-dependent. Very short inputs give it little to work with. Test with realistic clinical phrasing, not single tokens, and add regex patterns for structured identifiers like MRNs that follow a house format.</li>

<li><strong>Grounding check appears to do nothing.</strong> Check your interface shape against the supported use cases before assuming it is misconfigured, and check whether streaming is masking the result until the response completes.</li>

<li><strong>Retrieval returns plausible but wrong documents.</strong> Look at chunking before you look at the model. Clinical guidelines chunked mid-table or mid-criteria retrieve badly no matter what embedding you use.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Prototyping in a personal or sandbox account and promising to migrate later. The prototype is where the first real chart gets pasted.</li>

<li>Assuming the eligibility of Bedrock covers the whole stack. Every component that touches PHI needs to be eligible, including the vector store, the queue and the cache.</li>

<li>Treating PHI detection as de-identification. AWS says explicitly that it is not.</li>

<li>Leaving image and embedding log delivery enabled by copy-paste.</li>

<li>Logging blocked prompts to debug the guardrail.</li>

<li>Shipping a chat interface and citing contextual grounding as the hallucination control.</li>

<li>Filtering tenants in application code over a shared index.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Write the data-flow diagram with logs, backups and the vector index drawn as first-class destinations. If PHI can land there, it is on the diagram.</li>

<li>Define everything in Terraform or OpenTofu so the guardrail, the logging configuration and the SCP are reviewable artefacts rather than console state.</li>

<li>Keep a small evaluation set of realistic clinical questions with known-good answers, and run it on every prompt or model change.</li>

<li>Alarm on guardrail intervention rate. A sudden rise usually means a change upstream, not a change in users.</li>

<li>Dashboard invocation counts, latency and intervention rates somewhere your on-call actually looks, whether that is CloudWatch, Grafana or Datadog.</li>

<li>Scope IAM to specific model ARNs and specific guardrail identifiers. A wildcard on <code>bedrock:InvokeModel</code> means any model, including ones you never evaluated.</li>

<li>Rehearse the breach path once. Knowing which bucket, which log group and which key you would need to reason about is worth an afternoon.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Amazon Bedrock HIPAA compliant?</h3>



<p class="wp-block-paragraph">Bedrock is HIPAA-eligible, which is a different claim. Eligibility means AWS permits you to process PHI with the service under an executed BAA. Compliance is a property of your deployment: your account designation, encryption, access control, network isolation, logging and retention. A Bedrock workload with a signed BAA and public endpoints and unbounded plaintext logs is not compliant.</p>



<h3 class="wp-block-heading">Do I need a BAA if the tool is only for medical students?</h3>



<p class="wp-block-paragraph">If the tool genuinely never receives PHI, HIPAA is not engaged. The practical question is whether you can guarantee that, given that your users have clinical access and a free-text box. If you cannot, get the BAA. It costs you a self-service acceptance in AWS Artifact and removes an entire category of incident.</p>



<h3 class="wp-block-heading">Does AWS use my prompts to train the models?</h3>



<p class="wp-block-paragraph">AWS states that customer content submitted to Bedrock is not used to train the underlying foundation models or shared with model providers. That statement is the sort of thing a hospital security review will want quoted verbatim from the current AWS data protection documentation rather than from a blog, so pull the live wording when you write your assessment.</p>



<h3 class="wp-block-heading">Is Comprehend Medical enough to de-identify clinical text?</h3>



<p class="wp-block-paragraph">No. AWS documents that it may not identify PHI in all circumstances and does not by itself meet HIPAA&#8217;s de-identification requirements. Use it as a detection layer and a redaction aid inside a compliant environment. Formal de-identification, whether by the Safe Harbor method or expert determination, is a separate exercise with its own review.</p>



<h3 class="wp-block-heading">Where should the vector store live?</h3>



<p class="wp-block-paragraph">In the same account and Region as the rest of the workload, on a HIPAA-eligible service, encrypted with a customer-managed key, reachable only from private subnets. Embeddings derived from clinical text are not sanitised data and should not be treated as a lower-sensitivity artefact than the source.</p>



<h3 class="wp-block-heading">Should I use a global or geographic inference profile?</h3>



<p class="wp-block-paragraph">Geographic if you have any residency commitment, contractual or regulatory. Global if you have none and want the throughput and the lower per-token rate. Decide once, document the reasoning, and enforce it with a Service Control Policy rather than a configuration constant.</p>



<h3 class="wp-block-heading">Does an AI medical assistant count as a medical device?</h3>



<p class="wp-block-paragraph">It depends on your jurisdiction and on what you claim the software does. Software that surfaces reference information a clinician independently reviews has generally been treated differently from software that directs a clinical decision, but the boundary is fact-specific and moves. This is a question for regulatory counsel before launch, not a question for your architecture diagram.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A secure AI medical assistant on AWS is not mainly a model problem. Bedrock, Guardrails and Comprehend Medical are the easy part, and the documentation for them is good. The hard part is that PHI leaves through the paths you did not draw: an invocation log you enabled for good reasons, a global inference profile that routes wherever capacity exists, an index shared between tenants, a debug line added at two in the morning.</p>



<p class="wp-block-paragraph">So build the boundary first and the assistant second. Get the BAA accepted, pin the Regions in policy, decide consciously what your logs are allowed to hold, and isolate retrieval per tenant before there is a second tenant. Every one of those is cheap on day one and expensive in month six.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on your build?</h2>



<p class="wp-block-paragraph">I work with teams building AI on AWS where the data is sensitive and the failure modes are quiet. Things I can help with on a project like this:</p>



<ul class="wp-block-list">
<li>Reviewing a Bedrock architecture against the boundaries above and telling you where PHI can actually land</li>

<li>Setting up account separation, BAA scope, Region pinning with Service Control Policies, and VPC endpoint access to Bedrock</li>

<li>Designing and tuning Guardrails policies, including custom regex for house identifier formats, and the block-versus-mask decision per surface</li>

<li>Building the retrieval layer with per-tenant isolation, sensible clinical chunking, and citation-or-refuse behaviour</li>

<li>Auditing invocation logging, KMS key policies, retention and application-level log hygiene for accidental PHI capture</li>

<li>Putting the whole thing in Terraform or OpenTofu so your controls are reviewable instead of remembered</li>
</ul>



<p class="wp-block-paragraph">If any of that is on your plate, send me the piece you are least sure about. A redacted architecture diagram, a guardrail configuration, a logging policy, an <code>AccessDenied</code> you cannot explain. I would rather look at the real thing than talk in generalities.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/case-studies/secure-ai-medical-assistant-aws/">Build a Secure AI Medical Assistant on AWS: The Boundaries That Actually Leak</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
