That is the shape of every serious problem in this space. A personalized learning assistant on AWS almost never fails loudly. It fails by being helpful to the wrong person, or by being confidently wrong in a way that reads as authoritative because it came from the institution’s own system. Both failures are silent until a human reports them, and by then the assistant has been answering that way for a term.

This post covers the parts that are genuinely hard: getting identity and scope out of the learning platform instead of the browser, making retrieval respect the boundary, knowing what Bedrock Guardrails will and will not do for you, sizing a system whose traffic collapses every summer, and proving after the fact what the assistant saw. I assume you can already stand up a RAG pipeline. The pipeline is the easy part.

What “personalized” has to mean before you build anything

Personalization in an education setting is three separate questions that people tend to collapse into one:

• What is this learner allowed to see? Course enrollment, section, release dates, role. This is an authorization question and it belongs nowhere near the model.
• What has this learner already done? Submissions, attempts, viewed material. This is state, and it belongs in a database you control.
• How is this learner doing? Mastery, misconceptions, pace. This is inference, and it is the only part where a model earns its keep.

Teams that skip straight to the third question end up asking the model to enforce the first one through prompt instructions. That does not hold. A system prompt saying “do not reveal solutions before the due date” is a suggestion; a retrieval filter that never returns the solution chunk is a control. Keep the boundary in the retrieval layer and the personalization in the generation layer, and most of the scary failure modes stop being possible rather than becoming unlikely.

Where a personalized learning assistant on AWS actually breaks

Organized by failure family rather than by service, because the services are documented and the failures are not. Five of them, roughly in the order they bite.

Failure family one: identity that stops at the front door

The assistant lives inside Canvas, Moodle, Blackboard, Brightspace or Open edX. It gets there through LTI 1.3, which replaced the old shared-secret model with an OpenID Connect launch and an RSA-signed JWT. The platform initiates a login against your tool’s login endpoint, your tool responds with an auth request, the platform posts back a signed token, and you validate it against the platform’s published keys.

Almost everyone gets that far. The failure is what happens next.

The launch token carries a stable subject identifier, a roles claim, a context claim describing the course, and a deployment identifier, all namespaced under the 1EdTech spec URIs. Teams validate the signature, pull the user ID out, then build the rest of the session from whatever the front end sends afterwards. Course ID as a query parameter. Role in local storage. It works in testing because the front end sends the truth, and it stops working the moment a curious student opens developer tools. If your filter values arrive from the client, you have built an access control system that asks the attacker what they should be allowed to see.

Take the scope from the launch, then keep it server-side

The pattern that holds up:

1. Validate the launch JWT against the platform’s key set. Check the issuer, the audience, the nonce and the deployment identifier, not just the signature.
2. Extract subject, roles and context from the token and write them into a server-side session. Hand the browser an opaque session cookie and nothing else.
3. Derive every retrieval filter from that session record. The client never names a course, a section or a role.
4. If you need the full roster rather than just the launching user, call Names and Role Provisioning Services with an OAuth 2.0 client credentials token. That gives you learners who have never launched the tool, which matters for cohort-level features.
5. If you write grades back, do it through Assignment and Grade Services against a line item you created. A launch that works tells you nothing about whether passback works.

Test launch, roster fetch and grade passback as three independent contracts. They fail independently and they fail quietly. A term can start with a tool that launches beautifully and drops every grade on the floor because AGS was never wired to a line item.

Worth knowing before you commit: LTI 1.3 is meaningfully more setup than 1.1. Key registration, the OIDC handshake, per-deployment configuration on every platform instance. That is the price of not signing student traffic with a shared secret, and it is worth paying, but budget for it honestly. If you need to test against several platforms, a self-managed Moodle or Open edX instance on a cheap VPS from somewhere like Contabo or InterServer beats waiting on sandbox tenants, and you get to break it freely.

Failure family two: retrieval that ignores the boundary

This is the one from the opening. Vector search does not know about courses. It knows about similarity. If the answer key is semantically close to the question, it comes back, and the model uses it, and the response is fluent and correct and completely inappropriate.

Amazon Bedrock Knowledge Bases handles this through metadata filtering: you attach attributes to each document at ingestion, then pass a filter at query time that pre-filters the vector store before the similarity search runs. The filter is the control. Everything else is decoration.

The sidecar file is the only metadata that counts

For an S3 data source, attributes come from a companion file sitting next to the document with .metadata.json appended to the object key. This trips people up constantly, because S3 objects already have user-defined metadata and object tags, and neither of those is read for retrieval filtering. Keep lifecycle and cost allocation in S3 tags where they belong, and put filterable attributes in the sidecar.

{
  "metadataAttributes": {
    "course_id": "BIO201",
    "section_id": "BIO201-B",
    "institution_id": "uni-main",
    "material_type": "lecture_notes",
    "visibility": "student"
  }
}

Note visibility as a first-class attribute rather than an inference from folder structure. Rubrics, model answers and marking schemes get visibility: "instructor", and the student-facing filter excludes them by value, not by path convention. Path conventions get violated the first time a busy tutor uploads a file to the wrong folder.

At query time you compose the filter server-side from the session record. Bedrock’s retrieval configuration supports boolean composition, so the tenant condition and the visibility condition are separate clauses that must both hold:

{
  "vectorSearchConfiguration": {
    "filter": {
      "andAll": [
        { "equals": { "key": "institution_id", "value": "uni-main" } },
        { "equals": { "key": "course_id",      "value": "BIO201"   } },
        { "equals": { "key": "visibility",     "value": "student"  } }
      ]
    }
  }
}

Two habits make this survive production. Build the filter through a single function that always prepends institution and course, so no future endpoint can forget it. Then write a test that queries a known instructor-only chunk’s exact wording from a student session and asserts zero results. Manual smoke testing never catches this, because human testers ask reasonable questions and the failure only appears when someone asks an unreasonable one.

Silo, pool or bridge

AWS frames multi-tenant isolation as three patterns, and the trade-off is the same one you always get between blast radius and cost:

• Silo gives each institution its own knowledge base and its own vector index. Strongest isolation, easiest compliance conversation, and you multiply your idle infrastructure cost by the number of tenants. There is also a per-account service quota on knowledge bases, so check Service Quotas before you promise a hundred districts their own.
• Pool puts everyone in one index and relies entirely on metadata filtering. Cheapest by a wide margin, and one filter bug is a cross-institution disclosure rather than an embarrassment.
• Bridge splits the difference: separate knowledge bases sharing underlying vector infrastructure, which keeps the isolation story cleaner without paying for a full stack per tenant.

For a single university running many courses, pool with rigorous filtering is usually right, because the institution is the tenant and the courses are not. For an edtech vendor selling to multiple districts, I would start at bridge and only pool if the economics force it, because the conversation with a district’s data protection officer goes very differently when your answer is “we filter” rather than “they are separate indexes.”

Failure family three: guardrails that do less than you assume

Bedrock Guardrails is a useful layer and the first thing I reach for on an education workload: content filters, denied topics, word filters, PII detection and contextual grounding checks, applied at inference time and usable across models. For a system that teenagers will deliberately try to provoke, enforcement inside the invocation path beats anything bolted on afterwards. Three limits are documented and routinely missed.

Contextual grounding does not cover chatbots. AWS documents the supported use cases as summarization, paraphrasing and question answering, and states plainly that conversational QA and chatbot use cases are not supported. A tutoring assistant is a multi-turn conversation. If your hallucination story is “we turned on grounding checks,” read that page before it reaches a procurement document. The check still earns its place on single-shot retrieve-and-answer paths, which is where I would apply it, but a multi-turn tutor needs citation display and human review too.

PII detection is probabilistic and text-only. The sensitive information filter is a machine learning model that uses surrounding context to decide whether something is an identifier. It works better on prose than on bare fragments. AWS also documents that it operates on text output and does not inspect tool use parameters, so if your assistant calls a function with a student identifier in the arguments, that path is not covered. Do not treat it as a substitute for not putting identifiers in the prompt in the first place.

Blocked content lands in your logs. AWS documents that content blocked by guardrail policies appears as plain text in Bedrock model invocation logs when those logs are enabled. That is defensible for debugging and awkward when the blocked content is a distressed message from a fourteen year old. Decide deliberately whether invocation logging is on, where it lands, who can read it, and how long it lives. In education this is a governance decision, not a logging default.

None of this argues against Guardrails. It argues for knowing its edges, because the gap between what a feature name suggests and what the documentation says is where compliance commitments go to die.

Failure family four: the cost shape of an academic term

Education traffic is not a smooth curve. It is near zero over summer, spikes brutally in the week before every assessment deadline, and dies again. That shape punishes the default architecture in a specific way.

The console nudges you toward Amazon OpenSearch Serverless as the vector store. It bills per OpenSearch Compute Unit hour with a minimum baseline footprint that exists whether or not anyone queries it, so an empty index over a quiet August still bills. AWS has adjusted those minimums over time, so pull current figures from the OpenSearch Serverless capacity documentation rather than from any blog post, this one included. The mechanism is the part that matters: your floor is set by capacity that exists, not by traffic that happens.

Bedrock Knowledge Bases supports other backends, and the question that decides between them is idle cost:

• Aurora PostgreSQL with pgvector. A first-class documented option. Aurora Serverless v2 bills per Aurora Capacity Unit hour and can scale down to zero when idle, which fits the academic calendar almost too neatly. You take on index lifecycle work in exchange: choosing between HNSW and IVFFlat, keeping statistics fresh, watching bloat. If your institution already runs Postgres, that is work your DBAs recognize.
• Amazon S3 Vectors. Bills on storage and requests rather than provisioned capacity, so genuine idle costs approach nothing. The trade is retrieval latency, which is higher than a warm search cluster. For a tutor where a response already takes a second or two of model time, that may not be the bottleneck you think it is.
• OpenSearch Serverless. The right answer when retrieval latency is genuinely your constraint, when you need hybrid search and k-NN tuning, or when you already run collections for other workloads and the baseline is already paid for.

On the model side, the cost driver is retrieved context, not the question. Every turn ships the retrieved chunks as input tokens, so chunk size and the number of results retrieved multiply directly into your bill on every single message. Halving retrieved results is usually a bigger lever than switching to a cheaper model, and it often improves answer quality too, because the model has less irrelevant material to get distracted by.

Tag by course and faculty from day one. The first budget question you will be asked is which department is driving the spend, and retrofitting cost allocation tags after a pilot goes institution-wide is miserable. Cost Explorer answers it if the tags exist; tools like Vantage or CloudZero make the per-course view easier to hand to a finance team that does not want an AWS login.

Failure family five: no evidence when someone asks

FERPA governs education records in the United States, and the mechanism that lets a vendor touch them at all is the school official exception: the institution must maintain direct control over the vendor’s use and maintenance of the records. Similar logic applies under other regimes, and many districts and state contracts add data residency requirements that FERPA itself does not impose.

What that means architecturally is less exotic than it sounds, and it is mostly about being able to answer questions after the fact:

• Log the retrieval, not just the answer. Store which chunk identifiers were returned for each turn. When an instructor claims the assistant leaked a rubric, the retrieval log is the only thing that settles it.
• Keep student identifiers out of prompts. Pass an opaque session-scoped pseudonym and resolve it in your own system. The model does not need a real name to be helpful.
• Encrypt with customer managed KMS keys and use encryption context that names the tenant, so a key policy can actually constrain who decrypts what.
• Pin the region and enforce it with service control policies if residency is contractually required. An SCP is checkable evidence; a runbook is not.
• Write down retention for conversation history and enforce it with lifecycle rules. “We keep it forever because storage is cheap” is a defensible engineering position and an indefensible one in an education procurement review.
• Pull the current compliance reports and agreements from AWS Artifact rather than quoting a blog about what AWS certifies.

None of this is glamorous and all of it is cheaper to build in during the pilot than to retrofit when the institution’s legal team gets involved.

Troubleshooting the things that actually page you

“The new material isn’t showing up”

Almost always ingestion, not retrieval. Walk it in order rather than guessing. Did a sync actually run after the change? For an S3 data source, an edit is invisible until an ingestion job processes it.

aws bedrock-agent list-ingestion-jobs 
  --knowledge-base-id KB_ID 
  --data-source-id DS_ID

Look for a completed job starting after the upload. If the most recent job predates the change, your scheduler or event handler is the bug and retrieval is innocent. If a job did run, read its statistics: a modified-documents-indexed count of zero when you expected a change means change detection did not fire, which happens when an object is re-uploaded with identical bytes. A non-zero failed count means your file is probably the one that failed, and the failure reasons say why.

“The filter returns nothing at all”

Query once with no filter to confirm the content is indexed. If it comes back unfiltered and vanishes when filtered, it is a metadata problem: the sidecar was not ingested, the attribute name does not match exactly, or the value differs in case. These comparisons are literal. BIO201 and bio201 are two different courses as far as the filter is concerned, which is an argument for normalizing identifiers at both ingestion and query time in one shared helper.

“The launch shows a blank iframe”

Two suspects dominate. Cookies: your tool is in a cross-site iframe, so session cookies need the right SameSite and Secure attributes or the browser drops them and every launch looks like a fresh unauthenticated user. Registration mismatch: issuer, client ID or deployment ID differs between what the platform sends and what you stored. Log the received issuer and deployment ID on validation failure. Guessing at OIDC failures is slow; the values are right there.

“Answers are right but the citations are wrong”

Usually chunking. Slides and worksheets fragment badly under naive fixed-size chunking, and a chunk that starts mid-table cites a heading that belongs to something else. Before you change models, look at what the retriever actually returned for the query that misbehaved. Nine times out of ten the model was faithful to a bad chunk.

“Grades aren’t reaching the gradebook”

Check that a line item exists and that your service token carries the right AGS scopes. Passback failures are asynchronous, so they surface as an empty gradebook column weeks later rather than as an error the student sees. Alert on passback failure rate explicitly; nobody will report it in time.

Common mistakes

• Enforcing access control in the system prompt. Instructions are not permissions. If a chunk can be retrieved, assume it can be surfaced.
• Trusting client-supplied course or role values. Derive scope from the validated launch token and store it server-side, always.
• Ingesting the whole course folder. Instructor material and student material in one prefix is how rubrics leak. Separate at ingestion, tag at ingestion.
• Putting real names and student numbers in prompts. Pseudonymize before the model call and resolve afterwards.
• Sizing the vector store from peak week. The bill is set by the idle floor, and in education idle is most of the year.
• Assuming a guardrail feature does what its name implies. Read the supported use cases. Contextual grounding on a chatbot is the classic example.
• Piloting with one enthusiastic lecturer. That lecturer’s material is unusually well organized. The second department will not be.

Best practices

• Build the retrieval filter in exactly one function, and make the tenant and visibility clauses impossible to omit.
• Keep a small regression set of adversarial queries, including students asking for solutions and asking about classmates, and run it on every deployment.
• Show citations in the interface. Instructors trust the system faster when they can see the source, and it converts silent hallucinations into reportable ones.
• Give instructors a preview mode that answers as a student would see it. They will find scoping errors far faster than any test suite.
• Route the tool endpoint through a CDN and WAF such as Cloudflare, with rate limiting per session. Assessment week is a load test you did not schedule.
• Dashboard retrieval count, token spend per course and passback failure rate together. CloudWatch metrics fronted by Amazon Managed Grafana or Grafana Cloud gives faculty a view without handing out console access.
• Have an escalation path for distressing content that reaches a human being, and make sure the people on it know they are on it. This is a policy question you must answer before launch, not after.

Frequently asked questions

Do we need to fine-tune a model for our curriculum?

Almost certainly not. Curriculum content changes every term, and retrieval updates in minutes where fine-tuning is a rebuild. Fine-tuning earns its place for style and format consistency, not for facts. Start with retrieval, and revisit only if you have a specific behavior that prompting genuinely cannot produce.

How do we stop it from just doing the homework?

Partly retrieval scope, partly prompting, partly accepting you cannot fully win. Exclude solutions and rubrics from student-visible retrieval, prompt for Socratic responses on assessed material, flag the assessment context on the way in so the assistant knows which mode it is in. A determined student can paste the question into a consumer chatbot regardless, which is a course design question rather than an infrastructure one.

Is Bedrock the right choice versus calling a model API directly?

If the institution is already on AWS, Bedrock keeps the traffic inside your account boundary, gives you IAM and CloudTrail on the same footing as everything else, and can be reached over VPC endpoints. A direct provider API is often simpler and sometimes gets new models sooner. The deciding factor is usually procurement: one existing AWS agreement is a much shorter conversation than a new vendor review.

Does student data get used to train the model?

AWS states that Bedrock does not use customer inputs or outputs to train foundation models. Verify it against current AWS documentation and your agreement rather than any third-party summary, because this is exactly the clause a district will ask you to evidence in writing.

What does it cost to run a personalized learning assistant on AWS?

Model your bill as three parts: a vector store floor that bills whether or not anyone asks anything, a one-off embedding cost when you ingest the corpus, and per-turn inference dominated by retrieved context rather than by the question. For a small corpus with a term-shaped traffic curve, the floor usually dominates, which is why the vector store choice matters more than the model choice.

How do we know whether it actually helps anyone learn?

Decide the measure before launch and instrument for it. Usage is not learning. Pair the assistant’s logs with assessment outcomes on comparable cohorts and accept that a clean answer needs a study design, not a dashboard. Engineering can supply the data; it cannot supply the conclusion.

The one thing worth remembering

A personalized learning assistant on AWS will never tell you when it is wrong about who someone is. The model has no concept of enrollment, and retrieval has no concept of a due date. Both live entirely in the metadata you attach at ingestion and the filter you build server-side from a validated launch token. Get that right and what remains is ordinary engineering: chunking, cost curves, dashboards. Get it wrong and you ship something that works flawlessly right up until the week it embarrasses the institution.

Build the boundary first. Make it a control, not an instruction. Then go be clever with the tutoring.

Need a hand building or auditing one of these?

I work on the infrastructure side of AI assistants for education, which mostly means the parts that decide whether the thing is safe to put in front of students:

• Designing and reviewing the LTI 1.3 launch path, roster access and grade passback, including the cross-site cookie and registration issues that cause blank-iframe launches.
• Building the retrieval isolation layer: sidecar metadata schemas, server-side filter composition, and an adversarial regression suite that proves instructor material stays invisible to students.
• Choosing and migrating the vector store when the OpenSearch Serverless floor is eating a pilot budget, including Aurora pgvector and S3 Vectors options.
• Guardrail configuration that matches the documented behavior rather than the feature name, plus a logging and retention design you can defend in a procurement review.
• Cost allocation tagging by course and faculty, with dashboards that answer “which department is spending this” without handing out AWS console access.
• Infrastructure as code for the whole stack so a second institution is a variable file rather than a fortnight.

If you have something concrete, send it over: a retrieval filter you are not sure is airtight, an ingestion job that keeps skipping documents, a failing LTI launch log, or a Cost Explorer breakdown that stopped making sense. Easier to look at the real thing than to talk in the abstract.

The post Building a Personalized Learning Assistant on AWS Without Leaking Student Data appeared first on John Nessime.

That is what combining structured and unstructured data on AWS actually looks like when it goes wrong. Not a service outage. Not a permissions error. A quiet correctness bug that survives every test you wrote, because every individual component did exactly what it was told.

The storage side of this problem is solved. S3 holds the PDFs, Iceberg tables hold the rows, Bedrock Data Automation pulls fields out of the documents, S3 Vectors or OpenSearch holds the embeddings. None of that is hard any more. The hard part is the join, and the join is where nearly all the failures live.

What this post covers

Five failure families I keep running into on mixed-data architectures, why each one is invisible until it isn’t, and a working shape that avoids most of them. Then troubleshooting, mistakes, and the questions people actually ask before they build this.

Not a service tour. AWS documentation already covers what Glue, Athena, Lake Formation and Bedrock do. What it covers badly is what happens at the seams between them.

Failure family one: there is no join key, only a filename

Structured data arrives with identity built in. A row has a primary key, a foreign key, a surrogate key from the CDC pipeline. Unstructured data arrives with an object key, which is a path, which is a decision someone made about folder layout six months ago.

Object keys are not identity. They are location. The difference matters the moment anyone overwrites an object, moves a prefix, or uploads the same document twice under two names. Your join silently changes meaning and nothing raises an error, because to Athena a string is a string.

What to use instead

Mint a document ID at ingestion, before extraction, and carry it everywhere. Store it three places: as S3 object metadata on the source object, as a column on every extraction output row, and as vector metadata on every chunk you embed. That ID is the only thing that ties the three planes together.

Version the object too. S3 versioning gives you a version ID per object, and a document ID plus a version ID is a genuinely stable identity for “this exact bytes-on-disk artefact.” If you skip versioning on a bucket that feeds extraction, an overwrite destroys the evidence for every downstream row derived from it. You will be asked to reproduce that extraction eventually. Auditors, disputes, model retraining, all three.

S3 Metadata is worth knowing about here. It maintains queryable Apache Iceberg tables describing the objects in a general purpose bucket, so you can join object-level facts against your business tables in Athena without writing an inventory crawler. Two limits shape how you use it: metadata tables live in an AWS managed table bucket rather than one you own, and the journal table only records change events that happen after you create the configuration. Enable it on day one or you inherit a permanent blind spot for everything already in the bucket. The inventory table covers the existing objects, which is why you want both.

Failure family two: extraction confidence gets flattened into a fact

Bedrock Data Automation, Textract, and any model-based extraction return confidence alongside values. Every pipeline I have reviewed does the same thing with it: reads it once, applies a threshold, writes the value into a typed column, and drops the score.

Now a decimal that the model was 62% sure about sits in the same column as one it was 99% sure about, and every dashboard, every SQL aggregate, every model feature treats them identically. The uncertainty didn’t go away. You just stopped being able to see it.

Keep the score as a column. It costs one float per field and it buys three things: the ability to filter low-confidence rows out of a report without rerunning extraction, a review queue that ranks by expected damage rather than arrival order, and a way to measure whether a blueprint change actually improved anything. Add two more columns while you are there, one for the extractor or blueprint identifier and one for the timestamp. When a field goes wrong at scale the first question is always “which extractor produced these,” and without that column the answer is a week of archaeology.

-- Upsert extraction output into an Iceberg table, keyed on document
-- identity rather than object key. Athena performs merge-on-read for
-- MERGE INTO, so the write is cheap and the read pays the cost until
-- compaction catches up.

MERGE INTO contracts_extracted AS t
USING extraction_staging AS s
  ON t.document_id = s.document_id
 AND t.source_version_id = s.source_version_id
WHEN MATCHED THEN UPDATE SET
  contract_value       = s.contract_value,
  contract_value_conf  = s.contract_value_conf,
  extractor_id         = s.extractor_id,
  extracted_at         = s.extracted_at
WHEN NOT MATCHED THEN INSERT (
  document_id, source_version_id, contract_value,
  contract_value_conf, extractor_id, extracted_at
) VALUES (
  s.document_id, s.source_version_id, s.contract_value,
  s.contract_value_conf, s.extractor_id, s.extracted_at
);

The merge-on-read detail matters more than it looks. Athena writes delete files rather than rewriting data files, so a chatty extraction pipeline that merges every few minutes will quietly degrade query performance until compaction runs. On S3 Tables that maintenance is handled for you. On self-managed Iceberg in a general purpose bucket it is your job, and forgetting it is one of the more common reasons a lakehouse gets slower every week for no visible reason.

Failure family three: two catalogs, two permission models

This is the one that bites teams who did everything else right.

Your tables are governed by Lake Formation, with grants at table and column level. Your documents are governed by S3 bucket policies and IAM. Your vectors are governed by IAM on the vector bucket or index ARN. Three planes, three models, no shared enforcement point.

The result is predictable and unpleasant. An analyst who is correctly denied the salary column in Athena runs the same question against the RAG endpoint and gets the number read straight out of the source PDF, because nobody applied column-level thinking to the chunks. The retrieval layer had no idea it was quoting a restricted field.

There is no single AWS switch that fixes this. What works is applying the same classification at ingestion and enforcing it independently in each plane:

• Classify the document once, at ingestion, and write the classification into the sidecar metadata that the knowledge base ingests.
• Use metadata filters on retrieval so a caller only ever searches the chunks their tier permits.
• Keep restricted documents in a separate index or knowledge base where the classification is coarse and the blast radius matters.
• Redact at extraction time for anything you genuinely cannot let leak, rather than relying on a filter you might misconfigure later.

Bedrock Knowledge Bases reads per-document metadata from a sidecar JSON file that sits beside the source object and carries the same filename with .metadata.json appended. That file is where classification, tenant, document ID and effective dates belong.

{
  "metadataAttributes": {
    "document_id": "doc_01HX9K4RZP",
    "classification": "restricted",
    "tenant_id": "acme",
    "effective_year": 2031
  }
}

Filterable metadata is what makes tenant isolation possible on a shared index. It is also the thing most likely to be wrong on the first attempt, which brings us to the next failure family.

Failure family four: index decisions you cannot take back

S3 Vectors makes vector storage cheap enough that people stop thinking about it, and then hit a wall that has nothing to do with cost. When you create a vector index you fix three things permanently: the dimension, the distance metric, and the list of non-filterable metadata keys. Changing any of them means creating a new index and re-embedding everything.

By default every metadata key you attach is filterable. Filterable metadata is subject to a tighter size budget than the total metadata per vector, so if you push a large chunk of text through as a filterable key, writes start failing with validation errors rather than degrading gracefully. Query a key you declared non-filterable and you get a 400 back, not an empty result set.

If you are pointing Bedrock Knowledge Bases at an S3 Vectors index you created yourself, declare the Bedrock-managed text and metadata keys as non-filterable at creation time. They hold chunk text and are large. Leaving them filterable is the single most common reason a knowledge base sync fails partway through with validation exceptions and no obvious cause.

# Non-filterable keys are set once, at index creation, and cannot be
# changed afterwards. Keep the list short: the cap is small, and every
# key you spend here is one you can never filter on.

aws s3vectors create-index 
  --vector-bucket-name my-vector-bucket 
  --index-name documents 
  --data-type float32 
  --dimension 1024 
  --distance-metric cosine 
  --metadata-configuration '{"nonFilterableMetadataKeys":["AMAZON_BEDROCK_TEXT","AMAZON_BEDROCK_METADATA"]}'

The dimension has to match your embedding model’s output exactly, and the same model has to be used for ingestion and for query. Swapping embedding models mid-life is not a config change, it is a migration. Plan the index around the model you intend to keep, not the one that is convenient today.

Worth being even-handed here: S3 Vectors is the cheap, durable option and it is the right default for large corpora with modest query rates. If you need heavy filtering, hybrid keyword-plus-vector search, or very high query throughput, OpenSearch Serverless or a dedicated store like pgvector on Aurora will serve you better, and AWS supports tiering between them for exactly that reason. Pick on query pattern, not on storage price.

Failure family five: reprocessing rewrites history

You improve the extraction blueprint. You rerun it across the corpus. Numbers in last quarter’s board pack change.

This is the failure that ends trust in the platform, and it is entirely avoidable. Extraction output is a derived opinion about a document at a point in time. It is not a fact about the world. Model it that way.

Keep two timestamps on every extraction row: when the source document became effective, and when this extraction ran. Reports pin to the first, audits use the second. Now a reprocess adds rows rather than changing answers, and “why did this number move” has an answer you can query instead of guess at.

Iceberg gives you snapshot isolation and time travel for free on the table side, which is genuinely useful for reconciliation. It does not, on its own, tell you that the value changed because a different extractor ran. Only your columns do that.

A shape that works for combining structured and unstructured data on AWS

Nothing exotic. The ordering is what matters, because several of these decisions are hard to reverse.

1. Land raw, enable versioning, enable S3 Metadata. Do this before you ingest anything you care about. Both are cheap to turn on and impossible to backfill properly.
2. Mint the document ID at landing. Write it to object metadata and to the sidecar metadata file in the same step. One writer, one source of truth.
3. Classify before you extract. Classification drives which extraction path runs, which index the chunks land in, and which Lake Formation grants apply downstream.
4. Extract to a staging table, not straight to the curated table. Staging gives you somewhere to run validation before anything touches a table an analyst can query.
5. Merge into Iceberg on document identity, carrying confidence and provenance columns. Never merge on object key.
6. Embed separately, keyed by the same document ID. The vector plane and the table plane are different systems with different refresh rates. Treat them as such.
7. Join at query time in Athena. Retrieval returns document IDs, SQL turns those into rows. Resist the urge to materialise one giant denormalised table containing both.

Step seven is the one people argue with. The argument for denormalising is query simplicity. The argument against is that you now have a third copy that drifts from both parents, and drift in a table nobody owns is how you get back to the ticket at the top of this post.

Troubleshooting

Athena says the principal has no privilege on the resource

This is a Lake Formation grant, not an IAM one. IAM policy alone is not enough for catalog-governed tables. Grant on the catalog and database first, then the table. For S3 Metadata tables and S3 Tables specifically, the grant needs to reach the table bucket catalog and namespace, not just the table.

Queries against metadata tables fail with a syntax error

Metadata table namespaces need to be quoted with double quotes or backticks in Athena and Redshift. It looks like a parser bug and it is not.

Knowledge base sync fails partway with validation errors

Almost always filterable metadata size. Check whether the Bedrock-managed text keys were declared non-filterable when the index was created. If they were not, you need a new index, because that setting is immutable.

Retrieval returns documents the caller should not see

Check that the filter is actually being applied at query time and that the metadata key is filterable. A filter on a non-filterable key returns an error rather than silently failing, so if you are getting results, the filter is probably absent from the request entirely.

Athena gets slower week over week on a table that isn’t growing much

Delete files accumulating from merge-on-read writes, or small file proliferation from frequent micro-batches. Check snapshot and manifest counts before you start tuning queries.

Common mistakes

• Joining on the S3 object key because it was there and it worked in the prototype.
• Enabling S3 Metadata after the bucket is full, then wondering why the journal table is missing most of the history.
• Applying a confidence threshold and discarding the score.
• Creating the vector index before deciding on the embedding model.
• Assuming Lake Formation column grants protect the same field when it is quoted out of a PDF by a retrieval endpoint.
• Reprocessing in place, with no record of which extractor produced which value.
• Turning off S3 versioning on the raw bucket to save on storage, then being asked to reproduce an extraction.

Best practices

• One ingestion writer that mints identity. Everything downstream reads that ID and never invents its own.
• Confidence and provenance columns are not optional. They are the difference between debugging and guessing.
• Treat extraction output as append-mostly and bitemporal. Effective time and extraction time, always both.
• Decide the embedding model before the index, and budget non-filterable keys deliberately. The cap is small and permanent.
• Enforce classification in all three planes independently. There is no single choke point.
• Watch table maintenance the way you watch disk. Compaction and snapshot expiry on self-managed Iceberg; a monitoring stack such as Amazon Managed Grafana or Grafana Cloud pointed at query runtime and file counts will show the trend long before an analyst complains.
• Keep a small, cheap reprocessing rig. A modest VPS from a provider like Contabo or InterServer is enough to dry-run a blueprint change against a sample corpus before you spend Bedrock invocations on the full set.

FAQ

Do I need a lakehouse to combine structured and unstructured data on AWS?

No. You need a stable join key and a place to put extraction output. Athena over Iceberg tables in S3 is enough for most workloads. The lakehouse framing matters when you have several engines and warehouses to unify, not when you have documents and one analytics database.

Should extraction output live in the same table as the structured source data?

Separate tables, joined at query time. They have different refresh cadences, different failure modes and different trust levels. Merging them into one table means a bad extraction run corrupts data that had nothing wrong with it.

Is S3 Vectors a replacement for OpenSearch?

For cost-sensitive storage of large vector sets with moderate query volume, yes. For high query throughput, rich filtering or hybrid search, OpenSearch or pgvector remains the stronger fit. Tiering between them is a supported pattern and often the honest answer.

How do I keep the vector index in sync with the documents?

Drive both from the same ingestion event. S3 event notifications into a queue, then fan out to extraction and embedding. Reconcile periodically by comparing document IDs present in the table against those present in the index, because “mostly in sync” degrades quietly.

What breaks first when this architecture is done badly?

Trust. Long before anything errors, someone finds a number that disagrees with the source document and stops using the dashboard. The technical failure is downstream of the social one.

Can I use Textract instead of Bedrock Data Automation?

Yes, and for pure OCR and form or table extraction with predictable layouts it is still a good fit. Bedrock Data Automation is aimed at mixed modalities and schema-shaped custom output through blueprints. The architectural advice here does not change either way, because the failure families are about identity and governance, not about which extractor you picked.

Does Lake Formation govern my documents as well as my tables?

It governs the catalog and the tables registered in it. Raw objects in a general purpose bucket are governed by S3 and IAM, and vectors by IAM on the vector resources. Assume three enforcement points until you have tested otherwise.

The one thing worth remembering

Combining structured and unstructured data on AWS is not a storage problem and it has not been one for a while. Every service in the path works. What fails is identity: the assumption that a file path is a key, that a confidence score is a fact, and that a rerun is idempotent.

Mint an ID at the door, carry it into every plane, keep the score and the provenance, and model extraction output as an opinion with a timestamp. Do that and the rest of the architecture is ordinary engineering. Skip it and you will spend a year explaining numbers that were never wrong, exactly.

Need help wiring documents and tables together?

This is a large part of what I do. If you are standing one of these up, or unpicking one that has drifted, I can help with:

• Designing document identity and provenance so extraction output survives reprocessing, overwrites and audits.
• Building the Glue, Athena and Iceberg side of a mixed-data pipeline, including merge strategy and table maintenance that does not rot.
• Setting up Bedrock Data Automation or Textract extraction with confidence handling and a human review path that ranks by risk.
• Getting metadata filtering and tenant isolation right on S3 Vectors, OpenSearch or Bedrock Knowledge Bases before the index is locked in.
• Reconciling Lake Formation grants against what a retrieval endpoint can actually surface, and closing the gaps.
• Reviewing an existing lakehouse for the failure families above and giving you a prioritised list rather than a rewrite.

Send me a table schema, a sample extraction output, or the query that is returning the wrong number, and I will tell you what I see.

The post The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS appeared first on John Nessime.

The answer on screen was good. Fluent, specific, correctly hedged, three sources listed underneath it. Then a partner in the room asked a simple question: which matter is source three from? Nobody could answer it in the room, and when we went and looked, it was from a matter that half the people watching the demo were formally walled off from.

Nothing had crashed. No alarm fired. The retrieval layer had done exactly what a similarity search does, which is return the nearest vectors, and the nearest vectors did not care about the ethical wall. That is the thing about building a legal document intelligence platform on AWS: the failures that matter almost never look like failures. They look like a confident answer with a citation attached.

This post is about the parts of that build that are hard, organised around the five boundaries that actually have to hold. The extraction pipeline is the easy half. I will cover it, but quickly, because the AWS documentation is good and the failure modes are visible. The other four boundaries fail quietly, and that is where the engineering goes.

The shape of the thing

Before the boundaries, the skeleton. A legal document intelligence platform on AWS almost always ends up looking like this, whether you plan it or arrive at it:

• Documents land in Amazon S3, one prefix per matter, versioning on.
• An S3 event triggers AWS Lambda, which starts an asynchronous Amazon Textract job.
• Textract output lands back in S3 as JSON. Something normalises it into text plus page and bounding-box coordinates.
• Amazon Comprehend or a foundation model classifies the document and pulls entities: parties, dates, governing law, clause types.
• Chunks get embedded and written to a vector store, usually Amazon OpenSearch Serverless behind Amazon Bedrock Knowledge Bases.
• An application layer, typically API Gateway plus Lambda, takes a question, retrieves, and calls a model on Amazon Bedrock.

AWS Step Functions is worth reaching for once you have more than three stages, because Textract jobs are long-running and Lambda timeouts are not a retry strategy. That is the whole architecture. You can stand it up in a fortnight. Then you spend six months on everything below.

Boundary one: isolation, and why metadata filtering is not optional

This is the one from the demo, and it is the one that ends engagements.

A vector index has no native concept of a matter, a client, or an ethical wall. If you pool every document into one index and rely on the prompt to keep things separate, you have built a system where a well-phrased question can pull privileged material across a wall. Prompt instructions are not an access control. They are a suggestion that usually works.

There are two shapes that do work, and the choice between them is a real trade-off rather than a best practice.

Pooled index with server-side filtering

Every chunk carries a matter_id and a client_id as metadata. Every retrieval call attaches a filter. With an S3 data source in Bedrock Knowledge Bases, the metadata comes from a sidecar JSON file that sits next to the document and carries a metadataAttributes object.

{
  "metadataAttributes": {
    "matter_id": "M-4417",
    "client_id": "C-108",
    "doc_type": "engagement_letter",
    "privileged": true
  }
}

The retrieval call then narrows the search before the model ever sees a chunk:

"retrievalConfiguration": {
  "vectorSearchConfiguration": {
    "filter": {
      "equals": { "key": "matter_id", "value": "M-4417" }
    }
  }
}

Two rules make this safe, and both are the kind of thing that gets skipped under deadline. First, the filter value is derived on the server from the authenticated caller’s claims, never accepted from the client. If the browser can send you a matter_id, the browser can send you a different one. Second, the filter is applied by code that no feature request can bypass. A helper that builds every retrieval request, and a code review rule that no other path may call the retrieve API directly.

The trap here is ordering. You cannot filter on an attribute the index does not have. Add matter_id after ingestion and every chunk already in the index is invisible to that filter, which means it either returns for everyone or for no one, depending on how your filter is written. Design the metadata schema before the first ingestion run, not after the first demo.

Separate collection per client

The heavier option. A separate OpenSearch Serverless collection per client, which buys you a separate AWS KMS key per client, separate index settings, and a failure mode where a bug in the filtering logic cannot reach across clients at all.

It costs you sprawl. Every collection is a resource to provision, monitor, patch policy on, and eventually delete. Ingestion jobs run per data source per knowledge base, so freshness guarantees fragment. For a firm with twelve institutional clients this is fine. For a platform onboarding a hundred small clients it becomes the main operational burden of the product.

My default is pooled with strict server-side filtering, and per-client collections only where the client’s own contract demands a dedicated encryption key. If you cannot say out loud which of those two you are running, you are running neither properly.

Boundary two: extraction, and the confidence score everyone ignores

Textract returns a confidence score between 0 and 100 with each prediction. AWS is explicit in its own best-practice guidance that applications sensitive to detection errors should enforce a minimum threshold and route anything below it for human scrutiny, and that the right threshold depends entirely on how the output gets used.

Legal documents sit at the harsh end of that. A misread date on an archival scan is a curiosity. A misread date on a limitation period is a problem with a name on it.

Practical notes from this layer:

• Use the asynchronous operations for anything multipage. StartDocumentAnalysis and GetDocumentAnalysis handle long PDFs and TIFFs; the synchronous AnalyzeDocument path is for single pages and will fight you on a 400-page bundle.
• Textract’s Queries feature earns its place on structured instruments. Instead of extracting everything and grepping, you ask the document a direct question and get a scoped answer with its own confidence.
• Keep the page number and bounding box for every chunk. When a lawyer asks where a sentence came from, “page 14, second column” is an answer. “It’s in the corpus somewhere” is not, and the platform loses credibility the first time you say it.
• Amazon Augmented AI wires low-confidence predictions into a human review queue with a private workforce, so the reviewers are your people rather than an anonymous pool. For privileged material that distinction is the entire point.

What I would skip early: building a custom entity recognition model. Comprehend supports custom entity recognition and it is genuinely useful, but you need labelled data you do not have yet, and a foundation model with a decent prompt gets you far enough to find out which entities the users actually care about. Train the custom model once the requirement has stopped moving.

Boundary three: retention, where deletion is a distributed problem

Here is the failure that will not show up in any test suite you write.

A client asks for a document to be removed. Someone deletes the S3 object. The ticket closes. The document is still fully searchable, because its chunks are sitting in the vector index and will stay there until the data source is re-synced. Meanwhile the Textract JSON output is in a second bucket, the extracted text may be in a database, and if model invocation logging is on, whole passages of the document are sitting in CloudWatch Logs or another S3 bucket entirely.

One delete, at least four places holding a copy. Write the deletion path as a real workflow with an assertion at the end, not as a single API call:

1. Delete the source object and, if versioning is on, its versions.
2. Delete the derived artefacts: Textract JSON, normalised text, any thumbnails or page images.
3. Trigger an ingestion job so the knowledge base drops the orphaned chunks.
4. Confirm removal by querying the index for the document identifier and expecting nothing back.
5. Deal with the logs, which means either a retention policy short enough to make the problem expire or a deliberate decision, written down, that logs are out of scope.

Step four is the one people leave out, and it is the only step that actually proves anything.

The opposite problem: things that must not be deleted

Legal work has the reverse requirement too. When a matter goes into litigation hold, the documents need to survive an administrator with delete permissions and a bad afternoon.

S3 Object Lock is the mechanism, and it has two independent controls that people routinely conflate. A retention period protects an object version until a fixed date, in either governance mode, which privileged users can override, or compliance mode, which nobody can override and where the period cannot be shortened. A legal hold has no date at all. It stays until someone with s3:PutObjectLegalHold explicitly removes it. The two can be active at once, and while either is active the object version cannot be deleted or overwritten.

# Place an indefinite hold on a specific object version
aws s3api put-object-legal-hold 
  --bucket matters-archive 
  --key M-4417/exhibit-c.pdf 
  --version-id 3sL7f2Qz9pXvB1kR 
  --legal-hold Status=ON

Two things bite here. Object Lock requires versioning and turns it on automatically, and once enabled on a bucket you cannot turn it off or suspend versioning again. And compliance mode is genuinely permanent: if you set a seven-year retention when you meant seven days, that is the answer, for everyone, including the account root. Test in governance mode first. The bypass path exists and is deliberately awkward, requiring both the s3:BypassGovernanceRetention permission and an explicit x-amz-bypass-governance-retention:true header on the request, which is exactly the level of friction you want on that operation.

Design the hold model before you design the deletion model, because holds win. A deletion request that collides with an active hold is a legal question, not an engineering one, and the platform’s job is to surface the collision clearly rather than resolve it silently.

Boundary four: disclosure, or what leaves the account

This is the boundary the client’s general counsel will ask about, usually in writing, usually before signature. Three separate controls, and they are not interchangeable.

The AI services opt-out policy

AWS Organizations has a governance policy type that opts your accounts out of having content processed by certain AI services stored and used for service improvement. The list includes Amazon Textract and Amazon Comprehend, which are precisely the two doing the reading in this architecture. The policy type has to be enabled at the organisation root before you can attach anything:

aws organizations enable-policy-type 
  --root-id r-example 
  --policy-type AISERVICES_OPT_OUT_POLICY

Read the AWS documentation on this one carefully rather than taking my summary as gospel, because there is a caveat in it that matters: the services may still need to store your data operationally even when you have opted out of it being used for improvement. Opting out is not the same as the data never existing outside your account. Say that plainly in the client conversation. It is a much better position than being asked about it later.

Bedrock’s data position, and the log that undoes it

Amazon Bedrock’s published position is that inputs and outputs are not shared with third-party model providers and are not used to train the base models, and that fine-tuning operates on a private copy. That is a strong starting point for privileged content and it is the main reason Bedrock rather than a direct provider API shows up in these builds.

Then there is model invocation logging. It captures full prompt and response payloads to CloudWatch Logs or S3, and you want it on, because without it you cannot reconstruct what the system told someone. But understand what you have just built: a log that contains privileged document text, at a per-region setting, in a destination that probably has looser access controls than the document store it came from. Encrypt it with the same KMS key discipline, restrict it harder than you think you need to, and set a retention period on purpose.

Worth separating in your head: CloudTrail records that an API call happened and who made it. Model invocation logging records what was in it. You need both, for different questions, and only one of them contains client confidences.

Network path

Interface VPC endpoints via AWS PrivateLink keep traffic to Bedrock, Textract and the rest on the AWS network rather than out through an internet gateway or NAT. Add a gateway endpoint for S3 while you are there.

Endpoint policies are the part that gets forgotten. An endpoint without a policy is a private path to the whole service, including into accounts that are not yours. Scope it to the operations and resources you actually use, and pair it with an S3 bucket policy that rejects requests not arriving through your endpoint. The first protects what leaves; the second protects what can be reached.

The parts that are not on the architecture diagram

Two of these have caught people out badly, and neither is an AWS control.

The first is the development environment. You do not want real client documents on a laptop or in a scratch account, so build the parsing and chunking code against synthetic documents on a small separate box. A cheap VPS from a provider like Contabo or InterServer is fine for this, and the separation is worth more than the convenience of iterating in the production account.

The second is the reviewers. If your human review loop involves contractors working remotely, their network path and their disks are part of your boundary whether or not you drew them. A managed VPN such as NordVPN or Surfshark handles the network side, and when a matter closes and local copies have to go, a dedicated erasure tool from something like O&O Software does what dragging a folder to the bin does not. Unglamorous, and it is the layer that gets audited.

Boundary five: evidence, because someone will ask

At some point the question stops being “does it work” and becomes “who saw what, and when”. If you cannot answer that from stored records, the platform is not defensible regardless of how good the answers are.

• Turn on CloudTrail data events for the document buckets. They are off by default, they are billed separately from management events, and they are the only way to see individual object-level reads.
• Log the retrieval, not just the generation. Store which chunks came back and which filter was applied. When somebody asks whether the wall held on a specific query, this record is the answer and there is no reconstructing it later.
• Carry the end user’s identity through to the audit record. A Lambda execution role in the logs tells you the platform did something. It does not tell you who asked.
• Alarm on the filter, not just on errors. A retrieval that ran without a matter filter should page someone, even though it returned HTTP 200 and a perfectly good answer.

That last one is the closest thing to a single takeaway in this post. The dangerous events in this architecture are successful ones.

Troubleshooting the things that will go wrong

Retrieval returns nothing after you add a filter

Nearly always a metadata problem rather than a query problem. Either the sidecar metadata file was missing at ingestion, so the chunks carry no attribute to match, or the attribute was added after the chunks were indexed. Check whether the attribute exists on a known chunk before you touch the query. If it does not, you need a re-sync, not a different filter expression.

Textract returns text but the layout is scrambled

Common on two-column pleadings and on documents with headers and footers on every page. Raw text detection reads in a reading order that is not always yours. Use the layout and table features rather than plain detection, and hold onto the geometry so you can reassemble columns yourself if the default order is wrong for that document class.

The model cites a document that does not exist

Usually the citation is being generated rather than passed through. If document names are being written by the model instead of read from the retrieval response, it will invent plausible ones. Build citations from the retrieval result metadata in your application code and never from the generated text. Bedrock Guardrails helps with contextual grounding, but the structural fix is not asking the model to produce the reference in the first place.

Ingestion is slow and nobody knows where

Instrument per stage before you optimise anything. In most of these pipelines the time is in Textract for large scanned bundles, and the fix is parallelism at the document level rather than tuning the extraction itself. Step Functions with a distributed map over documents gets you there without a queue you have to babysit.

Costs climb faster than volume

Look at re-processing first. A pipeline that re-extracts a document every time anything downstream changes will quietly multiply your Textract bill against a static corpus. Key the extraction cache on the object’s content hash, not its path, and make re-ingestion an explicit action. The billing mechanisms differ per service, so read the current pricing pages rather than trusting a number from a blog post.

Common mistakes

• Treating the system prompt as an access control. It is a formatting instruction that happens to look like a rule.
• Taking the tenant identifier from the request body. If the client can name the matter, the client can name someone else’s.
• Designing the metadata schema after the first ingestion run. Retrofitting a filterable attribute means re-indexing the whole corpus.
• Assuming a deleted S3 object is gone from the platform. The chunks, the extraction output and the invocation logs all outlive it.
• Setting compliance-mode retention without testing in governance mode. There is no support ticket that fixes a seven-year mistake.
• Enabling model invocation logging without treating the log as privileged. You have just made a second copy of the documents in a less protected place.
• Building VPC endpoints and leaving the default policy on them. A private path with no policy is still a path.

Best practices worth the effort

• One retrieval helper, server-side filter injection, and a review rule that no other code path calls the retrieve API directly.
• Metadata schema agreed and frozen before the first production ingestion. Include the fields you might filter on later, even if unused today.
• Page-level provenance on every chunk, surfaced in the interface. It builds trust faster than any accuracy improvement.
• A confidence threshold chosen per document class, with a human review path for everything under it.
• Deletion implemented as a verified workflow across every store, ending in an assertion that the content is actually unretrievable.
• Object Lock legal holds for matters under litigation hold, tested in governance mode before anything runs in compliance mode.
• Customer-managed KMS keys across S3, the vector store and the logs, so key access is a control you can actually revoke.
• An alarm on unfiltered retrievals, because the failure mode is a success response.

Frequently asked questions

Is a legal document intelligence platform on AWS safe for privileged material?

The services support it. Bedrock’s stated position is that inputs and outputs are not shared with model providers or used to train base models, PrivateLink keeps traffic off the public internet, and KMS covers encryption at rest. What determines safety is your own configuration: tenant isolation, log handling and deletion behaviour. The platform is exactly as confidential as its weakest copy of the text.

Should I use Bedrock Knowledge Bases or build retrieval myself?

Start with Knowledge Bases. It handles chunking, embedding, sync and metadata filtering, and those are weeks of work with no differentiation in them. Build your own when you need retrieval behaviour it does not expose, such as unusual re-ranking or per-tenant chunking strategies. Going custom on day one usually means reimplementing the managed service badly.

How do I stop one client’s documents surfacing in another client’s answers?

Tag every chunk with a tenant identifier at ingestion, and inject the matching filter server-side on every retrieval from the authenticated caller’s identity. Never accept the identifier from the client. If the contract requires per-client encryption keys, use a separate collection per client instead, accepting the extra operational load.

Does deleting a document from S3 remove it from the search index?

No. The chunks stay in the vector store until an ingestion job runs and reconciles the data source. Until then the document is deleted and still fully searchable, which is the worst combination available. Always finish a deletion by querying the index and confirming nothing comes back.

What is the difference between an S3 legal hold and a retention period?

A retention period runs until a fixed date and comes in governance mode, which privileged users can override, or compliance mode, which nobody can. A legal hold has no end date and stays until someone explicitly removes it. They are independent, they can both be active on the same object version, and while either is active the object cannot be deleted or overwritten.

Do I need human review, or is the model accurate enough?

Accuracy is the wrong frame. The question is what happens to a wrong answer downstream. Where output feeds a decision with consequences, you want a confidence threshold and a review queue, and Amazon Augmented AI with a private workforce gives you that without building the review tooling yourself. Where output is a search aid a person will verify anyway, review is friction you do not need.

Can I run this in a single AWS account?

Technically yes, and for a pilot it is reasonable. It gets uncomfortable once you have production documents and a development environment in the same place, because an IAM mistake has nowhere to stop. Separate accounts under Organizations also gives you the policy layer, including the AI services opt-out policy, which only exists at the organisation level.

The one thing to take away

A legal document intelligence platform on AWS is not hard to build. Textract reads, Bedrock reasons, OpenSearch retrieves, and the tutorial version works on the first afternoon.

What is hard is that its failures are silent and well-formed. The cross-matter citation, the deleted document that still answers questions, the privileged passage sitting in a log bucket, the retrieval that ran without a filter and returned a beautiful paragraph. None of these throw an error. All of them are the kind of thing that ends a client relationship.

So build the boundaries first and the features second, and make sure every one of them is something you can prove from a stored record rather than something you believe about the code. If you can only take one habit from this: alarm on the successful requests that should not have been possible.

Need a second pair of eyes on your document pipeline?

I work on AWS document processing and retrieval systems where the confidentiality requirements are real. Typical things I get called in for:

• Reviewing tenant isolation in an existing RAG setup and finding the paths where the filter can be bypassed.
• Designing the metadata and chunking schema before ingestion, so filtering works without a re-index later.
• Building Textract pipelines with confidence thresholds, human review routing and page-level provenance.
• Implementing verified deletion across S3, derived artefacts, the vector index and logs, with a proof step at the end.
• Setting up retention and legal hold with S3 Object Lock, including the governance-mode rehearsal before compliance mode.
• Locking down the network and logging path: VPC endpoints with real policies, KMS key separation, and audit records that name the actual user.

If you have an architecture diagram, a Step Functions definition or a retrieval request you are unsure about, send it over and I will tell you what I would change and why.

The post Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold appeared first on John Nessime.

Then someone points out that this particular support lead has never had access to Opportunity records. Not through her profile, not through the role hierarchy, not through a sharing rule. And the assistant just read one out to her.

That is the failure that kills most CRM assistant projects, and it almost never shows up in testing, because you test as an admin. Everything you ask works. Everything you ask is supposed to work. The gap only opens when a real user with a real, narrow profile starts asking questions.

This post walks through building a CRM knowledge assistant on Salesforce and Amazon Bedrock Knowledge Bases: which of the two knowledge base paths you are actually allowed to use, how the Salesforce connector authenticates, what it does and does not carry across from your CRM, and the two architectures that survive a real permission model. The wiring is the easy part. The scoping is the work.

The three moving parts

Strip away the marketing and a CRM knowledge assistant is three things bolted together:

• Ingestion — something pulls Cases, Knowledge articles, Opportunities and Accounts out of Salesforce on a schedule, chunks them, embeds them, and writes the vectors somewhere.
• Retrieval — a query comes in, gets embedded, and the closest chunks come back. This is where filtering lives, and it is the security surface.
• Generation — a foundation model reads those chunks and writes an answer with citations.

Generation is the part everyone stares at and the part you should worry about least. It is a model choice and a prompt. Retrieval is where the assistant becomes either trustworthy or a liability.

Decision one: which knowledge base you are allowed to use

Bedrock now has two shapes of knowledge base, and this decision gets made for you the moment you say “Salesforce”.

Bedrock Managed Knowledge Base is the newer, opinionated one. AWS manages the vector store, picks the embedding and reranking models by default, handles parsing, and exposes agentic multi-hop retrieval. It ships with a small set of first-party connectors: Amazon S3, SharePoint, Confluence, Google Drive, OneDrive, Web Crawler, and a custom ingestion path. Salesforce is not on that list.

Customer-managed knowledge bases are the older shape. You bring the vector store, you pick the embedding model, you own more of the pipeline. This is where the Salesforce connector lives, and AWS still flags that connector as a preview release subject to change. It also has a hard constraint worth reading twice: with a Salesforce data source, Amazon OpenSearch Serverless is currently the only supported vector store. If your organisation has standardised on Aurora pgvector or something else, that standard does not apply here.

So the fork is real. Either you use the preview Salesforce connector on the customer-managed path, or you land Salesforce data in S3 yourself and use the managed path. I will come back to why the second option is often the right call, because it is not just a workaround.

One piece of context that matters if you are inheriting an older design: Amazon Kendra entered maintenance mode on 30 June 2026 and stopped accepting new customers on 30 July 2026. Kendra was the standard answer to “search my Salesforce org” for years, and AWS now points people at Bedrock Knowledge Bases instead. Existing indexes keep running and no end-of-service date has been set, but Kendra will not gain features. If a proposal on your desk starts with a Kendra index, that is worth raising before anyone writes code.

The permission model is the whole problem

Salesforce spends enormous effort on who can see what. Profiles, permission sets, the role hierarchy, org-wide defaults, criteria-based sharing rules, field-level security, restriction rules. A given Opportunity is visible to a specific, computed set of users, and that computation happens on every query.

None of that survives ingestion.

The AWS documentation is blunt about it, and this is the single most important sentence in the whole feature: everything you sync from your data source becomes available to anyone holding bedrock:Retrieve on that knowledge base, including content that had controlled permissions at the source. Three layers of Salesforce scoping collapse into one IAM action.

Worse, the connector authenticates as a single identity. You configure a Connected App using the OAuth 2.0 client credentials flow, and that flow runs as one named user. Whatever that user can see, the crawler ingests. Pick a System Administrator to “make sure the sync works” and you have just copied the entire org into a flat index that your whole application tier can read.

Why this is invisible until it isn’t

Nothing errors. There is no warning in the console, no failed sync, no denied API call. Retrieval returns chunks with good similarity scores and the model writes a fluent, cited answer. The only signal that something is wrong is a human recognising a record they should not be looking at, and that human is usually a customer-facing employee, not you.

Treat the knowledge base as a published dataset, not as a view over Salesforce. Ask the question you would ask about an S3 bucket: if every row of this landed in front of every user of the app, who gets hurt?

The two architectures that actually hold up

Option A: ingest only what everyone is allowed to see. Run the Connected App as a deliberately low-privilege integration user whose profile grants read access to Knowledge articles, published solutions, product data and price books, and nothing else. Exclude Contact, Lead, Case and Opportunity entirely. You get a genuinely useful support and enablement assistant, the permission question disappears, and you can ship it without a security review turning into a project.

This is the one I reach for first. Most of the value people want from a CRM knowledge assistant is “what do we know about this product, this policy, this recurring problem”, and none of that needs record-level data.

Option B: export to S3 with your own access metadata. If you genuinely need per-record answers, stop using the connector. Pull the objects you need through the Salesforce API or a scheduled extract, write each record to S3 as a document, and write a sidecar metadata file next to it carrying the access attribute you will filter on. Then point a knowledge base at the bucket and apply an explicit metadata filter on every retrieval, derived from the caller’s identity rather than from the query text.

The cost is honest: you now own an extraction pipeline, a metadata schema, and the job of keeping the access attribute correct when a record changes owner. That is real work. The benefit is that you can point at exactly where the boundary is enforced and test it.

What you cannot do is enforce the boundary in the prompt. “Only answer about accounts the user owns” is a suggestion to a language model, not an access control. Retrieval already happened by then.

Wiring the Salesforce connector

Assuming you have chosen Option A and want the connector, here is the shape of it.

On the Salesforce side

1. Create a Connected App in your org and enable the OAuth 2.0 client credentials flow.
2. Under the client credentials flow settings, set the Run As field to your integration user. This is the step people skip, and the sync fails without it. It is also the step that decides your entire blast radius, so pick that user before you pick anything else.
3. Copy the consumer key and consumer secret from the app’s OAuth settings.
4. Note your instance URL.

On the AWS side

Store the credentials in AWS Secrets Manager. The connector looks for three specific keys, and it will not fall back to alternatives if you name them differently:

{
  "consumerKey": "your Connected App client ID",
  "consumerSecret": "your Connected App client secret",
  "authenticationUrl": "https://yourcompany.my.salesforce.com/"
}

The secret has to live in the same AWS region as the knowledge base. Cross-region reads are not supported here, and the resulting error is generic enough to waste an hour. Do not reuse this secret for any other data source; rotate it on the same schedule as your other integration credentials.

Creating the data source

The data source configuration is a JSON document. The interesting part is patternObjectFilter, which is how you keep the crawler away from objects you never wanted:

{
  "salesforceConfiguration": {
    "sourceConfiguration": {
      "hostUrl": "https://yourcompany.my.salesforce.com/",
      "authType": "OAUTH2_CLIENT_CREDENTIALS",
      "credentialsSecretArn": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:NAME"
    },
    "crawlerConfiguration": {
      "filterConfiguration": {
        "type": "PATTERN",
        "patternObjectFilter": {
          "filters": [
            {
              "objectType": "Knowledge__kav",
              "inclusionFilters": [".*"]
            },
            {
              "objectType": "Case",
              "exclusionFilters": [".*"]
            }
          ]
        }
      }
    }
  },
  "type": "SALESFORCE"
}

Two things to internalise about these filters. First, they are regular expressions matched against a per-object identifier, not against record contents: for Knowledge__kav that is the article title, for Case the case number, for Attachment the file name. Second, when an inclusion and an exclusion filter both match the same document, exclusion wins. That precedence is what makes an allow-list pattern safe to write.

Then create the data source and attach it to an existing knowledge base:

aws bedrock create-data-source 
  --name "salesforce-knowledge" 
  --knowledge-base-id "YOUR_KB_ID" 
  --data-source-configuration file://salesforce-connector-config.json 
  --data-deletion-policy "DELETE" 
  --vector-ingestion-configuration '{"chunkingConfiguration":[{"chunkingStrategy":"FIXED_SIZE","fixedSizeChunkingConfiguration":[{"maxTokens":"300","overlapPercentage":"10"}]}]}'

A note on the command namespace: the agent-plane operations have historically lived under a separate CLI command group from the model-inference ones, and this has shifted between CLI versions. Run aws bedrock help against your installed version before assuming. The JSON payload is the stable part.

--data-deletion-policy DELETE means the vectors get removed from the store when you delete the data source. The alternative retains them. Choose DELETE unless you have a specific reason not to, because orphaned vectors in an OpenSearch Serverless collection are both a cost line and a surprise waiting for whoever inherits this.

After that, kick off a sync with StartIngestionJob, or hit Sync in the console. The connector does incremental syncs after the first run: added, updated and deleted content only. The first run crawls everything.

Chunking is a one-way door

You cannot change the chunking strategy after connecting the data source. Not tune it, not swap it. You delete the data source and start over, which means re-crawling and re-embedding everything.

So spend twenty minutes here rather than five. CRM content is not uniform. Knowledge articles are structured prose and behave like documentation. Case comment threads are short, conversational, and lose their meaning when split mid-exchange. Opportunity descriptions are often two sentences. A fixed-size chunk that suits one of these will mangle another.

The practical move is to stop trying to find one strategy that fits everything. Create separate data sources for materially different content shapes, each with its own chunking configuration, all pointing at the same knowledge base. It costs you a bit of configuration and saves you a full re-index later.

One more constraint: the Salesforce connector does not handle multimodal content. Tables, charts and diagrams embedded in your Knowledge articles are not going to be understood. If your best troubleshooting articles are mostly screenshots, this connector will quietly ingest the surrounding text and nothing else, and your retrieval quality will be worse than the article count suggests.

Retrieval, filtering, and one trap

Bedrock gives you two kinds of metadata filter at query time, and confusing them is a security bug.

Explicit filters are the ones you construct in code and pass in the retrieval configuration. Your application decides them from the authenticated caller. They are deterministic.

{
  "retrievalQuery": { "text": "renewal value on the Northwind account" },
  "retrievalConfiguration": {
    "vectorSearchConfiguration": {
      "numberOfResults": 8,
      "filter": {
        "andAll": [
          { "equals": { "key": "visibility_group", "value": "emea-enterprise" } },
          { "equals": { "key": "record_class", "value": "internal" } }
        ]
      }
    }
  }
}

Implicit filters are the other kind. You declare which metadata attributes exist and Bedrock uses a model to infer filters from the natural-language query. Someone asks about last quarter, it infers a date filter. This is a genuinely nice relevance feature and it is not an access control. A model inferring a filter from user-supplied text is a model that can be talked out of it. Use implicit filters to improve answers; use explicit filters to enforce boundaries.

The trap: filters on the Salesforce connector operate on auto-detected fields from your CRM records. You cannot inject your own visibility_group attribute at ingestion. That capability comes from sidecar metadata files, which is an S3 data source feature. This is the concrete reason Option B routes through a bucket. If per-user scoping is a requirement, the connector cannot give it to you no matter how carefully you configure it.

Where the money goes

Rates change, so learn the meters instead of memorising numbers.

• Embedding at ingestion — billed per token processed. A full re-crawl re-embeds everything, which is exactly why the chunking decision is expensive to reverse.
• Vector storage — on the customer-managed path this is your OpenSearch Serverless collection, billed on provisioned compute units and storage, with a floor whether or not anyone queries it. This is the line item that surprises people on a proof of concept that nobody uses. Managed Knowledge Base bills on indexed data size and retrieval count instead, with no idle collection sitting underneath.
• Generation — input and output tokens on whichever foundation model you chose. Retrieved chunks are input tokens, so numberOfResults is a direct cost lever and worth tuning down until quality actually degrades.
• Everything around it — the API layer, the chat frontend, and the observability stack. A small VPS from a provider like InterServer or a Lambda behind API Gateway both work; the choice barely matters next to the Bedrock line. Push retrieval latency and citation counts into Grafana Cloud or Datadog early, because “the assistant feels worse this week” is otherwise unfalsifiable.

Troubleshooting

• Sync fails immediately with an authentication error. Check the Run As user on the Connected App’s client credentials flow first. An empty Run As field is the most common cause. Then check that the secret keys are spelled exactly consumerKey, consumerSecret, authenticationUrl, and that the secret is in the knowledge base’s region.
• Sync succeeds but the document count is far lower than expected. Usually the integration user’s profile, not the filters. Query the object as that user in Salesforce and count. Also check the per-knowledge-base file count and file size quotas, which are separate limits and are silently enforced.
• Retrieval returns nothing for queries you know are covered. Test with no filter at all. If results appear, your filter keys do not match the auto-detected field names. If they do not, the content probably never made it past chunking; check whether those articles are mostly images.
• Deleted Salesforce records still turn up in answers. Incremental sync handles deletions, but only on the next sync. If you deleted something sensitive, run a sync immediately rather than waiting for the schedule.
• Answers are confidently wrong about numbers. Fixed-size chunking split a record away from its label. Retrieve the raw chunks for that query and read them. Nine times out of ten the problem is visible in the chunk, not in the model.

Common mistakes

• Running the Connected App as a System Administrator “just to get the sync working”. That decision is almost never revisited.
• Assuming sharing rules and field-level security follow the data into the index. They do not.
• Enforcing access in the system prompt. Retrieval has already happened by the time the model reads your instructions.
• Ingesting Contact and Lead because they were there. Both carry personal data and both are usually irrelevant to the questions people actually ask.
• Accepting default chunking, then discovering it is immutable.
• Building on a preview connector without telling stakeholders it is preview. The configuration surface can change under you.
• Testing only as an admin. Every acceptance test should run as the narrowest real profile you have.

Best practices

• Decide the integration user’s profile before you touch AWS. It defines the security boundary more than any IAM policy will.
• Start with Knowledge articles only. Ship that, get real queries, then argue about whether Opportunity data is worth the architecture it requires.
• Write filters as allow-lists. Exclusion beats inclusion, so an allow-list fails closed.
• Split content shapes across separate data sources so chunking suits each one.
• Keep a fixed evaluation set of about fifty real questions with known-good answers, and re-run it after every configuration change.
• Log the retrieved chunk IDs alongside every generated answer. When someone reports a bad answer months from now, that log is the only thing that makes it debuggable.
• Treat bedrock:Retrieve as a privileged permission and scope it to the specific knowledge base ARN, not to a wildcard.

Frequently asked questions

Does the Bedrock Salesforce connector respect sharing rules?

No. The crawler authenticates as one user and everything it can read becomes retrievable by anyone with bedrock:Retrieve on the knowledge base. Per-user scoping has to be built at the retrieval layer with explicit metadata filters, which in practice means routing through S3 rather than using the connector.

Can I use Aurora or pgvector instead of OpenSearch Serverless?

Not with the Salesforce data source. AWS currently restricts that connector to Amazon OpenSearch Serverless. If your vector store is non-negotiable, extract to S3 and use a data source that supports your chosen store.

Should I use Agentforce instead of building this on Bedrock?

If your questions are answerable entirely from Salesforce data and your users already live in Salesforce, Agentforce is the shorter path and it operates inside the platform’s existing permission model, which removes the hardest problem in this post. Bedrock earns its complexity when the assistant has to reason across CRM data plus sources that are not in Salesforce, or when you need a specific foundation model, or when the data must stay inside your AWS account. Cost profiles differ substantially and are worth modelling on your real query volume before committing.

How current will the assistant’s answers be?

As current as your last sync. Ingestion is scheduled, not live. For rapidly changing fields such as case status or pipeline stage, do not answer from the index at all: have the assistant call the Salesforce API for the live value and use the knowledge base only for the surrounding context.

What happens to the connector when it leaves preview?

Unknown, which is the point of the label. Preview means the configuration surface, supported objects and behaviour can change without the compatibility guarantees you would expect from a GA service. Keep your data source configuration in version control so a re-create is a rerun rather than an archaeology exercise.

Is a CRM knowledge assistant worth building if we already have Salesforce search?

It depends on the question shape. Search is better when people know what record they want. A CRM knowledge assistant wins on synthesis: “what usually causes this error for enterprise customers” pulls from twenty articles and cases at once, which no search result page does well. If your users mostly navigate to known records, you will not get much return.

Do I need Bedrock AgentCore for this?

Not for retrieval and generation. AgentCore matters when you want the assistant to take actions, chain tools, or be exposed to agent frameworks over MCP. A read-only question-answering assistant works fine against the knowledge base APIs directly, and skipping AgentCore removes a layer you would otherwise have to operate.

The one thing to remember

Connecting Salesforce to Amazon Bedrock takes an afternoon. The connector works, the syncs run, the answers are good. That speed is exactly what makes this dangerous, because nothing in the setup path forces you to think about who the answers are for.

A CRM knowledge assistant is a publishing decision wearing an engineering costume. Every record you ingest, you are publishing to everyone who can call retrieve. Decide what that audience is allowed to see, encode it in the integration user’s profile and in your inclusion filters, and test every question as your narrowest real user. Do that first and the rest of the build is routine.

Need help getting this into production?

I work with teams building retrieval systems on AWS, usually at the point where a working prototype has to survive a security review. Things I can help with on a CRM knowledge assistant:

• Reviewing an existing Bedrock knowledge base for records that should never have been ingested, and scoping the integration user’s Salesforce profile to match.
• Designing the S3 extraction path with sidecar access metadata when per-user filtering is a hard requirement.
• Choosing between the managed and customer-managed knowledge base paths, with the OpenSearch Serverless cost floor modelled against your actual query volume.
• Building the chunking and data source split before it becomes an immutable mistake, plus a repeatable evaluation set to measure changes against.
• Terraform or OpenTofu for the whole stack: knowledge base, data sources, Secrets Manager, IAM, and the vector store.
• Retrieval observability so you can tell whether answer quality moved, and why.
• Migration planning for teams currently on a Kendra index.

If you have a data source configuration, a failing sync, or a retrieval result that looks wrong, send it over and we can work out what it is doing.

The post Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records appeared first on John Nessime.