That is the shape of every serious problem in this space. A personalized learning assistant on AWS almost never fails loudly. It fails by being helpful to the wrong person, or by being confidently wrong in a way that reads as authoritative because it came from the institution’s own system. Both failures are silent until a human reports them, and by then the assistant has been answering that way for a term.

This post covers the parts that are genuinely hard: getting identity and scope out of the learning platform instead of the browser, making retrieval respect the boundary, knowing what Bedrock Guardrails will and will not do for you, sizing a system whose traffic collapses every summer, and proving after the fact what the assistant saw. I assume you can already stand up a RAG pipeline. The pipeline is the easy part.

What “personalized” has to mean before you build anything

Personalization in an education setting is three separate questions that people tend to collapse into one:

• What is this learner allowed to see? Course enrollment, section, release dates, role. This is an authorization question and it belongs nowhere near the model.
• What has this learner already done? Submissions, attempts, viewed material. This is state, and it belongs in a database you control.
• How is this learner doing? Mastery, misconceptions, pace. This is inference, and it is the only part where a model earns its keep.

Teams that skip straight to the third question end up asking the model to enforce the first one through prompt instructions. That does not hold. A system prompt saying “do not reveal solutions before the due date” is a suggestion; a retrieval filter that never returns the solution chunk is a control. Keep the boundary in the retrieval layer and the personalization in the generation layer, and most of the scary failure modes stop being possible rather than becoming unlikely.

Where a personalized learning assistant on AWS actually breaks

Organized by failure family rather than by service, because the services are documented and the failures are not. Five of them, roughly in the order they bite.

Failure family one: identity that stops at the front door

The assistant lives inside Canvas, Moodle, Blackboard, Brightspace or Open edX. It gets there through LTI 1.3, which replaced the old shared-secret model with an OpenID Connect launch and an RSA-signed JWT. The platform initiates a login against your tool’s login endpoint, your tool responds with an auth request, the platform posts back a signed token, and you validate it against the platform’s published keys.

Almost everyone gets that far. The failure is what happens next.

The launch token carries a stable subject identifier, a roles claim, a context claim describing the course, and a deployment identifier, all namespaced under the 1EdTech spec URIs. Teams validate the signature, pull the user ID out, then build the rest of the session from whatever the front end sends afterwards. Course ID as a query parameter. Role in local storage. It works in testing because the front end sends the truth, and it stops working the moment a curious student opens developer tools. If your filter values arrive from the client, you have built an access control system that asks the attacker what they should be allowed to see.

Take the scope from the launch, then keep it server-side

The pattern that holds up:

1. Validate the launch JWT against the platform’s key set. Check the issuer, the audience, the nonce and the deployment identifier, not just the signature.
2. Extract subject, roles and context from the token and write them into a server-side session. Hand the browser an opaque session cookie and nothing else.
3. Derive every retrieval filter from that session record. The client never names a course, a section or a role.
4. If you need the full roster rather than just the launching user, call Names and Role Provisioning Services with an OAuth 2.0 client credentials token. That gives you learners who have never launched the tool, which matters for cohort-level features.
5. If you write grades back, do it through Assignment and Grade Services against a line item you created. A launch that works tells you nothing about whether passback works.

Test launch, roster fetch and grade passback as three independent contracts. They fail independently and they fail quietly. A term can start with a tool that launches beautifully and drops every grade on the floor because AGS was never wired to a line item.

Worth knowing before you commit: LTI 1.3 is meaningfully more setup than 1.1. Key registration, the OIDC handshake, per-deployment configuration on every platform instance. That is the price of not signing student traffic with a shared secret, and it is worth paying, but budget for it honestly. If you need to test against several platforms, a self-managed Moodle or Open edX instance on a cheap VPS from somewhere like Contabo or InterServer beats waiting on sandbox tenants, and you get to break it freely.

Failure family two: retrieval that ignores the boundary

This is the one from the opening. Vector search does not know about courses. It knows about similarity. If the answer key is semantically close to the question, it comes back, and the model uses it, and the response is fluent and correct and completely inappropriate.

Amazon Bedrock Knowledge Bases handles this through metadata filtering: you attach attributes to each document at ingestion, then pass a filter at query time that pre-filters the vector store before the similarity search runs. The filter is the control. Everything else is decoration.

The sidecar file is the only metadata that counts

For an S3 data source, attributes come from a companion file sitting next to the document with .metadata.json appended to the object key. This trips people up constantly, because S3 objects already have user-defined metadata and object tags, and neither of those is read for retrieval filtering. Keep lifecycle and cost allocation in S3 tags where they belong, and put filterable attributes in the sidecar.

{
  "metadataAttributes": {
    "course_id": "BIO201",
    "section_id": "BIO201-B",
    "institution_id": "uni-main",
    "material_type": "lecture_notes",
    "visibility": "student"
  }
}

Note visibility as a first-class attribute rather than an inference from folder structure. Rubrics, model answers and marking schemes get visibility: "instructor", and the student-facing filter excludes them by value, not by path convention. Path conventions get violated the first time a busy tutor uploads a file to the wrong folder.

At query time you compose the filter server-side from the session record. Bedrock’s retrieval configuration supports boolean composition, so the tenant condition and the visibility condition are separate clauses that must both hold:

{
  "vectorSearchConfiguration": {
    "filter": {
      "andAll": [
        { "equals": { "key": "institution_id", "value": "uni-main" } },
        { "equals": { "key": "course_id",      "value": "BIO201"   } },
        { "equals": { "key": "visibility",     "value": "student"  } }
      ]
    }
  }
}

Two habits make this survive production. Build the filter through a single function that always prepends institution and course, so no future endpoint can forget it. Then write a test that queries a known instructor-only chunk’s exact wording from a student session and asserts zero results. Manual smoke testing never catches this, because human testers ask reasonable questions and the failure only appears when someone asks an unreasonable one.

Silo, pool or bridge

AWS frames multi-tenant isolation as three patterns, and the trade-off is the same one you always get between blast radius and cost:

• Silo gives each institution its own knowledge base and its own vector index. Strongest isolation, easiest compliance conversation, and you multiply your idle infrastructure cost by the number of tenants. There is also a per-account service quota on knowledge bases, so check Service Quotas before you promise a hundred districts their own.
• Pool puts everyone in one index and relies entirely on metadata filtering. Cheapest by a wide margin, and one filter bug is a cross-institution disclosure rather than an embarrassment.
• Bridge splits the difference: separate knowledge bases sharing underlying vector infrastructure, which keeps the isolation story cleaner without paying for a full stack per tenant.

For a single university running many courses, pool with rigorous filtering is usually right, because the institution is the tenant and the courses are not. For an edtech vendor selling to multiple districts, I would start at bridge and only pool if the economics force it, because the conversation with a district’s data protection officer goes very differently when your answer is “we filter” rather than “they are separate indexes.”

Failure family three: guardrails that do less than you assume

Bedrock Guardrails is a useful layer and the first thing I reach for on an education workload: content filters, denied topics, word filters, PII detection and contextual grounding checks, applied at inference time and usable across models. For a system that teenagers will deliberately try to provoke, enforcement inside the invocation path beats anything bolted on afterwards. Three limits are documented and routinely missed.

Contextual grounding does not cover chatbots. AWS documents the supported use cases as summarization, paraphrasing and question answering, and states plainly that conversational QA and chatbot use cases are not supported. A tutoring assistant is a multi-turn conversation. If your hallucination story is “we turned on grounding checks,” read that page before it reaches a procurement document. The check still earns its place on single-shot retrieve-and-answer paths, which is where I would apply it, but a multi-turn tutor needs citation display and human review too.

PII detection is probabilistic and text-only. The sensitive information filter is a machine learning model that uses surrounding context to decide whether something is an identifier. It works better on prose than on bare fragments. AWS also documents that it operates on text output and does not inspect tool use parameters, so if your assistant calls a function with a student identifier in the arguments, that path is not covered. Do not treat it as a substitute for not putting identifiers in the prompt in the first place.

Blocked content lands in your logs. AWS documents that content blocked by guardrail policies appears as plain text in Bedrock model invocation logs when those logs are enabled. That is defensible for debugging and awkward when the blocked content is a distressed message from a fourteen year old. Decide deliberately whether invocation logging is on, where it lands, who can read it, and how long it lives. In education this is a governance decision, not a logging default.

None of this argues against Guardrails. It argues for knowing its edges, because the gap between what a feature name suggests and what the documentation says is where compliance commitments go to die.

Failure family four: the cost shape of an academic term

Education traffic is not a smooth curve. It is near zero over summer, spikes brutally in the week before every assessment deadline, and dies again. That shape punishes the default architecture in a specific way.

The console nudges you toward Amazon OpenSearch Serverless as the vector store. It bills per OpenSearch Compute Unit hour with a minimum baseline footprint that exists whether or not anyone queries it, so an empty index over a quiet August still bills. AWS has adjusted those minimums over time, so pull current figures from the OpenSearch Serverless capacity documentation rather than from any blog post, this one included. The mechanism is the part that matters: your floor is set by capacity that exists, not by traffic that happens.

Bedrock Knowledge Bases supports other backends, and the question that decides between them is idle cost:

• Aurora PostgreSQL with pgvector. A first-class documented option. Aurora Serverless v2 bills per Aurora Capacity Unit hour and can scale down to zero when idle, which fits the academic calendar almost too neatly. You take on index lifecycle work in exchange: choosing between HNSW and IVFFlat, keeping statistics fresh, watching bloat. If your institution already runs Postgres, that is work your DBAs recognize.
• Amazon S3 Vectors. Bills on storage and requests rather than provisioned capacity, so genuine idle costs approach nothing. The trade is retrieval latency, which is higher than a warm search cluster. For a tutor where a response already takes a second or two of model time, that may not be the bottleneck you think it is.
• OpenSearch Serverless. The right answer when retrieval latency is genuinely your constraint, when you need hybrid search and k-NN tuning, or when you already run collections for other workloads and the baseline is already paid for.

On the model side, the cost driver is retrieved context, not the question. Every turn ships the retrieved chunks as input tokens, so chunk size and the number of results retrieved multiply directly into your bill on every single message. Halving retrieved results is usually a bigger lever than switching to a cheaper model, and it often improves answer quality too, because the model has less irrelevant material to get distracted by.

Tag by course and faculty from day one. The first budget question you will be asked is which department is driving the spend, and retrofitting cost allocation tags after a pilot goes institution-wide is miserable. Cost Explorer answers it if the tags exist; tools like Vantage or CloudZero make the per-course view easier to hand to a finance team that does not want an AWS login.

Failure family five: no evidence when someone asks

FERPA governs education records in the United States, and the mechanism that lets a vendor touch them at all is the school official exception: the institution must maintain direct control over the vendor’s use and maintenance of the records. Similar logic applies under other regimes, and many districts and state contracts add data residency requirements that FERPA itself does not impose.

What that means architecturally is less exotic than it sounds, and it is mostly about being able to answer questions after the fact:

• Log the retrieval, not just the answer. Store which chunk identifiers were returned for each turn. When an instructor claims the assistant leaked a rubric, the retrieval log is the only thing that settles it.
• Keep student identifiers out of prompts. Pass an opaque session-scoped pseudonym and resolve it in your own system. The model does not need a real name to be helpful.
• Encrypt with customer managed KMS keys and use encryption context that names the tenant, so a key policy can actually constrain who decrypts what.
• Pin the region and enforce it with service control policies if residency is contractually required. An SCP is checkable evidence; a runbook is not.
• Write down retention for conversation history and enforce it with lifecycle rules. “We keep it forever because storage is cheap” is a defensible engineering position and an indefensible one in an education procurement review.
• Pull the current compliance reports and agreements from AWS Artifact rather than quoting a blog about what AWS certifies.

None of this is glamorous and all of it is cheaper to build in during the pilot than to retrofit when the institution’s legal team gets involved.

Troubleshooting the things that actually page you

“The new material isn’t showing up”

Almost always ingestion, not retrieval. Walk it in order rather than guessing. Did a sync actually run after the change? For an S3 data source, an edit is invisible until an ingestion job processes it.

aws bedrock-agent list-ingestion-jobs 
  --knowledge-base-id KB_ID 
  --data-source-id DS_ID

Look for a completed job starting after the upload. If the most recent job predates the change, your scheduler or event handler is the bug and retrieval is innocent. If a job did run, read its statistics: a modified-documents-indexed count of zero when you expected a change means change detection did not fire, which happens when an object is re-uploaded with identical bytes. A non-zero failed count means your file is probably the one that failed, and the failure reasons say why.

“The filter returns nothing at all”

Query once with no filter to confirm the content is indexed. If it comes back unfiltered and vanishes when filtered, it is a metadata problem: the sidecar was not ingested, the attribute name does not match exactly, or the value differs in case. These comparisons are literal. BIO201 and bio201 are two different courses as far as the filter is concerned, which is an argument for normalizing identifiers at both ingestion and query time in one shared helper.

“The launch shows a blank iframe”

Two suspects dominate. Cookies: your tool is in a cross-site iframe, so session cookies need the right SameSite and Secure attributes or the browser drops them and every launch looks like a fresh unauthenticated user. Registration mismatch: issuer, client ID or deployment ID differs between what the platform sends and what you stored. Log the received issuer and deployment ID on validation failure. Guessing at OIDC failures is slow; the values are right there.

“Answers are right but the citations are wrong”

Usually chunking. Slides and worksheets fragment badly under naive fixed-size chunking, and a chunk that starts mid-table cites a heading that belongs to something else. Before you change models, look at what the retriever actually returned for the query that misbehaved. Nine times out of ten the model was faithful to a bad chunk.

“Grades aren’t reaching the gradebook”

Check that a line item exists and that your service token carries the right AGS scopes. Passback failures are asynchronous, so they surface as an empty gradebook column weeks later rather than as an error the student sees. Alert on passback failure rate explicitly; nobody will report it in time.

Common mistakes

• Enforcing access control in the system prompt. Instructions are not permissions. If a chunk can be retrieved, assume it can be surfaced.
• Trusting client-supplied course or role values. Derive scope from the validated launch token and store it server-side, always.
• Ingesting the whole course folder. Instructor material and student material in one prefix is how rubrics leak. Separate at ingestion, tag at ingestion.
• Putting real names and student numbers in prompts. Pseudonymize before the model call and resolve afterwards.
• Sizing the vector store from peak week. The bill is set by the idle floor, and in education idle is most of the year.
• Assuming a guardrail feature does what its name implies. Read the supported use cases. Contextual grounding on a chatbot is the classic example.
• Piloting with one enthusiastic lecturer. That lecturer’s material is unusually well organized. The second department will not be.

Best practices

• Build the retrieval filter in exactly one function, and make the tenant and visibility clauses impossible to omit.
• Keep a small regression set of adversarial queries, including students asking for solutions and asking about classmates, and run it on every deployment.
• Show citations in the interface. Instructors trust the system faster when they can see the source, and it converts silent hallucinations into reportable ones.
• Give instructors a preview mode that answers as a student would see it. They will find scoping errors far faster than any test suite.
• Route the tool endpoint through a CDN and WAF such as Cloudflare, with rate limiting per session. Assessment week is a load test you did not schedule.
• Dashboard retrieval count, token spend per course and passback failure rate together. CloudWatch metrics fronted by Amazon Managed Grafana or Grafana Cloud gives faculty a view without handing out console access.
• Have an escalation path for distressing content that reaches a human being, and make sure the people on it know they are on it. This is a policy question you must answer before launch, not after.

Frequently asked questions

Do we need to fine-tune a model for our curriculum?

Almost certainly not. Curriculum content changes every term, and retrieval updates in minutes where fine-tuning is a rebuild. Fine-tuning earns its place for style and format consistency, not for facts. Start with retrieval, and revisit only if you have a specific behavior that prompting genuinely cannot produce.

How do we stop it from just doing the homework?

Partly retrieval scope, partly prompting, partly accepting you cannot fully win. Exclude solutions and rubrics from student-visible retrieval, prompt for Socratic responses on assessed material, flag the assessment context on the way in so the assistant knows which mode it is in. A determined student can paste the question into a consumer chatbot regardless, which is a course design question rather than an infrastructure one.

Is Bedrock the right choice versus calling a model API directly?

If the institution is already on AWS, Bedrock keeps the traffic inside your account boundary, gives you IAM and CloudTrail on the same footing as everything else, and can be reached over VPC endpoints. A direct provider API is often simpler and sometimes gets new models sooner. The deciding factor is usually procurement: one existing AWS agreement is a much shorter conversation than a new vendor review.

Does student data get used to train the model?

AWS states that Bedrock does not use customer inputs or outputs to train foundation models. Verify it against current AWS documentation and your agreement rather than any third-party summary, because this is exactly the clause a district will ask you to evidence in writing.

What does it cost to run a personalized learning assistant on AWS?

Model your bill as three parts: a vector store floor that bills whether or not anyone asks anything, a one-off embedding cost when you ingest the corpus, and per-turn inference dominated by retrieved context rather than by the question. For a small corpus with a term-shaped traffic curve, the floor usually dominates, which is why the vector store choice matters more than the model choice.

How do we know whether it actually helps anyone learn?

Decide the measure before launch and instrument for it. Usage is not learning. Pair the assistant’s logs with assessment outcomes on comparable cohorts and accept that a clean answer needs a study design, not a dashboard. Engineering can supply the data; it cannot supply the conclusion.

The one thing worth remembering

A personalized learning assistant on AWS will never tell you when it is wrong about who someone is. The model has no concept of enrollment, and retrieval has no concept of a due date. Both live entirely in the metadata you attach at ingestion and the filter you build server-side from a validated launch token. Get that right and what remains is ordinary engineering: chunking, cost curves, dashboards. Get it wrong and you ship something that works flawlessly right up until the week it embarrasses the institution.

Build the boundary first. Make it a control, not an instruction. Then go be clever with the tutoring.

Need a hand building or auditing one of these?

I work on the infrastructure side of AI assistants for education, which mostly means the parts that decide whether the thing is safe to put in front of students:

• Designing and reviewing the LTI 1.3 launch path, roster access and grade passback, including the cross-site cookie and registration issues that cause blank-iframe launches.
• Building the retrieval isolation layer: sidecar metadata schemas, server-side filter composition, and an adversarial regression suite that proves instructor material stays invisible to students.
• Choosing and migrating the vector store when the OpenSearch Serverless floor is eating a pilot budget, including Aurora pgvector and S3 Vectors options.
• Guardrail configuration that matches the documented behavior rather than the feature name, plus a logging and retention design you can defend in a procurement review.
• Cost allocation tagging by course and faculty, with dashboards that answer “which department is spending this” without handing out AWS console access.
• Infrastructure as code for the whole stack so a second institution is a variable file rather than a fortnight.

If you have something concrete, send it over: a retrieval filter you are not sure is airtight, an ingestion job that keeps skipping documents, a failing LTI launch log, or a Cost Explorer breakdown that stopped making sense. Easier to look at the real thing than to talk in the abstract.

The post Building a Personalized Learning Assistant on AWS Without Leaking Student Data appeared first on John Nessime.

The user pool was set to MFA: Required. Somebody had screenshotted that console page for the security questionnaire. The screenshot was accurate. It just didn’t mean what everyone assumed it meant.

That gap is the whole problem with Cognito MFA for healthcare apps. The service gives you a setting that reads like a guarantee, and then quietly carves out entire classes of users, sessions, and sign-in paths that the setting never touches. Nothing errors. Nothing logs a warning. Your compliance evidence looks fine right up until someone actually tests it.

This post covers what the MFA switch in Amazon Cognito actually controls, the five failure families that show up in real healthcare deployments, how to wire a user pool so the enforcement matches the claim, and what to check when a clinician calls the help desk because they can’t get in.

What HIPAA actually requires from MFA right now

Worth getting this straight before touching any configuration, because a lot of vendor content is confidently wrong about it.

The HIPAA Security Rule as it stands today does not name multi-factor authentication. It requires person or entity authentication, and it leaves the mechanism to your risk analysis. MFA is how most people satisfy that in practice, but the rule does not spell it out.

What changes the picture is the Notice of Proposed Rulemaking that HHS published in the Federal Register on January 6, 2025. It would make MFA an explicit requirement for access to systems handling ePHI, with narrow exceptions, and it would remove the “addressable” escape hatch that currently lets organizations document a control as infeasible. This one matters enough to name the date, because timelines have moved: the comment period closed in March 2025, the rule is still proposed, and the Unified Agenda has pushed anticipated final action out to 2027. That estimate is not binding and has already slipped once.

Practical read: build for the proposed rule now, but don’t tell a client the mandate is already law. It isn’t, and auditors know the difference.

On the AWS side, Amazon Cognito is on the HIPAA Eligible Services list and is covered by the AWS Business Associate Agreement. Eligibility is a starting line, not a finish line. You still have to accept the BAA in each AWS account that touches PHI, and you still have to configure the service correctly. Nothing about creating a user pool makes it compliant.

Failure family one: half your users never see the MFA prompt

This is the one that produced the pen test finding, and it’s the most common serious gap I run into.

Cognito user pool MFA applies to local users. Those are the accounts that live in the user pool directory and authenticate against it. For federated users, the ones arriving through SAML or OIDC from a hospital identity provider, Okta, Entra ID, or an in-house ADFS, Cognito hands the entire authentication decision to the upstream IdP. It does not add a second factor. It cannot. It never sees the password.

So a portal with patients as local users and staff federated from a hospital directory has exactly the split you’d fear: patients get MFA, clinicians don’t, unless the hospital IdP is enforcing it independently. And the hospital IdP is usually somebody else’s system, run by a team you don’t have a Slack channel with.

There’s a second version of this. The app client, not the user pool, controls which authentication flows are permitted. If a legacy mobile client is still allowed to use a flow you thought you’d retired, that path stays open regardless of what the pool-level MFA setting says.

What to do about it:

• Inventory every app client attached to the pool. Not the ones you remember creating. Run list-user-pool-clients and account for all of them.
• For each federated IdP, get written confirmation of what the upstream enforces. A trust relationship is not a control. Put the answer in your risk analysis where an auditor will find it.
• Treat federation as a boundary you inherit risk across, and say so out loud in design reviews. This is the single most useful sentence you can contribute to a healthcare identity project.

Failure family two: the recovery attribute collision

This one is genuinely subtle, and it doesn’t bite during development. It bites during rollout, at scale, to the people least able to tolerate it.

Cognito refuses to let a user’s MFA delivery method be the same channel as their account recovery method. The logic is sound: if a password reset code and a second factor both land in the same inbox, an attacker with that inbox has everything, and the second factor is decoration.

The consequence is where it gets sharp. If MFA is required, a user has only an email address and no phone number, email is your account recovery channel, and they have not registered a TOTP authenticator, then that user cannot complete sign-in at all. Cognito returns an error. They are not locked out temporarily. There is no path forward from the login screen.

Now think about who that describes in a healthcare deployment. Part-time nursing staff enrolled from an HR export that had work email but no mobile number. Older patients who gave an email at registration and nothing else. Contractors. Locums. It is never a small number, and they all arrive at the help desk in the same week.

The same rule cuts the other way on password reset. A user whose preferred MFA is email cannot receive a reset code by email. If they have no phone number, self-service recovery is gone, and someone with administrative credentials has to intervene for every single one.

Three ways out, in the order I’d reach for them:

• Make TOTP the default factor. An authenticator app doesn’t consume either the email or phone attribute, so the collision never occurs. This is the one I reach for first in staff-facing apps.
• Require both email and phone_number as sign-up attributes. When a user has both, Cognito automatically routes reset codes to whichever channel isn’t their MFA factor. Clean, but it forces a phone number on people who may not want to give one.
• Configure a recovery fallback. Set account recovery to prefer email but fall back to SMS, so users who aren’t eligible for email recovery still have a route.

Whatever you choose, run a query against your existing user records for accounts missing a phone number before you flip MFA to required. Finding those users in a report is cheap. Finding them through a help desk queue is not.

Failure family three: required MFA and adaptive authentication are mutually exclusive

This trips up almost every security-conscious design I review, because both features are things a healthcare client will ask for by name.

Cognito’s threat protection includes adaptive authentication: risk scoring on each sign-in, with automated responses like requiring MFA or blocking outright. To use those automated responses in full-function mode, MFA must be set to optional in the user pool. That is a documented constraint, not a bug. The mechanism only makes sense if there’s a state where MFA is not already being demanded.

So you cannot check both boxes. Either MFA is required for everyone and adaptive responses have nothing to escalate to, or MFA is optional and risk scoring can escalate, which means low-risk sign-ins may proceed on a password alone.

For anything touching ePHI, I’d take required MFA and give up the adaptive escalation. A control that always applies beats a control that usually applies, and “usually” is a hard thing to defend in an audit.

You don’t have to give up the visibility, though. Threat protection runs in audit-only mode alongside required MFA. You still get risk signals, device fingerprinting, and compromised credential detection surfaced in CloudWatch and in the user event logs, without the enforcement conflict. That’s a genuinely good trade: enforcement from the required setting, detection from threat protection.

Two things to know before you budget for it. Threat protection lives in the Plus feature tier, and audit-only mode is billed the same as full-function. There’s no free observation period. Export those events somewhere you’ll actually look at them, whether that’s a CloudWatch Logs destination feeding your SIEM, or a platform like Datadog or Grafana Cloud if that’s where your on-call already lives.

Failure family four: the session outlives the shift

MFA protects the moment of sign-in. It does nothing about the eight hours after it. In healthcare that gap is unusually expensive, because the workstation is shared, the browser stays open, and the next person to touch the keyboard is a different human being.

Cognito’s defaults are built for consumer apps. Access and ID tokens last one hour. Refresh tokens last thirty days. Neither default is wrong in general and both are wrong for a clinical workstation.

The ranges you have to work with:

• Access token and ID token: five minutes to one day, set per app client.
• Refresh token: sixty minutes to ten years, set per app client. It cannot be shorter than the access token lifetime.

Now the part that catches people, and the reason I put this failure family on the list. If you use managed login, Cognito sets a browser session cookie valid for one hour. Setting your access token to fifteen minutes does not shorten the re-authentication window. During that hour the user’s session refreshes without presenting credentials again, because the cookie is still good. Teams configure short tokens, write “15-minute session timeout” in the security documentation, and the observable behavior is an hour.

The second trap is revocation. Revoking a token, or a global sign-out, invalidates refresh tokens. It does not invalidate an access token that has already been issued. That token stays valid until it expires, and any API accepting it will keep accepting it. So the practical blast radius of a compromised session is your access token TTL, not the moment you clicked “sign out this user.”

That’s the argument for short access tokens: it is the only lever that shortens the window between “we revoked access” and “access actually stopped.”

# Tighten an app client used by clinical staff.
# Access tokens live 15 minutes; refresh tokens expire at the end of a shift.
aws cognito-idp update-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --access-token-validity 15 
  --id-token-validity 15 
  --refresh-token-validity 9 
  --token-validity-units AccessToken=minutes,IdToken=minutes,RefreshToken=hours 
  --enable-token-revocation

Refresh token rotation is worth turning on where your client can support it. Each refresh returns a new refresh token and invalidates the old one, so a stolen refresh token is usable once at most. Two caveats that bite in practice: rotation does not extend the original lifetime, so a nine-hour refresh token still dies nine hours after sign-in no matter how often it rotated, and app clients with rotation enabled cannot use the older REFRESH_TOKEN_AUTH flow. That second one has broken more than one mobile release. Rotation requires the Essentials tier or higher.

None of this replaces workstation-level controls. Screen lock policy, OS session timeouts, and endpoint hardening tooling like the utilities from O&O Software do work your identity layer cannot do from the outside. If staff reach the admin console remotely, a managed VPN such as NordLayer or Surfshark reduces network exposure, but be honest in your documentation about what that is. It is not an authentication control, and treating it as one is exactly the reasoning auditors push back on.

Failure family five: the factor you picked is phishable

Cognito offers SMS, email, and TOTP as second factors, plus passkeys as a first factor under specific conditions. They are not equivalent, and the console presents them as though they were.

SMS is the weakest. SIM swap and SS7 interception are real, and an SMS code is trivially relayed through a phishing proxy. It has one genuine advantage worth naming: enrollment friction is near zero, and for elderly or low-technology-comfort patient populations that is not a small thing. An SMS factor that gets used beats a TOTP factor that gets abandoned. Just don’t put clinical staff behind it. There’s also an operational trap: if your account is in the Amazon SNS SMS sandbox in the relevant region, you must verify each destination number before delivery works, which turns a smooth rollout into a stalled one.

Email OTP is roughly SMS-equivalent in phishing resistance and worse if the mailbox is on the same laptop as the session you’re protecting. It needs the Essentials tier or higher and a configured Amazon SES sending identity. Useful as a fallback, weak as a primary.

TOTP is the sensible default for staff. It’s phishable, the code can be relayed in real time by an attacker-in-the-middle, but it’s immune to SIM swap and it doesn’t collide with your account recovery channel. Any standard authenticator works. If your organization already uses a password manager with built-in TOTP, such as 1Password or Bitwarden, enrollment gets meaningfully easier, at the cost of both factors living in one vault. That’s a trade-off to make deliberately rather than by accident.

Passkeys are the only phishing-resistant option here, and their integration has a shape you need to understand. A passkey cannot be a second factor on top of a password in Cognito. It can satisfy the MFA requirement on its own, as a first factor, when you set FactorConfiguration to MULTI_FACTOR_WITH_USER_VERIFICATION in the pool’s WebAuthn configuration. The user verification step, biometric or device PIN, is what makes it count as multi-factor.

That’s the path I’d push for privileged accounts: administrators, anyone with bulk record export, anyone who can change the pool configuration itself. Platform authenticators cover most staff; hardware keys like YubiKey are worth the cost for the handful of accounts whose compromise ends the conversation.

One hard constraint to plan around: you cannot set MFA to required in a pool that allows email or SMS one-time passwords as a first factor. Passwordless OTP sign-in and required MFA are incompatible. Passkeys with user verification are the documented exception.

Wiring up Cognito MFA for healthcare apps

An order of operations that avoids locking people out. Do this in a non-production pool first, with synthetic users only.

1. Accept the BAA in the AWS account through AWS Artifact, before any real user data lands in the pool. The BAA is per-account and is not inherited by new accounts.
2. Audit your user records for accounts missing phone_number or email. This is the population that will fail after you enable required MFA.
3. Decide the recovery configuration and set it explicitly, with a fallback channel, so the MFA factor and the recovery channel can never be the same for any user.
4. Enable MFA methods with the pool set to optional first. Let users enroll voluntarily. Track enrollment coverage as a number you report weekly.
5. Flip to required only when coverage is high and the stragglers have a support path. Managed login will prompt unenrolled users to set up a factor at next sign-in, but only when MFA is required.
6. Tighten token lifetimes per app client, then verify the observable behavior rather than the configuration. If you use managed login, remember the one-hour cookie.
7. Turn on threat protection in audit-only mode and stream the events out to wherever you review security signal.

The core enforcement setting, once you’re ready:

# Require MFA pool-wide, TOTP as the available method.
aws cognito-idp set-user-pool-mfa-config 
  --user-pool-id us-east-1_EXAMPLE 
  --mfa-configuration ON 
  --software-token-mfa-configuration Enabled=true

# Set a specific user's preferred factor administratively.
aws cognito-idp admin-set-user-mfa-preference 
  --user-pool-id us-east-1_EXAMPLE 
  --username clinician@example.org 
  --software-token-mfa-settings Enabled=true,PreferredMfa=true

Note that admin-set-user-mfa-preference does not reset an existing TOTP registration. To issue a new authenticator secret for a user who lost their phone, you go through AssociateSoftwareToken and VerifySoftwareToken. Build that runbook before you need it, because you will need it, and improvising an identity-proofing procedure at 2pm on a Tuesday is how bad help desk habits start.

A staging pool holding only synthetic patients doesn’t need to live inside the BAA account, and there’s no reason a test environment should carry production’s cost structure. A plain VPS from InterServer or Contabo is fine for the app tier in front of a throwaway pool, provided you are certain no real PHI can reach it. That certainty is the whole condition, and it’s worth writing down rather than assuming.

Troubleshooting

A user gets an error at sign-in and never reaches an MFA prompt

Almost always the recovery collision. Check whether the account has both email and phone_number, and compare the populated attribute against the pool’s account recovery setting. If the only available attribute is also the recovery channel, and there’s no TOTP registration, the user has no eligible factor. Register TOTP for them or add the missing attribute.

Sign-in succeeds without MFA even though the pool says required

Work backward from the user, not from the pool. Is this a federated user? Then Cognito is not in the authentication path and pool MFA does not apply. Is it their first sign-in after account creation? Cognito treats the initial verification message as the second factor on that first sign-in and prompts for factor registration afterward. Is a different app client involved with different permitted flows?

Password reset stopped working after enabling MFA

Expected behavior, not a regression. The reset code cannot go to the same channel as the user’s MFA factor. Users with only one contact attribute lose self-service recovery. Add the second attribute or configure a fallback recovery mechanism.

SMS codes never arrive

Check the Amazon SNS SMS sandbox status in the region holding your SNS resources. In the sandbox, destination numbers must be verified in SNS before delivery. Also confirm the IAM role Cognito assumes for SNS is present and correctly scoped.

Repeated MFA failures lock a user out

After five unsuccessful MFA code attempts, Cognito starts an exponential timeout lockout. It resolves on its own. Most real cases are clock drift on the user’s phone breaking TOTP validation, so check that before assuming an attack.

Tokens keep working after you revoked them

Correct and by design. Revocation kills refresh tokens; already-issued access tokens survive to their expiry. If that window is unacceptable for your risk profile, shorten the access token TTL. There is no other lever short of checking revocation on every API call.

Common mistakes

• Treating the pool-level MFA setting as coverage for federated users. It is not, and this is the gap most likely to survive to production.
• Storing PHI in Cognito custom attributes. Diagnosis codes, MRNs, and care team assignments do not belong in a directory record that gets copied into ID token claims, which then land in browser storage, proxy logs, and error reports. Keep the identity layer to identity.
• Enabling required MFA in one deploy without an enrollment ramp. You will manufacture a support incident.
• Assuming a short access token gives a short session when managed login is in play. The one-hour cookie governs re-authentication.
• Believing revocation is immediate.
• Leaving threat protection off because required MFA is on. Audit mode and required MFA coexist, and the detection signal is separately valuable.
• Enabling refresh token rotation without checking whether your client uses REFRESH_TOKEN_AUTH. That combination is not permitted and it breaks at runtime.

Best practices

• Separate app clients for patient-facing and staff-facing applications, with independently tuned token lifetimes. Staff sessions should not inherit consumer defaults.
• TOTP as the baseline factor for anyone with access to more than their own record.
• Passkeys with user verification for administrators and bulk-export roles.
• Both email and phone_number collected at enrollment where the population allows it, purely to keep recovery working.
• Access token TTL short enough that revocation means something. Fifteen minutes is a reasonable starting point for clinical apps.
• Threat protection in audit mode, with events exported and actually reviewed.
• Manage the pool in Terraform or OpenTofu so MFA settings, recovery configuration, and token lifetimes are reviewable in a diff instead of discovered in the console during an audit.
• Test the MFA path in CI against a non-production pool. Configuration drift in an identity provider is silent by nature.

FAQ

Is Amazon Cognito HIPAA compliant?

No service is HIPAA compliant by itself. Cognito is HIPAA eligible and covered under the AWS Business Associate Agreement, which means you may process PHI with it once you have accepted the BAA in that account and configured it appropriately. Compliance is a property of your whole system and your documentation, not of the service.

Does HIPAA require MFA today?

Not explicitly. The current Security Rule requires person or entity authentication and leaves the method to your risk analysis. The proposed 2025 update would make MFA an express requirement, but it remains proposed and the timeline for final action has moved more than once.

Can I use required MFA and adaptive authentication together?

Not in full-function mode. Adaptive authentication’s automated responses require MFA to be optional in the user pool. You can run threat protection in audit-only mode alongside required MFA, which keeps the detection signal without the conflict.

Which MFA factor should healthcare apps use?

TOTP as the baseline for staff, passkeys with user verification for privileged accounts. SMS is defensible for patient populations where enrollment friction genuinely determines whether MFA gets used at all, but it should not protect clinical access.

Does Cognito MFA apply to users signing in through our hospital SSO?

No. For federated users arriving via SAML or OIDC, Cognito delegates authentication entirely to the upstream identity provider and does not add a second factor. Enforcement has to happen at the IdP, and you should get that in writing.

How short should session timeouts be for a clinical application?

Set access tokens short, commonly around fifteen minutes, and refresh tokens to roughly a shift length rather than the thirty-day default. Then verify what users actually experience, because managed login’s one-hour browser cookie can keep a session alive past your token settings.

Can I store patient identifiers in Cognito user attributes?

You can, and I would avoid it. Custom attributes flow into ID token claims, which end up in browser storage, application logs, and error tracking. Keep an opaque internal identifier in the token and resolve clinical data from your application database behind its own authorization check.

Conclusion

If you remember one thing about Cognito MFA for healthcare apps, make it this: the MFA setting is a claim about local users at the moment of sign-in, and nothing else. It says nothing about your federated clinicians, nothing about the session that continues for the next eight hours, and nothing about whether the factor you chose survives a phishing proxy.

Verify enforcement by observing behavior, per user population, per app client. Not by screenshotting the console page. That screenshot is what got flagged in the pen test at the top of this post, and it was completely accurate.

Need help with Cognito and healthcare authentication?

I work with teams building and hardening authentication on AWS, particularly where a compliance obligation makes “it works” insufficient. Things I can help with:

• Auditing an existing user pool for the gaps in this post: federated coverage, recovery collisions, token lifetimes, app client sprawl
• Designing an MFA rollout that won’t generate a help desk queue, including the pre-flight query against your user records and the enrollment ramp
• Implementing passkeys with user verification for privileged accounts, alongside TOTP for general staff
• Moving user pool configuration into Terraform or OpenTofu so identity settings are reviewable and drift is visible
• Wiring threat protection events into your existing monitoring and building the alerts that are worth waking someone for
• Writing the technical evidence that maps your configuration to your risk analysis, in language an auditor accepts

If you have a user pool that isn’t behaving the way the console suggests it should, send me the output of describe-user-pool and describe-user-pool-client with the identifiers stripped, and I’ll tell you what I see.

The post Cognito MFA for Healthcare Apps: The Settings That Look Compliant but Aren’t appeared first on John Nessime.

Then somebody noticed that the person who asked couldn’t see half those opportunities in the Salesforce UI. Different role, different branch of the hierarchy. They saw them through the assistant.

Nothing errored. No 403, no exception in the logs, no alert. The integration user had access, so the API call succeeded, so the model got the records, so it summarised them. Every layer did exactly what it was configured to do. That is the failure mode that should worry you most when you build a Salesforce AI assistant with Amazon Bedrock: it is silent, it is correct-looking, and you will not find it in a load test.

This post covers what actually goes wrong when you wire a large language model into a CRM: which door AWS closed on the entry path, whose permissions your agent is really running under, how to shape the tool surface so the model can’t wander, and what happens to your org’s API budget when an agent starts looping. There’s a build walkthrough, but the build is the easy part.

Two doors closed while you were reading the old tutorial

Most of what ranks for this topic was written against an architecture you can no longer build in a fresh account. Both vendors changed the front door within a few months of each other, and the tutorials haven’t caught up. Check this before you follow anything, including this post.

On the AWS side: Bedrock Agents is now Bedrock Agents Classic

AWS renamed the original Bedrock Agents service to Bedrock Agents Classic and put it into maintenance mode. It stopped accepting new customers on 30 July 2026. This one is worth stating with the date attached, because the date is the whole point.

The mechanics matter more than the branding:

• Only CreateAgent and InvokeInlineAgent are restricted. Every other API stays available to everyone.
• Allowlisting is per AWS account, based on whether that account used Bedrock Agents in the previous twelve months. Multi-account estates get a split result: your old sandbox works, your new production account doesn’t.
• A non-allowlisted account calling CreateAgent gets an AccessDeniedException with HTTP 403, not a quota error. If you’re debugging IAM and seeing 403s, this is a plausible cause that no policy change will fix.
• There is no exception process. AWS documents this plainly.
• The model catalogue inside Classic is frozen. Bedrock itself keeps getting new models; the Classic orchestration layer does not.

The replacement is Amazon Bedrock AgentCore. It splits into a managed harness (config-driven, closest to the old experience) and code-defined agents on the AgentCore runtime (bring Strands, LangGraph, CrewAI, or your own loop). For a Salesforce assistant, the harness is usually enough, and you can move to code later without re-plumbing the tools.

Existing Classic agents keep running. There’s no announced end-of-life. But a frozen model catalogue sets a ceiling on the thing, and if your team can’t recreate the agent in a new account, your disaster recovery story has a hole in it.

On the Salesforce side: Connected Apps gave way to External Client Apps

Salesforce blocked creation of new Connected Apps through both UI and API, with package installation as the exception. External Client Apps are the replacement. Existing Connected Apps keep working and keep their OAuth flows, so this is a wall on new creation rather than a sunset.

The practical symptom is that the New Connected App button is greyed out with no explanation. Every AWS-to-Salesforce tutorial that opens with “first, create a Connected App” stops working right there.

One difference matters for agent work specifically: External Client Apps do not support the username-password OAuth flow at all. If your integration design assumed you’d stash a service account username and password somewhere and call it a day, that design is gone. Which is a good thing, and it forces the question the next section is about.

Failure family one: whose permissions is the assistant using?

This is the one from the opening, and it’s the reason most of these projects should slow down before they ship.

Salesforce’s security model is genuinely good. An OAuth token runs in a user context, and object permissions, field-level security, and sharing rules all still apply to API calls. A token with the api scope does not bypass CRUD, FLS, or sharing. People hear that and relax.

The catch is the phrase “a user context.” Which user?

If you authenticate the gateway to Salesforce with client credentials, every request from every person runs as one integration user. Salesforce enforces that user’s permissions perfectly. It has no idea a different human asked the question. You’ve built what’s sometimes called a multiplexor: one licence fronting many people, with all the real access control now living in your application layer instead of the platform’s.

At that point the sharing model is decoration. A prompt that talks the model into calling a query tool with a wider filter returns data the asker was never entitled to, and Salesforce logs it as a legitimate read by a legitimate user. There’s nothing to alert on, because nothing went wrong.

Two-legged versus three-legged, and when each is honest

AgentCore Gateway supports both client credentials (two-legged) and authorization code (three-legged) flows for reaching downstream targets, and handles ingress and egress authorisation separately. That distinction is the control you need.

• Two-legged is honest when the data is genuinely org-wide. Product catalogues, published knowledge articles, case deflection content. Nobody’s entitlement varies, so a single service identity represents the truth.
• Three-legged is the only correct answer for record data. Opportunities, accounts, contacts, custom objects with sharing rules on them. The token is minted for the actual human, Salesforce applies their sharing, and the answer is bounded by what they could have looked up themselves.

Three-legged costs you something real, and I’d rather say so than pretend it’s free. You need per-user consent, token storage, refresh handling, and a way to deal with a user whose token has expired mid-conversation. AgentCore Identity provides a token vault and handles the consent flow, including binding the authorisation URL to the session that started it so a shared link can’t be completed by someone else. It’s still more moving parts than a single secret in Secrets Manager.

Pay the cost anyway for record data. The alternative is a system whose blast radius is “everything the integration user can see,” discovered by a customer or an auditor rather than by you.

If you already run Okta, Auth0, or Ping as your identity provider, Gateway can sit in front of them for inbound auth rather than you standing up a parallel Cognito user pool. Reusing the IdP you already audit is worth more here than saving a config file.

Failure family two: the shape of the tool surface

The second mistake is giving the model a tool called run_soql that accepts an arbitrary query string. It demos beautifully. Ask anything, get anything.

You’ve now made the model your query planner and your authorisation layer at once. Every safety property of the system depends on prompt text. Guardrails help with content, not with whether a generated WHERE clause happened to be too generous.

Narrow tools are the fix, and they’re boring on purpose. get_open_cases_for_account(account_id) beats run_soql(query) because the query shape is fixed in your code, the model only supplies an identifier, and you can log and rate-limit each tool independently. When something goes wrong you’re reading a parameter, not parsing SOQL to work out what the model asked for.

AgentCore Gateway takes OpenAPI specs, Smithy models, and Lambda functions as tool input types, and exposes them to the agent as MCP tools. It also has one-click integrations for several SaaS products including Salesforce. Practically that gives you three routes:

1. The built-in Salesforce integration for the fastest path to something working. Good for a proof of concept. Look closely at which operations it exposes before you take it further, because breadth is exactly what you don’t want.
2. An OpenAPI target describing a small purpose-built API in front of Salesforce. This is the one I reach for first. You control the surface completely, and the spec doubles as the tool description the model reads.
3. Lambda targets when a tool needs to combine several Salesforce calls, or talk to something else at the same time.

MCP server targets are also supported, with the gateway aggregating tools from multiple targets into a single tools/list response. If you’re running your own MCP server as the Salesforce facade, it needs a stable HTTPS endpoint the gateway can reach; a small VPS from a provider like InterServer or Hetzner is plenty for the development and staging copies, with the production one behind whatever your standard ingress is.

Failure family three: the write path

Reads leak. Writes destroy. They deserve different treatment and usually get the same treatment, which is how you end up with three hundred opportunities silently restaged.

Start read-only and stay there longer than feels necessary. When you do add writes, the constraints that actually help:

• One record per call. No bulk update tool. If the model wants to change twenty records it makes twenty calls, each individually logged and individually rate-limited. Slow is a feature here.
• Separate the write identity. A distinct External Client App and permission set for writes means you can revoke the write path at 2am without taking the assistant offline.
• Human confirmation on anything irreversible. The AgentCore harness supports inline function tools that pause the agent and hand control back to your client code, equivalent to return-of-control. That pause is where a person says yes.
• Stamp the source. A custom field recording that a change came from the assistant, with the session and actor ID, is the difference between a ten-minute rollback and an afternoon of forensics.

Salesforce field history tracking is on a limited number of fields per object. If the field your assistant writes to isn’t tracked, you have no before-value. Check that before go-live, not after.

Failure family four: what happens under load

Salesforce meters API requests against a daily org-wide allocation that depends on your edition and licence count. Your assistant does not consume that budget the way a nightly sync does.

A single conversational turn can be several tool calls. A model that doesn’t get a clean answer will often retry with a different filter. Ten support agents having ten conversations each becomes a request pattern nobody modelled, and the org-wide limit means the integration that breaks first might be your ETL job, your marketing platform, or your billing sync. Not the assistant. That’s what makes it hard to diagnose.

Three things that help, in order of how much they buy you:

• Cap tool calls per session in your agent config, and return a clear message when the cap is hit rather than letting the loop run.
• Cache anything org-wide and slow-moving. Picklist values, record type metadata, product catalogue. These are a surprising share of calls and almost never change mid-day.
• Watch API usage as a leading indicator, not a post-mortem. Salesforce exposes remaining daily requests via the REST limits resource. Scrape it on a schedule into CloudWatch, Grafana Cloud, or whatever you already page from, and alert on the trend rather than the wall.

Wiring it up

With the design decisions settled, the build is short. The AgentCore CLI scaffolds and deploys the harness.

1. Create an External Client App in Salesforce with the narrowest OAuth scopes the tools need. Assign it via permission set, not profile.
2. Build the tool layer: an OpenAPI spec for your purpose-built endpoints, or Lambda functions for composite operations.
3. Create an AgentCore Gateway, attach the target, and configure the credential provider in AgentCore Identity for the egress side.
4. Create the harness, point it at the gateway, and deploy.

Scaffolding a harness and attaching a gateway looks like this. The --gateway-arn value is the gateway you created in step three, and the --type tells the CLI this tool is a gateway rather than one of the built-in tools such as the browser or code interpreter:

agentcore create --name sf-assistant

agentcore add tool --harness sf-assistant 
  --type agentcore_gateway --name sf-gateway 
  --gateway-arn arn:aws:bedrock-agentcore:REGION:ACCOUNT_ID:gateway/GATEWAY_ID

Then declare the model and the system prompt, and deploy. Check the current model identifiers in the Bedrock console rather than copying one from any blog post, this one included, because they change:

agentcore add harness 
  --name sf-assistant 
  --model-id YOUR_MODEL_ID 
  --system-prompt "You answer questions about CRM records using only the provided tools. Never guess a record value." 
  --tools sf-gateway

agentcore deploy

Invoking it takes a session ID, and an actor ID if you want memory scoped per user. Skipping --actor-id is the memory equivalent of the integration user problem: one shared memory store, everyone’s context in it.

agentcore invoke --harness sf-assistant 
  --session-id "$(uuidgen)" --actor-id alice 
  "Which of my open opportunities have had no activity in three weeks?"

Note “my” in that question. With three-legged auth it means something. With an integration user it means nothing, and the model will cheerfully invent a definition.

Troubleshooting

AccessDeniedException when creating an agent. If you’re calling CreateAgent or InvokeInlineAgent on the Classic API, check whether the account has prior Bedrock Agents usage before you touch IAM. No policy fixes this one. Build on AgentCore instead.

The New Connected App button is greyed out. Expected. Create an External Client App. If you have a documented technical requirement that only a Connected App satisfies, that’s a Salesforce Support request, not a setting you can toggle.

Tools appear in tools/list but calls fail. Gateway can cache tool schemas from a target without holding a working credential for calls. Discovery succeeding proves the schema synced, not that egress auth works. Look at the credential provider.

The assistant returns fewer records than the user expects. Under three-legged auth this is often correct behaviour that looks like a bug. Have the user run the same query in the Salesforce UI. If the UI shows more, check FLS on the specific fields; if it shows the same, the assistant is right and the user’s mental model is wrong.

Answers are confidently wrong about record values. Usually the model answering from training data instead of calling a tool. Tighten the system prompt to forbid answering record questions without a tool result, and check the traces to confirm which turns actually called out.

Everything works, then breaks at the same time daily. Look at org-wide API consumption before you look at your own code. Something else in the org may be eating the allocation on a schedule.

Common mistakes

• Using a System Administrator profile for the integration identity because it makes the errors go away.
• Exposing a generic SOQL tool and calling the prompt a security control.
• Testing only in a sandbox with one user, where the sharing model is invisible because there’s nothing to share.
• Treating Guardrails as an access control layer. They filter content; they don’t know who’s asking.
• Shipping writes in the first release because the read-only version felt underwhelming in the demo.
• Building on Bedrock Agents Classic in a new account without checking allowlist status first.

Best practices

• Use three-legged OAuth for anything governed by sharing rules. Reserve client credentials for genuinely org-wide data.
• Give each tool one job and a fixed query shape. The model supplies parameters, never structure.
• Separate read and write identities so you can kill one without the other.
• Scope memory per actor, not per deployment.
• Test with at least three users at different levels of the role hierarchy, and diff the answers. Identical answers to “show me my pipeline” is the signal that identity isn’t propagating.
• Keep end-to-end traces and ship them somewhere you already look, so agent behaviour lands in the same place as the rest of your telemetry.

Frequently asked questions

Can I still use Bedrock Agents for a new Salesforce project?

Only if the AWS account already used Bedrock Agents in the previous twelve months. Accounts without that history get an AccessDeniedException on CreateAgent, and AWS has said there’s no exception process. For anything new, build on AgentCore.

Do I need Agentforce to build a Salesforce AI assistant with Amazon Bedrock?

No. Agentforce is Salesforce’s own agent layer, and it can call out to AWS, which is useful when you want the assistant to live inside the Salesforce UI and inherit the Einstein Trust Layer. But an AgentCore agent calling Salesforce APIs through a gateway is a complete architecture on its own, and it’s the better fit when the assistant also needs to reach systems outside Salesforce.

Does the model see all my CRM data?

It sees whatever your tools return. That’s determined by the OAuth identity and the tool definitions, not by the model. This is why the identity decision matters more than the model choice.

Will Bedrock Guardrails stop the assistant leaking records?

No, and this is worth being blunt about. Guardrails filter content categories and can mask sensitive data patterns. They have no concept of which user is asking or which records that user is entitled to. A record correctly retrieved under an over-permissioned token passes every guardrail check, because there’s nothing objectionable about it. Guardrails are a good layer. They are not this layer.

How do I keep the assistant from burning the org’s API limit?

Cap tool calls per session, cache org-wide reference data, and monitor remaining daily requests via the REST limits resource on a schedule. The allocation is org-wide, so the symptom of overuse usually shows up in some other integration first.

Should I migrate an existing Bedrock Agents Classic setup?

There’s no deadline and no announced end-of-life, so there’s no emergency. The reasons to move are the frozen model catalogue and the fact that new accounts can’t recreate your agent. If you’re still actively developing the thing, move. If it’s stable and finished, it can wait. AWS publishes a capability mapping and a CLI import path for existing configurations.

Is an External Client App harder to set up than a Connected App?

Slightly different rather than harder. The main structural change is that developer settings and admin policies are separated, so the app definition and the per-org access controls are configured independently. The one real removal is the username-password flow, which no longer exists in the ECA model.

The one thing worth remembering

A Salesforce AI assistant with Amazon Bedrock is not really a model problem. Model quality is the part that will take care of itself, and the part you can change with one config line later.

The part that won’t take care of itself is identity. If the assistant answers as one integration user, you have quietly moved your entire access control model from a platform that enforces it well into a prompt that enforces it not at all, and nothing in your monitoring will tell you. Get that right first. Then argue about models.

Need help building or auditing this?

Most of the work on these projects is unglamorous: deciding what the agent is allowed to touch, and proving it can’t touch anything else. That’s the part I’m useful for.

• Auditing an existing Salesforce AI integration for identity propagation gaps, including testing across role hierarchy levels to find silent over-permissioning.
• Migrating a Bedrock Agents Classic deployment to AgentCore, including mapping action groups to gateway targets.
• Designing the tool surface: replacing generic query tools with narrow, logged, individually rate-limited operations.
• Setting up three-legged OAuth between AgentCore Gateway and Salesforce, including token vault and consent handling.
• Moving integrations from Connected Apps to External Client Apps without breaking existing OAuth flows.
• Instrumenting API consumption and agent traces so you see problems before an unrelated integration starts failing.

If you’ve got something concrete, send it over: a gateway target config, an agent trace, a tool schema, or just the error you’re stuck on. Easier to say something useful about a real setup than a hypothetical one.

The post Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM appeared first on John Nessime.

It is quick. You can have a working connection inside an hour. What isn’t quick is everything that surfaces three weeks later, when forty people are refreshing dashboards on a schedule nobody on your team controls and the Athena line on the invoice has quietly tripled.

Getting a BI tool to authenticate against AWS and return rows is a solved problem with decent vendor documentation. What is not well covered is what happens afterwards. This post covers how to connect BI tools to AWS data in a way that survives contact with real users: the four access patterns worth considering, why fine-grained security and cheap dashboards pull in opposite directions, how identity actually flows through a shared service account, and the cost levers in the order they pay off.

The failure that shows up on the invoice, not in the logs

Start with the shape of the problem, because it explains most of the design decisions that follow.

A dashboard is not one query. Every tile is a query. A twelve-tile dashboard opened by forty analysts, with an hourly scheduled refresh on top, is a query generator running against your data lake. Athena bills on bytes scanned, so the cost of that pattern is set almost entirely by how much data each tile touches, not by how “small” the answer looks in the browser.

Athena has a feature built for exactly this. Query result reuse lets a repeated query return the previous stored result instead of scanning again, within a maximum age you choose. If you don’t specify an age, Athena treats results older than 60 minutes as expired, and the longest age you can specify is the equivalent of seven days.

Here is the part that catches people, and it is the whole reason I wrote this post.

Result reuse is not available for tables that have Lake Formation row or column filters. It is also unavailable when the table’s S3 location is registered as a Lake Formation data location, for queries that reference more than one data catalog, for queries referencing more than twenty tables, and for federated catalogs or an external Hive metastore.

So the sequence goes like this. Dashboards are connected and cheap. A security review asks for row-level security on the customer dimension. You add Lake Formation data filters, which is exactly the right call. Everyone signs off. And the reuse rate on those tables silently drops to zero. No error. No alert. No code change. Just a larger bill the following month and a very awkward conversation about what changed.

You can still have both governance and cheap dashboards. You just can’t have both on the same table. The usual escape is to materialise pre-filtered, per-audience tables that carry no filters of their own, and grant access to those instead. That costs you a pipeline and some storage. It buys back the caching.

Pick the door before you pick the driver

Most guides start with driver installation. That’s backwards. The driver is a twenty-minute job once you’ve decided which door the BI tool walks through. There are four realistic doors, and they fail in different ways.

Athena over the AWS Glue Data Catalog

The default, and usually the right first answer. Files stay where they are, the Glue Data Catalog is the single source of truth for schema, and anything that speaks JDBC or ODBC can connect. Tableau, Power BI, Looker, Metabase, Apache Superset and Grafana all handle this without exotic configuration.

Where it wins: no cluster to size, no idle spend, and you can expose a new dataset by cataloguing it rather than by loading it anywhere.

Where it doesn’t: per-query billing punishes chatty dashboards, and interactive latency is measured in seconds rather than milliseconds. If your users expect a filter dropdown to respond instantly, they will be disappointed and they will tell you so.

One practical note that saves an afternoon: for Tableau Desktop, the Athena JDBC driver has to be dropped in a specific directory before the connector appears. On macOS that’s ~/Library/Tableau/Drivers, and on Windows it’s C:Program FilesTableauDrivers. If the analyst says “I don’t see Amazon Athena in the list,” that’s almost always why.

A Redshift datashare into a consumer warehouse

If the data already lives in Redshift, or if concurrency is the binding constraint, a datashare is the cleaner answer. The producer keeps ownership; the consumer gets live, transactionally consistent access and pays with its own compute, so a badly written dashboard can’t degrade your ETL warehouse.

-- On the producer
CREATE DATASHARE bi_share;
ALTER DATASHARE bi_share ADD SCHEMA reporting;
ALTER DATASHARE bi_share ADD TABLE reporting.fact_orders;
GRANT USAGE ON DATASHARE bi_share TO ACCOUNT '111122223333';

The schema has to be added before the tables inside it, which is a common first stumble. Cross-account sharing is a two-way handshake: the producer authorises the datashare, and an administrator in the consumer account associates it before anything is queryable. Both sides must be encrypted for cross-account sharing. Objects are read-only unless the producer explicitly grants write privileges and the consumer associates for writes, which is the behaviour you want for a BI audience.

Two things to price in. Cross-region sharing moves data and is billed accordingly. And the consumer warehouse doesn’t hold statistics on the producer’s data, so plan expectations around query planning rather than assuming parity with local tables.

The Iceberg REST catalog

If the consumer is another query engine rather than a dashboard tool, this is the interesting door. Amazon S3 Tables exposes an Iceberg REST endpoint at https://s3tables.<region>.amazonaws.com/iceberg, with your table bucket ARN as the warehouse location and SigV4 for authentication. Engines like Snowflake, Trino, StarRocks, Spark and PyIceberg can register that as an external catalog and read the tables in place.

Worth knowing: AWS positions the S3 Tables endpoint for basic read and write access against a single table bucket, and recommends the AWS Glue Iceberg REST endpoint when you need unified table management, centralised governance and fine-grained access control. If Lake Formation is already your control plane, go through Glue.

Where it doesn’t help: most BI tools don’t speak Iceberg REST. They speak SQL over JDBC. This is an engine-to-engine door, not a dashboard door, and treating it as the latter wastes a week.

A materialised extract

The unglamorous option that is frequently correct. Build a narrow, pre-aggregated table on a schedule and land it where the BI tool already keeps data: QuickSight SPICE, a Power BI import model, a Tableau extract, or a small Postgres instance. Tools like dbt handle the modelling side; Fivetran or Airbyte handle it if the source is a SaaS system rather than your own lake.

Where it wins: it is by a wide margin the cheapest and fastest option for stable reporting, because the expensive scan happens once per day instead of once per tile per user. If you self-host Metabase or Superset, a modest VPS from a provider like InterServer or Hetzner will comfortably serve a department off a nightly rollup.

Where it doesn’t: freshness. And you now own a pipeline that can fail silently, which is its own category of on-call pain.

Identity: the shared service role problem

Nearly every BI deployment starts with one IAM role for the whole tool. It works immediately, which is why it survives longer than it should.

The consequences accumulate quietly. CloudTrail records every query as the same principal, so audit answers stop being useful. Lake Formation row filters key off the principal, so if all traffic arrives as one role, your row-level security collapses to whatever that role can see. And cost attribution disappears, so when the bill jumps you cannot say which team caused it.

Three ways out, in increasing order of effort:

1. One role and one Athena workgroup per consumer group. Coarse, but honest. Finance gets a role, marketing gets a role, each with its own workgroup. Scan volume becomes attributable and you can set different limits per audience.
2. Real identity propagation. Federate the BI tool’s users through IAM Identity Center or SAML so Lake Formation grants apply to the actual person. This is the correct boundary and the most work, and driver and tool support varies. Test it with the exact tool version your users have before you promise it in a design review.
3. Push the filter into the data. Per-tenant tables, views or partitions with separate grants. Unfashionable and more to maintain, but it works with every tool, survives driver upgrades, and keeps result reuse eligible because the tables themselves carry no filters.

Option three is the one I reach for first when the audience is a handful of stable groups, and option two when the audience is the whole company.

The network path is a design decision, not a checkbox

For a SaaS BI tool, the public service endpoint with SigV4-signed requests over TLS is perfectly reasonable, and it is what most vendors document. Don’t let anyone talk you into complexity you don’t need.

You need more when the tool runs inside a VPC, or on-premises via Direct Connect or VPN, and traffic must not traverse the internet. Athena supports an interface VPC endpoint, and the Athena documentation is explicit that you should pair it with an AWS Glue endpoint, because the catalog lookup is a separate API call that will otherwise still go out to the public endpoint.

aws ec2 create-vpc-endpoint 
  --vpc-id <your-vpc-id> 
  --service-name com.amazonaws.<region>.athena 
  --vpc-endpoint-type Interface 
  --subnet-ids <subnet-id> 
  --security-group-ids <security-group-id> 
  --private-dns-enabled

With private DNS enabled, the standard athena.<region>.amazonaws.com hostname resolves to the endpoint, so no client configuration changes. Without it, VPC gives you a hostname in the form <endpoint-id>.athena.<region>.vpce.amazonaws.com, which you have to configure in the tool by hand. Attach an endpoint policy while you’re there; that’s where you restrict which principals and resources can be reached.

For Redshift, the equivalent is a Redshift-managed VPC endpoint, which requires either RA3 node types or Redshift Serverless. On provisioned clusters, cluster relocation has to be enabled first.

One cost item people forget: if the BI tool lives outside AWS, every row it pulls back is data transfer out. A live dashboard returning aggregates is cheap. A nightly extract refresh dragging millions of rows across the boundary is not. Add an S3 gateway endpoint for the query results bucket so the driver’s result download stays on the AWS network where it can.

Cost levers, in the order they pay off

Ordering matters here. Most teams reach for the guardrails first, which bounds the worst case without improving the base case.

1. Fix the storage layout. Columnar formats and sensible partitioning reduce bytes scanned on every query forever. Nothing else you do competes with this.
2. Pre-aggregate for the dashboard. A tile that reads a purpose-built rollup is a different animal from one that scans a fact table and aggregates on the fly.
3. Enable result reuse where governance allows it. Free wins on repeated queries, provided the tables aren’t disqualified by the limitations above.
4. Give every consumer its own workgroup with a per-query cutoff. This is the guardrail, and it belongs after the first three, not before them.
5. Move the highest-traffic dashboards to extract mode. Once a dashboard’s usage is predictable, importing beats querying live.

The workgroup guardrail in Terraform. The per-query cutoff cancels any single query that scans more than the limit, which is what stops an accidental SELECT * against a multi-terabyte table:

resource "aws_athena_workgroup" "bi_finance" {
  name  = "bi-finance"
  state = "ENABLED"

  configuration {
    enforce_workgroup_configuration    = true
    publish_cloudwatch_metrics_enabled = true
    bytes_scanned_cutoff_per_query     = 10737418240  # 10 GiB

    result_configuration {
      output_location = "s3://my-athena-results/bi-finance/"
    }
  }
}

Two flags there do real work. enforce_workgroup_configuration stops clients overriding the results location and encryption settings, which is what keeps a BI tool from writing results somewhere unmanaged. publish_cloudwatch_metrics_enabled is what makes the whole thing observable, and without it you’re back to reading the invoice to find out what happened.

Workgroup-wide limits are configured separately from the workgroup resource. Those let you set hourly or daily thresholds on aggregate data scanned and fire an SNS notification when a threshold is crossed, which can page someone or disable the workgroup outright.

Result reuse can be set per query through the API, which is the mechanism the drivers expose as a connection option:

aws athena start-query-execution 
  --work-group "bi-finance" 
  --query-string "SELECT region, SUM(net) FROM reporting.orders GROUP BY 1" 
  --result-reuse-configuration 
    "ResultReuseByAgeConfiguration={Enabled=true,MaxAgeInMinutes=60}"

Set the max age to match how often the underlying data actually changes. If the table is rebuilt once a night, an hour is far too conservative.

Finally, alarm on the scan metrics. Whether that lands in CloudWatch dashboards, Grafana or Datadog matters less than the alarm existing at all, owned by the team whose workgroup it is.

Troubleshooting the connections that half work

• The tool connects but lists no tables. Almost always Lake Formation rather than IAM. Check that the role has been granted on the database and tables, and note that the DESCRIBE permission on the default database is required for Athena.
• “Bytes scanned limit exceeded.” That’s your per-query cutoff doing its job. Look at what the tile is actually querying before raising the limit. Usually the partition filter is missing.
• Query works in the console, fails from the tool. Three usual suspects: a different workgroup, a results location the role can’t write to, or a role that has catalog access but not S3 access to the underlying data.
• A view works for you but not for the analyst. Lake Formation requires permissions on the tables, columns and S3 locations the view is built on. Column-level permissions are not available on views, so a view is not a substitute for column filtering.
• Costs jumped with no deployment. Check whether anyone added Lake Formation filters, and check whether partition pruning quietly stopped working after a schema or layout change.
• The first query of the morning is slow, the rest are fine. Partition metadata listing. Partition projection removes most of that cost on large tables.

Common mistakes

• Granting the BI role broad S3 read access and calling it least privilege. It sees every bucket prefix it can list, catalog or not.
• Running every consumer through the default primary workgroup, which makes attribution and per-team limits impossible.
• Exposing raw fact tables to business users and expecting them to write efficient SQL. They will drag columns into a canvas, which is what the tool is for.
• Setting dashboard refresh schedules without asking how fresh the underlying data actually is. Refreshing hourly against a nightly-loaded table burns money for identical results.
• Promising row-level security through a shared service role. It will not behave the way the security review thinks it does.
• Skipping the Glue VPC endpoint after building the Athena one, then spending a day wondering why traffic still leaves the VPC.

Best practices worth the effort

• Create the workgroup, the role and the tag before you hand over the connection string, not after the first invoice.
• Publish a curated reporting layer. The BI tool should see purpose-built tables with business column names, not your raw landing zone.
• Decide consciously whether a dataset is governed by Lake Formation filters or optimised for caching, and document which, because the two do not coexist on one table.
• Apply cost allocation tags at creation time. Retrofitting them across a live estate is thankless work.
• Version the connection setup as code. Terraform for the workgroup, roles and endpoints; a short runbook for the driver install on the client side.
• Re-test the whole path after any driver upgrade. Authentication behaviour is the thing that changes between driver major versions.

Frequently asked questions

What is the fastest way to connect Power BI or Tableau to AWS data?

Athena over the Glue Data Catalog, using the Athena JDBC or ODBC driver. Catalogue the data, create a workgroup with a results bucket, create a role with catalog and S3 read access, install the driver on the client, connect. The connection itself is the easy part; the workgroup and role are what you’ll wish you had done deliberately.

Should I use Athena or Redshift for BI dashboards?

Athena if query volume is moderate, the data already lives in S3, and users can tolerate a few seconds per query. Redshift if you need high concurrency, consistently low latency, or you already run a warehouse and can share into a separate consumer endpoint. The honest test is whether your peak is a spike or a plateau: Athena handles spikes without idle cost, Redshift handles plateaus without per-query cost.

Does Lake Formation row-level security work with any BI tool?

The filtering itself is enforced by the query engine, so it works regardless of the tool. What varies is whether the tool can present each end user as a distinct principal. Through a single shared service role, every user gets the same filtered view, which is not what row-level security is for. Verify the identity path with your actual tool before designing around it.

Why did my Athena bill increase after enabling fine-grained access control?

Because query result reuse is not supported for tables with Lake Formation row or column filters, or where the S3 location is registered as a Lake Formation data location. Queries that were being served from cached results started scanning again. Nothing errors, which is why it takes a billing cycle to notice.

Do I need PrivateLink to connect a BI tool to AWS?

No, unless your requirements say traffic must not traverse the public internet, or the tool runs inside a VPC or on-premises network. Public endpoints with signed requests over TLS are the normal path for SaaS BI tools. Add interface endpoints when there’s a stated requirement, and remember to add the Glue endpoint alongside the Athena one.

Can BI tools read Amazon S3 Tables directly?

Indirectly. S3 Tables integrates with the Glue Data Catalog, so Athena, EMR and Redshift can query them, and any BI tool that connects through Athena inherits that. The Iceberg REST endpoint is aimed at query engines and Iceberg clients rather than dashboard tools, which generally expect a SQL connection instead.

Is QuickSight a better fit than an external BI tool?

It removes the network, driver and identity plumbing described above, and SPICE gives you an in-memory cache without building an extract pipeline. Against that, your organisation may already be committed to Tableau, Power BI or Looker for reasons that have nothing to do with infrastructure, and retraining an analytics team is a genuine cost. Evaluate it on that basis rather than on the connection mechanics alone.

The one thing to take away

When you connect BI tools to AWS data, the connection is not the project. The project is deciding, deliberately, which datasets are governed by fine-grained filters and which are optimised for repeated cheap reads, then giving every consumer its own role, workgroup and cost tag before anyone gets a connection string.

Do that and a surprise bill becomes a question with an answer. Skip it and you’ll be reading CloudTrail on a Friday afternoon trying to work out which dashboard scanned four terabytes, with no way to prove it.

Need help wiring this up properly?

This is a large part of what I do. If you’re opening up AWS data to an analytics team and want it done so it still makes sense in six months, I can help with:

• Choosing between Athena, a Redshift datashare, an Iceberg REST catalog or a scheduled extract for your specific query pattern and user count
• Setting up per-consumer IAM roles, Athena workgroups, scan limits and cost allocation tags as Terraform rather than console clicks
• Designing a Lake Formation permission model that gives you row and column security without silently destroying your caching
• Building the curated reporting layer and partition strategy that makes dashboards fast and cheap at the source
• Private network paths with interface VPC endpoints, Redshift-managed endpoints and endpoint policies for tools that can’t use public endpoints
• Diagnosing an Athena or Redshift bill that grew without an obvious cause, and putting alarms in place so it doesn’t happen twice

Send me a workgroup config, a slow dashboard query, or a Cost Explorer screenshot you can’t explain, and I’ll tell you what I’d look at first.

The post How to Connect BI Tools to AWS Data Without a Runaway Bill appeared first on John Nessime.