“Salesforce handles it” is not an answer. Neither is “it’s in our VPC,” because it usually isn’t, and the parts that are in your VPC are the parts nobody has threat-modelled.

This post traces a single Agentforce request end to end and stops at each boundary it crosses. Agentforce and AWS get wired together in three or four standard patterns, and every one of them has a specific point where Salesforce’s guarantees stop and yours start. I will cover where that line sits, what changes about the security model when you cross it, and one default that surprises almost everyone.

The boundaries a single request crosses

One user turn touches more systems than the diagrams suggest. Roughly:

1. Retrieval. The agent grounds itself in CRM records, Data Cloud objects and knowledge content.
2. Reasoning. The request gets classified to a topic, then an action gets chosen. Both steps involve a model call.
3. Generation. A prompt is assembled and sent through the LLM gateway to a model provider.
4. Action. If the agent decides to do something, it calls out. That callout is where AWS usually enters the picture.

Steps one to three happen inside Salesforce’s trust boundary, and the Einstein Trust Layer applies. Step four does not, and it doesn’t. That sentence is the whole post, but the detail matters.

Retrieval: the part that’s better than you’d expect

Grounding is where Agentforce gets its context, and the architecturally interesting decision is that vector search lives inside Data Cloud rather than in a separate vector database. Structured records and unstructured content are retrieved from the same layer, which removes a whole class of sync problem you would otherwise be building.

The security-relevant property is that retrieval is permission-aware. The agent retrieves as the running user, so sharing rules and field-level security apply. If a rep cannot see an opportunity, the agent acting on their behalf should not be able to summarise it either.

Two caveats worth putting in your notes. First, this only holds while retrieval goes through Salesforce; the moment you ground from an external source, you are enforcing access control yourself. Second, if your CRM sharing model is loose, the agent inherits that looseness and makes it much easier to exploit, because a model will cheerfully summarise a thousand records a human would never have opened one by one. Agentforce does not create the over-permissioning problem, it just removes the friction that was hiding it.

Where AWS shows up here: if your analytical data lives in Redshift, an Iceberg lake or Databricks, zero-copy federation lets Data Cloud register those tables and query them in place rather than replicating them. Good for cost and duplication, and it moves an access-control decision into the federation configuration, which is a place people forget to audit.

The Trust Layer, and the default nobody expects

The Einstein Trust Layer sits between every Salesforce-originated prompt and the model. Its components are well documented: secure grounding, data masking, system policies against prompt injection, the LLM gateway, zero data retention agreements with model providers, toxicity checks on the way back, and an audit trail of the whole journey.

Here is the part that gets misreported constantly, including in a lot of otherwise decent write-ups: LLM data masking is disabled for agents. Salesforce says so plainly in its own documentation on the Trust Layer architecture and repeats it in the Trailhead module on masking. Masking remains available and configurable for embedded generative features like service replies and work summaries. For Agentforce agents, it is off.

The reason is not hard to work out once you think about what an agent does. Masking swaps real values for placeholders. An agent that has to actually perform an action needs the real record ID, the real amount, the real email address, because those become tool arguments. You cannot pass <Person_0> to a Flow and expect it to update a contact. Masking and acting are in genuine tension, and Salesforce resolved it in favour of acting.

What this changes in practice:

• Real PII reaches the model provider. The zero data retention commitment is what protects it, not de-identification. Those are different controls with different failure modes: masking is technical and verifiable, a retention agreement is contractual.
• If your compliance position was “no personal data leaves our boundary in identifiable form,” Agentforce does not satisfy it the way you may have assumed, and this is worth raising before your security review rather than during it.
• Controlling exposure moves upstream. What the model sees is now decided by what you ground on and what your sharing model permits, not by a masking filter downstream.

None of this makes the Trust Layer weak. Zero retention, the gateway, system policies and the audit trail all still apply to agent traffic, and the Trust Layer typically runs several times in a single user turn, once for topic classification and again for generation. It is a real control plane. Just know which control is doing the work.

Connecting Agentforce and AWS: four patterns

There are four ways these two systems talk, and they have different latency, failure and security characteristics.

Synchronous outbound: agent action calls AWS

A custom agent action, implemented via External Services with an OpenAPI schema, an Apex invocable method, or a Flow, calls an endpoint you own. Behind it sits API Gateway and Lambda, and often a Bedrock agent or knowledge base.

Authentication uses named credentials, which keep the secret out of your Apex and centralise the auth config. Use them; hardcoding a key into an Apex class is the version of this that ends up in a git history.

The trade-off is that you have put a network call and a second model invocation inside a conversational turn. Users notice. Budget for it, and set the timeout deliberately rather than inheriting a default.

Asynchronous outbound: Event Relay to EventBridge

Salesforce Event Relay delivers platform events from the Salesforce event bus to Amazon EventBridge without code, and EventBridge routes onward to Lambda, S3, Step Functions or a Bedrock agent.

This is the right pattern for anything that does not need to complete inside the conversation: enrichment, downstream processing, long-running work. It also fails better, because a retry is a queue concern rather than a user watching a spinner.

Inbound: AWS calls Agentforce

The Agent API lets something on the AWS side invoke an Agentforce agent, typically through EventBridge API destinations. That gives you bidirectional multi-agent flows: a Bedrock agent that detects a problem can open a case through Agentforce rather than writing to the Salesforce API directly and bypassing the agent’s own logic.

Worth being deliberate here. An inbound path means an AWS workload can trigger agent reasoning, which means anything that can inject content into that workload can influence a prompt. Treat the payload as untrusted input, because it is.

MCP: tools rather than integrations

Agentforce has a native MCP client, so an agent can connect to any MCP-compliant server without a bespoke integration, alongside a server registry intended to let admins govern which agents connect to which tools under central policy.

This is the pattern that will cause the most trouble over the next couple of years, and not because the protocol is bad. It is because MCP makes adding a tool trivial, and every tool is a new path by which text from somewhere else reaches your agent’s context and by which your agent’s decisions reach someone else’s system. The registry exists precisely so that “which tools can this agent reach” is an administered answer rather than an emergent one. Use it as a control, not a catalogue.

Keeping the traffic private

By default these callouts traverse the public internet with TLS. Salesforce Private Connect uses AWS PrivateLink to give you a private path to your own AWS endpoints instead. Whether that is required is a policy question rather than a technical one, but it is much easier to set up at design time than to retrofit after an auditor asks.

What changes the moment you cross into AWS

This is the seam, and it is where most real incidents will come from. Four things stop being true simultaneously.

• Zero retention becomes your policy, not Salesforce’s. If your Lambda logs its input at INFO, you now have customer PII in CloudWatch Logs with whatever retention that log group happens to have, which by default is forever. Your own debug logging is the single most likely leak path in this entire architecture, and it will not show up in any Salesforce audit report.
• The audit trail splits. Salesforce logs the prompt journey. AWS logs the invocation. Nothing correlates them unless you pass a request identifier through the callout and log it on both sides. Do that from day one; reconstructing an incident across two disconnected audit systems is genuinely unpleasant.
• User identity does not travel. This is the important one. Inside Salesforce the agent acts as the running user and sharing rules apply. Your Lambda runs under an IAM execution role with fixed permissions and no idea who asked. If that role can read a whole DynamoDB table, then every user of the agent can effectively read the whole table through it, regardless of their CRM permissions. That is a classic confused deputy, and it is easy to build without noticing. If the action needs to be scoped per user, pass the identity explicitly and enforce it in your own code.
• Model governance forks. A Bedrock model you invoke from your own Lambda is outside the Trust Layer. Its guardrails, logging and retention are configured in your account and are yours to get right. If you want prompt filtering or PII detection on that leg, you are building or buying it.

Note the difference between two things people conflate. Salesforce running on Hyperforce, which is built on public cloud infrastructure including AWS, means Salesforce’s own workloads run there under Salesforce’s controls. It does not mean your Salesforce data sits in your AWS account. When an agent action calls your Lambda, that is the first moment data lands in infrastructure you control and are responsible for.

Common mistakes

• Assuming masking protects agent traffic to the model. It does not; check the current documentation yourself rather than taking a blog’s word for it, including this one.
• Logging full request payloads in Lambda during development and never turning it off.
• Giving the Lambda execution role broad data access because “the agent is trusted,” when the agent is a proxy for every user who can talk to it.
• Putting a slow AWS call inside a synchronous agent action when the work did not need to be synchronous.
• Not passing a correlation identifier across the boundary, so Salesforce and CloudWatch tell two unlinked stories.
• Treating content returned from an external tool or MCP server as trusted context rather than as untrusted input that will be read by a model.
• Deploying agents against a permissive sharing model and discovering the blast radius later.

Frequently asked questions

Does Agentforce data get used to train models?

Not on the Salesforce leg. Salesforce’s zero data retention arrangements with model providers mean prompts and responses are not stored or used for training. That commitment covers traffic going through the LLM gateway. It says nothing about a model you invoke yourself from your own AWS account, where the retention and logging configuration is entirely yours.

Can I use my own Bedrock model inside Agentforce?

There are two distinct things here. Bedrock-hosted models can be brought into Salesforce through the Einstein and Bedrock integration, in which case they sit behind the Trust Layer. Separately, you can call a Bedrock model or agent from a custom agent action, in which case you are outside the Trust Layer and own the controls. Be clear which one you are building, because they have completely different compliance stories.

Sync or async for AWS integrations?

Default to async through Event Relay and EventBridge unless the agent genuinely needs the result to continue the conversation. Synchronous callouts put a network round trip and often a second inference call inside a user’s turn, and they fail in front of the user rather than into a retry queue.

How do I stop the agent seeing data a user shouldn’t?

Inside Salesforce, fix the sharing model, because retrieval respects it and inherits its flaws. Outside Salesforce, sharing rules do not apply at all: pass the user identity through the callout and enforce authorisation in your own code, or scope the integration to data that is safe for every user of that agent.

Do I need PrivateLink for this?

Technically no; callouts work over TLS on the public internet. Practically it depends on your regulatory position and whether your AWS endpoints should be publicly reachable at all. If they should not be, Private Connect over PrivateLink is the supported path and is far cheaper to design in than to add later.

What should I log, and where?

On the AWS side, log the correlation identifier, the action name, the outcome and the timing. Do not log the payload. If you need payload-level debugging, gate it behind a flag that is off in production and set an explicit short retention on the log group. Ship the operational metrics somewhere you already watch, whether that is CloudWatch alarms, Grafana or Datadog, because an agent action that silently starts failing looks like nothing at all from the Salesforce side.

The one thing to take away

Wiring up Agentforce and AWS is not hard. The patterns are documented, the connectors exist, and a working proof of concept is a day’s work.

What is hard is knowing precisely where Salesforce’s guarantees end. They end at the callout. Everything before it is covered by the Trust Layer, with the significant exception that masking is off for agents, so what reaches the model is real data protected by a retention agreement rather than by de-identification. Everything after it is yours: your retention, your audit trail, your authorisation, your logs.

Draw that line on your architecture diagram before the security review, not during it.

Need help designing or reviewing this?

I work on the AWS side of Salesforce integrations, which is usually the side nobody owns. Things I can help with:

• Threat-modelling the seam between Agentforce and your AWS account, and writing down what each side actually guarantees.
• Building agent actions on API Gateway and Lambda with named credentials, sane timeouts and no payload logging.
• Event-driven integration via Event Relay and EventBridge, including retry, dead-letter and replay design.
• Scoping IAM execution roles so an agent action cannot become a confused deputy for every user who can reach it.
• Correlated observability across Salesforce and CloudWatch so an incident can be reconstructed in one timeline.
• Private Connect and PrivateLink setup where callouts should not touch the public internet.

If you have an integration already built, send me the agent action definition and the Lambda’s IAM policy. Those two together usually tell the whole story.

The post Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin appeared first on John Nessime.

A Salesforce admin renames a custom field on Tuesday afternoon. Your connector keeps running, your Glue job keeps succeeding, the crawler happily adds a new column and quietly stops populating the old one. Three weeks later someone asks why the pipeline report has a cliff in it, and you spend an afternoon reconstructing a change nobody in engineering made and nobody logged.

This post is about using AWS Glue Data Quality to catch that class of breakage. It is aimed at people pulling from SaaS sources into S3 or a lakehouse: Salesforce, HubSpot, Zendesk, Stripe, Shopify, Zoho, an internal partner API. I will cover where to run the checks, the flattening constraint that will bite you before anything else does, which DQDL rule actually catches which SaaS failure, when dynamic rules and anomaly detection quietly do nothing, and what should happen when a rule fails.

Why SaaS data breaks in ways your own database doesn’t

When you own the source database, the schema is under change control: someone writes a migration, it gets reviewed, and if it breaks a consumer there is a commit to point at. A SaaS source has none of that. Three consequences:

• The schema is edited by people who are not engineers. A sales ops admin adding a picklist value is a production schema change with no review, no notice and no version number.
• The payload is nested. SaaS APIs return structs and arrays: line items, related objects, custom field bags. That shape matters more than you would expect, for reasons in the next section but one.
• Absence is ambiguous. A field missing from a payload might mean “not set,” “not permitted for this user,” or “we removed it.” The connector usually cannot tell you which, and neither can your table.

Glue Data Quality is a rules engine built on Deequ that evaluates declarative checks written in DQDL, the Data Quality Definition Language, against a dataset. It is well suited to the first and third problems. The second one needs work before it is useful at all.

Decide where the checks run before you write a rule

There are two places to evaluate a ruleset, and they answer different questions.

Inside the Glue ETL job

The EvaluateDataQuality transform runs mid-pipeline against a DynamicFrame. It answers “is this batch safe to load?” and it can act on the answer: stop the job, or split failing rows into a quarantine prefix. This is also the only placement where row-level results exist, which matters because “which 400 rows are bad” is a far more useful answer than “the rule failed.”

Against a Data Catalog table

You can attach a ruleset to a catalog table and run it on a schedule, independent of any job. This answers “is the table currently trustworthy?” and surfaces a quality score analysts can see before they query. The trade-off is that it is after the fact: by the time a catalog evaluation fails, the bad batch is already queryable. Analyzers are also not supported for catalog evaluations, so some dynamic-rule tooling behaves differently there.

In practice I run both. The in-job ruleset is small and strict, and exists to stop garbage landing. The catalog ruleset is broader and looser, and exists so a table’s state is visible to people who will never open Glue Studio.

The flattening problem that comes first

Here is the constraint that catches most people on SaaS data, stated plainly in the AWS documentation and easy to skim past: DQDL cannot evaluate nested or list-type columns. No structs, no arrays.

Almost every interesting SaaS payload is nested. A Salesforce opportunity carries a related account struct, a Stripe invoice carries an array of line items, a Zendesk ticket carries a custom fields array of key-value pairs. Point a ruleset at the raw landed data and the fields you most want to check are the ones the engine cannot see.

So you flatten first, with Glue’s relationalize or an explicit projection, then evaluate. That works, but it changes what your rules mean:

• Your rules now describe the projection, not the payload. If your flattening step drops a field, no rule will notice, because from DQDL’s point of view that field was never there.
• Exploding an array changes the grain. One invoice with five line items becomes five rows. Every RowCount rule you write against that table is now counting line items, and a customer who starts itemising differently will move your row count without any data being wrong.
• Custom-field bags need pivoting, not flattening. A key-value array flattened naively gives you rows, not columns, and you cannot write a ColumnValues rule against a value whose meaning depends on a sibling key column. Pivot the ones you care about into real columns and check those.

The practical rule: put your quality checks immediately after the flattening step, and treat the flattening step itself as code that needs its own test. Glue Data Quality will not cover it for you.

Matching SaaS breakage to the rule that catches it

DQDL has a long rule list and most guides just enumerate it. More useful is the other direction: what actually goes wrong with SaaS sources, and what catches each one.

A field is renamed, removed, or added

This is the Tuesday-afternoon scenario. Two rules, doing two different jobs:

Rules = [
    ColumnExists "annual_revenue__c",
    ColumnCount = 47
]

ColumnExists protects the specific fields your downstream models depend on. ColumnCount protects against everything else: a field silently added, or one you forgot you needed. The first is precise and the second is a tripwire, and you want both because a rename shows up as one column gone and one column arrived, which leaves the count unchanged.

SchemaMatch compares against a reference dataset’s schema, which is stricter and useful if you maintain a golden reference table. Bear in mind a hard column count on a SaaS object fails every time an admin adds a field, so decide whether that blocks or just warns.

A field still exists but stopped being populated

Permissions change, a workflow gets disabled, an integration user loses field-level access. The column is still in the schema and is now all nulls.

Rules = [
    IsComplete "opportunity_id",
    Completeness "lead_source" > 0.85,
    Completeness "email" > 0.60 where "record_type = 'Contact'"
]

IsComplete is the absolute version: no nulls at all, right for keys. Completeness takes a threshold, which is what you want for optional business fields where a hard rule would fail constantly. The where clause is the one people miss: SaaS objects are usually polymorphic, and a completeness threshold that makes sense for contacts is nonsense across the whole table. Scope the rule to the record type.

The sync ran short

A paginated extract stops early, a rate limit gets absorbed by retries, an incremental cursor drifts. The job succeeds and delivers less data than it should have. A static RowCount > 1000 stops catching this the moment you cross a thousand rows. Dynamic rules compare against your own history instead:

Rules = [
    RowCount > min(last(3)),
    Sum "amount" > avg(last(5)) * 0.7,
    DetectAnomalies "RowCount"
]

Analyzers = [
    RowCount,
    DistinctValuesCount "owner_id"
]

last(k) pulls the last k recorded values of that metric for the same dataset; with k greater than one you need an aggregation like avg, min or max to reduce them to one number. DetectAnomalies goes further, using forecasting over stored history to flag deviations with no threshold from you at all. That is the right tool for anything seasonal: a B2B pipeline that dips every weekend trips a naive trailing-average rule every Monday, while anomaly detection learns the shape.

Analyzers deserve their own mention: they gather a metric without asserting anything, which builds the history dynamic rules and anomaly detection need. DistinctValuesCount on an owner or account column is a strong early signal for a partial sync, because row count can look fine while the number of distinct accounts represented quietly halves.

Records arrive twice, or the key stops being a key

Replays, overlapping incremental windows and connector retries all produce duplicates, and SaaS “IDs” are less unique than vendor docs imply once you have merged sandboxes or multiple accounts.

Rules = [
    IsPrimaryKey "id",
    Uniqueness "external_reference" > 0.99
]

IsPrimaryKey asserts unique and non-null together. Uniqueness with a threshold is the softer version for fields that are mostly-but-not-perfectly unique, which describes a lot of SaaS reference fields honestly.

Objects stop agreeing with each other

Multi-object extracts rot quietly. Opportunities reference accounts that were never synced; invoice totals stop matching line item sums because one object’s extract failed and the other’s didn’t.

ReferentialIntegrity checks that values in one dataset’s column exist in another’s. RowCountMatch and AggregateMatch compare counts and summary metrics across two datasets. These catch a broken join before an analyst does, and they need the multi-frame form of the transform with the second dataset supplied as an additional source.

The sync stopped and nobody noticed

The quietest failure of all: nothing is wrong with the data, there just isn’t any new data. DataFreshness checks a timestamp column against a window. It is deliberately simple and its comparison granularity is limited, so for anything more nuanced the custom SQL rule is the escape hatch, running a query against the dataset and evaluating the result against an expression. DQDL is case sensitive and that rule type’s exact capitalisation has appeared more than one way across the docs and the visual builder, so copy it from the current DQDL reference rather than from memory.

Static, dynamic and ML: when each one lies to you

All three check styles have a mode where they pass while the data is wrong. Knowing which is which is most of the skill.

• Static rules lie by going stale. A threshold that was correct when you wrote it stops being correct as the business grows, and a rule that can no longer fail is worse than no rule because it looks like coverage.
• Dynamic rules lie when they have no history. This one is genuinely nasty. last(k) reads previously recorded values for that metric, and if there aren’t any, it falls back to a default of zero. Add RowCount > avg(last(3)) to an existing job today and it will pass trivially for the first few runs, because almost any row count beats zero. It looks like a working check from the moment you deploy it. It isn’t one until it has accumulated runs.
• Anomaly detection lies while it is warming up, and after you retrain it badly. It needs a run of history before its forecasts mean anything, and if you leave a period of genuinely bad data in the training window, it learns that the bad shape is normal. You can exclude specific statistics from the calculation in the Data Quality tab, and you should, after any incident.

So when you add a dynamic or ML-based rule, write down the date and treat it as unarmed for the first couple of weeks. Keep a crude static rule alongside it in the meantime.

What should happen when a rule fails

The transform returns a collection. The two members you care about are the row-level outcomes, which give you the original data with an evaluation result column appended per row, and the rule outcomes, which give you one row per rule with its verdict.

from awsgluedq.transforms import EvaluateDataQuality

dq_results = EvaluateDataQuality().process_rows(
    frame=flattened_opportunities,
    ruleset=ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "salesforce_opportunity",
        "enableDataQualityCloudWatchMetrics": "true",
        "enableDataQualityResultsPublishing": "true",
    },
)

row_level = SelectFromCollection.apply(
    dfc=dq_results, key="rowLevelOutcomes"
)

dataQualityEvaluationContext is the namespace your CloudWatch metrics land under. Name it after the source object rather than the job, because you will eventually want to alert per object.

From there, three response patterns:

1. Fail fast. Stop the job on failure so nothing lands. Right for structural rules: if the primary key isn’t unique, downstream is going to be wrong no matter what you do next.
2. Quarantine. Filter the row-level outcomes on the evaluation result column, write passing rows to the target and failing rows to a separate prefix. Right for value-level rules where 2% bad records shouldn’t block 98% good ones. The important discipline is that somebody has to actually look at the quarantine prefix; an unmonitored quarantine is a delete with extra steps.
3. Warn and continue. Let everything land, publish the metrics, alert on the score. Right for new rules you don’t trust yet, and for the catalog-side ruleset.

Whichever you pick, get the failure out of Glue and into wherever your team actually looks. Results and anomaly predictions can be written back to catalog tables so they are queryable with plain SQL, and CloudWatch metrics plus EventBridge events give you the hook for paging or pausing a downstream pipeline. A Grafana or Datadog dashboard fed from those metrics beats a score in a console tab nobody opens.

Troubleshooting AWS Glue Data Quality on SaaS pipelines

“Input data does not include column X”

A rule references a column the engine cannot find. On SaaS data that means one of three things: the field disappeared from the source, your flattening step renamed it, or the Data Catalog is out of sync with the underlying table. Check the catalog schema against the landed files before touching the ruleset; rerunning the crawler resolves a surprising share of these.

A rule type isn’t recognised

DQDL is case sensitive and the rule set has grown over releases, so a rule copied from an older blog post can parse fine in your head and not at all in the engine. Check the rule name against the current DQDL rule type reference, and check that the feature is available on your Glue version. Anomaly detection in particular has a version floor.

A dynamic rule never fails

Almost always the empty-history problem described above. Look at the statistics for that metric in the Data Quality tab. If there are fewer recorded points than your last(k) asks for, the rule is not doing what you think. It will start working on its own; the mistake is trusting it in the meantime.

Evaluation is slow or the job runs out of memory

Each rule triggers analyzer computation over the dataset, and a large ruleset on a wide flattened SaaS table adds up fast. The transform exposes caching and observations-scope options to tune this. Before reaching for a bigger worker type, cut the ruleset down: most contain a dozen recommended rules nobody reads the results of.

A “column reference is ambiguous” error

Classic flattening artefact. Two nested paths collapsed to the same leaf name, so datechecked now matches two things. Fix it in the projection with explicit aliases rather than trying to disambiguate inside DQDL.

Common mistakes

• Accepting the recommended ruleset wholesale. Recommendations are generated from a sample of current data, so they encode today’s accidents as tomorrow’s requirements.
• Writing rules against nested data and assuming they cover the whole payload.
• Deploying a dynamic rule and treating it as armed on day one.
• Applying whole-table completeness thresholds to a polymorphic object instead of scoping with a where clause.
• Quarantining failed rows to a prefix nobody monitors.
• Writing RowCount rules against an exploded array without noticing the grain changed.
• Failing the job on every rule, including cosmetic ones, until the team starts rerunning with checks disabled.
• Keeping the ruleset in the console instead of in version control next to the job.

Best practices

• Split rules into blocking and reporting tiers, and be honest about which is which. A rule that always fails and never stops anything trains people to ignore the tab.
• Pair ColumnExists with ColumnCount. One protects the fields you named, the other catches the drift you did not anticipate.
• Add analyzers early, even for metrics you have no rule for yet. History has to accumulate before dynamic rules and anomaly detection are worth anything, and you cannot backfill it.
• Keep DQDL in Git alongside the job definition and deploy it with Terraform or CloudFormation. Rulesets are code and should move through the same review as the pipeline they guard.
• Namespace the evaluation context per source object so CloudWatch metrics and alerts can be routed per vendor.
• Exclude incident periods from the anomaly detection training window once the incident is resolved.
• Surface results where stakeholders already look. Writing outcomes back to catalog tables makes them queryable from Athena, which beats a score in a console tab.

Frequently asked questions

How much does AWS Glue Data Quality cost?

It bills on Glue DPU-hours like any other Glue workload, so cost is a function of evaluation time and DPUs used. The dimension that surprises people is anomaly detection, which consumes additional capacity per statistic analysed, so enabling it broadly across many columns and tables is where the bill moves. Model it against the current Glue pricing page; rates change and vary by region.

Can it check data before it lands in S3?

Yes, if the check runs inside the Glue job. The transform evaluates a DynamicFrame in memory, so you can validate after extraction and before the write. That is the placement that lets you stop a bad load rather than report on one.

Does it work on nested JSON from a SaaS API?

Not directly. DQDL does not evaluate nested or list-type columns, so you have to flatten the structure first and run the ruleset against the flattened frame. Plan the flattening deliberately, because your rules only ever see what the projection exposes.

How does it compare to Great Expectations, Soda or Monte Carlo?

Glue Data Quality’s real advantage is that it is already inside the runtime and the catalog: no extra infrastructure, no separate agent, one IAM story. The disadvantages are real too. DQDL is less expressive than Python-based assertions, the nested-data limitation is a hard wall, and it only covers what Glue can read. Great Expectations gives you more expressive checks at the cost of running it yourself; Monte Carlo and similar platforms add lineage and cross-warehouse coverage Glue does not attempt. Glue-centric stack, start here. Several engines, a standalone tool fits better.

Should I use the recommended rules?

As a starting draft, yes. As a ruleset, no. Recommendations are inferred from a sample of the data as it is today, which means any existing quality problem gets encoded as the expected baseline. Read every generated rule and delete the ones you would not have written yourself.

Can I reconcile row counts against the SaaS source itself?

Not directly; it only sees datasets it can read. The usual pattern is to have the extraction step record the count the API reported, land that alongside the data as a control table, then use RowCountMatch or AggregateMatch to compare the two. That turns “did we get everything” into a rule the engine can evaluate.

The one thing worth remembering

AWS Glue Data Quality is not really a data quality product. It is a way of writing down what you believe about a dataset so a machine can check it every run, and its value is entirely determined by whether those beliefs are written precisely enough to be wrong.

For SaaS sources that means two habits. Flatten deliberately and check immediately after, because your rules only see the projection. And treat every new dynamic or ML-based rule as unarmed until it has history, because a rule that cannot fail yet looks exactly like a rule that is passing.

Need help with data quality on your Glue pipelines?

I work with teams whose SaaS pipelines run green while the numbers drift. Things I can help with:

• Writing a DQDL ruleset for a specific SaaS object that catches schema drift instead of just restating the current schema.
• Designing the flattening and pivoting step for nested payloads and custom-field bags so the checks actually cover what matters.
• Setting up quarantine and fail-fast paths in Glue ETL, with the routing and alerting that makes a quarantine prefix something people act on.
• Cross-object reconciliation between SaaS extracts and control counts recorded at extraction time.
• Tuning anomaly detection and dynamic rules, including cleaning incident periods out of the training window.
• Getting data quality results out of the console and into Grafana, CloudWatch alarms or an Athena-queryable table.

If you have a ruleset that keeps passing while the data drifts, send me the DQDL and a schema dump. That is usually enough to see where the gap is.

The post AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed appeared first on John Nessime.

Integrations very rarely fail on launch day. They fail in week six. That is roughly when the OAuth token first expires, when an admin adds a field nobody mentioned, when a second tool starts consuming the same API allowance, and when enough records have been deleted for the row counts to visibly diverge. None of those conditions exist in staging, which is why staging said yes.

So this is a Salesforce AWS integration checklist aimed at week six rather than at Tuesday. Every item has a pass test, because “we thought about that” is not the same as “we checked.”

1. Credentials and identity

• The integration has its own Salesforce user. Pass: the username is not a person’s, and nobody uses it interactively.
• That user is API-only and cannot log in through the UI. Pass: someone tried and was refused.
• Its profile grants only the objects and fields you actually read. Pass: you have the field-level permission list and can explain every entry.
• Credentials live in a secrets manager, not in environment variables or config files. Pass: the secret is retrievable by ARN and nothing in the repo contains it.
• You have rotated the credential at least once, in production, and the pipeline survived. Pass: there is a dated record of the rotation and no incident attached to it.
• You know what invalidates the refresh token. Pass: you can name the events that revoke it, including password resets and session policy changes on the integration user.

That last item is the classic week-four failure. A refresh token feels permanent right up until an admin does something routine to the user account it belongs to.

2. Network, versions and deadlines

• You know whether the compute needs to be in a VPC, and why. Pass: if it is, you can name the private resource it reaches.
• If it is in a VPC, S3 traffic goes through a gateway endpoint. Pass: the endpoint exists and the route table references it.
• The Salesforce API version in your endpoints is pinned and current. Pass: it is not a version on a retirement list.
• IAM permissions are scoped to the specific resources involved. Pass: no wildcard on S3 buckets or Secrets Manager paths.
• Login IP restrictions and connected app policies are configured deliberately. Pass: you know whether IP relaxation is on and made that choice on purpose.

The API version item has real dates attached, so it is worth stating plainly rather than leaving as a vague “keep current”. Salesforce retired versions 7.0 through 20.0 in Summer ’22 and versions 21.0 through 30.0 in Summer ’25, and has announced that 31.0 through 40.0 retire in June 2028, with the SOAP login() call retiring separately in June 2027. Retired versions return HTTP 410 on REST rather than degrading gracefully.

Usefully, Salesforce warns you first. Calls to legacy versions come back with a Warning header, so this is detectable today rather than on the morning it breaks:

# If this matches anything, you have a deadline you did not know about.
grep -i "Warning:" integration.log | sort -u

3. Correctness

• Deletes are handled, not just creates and updates. Pass: you can point at the code path and name the mechanism it uses.
• Merged records do not double-count. Pass: you have checked what happens to the losing record of a merge.
• Formula and computed fields are either recalculated downstream or documented as snapshots. Pass: there is a written list of which columns are derived.
• Re-running the same window produces the same result. Pass: you ran it twice and diffed the output.
• One real aggregate reconciles against Salesforce for a closed period. Pass: daily and monthly totals both match, not just daily.
• Timezone handling is explicit at the boundary. Pass: someone can say which timezone the date column is in without guessing.

If daily figures reconcile and monthly ones do not, that is a timezone problem at period boundaries, not missing data. Find it now, because finding it inside somebody’s board pack is a worse day.

4. Failure handling

• Retries are scoped to failures that deserve them. Pass: transient errors retry, data errors do not, and you can show which is which.
• Backoff is exponential with jitter. Pass: a fixed sleep does not appear anywhere in the retry path.
• A partial failure leaves recoverable state. Pass: the watermark advances only after data lands, not when the API call returns.
• You can replay an arbitrary historical window on demand. Pass: someone has done it, in production, with a command you could hand to a colleague.
• Rate limiting is treated as an expected condition. Pass: a 429 or a limit error produces a backoff, not an alert and a failed run.

5. Observability

• The pipeline emits a metric on successful completion. Pass: the metric exists and you can graph it.
• An alarm fires when that metric stops arriving. Pass: you disabled the schedule on purpose and the alarm went red.
• Data freshness is measured on the destination side. Pass: you can answer “how far behind are we right now” from a dashboard.
• Salesforce API allowance consumption is graphed. Pass: there is a percentage on a chart with a threshold line.
• Alerts describe symptoms and link to a runbook. Pass: the alarm description contains a URL somebody could follow at 3am.

The second item is the one people skip and it is the only one that matters. An alarm you have never seen fire is a belief, not a control.

6. Cost

• Every resource carries a cost allocation tag. Pass: Cost Explorer filtered to that tag returns a number.
• Log groups have a retention period. Pass: none of them say “Never expire”.
• The schedule matches how the output is actually used. Pass: somebody named the decision the data supports and the latency it tolerates.
• Average object size in the destination is sensible. Pass: you have looked, and it is not thousands of tiny files per partition.
• A budget alert exists on the pipeline’s tag. Pass: it has a threshold and a recipient who is not on holiday.

7. Operations and handover

• The integration has a named owner. Pass: a person, not a team inbox, and they know.
• A runbook exists covering the three most likely failures. Pass: someone who did not build it followed the runbook successfully.
• Infrastructure is defined as code. Pass: you could rebuild it in a fresh account from the repository.
• Schema drift produces a notification. Pass: adding a field in a sandbox generated an alert somewhere.
• Personal data has an access policy and a retention policy. Pass: both are written down and somebody outside the team has read them.
• You know what a Salesforce sandbox refresh does to it. Pass: you have been through one, or you know what will break when you do.

If you only do five

Nobody clears a thirty-item list before a deadline. These five catch most of what actually goes wrong:

1. Rotate the credential once before go-live. It proves the whole credential path works and it is the failure with the longest fuse.
2. Break it deliberately and confirm someone is told. Disable the schedule, watch the alarm, put it back.
3. Reconcile one real number against Salesforce for a closed period, daily and monthly.
4. Set log retention and one cost allocation tag. Five minutes, and it is the difference between a knowable bill and a mystery.
5. Write the runbook and name the owner. If it is nobody’s, it is nobody’s at 3am too.

FAQ

What actually breaks first in production?

Credentials, usually around the first expiry or the first time an admin touches the integration user. After that, schema drift and API allowance contention with a tool somebody else added.

How do I know if I’m on a retiring API version?

Look at the version string in your endpoint URLs, and search your logs for the Warning header Salesforce returns on legacy versions. Retired versions fail with an HTTP 410 rather than degrading, so there is no soft landing.

Is a staging environment enough to sign off?

No, because the conditions that break integrations barely exist there: real volume, real credential lifecycles, real deletions, and a shared API budget. Staging proves the code runs. It cannot prove the thing survives contact with an organisation.

How often should I revisit the list?

Quarterly for the alarm test and the reconciliation, annually for permissions, credentials and API versions. Put both in a calendar, because an integration that works is one nobody thinks about until it does not.

Does this apply to managed connectors too?

Most of it. You inherit retries and pagination from the vendor, but credentials, permissions, reconciliation, cost, alerting and ownership are all still yours. A managed connector removes code, not accountability.

The one thing to remember

“It works” is a statement about today. “It’s ready” is a claim about week six, and the only honest way to make it is to have tested the things that only happen later: rotate the credential, break the schedule, reconcile a number, follow the runbook.

If you cannot point at evidence for an item, it is not done. It is intended, which is a different thing, and the gap between the two is where you will spend your next quarter.

Want a second pair of eyes before go-live?

Readiness reviews are cheap compared with the incidents they prevent, and they go faster with someone who has seen the same six failures repeatedly. Work I take on:

• Pre-launch readiness review against this checklist, with evidence collected rather than assurances taken.
• Credential and permission audit: integration users, profiles, field-level security, secret storage and rotation.
• Reconciliation harness so row counts and aggregates are checked automatically rather than when someone complains.
• Alerting and runbooks, including a failure drill that proves the alerts reach a human.
• Legacy API version audits ahead of the announced retirement deadlines.
• Cost tagging, log retention and budget alerts so the first month’s bill is not a surprise.

Tell me which items on this list you have evidence for, and I will tell you which of the rest matter most for your setup.

The post It Works Isn’t Ready: A Production Checklist for Salesforce AWS Integrations appeared first on John Nessime.

That is the failure mode worth planning for. Not a jailbreak, not a model saying something clinically wrong on stage. A user typing something perfectly reasonable into a box you built, and the data ending up somewhere you never drew on the diagram. This post walks through how to build a secure AI medical assistant on AWS, organised by the boundary the data actually crosses: the input box, your own logs, the Region the inference runs in, the retrieval index, and finally the answer itself. Commands are included where they explain something. Where a value depends on your account or your counsel, I say so instead of making one up.

Two different products hiding behind one request

“An AI assistant for medical students and doctors” is two builds with two risk profiles, and conflating them is the root of most of the trouble.

• The study tool. Question banks, guideline summaries, differential drills, spaced repetition. In theory it never touches patient data. Its real risk is confident wrongness and unattributed answers, not privacy.
• The clinical assistant. Note summarisation, coding support, chart question answering. It handles PHI by design, so the whole thing has to sit inside a HIPAA-designated account from day one.

The trap is that the study tool becomes the clinical assistant without anyone shipping a release. A student rehearses a case they saw on the ward. A doctor tries the study tool on a real chart because it is the one that is already open. The moment your input box accepts free text from someone with clinical access, you should assume PHI will arrive in it. Build accordingly, or put the study tool on infrastructure where PHI arriving is survivable.

My default is to run both in the same HIPAA-designated account with the same controls, and keep only the marketing site, the docs and the waitlist form somewhere ordinary and cheap like InterServer or any commodity host, entirely outside the AWS organisation. Small blast radius beats clever separation you have to explain to an auditor.

Boundary one: PHI arrives before you decide to accept it

Start with the thing that trips up almost every first build: HIPAA-eligible and HIPAA-compliant are not the same word. AWS designating a service as HIPAA-eligible means you are permitted to process PHI with it under an executed Business Associate Addendum. It says nothing about whether your deployment is compliant. That part is entirely yours.

Three conditions have to hold together before any PHI touches a service: the service is on the AWS HIPAA Eligible Services Reference, you have an executed BAA, and the account is designated for HIPAA use. Amazon Bedrock and Amazon Comprehend Medical both appear on that list, and the BAA is accepted through AWS Artifact rather than negotiated by email. Check the reference page yourself before you commit to an architecture, because the list changes and every component in your diagram has to be on it, not just the model.

Detecting PHI at the door

Amazon Comprehend Medical has an operation specifically for finding protected health information in unstructured clinical text. You hand it free text, it returns detected entities with a type, a confidence score and character offsets.

aws comprehendmedical detect-phi 
  --region us-east-1 
  --text "Patient seen for chest pain, MRN 004512, discharged Tuesday."

The offsets are the useful part. They let you redact or tokenise in place before the text goes anywhere else, rather than throwing the whole message away and telling the user to try again.

Now the caveat that matters more than the feature. AWS states plainly that Comprehend Medical may not identify PHI in all circumstances and that it does not, on its own, meet HIPAA’s requirements for de-identification. Read that as: it is a good filter and a terrible guarantee. If your compliance story is “we strip PHI before the model sees it, so we are outside HIPAA scope,” that story does not hold. Treat detection as defence in depth inside a compliant account, not as an escape hatch from needing one.

Boundary two: your own logs are the most likely leak

This is the invisible one. Bedrock model invocation logging is disabled by default and captures the full request data, response data and metadata for every call in the account, in that Region. You turn it on for a good reason, usually because someone in security asked who prompted what and when. Then it quietly becomes the largest concentration of raw clinical text you own, and it is nowhere on the architecture diagram because it is a checkbox rather than a component.

Check whether it is on before you assume anything:

aws bedrock get-model-invocation-logging-configuration --region us-east-1

The configuration is per account per Region, so run it in every Region where anyone has ever opened the Bedrock console. Setting it deliberately looks like this:

aws bedrock put-model-invocation-logging-configuration 
  --region us-east-1 
  --logging-config '{
    "s3Config": {
      "bucketName": "med-assistant-invocation-logs",
      "keyPrefix": "bedrock/"
    },
    "textDataDeliveryEnabled": true,
    "imageDataDeliveryEnabled": false,
    "embeddingDataDeliveryEnabled": false
  }'

Each delivery flag is a separate decision about a separate category of PHI. Text is the obvious one. Image delivery matters the moment anyone uploads a photographed chart or a scan, because burned-in identifiers travel with the pixels and no text filter will ever see them. Embedding delivery is the one people leave on without thinking; vectors derived from clinical text are not a safe artefact, and they are large.

The sharpest detail is buried in the Guardrails documentation: AWS notes you can disable invocation logs if you do not want blocked content appearing as plaintext in them. Read the implication. A guardrail can refuse a prompt, mask the identifiers, and stop the model ever seeing them, and the original text still lands in your log destination. The guardrail protects the model call. It does not protect the log.

• Encrypt the log destination with a customer-managed KMS key, and keep the key policy tight enough that “everyone with S3 read” is not also “everyone with chart access”.
• Set a retention period that reflects a legal decision someone actually made, not the CloudWatch default of never expiring.
• Send logs to a separate, tightly scoped account if your organisation is large enough to have people who need dashboards but not records.
• Remember your application logs too. A framework that logs request bodies on error will do this to you long before Bedrock does.

Boundary three: where the inference actually runs

Cross-Region inference in Bedrock exists because on-demand capacity is uneven and bursts happen. It routes your request to another Region to get it served. There are two flavours and the difference is not cosmetic.

• Geographic profiles keep routing inside a defined geography such as the US or the EU. A request that starts in the EU stays in EU Regions. This is the one built for residency requirements.
• Global profiles route to supported commercial Regions worldwide for maximum throughput. AWS documents that a request can be routed to a destination Region even if you never opted that Region into your account.

To be fair to global profiles: data is not stored in the destination Region, transfer happens encrypted across the AWS network, and your invocation logs, knowledge bases and configuration all stay in the source Region. For a workload with no geographic constraint it is a genuinely good default, and it typically carries a lower per-token rate than staying in-geography. For a clinical workload with a residency commitment in a contract, it is the wrong tool, and “the prompt left the geography but was not stored there” is a sentence you do not want to be constructing during an audit.

Pin it in policy rather than trusting a config value in a repo. A Service Control Policy denying the Bedrock API outside your approved Regions closes both doors at once, because invoking a cross-Region profile requires model access in the destination Regions as well as the source:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyBedrockOutsideApprovedRegions",
      "Effect": "Deny",
      "Action": "bedrock:*",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    }
  ]
}

The trade-off is real and you should know it before you apply this. If you later adopt a geographic profile whose destination list includes a Region you denied, invocations fail with an access error that looks nothing like a routing problem. Keep the approved Region list and the profile’s destination list in the same review, not in two different tickets.

Boundary four: retrieval, and the grounding check that will not save you

A useful medical assistant is retrieval-augmented. The model alone is a fluent generalist; the value comes from grounding answers in a curated corpus, whether that is institutional guidelines, formulary rules or a licensed reference set. Bedrock Knowledge Bases will manage the ingestion and the vector store for you, or you can run your own index in Amazon OpenSearch Service.

Two things bite here.

Tenant isolation in the index. If you serve more than one hospital, department or study cohort, a shared index with a filter applied in application code is one refactor away from cross-tenant retrieval. Filters are easy to forget and impossible to notice, because a wrong answer that cites a real document looks exactly like a right one. Separate indexes per tenant, or fine-grained access control enforced below your application, cost more and fail safe.

The grounding check has a scope limit. Guardrails contextual grounding checks detect responses that are not supported by the source material or not relevant to the question. Excellent feature. The documentation states the supported use cases are summarisation, paraphrasing and question answering, and that conversational chatbot use cases are not supported. If your product is a chat interface with turn history, do not assume this check is covering you. There is also a streaming wrinkle: relevance is assessed per chunk, so an irrelevant response can reach the user before it is marked irrelevant at the end of the stream.

What you can rely on regardless of interface is applying a guardrail to arbitrary text directly, which is how you check an input before it enters your own pipeline:

import boto3

client = boto3.client("bedrock-runtime", region_name="us-east-1")

response = client.apply_guardrail(
    guardrailIdentifier=GUARDRAIL_ID,
    guardrailVersion="DRAFT",
    source="INPUT",
    content=[{"text": {"text": user_message}}],
)

if response["action"] == "GUARDRAIL_INTERVENED":
    # Stop here. Do not forward, and do not write the raw text anywhere.
    handle_blocked(response["assessments"])

Note what the comment is doing. The most common bug in this pattern is catching the intervention and then logging the offending input “for debugging”, which reintroduces exactly the leak the guardrail just prevented. Log the assessment, log a request identifier, never the text.

Sensitive information filters give you two handling modes: block, which rejects the request outright, and mask, which replaces detected entities with placeholder tags. For a study tool, mask is usually right, because a student who typed a name by reflex gets a useful answer instead of a wall. For a clinical assistant working over charts, blocking on unexpected identifiers in an input that should have arrived structured is a better signal that something upstream is wrong.

Boundary five: being confidently wrong

Everything above is about data leaving. This one is about a bad answer arriving, and for a medical audience it is the reputational failure that actually ends products.

• Cite or refuse. Return the retrieved passages alongside the answer. If retrieval returned nothing above your relevance threshold, say so rather than letting the model answer from parametric memory. A student cannot verify what they cannot see.
• Version the corpus, not just the model. When a guideline changes, you need to know which answers were generated against the old text. Store a corpus revision identifier with every logged response.
• Use denied topics deliberately. Dosing for a named patient, and anything that reads as a treatment directive rather than reference information, are reasonable things to route to a refusal with a clear explanation.
• Get regulatory advice early. Whether clinical decision support software is regulated as a medical device depends on your jurisdiction and, critically, on the claims you make about it. This is a legal question with engineering consequences, and it is much cheaper to answer before the interface exists.

A build order for a secure AI medical assistant on AWS

Sequence matters here more than in most builds, because several of these are painful to retrofit.

1. Accept the BAA through AWS Artifact and designate the account for HIPAA use. Do this before the first prototype, not before the first customer.
2. Pin Regions with a Service Control Policy, and decide the geographic-versus-global inference profile question in writing.
3. Configure invocation logging deliberately, with a customer-managed KMS key, an explicit retention period, and each data-type delivery flag chosen rather than defaulted.
4. Put the application in private subnets and reach Bedrock over VPC endpoints so PHI-bearing traffic does not traverse the public internet.
5. Build the guardrail before the prompt. Sensitive information filters, denied topics, and grounding checks where they apply.
6. Add Comprehend Medical detection in the ingestion path for anything you are storing, and at the input boundary for anything a user types.
7. Build retrieval with tenant isolation from the first index, not the second.
8. Only now write the assistant’s prompt and interface, and put a WAF such as Cloudflare or AWS WAF in front of the public endpoint.

Troubleshooting

• AccessDenied on a model that clearly works elsewhere. Usually one of three things: model access not requested in this Region, a cross-Region profile whose destination Regions your SCP denies, or an IAM policy that grants the model in the source Region only.
• No invocation logs appearing. The configuration is per Region and disabled by default. Confirm you queried the same Region the application calls, and that the delivery flag for the data type you expect is enabled.
• Guardrail passes text you expected it to catch. Sensitive information detection is probabilistic and context-dependent. Very short inputs give it little to work with. Test with realistic clinical phrasing, not single tokens, and add regex patterns for structured identifiers like MRNs that follow a house format.
• Grounding check appears to do nothing. Check your interface shape against the supported use cases before assuming it is misconfigured, and check whether streaming is masking the result until the response completes.
• Retrieval returns plausible but wrong documents. Look at chunking before you look at the model. Clinical guidelines chunked mid-table or mid-criteria retrieve badly no matter what embedding you use.

Common mistakes

• Prototyping in a personal or sandbox account and promising to migrate later. The prototype is where the first real chart gets pasted.
• Assuming the eligibility of Bedrock covers the whole stack. Every component that touches PHI needs to be eligible, including the vector store, the queue and the cache.
• Treating PHI detection as de-identification. AWS says explicitly that it is not.
• Leaving image and embedding log delivery enabled by copy-paste.
• Logging blocked prompts to debug the guardrail.
• Shipping a chat interface and citing contextual grounding as the hallucination control.
• Filtering tenants in application code over a shared index.

Best practices

• Write the data-flow diagram with logs, backups and the vector index drawn as first-class destinations. If PHI can land there, it is on the diagram.
• Define everything in Terraform or OpenTofu so the guardrail, the logging configuration and the SCP are reviewable artefacts rather than console state.
• Keep a small evaluation set of realistic clinical questions with known-good answers, and run it on every prompt or model change.
• Alarm on guardrail intervention rate. A sudden rise usually means a change upstream, not a change in users.
• Dashboard invocation counts, latency and intervention rates somewhere your on-call actually looks, whether that is CloudWatch, Grafana or Datadog.
• Scope IAM to specific model ARNs and specific guardrail identifiers. A wildcard on bedrock:InvokeModel means any model, including ones you never evaluated.
• Rehearse the breach path once. Knowing which bucket, which log group and which key you would need to reason about is worth an afternoon.

Frequently asked questions

Is Amazon Bedrock HIPAA compliant?

Bedrock is HIPAA-eligible, which is a different claim. Eligibility means AWS permits you to process PHI with the service under an executed BAA. Compliance is a property of your deployment: your account designation, encryption, access control, network isolation, logging and retention. A Bedrock workload with a signed BAA and public endpoints and unbounded plaintext logs is not compliant.

Do I need a BAA if the tool is only for medical students?

If the tool genuinely never receives PHI, HIPAA is not engaged. The practical question is whether you can guarantee that, given that your users have clinical access and a free-text box. If you cannot, get the BAA. It costs you a self-service acceptance in AWS Artifact and removes an entire category of incident.

Does AWS use my prompts to train the models?

AWS states that customer content submitted to Bedrock is not used to train the underlying foundation models or shared with model providers. That statement is the sort of thing a hospital security review will want quoted verbatim from the current AWS data protection documentation rather than from a blog, so pull the live wording when you write your assessment.

Is Comprehend Medical enough to de-identify clinical text?

No. AWS documents that it may not identify PHI in all circumstances and does not by itself meet HIPAA’s de-identification requirements. Use it as a detection layer and a redaction aid inside a compliant environment. Formal de-identification, whether by the Safe Harbor method or expert determination, is a separate exercise with its own review.

Where should the vector store live?

In the same account and Region as the rest of the workload, on a HIPAA-eligible service, encrypted with a customer-managed key, reachable only from private subnets. Embeddings derived from clinical text are not sanitised data and should not be treated as a lower-sensitivity artefact than the source.

Should I use a global or geographic inference profile?

Geographic if you have any residency commitment, contractual or regulatory. Global if you have none and want the throughput and the lower per-token rate. Decide once, document the reasoning, and enforce it with a Service Control Policy rather than a configuration constant.

Does an AI medical assistant count as a medical device?

It depends on your jurisdiction and on what you claim the software does. Software that surfaces reference information a clinician independently reviews has generally been treated differently from software that directs a clinical decision, but the boundary is fact-specific and moves. This is a question for regulatory counsel before launch, not a question for your architecture diagram.

The one thing worth remembering

A secure AI medical assistant on AWS is not mainly a model problem. Bedrock, Guardrails and Comprehend Medical are the easy part, and the documentation for them is good. The hard part is that PHI leaves through the paths you did not draw: an invocation log you enabled for good reasons, a global inference profile that routes wherever capacity exists, an index shared between tenants, a debug line added at two in the morning.

So build the boundary first and the assistant second. Get the BAA accepted, pin the Regions in policy, decide consciously what your logs are allowed to hold, and isolate retrieval per tenant before there is a second tenant. Every one of those is cheap on day one and expensive in month six.

Need a second pair of eyes on your build?

I work with teams building AI on AWS where the data is sensitive and the failure modes are quiet. Things I can help with on a project like this:

• Reviewing a Bedrock architecture against the boundaries above and telling you where PHI can actually land
• Setting up account separation, BAA scope, Region pinning with Service Control Policies, and VPC endpoint access to Bedrock
• Designing and tuning Guardrails policies, including custom regex for house identifier formats, and the block-versus-mask decision per surface
• Building the retrieval layer with per-tenant isolation, sensible clinical chunking, and citation-or-refuse behaviour
• Auditing invocation logging, KMS key policies, retention and application-level log hygiene for accidental PHI capture
• Putting the whole thing in Terraform or OpenTofu so your controls are reviewable instead of remembered

If any of that is on your plate, send me the piece you are least sure about. A redacted architecture diagram, a guardrail configuration, a logging policy, an AccessDenied you cannot explain. I would rather look at the real thing than talk in generalities.

The post Build a Secure AI Medical Assistant on AWS: The Boundaries That Actually Leak appeared first on John Nessime.

Then you check the invocation count and it is also zero, and has been since the day somebody disabled an EventBridge rule while cleaning up a different stack. The job has not run in three weeks. It never errored because it never started.

This is the shape of almost every integration monitoring failure: a healthy graph and a stopped job produce identical output. Zero errors is what success looks like and it is also what absence looks like, and if the only thing you measure is failure, the two are indistinguishable.

This is about how to monitor Salesforce integrations so that stopping is as loud as breaking. Four signals worth emitting, how to get them into CloudWatch cheaply, the alarm configuration that actually fires, and what belongs on a Grafana dashboard once you have them.

Four signals, not one

Most integration monitoring stops at errors and duration, because those come free from Lambda or your container platform. Both are worth having and neither answers the question anybody actually asks, which is “is the data right”.

The four that do:

• Liveness. Did it run at all?
• Volume. Did it move a plausible amount of data?
• Freshness. How old is the newest record on the destination side?
• Budget. How much of Salesforce’s daily API allowance have you spent?

Errors are a fifth, and the least interesting, because errors are the failure mode that already announces itself.

Liveness: the alarm that has to fire on silence

Emit a metric on every successful completion. A single count, value 1. Then alarm when it stops arriving.

The trap is in the CloudWatch defaults. TreatMissingData has four settings, and the default is missing, which sends the alarm to INSUFFICIENT_DATA when nothing arrives. That state is not ALARM. Nothing pages. Your dashboard shows a grey alarm that most people read as “fine”.

So heartbeat alarms need breaching. That much is standard advice. Here is the part that is not: even with breaching set, a heartbeat alarm can still fail to fire. CloudWatch evaluates over a range wider than your evaluation periods, and if it finds any real data point in that wider range, those override the missing ones. On a job that runs hourly, a successful run from earlier can keep the alarm quiet through several missed runs.

The robust version uses metric math to turn absence into a real zero, so there is no missing data to interpret:

aws cloudwatch put-metric-alarm 
  --alarm-name "opportunity-sync-not-running" 
  --alarm-description "No completed run in the last 90 minutes" 
  --comparison-operator LessThanThreshold 
  --threshold 1 
  --evaluation-periods 1 
  --treat-missing-data breaching 
  --alarm-actions "$SNS_TOPIC_ARN" 
  --metrics '[
    {
      "Id": "runs",
      "MetricStat": {
        "Metric": {
          "Namespace": "SalesforceSync",
          "MetricName": "RunCompleted",
          "Dimensions": [{"Name": "Integration", "Value": "opportunity-sync"}]
        },
        "Period": 5400,
        "Stat": "Sum"
      },
      "ReturnData": false
    },
    {
      "Id": "filled",
      "Expression": "FILL(runs, 0)",
      "ReturnData": true
    }
  ]'

FILL(runs, 0) substitutes a zero wherever the metric has no data point, so the alarm always has something real to compare against the threshold. The window is deliberately longer than the schedule: an hourly job gets ninety minutes, so one late run does not wake anybody.

Then do the thing everyone skips: disable the schedule in a test account and confirm the alarm actually goes red. An untested alarm is a belief, not a control.

Volume and freshness: is the data actually moving

A job can complete successfully and process nothing. A credential with the wrong field-level permissions, a filter that silently matches nothing, a watermark that got written before the data landed: all of these produce a clean run and an empty result.

Volume is the count of records read and written per run. Alarming on it is harder than liveness because the right number varies: a quiet Sunday legitimately looks like a broken Tuesday. Two approaches that work. Use CloudWatch anomaly detection, which learns the daily and weekly shape and alarms on departures from it. Or set a crude floor that only catches the catastrophic case, which is usually zero, and accept that you will not catch a fifty percent drop.

I would start with the crude floor. It is five minutes of work and catches the failure that actually happens.

Freshness is the better metric and almost nobody emits it. At the end of each run, query the destination for the newest record’s modified timestamp, subtract it from now, and publish the difference in seconds. That single number answers the business question directly: how far behind Salesforce are we right now?

It also collapses several failure modes into one signal. A stopped job, a job that runs but writes nothing, a job stuck retrying, a job silently filtered down to zero rows: all of them show up as lag climbing. If you only add one metric from this post, add this one.

Budget: watch the Salesforce allowance

Your org has a daily API allowance shared across every integration touching it. Exceeding it does not just break your sync; it breaks marketing automation, support tooling, and whatever else somebody connected two years ago.

Enforcement is initially soft, and then it is not: past a protection threshold, calls come back as 403 with REQUEST_LIMIT_EXCEEDED until the rolling window drains. By that point you are in an incident that spans several teams.

The good news is that this costs nothing to observe. Salesforce returns your current consumption on ordinary REST responses in a header, so you get it on calls you were making anyway:

# Sforce-Limit-Info: api-usage=1212/15000
#
# Free: no extra API call, which matters when the thing you are
# measuring is an API budget. Add real error handling before
# shipping this; the header is not guaranteed on every response.
raw = response.headers.get("Sforce-Limit-Info", "")
used, allowed = (int(v) for v in raw.split("api-usage=")[1].split("/"))
emit("ApiUsagePercent", used / allowed * 100)

For a fuller picture, the /services/data/vXX.X/limits endpoint returns every allocation in the org, including DailyApiRequests with its max and remaining values. It needs the View Setup and Configuration permission and the numbers lag by a few minutes. Poll it on a schedule rather than per request, and alarm on percentage consumed rather than absolute calls, so the alarm survives a licence change.

Graph consumption by integration if you can attribute it. The conversation about which team is burning the allowance goes very differently when there is a chart.

Getting the metrics in without a bill shock

You can call PutMetricData directly, and it works, and it is a synchronous API call in the hot path of your job that can fail or add latency. Custom metrics are also charged per metric per month, and a metric is every unique combination of name and dimensions, so a dimension with high cardinality gets expensive quietly.

The better default is Embedded Metric Format: write structured JSON to stdout and CloudWatch extracts the metrics from your logs. No API call, no added latency, and the log line stays queryable in Logs Insights alongside the metric.

{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [{
      "Namespace": "SalesforceSync",
      "Dimensions": [["Integration"]],
      "Metrics": [
        { "Name": "RunCompleted",     "Unit": "Count"   },
        { "Name": "RecordsWritten",   "Unit": "Count"   },
        { "Name": "SourceLagSeconds", "Unit": "Seconds" },
        { "Name": "ApiUsagePercent",  "Unit": "Percent" }
      ]
    }]
  },
  "Integration": "opportunity-sync",
  "RunId": "a41c9f",
  "RunCompleted": 1,
  "RecordsWritten": 4127,
  "SourceLagSeconds": 312,
  "ApiUsagePercent": 8.1
}

Note what is a dimension and what is not. Integration is a dimension because it has a handful of values and you want to alarm per integration. RunId is a plain field: searchable in the logs, and not a dimension, because making it one would create a new metric on every run. That distinction is the whole cost story.

Grafana on top

CloudWatch dashboards are fine and Grafana is better for this, for three reasons: you can put Salesforce metrics next to your warehouse and application metrics on one screen, the alerting is more expressive, and non-engineers will actually open it.

Add CloudWatch as a data source using an IAM role rather than access keys, scoped to cloudwatch:GetMetricData, cloudwatch:ListMetrics and the Logs Insights permissions if you want log panels. One honest cost note: Grafana queries CloudWatch through the metric data API, which is billed per metric requested, so a busy dashboard on a short refresh interval is a real line item. Set a sane refresh, avoid auto-refresh on wall displays, and use the caching in Grafana’s CloudWatch data source.

What goes on the dashboard, in order down the page:

• Freshness per integration, as a stat panel with thresholds. This is the panel people look at.
• Time since last successful run, per integration.
• Records processed, over a window long enough to show the weekly shape.
• API allowance consumed, as a percentage with a threshold line.
• Errors and duration, at the bottom, where they belong.

One dashboard, one screen, no scrolling. A dashboard nobody can read at a glance during an incident is decoration.

Alerts people don’t ignore

Decide deliberately where alerting lives. CloudWatch alarms are more reliable, because they keep working when Grafana is down, and Grafana alerts are more flexible and can span data sources. My default is CloudWatch for the small number of alerts that page someone, and Grafana for everything informational.

Three things that separate a useful alert from noise. Alarm on the symptom, not the cause: “Opportunity data is more than two hours stale” is actionable in a way “Lambda errors greater than zero” is not. Put the runbook link in the alarm description, since that field ends up in the notification and is the only documentation anybody reads at midnight. And use composite alarms to suppress the cascade, so a Salesforce outage produces one page rather than nine.

Common mistakes

• Monitoring only errors, so a stopped job looks identical to a healthy one.
• Leaving TreatMissingData at its default on a heartbeat alarm.
• Setting it to breaching and assuming that is sufficient, without handling the evaluation range.
• Never testing that an alarm fires by actually breaking something.
• No freshness metric, so nobody can answer how far behind the data is.
• Ignoring API allowance until an integration you do not own breaks.
• High-cardinality dimensions such as record ID or run ID, and the bill that follows.
• Calling PutMetricData synchronously in the job’s critical path.
• Alerting on causes rather than on user-visible symptoms.
• A dashboard that requires scrolling and interpretation during an incident.

Best practices

• Emit liveness, volume, freshness and API budget from every integration, as a standard.
• Heartbeat alarms with breaching plus FILL(), and a window longer than the schedule.
• Freshness as the headline metric, because it maps to a question the business asks.
• Embedded Metric Format rather than direct API calls.
• Low-cardinality dimensions; everything else stays a log field.
• Alarm on percentage of the API allowance, not absolute calls.
• IAM roles for the Grafana data source, and a refresh interval you have costed.
• Runbook links in alarm descriptions.
• Composite alarms to collapse cascades into one page.
• A quarterly test that breaks each integration on purpose and confirms someone gets told.

FAQ

Why didn’t my CloudWatch alarm fire when the job stopped?

Almost certainly TreatMissingData. The default sends the alarm to INSUFFICIENT_DATA, which is not ALARM and pages nobody. Set it to breaching, and wrap the metric in FILL() so there is no missing data for CloudWatch to reinterpret.

What’s the single most useful metric to add?

Freshness: how old the newest record on the destination side is. It catches stopped jobs, empty runs, stuck retries and silent filtering with one number, and it is the only one of these metrics a non-engineer can interpret.

CloudWatch dashboards or Grafana?

Grafana if you already run it, because you can put Salesforce, warehouse and application metrics on one screen. CloudWatch if you do not, because a second system to operate is not free. Either way keep the paging alarms in CloudWatch so they survive Grafana being down.

Will custom metrics be expensive?

Only if you make them so. Cost scales with unique name-and-dimension combinations, so a handful of metrics dimensioned by integration name is negligible. Adding a run ID or record ID as a dimension is how the bill grows without anyone noticing.

How do I monitor a third-party connector I can’t add code to?

Monitor the destination instead. A scheduled job that queries the target for the newest record’s timestamp and emits it as a freshness metric works regardless of what wrote the data, and it is arguably a better test because it measures the outcome rather than the process.

The one thing to remember

Absence of failure is not evidence of success. An integration that stopped produces exactly the same error graph as one working perfectly, and every default in your monitoring stack is tuned to stay quiet when data stops arriving rather than to shout about it.

So measure the thing you actually care about. Not “did it error” but “how stale is the data right now”, alarmed in a way that fires on silence, and tested by deliberately breaking it. Everything else on the dashboard is supporting evidence.

Want this built properly?

Integration monitoring tends to get added after the first silent failure, which is one failure too late. Work I take on:

• Instrumenting Salesforce integrations with liveness, volume, freshness and API budget metrics via CloudWatch.
• Auditing existing alarms for the ones that cannot fire, and fixing the missing-data handling.
• Building the Grafana dashboard and data source, including cost-aware query and refresh configuration.
• Alert design: symptom-based alarms, composite alarms to suppress cascades, runbooks attached where people will read them.
• API allowance monitoring and attribution across multiple integrations sharing one org.
• Running a failure drill so you know the alerting works before you need it.

Tell me how you would currently find out that a sync stopped, and I will tell you how long it would take.

The post Zero Errors, Zero Records: Monitoring Salesforce Integrations with CloudWatch and Grafana appeared first on John Nessime.

“EC2 – Other” is where AWS files NAT gateway charges, along with EBS, elastic IPs and assorted data transfer. Your Salesforce sync runs in a private subnet, so every byte it pulls from Salesforce and every byte it writes to S3 goes through a NAT gateway, which bills an hourly rate plus a charge per gigabyte processed. None of that appears under Lambda, or under S3, or under the tag you put on the pipeline.

That is the recurring theme when you look at the AWS cost of Salesforce data pipelines: the spend is almost never in the place you have been optimising. People tune Lambda memory settings while the bill is being driven by data movement, request counts, log ingestion and scanned bytes.

Five levers, in the order I would pull them.

First, find out where it actually goes

Every hour spent guessing is an hour not spent on the line item that matters. Tag everything the pipeline owns with a consistent cost allocation tag, activate it in the billing console, and wait for it to backfill.

Then group Cost Explorer by usage type rather than by service. Service-level grouping is what hides NAT charges inside a generic bucket; usage type is where the truth lives.

# Substitute a whole billing month. Grouping by USAGE_TYPE is the
# point: "EC2 - Other" tells you nothing, "NatGateway-Bytes" does.
aws ce get-cost-and-usage 
  --time-period Start=YYYY-MM-01,End=YYYY-MM-01 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["EC2 - Other"]}}'

Add S3 Storage Lens for the bucket. It tells you object count and average object size per prefix, which is the number that predicts your request bill and your Athena bill better than total storage does.

Lever 1: get the traffic off the NAT gateway

Usually the largest single saving available, and often a change you can make before lunch.

Start with the question nobody asks: does this function need to be in a VPC at all? A Lambda that calls the Salesforce REST API and writes to S3 touches nothing private. If it is in a VPC because that felt like the secure default, taking it out removes the NAT gateway from the path entirely and costs you nothing.

If it genuinely needs the VPC, because it also talks to RDS or something inside your network, add a gateway VPC endpoint for S3. AWS charges no hourly fee and no data processing fee for gateway endpoints, so S3 traffic simply stops flowing through NAT:

aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0

Two honest caveats. Gateway endpoints exist only for S3 and DynamoDB; everything else needs an interface endpoint, which has an hourly charge per availability zone plus a per-gigabyte charge. Still cheaper than NAT for AWS-service traffic, not free.

And the one specific to this workload: endpoints do nothing for your Salesforce traffic. Salesforce is not an AWS service, so those calls still traverse NAT. If you are pulling large volumes out of Salesforce from inside a VPC, the endpoint fixes the S3 half of the bill and leaves the other half untouched. That is the strongest argument for getting the function out of the VPC rather than papering over it.

Lever 2: fewer, bigger files

A sync running every five minutes across twelve Salesforce objects writes a lot of very small files. Storage for those is negligible. The costs are elsewhere and they compound.

• PUT requests are billed per request and cost meaningfully more than GETs. Millions of tiny writes is a request bill, not a storage bill.
• Athena opens every file it reads. Thousands of small objects per partition turn a fast scan into a slow, expensive one.
• Glue crawlers take longer over more objects, and they are billed by the second.

The fix is compaction: a scheduled job that rewrites each day’s partition into a small number of larger Parquet files and deletes the fragments. If you are on Iceberg, its compaction procedure does this for you and is the better reason to adopt it than any of the ones usually cited.

One trap worth flagging, because it is the opposite of what people expect. Lifecycle rules that move small objects to S3 Standard-IA can increase your bill: those classes have a minimum billable object size of 128 KB, so a 10 KB file is billed as 128 KB, plus a 30-day minimum duration. And S3 Intelligent-Tiering will not help either, since objects under 128 KB are never auto-tiered and stay at Frequent Access rates. Compact first, then think about storage classes. In the other order you are paying more for the privilege.

Lever 3: log less

CloudWatch Logs is billed primarily on ingestion volume, and on a high-frequency pipeline it is routinely larger than the compute it is logging. A sync that logs every record it processes, running every five minutes, generates an astonishing amount of data whose only reader is a retention policy.

• Set a retention period on every log group. The default is to keep logs forever, and log groups created automatically by Lambda inherit it. This is the single most common unclaimed saving on an AWS account.
• Log per run, not per record. One structured summary line with counts, durations and outcome tells you more during an incident than ten thousand lines of per-record chatter.
• Turn off debug logging in production, and make the level an environment variable so raising it is a deliberate act rather than a redeploy.

Check the log classes available before you settle. There is a cheaper class aimed at logs you rarely query, which suits pipeline output well, with fewer features in exchange.

Lever 4: scan less

Athena bills on bytes scanned, so its cost is a property of your data layout, not your SQL. Three things move it, in order.

Parquet rather than JSON or CSV. Columnar means a query touching four columns reads four columns. This is usually a large multiple, not a percentage.

Partitions that match how people filter. If every query has a date range, partition by date. If nobody ever filters by region, partitioning by region just creates more small files.

Partition projection instead of a crawler. Athena works partitions out from the path pattern rather than reading them from the catalog, which removes the crawler cost, removes the repair step, and means partitions are never stale:

ALTER TABLE salesforce.opportunity SET TBLPROPERTIES (
  'projection.enabled'             = 'true',
  'projection.extract_date.type'   = 'date',
  'projection.extract_date.range'  = '2024-01-01,NOW',
  'projection.extract_date.format' = 'yyyy-MM-dd',
  'storage.location.template' =
    's3://acme-lake/raw/salesforce/opportunity/extract_date=${extract_date}/'
);

Set a per-query and per-workgroup data scanned limit while you are in there. It converts a runaway SELECT * from a surprise on the bill into an error someone sees immediately.

Lever 5: run it less often

The cheapest work is work you do not do. A sync running every five minutes because five minutes felt responsive, feeding a dashboard people open once each morning, is paying for freshness nobody consumes.

Ask what decision the data supports and how quickly it needs to be right. Hourly is enough for most reporting, and dropping from five minutes to hourly cuts invocations, requests, log volume and small-file creation all at once. It is the only lever here that improves four line items with one change.

Two false economies while you are looking. Cutting Lambda memory usually costs more, not less, because the function runs proportionally longer and you also lose CPU. And moving from Athena to a provisioned engine to escape per-query pricing only pays off at genuinely high query volume; below that you have swapped a variable cost for a fixed one and called it a saving.

Common mistakes

• Optimising compute because it is the part you understand, while data movement drives the bill.
• Reading Cost Explorer grouped by service, so NAT charges stay hidden inside “EC2 – Other”.
• Running a Lambda in a VPC when nothing it touches is private.
• Adding a gateway endpoint and expecting it to help traffic bound for Salesforce.
• Millions of small objects, and treating it as a storage problem rather than a request problem.
• Lifecycle rules that move sub-128 KB objects to an infrequent access class and increase the bill.
• No retention policy on log groups.
• Logging per record on a five-minute schedule.
• JSON in the lake because it was easier to write.
• Running a Glue crawler on a schedule when partition projection would do it for nothing.
• No cost allocation tags, so nobody can attribute anything.
• Syncing far more frequently than anybody reads the output.

Best practices

• Tag every resource the pipeline owns, and group Cost Explorer by usage type.
• Keep functions out of VPCs unless they need private resources.
• Gateway endpoints for S3 and DynamoDB on every VPC that touches them.
• Compact partitions on a schedule; measure average object size, not just total bytes.
• Compact before choosing storage classes, never the other way round.
• Retention on every log group, set at creation, enforced in your IaC.
• One structured summary log line per run.
• Parquet, partitioned the way people actually filter, with partition projection.
• Per-query and per-workgroup scan limits in Athena.
• Match the schedule to the decision the data supports.
• A budget alert on the pipeline’s tag, so the next surprise arrives as a notification.

FAQ

Why is “EC2 – Other” large when I run no EC2?

It is a catch-all that includes NAT gateway hours and data processing, EBS, elastic IPs and data transfer. Filter Cost Explorer to that service and group by usage type; if you see NAT gateway bytes near the top, that is your pipeline paying to move its own data.

Should my sync Lambda be in a VPC?

Only if it needs to reach something private, such as a database in your own subnets. Calling an external API and writing to S3 does not require it, and putting it in a VPC adds NAT charges and cold start complexity for no security benefit you can articulate.

Will Intelligent-Tiering fix my small files?

No. Objects under 128 KB are never auto-tiered and stay at Frequent Access rates. Standard-IA is worse, because it has a 128 KB minimum billable size, so small objects are billed as if they were larger. Compaction is the fix; storage class is a decision for afterwards.

Glue or Lambda for the extraction?

Lambda for anything that fits in its execution limits, which covers most incremental CRM syncs, and it is cheaper for short frequent runs. Glue when you genuinely need Spark for volume or transformation. Glue billed by the second for a job that runs for ninety seconds is a poor trade.

What is the fastest saving to make?

Retention policies on log groups, then the S3 gateway endpoint. Both take minutes, neither carries risk, and between them they usually account for more than whatever you were planning to spend a week optimising.

The one thing to remember

Data pipeline bills are dominated by moving bytes, counting requests, ingesting logs and scanning storage. Compute is the part you think about and rarely the part you pay for.

So do the boring thing first: tag it, group Cost Explorer by usage type, and read the top five lines. The answer is almost always sitting there in a category you have been scrolling past, and it is usually cheaper to fix than to keep paying for.

Bill larger than it should be?

Pipeline cost work has a good ratio: a few days of investigation usually pays for itself in the first month. Work I take on:

• Cost breakdown of an existing Salesforce or CRM pipeline, by usage type, with a ranked list of what to fix and what it saves.
• Network path work: VPC endpoints, getting functions out of VPCs, and removing NAT from the data path.
• Storage layout: compaction jobs, partitioning strategy, Parquet or Iceberg conversion.
• Athena cost control: partition projection, workgroup scan limits, replacing scheduled crawlers.
• Logging and retention cleanup across accounts, which is usually the quickest win available.
• Cost allocation tagging and budget alerts so the next increase is noticed early.

Send me a month of Cost Explorer grouped by usage type and I will tell you where the money is going.

The post You Don’t Run Any EC2: Cutting the AWS Cost of Salesforce Data Pipelines appeared first on John Nessime.

Then someone points out that a nightly job has been copying a de-identified extract into that account for eight months, the de-identification script never removed the admission dates, and the account was spun up before anyone thought about the Business Associate Addendum. Nothing was misconfigured. Every control you built worked exactly as designed. And you have been out of compliance the entire time.

That is the shape of most real failures here. Not a breach, not a misconfiguration, but a mismatch between the boundary your tooling checks and the boundary your obligation actually follows. This post covers HIPAA compliance on AWS organised by those gaps: where the contract stops, where encryption stops being a control, what “six years” genuinely applies to, and which parts of the estate people forget are in scope at all.

Eligible is not compliant, and the difference is the whole job

AWS does not sell HIPAA compliance. It sells HIPAA eligible services, which is a genuinely different thing. Eligible means AWS has built the service so it can lawfully handle electronic protected health information and has agreed to cover it under a Business Associate Addendum. Compliant describes an entire system: your architecture, your key management, your access reviews, your policies, your staff, your vendors.

Under the shared responsibility model, AWS secures the infrastructure. You secure everything you build on it. Nothing about signing the BAA transfers a single obligation off your side of the line. An unencrypted RDS instance, an overly broad IAM policy or an application that logs a patient identifier into stdout is your problem in exactly the same way it would be in a rack you own.

People know this in the abstract. Where it bites is in the specifics below.

Gap one: the BAA is a contract boundary, and nothing enforces it

This is the one I would fix first, because it is invisible to every security tool you own.

The AWS BAA is self-service through AWS Artifact, at no extra cost. You can accept it for a single account, or, if you are in the management account of an AWS Organization, accept it once so that existing and future member accounts are covered. That organization-level option is the one worth using, because the per-account version quietly rots: someone creates a new account for a proof of concept, nobody repeats the Artifact step, and six months later that account is running something real.

The second half of the boundary is the HIPAA Eligible Services Reference that AWS publishes. Only services on that list may create, receive, process, maintain or transmit ePHI under the BAA. The list is long, it changes, and some entries carry carve-outs where the service is eligible but a specific feature is not. Reading a service name on the list and assuming every feature inside it is covered is the kind of mistake that only surfaces during an audit.

Here is the part worth internalising: there is no AWS control that stops you putting PHI into a non-eligible service. No API error, no Config rule out of the box, no GuardDuty finding. The eligible services list is a contractual construct. Your infrastructure has no idea it exists.

Turning a contract boundary into a technical one

The mechanism that actually helps is Service Control Policies on the organizational unit that holds your PHI accounts. SCPs set the ceiling on what any principal in those accounts can do, including the root user, so they work as a guardrail rather than a suggestion.

Start with the easy one. Pin the accounts to the Regions you have actually assessed, because data residency assumptions fall apart the moment someone launches something in a Region you never reviewed:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnapprovedRegions",
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "organizations:*",
        "route53:*",
        "cloudfront:*",
        "support:*",
        "sts:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    }
  ]
}

The NotAction list matters. Global services are backed by endpoints in specific Regions, so denying them wholesale by Region locks you out of IAM and breaks Route 53 and CloudFront. Those entries are exemptions, not an allow-list.

The harder one is restricting which services can be used at all. The same NotAction pattern works, with the services you have approved for PHI listed as the exemptions and everything else denied. It is effective and it is blunt: every new service anyone wants becomes a change request against the policy, and if you forget a dependency you find out through a failure in production. I would only reach for it on a dedicated PHI OU where the workload is well understood, not across a general-purpose organization.

Whichever route you take, write down the approved service list somewhere a human reviews on a schedule, and diff it against the AWS reference periodically. That review is itself a compliance artefact.

Gap two: encryption is a checkbox, the key policy is the control

Almost every guide to HIPAA compliance on AWS tells you to encrypt at rest and in transit. Almost none of them explain why it is worth doing properly rather than minimally, so teams enable default encryption with an AWS-managed key, watch the Config rule turn green, and move on.

The reason to care is the Breach Notification Rule. It applies to unsecured PHI, meaning PHI that has not been rendered unusable, unreadable or indecipherable through a method HHS has specified. HHS guidance points at NIST-validated encryption. If PHI is encrypted to that standard and the decryption keys were not compromised alongside it, an incident involving that data generally does not trigger the notification machinery at all. No individual letters, no HHS portal submission, no press release for a large incident.

Read that second condition again, because it is where the architecture decision lives. The safe harbour depends on the keys not being compromised with the data. If your encryption key is one an attacker inherits automatically the moment they compromise a role in the account, you have encryption but you may not have the argument.

What that means in practice

• Use customer managed KMS keys for anything holding PHI, not AWS-managed keys. Only a customer managed key gives you a key policy you can write, and only a key policy lets you deny decryption independently of the resource policy.
• Separate the key administrators from the key users. The people who can schedule deletion of a key should not be the people whose application role uses it every second.
• Use a distinct key per data domain rather than one key for the whole account. Blast radius and audit trail both improve, and you get the ability to revoke access to one dataset without touching another.
• Constrain key usage with the kms:ViaService condition so a key that exists to encrypt RDS storage cannot be used to decrypt something a role dragged into Lambda.
• Turn on key rotation and leave it on. It costs nothing operationally and it is the kind of thing an assessor asks about by reflex.

Pull the current key policy before you assume it says what you think:

aws kms get-key-policy 
  --key-id alias/phi-rds 
  --policy-name default 
  --output text

# Find storage that slipped through unencrypted
aws ec2 describe-volumes 
  --filters Name=encrypted,Values=false 
  --query 'Volumes[].{Id:VolumeId,AZ:AvailabilityZone}' 
  --output table

aws rds describe-db-instances 
  --query 'DBInstances[?StorageEncrypted==`false`].DBInstanceIdentifier' 
  --output text

The RDS query is the important one, because RDS encryption cannot be enabled in place. If that command returns anything, the fix is a snapshot, an encrypted copy of the snapshot, a restore, and a cutover. Plan for downtime or a replication strategy. This is the single most common “we will fix it later” item I see, and later gets expensive.

Also switch on EBS encryption by default in every Region you use, so the next instance somebody launches from a console wizard is not a new exception:

aws ec2 enable-ebs-encryption-by-default --region us-east-1
aws ec2 get-ebs-encryption-by-default --region us-east-1

Gap three: you have logs, but you may not have evidence

The Security Rule requires audit controls: mechanisms that record and examine activity in systems containing ePHI. It also requires you to regularly review records of information system activity. Both of those are about having and using the records.

Now the correction, because this one is repeated everywhere and it is wrong in a way that costs money. You will read that HIPAA requires six years of audit logs. It does not. The six-year requirement sits in the documentation standard, and it applies to the policies, procedures and records of actions, activities and assessments that the Security Rule requires you to keep, retained for six years from creation or from when the document was last in effect, whichever is later. There is no clause anywhere in the Security Rule that names a retention period for CloudTrail events.

What this actually means is more demanding, not less. You have to decide your log retention period, write it into a policy, justify it against your risk analysis, and then keep that policy for six years. And an assessor will hold you to the number you wrote. Setting a CloudWatch Logs retention of thirty days while your policy claims one year is a finding. Storing seven years of everything because a blog told you to, when your policy says two, is not compliance, it is just a bill.

So: pick a period you can defend, make the infrastructure match it exactly, and treat any gap between policy and configuration as a defect.

Making logs into evidence

Retention is only half of it. The other half is being able to show that the records were not altered. CloudTrail has log file validation for exactly this, and it is off unless you turn it on:

aws cloudtrail update-trail 
  --name org-phi-trail 
  --enable-log-file-validation

# Later, prove a window of logs is intact
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:us-east-1:111122223333:trail/org-phi-trail 
  --start-time "$(date -u -d '90 days ago' +%Y-%m-%dT%H:%M:%SZ)"

With validation enabled, CloudTrail writes signed digest files alongside the log files, and validate-logs checks them. The difference between “here are our logs” and “here are our logs, and here is a cryptographic check that nothing was modified or deleted” is the difference between an assertion and evidence.

Put the archive bucket in a separate account that the workload accounts cannot write to or delete from, and apply S3 Object Lock in compliance mode for the retention window you committed to. Object Lock in compliance mode cannot be shortened or bypassed by anyone, including the root user, which is exactly the property you want and exactly the property that will hurt if you set the period carelessly. Test it in governance mode first.

For the review obligation, a query interface matters more than raw storage. Athena over the CloudTrail bucket is the cheap default. If you want alerting and dashboards on top of access patterns, this is a natural place for a platform such as Grafana, Datadog or Splunk, and any of them will hold access records for you. Just remember that if those records contain PHI, that vendor needs a BAA too. See the subprocessor section below.

Gap four: backups, snapshots and the parts of scope people forget

The Security Rule’s contingency plan standard is not optional decoration. It requires a data backup plan, a disaster recovery plan and an emergency mode operation plan, plus testing and revision procedures. Most teams have the backups. Far fewer have the tested restore, and the tested restore is the part that gets asked about.

Three things routinely go wrong here.

1. Copies leave the boundary. A cross-Region snapshot copy lands in a Region you did not assess. A cross-account copy for the DR account lands somewhere outside the OU your SCPs protect. The data is still PHI. The controls did not travel with it.
2. Re-encryption changes the key, not just the copy. Copying an encrypted snapshot to another account requires a key the destination can use. It is easy to end up with a shared or less restrictive key protecting your backups than protects production, which inverts the risk model.
3. The restore is never rehearsed. A backup you have never restored is a hypothesis. Schedule a restore into an isolated account, record the elapsed time, and file the result. That record is your evidence for the testing requirement, and it is the single easiest compliance artefact to produce for free.

While you are inventorying, remember the places PHI ends up without anyone deciding it should: application logs that include request bodies, database slow query logs capturing parameter values, support tickets with screenshots attached, CSV extracts in an analyst’s bucket, and non-production environments seeded from a production dump. That last one is the classic. If your staging database is a copy of production, staging is in scope, and staging is almost never built to the same standard.

Gap five: the business associate chain does not stop at AWS

Your BAA with AWS covers AWS. It covers nothing else in your stack.

Every vendor that can create, receive, maintain or transmit PHI on your behalf is a business associate and needs an agreement. In a typical AWS estate that means the error tracker holding stack traces, the log aggregation platform, the APM tool, the transactional email provider, the customer support desk, the CI system if it ever touches a production dataset, and any AI or analytics service you have wired in.

Build the inventory as a table with three columns: vendor, what PHI it can see, and whether a signed agreement exists. The third column is usually where the surprises are. Some vendors sign readily, some only on higher-priced tiers, and some decline entirely, at which point you have an architecture decision rather than a procurement one.

One structural move that reduces this surface considerably: keep everything that does not need PHI out of the PHI accounts entirely. Your marketing site, your docs, your status page and your public API gateway for non-clinical traffic do not belong in a regulated account. Running them on ordinary infrastructure, whether that is a separate AWS account, a straightforward VPS from a host like InterServer, or a static site behind Cloudflare, shrinks the estate you have to assess, evidence and defend. Fewer things in scope is the cheapest compliance win available.

For tracking the paperwork side, compliance automation platforms such as Vanta, Drata or Secureframe pull evidence from AWS on a schedule and keep the vendor register current. They are genuinely useful for the collection and reminder burden. They do not design your architecture, and I have seen teams treat a green dashboard in one of those tools as though it were an assessment. It is not. It is a checklist that knows what you told it.

What is changing, and why “addressable” is a bad thing to build on

Since it was adopted, the Security Rule has split implementation specifications into required and addressable. Addressable never meant optional. It meant you assess whether the specification is reasonable and appropriate, and if not, you implement an equivalent alternative or document why neither is necessary. In practice, a lot of organisations turned the documented justification into the deliverable and skipped the control.

HHS published a Notice of Proposed Rulemaking in the Federal Register in January 2025 that would remove that distinction, making implementation specifications required with limited exceptions, and would explicitly require encryption of ePHI at rest and in transit and multi-factor authentication, again with limited exceptions. The comment period closed in March 2025.

Be precise about the status, because a lot of vendor content is not: this is a proposed rule and it is not final. The expected timeline for final action has slipped more than once, and the requirements could still change or be withdrawn. Nobody should be telling you a compliance deadline as though it were settled.

What is worth taking from it is the direction of travel. If your current position depends on having documented that encryption or MFA was not reasonable and appropriate, that position is fragile regardless of what the final rule says. On AWS specifically, encryption at rest and MFA are both cheap and both already best practice. Building the architecture on an addressable deferral is an unforced risk.

Troubleshooting the findings you will actually hit

“An assessor asked which accounts are in BAA scope and nobody could answer”

Sign in to AWS Artifact from the management account and check the organization agreements tab to see whether the BAA was accepted at the organization level or per account. If it is per account, list your accounts, work out which hold PHI, and confirm each one individually. Then move to the organization-level agreement so this question has one answer forever.

“Config says the bucket is encrypted but we cannot prove who read the objects”

Bucket encryption and object-level access logging are unrelated. CloudTrail management events do not record S3 object reads by default. You need CloudTrail data events for that bucket, or S3 server access logging, and both cost money proportional to request volume. Enable data events selectively on the buckets that hold PHI rather than account-wide.

“We enabled an SCP and production broke”

Almost always a Region deny catching a global service endpoint, or a service allow-list missing a dependency the workload calls indirectly. Check CloudTrail for AccessDenied events with an explicit deny from an SCP, and look at the service name in the event rather than the one you expected. Attach new SCPs to a test OU with a representative workload before the PHI OU.

“Snapshot copy to the DR account fails with a KMS error”

The destination account cannot use the source key. The source key policy has to grant the destination principal permission to use it, and the copy has to specify a key the destination can decrypt with. Fix it by granting explicitly on a key you control, not by falling back to an AWS-managed key, which is the tempting shortcut and gives up the key policy control you needed.

“CloudWatch Logs retention was never set”

New log groups default to never expiring, which is both a cost problem and a policy mismatch. Audit them with aws logs describe-log-groups and look for groups with no retentionInDays value, then set the period your policy specifies with aws logs put-retention-policy.

Common mistakes

• Treating the signed BAA as the finish line rather than the prerequisite. It is the thing you need before the first byte of PHI arrives, not evidence that anything is configured correctly.
• Assuming a service is fully eligible because its name appears on the list, without reading the feature-level carve-outs next to it.
• Quoting “six years” as a log retention requirement, then either overspending on storage or writing a policy that contradicts the actual configuration.
• Using AWS-managed KMS keys for PHI, which leaves no key policy to write and no independent revocation path.
• Seeding staging or test environments from production data and then holding those environments to a lower standard.
• Forgetting that the risk analysis is a required, recurring, documented activity, not a one-off spreadsheet from the year you launched.
• Signing a BAA with AWS and none of the ten other vendors that can see the same data.
• Letting a compliance automation dashboard stand in for an architecture review.

Best practices worth the effort

• Isolate PHI in its own accounts and its own OU. Account boundaries are the strongest isolation AWS offers, and they make the scope question answerable in one sentence.
• Accept the BAA at the organization level. It removes an ongoing manual step that fails silently.
• Customer managed keys, one per data domain, with split admin and usage roles. This is what makes the breach safe harbour argument defensible rather than theoretical.
• Write the retention period down first, configure second. Policy and infrastructure should agree exactly, in both directions.
• Ship audit logs to a separate account with Object Lock and CloudTrail validation enabled. Immutability and integrity are what turn logs into evidence.
• Keep PHI out of everything that does not need it. De-identify early, tokenise where you can, and route non-clinical traffic through infrastructure that is not in scope.
• Define everything in Terraform or OpenTofu. A reviewable, version-controlled definition of your controls is worth more to an assessor than any screenshot, and it stops drift being invisible.
• Rehearse the restore and the breach response. Both are required, both are tested by asking for the record, and both are cheap to evidence if you actually do them.

Frequently asked questions

Is AWS HIPAA compliant?

Not on its own, and the phrasing is the problem. AWS offers HIPAA-eligible services and will sign a Business Associate Addendum, which means you can build a compliant system on it. Compliance is a property of your whole environment, including configuration, policies, vendors and staff. No provider can sell it to you as a finished product.

How do I sign a BAA with AWS?

Through AWS Artifact in the console. It is self-service and there is no additional charge. Accept it for an individual account under account agreements, or from the management account of an AWS Organization under organization agreements so all current and future member accounts are covered. It should be accepted by someone with authority to bind your organisation, and it must be in place before any PHI reaches AWS.

Does HIPAA require six years of CloudTrail logs?

No. The six-year requirement is a documentation retention rule covering the policies, procedures and records the Security Rule requires you to maintain, kept for six years from creation or from when they were last in effect. The audit controls standard requires the mechanism to record and examine activity but does not name a retention period for the logs themselves. You set that period in your own policy, justify it, and make the configuration match.

Which AWS services can I use with PHI?

Only those on the AWS HIPAA Eligible Services Reference, and only in accounts covered by your BAA. Check the list before adopting anything new, read the feature-level exclusions noted against individual services, and re-check periodically because entries are added over time. Nothing in AWS will stop you using a non-eligible service with PHI, so this has to be an explicit process on your side.

If encrypted PHI is exposed, do I still have to report a breach?

Generally no, provided the encryption meets the standard in HHS guidance and the decryption keys were not compromised along with the data. The Breach Notification Rule applies to unsecured PHI, and properly encrypted data does not meet that definition. This is why key management, not just enabling encryption, is the part that determines whether the protection is real. You still document the incident and the assessment.

Are the new HIPAA Security Rule requirements in force?

Not at the time of writing. The proposals to make all implementation specifications required and to mandate encryption and multi-factor authentication came from a Notice of Proposed Rulemaking published in January 2025. The comment period has closed, but no final rule has been issued and the timeline has moved. Treat any specific compliance deadline you see quoted with suspicion and check the current status directly.

Is a HIPAA-compliant AWS environment expensive to run?

The controls themselves are mostly cheap. KMS keys, CloudTrail validation, Config rules and account separation cost very little. The real costs are log storage volume, CloudTrail data events on busy buckets, running non-production environments to production standard, and staff time on risk analysis and evidence collection. Reducing what is in scope is the most effective cost lever, which is another reason to keep non-clinical workloads out of the regulated accounts.

The one thing to take away

HIPAA compliance on AWS fails at the seams, not at the controls. Your encryption works. Your IAM policies are tight. What goes wrong is that the obligation follows the data into an account nobody added to the BAA, a Region nobody assessed, a staging database seeded from production, a vendor nobody signed an agreement with, or a retention period nobody wrote down.

So build the boundary technically rather than trusting it contractually. Isolate PHI into its own accounts, wrap those accounts in guardrails that make the contract boundary enforceable, own your keys so the encryption means something legally, and make your logs provable rather than merely present. Then write the whole thing down, because in this domain the documentation genuinely is part of the control.

None of that is exotic engineering. It is ordinary AWS work applied to a boundary that no dashboard draws for you.

Working on a healthcare workload on AWS?

This is the kind of work I do. If you are building or inheriting a PHI environment on AWS, I can help with:

• Scope and boundary review: mapping which accounts, Regions, services and vendors actually touch PHI, and finding the ones nobody knew about.
• Account and OU design with enforceable guardrails: SCPs, organization-level BAA coverage, and Region and service restrictions that hold without breaking your workloads.
• KMS key architecture: customer managed keys per data domain, split administration and usage, and key policies written so the breach safe harbour argument stands up.
• Audit logging that produces evidence: centralised CloudTrail with log file validation, an isolated archive account with Object Lock, and retention that matches your written policy exactly.
• Backup, restore and contingency testing: encrypted cross-account copies that stay inside your boundary, plus rehearsed restores documented as compliance artefacts.
• Terraform or OpenTofu modules for the whole control set, so your posture is reviewable, repeatable and does not drift between audits.

If you would rather start with something concrete than a discovery call, send me a redacted account structure, an SCP that is causing trouble, or the output of a Config or Security Hub run, and I will tell you what I would look at first.

The post HIPAA Compliance on AWS: The Gaps That Pass Every Security Check appeared first on John Nessime.