You look at the CRM reporting tab. It cannot do that. So now you are picking a platform, and within about ten minutes someone will say “we should build a data lake,” and someone else will say “just put it in the warehouse,” and the deciding argument will be about how much data there is.

That argument is almost always the wrong one. This post covers data lake vs data warehouse specifically for CRM analytics: why volume rarely decides it, what actually does, a decision procedure you can run in an afternoon, and where zero-ETL has changed the maths recently enough that older advice is stale.

Start with the number, then stop using it

Go and count your rows. Not estimate. Count.

A mid-sized company’s entire Salesforce org is usually a few million rows across the objects anyone actually reports on: accounts, contacts, opportunities, activities, cases. Add HubSpot, Zendesk and a billing system and you are still typically in the low tens of gigabytes once it is written as compressed Parquet. Activity and event history is the one object that can genuinely run large, and it is worth checking separately.

Numbers at that scale do not stress either architecture. A data warehouse will not break a sweat. A lake will not break a sweat. The lake-versus-warehouse debate as it is usually framed, the one about petabytes and elasticity and separating storage from compute, is a debate about a scale problem you do not have.

So count the rows, confirm you are small, and then stop using volume as an argument. Everything below is about what decides it instead.

Data lake vs data warehouse: the three options, honestly

Warehouse only

Load CRM objects straight into Redshift, Snowflake, BigQuery or Postgres. Model with SQL. Point the BI tool at it.

Where it wins: it is the fastest path to an analyst being productive, and that is not a small thing. Query performance on joins is better than a lake gives you without tuning. Access control is mature and row-level. Every BI tool connects natively. The people who will maintain this already know SQL and probably do not know Spark.

Where it doesn’t: semi-structured data is awkward, and CRM payloads are full of it. Custom field bags, JSON blobs, email bodies, call transcripts. You can store JSON in a modern warehouse but you will not enjoy querying it at any depth. Storing many years of daily snapshots gets expensive on warehouse storage in a way it does not on object storage. And you are more locked in: moving between warehouses is a migration project, not a repoint.

Lake only

Land raw API responses to S3, catalogue them, query with Athena or Spark. Iceberg or Delta as the table format if you want anything resembling a database.

Where it wins: raw retention is nearly free, so keeping every version of every record for years is a storage line item rather than a decision. Schema evolution is genuinely easier: when a SaaS admin adds a field, the lake absorbs it. Unstructured content lives alongside structured tables, which matters more now that people want to run models over call notes and support tickets. And an open table format means the data is not hostage to one query engine.

Where it doesn’t: you have taken on work a warehouse does for you. Small-file compaction, partitioning, table maintenance, catalogue hygiene. Query latency on interactive dashboards is worse unless you invest in layout. Governance is a build, not a setting. And the per-query cost model punishes exactly the workload CRM analytics generates: lots of small, repeated, unpredictable queries from a BI tool. Athena bills on data scanned with a per-query minimum, which means a dashboard refreshing thirty tiles is thirty billable scans whether or not anyone is looking.

Lakehouse

Raw and historical data in object storage as Iceberg tables; curated marts in the warehouse or queried in place by a warehouse engine. This is where most serious setups end up, and it is what AWS, Databricks and Snowflake have all converged on.

Where it wins: you get cheap deep history and expensive fast queries in the right places. Raw payloads stay replayable when a vendor changes a field type. Analysts still get warehouse-shaped tables.

Where it doesn’t: it is two systems. Two cost models, two sets of permissions, two places a table can be defined and drift. For a five-person company with one CRM, this is architecture as procrastination. The complexity is real and it should be earning something.

What actually decides it

Four levers, in the order they usually matter for CRM data.

1. How much history you need, and whether deletes matter

This is the big one and it is the one people skip.

CRM systems mutate in place. An opportunity’s stage, amount and close date are overwritten as the deal moves. If you only ever sync current state, you can report on the pipeline as it is today and you cannot report on what it looked like last March. Every question about pipeline movement, forecast accuracy, stage velocity or “what changed since the QBR” needs point-in-time history, and the CRM will not give it to you retroactively. Deleted records are worse: they are simply gone.

If you need history, you need somewhere cheap to keep daily snapshots or a change feed. That pushes toward object storage, or toward a warehouse with a genuinely cheap storage tier. If a rolling twelve months of current-state data covers every question anyone asks, the warehouse handles it alone.

2. How fast the schema churns

Count the custom fields added to your main CRM object in the last six months. If it is two, schema churn is not your problem. If it is forty, and half of them are from a business unit you do not talk to, a rigid warehouse schema becomes a standing maintenance tax and the lake’s tolerance for drift starts paying for itself.

3. The shape of the queries

Scheduled dashboards hitting the same tables repeatedly favour a warehouse, because per-second compute amortises across many queries and results are cached. Redshift Serverless, for example, bills in processing-unit-hours per second with a short minimum and shuts down when idle, which suits bursty BI traffic.

Exploratory one-off analysis across wide, rarely-touched history favours a lake, because you only pay when someone asks. Most CRM analytics is overwhelmingly the first kind. That fact alone settles a lot of these arguments.

4. Who is going to maintain it

The least technical criterion and often the most decisive. If the answer is “a RevOps analyst who is very good at SQL and has no platform team behind them,” build the warehouse. A lake handed to someone who cannot maintain it becomes an unqueryable pile of Parquet in about eight months. That is not a hypothetical failure mode; it is the normal one.

A decision procedure you can run this afternoon

1. Count rows per object for everything anyone reports on. Note which objects are over about ten million rows. Usually it is activities and nothing else.
2. Write down the ten questions the business actually asks. Not the ones you imagine. Mark each one as needing current state or point-in-time history.
3. Count schema changes on your main objects over the last six months.
4. Estimate query volume: number of dashboards, tiles per dashboard, refresh frequency, plus rough ad-hoc queries per week. This is the number that drives cost, not storage.
5. Model both costs against those numbers. For the lake: bytes scanned per query times queries per month. For the warehouse: compute-hours per month plus storage. Do it on a napkin. At CRM scale the answer is usually obvious and usually favours the warehouse.
6. Name the maintainer. If you cannot name a person, pick the option that needs less of them.

Read together, most CRM-only shops land on: warehouse for the curated layer, cheap object storage for raw history, and no ambition beyond that until something forces it. That is a lakehouse, but the small honest kind, and it is a very different commitment from standing up a full lake platform.

Where zero-ETL changed the maths

A lot of writing on this topic predates a genuine shift, so it is worth stating plainly.

Managed zero-ETL integrations now replicate CRM and support platforms into AWS targets without you building a pipeline. AWS Glue supports SaaS sources including Salesforce, SAP, ServiceNow, Zendesk and Zoho CRM, landing into Amazon Redshift, the SageMaker lakehouse, or S3 Tables with Iceberg underneath. Salesforce has pushed the other direction with zero-copy federation, letting Data Cloud read Iceberg tables in Databricks, Snowflake, BigQuery or Redshift in place, and letting those platforms read Data Cloud objects back.

Two consequences for this decision:

• Ingestion is no longer the differentiator. “The lake is easier to load into” was a real argument when you were hand-rolling connectors. It is much weaker when the same managed integration targets either destination.
• Iceberg makes the choice less permanent. If your raw layer is Iceberg tables in object storage, adding a warehouse engine later is a configuration change rather than a migration. That is a genuine reason to prefer open table formats even if you never build a “lake” as such.

Two honest caveats. Zero-ETL integrations cover a curated list of sources and regions, so check yours before designing around them. And a managed replica is still current-state replication plus change capture, not a modelled history; you still have to decide how snapshots and slowly changing dimensions get built.

Arguments that don’t survive contact

• “We need a lake because we’ll do AI later.” Reasonable in spirit, wrong in sequence. Build the thing that answers this quarter’s questions, keep raw payloads in object storage so you have not destroyed your options, and revisit. Raw retention is the cheap insurance; a lake platform is not.
• “Storage is cheaper in a lake.” True and almost always irrelevant. At CRM volumes, storage is a rounding error next to compute and, more to the point, next to the salary of whoever maintains the thing.
• “The warehouse is a single point of vendor lock-in.” Fair, and it is why open table formats matter. But if you replicate raw data to object storage, the lock-in is on your transformation logic, not your data, and that is portable if you keep it in SQL and version control.
• “We’ll just query the CRM API directly and skip both.” Works until a dashboard fans out into rate limits and the whole thing becomes a data pipeline you did not design. If more than one person is asking questions, you need a copy.
• “Let’s do both properly from day one.” This is the expensive mistake. Two systems, two cost models, and tables that quietly disagree, in service of a workload that fits comfortably in one.

Frequently asked questions

Is a data lake overkill for one CRM?

For a single CRM with current-state reporting needs, usually yes. The point at which it stops being overkill is when you need years of point-in-time history, when unstructured content like call transcripts becomes part of the analysis, or when you are integrating enough systems that a common raw layer is cheaper than a dozen warehouse loaders.

How do I get historical CRM snapshots if I only have current state?

You mostly cannot get it retroactively, which is why this decision is urgent rather than academic. Start capturing now: either daily full snapshots to partitioned object storage, or a change feed if the platform offers one, then build slowly changing dimension tables from that. Field history tracking inside the CRM covers a limited number of fields and is not a substitute.

Does the lakehouse make this comparison obsolete?

It blurs the storage layer, not the operational reality. You still choose a query engine, a cost model, a governance model and a maintainer. The useful question has shifted from “lake or warehouse” to “how much of this do I want to operate myself,” which is a better question anyway.

Where does reverse ETL fit?

If you plan to push computed values back into the CRM, such as a lead score or account health field, that pushes toward the warehouse. Reverse ETL tools connect to warehouses far more readily than to query engines over object storage, and the write path wants a stable, modelled table rather than a raw one.

Can I start with one and move later?

Moving from lake to warehouse is straightforward: you are loading modelled tables into a new engine. Moving from warehouse to lake is harder, because the history only exists inside the warehouse and the transformations are written in its dialect. The asymmetric fix is to land raw data in object storage regardless of which you query from. That single habit preserves the option cheaply.

What about just using the CRM’s own analytics product?

Worth pricing seriously before building anything. It is genuinely the right answer when all your questions are about one system, and it removes an entire pipeline from your life. It stops being the right answer the moment a question spans CRM plus billing plus product usage, because you will be paying to ingest the other systems into a platform priced for CRM data.

The short version

In the data lake vs data warehouse question for CRM analytics, volume is the argument everyone reaches for and it is the one that matters least. Your CRM data is small. Both options handle it.

What decides it is how much history you need, how fast the schema moves, what shape the queries are, and who is going to keep it running. Answer those four and the architecture falls out on its own. And whatever you pick, land the raw payloads in object storage from day one, because that is the cheap decision that keeps every other decision reversible.

Need a second opinion on your CRM analytics stack?

I help teams make this call without over-building, and unpick it when it has already been over-built. Things I can help with:

• Running the sizing and cost model above against your real query volume, not a vendor calculator.
• Designing point-in-time history capture for CRM objects, including deletes and slowly changing dimensions.
• Setting up zero-ETL or connector-based ingestion from Salesforce, HubSpot, Zendesk or Zoho into Redshift, Snowflake or an Iceberg lake.
• Migrating an unmaintainable pile of Parquet into something an analyst can actually query.
• Cutting query costs on an existing setup: partitioning, table layout, dashboard refresh patterns.
• Building the curated CRM models that BI tools sit on, in SQL and in version control.

If you are mid-argument about this, send me your row counts and the ten questions the business wants answered. That is usually enough to settle it.

The post Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question appeared first on John Nessime.

“Salesforce handles it” is not an answer. Neither is “it’s in our VPC,” because it usually isn’t, and the parts that are in your VPC are the parts nobody has threat-modelled.

This post traces a single Agentforce request end to end and stops at each boundary it crosses. Agentforce and AWS get wired together in three or four standard patterns, and every one of them has a specific point where Salesforce’s guarantees stop and yours start. I will cover where that line sits, what changes about the security model when you cross it, and one default that surprises almost everyone.

The boundaries a single request crosses

One user turn touches more systems than the diagrams suggest. Roughly:

1. Retrieval. The agent grounds itself in CRM records, Data Cloud objects and knowledge content.
2. Reasoning. The request gets classified to a topic, then an action gets chosen. Both steps involve a model call.
3. Generation. A prompt is assembled and sent through the LLM gateway to a model provider.
4. Action. If the agent decides to do something, it calls out. That callout is where AWS usually enters the picture.

Steps one to three happen inside Salesforce’s trust boundary, and the Einstein Trust Layer applies. Step four does not, and it doesn’t. That sentence is the whole post, but the detail matters.

Retrieval: the part that’s better than you’d expect

Grounding is where Agentforce gets its context, and the architecturally interesting decision is that vector search lives inside Data Cloud rather than in a separate vector database. Structured records and unstructured content are retrieved from the same layer, which removes a whole class of sync problem you would otherwise be building.

The security-relevant property is that retrieval is permission-aware. The agent retrieves as the running user, so sharing rules and field-level security apply. If a rep cannot see an opportunity, the agent acting on their behalf should not be able to summarise it either.

Two caveats worth putting in your notes. First, this only holds while retrieval goes through Salesforce; the moment you ground from an external source, you are enforcing access control yourself. Second, if your CRM sharing model is loose, the agent inherits that looseness and makes it much easier to exploit, because a model will cheerfully summarise a thousand records a human would never have opened one by one. Agentforce does not create the over-permissioning problem, it just removes the friction that was hiding it.

Where AWS shows up here: if your analytical data lives in Redshift, an Iceberg lake or Databricks, zero-copy federation lets Data Cloud register those tables and query them in place rather than replicating them. Good for cost and duplication, and it moves an access-control decision into the federation configuration, which is a place people forget to audit.

The Trust Layer, and the default nobody expects

The Einstein Trust Layer sits between every Salesforce-originated prompt and the model. Its components are well documented: secure grounding, data masking, system policies against prompt injection, the LLM gateway, zero data retention agreements with model providers, toxicity checks on the way back, and an audit trail of the whole journey.

Here is the part that gets misreported constantly, including in a lot of otherwise decent write-ups: LLM data masking is disabled for agents. Salesforce says so plainly in its own documentation on the Trust Layer architecture and repeats it in the Trailhead module on masking. Masking remains available and configurable for embedded generative features like service replies and work summaries. For Agentforce agents, it is off.

The reason is not hard to work out once you think about what an agent does. Masking swaps real values for placeholders. An agent that has to actually perform an action needs the real record ID, the real amount, the real email address, because those become tool arguments. You cannot pass <Person_0> to a Flow and expect it to update a contact. Masking and acting are in genuine tension, and Salesforce resolved it in favour of acting.

What this changes in practice:

• Real PII reaches the model provider. The zero data retention commitment is what protects it, not de-identification. Those are different controls with different failure modes: masking is technical and verifiable, a retention agreement is contractual.
• If your compliance position was “no personal data leaves our boundary in identifiable form,” Agentforce does not satisfy it the way you may have assumed, and this is worth raising before your security review rather than during it.
• Controlling exposure moves upstream. What the model sees is now decided by what you ground on and what your sharing model permits, not by a masking filter downstream.

None of this makes the Trust Layer weak. Zero retention, the gateway, system policies and the audit trail all still apply to agent traffic, and the Trust Layer typically runs several times in a single user turn, once for topic classification and again for generation. It is a real control plane. Just know which control is doing the work.

Connecting Agentforce and AWS: four patterns

There are four ways these two systems talk, and they have different latency, failure and security characteristics.

Synchronous outbound: agent action calls AWS

A custom agent action, implemented via External Services with an OpenAPI schema, an Apex invocable method, or a Flow, calls an endpoint you own. Behind it sits API Gateway and Lambda, and often a Bedrock agent or knowledge base.

Authentication uses named credentials, which keep the secret out of your Apex and centralise the auth config. Use them; hardcoding a key into an Apex class is the version of this that ends up in a git history.

The trade-off is that you have put a network call and a second model invocation inside a conversational turn. Users notice. Budget for it, and set the timeout deliberately rather than inheriting a default.

Asynchronous outbound: Event Relay to EventBridge

Salesforce Event Relay delivers platform events from the Salesforce event bus to Amazon EventBridge without code, and EventBridge routes onward to Lambda, S3, Step Functions or a Bedrock agent.

This is the right pattern for anything that does not need to complete inside the conversation: enrichment, downstream processing, long-running work. It also fails better, because a retry is a queue concern rather than a user watching a spinner.

Inbound: AWS calls Agentforce

The Agent API lets something on the AWS side invoke an Agentforce agent, typically through EventBridge API destinations. That gives you bidirectional multi-agent flows: a Bedrock agent that detects a problem can open a case through Agentforce rather than writing to the Salesforce API directly and bypassing the agent’s own logic.

Worth being deliberate here. An inbound path means an AWS workload can trigger agent reasoning, which means anything that can inject content into that workload can influence a prompt. Treat the payload as untrusted input, because it is.

MCP: tools rather than integrations

Agentforce has a native MCP client, so an agent can connect to any MCP-compliant server without a bespoke integration, alongside a server registry intended to let admins govern which agents connect to which tools under central policy.

This is the pattern that will cause the most trouble over the next couple of years, and not because the protocol is bad. It is because MCP makes adding a tool trivial, and every tool is a new path by which text from somewhere else reaches your agent’s context and by which your agent’s decisions reach someone else’s system. The registry exists precisely so that “which tools can this agent reach” is an administered answer rather than an emergent one. Use it as a control, not a catalogue.

Keeping the traffic private

By default these callouts traverse the public internet with TLS. Salesforce Private Connect uses AWS PrivateLink to give you a private path to your own AWS endpoints instead. Whether that is required is a policy question rather than a technical one, but it is much easier to set up at design time than to retrofit after an auditor asks.

What changes the moment you cross into AWS

This is the seam, and it is where most real incidents will come from. Four things stop being true simultaneously.

• Zero retention becomes your policy, not Salesforce’s. If your Lambda logs its input at INFO, you now have customer PII in CloudWatch Logs with whatever retention that log group happens to have, which by default is forever. Your own debug logging is the single most likely leak path in this entire architecture, and it will not show up in any Salesforce audit report.
• The audit trail splits. Salesforce logs the prompt journey. AWS logs the invocation. Nothing correlates them unless you pass a request identifier through the callout and log it on both sides. Do that from day one; reconstructing an incident across two disconnected audit systems is genuinely unpleasant.
• User identity does not travel. This is the important one. Inside Salesforce the agent acts as the running user and sharing rules apply. Your Lambda runs under an IAM execution role with fixed permissions and no idea who asked. If that role can read a whole DynamoDB table, then every user of the agent can effectively read the whole table through it, regardless of their CRM permissions. That is a classic confused deputy, and it is easy to build without noticing. If the action needs to be scoped per user, pass the identity explicitly and enforce it in your own code.
• Model governance forks. A Bedrock model you invoke from your own Lambda is outside the Trust Layer. Its guardrails, logging and retention are configured in your account and are yours to get right. If you want prompt filtering or PII detection on that leg, you are building or buying it.

Note the difference between two things people conflate. Salesforce running on Hyperforce, which is built on public cloud infrastructure including AWS, means Salesforce’s own workloads run there under Salesforce’s controls. It does not mean your Salesforce data sits in your AWS account. When an agent action calls your Lambda, that is the first moment data lands in infrastructure you control and are responsible for.

Common mistakes

• Assuming masking protects agent traffic to the model. It does not; check the current documentation yourself rather than taking a blog’s word for it, including this one.
• Logging full request payloads in Lambda during development and never turning it off.
• Giving the Lambda execution role broad data access because “the agent is trusted,” when the agent is a proxy for every user who can talk to it.
• Putting a slow AWS call inside a synchronous agent action when the work did not need to be synchronous.
• Not passing a correlation identifier across the boundary, so Salesforce and CloudWatch tell two unlinked stories.
• Treating content returned from an external tool or MCP server as trusted context rather than as untrusted input that will be read by a model.
• Deploying agents against a permissive sharing model and discovering the blast radius later.

Frequently asked questions

Does Agentforce data get used to train models?

Not on the Salesforce leg. Salesforce’s zero data retention arrangements with model providers mean prompts and responses are not stored or used for training. That commitment covers traffic going through the LLM gateway. It says nothing about a model you invoke yourself from your own AWS account, where the retention and logging configuration is entirely yours.

Can I use my own Bedrock model inside Agentforce?

There are two distinct things here. Bedrock-hosted models can be brought into Salesforce through the Einstein and Bedrock integration, in which case they sit behind the Trust Layer. Separately, you can call a Bedrock model or agent from a custom agent action, in which case you are outside the Trust Layer and own the controls. Be clear which one you are building, because they have completely different compliance stories.

Sync or async for AWS integrations?

Default to async through Event Relay and EventBridge unless the agent genuinely needs the result to continue the conversation. Synchronous callouts put a network round trip and often a second inference call inside a user’s turn, and they fail in front of the user rather than into a retry queue.

How do I stop the agent seeing data a user shouldn’t?

Inside Salesforce, fix the sharing model, because retrieval respects it and inherits its flaws. Outside Salesforce, sharing rules do not apply at all: pass the user identity through the callout and enforce authorisation in your own code, or scope the integration to data that is safe for every user of that agent.

Do I need PrivateLink for this?

Technically no; callouts work over TLS on the public internet. Practically it depends on your regulatory position and whether your AWS endpoints should be publicly reachable at all. If they should not be, Private Connect over PrivateLink is the supported path and is far cheaper to design in than to add later.

What should I log, and where?

On the AWS side, log the correlation identifier, the action name, the outcome and the timing. Do not log the payload. If you need payload-level debugging, gate it behind a flag that is off in production and set an explicit short retention on the log group. Ship the operational metrics somewhere you already watch, whether that is CloudWatch alarms, Grafana or Datadog, because an agent action that silently starts failing looks like nothing at all from the Salesforce side.

The one thing to take away

Wiring up Agentforce and AWS is not hard. The patterns are documented, the connectors exist, and a working proof of concept is a day’s work.

What is hard is knowing precisely where Salesforce’s guarantees end. They end at the callout. Everything before it is covered by the Trust Layer, with the significant exception that masking is off for agents, so what reaches the model is real data protected by a retention agreement rather than by de-identification. Everything after it is yours: your retention, your audit trail, your authorisation, your logs.

Draw that line on your architecture diagram before the security review, not during it.

Need help designing or reviewing this?

I work on the AWS side of Salesforce integrations, which is usually the side nobody owns. Things I can help with:

• Threat-modelling the seam between Agentforce and your AWS account, and writing down what each side actually guarantees.
• Building agent actions on API Gateway and Lambda with named credentials, sane timeouts and no payload logging.
• Event-driven integration via Event Relay and EventBridge, including retry, dead-letter and replay design.
• Scoping IAM execution roles so an agent action cannot become a confused deputy for every user who can reach it.
• Correlated observability across Salesforce and CloudWatch so an incident can be reconstructed in one timeline.
• Private Connect and PrivateLink setup where callouts should not touch the public internet.

If you have an integration already built, send me the agent action definition and the Lambda’s IAM policy. Those two together usually tell the whole story.

The post Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin appeared first on John Nessime.

A Salesforce admin renames a custom field on Tuesday afternoon. Your connector keeps running, your Glue job keeps succeeding, the crawler happily adds a new column and quietly stops populating the old one. Three weeks later someone asks why the pipeline report has a cliff in it, and you spend an afternoon reconstructing a change nobody in engineering made and nobody logged.

This post is about using AWS Glue Data Quality to catch that class of breakage. It is aimed at people pulling from SaaS sources into S3 or a lakehouse: Salesforce, HubSpot, Zendesk, Stripe, Shopify, Zoho, an internal partner API. I will cover where to run the checks, the flattening constraint that will bite you before anything else does, which DQDL rule actually catches which SaaS failure, when dynamic rules and anomaly detection quietly do nothing, and what should happen when a rule fails.

Why SaaS data breaks in ways your own database doesn’t

When you own the source database, the schema is under change control: someone writes a migration, it gets reviewed, and if it breaks a consumer there is a commit to point at. A SaaS source has none of that. Three consequences:

• The schema is edited by people who are not engineers. A sales ops admin adding a picklist value is a production schema change with no review, no notice and no version number.
• The payload is nested. SaaS APIs return structs and arrays: line items, related objects, custom field bags. That shape matters more than you would expect, for reasons in the next section but one.
• Absence is ambiguous. A field missing from a payload might mean “not set,” “not permitted for this user,” or “we removed it.” The connector usually cannot tell you which, and neither can your table.

Glue Data Quality is a rules engine built on Deequ that evaluates declarative checks written in DQDL, the Data Quality Definition Language, against a dataset. It is well suited to the first and third problems. The second one needs work before it is useful at all.

Decide where the checks run before you write a rule

There are two places to evaluate a ruleset, and they answer different questions.

Inside the Glue ETL job

The EvaluateDataQuality transform runs mid-pipeline against a DynamicFrame. It answers “is this batch safe to load?” and it can act on the answer: stop the job, or split failing rows into a quarantine prefix. This is also the only placement where row-level results exist, which matters because “which 400 rows are bad” is a far more useful answer than “the rule failed.”

Against a Data Catalog table

You can attach a ruleset to a catalog table and run it on a schedule, independent of any job. This answers “is the table currently trustworthy?” and surfaces a quality score analysts can see before they query. The trade-off is that it is after the fact: by the time a catalog evaluation fails, the bad batch is already queryable. Analyzers are also not supported for catalog evaluations, so some dynamic-rule tooling behaves differently there.

In practice I run both. The in-job ruleset is small and strict, and exists to stop garbage landing. The catalog ruleset is broader and looser, and exists so a table’s state is visible to people who will never open Glue Studio.

The flattening problem that comes first

Here is the constraint that catches most people on SaaS data, stated plainly in the AWS documentation and easy to skim past: DQDL cannot evaluate nested or list-type columns. No structs, no arrays.

Almost every interesting SaaS payload is nested. A Salesforce opportunity carries a related account struct, a Stripe invoice carries an array of line items, a Zendesk ticket carries a custom fields array of key-value pairs. Point a ruleset at the raw landed data and the fields you most want to check are the ones the engine cannot see.

So you flatten first, with Glue’s relationalize or an explicit projection, then evaluate. That works, but it changes what your rules mean:

• Your rules now describe the projection, not the payload. If your flattening step drops a field, no rule will notice, because from DQDL’s point of view that field was never there.
• Exploding an array changes the grain. One invoice with five line items becomes five rows. Every RowCount rule you write against that table is now counting line items, and a customer who starts itemising differently will move your row count without any data being wrong.
• Custom-field bags need pivoting, not flattening. A key-value array flattened naively gives you rows, not columns, and you cannot write a ColumnValues rule against a value whose meaning depends on a sibling key column. Pivot the ones you care about into real columns and check those.

The practical rule: put your quality checks immediately after the flattening step, and treat the flattening step itself as code that needs its own test. Glue Data Quality will not cover it for you.

Matching SaaS breakage to the rule that catches it

DQDL has a long rule list and most guides just enumerate it. More useful is the other direction: what actually goes wrong with SaaS sources, and what catches each one.

A field is renamed, removed, or added

This is the Tuesday-afternoon scenario. Two rules, doing two different jobs:

Rules = [
    ColumnExists "annual_revenue__c",
    ColumnCount = 47
]

ColumnExists protects the specific fields your downstream models depend on. ColumnCount protects against everything else: a field silently added, or one you forgot you needed. The first is precise and the second is a tripwire, and you want both because a rename shows up as one column gone and one column arrived, which leaves the count unchanged.

SchemaMatch compares against a reference dataset’s schema, which is stricter and useful if you maintain a golden reference table. Bear in mind a hard column count on a SaaS object fails every time an admin adds a field, so decide whether that blocks or just warns.

A field still exists but stopped being populated

Permissions change, a workflow gets disabled, an integration user loses field-level access. The column is still in the schema and is now all nulls.

Rules = [
    IsComplete "opportunity_id",
    Completeness "lead_source" > 0.85,
    Completeness "email" > 0.60 where "record_type = 'Contact'"
]

IsComplete is the absolute version: no nulls at all, right for keys. Completeness takes a threshold, which is what you want for optional business fields where a hard rule would fail constantly. The where clause is the one people miss: SaaS objects are usually polymorphic, and a completeness threshold that makes sense for contacts is nonsense across the whole table. Scope the rule to the record type.

The sync ran short

A paginated extract stops early, a rate limit gets absorbed by retries, an incremental cursor drifts. The job succeeds and delivers less data than it should have. A static RowCount > 1000 stops catching this the moment you cross a thousand rows. Dynamic rules compare against your own history instead:

Rules = [
    RowCount > min(last(3)),
    Sum "amount" > avg(last(5)) * 0.7,
    DetectAnomalies "RowCount"
]

Analyzers = [
    RowCount,
    DistinctValuesCount "owner_id"
]

last(k) pulls the last k recorded values of that metric for the same dataset; with k greater than one you need an aggregation like avg, min or max to reduce them to one number. DetectAnomalies goes further, using forecasting over stored history to flag deviations with no threshold from you at all. That is the right tool for anything seasonal: a B2B pipeline that dips every weekend trips a naive trailing-average rule every Monday, while anomaly detection learns the shape.

Analyzers deserve their own mention: they gather a metric without asserting anything, which builds the history dynamic rules and anomaly detection need. DistinctValuesCount on an owner or account column is a strong early signal for a partial sync, because row count can look fine while the number of distinct accounts represented quietly halves.

Records arrive twice, or the key stops being a key

Replays, overlapping incremental windows and connector retries all produce duplicates, and SaaS “IDs” are less unique than vendor docs imply once you have merged sandboxes or multiple accounts.

Rules = [
    IsPrimaryKey "id",
    Uniqueness "external_reference" > 0.99
]

IsPrimaryKey asserts unique and non-null together. Uniqueness with a threshold is the softer version for fields that are mostly-but-not-perfectly unique, which describes a lot of SaaS reference fields honestly.

Objects stop agreeing with each other

Multi-object extracts rot quietly. Opportunities reference accounts that were never synced; invoice totals stop matching line item sums because one object’s extract failed and the other’s didn’t.

ReferentialIntegrity checks that values in one dataset’s column exist in another’s. RowCountMatch and AggregateMatch compare counts and summary metrics across two datasets. These catch a broken join before an analyst does, and they need the multi-frame form of the transform with the second dataset supplied as an additional source.

The sync stopped and nobody noticed

The quietest failure of all: nothing is wrong with the data, there just isn’t any new data. DataFreshness checks a timestamp column against a window. It is deliberately simple and its comparison granularity is limited, so for anything more nuanced the custom SQL rule is the escape hatch, running a query against the dataset and evaluating the result against an expression. DQDL is case sensitive and that rule type’s exact capitalisation has appeared more than one way across the docs and the visual builder, so copy it from the current DQDL reference rather than from memory.

Static, dynamic and ML: when each one lies to you

All three check styles have a mode where they pass while the data is wrong. Knowing which is which is most of the skill.

• Static rules lie by going stale. A threshold that was correct when you wrote it stops being correct as the business grows, and a rule that can no longer fail is worse than no rule because it looks like coverage.
• Dynamic rules lie when they have no history. This one is genuinely nasty. last(k) reads previously recorded values for that metric, and if there aren’t any, it falls back to a default of zero. Add RowCount > avg(last(3)) to an existing job today and it will pass trivially for the first few runs, because almost any row count beats zero. It looks like a working check from the moment you deploy it. It isn’t one until it has accumulated runs.
• Anomaly detection lies while it is warming up, and after you retrain it badly. It needs a run of history before its forecasts mean anything, and if you leave a period of genuinely bad data in the training window, it learns that the bad shape is normal. You can exclude specific statistics from the calculation in the Data Quality tab, and you should, after any incident.

So when you add a dynamic or ML-based rule, write down the date and treat it as unarmed for the first couple of weeks. Keep a crude static rule alongside it in the meantime.

What should happen when a rule fails

The transform returns a collection. The two members you care about are the row-level outcomes, which give you the original data with an evaluation result column appended per row, and the rule outcomes, which give you one row per rule with its verdict.

from awsgluedq.transforms import EvaluateDataQuality

dq_results = EvaluateDataQuality().process_rows(
    frame=flattened_opportunities,
    ruleset=ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "salesforce_opportunity",
        "enableDataQualityCloudWatchMetrics": "true",
        "enableDataQualityResultsPublishing": "true",
    },
)

row_level = SelectFromCollection.apply(
    dfc=dq_results, key="rowLevelOutcomes"
)

dataQualityEvaluationContext is the namespace your CloudWatch metrics land under. Name it after the source object rather than the job, because you will eventually want to alert per object.

From there, three response patterns:

1. Fail fast. Stop the job on failure so nothing lands. Right for structural rules: if the primary key isn’t unique, downstream is going to be wrong no matter what you do next.
2. Quarantine. Filter the row-level outcomes on the evaluation result column, write passing rows to the target and failing rows to a separate prefix. Right for value-level rules where 2% bad records shouldn’t block 98% good ones. The important discipline is that somebody has to actually look at the quarantine prefix; an unmonitored quarantine is a delete with extra steps.
3. Warn and continue. Let everything land, publish the metrics, alert on the score. Right for new rules you don’t trust yet, and for the catalog-side ruleset.

Whichever you pick, get the failure out of Glue and into wherever your team actually looks. Results and anomaly predictions can be written back to catalog tables so they are queryable with plain SQL, and CloudWatch metrics plus EventBridge events give you the hook for paging or pausing a downstream pipeline. A Grafana or Datadog dashboard fed from those metrics beats a score in a console tab nobody opens.

Troubleshooting AWS Glue Data Quality on SaaS pipelines

“Input data does not include column X”

A rule references a column the engine cannot find. On SaaS data that means one of three things: the field disappeared from the source, your flattening step renamed it, or the Data Catalog is out of sync with the underlying table. Check the catalog schema against the landed files before touching the ruleset; rerunning the crawler resolves a surprising share of these.

A rule type isn’t recognised

DQDL is case sensitive and the rule set has grown over releases, so a rule copied from an older blog post can parse fine in your head and not at all in the engine. Check the rule name against the current DQDL rule type reference, and check that the feature is available on your Glue version. Anomaly detection in particular has a version floor.

A dynamic rule never fails

Almost always the empty-history problem described above. Look at the statistics for that metric in the Data Quality tab. If there are fewer recorded points than your last(k) asks for, the rule is not doing what you think. It will start working on its own; the mistake is trusting it in the meantime.

Evaluation is slow or the job runs out of memory

Each rule triggers analyzer computation over the dataset, and a large ruleset on a wide flattened SaaS table adds up fast. The transform exposes caching and observations-scope options to tune this. Before reaching for a bigger worker type, cut the ruleset down: most contain a dozen recommended rules nobody reads the results of.

A “column reference is ambiguous” error

Classic flattening artefact. Two nested paths collapsed to the same leaf name, so datechecked now matches two things. Fix it in the projection with explicit aliases rather than trying to disambiguate inside DQDL.

Common mistakes

• Accepting the recommended ruleset wholesale. Recommendations are generated from a sample of current data, so they encode today’s accidents as tomorrow’s requirements.
• Writing rules against nested data and assuming they cover the whole payload.
• Deploying a dynamic rule and treating it as armed on day one.
• Applying whole-table completeness thresholds to a polymorphic object instead of scoping with a where clause.
• Quarantining failed rows to a prefix nobody monitors.
• Writing RowCount rules against an exploded array without noticing the grain changed.
• Failing the job on every rule, including cosmetic ones, until the team starts rerunning with checks disabled.
• Keeping the ruleset in the console instead of in version control next to the job.

Best practices

• Split rules into blocking and reporting tiers, and be honest about which is which. A rule that always fails and never stops anything trains people to ignore the tab.
• Pair ColumnExists with ColumnCount. One protects the fields you named, the other catches the drift you did not anticipate.
• Add analyzers early, even for metrics you have no rule for yet. History has to accumulate before dynamic rules and anomaly detection are worth anything, and you cannot backfill it.
• Keep DQDL in Git alongside the job definition and deploy it with Terraform or CloudFormation. Rulesets are code and should move through the same review as the pipeline they guard.
• Namespace the evaluation context per source object so CloudWatch metrics and alerts can be routed per vendor.
• Exclude incident periods from the anomaly detection training window once the incident is resolved.
• Surface results where stakeholders already look. Writing outcomes back to catalog tables makes them queryable from Athena, which beats a score in a console tab.

Frequently asked questions

How much does AWS Glue Data Quality cost?

It bills on Glue DPU-hours like any other Glue workload, so cost is a function of evaluation time and DPUs used. The dimension that surprises people is anomaly detection, which consumes additional capacity per statistic analysed, so enabling it broadly across many columns and tables is where the bill moves. Model it against the current Glue pricing page; rates change and vary by region.

Can it check data before it lands in S3?

Yes, if the check runs inside the Glue job. The transform evaluates a DynamicFrame in memory, so you can validate after extraction and before the write. That is the placement that lets you stop a bad load rather than report on one.

Does it work on nested JSON from a SaaS API?

Not directly. DQDL does not evaluate nested or list-type columns, so you have to flatten the structure first and run the ruleset against the flattened frame. Plan the flattening deliberately, because your rules only ever see what the projection exposes.

How does it compare to Great Expectations, Soda or Monte Carlo?

Glue Data Quality’s real advantage is that it is already inside the runtime and the catalog: no extra infrastructure, no separate agent, one IAM story. The disadvantages are real too. DQDL is less expressive than Python-based assertions, the nested-data limitation is a hard wall, and it only covers what Glue can read. Great Expectations gives you more expressive checks at the cost of running it yourself; Monte Carlo and similar platforms add lineage and cross-warehouse coverage Glue does not attempt. Glue-centric stack, start here. Several engines, a standalone tool fits better.

Should I use the recommended rules?

As a starting draft, yes. As a ruleset, no. Recommendations are inferred from a sample of the data as it is today, which means any existing quality problem gets encoded as the expected baseline. Read every generated rule and delete the ones you would not have written yourself.

Can I reconcile row counts against the SaaS source itself?

Not directly; it only sees datasets it can read. The usual pattern is to have the extraction step record the count the API reported, land that alongside the data as a control table, then use RowCountMatch or AggregateMatch to compare the two. That turns “did we get everything” into a rule the engine can evaluate.

The one thing worth remembering

AWS Glue Data Quality is not really a data quality product. It is a way of writing down what you believe about a dataset so a machine can check it every run, and its value is entirely determined by whether those beliefs are written precisely enough to be wrong.

For SaaS sources that means two habits. Flatten deliberately and check immediately after, because your rules only see the projection. And treat every new dynamic or ML-based rule as unarmed until it has history, because a rule that cannot fail yet looks exactly like a rule that is passing.

Need help with data quality on your Glue pipelines?

I work with teams whose SaaS pipelines run green while the numbers drift. Things I can help with:

• Writing a DQDL ruleset for a specific SaaS object that catches schema drift instead of just restating the current schema.
• Designing the flattening and pivoting step for nested payloads and custom-field bags so the checks actually cover what matters.
• Setting up quarantine and fail-fast paths in Glue ETL, with the routing and alerting that makes a quarantine prefix something people act on.
• Cross-object reconciliation between SaaS extracts and control counts recorded at extraction time.
• Tuning anomaly detection and dynamic rules, including cleaning incident periods out of the training window.
• Getting data quality results out of the console and into Grafana, CloudWatch alarms or an Athena-queryable table.

If you have a ruleset that keeps passing while the data drifts, send me the DQDL and a schema dump. That is usually enough to see where the gap is.

The post AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed appeared first on John Nessime.

That is the shape of most API pipeline incidents. Not a crash. A success that isn’t one.

This post covers running Apache Airflow on AWS specifically for SaaS and API workloads: pulling from HubSpot, Salesforce, Stripe, Zendesk, Shopify, an internal partner API, whatever. It is organised by failure family rather than by feature, because the Airflow documentation already explains what an operator is and does a poor job of explaining which of these things will page you at 3am. I will cover choosing a deployment model, the concurrency trap that catches almost everyone, retry design, incremental state, secrets, and where the money actually goes.

Why SaaS and API sources break differently

When your source is a database you control, failure is loud: connection refused, deadlock, disk full. When it is somebody else’s SaaS API, three things change.

• You are a guest. The vendor decides your rate limit, and they can change it without telling you. Your pipeline’s correctness now depends on a number in someone else’s config file.
• Errors arrive as valid HTTP. A 429, a 200 with a truncated page, a 200 with an error object in the body. Your HTTP client is happy. Your data is not.
• Tasks spend most of their life waiting. API extraction is I/O bound almost end to end. That sounds harmless and is the root of the most expensive mistakes.

Pick the deployment model before you write a DAG

This decision constrains everything after it and is harder to reverse than people expect. Three realistic options.

Amazon MWAA, provisioned

AWS runs the scheduler, web server, workers, triggerer and metadata database on Fargate; you drop DAGs into an S3 bucket and they get picked up.

Where it wins: real Airflow, custom providers, custom plugins, full control over environment configuration. If your DAGs need arbitrary Python libraries, this option will not fight you.

Where it doesn’t: the environment bills by the hour whether or not anything is running. There is no scale to zero on the base environment. If you sync six APIs once a day and each run takes twenty minutes, you are paying for a mostly idle cluster around the clock. The mw1.micro class exists precisely for the small case, but it collapses the scheduler and worker into a single Fargate task and caps worker autoscale low, so treat it as a dev or isolation tier rather than a cheap production tier.

Amazon MWAA Serverless

You submit workflow definitions and AWS runs each task in its own Fargate container, billing per task duration with a one-minute minimum rather than per environment hour.

Where it wins: spiky or infrequent schedules. If the workload is “six syncs a day, nothing overnight,” the cost profile beats a permanently running environment by a wide margin. Each workflow also gets its own IAM execution role, which is a real security improvement over one shared role per environment.

Where it doesn’t: it leans on declarative YAML workflow definitions based on the DAG Factory format and a curated set of AWS operators. That is a deliberate trade: because the definition is declarative, the service can schedule tasks without executing your DAG code. It also means custom operators, exotic third-party providers and clever Python at parse time are not the sweet spot. It is also available in fewer regions than provisioned MWAA, so check your region before you design around it.

Self-managed on ECS, EKS or a VPS

On EKS with the Kubernetes executor you get per-task pods and tight cost control. On a single VPS from a provider like InterServer or Hetzner, a Docker Compose stack with a Postgres metadata database will run a modest set of API syncs for a fraction of any managed price.

Where it wins: cost at both extremes, and total control. Where it doesn’t: you now own metadata database upgrades, major version migrations, log retention and the 2am scheduler restart. Managed Airflow is a bet that your time is worth more than the hourly premium. For a solo engineer with three pipelines that bet often loses; for a data team of eight it usually wins. Astronomer is the main non-AWS managed option worth pricing alongside these.

Failure family one: the throttle that silently stops throttling

You start with a normal setup: an Airflow pool named crm_api with four slots, and every task that touches the vendor assigned to it. Four concurrent requests, comfortably under the vendor’s limit. This works.

Then you notice those tasks spend nearly all their runtime waiting on HTTP, burning worker slots to sit still. So you switch them to deferrable operators. A deferrable task suspends itself while waiting, releases its worker slot, and hands the waiting to the triggerer, which polls asynchronously. Worker pressure drops. Everything looks better.

And your rate limiting quietly stops working.

By default, a pool does not count tasks in the deferred state as occupying slots. That was deliberate, and the logic is sound in the abstract: a deferred task is not consuming a worker. But if you were using the pool to protect an external API rather than your own workers, it has just stopped doing the job you gave it. Every task can defer at once, and the vendor sees the full fan-out.

The fix is a per-pool flag, include_deferred, which tells the scheduler to count deferred tasks against the slot budget. It is off by default. You can set it when editing the pool in the Airflow UI, or through the API.

The failure signature is what makes this nasty. Nothing errors. Your DAG gets faster. The vendor starts returning 429s that your retry logic absorbs, and the only symptom is that runs take a little longer and occasionally a page goes missing. Weeks can pass. Two related traps in the same family:

• max_active_tasks at the DAG level has the same blind spot with deferred tasks, and there is no equivalent opt-in flag. If you need a hard external concurrency cap, use a pool with include_deferred enabled, not DAG-level concurrency.
• On MWAA, the triggerer runs alongside the scheduler on the same Fargate task, so scheduler count and triggerer capacity are linked. If you go heavily deferrable and your deferred tasks start stalling, scheduler capacity is the thing to look at.

Failure family two: retries that make the outage worse

The default instinct is to set retries high and move on. Against a rate-limited API, a fixed retry delay across many parallel tasks is just a slower version of the same stampede.

What you want is exponential backoff with a ceiling. The shape:

from datetime import timedelta

from airflow.sdk import dag, task

@dag(
    schedule="0 5 * * *",
    catchup=False,
    max_active_runs=1,          # never let two runs of this DAG overlap
    default_args={
        "retries": 5,
        "retry_delay": timedelta(seconds=30),
        "retry_exponential_backoff": True,   # 30s, 60s, 120s, 240s...
        "max_retry_delay": timedelta(minutes=15),  # stop doubling here
        "pool": "crm_api",      # shared budget across every task touching this vendor
    },
    tags=["crm", "extract"],
)
def crm_extract():

    @task(max_active_tis_per_dag=4)
    def fetch_page(page_token: str) -> str:
        ...

crm_extract()

The lines that matter:

• retry_exponential_backoff turns retry_delay into a base rather than a constant, so repeated failures spread out instead of hammering in lockstep.
• max_retry_delay caps the doubling. Without it, a task that fails five times can sit idle for hours and blow past the window you actually cared about.
• max_active_runs=1 is the one people skip. If a run overruns its schedule, the next one starts anyway, and now two runs are fetching the same pages from the same vendor with the same credentials. This is a common way to trigger a rate limit you have never hit before.
• max_active_tis_per_dag limits how many instances of that specific task run concurrently across DAG runs, which is the right knob for dynamically mapped extraction tasks.

One thing Airflow will not do for you: honour a Retry-After header. Airflow’s retry timing is computed from your config, not from the vendor’s response. If the API tells you exactly how long to wait, you have to catch that in your own code and sleep or reschedule accordingly. Ignoring a header the vendor bothered to send is a good way to get your API key throttled harder.

Failure family three: pagination, cursors and the empty page

Back to the Monday message. The specific bug behind most “the numbers are wrong but nothing failed” incidents is a loop that treats any non-error response as a terminating condition. Three rules prevent it:

1. Never infer “done” from an empty result. Terminate on the explicit signal the API gives you: a null next_cursor, a missing Link header, a page count. An empty array with a valid cursor still has more data behind it.
2. Assert the response shape before you use it. Check the status code explicitly and validate that the fields you depend on exist. A 200 carrying {"error": "..."} should raise, not return zero rows.
3. Land raw, transform later. Write the untouched API response to S3 first, then parse from S3. When the vendor changes a field type, you can replay from raw instead of re-extracting from an API that no longer serves that window.

Where to keep incremental state

The tempting pattern is to store the last-seen timestamp in an Airflow Variable and update it at the end of a run. Do not make that your source of truth. If a run dies midway, the Variable is in an undefined state, and clearing and re-running the DAG will not restore it. Airflow’s retry and backfill machinery has no idea it exists.

Better: make each run’s window a function of the run itself, and write output to a deterministic, run-scoped location such as s3://bucket/source=crm/dt=<logical-date>/. Re-running the same interval overwrites the same prefix. That is what makes a task idempotent, and idempotency is the difference between “clear the task and let it rerun” and a two-hour manual repair.

Then overlap your windows deliberately. Many SaaS APIs order results by modified time with eventual consistency, so a record edited at the boundary can appear after you have already moved on. Query a window slightly wider than your schedule interval and rely on an idempotent upsert downstream to absorb the duplicates. Late-arriving data is not an edge case with SaaS sources. It is the normal case.

Airflow’s asset-based scheduling is the clean way to trigger downstream DAGs from this: the extract DAG produces an asset, and the transform DAG runs when the asset updates, rather than being scheduled at a time you hope is late enough.

Failure family four: credentials

API tokens rotate, sometimes on the vendor’s schedule rather than yours. Storing an API key in an Airflow Connection through the UI works, and is the wrong long-term answer: the value lives in the metadata database and there is no rotation story. On AWS, point Airflow’s secrets backend at AWS Secrets Manager. On MWAA that is an environment configuration option:

secrets.backend
  airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

secrets.backend_kwargs
  {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}

With that in place, a connection lookup for crm_default resolves against the secret at airflow/connections/crm_default. Rotating the credential is a Secrets Manager operation with no Airflow deployment involved.

Two things to know before you turn it on. First, every connection and variable lookup becomes a Secrets Manager API call, and lookups fall through to the backend before hitting the metadata database, so a DAG that reads a Variable at parse time will generate a call on every parse cycle. Move those reads inside tasks. Second, the environment’s execution role needs explicit read permission on the relevant secret ARNs, and if you use a customer-managed KMS key, decrypt permission on that key too.

Worth knowing if you are on Airflow 3: task code can no longer reach the metadata database directly. All runtime interaction goes through the Task Execution API. If you inherited custom operators that open a session and query Airflow’s own tables, that is a migration blocker, not a warning.

Failure family five: the bill

Nobody is surprised by the environment line item. They are surprised by the other four.

• Idle time. A provisioned MWAA environment bills continuously. Compute the ratio of hours billed to hours doing work. If it is bad, that is the argument for MWAA Serverless or for consolidating several thin pipelines into one environment.
• NAT Gateway. This is the classic one. Private-subnet workers calling public SaaS APIs route through a NAT Gateway, which charges hourly and per gigabyte processed. A high-volume extraction pipeline can spend more on NAT than on Airflow. VPC endpoints remove that cost for AWS service traffic, but they do nothing for calls to a third-party API, which is exactly the traffic an API pipeline generates.
• CloudWatch Logs. Task logs go to CloudWatch, and ingestion is billed per gigabyte. Set the Airflow log level per component rather than globally at DEBUG, and set a retention policy on the log groups. The default is to keep logs forever.
• S3 requests. Landing raw API responses one small object per page generates a lot of PUTs. Batch pages into larger objects where you can.

Rates and dimensions change, so model your own workload against the current pricing page rather than trusting a number from a blog post. The point is knowing which four lines to look at.

Troubleshooting Apache Airflow on AWS when API pipelines misbehave

Tasks sit in “queued” and never start

Usually a slot problem, not a broken scheduler. Check, in order: is the pool full; has DAG-level max_active_tasks been hit; is worker autoscaling at its configured maximum. On MWAA, the container and queue utilisation metrics published to CloudWatch tell you which of the three it is far faster than reading scheduler logs.

DAG file changes don’t appear

On MWAA, DAGs sync from S3 on an interval; it is not instant. If a file has been there for several minutes and still hasn’t appeared, it almost always failed to parse. Check the DAG processing logs in CloudWatch rather than the scheduler logs, because a broken import raises there and never reaches the scheduler.

A new provider package won’t install

MWAA installs from your requirements.txt in the DAGs bucket, and from Airflow 2.7.2 onward that file must include a constraint line. Without one, MWAA picks a constraint for you, and pip is free to resolve a provider version that conflicts with the Airflow build in the image.

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<AIRFLOW_VERSION>/constraints-<PYTHON_VERSION>.txt"

apache-airflow-providers-http
apache-airflow-providers-salesforce

Substitute the literal Airflow version your environment runs and the Python version bundled with it. MWAA does not expand shell variables in that file. Test the requirements file against a local Airflow image before you upload it, because a failed install on MWAA surfaces as a partially working environment rather than a clean error.

A backfill is stuck and you need to clear it

You do not need a web login token for this. MWAA exposes the Airflow REST API through a signed AWS API call, so you can drive it from CI or a runbook with normal IAM credentials:

aws mwaa invoke-rest-api 
  --name MyMWAAEnvironment 
  --path "/dags/crm_extract/clearTaskInstances" 
  --method POST 
  --body '{"dry_run": true}'

Start with dry_run set to true so the response tells you which task instances would be cleared before you actually clear them. Note that the resource paths differ between Airflow 2 and Airflow 3 environments, so confirm against the API version your environment exposes.

Deferred tasks stall forever

If deferred tasks stop resuming while the environment reports healthy, suspect the triggerer rather than your DAG. A triggerer that has lost its ability to process triggers can keep heartbeating normally, so the scheduler sees nothing wrong while every deferred task drifts toward timeout. This class of bug has been fixed and re-fixed upstream, so check your Airflow version’s release notes before assuming it is your code.

Common mistakes

• Switching to deferrable operators without enabling include_deferred on the pools that were protecting the API.
• Leaving max_active_runs unset, so a slow run and the next scheduled run compete for the same rate limit budget.
• Treating an empty response page as the end of pagination.
• Storing the incremental watermark in an Airflow Variable and updating it mid-run.
• Calling an API or reading a Variable at DAG parse time, which executes on every parse cycle rather than once per run.
• Transforming during extraction, so a vendor schema change means re-pulling data the API may no longer serve.
• Sizing the environment for peak concurrency when the actual constraint is the vendor’s rate limit.

Best practices

• One pool per vendor, sized to their published limit with headroom, and include_deferred enabled on every one of them.
• Land raw responses to S3 before parsing. Extraction and transformation are separate tasks with separate failure modes.
• Make every task idempotent and window-scoped, so “clear and rerun” is always a safe repair.
• Overlap extraction windows and deduplicate downstream rather than trusting a vendor’s timestamps to be exact.
• Secrets Manager for credentials, with the execution role scoped to specific secret ARNs.
• Alert on row counts and freshness, not just task state. A green DAG that produced 40% of yesterday’s rows is the failure you actually care about. Shipping Airflow’s StatsD metrics into Prometheus, Grafana Cloud or Datadog makes that a dashboard rather than a discovery.
• Define the environment in Terraform or OpenTofu. Recreating an MWAA environment by hand after a bad configuration change is a bad afternoon.

Frequently asked questions

Is MWAA worth it compared to self-hosting Airflow on EC2?

It depends almost entirely on how many people share the platform. MWAA’s premium buys you managed metadata database upgrades, patched images and version migration support. If one engineer maintains three DAGs, self-hosting on a modest VPS is cheaper and the operational load is real but small. Once several teams depend on the scheduler being up, the premium is easy to justify.

Should I use Step Functions instead of Airflow for API pipelines?

Step Functions is genuinely better for event-driven, AWS-service-centric orchestration with modest branching, and it scales to zero. Airflow wins when you need scheduled batch semantics, backfills over historical windows, dependencies between many pipelines, and a UI that non-platform engineers can use to see why last Tuesday failed. Backfill is usually the deciding feature.

Do deferrable operators reduce my AWS bill?

On provisioned MWAA, they reduce worker pressure, which reduces autoscaling into additional worker instances. The base environment cost is unchanged. On a Kubernetes executor setup where each task is a pod, the saving is more direct. Either way, do not adopt them purely for cost without revisiting your pool configuration first.

How do I handle a vendor with no documented rate limit?

Start conservative, one or two concurrent requests, and instrument the response status codes. Raise the pool size gradually and watch for 429s or rising latency. Latency creeping up under load is often the earlier signal, because some vendors throttle by slowing you down before they start rejecting.

Can Airflow read a Retry-After header automatically?

No. Airflow computes retry timing from retry_delay and the backoff settings on the task. If a vendor sends Retry-After, you need to handle it in your own request code or in a custom operator.

What breaks when upgrading to Airflow 3?

The big one for API pipelines is that task code can no longer access the metadata database directly; everything goes through the Task Execution API. Imports also move to the airflow.sdk namespace, and several core operators now live in the standard provider package. Audit custom operators first, since that is where direct database access hides. MWAA requires you to be on the latest Airflow 2 minor version before a major upgrade, so plan two steps.

How many DAGs can one MWAA environment handle?

The binding constraint is usually the metadata database and scheduler CPU, not DAG count. Watch metadata database memory and scheduler CPU utilisation; when either saturates, you either move up an environment class or split into multiple environments. Splitting also gives you blast-radius isolation, which matters more than people expect.

Wrapping up

Running Apache Airflow on AWS for SaaS and API pipelines is mostly not an Airflow problem. The scheduler works. The operators work. What bites is the gap between “the task succeeded” and “the data is correct,” and that gap lives in concurrency settings, pagination logic and retry design rather than anywhere Airflow will warn you about.

If you take one thing away: a green DAG is not a signal that your data is complete. Enable include_deferred on the pools protecting your vendors, terminate pagination on an explicit signal instead of an empty page, make every task idempotent, and alert on row counts. Those four things prevent most of the incidents that never show up as a failed task.

Need help with your Airflow pipelines on AWS?

I work with teams running data and API pipelines on AWS, usually somewhere between “it works but nobody trusts it” and “we need to move off cron.” Things I can help with:

• Reviewing existing DAGs for silent data loss: pagination logic, retry behaviour, pool and concurrency configuration.
• Choosing between MWAA provisioned, MWAA Serverless and self-managed Airflow, with a cost model for your actual schedule rather than a generic comparison.
• Building SaaS extraction pipelines that are idempotent and safely re-runnable, landing raw to S3 with incremental windows that survive failure.
• Cutting MWAA cost: environment right-sizing, NAT Gateway traffic, CloudWatch log volume and dependency install time.
• Airflow 2 to 3 migration audits, focused on custom operators and direct metadata database access.
• Data freshness and volume alerting in Grafana or CloudWatch, so you learn about a partial sync before the business does.

If something specific is broken, send me the DAG file, the task log, or the CloudWatch metrics for the run that went wrong. It is usually faster to look at the real thing than to describe it.

The post Apache Airflow on AWS: Building SaaS and API Pipelines That Don’t Lie to You appeared first on John Nessime.

Acme has nineteen. The assistant retrieved five chunks that happened to mention Acme, because five is roughly how many chunks a retrieval returns, and then it counted them. There is no bug. Nothing failed. Similarity search returned the most similar things and the model described what it was given.

That is the defining problem with preparing CRM data for Amazon Bedrock, and it is upstream of chunk sizes and embedding models. CRM data is structured. Retrieval-augmented generation is built for unstructured text. Feed a table of opportunities through an embedding pipeline and you get a system that is genuinely good at “what did we discuss with this account” and quietly terrible at “how many”, “how much” and “since when”.

This covers the split that fixes it, what a good document actually looks like, the permissions problem nobody notices until it is a problem, and how to know whether any of it works.

Route the question before you build anything

Your users will ask two kinds of question and they need two different mechanisms.

• Narrative questions. “What were the objections on the Acme renewal?” “Summarise our history with this account.” “What did the customer say about pricing?” These live in notes, emails, call summaries and descriptions. Semantic retrieval is exactly right for them.
• Analytical questions. “How many open deals over fifty thousand?” “What is total pipeline this quarter?” “Which accounts have had no activity in ninety days?” These need aggregation, filtering and joins. Retrieval cannot do them and will not tell you it cannot.

The useful part is that you do not have to build the second half yourself. Bedrock Knowledge Bases supports structured data retrieval: you point it at Amazon Redshift or a Glue Data Catalog, and it generates and executes SQL from natural language against your data in place, with no copying or embedding. The counting question goes to SQL, the narrative question goes to the vector store, and both answers are right for the right reasons.

So the first architectural decision is not which chunking strategy to use. It is: which of your fields are text worth embedding, and which are facts worth querying. Get that wrong and no amount of tuning downstream will save it.

One practical note if you take the structured route. Bedrock’s execution role authenticates to Redshift as an IAM identity, and you grant it access the same way you would any other reader. Point it at curated views rather than base tables, exactly as you would a BI tool:

-- Give the knowledge base the same narrow access you'd give a
-- reporting user. A view is a contract; a schema grant is not.
GRANT USAGE ON SCHEMA analytics
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

GRANT SELECT ON analytics.v_opportunities
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

Named columns and clear table names matter more here than anywhere else, because the model is reading your schema to write the SQL. A column called flag_2 will produce exactly the query quality it deserves.

What a good document looks like

For the narrative half, the instinct is to export records to CSV and let the chunker deal with it. That produces chunks that are half a row of one record and half a row of another, with no context about which is which.

Build documents deliberately instead. Three rules that do most of the work:

One record, one document. An opportunity and its notes become a single file. A chunk from it is then always about one thing.

Denormalise the context in. A note that says “they want a two-year term” is useless in isolation. Embed the account name, the opportunity name, the stage and the owner into the same document, so a chunk carries enough context to be retrieved and to be understood once retrieved.

Write it as prose, not as fields. Embedding models were trained on language. Stage__c: Negotiation embeds worse than “The opportunity is at the Negotiation stage.” This feels silly and it measurably improves retrieval.

On chunking itself: start with the default strategy and change it only when evaluation tells you to. If your documents are one record each and reasonably short, semantic or hierarchical chunking buys you less than people expect. The gains that matter come from document construction and metadata, not from chunk size.

The permissions problem

This is the one that turns a nice project into an incident, and it is easy to miss because nothing about it looks broken.

Salesforce has an elaborate sharing model: role hierarchies, sharing rules, territory management, field-level security. A rep sees their accounts and not everyone else’s. Your vector store has none of that. Once a record is embedded, it is a vector like any other, and anyone who can query the knowledge base can retrieve it.

So a rep asks about a competitor deal they were never on the team for, and the assistant helpfully summarises it. Or someone asks about compensation-adjacent records and gets an answer. No alert fires, because from the system’s perspective retrieval worked perfectly.

Three ways out, in increasing order of effort:

1. Only embed what everyone may see. Blunt, easy, and often correct for a first version. Exclude anything sensitive and be explicit that the assistant does not know about it.
2. Filter at retrieval time using metadata. Attach owner, team or region to every document, and pass a filter with each query derived from the caller’s identity. This is the practical answer for most teams.
3. Separate knowledge bases per audience. Heavier to operate, and the only option that gives you a hard boundary rather than a filter you have to remember to apply.

Whichever you pick, decide it before ingestion, because metadata has to be configured at ingest time. Retrofitting a filter you did not plan for means rebuilding the index.

Metadata is most of the value

Metadata filtering narrows the candidate set before similarity search runs. That improves relevance, reduces tokens in the prompt, and gives you the access control lever above. For S3 sources it lives in a sidecar file named after the object it describes:

// acme-renewal.txt.metadata.json, alongside acme-renewal.txt in S3
{
  "metadataAttributes": {
    "object_type": "Opportunity",
    "owner_id": "0051t00000XYZ",
    "account_id": "0011t00000ABC",
    "region": "EMEA",
    "is_closed": false,
    "last_modified_epoch": 1750000000
  }
}

Include, at minimum: what kind of record it is, who owns it, which account it belongs to, whether it is still open, and when it last changed. Those five turn “search everything” into “search the open opportunities this person owns, changed recently”, which is a different product.

Freshness, and the vectors that outlive the record

An embedding is a copy. When the source record changes, the copy is stale. When the source record is deleted, the copy is not.

The consequences are worse in a CRM than in a document store, because CRM records change constantly and confidently answering with last quarter’s stage is indistinguishable from answering correctly. An assistant that cites a deal which closed-lost in March, or a contact who left the company, is not obviously wrong to the person reading it.

• Sync on a schedule that matches how the data is used. Daily is fine for account summaries and useless for live pipeline questions, which should be going to SQL anyway.
• Handle deletes explicitly. Whatever removes a record from your lake must also remove its document from the source bucket, or the vector survives.
• Put the record’s last-modified timestamp in the metadata and surface it in the answer. “As of three days ago” is a small change that prevents a large class of misunderstanding.
• Reconcile document count against source record count periodically. Divergence means orphans.

What not to embed

CRM free-text fields are where people put things. Notes fields contain personal details, occasionally payment information, opinions about customers that were never meant to leave a private conversation, and years of accumulated noise.

Once embedded, all of it is retrievable by anyone with query access, and a vector is not something you can easily grep for a mistake.

• Run detection over free-text fields before ingestion and redact rather than hope.
• Exclude fields nobody asks questions about. Every extra field is noise competing for retrieval slots.
• Consider excluding records closed long ago; they dilute results and rarely answer anything.
• Add Bedrock Guardrails on the output side as a second layer, not as your only control.
• Keep the whole thing on private networking with VPC endpoints if the data warrants it.

There is a governance question here too, and it is worth raising before someone else does: your customers’ personal data is now in a vector store feeding a language model. Whether your privacy notice covers that is not an engineering decision, but it is an engineering responsibility to ask.

Evaluate it before you show anyone

A demo proves the pipeline runs. It does not tell you whether the answers are right, and this is a system whose failure mode is being wrong fluently.

Write thirty real questions from the people who will use it, with correct answers you have verified by hand. Include the analytical ones specifically, because those are where retrieval fails silently. Then run the set after every meaningful change to chunking, metadata or the document format, and compare.

Use the Retrieve API rather than RetrieveAndGenerate while you are tuning. Seeing which chunks came back tells you whether a bad answer is a retrieval problem or a generation problem, and those have completely different fixes.

Common mistakes

• Embedding structured records and expecting aggregation to work.
• Exporting to CSV and letting the chunker split rows arbitrarily.
• Field-value dumps instead of prose, so embeddings have little language to work with.
• No metadata, so every query searches everything and access control is impossible.
• Assuming Salesforce sharing rules follow the data into the vector store.
• Deciding on filtering after ingestion, then discovering it has to be configured at ingest.
• Syncing changes but never removing documents for deleted records.
• Embedding every field because it was easier than choosing.
• No PII detection over free-text notes.
• Tuning chunk size before fixing document construction.
• Judging the system on a demo instead of a scored question set.
• Granting the execution role a whole schema rather than specific views.

Best practices

• Route analytical questions to structured retrieval and narrative questions to the vector store.
• One record per document, with context denormalised in, written as prose.
• Metadata on every document: type, owner, account, open state, last modified.
• Decide the access model before ingestion, and filter at retrieval time by the caller’s identity.
• Redact free-text fields before embedding, and exclude fields nobody queries.
• Delete documents when source records are deleted, and reconcile the counts.
• Surface record age in answers so staleness is visible rather than implied.
• Expose curated views to the execution role, never whole schemas.
• Meaningful table and column names, because the model reads your schema to write SQL.
• A scored evaluation set, run on every change, including questions you expect it to fail.

FAQ

Why does it get counts wrong?

Because retrieval returns the most similar chunks, not all matching records, and the model counts what it was handed. Counting is a query, not a similarity search. Send those questions to structured retrieval over your warehouse instead.

Should I use the Salesforce connector or export to S3 first?

The connector is quicker to stand up. Going via S3 gives you control over document construction, metadata and redaction, which is where most of the quality lives. Start with the connector to learn what people ask, then move to S3 once you know what the documents should look like.

Which chunking strategy is best?

Usually the default, honestly. If each document is one record, chunking has less work to do. Change it in response to evaluation results rather than in advance, because chunking is the most over-discussed and least decisive variable in a CRM RAG setup.

How do I stop people seeing records they shouldn’t?

Metadata filters applied per query based on the caller’s identity, separate knowledge bases per audience, or simply not embedding sensitive records. There is no automatic inheritance of CRM sharing rules, and assuming otherwise is the most consequential mistake in this post.

How often should I re-sync?

Match the questions. Account histories and notes tolerate a daily sync comfortably. Anything needing current numbers should not be coming from embeddings at all. Whatever the interval, make sure deletions propagate.

Do I need a vector database at all?

Only for the narrative half. If every question your users ask is analytical, structured retrieval over Redshift answers all of them with no embedding pipeline, no vector store and no staleness. Plenty of CRM assistants would be better products if they had noticed that.

The one thing to remember

The model will answer either way. It will answer the narrative question well and the counting question badly, in the same tone, with the same citations, and nothing in the response distinguishes them.

So the preparation work is mostly triage: decide which questions are retrieval and which are queries, build documents that carry their own context, attach metadata before you ingest rather than after, and test with real questions whose answers you already know. Chunk size is the last thing to worry about and the first thing everybody argues about.

Building one of these?

The pipeline is usually the easy part; the decisions before it are where these projects succeed or quietly disappoint. Work I take on:

• Designing the split between semantic retrieval and structured querying, so counting questions get correct answers.
• Building the document preparation pipeline from CRM data: record-level documents, denormalised context, metadata, redaction.
• Access control design for knowledge bases, including per-caller metadata filtering and audience separation.
• Freshness and deletion handling so the index does not diverge from the source.
• Curated Redshift views and IAM grants for structured data retrieval.
• Evaluation harnesses with scored question sets, run on every change rather than once before launch.

Send me ten questions your users would actually ask, and I will tell you which of them retrieval can answer.

The post It Counted the Chunks: Preparing CRM Data for Amazon Bedrock appeared first on John Nessime.

Integrations very rarely fail on launch day. They fail in week six. That is roughly when the OAuth token first expires, when an admin adds a field nobody mentioned, when a second tool starts consuming the same API allowance, and when enough records have been deleted for the row counts to visibly diverge. None of those conditions exist in staging, which is why staging said yes.

So this is a Salesforce AWS integration checklist aimed at week six rather than at Tuesday. Every item has a pass test, because “we thought about that” is not the same as “we checked.”

1. Credentials and identity

• The integration has its own Salesforce user. Pass: the username is not a person’s, and nobody uses it interactively.
• That user is API-only and cannot log in through the UI. Pass: someone tried and was refused.
• Its profile grants only the objects and fields you actually read. Pass: you have the field-level permission list and can explain every entry.
• Credentials live in a secrets manager, not in environment variables or config files. Pass: the secret is retrievable by ARN and nothing in the repo contains it.
• You have rotated the credential at least once, in production, and the pipeline survived. Pass: there is a dated record of the rotation and no incident attached to it.
• You know what invalidates the refresh token. Pass: you can name the events that revoke it, including password resets and session policy changes on the integration user.

That last item is the classic week-four failure. A refresh token feels permanent right up until an admin does something routine to the user account it belongs to.

2. Network, versions and deadlines

• You know whether the compute needs to be in a VPC, and why. Pass: if it is, you can name the private resource it reaches.
• If it is in a VPC, S3 traffic goes through a gateway endpoint. Pass: the endpoint exists and the route table references it.
• The Salesforce API version in your endpoints is pinned and current. Pass: it is not a version on a retirement list.
• IAM permissions are scoped to the specific resources involved. Pass: no wildcard on S3 buckets or Secrets Manager paths.
• Login IP restrictions and connected app policies are configured deliberately. Pass: you know whether IP relaxation is on and made that choice on purpose.

The API version item has real dates attached, so it is worth stating plainly rather than leaving as a vague “keep current”. Salesforce retired versions 7.0 through 20.0 in Summer ’22 and versions 21.0 through 30.0 in Summer ’25, and has announced that 31.0 through 40.0 retire in June 2028, with the SOAP login() call retiring separately in June 2027. Retired versions return HTTP 410 on REST rather than degrading gracefully.

Usefully, Salesforce warns you first. Calls to legacy versions come back with a Warning header, so this is detectable today rather than on the morning it breaks:

# If this matches anything, you have a deadline you did not know about.
grep -i "Warning:" integration.log | sort -u

3. Correctness

• Deletes are handled, not just creates and updates. Pass: you can point at the code path and name the mechanism it uses.
• Merged records do not double-count. Pass: you have checked what happens to the losing record of a merge.
• Formula and computed fields are either recalculated downstream or documented as snapshots. Pass: there is a written list of which columns are derived.
• Re-running the same window produces the same result. Pass: you ran it twice and diffed the output.
• One real aggregate reconciles against Salesforce for a closed period. Pass: daily and monthly totals both match, not just daily.
• Timezone handling is explicit at the boundary. Pass: someone can say which timezone the date column is in without guessing.

If daily figures reconcile and monthly ones do not, that is a timezone problem at period boundaries, not missing data. Find it now, because finding it inside somebody’s board pack is a worse day.

4. Failure handling

• Retries are scoped to failures that deserve them. Pass: transient errors retry, data errors do not, and you can show which is which.
• Backoff is exponential with jitter. Pass: a fixed sleep does not appear anywhere in the retry path.
• A partial failure leaves recoverable state. Pass: the watermark advances only after data lands, not when the API call returns.
• You can replay an arbitrary historical window on demand. Pass: someone has done it, in production, with a command you could hand to a colleague.
• Rate limiting is treated as an expected condition. Pass: a 429 or a limit error produces a backoff, not an alert and a failed run.

5. Observability

• The pipeline emits a metric on successful completion. Pass: the metric exists and you can graph it.
• An alarm fires when that metric stops arriving. Pass: you disabled the schedule on purpose and the alarm went red.
• Data freshness is measured on the destination side. Pass: you can answer “how far behind are we right now” from a dashboard.
• Salesforce API allowance consumption is graphed. Pass: there is a percentage on a chart with a threshold line.
• Alerts describe symptoms and link to a runbook. Pass: the alarm description contains a URL somebody could follow at 3am.

The second item is the one people skip and it is the only one that matters. An alarm you have never seen fire is a belief, not a control.

6. Cost

• Every resource carries a cost allocation tag. Pass: Cost Explorer filtered to that tag returns a number.
• Log groups have a retention period. Pass: none of them say “Never expire”.
• The schedule matches how the output is actually used. Pass: somebody named the decision the data supports and the latency it tolerates.
• Average object size in the destination is sensible. Pass: you have looked, and it is not thousands of tiny files per partition.
• A budget alert exists on the pipeline’s tag. Pass: it has a threshold and a recipient who is not on holiday.

7. Operations and handover

• The integration has a named owner. Pass: a person, not a team inbox, and they know.
• A runbook exists covering the three most likely failures. Pass: someone who did not build it followed the runbook successfully.
• Infrastructure is defined as code. Pass: you could rebuild it in a fresh account from the repository.
• Schema drift produces a notification. Pass: adding a field in a sandbox generated an alert somewhere.
• Personal data has an access policy and a retention policy. Pass: both are written down and somebody outside the team has read them.
• You know what a Salesforce sandbox refresh does to it. Pass: you have been through one, or you know what will break when you do.

If you only do five

Nobody clears a thirty-item list before a deadline. These five catch most of what actually goes wrong:

1. Rotate the credential once before go-live. It proves the whole credential path works and it is the failure with the longest fuse.
2. Break it deliberately and confirm someone is told. Disable the schedule, watch the alarm, put it back.
3. Reconcile one real number against Salesforce for a closed period, daily and monthly.
4. Set log retention and one cost allocation tag. Five minutes, and it is the difference between a knowable bill and a mystery.
5. Write the runbook and name the owner. If it is nobody’s, it is nobody’s at 3am too.

FAQ

What actually breaks first in production?

Credentials, usually around the first expiry or the first time an admin touches the integration user. After that, schema drift and API allowance contention with a tool somebody else added.

How do I know if I’m on a retiring API version?

Look at the version string in your endpoint URLs, and search your logs for the Warning header Salesforce returns on legacy versions. Retired versions fail with an HTTP 410 rather than degrading, so there is no soft landing.

Is a staging environment enough to sign off?

No, because the conditions that break integrations barely exist there: real volume, real credential lifecycles, real deletions, and a shared API budget. Staging proves the code runs. It cannot prove the thing survives contact with an organisation.

How often should I revisit the list?

Quarterly for the alarm test and the reconciliation, annually for permissions, credentials and API versions. Put both in a calendar, because an integration that works is one nobody thinks about until it does not.

Does this apply to managed connectors too?

Most of it. You inherit retries and pagination from the vendor, but credentials, permissions, reconciliation, cost, alerting and ownership are all still yours. A managed connector removes code, not accountability.

The one thing to remember

“It works” is a statement about today. “It’s ready” is a claim about week six, and the only honest way to make it is to have tested the things that only happen later: rotate the credential, break the schedule, reconcile a number, follow the runbook.

If you cannot point at evidence for an item, it is not done. It is intended, which is a different thing, and the gap between the two is where you will spend your next quarter.

Want a second pair of eyes before go-live?

Readiness reviews are cheap compared with the incidents they prevent, and they go faster with someone who has seen the same six failures repeatedly. Work I take on:

• Pre-launch readiness review against this checklist, with evidence collected rather than assurances taken.
• Credential and permission audit: integration users, profiles, field-level security, secret storage and rotation.
• Reconciliation harness so row counts and aggregates are checked automatically rather than when someone complains.
• Alerting and runbooks, including a failure drill that proves the alerts reach a human.
• Legacy API version audits ahead of the announced retirement deadlines.
• Cost tagging, log retention and budget alerts so the first month’s bill is not a surprise.

Tell me which items on this list you have evidence for, and I will tell you which of the rest matter most for your setup.

The post It Works Isn’t Ready: A Production Checklist for Salesforce AWS Integrations appeared first on John Nessime.

Eighteen months later there are forty flows, several of them on five-minute schedules because someone wanted fresher dashboards, and AppFlow is a line item that gets mentioned in the monthly review. Nothing was done wrong. The spreadsheet modelled the situation at the moment of the decision and that situation changed.

This is what makes an honest AppFlow vs Lambda vs MuleSoft cost comparison hard: you are not comparing three prices, you are comparing three different shapes, and the cheapest one at two integrations is frequently not the cheapest one at forty. The crossover is the entire decision, and almost nobody models it.

A note on numbers before we start. I am not quoting rate cards. AWS pricing varies by region and changes, MuleSoft publishes no list prices at all, and every comparison article with a neat table of dollar figures is quietly out of date. What does not change is the mechanism each one bills on, and that is what determines whether your bill grows with volume, with integration count, or not at all.

Three shapes

• AppFlow is pure consumption with no floor. You pay per flow run and per gigabyte processed. Two flows cost almost nothing; the bill scales with flows multiplied by frequency.
• Lambda is consumption too, but the meter is nearly irrelevant. The real cost is engineering time to build and own it, and the surrounding AWS services it drags along.
• MuleSoft is a floor. An annual contract sized on capacity or usage entitlements, paid whether you use it or not, largely independent of how many integrations you run inside it.

Draw those as lines against integration count and you get the whole argument. One starts near zero and climbs steeply. One starts moderate and climbs gently. One starts high and is close to flat. Where they cross depends entirely on your situation, which is why a generic answer is worthless and a framework is not.

AppFlow: cheap until the schedule tightens

AppFlow bills per successful flow run plus data processed. The detail that catches people, and it is stated plainly in AWS’s own pricing documentation, is this: a flow run that checks for new data and finds none still counts. You pay for polling, not for movement.

That turns schedule frequency into the dominant variable, and it is the one nobody models because at decision time everything is daily:

Flow runs per month, per flow:

  every 5 minutes   12 x 24 x 30  =  8,640
  every 15 minutes   4 x 24 x 30  =  2,880
  hourly                 24 x 30  =    720
  daily                       30  =     30

Twenty flows, five-minute schedule:  172,800 runs
The same twenty flows, hourly:        14,400 runs

Identical data. Twelve times the flow run charge.

Two more things that are not on the AppFlow line item. Standard S3 request and storage charges apply to whatever it writes, and KMS charges apply to the keys it uses. And on some connectors, running additional concurrent processes to speed a transfer is billed as additional flows.

Where AppFlow wins: a small number of standard SaaS-to-AWS movements on relaxed schedules, where nobody wants to own code. It is genuinely excellent at that, it goes live in an afternoon, and there is no floor to justify.

Where it stops winning: many flows, tight schedules, or transformation logic it cannot express. Schema drift handling is basic, and there is a point where you are working around the tool more than using it.

Lambda: the invoice is not the cost

For CRM-sized workloads, Lambda’s compute charge is close to a rounding error. A function that runs hourly for thirty seconds costs so little that optimising it is a waste of an afternoon.

The costs are elsewhere, and two of them are large.

The surrounding AWS services. NAT gateway data processing if the function sits in a VPC, CloudWatch Logs ingestion, S3 requests from small-file writes, Athena scans downstream. On a real pipeline these routinely exceed the compute charge several times over, and none of them appear under “Lambda” in Cost Explorer.

Engineering time, forever. Not just the build. Pagination, retries, watermarks, delete handling, schema drift, credential rotation, alerting, and someone available when it breaks at an awkward hour. This is the line that gets costed at zero in every comparison and is usually the largest number in the exercise.

The interesting property, though, is that the marginal cost of integration number eleven is much lower than integration number one, if you built a framework rather than eleven scripts. Shared extraction, shared state, shared error handling, one deployment pipeline. That is what makes the Lambda line climb gently instead of steeply. Teams that build eleven bespoke functions get the worst of both worlds: consumption pricing with none of the amortisation.

Where Lambda wins: you already have AWS engineers, you need control over the output shape, and the integration count is going up. Also anywhere the requirement is genuinely unusual, because a managed connector cannot express what it does not have a checkbox for.

Where it does not: a team of two who would rather ship product, or an organisation where the integration needs to be maintainable by people who are not engineers.

MuleSoft: a floor, and what it buys

MuleSoft publishes no list prices. Everything is quote-based, annual, and negotiated, which means any figure you find online is somebody else’s contract and tells you little about yours. It is also worth knowing that the commercial model has been repackaged relatively recently, moving new customers away from the older capacity-based structure toward usage entitlements, so a pricing article from a couple of years ago may describe a product you cannot buy.

Three structural things matter more than the number.

It is a commitment, not a meter. You size it up front and pay for it regardless of use, which is bad if you are unsure and fine if you are certain. Undersizing means overages; oversizing means paying for headroom.

Separate meters do not offset each other. Integration capacity, API traffic and add-on modules are metered independently, so spare room in one does not cover a spike in another. Model each axis separately or the renewal will be a surprise.

Renewal escalation is normal. Enterprise contracts typically carry annual uplift clauses, so the three-year total is meaningfully above three times year one. Negotiate that at signature, because the leverage is gone afterwards.

Now the fair part, because a post that treats MuleSoft as merely expensive is not being honest. What the floor actually buys: API management and governance as a first-class capability rather than something you assemble; a connector catalogue covering legacy and on-premise systems that neither AWS option touches; integrations that a trained analyst can maintain without a software team; audit trails, support contracts and SLAs that satisfy people who ask for them in writing. At a hundred integrations across several business units, the fixed cost is spread thin and the governance is the point.

The cost that is easy to miss is skills. DataWeave and the Anypoint tooling are proprietary, so you are hiring for a specific platform and your switching cost rises with every integration you build. That is not an argument against it; it is an argument for deciding deliberately rather than drifting in.

Where MuleSoft wins: many integrations, many teams, on-premise or legacy systems in the mix, and governance requirements with a compliance function attached to them.

Where it does not: three integrations, one team, all cloud. The floor will never amortise and you will spend the first year justifying it.

Modelling the crossover

Build the comparison over three years rather than one month, and project the inputs rather than freezing them.

1. Count integrations at year three, not today. Ask the people requesting them. The answer is always higher than the current number and it is the variable the whole model turns on.
2. Write down the schedule for each one, and defend it. “Every five minutes” is usually a preference, not a requirement. For AppFlow this is the difference between a small bill and a large one.
3. For AppFlow, multiply flows by runs per month. Apply current rates yourself, then add S3 requests and KMS.
4. For Lambda, cost the engineering explicitly. Build days plus ongoing maintenance days per year, at a loaded rate. If that number embarrasses you, it is probably the accurate one.
5. For MuleSoft, get an actual quote, model each meter separately, and include renewal escalation across the term.
6. Add the cost of being wrong. How much does switching cost in each direction? Consumption services you can walk away from. An annual contract and a proprietary skill set you cannot.

Then plot all three. You are looking for where the lines cross and whether your projected integration count sits comfortably on one side of it or uncomfortably near it. Near a crossover, pick the option that is cheaper to reverse.

Common mistakes

• Comparing one month at today’s integration count.
• Costing engineering time at zero on the build-it-yourself option.
• Not modelling flow run frequency, which is the single largest AppFlow variable.
• Forgetting that AppFlow charges for polling runs that find nothing.
• Attributing only the Lambda invoice to the Lambda option, ignoring NAT, logs and S3 requests.
• Building eleven bespoke Lambdas instead of one framework, so nothing amortises.
• Using published MuleSoft prices from an article, when no list prices exist and the packaging has changed.
• Ignoring renewal escalation when comparing an annual contract against consumption pricing.
• Treating a hybrid as a failure of nerve rather than the usual right answer.
• Choosing for the architecture you want rather than the team you have.

How I’d decide

Start with AppFlow if the integrations are standard, few, and not urgent. It is the lowest-commitment option and you can leave whenever you like.

Move to Lambda when you have AWS engineers, the count is climbing, and you need control over the output. Build it once as a framework, not repeatedly as scripts, or you get the cost profile without the benefit.

Consider MuleSoft when the problem is organisational rather than technical: many teams, legacy systems, governance requirements, and integrations that need to outlive the person who built them.

And be relaxed about mixing them. Plenty of sensible estates run AppFlow for the boring standard movements and custom code for the two integrations that are genuinely specific. Insisting on one tool for everything is a preference, not an architecture.

FAQ

Which is cheapest overall?

None of them, universally. AppFlow is cheapest at low integration counts and relaxed schedules. Lambda is cheapest at scale if you already have the engineers and build a shared framework. MuleSoft is cheapest only when the fixed cost is spread across many integrations and teams. The question is which shape fits your trajectory.

Why did my AppFlow bill grow when data volume didn’t?

Because you are billed per flow run, including runs that check for new data and find none. Somebody tightened a schedule, or the flow count grew. Both multiply the run charge without moving a single extra record.

Is Lambda really cheaper than AppFlow?

On the invoice, usually yes. In total cost, only once the engineering amortises across several integrations. For one or two simple flows, AppFlow is almost always the better economic answer, and the “we’ll just write a Lambda” instinct is a false economy at that scale.

Why can’t I find MuleSoft’s prices?

Because there aren’t any published. It is sold through quotes with negotiated discounts, so figures you find online are other people’s contracts. Get a quote, model each meter separately, and negotiate the renewal escalation before you sign rather than after.

Can I start on one and move later?

Between the two consumption options, fairly easily, especially if your destination schema is defined by you rather than by the tool. Off an annual contract with proprietary tooling, much less easily, which is exactly why the switching cost belongs in the model rather than as a footnote.

The one thing to remember

You are not choosing the cheapest tool. You are choosing a cost shape, and committing to how it behaves as your integration count and your schedules change over the next three years.

So model the trajectory, not the snapshot. Put engineering time in the spreadsheet at a real rate. And when the lines are close together, take the option you can walk away from, because being wrong cheaply is worth more than being right narrowly.

Need the model built for your numbers?

This decision usually gets made under time pressure with incomplete inputs, and then lived with for years. Work I take on:

• Building the three-year cost model for your actual integration inventory, schedules and volumes, with engineering time costed honestly.
• Auditing an existing AppFlow estate for flow count, schedule frequency and the charges that sit outside the AppFlow line item.
• Designing a shared extraction framework so custom integrations amortise instead of multiplying.
• Migrating between approaches without downtime, including keeping the destination schema stable so the choice stays reversible.
• Reviewing an integration platform contract for meter structure, overage terms and renewal escalation before you sign.
• Hybrid architectures where managed connectors handle the standard movements and code handles the awkward ones.

Send me your integration list with schedules and rough volumes, and I will tell you which shape you are actually buying.

The post Cheapest at Two Flows, Worst at Forty: AppFlow vs Lambda vs MuleSoft on Cost appeared first on John Nessime.

Then you check the invocation count and it is also zero, and has been since the day somebody disabled an EventBridge rule while cleaning up a different stack. The job has not run in three weeks. It never errored because it never started.

This is the shape of almost every integration monitoring failure: a healthy graph and a stopped job produce identical output. Zero errors is what success looks like and it is also what absence looks like, and if the only thing you measure is failure, the two are indistinguishable.

This is about how to monitor Salesforce integrations so that stopping is as loud as breaking. Four signals worth emitting, how to get them into CloudWatch cheaply, the alarm configuration that actually fires, and what belongs on a Grafana dashboard once you have them.

Four signals, not one

Most integration monitoring stops at errors and duration, because those come free from Lambda or your container platform. Both are worth having and neither answers the question anybody actually asks, which is “is the data right”.

The four that do:

• Liveness. Did it run at all?
• Volume. Did it move a plausible amount of data?
• Freshness. How old is the newest record on the destination side?
• Budget. How much of Salesforce’s daily API allowance have you spent?

Errors are a fifth, and the least interesting, because errors are the failure mode that already announces itself.

Liveness: the alarm that has to fire on silence

Emit a metric on every successful completion. A single count, value 1. Then alarm when it stops arriving.

The trap is in the CloudWatch defaults. TreatMissingData has four settings, and the default is missing, which sends the alarm to INSUFFICIENT_DATA when nothing arrives. That state is not ALARM. Nothing pages. Your dashboard shows a grey alarm that most people read as “fine”.

So heartbeat alarms need breaching. That much is standard advice. Here is the part that is not: even with breaching set, a heartbeat alarm can still fail to fire. CloudWatch evaluates over a range wider than your evaluation periods, and if it finds any real data point in that wider range, those override the missing ones. On a job that runs hourly, a successful run from earlier can keep the alarm quiet through several missed runs.

The robust version uses metric math to turn absence into a real zero, so there is no missing data to interpret:

aws cloudwatch put-metric-alarm 
  --alarm-name "opportunity-sync-not-running" 
  --alarm-description "No completed run in the last 90 minutes" 
  --comparison-operator LessThanThreshold 
  --threshold 1 
  --evaluation-periods 1 
  --treat-missing-data breaching 
  --alarm-actions "$SNS_TOPIC_ARN" 
  --metrics '[
    {
      "Id": "runs",
      "MetricStat": {
        "Metric": {
          "Namespace": "SalesforceSync",
          "MetricName": "RunCompleted",
          "Dimensions": [{"Name": "Integration", "Value": "opportunity-sync"}]
        },
        "Period": 5400,
        "Stat": "Sum"
      },
      "ReturnData": false
    },
    {
      "Id": "filled",
      "Expression": "FILL(runs, 0)",
      "ReturnData": true
    }
  ]'

FILL(runs, 0) substitutes a zero wherever the metric has no data point, so the alarm always has something real to compare against the threshold. The window is deliberately longer than the schedule: an hourly job gets ninety minutes, so one late run does not wake anybody.

Then do the thing everyone skips: disable the schedule in a test account and confirm the alarm actually goes red. An untested alarm is a belief, not a control.

Volume and freshness: is the data actually moving

A job can complete successfully and process nothing. A credential with the wrong field-level permissions, a filter that silently matches nothing, a watermark that got written before the data landed: all of these produce a clean run and an empty result.

Volume is the count of records read and written per run. Alarming on it is harder than liveness because the right number varies: a quiet Sunday legitimately looks like a broken Tuesday. Two approaches that work. Use CloudWatch anomaly detection, which learns the daily and weekly shape and alarms on departures from it. Or set a crude floor that only catches the catastrophic case, which is usually zero, and accept that you will not catch a fifty percent drop.

I would start with the crude floor. It is five minutes of work and catches the failure that actually happens.

Freshness is the better metric and almost nobody emits it. At the end of each run, query the destination for the newest record’s modified timestamp, subtract it from now, and publish the difference in seconds. That single number answers the business question directly: how far behind Salesforce are we right now?

It also collapses several failure modes into one signal. A stopped job, a job that runs but writes nothing, a job stuck retrying, a job silently filtered down to zero rows: all of them show up as lag climbing. If you only add one metric from this post, add this one.

Budget: watch the Salesforce allowance

Your org has a daily API allowance shared across every integration touching it. Exceeding it does not just break your sync; it breaks marketing automation, support tooling, and whatever else somebody connected two years ago.

Enforcement is initially soft, and then it is not: past a protection threshold, calls come back as 403 with REQUEST_LIMIT_EXCEEDED until the rolling window drains. By that point you are in an incident that spans several teams.

The good news is that this costs nothing to observe. Salesforce returns your current consumption on ordinary REST responses in a header, so you get it on calls you were making anyway:

# Sforce-Limit-Info: api-usage=1212/15000
#
# Free: no extra API call, which matters when the thing you are
# measuring is an API budget. Add real error handling before
# shipping this; the header is not guaranteed on every response.
raw = response.headers.get("Sforce-Limit-Info", "")
used, allowed = (int(v) for v in raw.split("api-usage=")[1].split("/"))
emit("ApiUsagePercent", used / allowed * 100)

For a fuller picture, the /services/data/vXX.X/limits endpoint returns every allocation in the org, including DailyApiRequests with its max and remaining values. It needs the View Setup and Configuration permission and the numbers lag by a few minutes. Poll it on a schedule rather than per request, and alarm on percentage consumed rather than absolute calls, so the alarm survives a licence change.

Graph consumption by integration if you can attribute it. The conversation about which team is burning the allowance goes very differently when there is a chart.

Getting the metrics in without a bill shock

You can call PutMetricData directly, and it works, and it is a synchronous API call in the hot path of your job that can fail or add latency. Custom metrics are also charged per metric per month, and a metric is every unique combination of name and dimensions, so a dimension with high cardinality gets expensive quietly.

The better default is Embedded Metric Format: write structured JSON to stdout and CloudWatch extracts the metrics from your logs. No API call, no added latency, and the log line stays queryable in Logs Insights alongside the metric.

{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [{
      "Namespace": "SalesforceSync",
      "Dimensions": [["Integration"]],
      "Metrics": [
        { "Name": "RunCompleted",     "Unit": "Count"   },
        { "Name": "RecordsWritten",   "Unit": "Count"   },
        { "Name": "SourceLagSeconds", "Unit": "Seconds" },
        { "Name": "ApiUsagePercent",  "Unit": "Percent" }
      ]
    }]
  },
  "Integration": "opportunity-sync",
  "RunId": "a41c9f",
  "RunCompleted": 1,
  "RecordsWritten": 4127,
  "SourceLagSeconds": 312,
  "ApiUsagePercent": 8.1
}

Note what is a dimension and what is not. Integration is a dimension because it has a handful of values and you want to alarm per integration. RunId is a plain field: searchable in the logs, and not a dimension, because making it one would create a new metric on every run. That distinction is the whole cost story.

Grafana on top

CloudWatch dashboards are fine and Grafana is better for this, for three reasons: you can put Salesforce metrics next to your warehouse and application metrics on one screen, the alerting is more expressive, and non-engineers will actually open it.

Add CloudWatch as a data source using an IAM role rather than access keys, scoped to cloudwatch:GetMetricData, cloudwatch:ListMetrics and the Logs Insights permissions if you want log panels. One honest cost note: Grafana queries CloudWatch through the metric data API, which is billed per metric requested, so a busy dashboard on a short refresh interval is a real line item. Set a sane refresh, avoid auto-refresh on wall displays, and use the caching in Grafana’s CloudWatch data source.

What goes on the dashboard, in order down the page:

• Freshness per integration, as a stat panel with thresholds. This is the panel people look at.
• Time since last successful run, per integration.
• Records processed, over a window long enough to show the weekly shape.
• API allowance consumed, as a percentage with a threshold line.
• Errors and duration, at the bottom, where they belong.

One dashboard, one screen, no scrolling. A dashboard nobody can read at a glance during an incident is decoration.

Alerts people don’t ignore

Decide deliberately where alerting lives. CloudWatch alarms are more reliable, because they keep working when Grafana is down, and Grafana alerts are more flexible and can span data sources. My default is CloudWatch for the small number of alerts that page someone, and Grafana for everything informational.

Three things that separate a useful alert from noise. Alarm on the symptom, not the cause: “Opportunity data is more than two hours stale” is actionable in a way “Lambda errors greater than zero” is not. Put the runbook link in the alarm description, since that field ends up in the notification and is the only documentation anybody reads at midnight. And use composite alarms to suppress the cascade, so a Salesforce outage produces one page rather than nine.

Common mistakes

• Monitoring only errors, so a stopped job looks identical to a healthy one.
• Leaving TreatMissingData at its default on a heartbeat alarm.
• Setting it to breaching and assuming that is sufficient, without handling the evaluation range.
• Never testing that an alarm fires by actually breaking something.
• No freshness metric, so nobody can answer how far behind the data is.
• Ignoring API allowance until an integration you do not own breaks.
• High-cardinality dimensions such as record ID or run ID, and the bill that follows.
• Calling PutMetricData synchronously in the job’s critical path.
• Alerting on causes rather than on user-visible symptoms.
• A dashboard that requires scrolling and interpretation during an incident.

Best practices

• Emit liveness, volume, freshness and API budget from every integration, as a standard.
• Heartbeat alarms with breaching plus FILL(), and a window longer than the schedule.
• Freshness as the headline metric, because it maps to a question the business asks.
• Embedded Metric Format rather than direct API calls.
• Low-cardinality dimensions; everything else stays a log field.
• Alarm on percentage of the API allowance, not absolute calls.
• IAM roles for the Grafana data source, and a refresh interval you have costed.
• Runbook links in alarm descriptions.
• Composite alarms to collapse cascades into one page.
• A quarterly test that breaks each integration on purpose and confirms someone gets told.

FAQ

Why didn’t my CloudWatch alarm fire when the job stopped?

Almost certainly TreatMissingData. The default sends the alarm to INSUFFICIENT_DATA, which is not ALARM and pages nobody. Set it to breaching, and wrap the metric in FILL() so there is no missing data for CloudWatch to reinterpret.

What’s the single most useful metric to add?

Freshness: how old the newest record on the destination side is. It catches stopped jobs, empty runs, stuck retries and silent filtering with one number, and it is the only one of these metrics a non-engineer can interpret.

CloudWatch dashboards or Grafana?

Grafana if you already run it, because you can put Salesforce, warehouse and application metrics on one screen. CloudWatch if you do not, because a second system to operate is not free. Either way keep the paging alarms in CloudWatch so they survive Grafana being down.

Will custom metrics be expensive?

Only if you make them so. Cost scales with unique name-and-dimension combinations, so a handful of metrics dimensioned by integration name is negligible. Adding a run ID or record ID as a dimension is how the bill grows without anyone noticing.

How do I monitor a third-party connector I can’t add code to?

Monitor the destination instead. A scheduled job that queries the target for the newest record’s timestamp and emits it as a freshness metric works regardless of what wrote the data, and it is arguably a better test because it measures the outcome rather than the process.

The one thing to remember

Absence of failure is not evidence of success. An integration that stopped produces exactly the same error graph as one working perfectly, and every default in your monitoring stack is tuned to stay quiet when data stops arriving rather than to shout about it.

So measure the thing you actually care about. Not “did it error” but “how stale is the data right now”, alarmed in a way that fires on silence, and tested by deliberately breaking it. Everything else on the dashboard is supporting evidence.

Want this built properly?

Integration monitoring tends to get added after the first silent failure, which is one failure too late. Work I take on:

• Instrumenting Salesforce integrations with liveness, volume, freshness and API budget metrics via CloudWatch.
• Auditing existing alarms for the ones that cannot fire, and fixing the missing-data handling.
• Building the Grafana dashboard and data source, including cost-aware query and refresh configuration.
• Alert design: symptom-based alarms, composite alarms to suppress cascades, runbooks attached where people will read them.
• API allowance monitoring and attribution across multiple integrations sharing one org.
• Running a failure drill so you know the alerting works before you need it.

Tell me how you would currently find out that a sync stopped, and I will tell you how long it would take.

The post Zero Errors, Zero Records: Monitoring Salesforce Integrations with CloudWatch and Grafana appeared first on John Nessime.

“EC2 – Other” is where AWS files NAT gateway charges, along with EBS, elastic IPs and assorted data transfer. Your Salesforce sync runs in a private subnet, so every byte it pulls from Salesforce and every byte it writes to S3 goes through a NAT gateway, which bills an hourly rate plus a charge per gigabyte processed. None of that appears under Lambda, or under S3, or under the tag you put on the pipeline.

That is the recurring theme when you look at the AWS cost of Salesforce data pipelines: the spend is almost never in the place you have been optimising. People tune Lambda memory settings while the bill is being driven by data movement, request counts, log ingestion and scanned bytes.

Five levers, in the order I would pull them.

First, find out where it actually goes

Every hour spent guessing is an hour not spent on the line item that matters. Tag everything the pipeline owns with a consistent cost allocation tag, activate it in the billing console, and wait for it to backfill.

Then group Cost Explorer by usage type rather than by service. Service-level grouping is what hides NAT charges inside a generic bucket; usage type is where the truth lives.

# Substitute a whole billing month. Grouping by USAGE_TYPE is the
# point: "EC2 - Other" tells you nothing, "NatGateway-Bytes" does.
aws ce get-cost-and-usage 
  --time-period Start=YYYY-MM-01,End=YYYY-MM-01 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["EC2 - Other"]}}'

Add S3 Storage Lens for the bucket. It tells you object count and average object size per prefix, which is the number that predicts your request bill and your Athena bill better than total storage does.

Lever 1: get the traffic off the NAT gateway

Usually the largest single saving available, and often a change you can make before lunch.

Start with the question nobody asks: does this function need to be in a VPC at all? A Lambda that calls the Salesforce REST API and writes to S3 touches nothing private. If it is in a VPC because that felt like the secure default, taking it out removes the NAT gateway from the path entirely and costs you nothing.

If it genuinely needs the VPC, because it also talks to RDS or something inside your network, add a gateway VPC endpoint for S3. AWS charges no hourly fee and no data processing fee for gateway endpoints, so S3 traffic simply stops flowing through NAT:

aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0

Two honest caveats. Gateway endpoints exist only for S3 and DynamoDB; everything else needs an interface endpoint, which has an hourly charge per availability zone plus a per-gigabyte charge. Still cheaper than NAT for AWS-service traffic, not free.

And the one specific to this workload: endpoints do nothing for your Salesforce traffic. Salesforce is not an AWS service, so those calls still traverse NAT. If you are pulling large volumes out of Salesforce from inside a VPC, the endpoint fixes the S3 half of the bill and leaves the other half untouched. That is the strongest argument for getting the function out of the VPC rather than papering over it.

Lever 2: fewer, bigger files

A sync running every five minutes across twelve Salesforce objects writes a lot of very small files. Storage for those is negligible. The costs are elsewhere and they compound.

• PUT requests are billed per request and cost meaningfully more than GETs. Millions of tiny writes is a request bill, not a storage bill.
• Athena opens every file it reads. Thousands of small objects per partition turn a fast scan into a slow, expensive one.
• Glue crawlers take longer over more objects, and they are billed by the second.

The fix is compaction: a scheduled job that rewrites each day’s partition into a small number of larger Parquet files and deletes the fragments. If you are on Iceberg, its compaction procedure does this for you and is the better reason to adopt it than any of the ones usually cited.

One trap worth flagging, because it is the opposite of what people expect. Lifecycle rules that move small objects to S3 Standard-IA can increase your bill: those classes have a minimum billable object size of 128 KB, so a 10 KB file is billed as 128 KB, plus a 30-day minimum duration. And S3 Intelligent-Tiering will not help either, since objects under 128 KB are never auto-tiered and stay at Frequent Access rates. Compact first, then think about storage classes. In the other order you are paying more for the privilege.

Lever 3: log less

CloudWatch Logs is billed primarily on ingestion volume, and on a high-frequency pipeline it is routinely larger than the compute it is logging. A sync that logs every record it processes, running every five minutes, generates an astonishing amount of data whose only reader is a retention policy.

• Set a retention period on every log group. The default is to keep logs forever, and log groups created automatically by Lambda inherit it. This is the single most common unclaimed saving on an AWS account.
• Log per run, not per record. One structured summary line with counts, durations and outcome tells you more during an incident than ten thousand lines of per-record chatter.
• Turn off debug logging in production, and make the level an environment variable so raising it is a deliberate act rather than a redeploy.

Check the log classes available before you settle. There is a cheaper class aimed at logs you rarely query, which suits pipeline output well, with fewer features in exchange.

Lever 4: scan less

Athena bills on bytes scanned, so its cost is a property of your data layout, not your SQL. Three things move it, in order.

Parquet rather than JSON or CSV. Columnar means a query touching four columns reads four columns. This is usually a large multiple, not a percentage.

Partitions that match how people filter. If every query has a date range, partition by date. If nobody ever filters by region, partitioning by region just creates more small files.

Partition projection instead of a crawler. Athena works partitions out from the path pattern rather than reading them from the catalog, which removes the crawler cost, removes the repair step, and means partitions are never stale:

ALTER TABLE salesforce.opportunity SET TBLPROPERTIES (
  'projection.enabled'             = 'true',
  'projection.extract_date.type'   = 'date',
  'projection.extract_date.range'  = '2024-01-01,NOW',
  'projection.extract_date.format' = 'yyyy-MM-dd',
  'storage.location.template' =
    's3://acme-lake/raw/salesforce/opportunity/extract_date=${extract_date}/'
);

Set a per-query and per-workgroup data scanned limit while you are in there. It converts a runaway SELECT * from a surprise on the bill into an error someone sees immediately.

Lever 5: run it less often

The cheapest work is work you do not do. A sync running every five minutes because five minutes felt responsive, feeding a dashboard people open once each morning, is paying for freshness nobody consumes.

Ask what decision the data supports and how quickly it needs to be right. Hourly is enough for most reporting, and dropping from five minutes to hourly cuts invocations, requests, log volume and small-file creation all at once. It is the only lever here that improves four line items with one change.

Two false economies while you are looking. Cutting Lambda memory usually costs more, not less, because the function runs proportionally longer and you also lose CPU. And moving from Athena to a provisioned engine to escape per-query pricing only pays off at genuinely high query volume; below that you have swapped a variable cost for a fixed one and called it a saving.

Common mistakes

• Optimising compute because it is the part you understand, while data movement drives the bill.
• Reading Cost Explorer grouped by service, so NAT charges stay hidden inside “EC2 – Other”.
• Running a Lambda in a VPC when nothing it touches is private.
• Adding a gateway endpoint and expecting it to help traffic bound for Salesforce.
• Millions of small objects, and treating it as a storage problem rather than a request problem.
• Lifecycle rules that move sub-128 KB objects to an infrequent access class and increase the bill.
• No retention policy on log groups.
• Logging per record on a five-minute schedule.
• JSON in the lake because it was easier to write.
• Running a Glue crawler on a schedule when partition projection would do it for nothing.
• No cost allocation tags, so nobody can attribute anything.
• Syncing far more frequently than anybody reads the output.

Best practices

• Tag every resource the pipeline owns, and group Cost Explorer by usage type.
• Keep functions out of VPCs unless they need private resources.
• Gateway endpoints for S3 and DynamoDB on every VPC that touches them.
• Compact partitions on a schedule; measure average object size, not just total bytes.
• Compact before choosing storage classes, never the other way round.
• Retention on every log group, set at creation, enforced in your IaC.
• One structured summary log line per run.
• Parquet, partitioned the way people actually filter, with partition projection.
• Per-query and per-workgroup scan limits in Athena.
• Match the schedule to the decision the data supports.
• A budget alert on the pipeline’s tag, so the next surprise arrives as a notification.

FAQ

Why is “EC2 – Other” large when I run no EC2?

It is a catch-all that includes NAT gateway hours and data processing, EBS, elastic IPs and data transfer. Filter Cost Explorer to that service and group by usage type; if you see NAT gateway bytes near the top, that is your pipeline paying to move its own data.

Should my sync Lambda be in a VPC?

Only if it needs to reach something private, such as a database in your own subnets. Calling an external API and writing to S3 does not require it, and putting it in a VPC adds NAT charges and cold start complexity for no security benefit you can articulate.

Will Intelligent-Tiering fix my small files?

No. Objects under 128 KB are never auto-tiered and stay at Frequent Access rates. Standard-IA is worse, because it has a 128 KB minimum billable size, so small objects are billed as if they were larger. Compaction is the fix; storage class is a decision for afterwards.

Glue or Lambda for the extraction?

Lambda for anything that fits in its execution limits, which covers most incremental CRM syncs, and it is cheaper for short frequent runs. Glue when you genuinely need Spark for volume or transformation. Glue billed by the second for a job that runs for ninety seconds is a poor trade.

What is the fastest saving to make?

Retention policies on log groups, then the S3 gateway endpoint. Both take minutes, neither carries risk, and between them they usually account for more than whatever you were planning to spend a week optimising.

The one thing to remember

Data pipeline bills are dominated by moving bytes, counting requests, ingesting logs and scanning storage. Compute is the part you think about and rarely the part you pay for.

So do the boring thing first: tag it, group Cost Explorer by usage type, and read the top five lines. The answer is almost always sitting there in a category you have been scrolling past, and it is usually cheaper to fix than to keep paying for.

Bill larger than it should be?

Pipeline cost work has a good ratio: a few days of investigation usually pays for itself in the first month. Work I take on:

• Cost breakdown of an existing Salesforce or CRM pipeline, by usage type, with a ranked list of what to fix and what it saves.
• Network path work: VPC endpoints, getting functions out of VPCs, and removing NAT from the data path.
• Storage layout: compaction jobs, partitioning strategy, Parquet or Iceberg conversion.
• Athena cost control: partition projection, workgroup scan limits, replacing scheduled crawlers.
• Logging and retention cleanup across accounts, which is usually the quickest win available.
• Cost allocation tagging and budget alerts so the next increase is noticed early.

Send me a month of Cost Explorer grouped by usage type and I will tell you where the money is going.

The post You Don’t Run Any EC2: Cutting the AWS Cost of Salesforce Data Pipelines appeared first on John Nessime.

You pull a sample of the extra rows and they all look fine. Real opportunity IDs, real amounts, real owners. You paste one into Salesforce and get nothing. The record was deleted eight months ago. So were the others.

Your incremental extract has never deleted a row in its life. It asks Salesforce for everything where SystemModstamp is greater than the last run, and a record that no longer exists cannot come back in that result set. So the lake only ever grows, the gap widens a little every week, and nothing anywhere reports an error.

That is the defining problem when you build a Salesforce data lake on AWS, and it is barely mentioned in the tutorials, which mostly stop once data lands in S3. Landing the data is the easy part. Keeping it equal to the source is the work.

This covers the ingestion options and their honest trade-offs, how to structure the lake, how to handle deletes properly (harder than it sounds), the fields that go stale without telling you, staying inside your API budget, and the query layer on top.

Why Salesforce is an awkward source

Four properties make it different from replicating a database, and every design decision below follows from them.

• Deletes are soft, then permanent. A deleted record moves to the Recycle Bin, where it is visible for a limited window and then gone. Your ability to detect a deletion expires.
• Some fields are computed at read time. Formula fields are not stored. Salesforce works them out when you query, which means what you extracted is a snapshot, not a value that stays true.
• The API is a metered resource. Your org has a daily request allowance shared with every other integration. A careless extract can starve the tools sales actually uses.
• The schema changes without warning. Admins add fields as part of their normal job. Nobody tells the data team.

Choosing how to get the data out

Four realistic paths, roughly in order of how much you have to operate yourself.

Amazon AppFlow is the native option: a managed connector that moves Salesforce objects into S3 on a schedule or on events, with no infrastructure to run. It handles both scheduled batch extracts and Salesforce Change Data Capture event flows, and it supports AWS PrivateLink to Salesforce, so the traffic does not have to cross the public internet. That last point matters more than it usually gets credit for.

Where it gets thin: schema drift handling is basic, and mapping large numbers of objects through the console gets tedious fast. Define the flows in Terraform or CloudFormation from the start rather than clicking them, or you will end up with production configuration nobody can reproduce.

A managed ELT vendor such as Fivetran or Airbyte handles schema drift, deletes and incremental logic for you, which is genuinely most of the hard work in this post. You pay per row or per connector, and on a large Salesforce org that number gets attention. Worth pricing against the engineering time before dismissing it.

Rolling your own on the Bulk API gives you total control and makes you responsible for everything: pagination, retries, API budget, delete detection, schema evolution. I would only choose this when an existing tool genuinely cannot express what you need, and I would expect it to be a real service with monitoring rather than a Lambda someone wrote in an afternoon.

Zero-copy sharing via Salesforce Data Cloud avoids replication entirely by querying Salesforce-held data from your warehouse. Attractive when compliance objects to copying CRM data, and it carries its own licensing and latency questions. Check what your Salesforce contract actually includes before designing around it.

Structure the lake so mistakes are cheap

Two layers, and the discipline to keep them separate.

Raw is append-only and never edited. Every extract lands as a new partition, in Parquet, exactly as Salesforce returned it. You do not deduplicate here, you do not fix types here, and you certainly do not delete here. Raw is your ability to rebuild everything downstream when you discover a logic bug six months in, which you will.

# Partition by extraction date, not by a business date. A bad run
# is then one partition to drop rather than a full-table repair.
s3://acme-lake/raw/salesforce/opportunity/extract_date=YYYY-MM-DD/

Curated is what people query: deduplicated to current state, typed properly, deletes applied. Build it as views over raw to begin with, and only materialise the ones that are genuinely too slow. Materialising early gives you a second copy of the truth to keep in sync, which is the problem you are already trying to solve.

-- Current state from an append-only raw layer: keep the most
-- recent version of each record and drop anything flagged deleted.
CREATE OR REPLACE VIEW curated.opportunity AS
SELECT *
FROM (
    SELECT o.*,
           ROW_NUMBER() OVER (
               PARTITION BY id ORDER BY systemmodstamp DESC
           ) AS rn
    FROM raw.opportunity o
)
WHERE rn = 1
  AND is_deleted = false;

Register both in the Glue Data Catalog so Athena and Redshift Spectrum see the same definitions. If you expect frequent updates and deletes rather than pure appends, an open table format like Apache Iceberg is worth the extra setup: row-level deletes, schema evolution and time travel are exactly the operations this workload needs, and they are painful to hand-roll on plain Parquet.

Handling deletes properly

Here is the section that matters. There are three mechanisms and you need more than one, because each has a hole.

1. Query the Recycle Bin

Standard SOQL hides deleted records. The REST API’s queryAll endpoint, the SOAP queryAll() call, or ALL ROWS in Apex all return them, and IsDeleted exists on virtually every object even though it does not appear in Setup.

-- Against /services/data/vXX.X/queryAll, not /query.
-- Substitute your own high-water mark for the placeholder.
SELECT Id, IsDeleted, SystemModstamp
FROM Opportunity
WHERE IsDeleted = true
  AND SystemModstamp > {last_run_utc}

The hole: records sit in the Recycle Bin for fifteen days by default. Salesforce Classic offers an extended retention setting that pushes it to thirty. But the bin also has a capacity tied to your org’s storage allocation, and when it fills, the oldest records are purged early to make room. No alert, no warning. So your detection window is not fifteen days. It is fifteen days or less, and you find out which during a mass-delete week.

2. Change Data Capture

Salesforce CDC publishes create, update, delete and undelete events as they happen, and AppFlow can consume them. This is the cleanest mechanism, because a delete arrives as an event rather than being inferred from an absence.

The hole: it is a streaming subscription, so if your consumer is down long enough, you miss events. Event replay windows are finite. CDC is a good primary mechanism and a bad only mechanism.

3. Periodic full ID reconciliation

This is the safety net, it catches everything the other two miss, and it is the one people skip because it feels crude. Pull nothing but the Id column for the whole object. One narrow field over the Bulk API is cheap even on millions of rows.

SELECT Id FROM Opportunity

Then diff it against the lake. Whatever is on your side and not on theirs no longer exists, whatever the reason:

-- Rows the lake still believes in and Salesforce does not.
SELECT l.id
FROM curated.opportunity l
LEFT JOIN staging.opportunity_ids s
       ON s.id = l.id
WHERE s.id IS NULL;

Run it weekly on your important objects. Soft-delete the differences in the curated layer rather than hard-deleting from raw, so you keep the audit trail.

One more case this catches that nothing else does cleanly: merges. When an admin merges two Accounts, Contacts or Leads, the losing record is deleted and gets a MasterRecordId pointing at the survivor. If your lake keeps the loser, you are double-counting a customer, and the arithmetic is wrong in a way that looks like a data quality problem rather than a pipeline problem.

The other thing that goes stale silently

Formula fields are not stored anywhere. Salesforce computes them when you ask. So the value that landed in your lake is a snapshot from extraction time, and Salesforce will happily return something different tomorrow without the record’s SystemModstamp moving at all.

Cross-object formulas are the worst version. A formula on Opportunity that references a field on its Account recalculates when the Account changes. The Opportunity did not change, so your incremental extract never picks it up. That column in your lake can be wrong indefinitely and nothing detects it.

Three ways out, in order of preference:

1. Do not extract them. Pull the underlying fields and reimplement the calculation in your transformation layer, where it is version-controlled and testable. More work up front, correct forever after.
2. Refresh them on a schedule. A narrow periodic extract of just the ID plus the formula columns, full-table. Cheap if the column list is short.
3. Accept it and document it. Fine for a formula nobody reports on. Not fine for anything in a revenue calculation.

Keep a list of which columns in your lake are formula-derived. Six months from now, when a number disagrees with Salesforce, that list is the first thing you will want and the last thing anybody wrote down.

Stay inside the API budget

Your Salesforce org has a daily API request allowance determined by edition and licence count, and it is shared. Exceed it and everything integrated with Salesforce stops working, not just your pipeline. That includes whatever marketing and support depend on, which is a conversation you would rather not have.

• Use the Bulk API for extracts, not the REST query API. Bulk is designed for volume and consumes the allowance very differently.
• Incremental by default, full refresh by exception. A nightly full extract of a large object is the single most common way to blow the budget.
• Give the pipeline its own integration user. That way the API usage reports attribute consumption to it, and you can see your own footprint instead of arguing about it.
• Watch consumption as a metric. Salesforce exposes API usage; graph it. Finding out by outage is expensive.

One thing that surprises people: field-level security on that integration user determines which fields come back. A field can exist in Salesforce, be populated, and arrive in your lake as consistently null because the integration user’s profile cannot see it. Check the profile before debugging the pipeline.

Query layer and access

Athena over the Glue catalog is the default answer and a good one: no cluster to run, pay per query, and it reads the curated views directly. Redshift Spectrum makes sense when you already run Redshift and want to join CRM data to warehouse tables.

Cost control on Athena is mostly about scan volume, which means partitioning and columnar storage rather than query tuning. Parquet plus sensible partitions does more than any amount of SQL cleverness.

On access: you have just replicated your customer database into object storage. Contacts and Leads are personal data, and Salesforce’s own field-level security does not follow it across. Encrypt the bucket, keep it private, use Lake Formation for column-level grants if different teams need different views, and decide the retention policy deliberately rather than by default. “We keep everything forever in S3 because it’s cheap” is a defensible engineering position and a poor compliance one.

Troubleshooting

Row counts drift upward over time

Deletes are not being applied. Run the full ID reconciliation and see how many rows come back. If it is a lot, also check for merged records via MasterRecordId before assuming the extract is at fault.

A column is always null in the lake, populated in Salesforce

Field-level security on the integration user’s profile. Log in as that user, or check the profile’s field permissions, before touching the pipeline.

A value disagrees with Salesforce but the record looks current

Almost certainly a formula field, especially a cross-object one. Check whether the column is formula-derived; if it is, the record’s modstamp never moved and your incremental extract had no reason to refetch it.

Extract fails partway on a large object

Query timeouts or governor limits. Chunk by date range or by ID range rather than pulling the object in one go, and make each chunk independently retryable so a failure costs you one slice rather than the whole run.

New Salesforce fields never appear

Most connectors map fields explicitly at configuration time and do not add new ones on their own. Poll the object’s describe metadata on a schedule and alert when the field list changes, so schema drift is a notification rather than a discovery.

Athena queries are slow or expensive

You are scanning too much. Check partitioning first, then whether the files are Parquet rather than JSON or CSV, then whether many tiny files are being read per query. Small-file proliferation is the usual culprit on frequently-run extracts, and compaction fixes it.

Common mistakes

• Building incremental sync on SystemModstamp alone and never handling deletes.
• Assuming the Recycle Bin gives you a guaranteed fifteen-day detection window.
• Treating CDC as sufficient on its own, with no reconciliation behind it.
• Ignoring merges, so a merged customer is counted twice forever.
• Extracting formula fields and treating them as durable values.
• Nightly full extracts of large objects, and the API outage that follows.
• Sharing an integration user with other tools, so nobody can attribute API consumption.
• Mutating the raw layer, which destroys your ability to rebuild.
• Clicking AppFlow flows together in the console with no infrastructure as code.
• Landing JSON or CSV instead of Parquet, then paying for it on every Athena query.
• No alerting on schema drift, so new fields are found by a user asking.
• Replicating Contacts and Leads into S3 with no retention policy or access controls.

Best practices

• Append-only raw layer, curated views on top, and never edit raw.
• Use at least two delete mechanisms, one of which is periodic full ID reconciliation.
• Soft-delete in curated rather than hard-deleting, so the audit trail survives.
• Reimplement formula logic downstream instead of trusting extracted formula values.
• Bulk API for extracts, incremental by default.
• A dedicated integration user with a documented, deliberately scoped profile.
• Partition by extraction date and store Parquet, or use Iceberg if you need row-level updates.
• Define flows, catalogs and permissions as code.
• Monitor API consumption and row-count delta against Salesforce as first-class metrics.
• Alert on schema changes rather than discovering them.
• Encrypt, restrict and set a retention policy on personal data the day you land it, not later.

FAQ

How do I handle deleted Salesforce records in a data lake?

Combine mechanisms. Query the Recycle Bin with queryAll and IsDeleted for recent deletions, consume Change Data Capture events for real-time coverage, and run a periodic full ID reconciliation as the backstop. No single one of the three is complete on its own.

Should I use AppFlow or a third-party ELT tool?

AppFlow if you want to stay inside AWS, value PrivateLink connectivity, and are willing to handle schema drift and deletes yourself. A managed vendor if you would rather buy those solved and can live with per-row pricing. Price both against the engineering time honestly; the build-it-yourself option is usually costed at zero and is not.

Parquet or Iceberg?

Parquet with date partitions is fine for append-only raw. Iceberg earns its complexity in the curated layer, where you want row-level updates and deletes, schema evolution and time travel. Salesforce data changes constantly, so that need is real rather than theoretical.

How often should I sync?

Match the decisions people actually make with the data. Hourly incrementals suit most reporting; sub-minute freshness usually means CDC and a real streaming consumer, which is a much larger commitment. Schedule the ID reconciliation separately and less often, weekly is normally enough.

Will this use up my Salesforce API limit?

It can, and the failure is shared with every other integration on the org. Use the Bulk API, stay incremental, give the pipeline its own user so consumption is attributable, and graph usage. Do not find out from an outage.

Do I need Salesforce Data Cloud?

Not to build a lake on AWS. It becomes interesting when compliance objects to copying CRM data at all, since zero-copy sharing avoids replication. Check what your existing Salesforce agreement covers before designing around it, because the licensing is a real factor.

Why do my numbers differ from Salesforce reports?

Check in this order: missing deletes, merged records counted twice, stale formula fields, fields the integration user cannot see, and only then your own transformation logic. The first two account for most of it, and the last is where people look first.

The one thing to remember

Getting Salesforce data into S3 is a configuration exercise you can finish in an afternoon. Keeping that data equal to Salesforce is an ongoing engineering problem, and the parts that break do not raise errors. They produce numbers that are slightly wrong, in a consistent direction, for months.

So build the reconciliation before you build the dashboards. A weekly job that pulls every ID and diffs it against the lake is unglamorous, cheap, and the only thing that will tell you the truth when someone asks why the two systems disagree.

Building or fixing one of these?

Most of the Salesforce lakes I get asked to look at work correctly for the first quarter and then quietly diverge. Work I take on:

• Building a Salesforce to S3 pipeline end to end: AppFlow or Bulk API ingestion, Glue catalog, Athena or Redshift Spectrum query layer.
• Auditing an existing lake for drift, and reporting exactly how many rows disagree with Salesforce and why.
• Implementing proper delete handling: CDC consumption, Recycle Bin queries and scheduled ID reconciliation.
• Reworking formula-derived columns into version-controlled transformations that stay correct.
• API budget work: moving extracts to Bulk, splitting integration users, and monitoring consumption before it causes an outage.
• Access and retention design for CRM data in S3, including encryption, Lake Formation grants and deletion policies.

Tell me which objects you replicate and how you currently detect deletes, and I will tell you where the drift is coming from.

The post The Row Count That Only Goes Up: Building a Salesforce Data Lake on AWS appeared first on John Nessime.