You look at the CRM reporting tab. It cannot do that. So now you are picking a platform, and within about ten minutes someone will say “we should build a data lake,” and someone else will say “just put it in the warehouse,” and the deciding argument will be about how much data there is.

That argument is almost always the wrong one. This post covers data lake vs data warehouse specifically for CRM analytics: why volume rarely decides it, what actually does, a decision procedure you can run in an afternoon, and where zero-ETL has changed the maths recently enough that older advice is stale.

Start with the number, then stop using it

Go and count your rows. Not estimate. Count.

A mid-sized company’s entire Salesforce org is usually a few million rows across the objects anyone actually reports on: accounts, contacts, opportunities, activities, cases. Add HubSpot, Zendesk and a billing system and you are still typically in the low tens of gigabytes once it is written as compressed Parquet. Activity and event history is the one object that can genuinely run large, and it is worth checking separately.

Numbers at that scale do not stress either architecture. A data warehouse will not break a sweat. A lake will not break a sweat. The lake-versus-warehouse debate as it is usually framed, the one about petabytes and elasticity and separating storage from compute, is a debate about a scale problem you do not have.

So count the rows, confirm you are small, and then stop using volume as an argument. Everything below is about what decides it instead.

Data lake vs data warehouse: the three options, honestly

Warehouse only

Load CRM objects straight into Redshift, Snowflake, BigQuery or Postgres. Model with SQL. Point the BI tool at it.

Where it wins: it is the fastest path to an analyst being productive, and that is not a small thing. Query performance on joins is better than a lake gives you without tuning. Access control is mature and row-level. Every BI tool connects natively. The people who will maintain this already know SQL and probably do not know Spark.

Where it doesn’t: semi-structured data is awkward, and CRM payloads are full of it. Custom field bags, JSON blobs, email bodies, call transcripts. You can store JSON in a modern warehouse but you will not enjoy querying it at any depth. Storing many years of daily snapshots gets expensive on warehouse storage in a way it does not on object storage. And you are more locked in: moving between warehouses is a migration project, not a repoint.

Lake only

Land raw API responses to S3, catalogue them, query with Athena or Spark. Iceberg or Delta as the table format if you want anything resembling a database.

Where it wins: raw retention is nearly free, so keeping every version of every record for years is a storage line item rather than a decision. Schema evolution is genuinely easier: when a SaaS admin adds a field, the lake absorbs it. Unstructured content lives alongside structured tables, which matters more now that people want to run models over call notes and support tickets. And an open table format means the data is not hostage to one query engine.

Where it doesn’t: you have taken on work a warehouse does for you. Small-file compaction, partitioning, table maintenance, catalogue hygiene. Query latency on interactive dashboards is worse unless you invest in layout. Governance is a build, not a setting. And the per-query cost model punishes exactly the workload CRM analytics generates: lots of small, repeated, unpredictable queries from a BI tool. Athena bills on data scanned with a per-query minimum, which means a dashboard refreshing thirty tiles is thirty billable scans whether or not anyone is looking.

Lakehouse

Raw and historical data in object storage as Iceberg tables; curated marts in the warehouse or queried in place by a warehouse engine. This is where most serious setups end up, and it is what AWS, Databricks and Snowflake have all converged on.

Where it wins: you get cheap deep history and expensive fast queries in the right places. Raw payloads stay replayable when a vendor changes a field type. Analysts still get warehouse-shaped tables.

Where it doesn’t: it is two systems. Two cost models, two sets of permissions, two places a table can be defined and drift. For a five-person company with one CRM, this is architecture as procrastination. The complexity is real and it should be earning something.

What actually decides it

Four levers, in the order they usually matter for CRM data.

1. How much history you need, and whether deletes matter

This is the big one and it is the one people skip.

CRM systems mutate in place. An opportunity’s stage, amount and close date are overwritten as the deal moves. If you only ever sync current state, you can report on the pipeline as it is today and you cannot report on what it looked like last March. Every question about pipeline movement, forecast accuracy, stage velocity or “what changed since the QBR” needs point-in-time history, and the CRM will not give it to you retroactively. Deleted records are worse: they are simply gone.

If you need history, you need somewhere cheap to keep daily snapshots or a change feed. That pushes toward object storage, or toward a warehouse with a genuinely cheap storage tier. If a rolling twelve months of current-state data covers every question anyone asks, the warehouse handles it alone.

2. How fast the schema churns

Count the custom fields added to your main CRM object in the last six months. If it is two, schema churn is not your problem. If it is forty, and half of them are from a business unit you do not talk to, a rigid warehouse schema becomes a standing maintenance tax and the lake’s tolerance for drift starts paying for itself.

3. The shape of the queries

Scheduled dashboards hitting the same tables repeatedly favour a warehouse, because per-second compute amortises across many queries and results are cached. Redshift Serverless, for example, bills in processing-unit-hours per second with a short minimum and shuts down when idle, which suits bursty BI traffic.

Exploratory one-off analysis across wide, rarely-touched history favours a lake, because you only pay when someone asks. Most CRM analytics is overwhelmingly the first kind. That fact alone settles a lot of these arguments.

4. Who is going to maintain it

The least technical criterion and often the most decisive. If the answer is “a RevOps analyst who is very good at SQL and has no platform team behind them,” build the warehouse. A lake handed to someone who cannot maintain it becomes an unqueryable pile of Parquet in about eight months. That is not a hypothetical failure mode; it is the normal one.

A decision procedure you can run this afternoon

1. Count rows per object for everything anyone reports on. Note which objects are over about ten million rows. Usually it is activities and nothing else.
2. Write down the ten questions the business actually asks. Not the ones you imagine. Mark each one as needing current state or point-in-time history.
3. Count schema changes on your main objects over the last six months.
4. Estimate query volume: number of dashboards, tiles per dashboard, refresh frequency, plus rough ad-hoc queries per week. This is the number that drives cost, not storage.
5. Model both costs against those numbers. For the lake: bytes scanned per query times queries per month. For the warehouse: compute-hours per month plus storage. Do it on a napkin. At CRM scale the answer is usually obvious and usually favours the warehouse.
6. Name the maintainer. If you cannot name a person, pick the option that needs less of them.

Read together, most CRM-only shops land on: warehouse for the curated layer, cheap object storage for raw history, and no ambition beyond that until something forces it. That is a lakehouse, but the small honest kind, and it is a very different commitment from standing up a full lake platform.

Where zero-ETL changed the maths

A lot of writing on this topic predates a genuine shift, so it is worth stating plainly.

Managed zero-ETL integrations now replicate CRM and support platforms into AWS targets without you building a pipeline. AWS Glue supports SaaS sources including Salesforce, SAP, ServiceNow, Zendesk and Zoho CRM, landing into Amazon Redshift, the SageMaker lakehouse, or S3 Tables with Iceberg underneath. Salesforce has pushed the other direction with zero-copy federation, letting Data Cloud read Iceberg tables in Databricks, Snowflake, BigQuery or Redshift in place, and letting those platforms read Data Cloud objects back.

Two consequences for this decision:

• Ingestion is no longer the differentiator. “The lake is easier to load into” was a real argument when you were hand-rolling connectors. It is much weaker when the same managed integration targets either destination.
• Iceberg makes the choice less permanent. If your raw layer is Iceberg tables in object storage, adding a warehouse engine later is a configuration change rather than a migration. That is a genuine reason to prefer open table formats even if you never build a “lake” as such.

Two honest caveats. Zero-ETL integrations cover a curated list of sources and regions, so check yours before designing around them. And a managed replica is still current-state replication plus change capture, not a modelled history; you still have to decide how snapshots and slowly changing dimensions get built.

Arguments that don’t survive contact

• “We need a lake because we’ll do AI later.” Reasonable in spirit, wrong in sequence. Build the thing that answers this quarter’s questions, keep raw payloads in object storage so you have not destroyed your options, and revisit. Raw retention is the cheap insurance; a lake platform is not.
• “Storage is cheaper in a lake.” True and almost always irrelevant. At CRM volumes, storage is a rounding error next to compute and, more to the point, next to the salary of whoever maintains the thing.
• “The warehouse is a single point of vendor lock-in.” Fair, and it is why open table formats matter. But if you replicate raw data to object storage, the lock-in is on your transformation logic, not your data, and that is portable if you keep it in SQL and version control.
• “We’ll just query the CRM API directly and skip both.” Works until a dashboard fans out into rate limits and the whole thing becomes a data pipeline you did not design. If more than one person is asking questions, you need a copy.
• “Let’s do both properly from day one.” This is the expensive mistake. Two systems, two cost models, and tables that quietly disagree, in service of a workload that fits comfortably in one.

Frequently asked questions

Is a data lake overkill for one CRM?

For a single CRM with current-state reporting needs, usually yes. The point at which it stops being overkill is when you need years of point-in-time history, when unstructured content like call transcripts becomes part of the analysis, or when you are integrating enough systems that a common raw layer is cheaper than a dozen warehouse loaders.

How do I get historical CRM snapshots if I only have current state?

You mostly cannot get it retroactively, which is why this decision is urgent rather than academic. Start capturing now: either daily full snapshots to partitioned object storage, or a change feed if the platform offers one, then build slowly changing dimension tables from that. Field history tracking inside the CRM covers a limited number of fields and is not a substitute.

Does the lakehouse make this comparison obsolete?

It blurs the storage layer, not the operational reality. You still choose a query engine, a cost model, a governance model and a maintainer. The useful question has shifted from “lake or warehouse” to “how much of this do I want to operate myself,” which is a better question anyway.

Where does reverse ETL fit?

If you plan to push computed values back into the CRM, such as a lead score or account health field, that pushes toward the warehouse. Reverse ETL tools connect to warehouses far more readily than to query engines over object storage, and the write path wants a stable, modelled table rather than a raw one.

Can I start with one and move later?

Moving from lake to warehouse is straightforward: you are loading modelled tables into a new engine. Moving from warehouse to lake is harder, because the history only exists inside the warehouse and the transformations are written in its dialect. The asymmetric fix is to land raw data in object storage regardless of which you query from. That single habit preserves the option cheaply.

What about just using the CRM’s own analytics product?

Worth pricing seriously before building anything. It is genuinely the right answer when all your questions are about one system, and it removes an entire pipeline from your life. It stops being the right answer the moment a question spans CRM plus billing plus product usage, because you will be paying to ingest the other systems into a platform priced for CRM data.

The short version

In the data lake vs data warehouse question for CRM analytics, volume is the argument everyone reaches for and it is the one that matters least. Your CRM data is small. Both options handle it.

What decides it is how much history you need, how fast the schema moves, what shape the queries are, and who is going to keep it running. Answer those four and the architecture falls out on its own. And whatever you pick, land the raw payloads in object storage from day one, because that is the cheap decision that keeps every other decision reversible.

Need a second opinion on your CRM analytics stack?

I help teams make this call without over-building, and unpick it when it has already been over-built. Things I can help with:

• Running the sizing and cost model above against your real query volume, not a vendor calculator.
• Designing point-in-time history capture for CRM objects, including deletes and slowly changing dimensions.
• Setting up zero-ETL or connector-based ingestion from Salesforce, HubSpot, Zendesk or Zoho into Redshift, Snowflake or an Iceberg lake.
• Migrating an unmaintainable pile of Parquet into something an analyst can actually query.
• Cutting query costs on an existing setup: partitioning, table layout, dashboard refresh patterns.
• Building the curated CRM models that BI tools sit on, in SQL and in version control.

If you are mid-argument about this, send me your row counts and the ten questions the business wants answered. That is usually enough to settle it.

The post Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question appeared first on John Nessime.

“Salesforce handles it” is not an answer. Neither is “it’s in our VPC,” because it usually isn’t, and the parts that are in your VPC are the parts nobody has threat-modelled.

This post traces a single Agentforce request end to end and stops at each boundary it crosses. Agentforce and AWS get wired together in three or four standard patterns, and every one of them has a specific point where Salesforce’s guarantees stop and yours start. I will cover where that line sits, what changes about the security model when you cross it, and one default that surprises almost everyone.

The boundaries a single request crosses

One user turn touches more systems than the diagrams suggest. Roughly:

1. Retrieval. The agent grounds itself in CRM records, Data Cloud objects and knowledge content.
2. Reasoning. The request gets classified to a topic, then an action gets chosen. Both steps involve a model call.
3. Generation. A prompt is assembled and sent through the LLM gateway to a model provider.
4. Action. If the agent decides to do something, it calls out. That callout is where AWS usually enters the picture.

Steps one to three happen inside Salesforce’s trust boundary, and the Einstein Trust Layer applies. Step four does not, and it doesn’t. That sentence is the whole post, but the detail matters.

Retrieval: the part that’s better than you’d expect

Grounding is where Agentforce gets its context, and the architecturally interesting decision is that vector search lives inside Data Cloud rather than in a separate vector database. Structured records and unstructured content are retrieved from the same layer, which removes a whole class of sync problem you would otherwise be building.

The security-relevant property is that retrieval is permission-aware. The agent retrieves as the running user, so sharing rules and field-level security apply. If a rep cannot see an opportunity, the agent acting on their behalf should not be able to summarise it either.

Two caveats worth putting in your notes. First, this only holds while retrieval goes through Salesforce; the moment you ground from an external source, you are enforcing access control yourself. Second, if your CRM sharing model is loose, the agent inherits that looseness and makes it much easier to exploit, because a model will cheerfully summarise a thousand records a human would never have opened one by one. Agentforce does not create the over-permissioning problem, it just removes the friction that was hiding it.

Where AWS shows up here: if your analytical data lives in Redshift, an Iceberg lake or Databricks, zero-copy federation lets Data Cloud register those tables and query them in place rather than replicating them. Good for cost and duplication, and it moves an access-control decision into the federation configuration, which is a place people forget to audit.

The Trust Layer, and the default nobody expects

The Einstein Trust Layer sits between every Salesforce-originated prompt and the model. Its components are well documented: secure grounding, data masking, system policies against prompt injection, the LLM gateway, zero data retention agreements with model providers, toxicity checks on the way back, and an audit trail of the whole journey.

Here is the part that gets misreported constantly, including in a lot of otherwise decent write-ups: LLM data masking is disabled for agents. Salesforce says so plainly in its own documentation on the Trust Layer architecture and repeats it in the Trailhead module on masking. Masking remains available and configurable for embedded generative features like service replies and work summaries. For Agentforce agents, it is off.

The reason is not hard to work out once you think about what an agent does. Masking swaps real values for placeholders. An agent that has to actually perform an action needs the real record ID, the real amount, the real email address, because those become tool arguments. You cannot pass <Person_0> to a Flow and expect it to update a contact. Masking and acting are in genuine tension, and Salesforce resolved it in favour of acting.

What this changes in practice:

• Real PII reaches the model provider. The zero data retention commitment is what protects it, not de-identification. Those are different controls with different failure modes: masking is technical and verifiable, a retention agreement is contractual.
• If your compliance position was “no personal data leaves our boundary in identifiable form,” Agentforce does not satisfy it the way you may have assumed, and this is worth raising before your security review rather than during it.
• Controlling exposure moves upstream. What the model sees is now decided by what you ground on and what your sharing model permits, not by a masking filter downstream.

None of this makes the Trust Layer weak. Zero retention, the gateway, system policies and the audit trail all still apply to agent traffic, and the Trust Layer typically runs several times in a single user turn, once for topic classification and again for generation. It is a real control plane. Just know which control is doing the work.

Connecting Agentforce and AWS: four patterns

There are four ways these two systems talk, and they have different latency, failure and security characteristics.

Synchronous outbound: agent action calls AWS

A custom agent action, implemented via External Services with an OpenAPI schema, an Apex invocable method, or a Flow, calls an endpoint you own. Behind it sits API Gateway and Lambda, and often a Bedrock agent or knowledge base.

Authentication uses named credentials, which keep the secret out of your Apex and centralise the auth config. Use them; hardcoding a key into an Apex class is the version of this that ends up in a git history.

The trade-off is that you have put a network call and a second model invocation inside a conversational turn. Users notice. Budget for it, and set the timeout deliberately rather than inheriting a default.

Asynchronous outbound: Event Relay to EventBridge

Salesforce Event Relay delivers platform events from the Salesforce event bus to Amazon EventBridge without code, and EventBridge routes onward to Lambda, S3, Step Functions or a Bedrock agent.

This is the right pattern for anything that does not need to complete inside the conversation: enrichment, downstream processing, long-running work. It also fails better, because a retry is a queue concern rather than a user watching a spinner.

Inbound: AWS calls Agentforce

The Agent API lets something on the AWS side invoke an Agentforce agent, typically through EventBridge API destinations. That gives you bidirectional multi-agent flows: a Bedrock agent that detects a problem can open a case through Agentforce rather than writing to the Salesforce API directly and bypassing the agent’s own logic.

Worth being deliberate here. An inbound path means an AWS workload can trigger agent reasoning, which means anything that can inject content into that workload can influence a prompt. Treat the payload as untrusted input, because it is.

MCP: tools rather than integrations

Agentforce has a native MCP client, so an agent can connect to any MCP-compliant server without a bespoke integration, alongside a server registry intended to let admins govern which agents connect to which tools under central policy.

This is the pattern that will cause the most trouble over the next couple of years, and not because the protocol is bad. It is because MCP makes adding a tool trivial, and every tool is a new path by which text from somewhere else reaches your agent’s context and by which your agent’s decisions reach someone else’s system. The registry exists precisely so that “which tools can this agent reach” is an administered answer rather than an emergent one. Use it as a control, not a catalogue.

Keeping the traffic private

By default these callouts traverse the public internet with TLS. Salesforce Private Connect uses AWS PrivateLink to give you a private path to your own AWS endpoints instead. Whether that is required is a policy question rather than a technical one, but it is much easier to set up at design time than to retrofit after an auditor asks.

What changes the moment you cross into AWS

This is the seam, and it is where most real incidents will come from. Four things stop being true simultaneously.

• Zero retention becomes your policy, not Salesforce’s. If your Lambda logs its input at INFO, you now have customer PII in CloudWatch Logs with whatever retention that log group happens to have, which by default is forever. Your own debug logging is the single most likely leak path in this entire architecture, and it will not show up in any Salesforce audit report.
• The audit trail splits. Salesforce logs the prompt journey. AWS logs the invocation. Nothing correlates them unless you pass a request identifier through the callout and log it on both sides. Do that from day one; reconstructing an incident across two disconnected audit systems is genuinely unpleasant.
• User identity does not travel. This is the important one. Inside Salesforce the agent acts as the running user and sharing rules apply. Your Lambda runs under an IAM execution role with fixed permissions and no idea who asked. If that role can read a whole DynamoDB table, then every user of the agent can effectively read the whole table through it, regardless of their CRM permissions. That is a classic confused deputy, and it is easy to build without noticing. If the action needs to be scoped per user, pass the identity explicitly and enforce it in your own code.
• Model governance forks. A Bedrock model you invoke from your own Lambda is outside the Trust Layer. Its guardrails, logging and retention are configured in your account and are yours to get right. If you want prompt filtering or PII detection on that leg, you are building or buying it.

Note the difference between two things people conflate. Salesforce running on Hyperforce, which is built on public cloud infrastructure including AWS, means Salesforce’s own workloads run there under Salesforce’s controls. It does not mean your Salesforce data sits in your AWS account. When an agent action calls your Lambda, that is the first moment data lands in infrastructure you control and are responsible for.

Common mistakes

• Assuming masking protects agent traffic to the model. It does not; check the current documentation yourself rather than taking a blog’s word for it, including this one.
• Logging full request payloads in Lambda during development and never turning it off.
• Giving the Lambda execution role broad data access because “the agent is trusted,” when the agent is a proxy for every user who can talk to it.
• Putting a slow AWS call inside a synchronous agent action when the work did not need to be synchronous.
• Not passing a correlation identifier across the boundary, so Salesforce and CloudWatch tell two unlinked stories.
• Treating content returned from an external tool or MCP server as trusted context rather than as untrusted input that will be read by a model.
• Deploying agents against a permissive sharing model and discovering the blast radius later.

Frequently asked questions

Does Agentforce data get used to train models?

Not on the Salesforce leg. Salesforce’s zero data retention arrangements with model providers mean prompts and responses are not stored or used for training. That commitment covers traffic going through the LLM gateway. It says nothing about a model you invoke yourself from your own AWS account, where the retention and logging configuration is entirely yours.

Can I use my own Bedrock model inside Agentforce?

There are two distinct things here. Bedrock-hosted models can be brought into Salesforce through the Einstein and Bedrock integration, in which case they sit behind the Trust Layer. Separately, you can call a Bedrock model or agent from a custom agent action, in which case you are outside the Trust Layer and own the controls. Be clear which one you are building, because they have completely different compliance stories.

Sync or async for AWS integrations?

Default to async through Event Relay and EventBridge unless the agent genuinely needs the result to continue the conversation. Synchronous callouts put a network round trip and often a second inference call inside a user’s turn, and they fail in front of the user rather than into a retry queue.

How do I stop the agent seeing data a user shouldn’t?

Inside Salesforce, fix the sharing model, because retrieval respects it and inherits its flaws. Outside Salesforce, sharing rules do not apply at all: pass the user identity through the callout and enforce authorisation in your own code, or scope the integration to data that is safe for every user of that agent.

Do I need PrivateLink for this?

Technically no; callouts work over TLS on the public internet. Practically it depends on your regulatory position and whether your AWS endpoints should be publicly reachable at all. If they should not be, Private Connect over PrivateLink is the supported path and is far cheaper to design in than to add later.

What should I log, and where?

On the AWS side, log the correlation identifier, the action name, the outcome and the timing. Do not log the payload. If you need payload-level debugging, gate it behind a flag that is off in production and set an explicit short retention on the log group. Ship the operational metrics somewhere you already watch, whether that is CloudWatch alarms, Grafana or Datadog, because an agent action that silently starts failing looks like nothing at all from the Salesforce side.

The one thing to take away

Wiring up Agentforce and AWS is not hard. The patterns are documented, the connectors exist, and a working proof of concept is a day’s work.

What is hard is knowing precisely where Salesforce’s guarantees end. They end at the callout. Everything before it is covered by the Trust Layer, with the significant exception that masking is off for agents, so what reaches the model is real data protected by a retention agreement rather than by de-identification. Everything after it is yours: your retention, your audit trail, your authorisation, your logs.

Draw that line on your architecture diagram before the security review, not during it.

Need help designing or reviewing this?

I work on the AWS side of Salesforce integrations, which is usually the side nobody owns. Things I can help with:

• Threat-modelling the seam between Agentforce and your AWS account, and writing down what each side actually guarantees.
• Building agent actions on API Gateway and Lambda with named credentials, sane timeouts and no payload logging.
• Event-driven integration via Event Relay and EventBridge, including retry, dead-letter and replay design.
• Scoping IAM execution roles so an agent action cannot become a confused deputy for every user who can reach it.
• Correlated observability across Salesforce and CloudWatch so an incident can be reconstructed in one timeline.
• Private Connect and PrivateLink setup where callouts should not touch the public internet.

If you have an integration already built, send me the agent action definition and the Lambda’s IAM policy. Those two together usually tell the whole story.

The post Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin appeared first on John Nessime.

A Salesforce admin renames a custom field on Tuesday afternoon. Your connector keeps running, your Glue job keeps succeeding, the crawler happily adds a new column and quietly stops populating the old one. Three weeks later someone asks why the pipeline report has a cliff in it, and you spend an afternoon reconstructing a change nobody in engineering made and nobody logged.

This post is about using AWS Glue Data Quality to catch that class of breakage. It is aimed at people pulling from SaaS sources into S3 or a lakehouse: Salesforce, HubSpot, Zendesk, Stripe, Shopify, Zoho, an internal partner API. I will cover where to run the checks, the flattening constraint that will bite you before anything else does, which DQDL rule actually catches which SaaS failure, when dynamic rules and anomaly detection quietly do nothing, and what should happen when a rule fails.

Why SaaS data breaks in ways your own database doesn’t

When you own the source database, the schema is under change control: someone writes a migration, it gets reviewed, and if it breaks a consumer there is a commit to point at. A SaaS source has none of that. Three consequences:

• The schema is edited by people who are not engineers. A sales ops admin adding a picklist value is a production schema change with no review, no notice and no version number.
• The payload is nested. SaaS APIs return structs and arrays: line items, related objects, custom field bags. That shape matters more than you would expect, for reasons in the next section but one.
• Absence is ambiguous. A field missing from a payload might mean “not set,” “not permitted for this user,” or “we removed it.” The connector usually cannot tell you which, and neither can your table.

Glue Data Quality is a rules engine built on Deequ that evaluates declarative checks written in DQDL, the Data Quality Definition Language, against a dataset. It is well suited to the first and third problems. The second one needs work before it is useful at all.

Decide where the checks run before you write a rule

There are two places to evaluate a ruleset, and they answer different questions.

Inside the Glue ETL job

The EvaluateDataQuality transform runs mid-pipeline against a DynamicFrame. It answers “is this batch safe to load?” and it can act on the answer: stop the job, or split failing rows into a quarantine prefix. This is also the only placement where row-level results exist, which matters because “which 400 rows are bad” is a far more useful answer than “the rule failed.”

Against a Data Catalog table

You can attach a ruleset to a catalog table and run it on a schedule, independent of any job. This answers “is the table currently trustworthy?” and surfaces a quality score analysts can see before they query. The trade-off is that it is after the fact: by the time a catalog evaluation fails, the bad batch is already queryable. Analyzers are also not supported for catalog evaluations, so some dynamic-rule tooling behaves differently there.

In practice I run both. The in-job ruleset is small and strict, and exists to stop garbage landing. The catalog ruleset is broader and looser, and exists so a table’s state is visible to people who will never open Glue Studio.

The flattening problem that comes first

Here is the constraint that catches most people on SaaS data, stated plainly in the AWS documentation and easy to skim past: DQDL cannot evaluate nested or list-type columns. No structs, no arrays.

Almost every interesting SaaS payload is nested. A Salesforce opportunity carries a related account struct, a Stripe invoice carries an array of line items, a Zendesk ticket carries a custom fields array of key-value pairs. Point a ruleset at the raw landed data and the fields you most want to check are the ones the engine cannot see.

So you flatten first, with Glue’s relationalize or an explicit projection, then evaluate. That works, but it changes what your rules mean:

• Your rules now describe the projection, not the payload. If your flattening step drops a field, no rule will notice, because from DQDL’s point of view that field was never there.
• Exploding an array changes the grain. One invoice with five line items becomes five rows. Every RowCount rule you write against that table is now counting line items, and a customer who starts itemising differently will move your row count without any data being wrong.
• Custom-field bags need pivoting, not flattening. A key-value array flattened naively gives you rows, not columns, and you cannot write a ColumnValues rule against a value whose meaning depends on a sibling key column. Pivot the ones you care about into real columns and check those.

The practical rule: put your quality checks immediately after the flattening step, and treat the flattening step itself as code that needs its own test. Glue Data Quality will not cover it for you.

Matching SaaS breakage to the rule that catches it

DQDL has a long rule list and most guides just enumerate it. More useful is the other direction: what actually goes wrong with SaaS sources, and what catches each one.

A field is renamed, removed, or added

This is the Tuesday-afternoon scenario. Two rules, doing two different jobs:

Rules = [
    ColumnExists "annual_revenue__c",
    ColumnCount = 47
]

ColumnExists protects the specific fields your downstream models depend on. ColumnCount protects against everything else: a field silently added, or one you forgot you needed. The first is precise and the second is a tripwire, and you want both because a rename shows up as one column gone and one column arrived, which leaves the count unchanged.

SchemaMatch compares against a reference dataset’s schema, which is stricter and useful if you maintain a golden reference table. Bear in mind a hard column count on a SaaS object fails every time an admin adds a field, so decide whether that blocks or just warns.

A field still exists but stopped being populated

Permissions change, a workflow gets disabled, an integration user loses field-level access. The column is still in the schema and is now all nulls.

Rules = [
    IsComplete "opportunity_id",
    Completeness "lead_source" > 0.85,
    Completeness "email" > 0.60 where "record_type = 'Contact'"
]

IsComplete is the absolute version: no nulls at all, right for keys. Completeness takes a threshold, which is what you want for optional business fields where a hard rule would fail constantly. The where clause is the one people miss: SaaS objects are usually polymorphic, and a completeness threshold that makes sense for contacts is nonsense across the whole table. Scope the rule to the record type.

The sync ran short

A paginated extract stops early, a rate limit gets absorbed by retries, an incremental cursor drifts. The job succeeds and delivers less data than it should have. A static RowCount > 1000 stops catching this the moment you cross a thousand rows. Dynamic rules compare against your own history instead:

Rules = [
    RowCount > min(last(3)),
    Sum "amount" > avg(last(5)) * 0.7,
    DetectAnomalies "RowCount"
]

Analyzers = [
    RowCount,
    DistinctValuesCount "owner_id"
]

last(k) pulls the last k recorded values of that metric for the same dataset; with k greater than one you need an aggregation like avg, min or max to reduce them to one number. DetectAnomalies goes further, using forecasting over stored history to flag deviations with no threshold from you at all. That is the right tool for anything seasonal: a B2B pipeline that dips every weekend trips a naive trailing-average rule every Monday, while anomaly detection learns the shape.

Analyzers deserve their own mention: they gather a metric without asserting anything, which builds the history dynamic rules and anomaly detection need. DistinctValuesCount on an owner or account column is a strong early signal for a partial sync, because row count can look fine while the number of distinct accounts represented quietly halves.

Records arrive twice, or the key stops being a key

Replays, overlapping incremental windows and connector retries all produce duplicates, and SaaS “IDs” are less unique than vendor docs imply once you have merged sandboxes or multiple accounts.

Rules = [
    IsPrimaryKey "id",
    Uniqueness "external_reference" > 0.99
]

IsPrimaryKey asserts unique and non-null together. Uniqueness with a threshold is the softer version for fields that are mostly-but-not-perfectly unique, which describes a lot of SaaS reference fields honestly.

Objects stop agreeing with each other

Multi-object extracts rot quietly. Opportunities reference accounts that were never synced; invoice totals stop matching line item sums because one object’s extract failed and the other’s didn’t.

ReferentialIntegrity checks that values in one dataset’s column exist in another’s. RowCountMatch and AggregateMatch compare counts and summary metrics across two datasets. These catch a broken join before an analyst does, and they need the multi-frame form of the transform with the second dataset supplied as an additional source.

The sync stopped and nobody noticed

The quietest failure of all: nothing is wrong with the data, there just isn’t any new data. DataFreshness checks a timestamp column against a window. It is deliberately simple and its comparison granularity is limited, so for anything more nuanced the custom SQL rule is the escape hatch, running a query against the dataset and evaluating the result against an expression. DQDL is case sensitive and that rule type’s exact capitalisation has appeared more than one way across the docs and the visual builder, so copy it from the current DQDL reference rather than from memory.

Static, dynamic and ML: when each one lies to you

All three check styles have a mode where they pass while the data is wrong. Knowing which is which is most of the skill.

• Static rules lie by going stale. A threshold that was correct when you wrote it stops being correct as the business grows, and a rule that can no longer fail is worse than no rule because it looks like coverage.
• Dynamic rules lie when they have no history. This one is genuinely nasty. last(k) reads previously recorded values for that metric, and if there aren’t any, it falls back to a default of zero. Add RowCount > avg(last(3)) to an existing job today and it will pass trivially for the first few runs, because almost any row count beats zero. It looks like a working check from the moment you deploy it. It isn’t one until it has accumulated runs.
• Anomaly detection lies while it is warming up, and after you retrain it badly. It needs a run of history before its forecasts mean anything, and if you leave a period of genuinely bad data in the training window, it learns that the bad shape is normal. You can exclude specific statistics from the calculation in the Data Quality tab, and you should, after any incident.

So when you add a dynamic or ML-based rule, write down the date and treat it as unarmed for the first couple of weeks. Keep a crude static rule alongside it in the meantime.

What should happen when a rule fails

The transform returns a collection. The two members you care about are the row-level outcomes, which give you the original data with an evaluation result column appended per row, and the rule outcomes, which give you one row per rule with its verdict.

from awsgluedq.transforms import EvaluateDataQuality

dq_results = EvaluateDataQuality().process_rows(
    frame=flattened_opportunities,
    ruleset=ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "salesforce_opportunity",
        "enableDataQualityCloudWatchMetrics": "true",
        "enableDataQualityResultsPublishing": "true",
    },
)

row_level = SelectFromCollection.apply(
    dfc=dq_results, key="rowLevelOutcomes"
)

dataQualityEvaluationContext is the namespace your CloudWatch metrics land under. Name it after the source object rather than the job, because you will eventually want to alert per object.

From there, three response patterns:

1. Fail fast. Stop the job on failure so nothing lands. Right for structural rules: if the primary key isn’t unique, downstream is going to be wrong no matter what you do next.
2. Quarantine. Filter the row-level outcomes on the evaluation result column, write passing rows to the target and failing rows to a separate prefix. Right for value-level rules where 2% bad records shouldn’t block 98% good ones. The important discipline is that somebody has to actually look at the quarantine prefix; an unmonitored quarantine is a delete with extra steps.
3. Warn and continue. Let everything land, publish the metrics, alert on the score. Right for new rules you don’t trust yet, and for the catalog-side ruleset.

Whichever you pick, get the failure out of Glue and into wherever your team actually looks. Results and anomaly predictions can be written back to catalog tables so they are queryable with plain SQL, and CloudWatch metrics plus EventBridge events give you the hook for paging or pausing a downstream pipeline. A Grafana or Datadog dashboard fed from those metrics beats a score in a console tab nobody opens.

Troubleshooting AWS Glue Data Quality on SaaS pipelines

“Input data does not include column X”

A rule references a column the engine cannot find. On SaaS data that means one of three things: the field disappeared from the source, your flattening step renamed it, or the Data Catalog is out of sync with the underlying table. Check the catalog schema against the landed files before touching the ruleset; rerunning the crawler resolves a surprising share of these.

A rule type isn’t recognised

DQDL is case sensitive and the rule set has grown over releases, so a rule copied from an older blog post can parse fine in your head and not at all in the engine. Check the rule name against the current DQDL rule type reference, and check that the feature is available on your Glue version. Anomaly detection in particular has a version floor.

A dynamic rule never fails

Almost always the empty-history problem described above. Look at the statistics for that metric in the Data Quality tab. If there are fewer recorded points than your last(k) asks for, the rule is not doing what you think. It will start working on its own; the mistake is trusting it in the meantime.

Evaluation is slow or the job runs out of memory

Each rule triggers analyzer computation over the dataset, and a large ruleset on a wide flattened SaaS table adds up fast. The transform exposes caching and observations-scope options to tune this. Before reaching for a bigger worker type, cut the ruleset down: most contain a dozen recommended rules nobody reads the results of.

A “column reference is ambiguous” error

Classic flattening artefact. Two nested paths collapsed to the same leaf name, so datechecked now matches two things. Fix it in the projection with explicit aliases rather than trying to disambiguate inside DQDL.

Common mistakes

• Accepting the recommended ruleset wholesale. Recommendations are generated from a sample of current data, so they encode today’s accidents as tomorrow’s requirements.
• Writing rules against nested data and assuming they cover the whole payload.
• Deploying a dynamic rule and treating it as armed on day one.
• Applying whole-table completeness thresholds to a polymorphic object instead of scoping with a where clause.
• Quarantining failed rows to a prefix nobody monitors.
• Writing RowCount rules against an exploded array without noticing the grain changed.
• Failing the job on every rule, including cosmetic ones, until the team starts rerunning with checks disabled.
• Keeping the ruleset in the console instead of in version control next to the job.

Best practices

• Split rules into blocking and reporting tiers, and be honest about which is which. A rule that always fails and never stops anything trains people to ignore the tab.
• Pair ColumnExists with ColumnCount. One protects the fields you named, the other catches the drift you did not anticipate.
• Add analyzers early, even for metrics you have no rule for yet. History has to accumulate before dynamic rules and anomaly detection are worth anything, and you cannot backfill it.
• Keep DQDL in Git alongside the job definition and deploy it with Terraform or CloudFormation. Rulesets are code and should move through the same review as the pipeline they guard.
• Namespace the evaluation context per source object so CloudWatch metrics and alerts can be routed per vendor.
• Exclude incident periods from the anomaly detection training window once the incident is resolved.
• Surface results where stakeholders already look. Writing outcomes back to catalog tables makes them queryable from Athena, which beats a score in a console tab.

Frequently asked questions

How much does AWS Glue Data Quality cost?

It bills on Glue DPU-hours like any other Glue workload, so cost is a function of evaluation time and DPUs used. The dimension that surprises people is anomaly detection, which consumes additional capacity per statistic analysed, so enabling it broadly across many columns and tables is where the bill moves. Model it against the current Glue pricing page; rates change and vary by region.

Can it check data before it lands in S3?

Yes, if the check runs inside the Glue job. The transform evaluates a DynamicFrame in memory, so you can validate after extraction and before the write. That is the placement that lets you stop a bad load rather than report on one.

Does it work on nested JSON from a SaaS API?

Not directly. DQDL does not evaluate nested or list-type columns, so you have to flatten the structure first and run the ruleset against the flattened frame. Plan the flattening deliberately, because your rules only ever see what the projection exposes.

How does it compare to Great Expectations, Soda or Monte Carlo?

Glue Data Quality’s real advantage is that it is already inside the runtime and the catalog: no extra infrastructure, no separate agent, one IAM story. The disadvantages are real too. DQDL is less expressive than Python-based assertions, the nested-data limitation is a hard wall, and it only covers what Glue can read. Great Expectations gives you more expressive checks at the cost of running it yourself; Monte Carlo and similar platforms add lineage and cross-warehouse coverage Glue does not attempt. Glue-centric stack, start here. Several engines, a standalone tool fits better.

Should I use the recommended rules?

As a starting draft, yes. As a ruleset, no. Recommendations are inferred from a sample of the data as it is today, which means any existing quality problem gets encoded as the expected baseline. Read every generated rule and delete the ones you would not have written yourself.

Can I reconcile row counts against the SaaS source itself?

Not directly; it only sees datasets it can read. The usual pattern is to have the extraction step record the count the API reported, land that alongside the data as a control table, then use RowCountMatch or AggregateMatch to compare the two. That turns “did we get everything” into a rule the engine can evaluate.

The one thing worth remembering

AWS Glue Data Quality is not really a data quality product. It is a way of writing down what you believe about a dataset so a machine can check it every run, and its value is entirely determined by whether those beliefs are written precisely enough to be wrong.

For SaaS sources that means two habits. Flatten deliberately and check immediately after, because your rules only see the projection. And treat every new dynamic or ML-based rule as unarmed until it has history, because a rule that cannot fail yet looks exactly like a rule that is passing.

Need help with data quality on your Glue pipelines?

I work with teams whose SaaS pipelines run green while the numbers drift. Things I can help with:

• Writing a DQDL ruleset for a specific SaaS object that catches schema drift instead of just restating the current schema.
• Designing the flattening and pivoting step for nested payloads and custom-field bags so the checks actually cover what matters.
• Setting up quarantine and fail-fast paths in Glue ETL, with the routing and alerting that makes a quarantine prefix something people act on.
• Cross-object reconciliation between SaaS extracts and control counts recorded at extraction time.
• Tuning anomaly detection and dynamic rules, including cleaning incident periods out of the training window.
• Getting data quality results out of the console and into Grafana, CloudWatch alarms or an Athena-queryable table.

If you have a ruleset that keeps passing while the data drifts, send me the DQDL and a schema dump. That is usually enough to see where the gap is.

The post AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed appeared first on John Nessime.

That is the shape of most API pipeline incidents. Not a crash. A success that isn’t one.

This post covers running Apache Airflow on AWS specifically for SaaS and API workloads: pulling from HubSpot, Salesforce, Stripe, Zendesk, Shopify, an internal partner API, whatever. It is organised by failure family rather than by feature, because the Airflow documentation already explains what an operator is and does a poor job of explaining which of these things will page you at 3am. I will cover choosing a deployment model, the concurrency trap that catches almost everyone, retry design, incremental state, secrets, and where the money actually goes.

Why SaaS and API sources break differently

When your source is a database you control, failure is loud: connection refused, deadlock, disk full. When it is somebody else’s SaaS API, three things change.

• You are a guest. The vendor decides your rate limit, and they can change it without telling you. Your pipeline’s correctness now depends on a number in someone else’s config file.
• Errors arrive as valid HTTP. A 429, a 200 with a truncated page, a 200 with an error object in the body. Your HTTP client is happy. Your data is not.
• Tasks spend most of their life waiting. API extraction is I/O bound almost end to end. That sounds harmless and is the root of the most expensive mistakes.

Pick the deployment model before you write a DAG

This decision constrains everything after it and is harder to reverse than people expect. Three realistic options.

Amazon MWAA, provisioned

AWS runs the scheduler, web server, workers, triggerer and metadata database on Fargate; you drop DAGs into an S3 bucket and they get picked up.

Where it wins: real Airflow, custom providers, custom plugins, full control over environment configuration. If your DAGs need arbitrary Python libraries, this option will not fight you.

Where it doesn’t: the environment bills by the hour whether or not anything is running. There is no scale to zero on the base environment. If you sync six APIs once a day and each run takes twenty minutes, you are paying for a mostly idle cluster around the clock. The mw1.micro class exists precisely for the small case, but it collapses the scheduler and worker into a single Fargate task and caps worker autoscale low, so treat it as a dev or isolation tier rather than a cheap production tier.

Amazon MWAA Serverless

You submit workflow definitions and AWS runs each task in its own Fargate container, billing per task duration with a one-minute minimum rather than per environment hour.

Where it wins: spiky or infrequent schedules. If the workload is “six syncs a day, nothing overnight,” the cost profile beats a permanently running environment by a wide margin. Each workflow also gets its own IAM execution role, which is a real security improvement over one shared role per environment.

Where it doesn’t: it leans on declarative YAML workflow definitions based on the DAG Factory format and a curated set of AWS operators. That is a deliberate trade: because the definition is declarative, the service can schedule tasks without executing your DAG code. It also means custom operators, exotic third-party providers and clever Python at parse time are not the sweet spot. It is also available in fewer regions than provisioned MWAA, so check your region before you design around it.

Self-managed on ECS, EKS or a VPS

On EKS with the Kubernetes executor you get per-task pods and tight cost control. On a single VPS from a provider like InterServer or Hetzner, a Docker Compose stack with a Postgres metadata database will run a modest set of API syncs for a fraction of any managed price.

Where it wins: cost at both extremes, and total control. Where it doesn’t: you now own metadata database upgrades, major version migrations, log retention and the 2am scheduler restart. Managed Airflow is a bet that your time is worth more than the hourly premium. For a solo engineer with three pipelines that bet often loses; for a data team of eight it usually wins. Astronomer is the main non-AWS managed option worth pricing alongside these.

Failure family one: the throttle that silently stops throttling

You start with a normal setup: an Airflow pool named crm_api with four slots, and every task that touches the vendor assigned to it. Four concurrent requests, comfortably under the vendor’s limit. This works.

Then you notice those tasks spend nearly all their runtime waiting on HTTP, burning worker slots to sit still. So you switch them to deferrable operators. A deferrable task suspends itself while waiting, releases its worker slot, and hands the waiting to the triggerer, which polls asynchronously. Worker pressure drops. Everything looks better.

And your rate limiting quietly stops working.

By default, a pool does not count tasks in the deferred state as occupying slots. That was deliberate, and the logic is sound in the abstract: a deferred task is not consuming a worker. But if you were using the pool to protect an external API rather than your own workers, it has just stopped doing the job you gave it. Every task can defer at once, and the vendor sees the full fan-out.

The fix is a per-pool flag, include_deferred, which tells the scheduler to count deferred tasks against the slot budget. It is off by default. You can set it when editing the pool in the Airflow UI, or through the API.

The failure signature is what makes this nasty. Nothing errors. Your DAG gets faster. The vendor starts returning 429s that your retry logic absorbs, and the only symptom is that runs take a little longer and occasionally a page goes missing. Weeks can pass. Two related traps in the same family:

• max_active_tasks at the DAG level has the same blind spot with deferred tasks, and there is no equivalent opt-in flag. If you need a hard external concurrency cap, use a pool with include_deferred enabled, not DAG-level concurrency.
• On MWAA, the triggerer runs alongside the scheduler on the same Fargate task, so scheduler count and triggerer capacity are linked. If you go heavily deferrable and your deferred tasks start stalling, scheduler capacity is the thing to look at.

Failure family two: retries that make the outage worse

The default instinct is to set retries high and move on. Against a rate-limited API, a fixed retry delay across many parallel tasks is just a slower version of the same stampede.

What you want is exponential backoff with a ceiling. The shape:

from datetime import timedelta

from airflow.sdk import dag, task

@dag(
    schedule="0 5 * * *",
    catchup=False,
    max_active_runs=1,          # never let two runs of this DAG overlap
    default_args={
        "retries": 5,
        "retry_delay": timedelta(seconds=30),
        "retry_exponential_backoff": True,   # 30s, 60s, 120s, 240s...
        "max_retry_delay": timedelta(minutes=15),  # stop doubling here
        "pool": "crm_api",      # shared budget across every task touching this vendor
    },
    tags=["crm", "extract"],
)
def crm_extract():

    @task(max_active_tis_per_dag=4)
    def fetch_page(page_token: str) -> str:
        ...

crm_extract()

The lines that matter:

• retry_exponential_backoff turns retry_delay into a base rather than a constant, so repeated failures spread out instead of hammering in lockstep.
• max_retry_delay caps the doubling. Without it, a task that fails five times can sit idle for hours and blow past the window you actually cared about.
• max_active_runs=1 is the one people skip. If a run overruns its schedule, the next one starts anyway, and now two runs are fetching the same pages from the same vendor with the same credentials. This is a common way to trigger a rate limit you have never hit before.
• max_active_tis_per_dag limits how many instances of that specific task run concurrently across DAG runs, which is the right knob for dynamically mapped extraction tasks.

One thing Airflow will not do for you: honour a Retry-After header. Airflow’s retry timing is computed from your config, not from the vendor’s response. If the API tells you exactly how long to wait, you have to catch that in your own code and sleep or reschedule accordingly. Ignoring a header the vendor bothered to send is a good way to get your API key throttled harder.

Failure family three: pagination, cursors and the empty page

Back to the Monday message. The specific bug behind most “the numbers are wrong but nothing failed” incidents is a loop that treats any non-error response as a terminating condition. Three rules prevent it:

1. Never infer “done” from an empty result. Terminate on the explicit signal the API gives you: a null next_cursor, a missing Link header, a page count. An empty array with a valid cursor still has more data behind it.
2. Assert the response shape before you use it. Check the status code explicitly and validate that the fields you depend on exist. A 200 carrying {"error": "..."} should raise, not return zero rows.
3. Land raw, transform later. Write the untouched API response to S3 first, then parse from S3. When the vendor changes a field type, you can replay from raw instead of re-extracting from an API that no longer serves that window.

Where to keep incremental state

The tempting pattern is to store the last-seen timestamp in an Airflow Variable and update it at the end of a run. Do not make that your source of truth. If a run dies midway, the Variable is in an undefined state, and clearing and re-running the DAG will not restore it. Airflow’s retry and backfill machinery has no idea it exists.

Better: make each run’s window a function of the run itself, and write output to a deterministic, run-scoped location such as s3://bucket/source=crm/dt=<logical-date>/. Re-running the same interval overwrites the same prefix. That is what makes a task idempotent, and idempotency is the difference between “clear the task and let it rerun” and a two-hour manual repair.

Then overlap your windows deliberately. Many SaaS APIs order results by modified time with eventual consistency, so a record edited at the boundary can appear after you have already moved on. Query a window slightly wider than your schedule interval and rely on an idempotent upsert downstream to absorb the duplicates. Late-arriving data is not an edge case with SaaS sources. It is the normal case.

Airflow’s asset-based scheduling is the clean way to trigger downstream DAGs from this: the extract DAG produces an asset, and the transform DAG runs when the asset updates, rather than being scheduled at a time you hope is late enough.

Failure family four: credentials

API tokens rotate, sometimes on the vendor’s schedule rather than yours. Storing an API key in an Airflow Connection through the UI works, and is the wrong long-term answer: the value lives in the metadata database and there is no rotation story. On AWS, point Airflow’s secrets backend at AWS Secrets Manager. On MWAA that is an environment configuration option:

secrets.backend
  airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

secrets.backend_kwargs
  {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}

With that in place, a connection lookup for crm_default resolves against the secret at airflow/connections/crm_default. Rotating the credential is a Secrets Manager operation with no Airflow deployment involved.

Two things to know before you turn it on. First, every connection and variable lookup becomes a Secrets Manager API call, and lookups fall through to the backend before hitting the metadata database, so a DAG that reads a Variable at parse time will generate a call on every parse cycle. Move those reads inside tasks. Second, the environment’s execution role needs explicit read permission on the relevant secret ARNs, and if you use a customer-managed KMS key, decrypt permission on that key too.

Worth knowing if you are on Airflow 3: task code can no longer reach the metadata database directly. All runtime interaction goes through the Task Execution API. If you inherited custom operators that open a session and query Airflow’s own tables, that is a migration blocker, not a warning.

Failure family five: the bill

Nobody is surprised by the environment line item. They are surprised by the other four.

• Idle time. A provisioned MWAA environment bills continuously. Compute the ratio of hours billed to hours doing work. If it is bad, that is the argument for MWAA Serverless or for consolidating several thin pipelines into one environment.
• NAT Gateway. This is the classic one. Private-subnet workers calling public SaaS APIs route through a NAT Gateway, which charges hourly and per gigabyte processed. A high-volume extraction pipeline can spend more on NAT than on Airflow. VPC endpoints remove that cost for AWS service traffic, but they do nothing for calls to a third-party API, which is exactly the traffic an API pipeline generates.
• CloudWatch Logs. Task logs go to CloudWatch, and ingestion is billed per gigabyte. Set the Airflow log level per component rather than globally at DEBUG, and set a retention policy on the log groups. The default is to keep logs forever.
• S3 requests. Landing raw API responses one small object per page generates a lot of PUTs. Batch pages into larger objects where you can.

Rates and dimensions change, so model your own workload against the current pricing page rather than trusting a number from a blog post. The point is knowing which four lines to look at.

Troubleshooting Apache Airflow on AWS when API pipelines misbehave

Tasks sit in “queued” and never start

Usually a slot problem, not a broken scheduler. Check, in order: is the pool full; has DAG-level max_active_tasks been hit; is worker autoscaling at its configured maximum. On MWAA, the container and queue utilisation metrics published to CloudWatch tell you which of the three it is far faster than reading scheduler logs.

DAG file changes don’t appear

On MWAA, DAGs sync from S3 on an interval; it is not instant. If a file has been there for several minutes and still hasn’t appeared, it almost always failed to parse. Check the DAG processing logs in CloudWatch rather than the scheduler logs, because a broken import raises there and never reaches the scheduler.

A new provider package won’t install

MWAA installs from your requirements.txt in the DAGs bucket, and from Airflow 2.7.2 onward that file must include a constraint line. Without one, MWAA picks a constraint for you, and pip is free to resolve a provider version that conflicts with the Airflow build in the image.

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<AIRFLOW_VERSION>/constraints-<PYTHON_VERSION>.txt"

apache-airflow-providers-http
apache-airflow-providers-salesforce

Substitute the literal Airflow version your environment runs and the Python version bundled with it. MWAA does not expand shell variables in that file. Test the requirements file against a local Airflow image before you upload it, because a failed install on MWAA surfaces as a partially working environment rather than a clean error.

A backfill is stuck and you need to clear it

You do not need a web login token for this. MWAA exposes the Airflow REST API through a signed AWS API call, so you can drive it from CI or a runbook with normal IAM credentials:

aws mwaa invoke-rest-api 
  --name MyMWAAEnvironment 
  --path "/dags/crm_extract/clearTaskInstances" 
  --method POST 
  --body '{"dry_run": true}'

Start with dry_run set to true so the response tells you which task instances would be cleared before you actually clear them. Note that the resource paths differ between Airflow 2 and Airflow 3 environments, so confirm against the API version your environment exposes.

Deferred tasks stall forever

If deferred tasks stop resuming while the environment reports healthy, suspect the triggerer rather than your DAG. A triggerer that has lost its ability to process triggers can keep heartbeating normally, so the scheduler sees nothing wrong while every deferred task drifts toward timeout. This class of bug has been fixed and re-fixed upstream, so check your Airflow version’s release notes before assuming it is your code.

Common mistakes

• Switching to deferrable operators without enabling include_deferred on the pools that were protecting the API.
• Leaving max_active_runs unset, so a slow run and the next scheduled run compete for the same rate limit budget.
• Treating an empty response page as the end of pagination.
• Storing the incremental watermark in an Airflow Variable and updating it mid-run.
• Calling an API or reading a Variable at DAG parse time, which executes on every parse cycle rather than once per run.
• Transforming during extraction, so a vendor schema change means re-pulling data the API may no longer serve.
• Sizing the environment for peak concurrency when the actual constraint is the vendor’s rate limit.

Best practices

• One pool per vendor, sized to their published limit with headroom, and include_deferred enabled on every one of them.
• Land raw responses to S3 before parsing. Extraction and transformation are separate tasks with separate failure modes.
• Make every task idempotent and window-scoped, so “clear and rerun” is always a safe repair.
• Overlap extraction windows and deduplicate downstream rather than trusting a vendor’s timestamps to be exact.
• Secrets Manager for credentials, with the execution role scoped to specific secret ARNs.
• Alert on row counts and freshness, not just task state. A green DAG that produced 40% of yesterday’s rows is the failure you actually care about. Shipping Airflow’s StatsD metrics into Prometheus, Grafana Cloud or Datadog makes that a dashboard rather than a discovery.
• Define the environment in Terraform or OpenTofu. Recreating an MWAA environment by hand after a bad configuration change is a bad afternoon.

Frequently asked questions

Is MWAA worth it compared to self-hosting Airflow on EC2?

It depends almost entirely on how many people share the platform. MWAA’s premium buys you managed metadata database upgrades, patched images and version migration support. If one engineer maintains three DAGs, self-hosting on a modest VPS is cheaper and the operational load is real but small. Once several teams depend on the scheduler being up, the premium is easy to justify.

Should I use Step Functions instead of Airflow for API pipelines?

Step Functions is genuinely better for event-driven, AWS-service-centric orchestration with modest branching, and it scales to zero. Airflow wins when you need scheduled batch semantics, backfills over historical windows, dependencies between many pipelines, and a UI that non-platform engineers can use to see why last Tuesday failed. Backfill is usually the deciding feature.

Do deferrable operators reduce my AWS bill?

On provisioned MWAA, they reduce worker pressure, which reduces autoscaling into additional worker instances. The base environment cost is unchanged. On a Kubernetes executor setup where each task is a pod, the saving is more direct. Either way, do not adopt them purely for cost without revisiting your pool configuration first.

How do I handle a vendor with no documented rate limit?

Start conservative, one or two concurrent requests, and instrument the response status codes. Raise the pool size gradually and watch for 429s or rising latency. Latency creeping up under load is often the earlier signal, because some vendors throttle by slowing you down before they start rejecting.

Can Airflow read a Retry-After header automatically?

No. Airflow computes retry timing from retry_delay and the backoff settings on the task. If a vendor sends Retry-After, you need to handle it in your own request code or in a custom operator.

What breaks when upgrading to Airflow 3?

The big one for API pipelines is that task code can no longer access the metadata database directly; everything goes through the Task Execution API. Imports also move to the airflow.sdk namespace, and several core operators now live in the standard provider package. Audit custom operators first, since that is where direct database access hides. MWAA requires you to be on the latest Airflow 2 minor version before a major upgrade, so plan two steps.

How many DAGs can one MWAA environment handle?

The binding constraint is usually the metadata database and scheduler CPU, not DAG count. Watch metadata database memory and scheduler CPU utilisation; when either saturates, you either move up an environment class or split into multiple environments. Splitting also gives you blast-radius isolation, which matters more than people expect.

Wrapping up

Running Apache Airflow on AWS for SaaS and API pipelines is mostly not an Airflow problem. The scheduler works. The operators work. What bites is the gap between “the task succeeded” and “the data is correct,” and that gap lives in concurrency settings, pagination logic and retry design rather than anywhere Airflow will warn you about.

If you take one thing away: a green DAG is not a signal that your data is complete. Enable include_deferred on the pools protecting your vendors, terminate pagination on an explicit signal instead of an empty page, make every task idempotent, and alert on row counts. Those four things prevent most of the incidents that never show up as a failed task.

Need help with your Airflow pipelines on AWS?

I work with teams running data and API pipelines on AWS, usually somewhere between “it works but nobody trusts it” and “we need to move off cron.” Things I can help with:

• Reviewing existing DAGs for silent data loss: pagination logic, retry behaviour, pool and concurrency configuration.
• Choosing between MWAA provisioned, MWAA Serverless and self-managed Airflow, with a cost model for your actual schedule rather than a generic comparison.
• Building SaaS extraction pipelines that are idempotent and safely re-runnable, landing raw to S3 with incremental windows that survive failure.
• Cutting MWAA cost: environment right-sizing, NAT Gateway traffic, CloudWatch log volume and dependency install time.
• Airflow 2 to 3 migration audits, focused on custom operators and direct metadata database access.
• Data freshness and volume alerting in Grafana or CloudWatch, so you learn about a partial sync before the business does.

If something specific is broken, send me the DAG file, the task log, or the CloudWatch metrics for the run that went wrong. It is usually faster to look at the real thing than to describe it.

The post Apache Airflow on AWS: Building SaaS and API Pipelines That Don’t Lie to You appeared first on John Nessime.

The frustrating part is that the pipeline is fine. Webhooks are arriving, Lambda is running clean, S3 has the files, Athena returns rows. Nothing is broken in the way monitoring understands “broken”. The pipeline is faithfully recording something that is no longer true.

This post is about building a Shopify sales dashboard with AWS that survives that conversation. Not the wiring, which is well documented and mostly straightforward, but the design decisions that determine whether your numbers still hold up six months in. I’ll cover the three ingestion paths and when each one is the right call, why append-only pipelines drift, how to lay out S3 and Athena so recomputation is cheap, and what to do when the totals are already wrong.

Why a Shopify sales dashboard with AWS drifts from the admin

Here’s the thing that catches almost everyone: a Shopify order is not an event, it’s a mutable record.

An event pipeline assumes facts are immutable once written. A payment happened. A shipment left. You append it, you never touch it again, and the sum of the log is the truth. That model is why streaming architectures are so clean, and it’s exactly wrong for order data.

An order created on Monday can be edited on Tuesday, partially refunded on Friday, and fully refunded three weeks later. Every one of those changes belongs, financially, to Monday. If your pipeline appends the orders/create payload and never revisits it, Monday’s revenue is frozen at the moment of checkout and it will only ever be too high.

This is the invisible failure. Nothing alerts. No queue backs up. Your dashboard is confidently wrong, and the gap widens roughly in proportion to your return rate. A store with a two percent return rate takes a long time to notice. A fashion store running thirty percent returns notices in about a month, usually via an angry accountant.

The four adjustments that move historical numbers

• Refunds. Full or partial. A refund carries its own created_at, which is when the money moved back. The order it belongs to has a different, earlier date. You need both, and which one you attribute to depends on whether finance wants cash-basis or order-basis reporting. Ask before you build.
• Order edits. A merchant adds a line item or adjusts a quantity after the fact. The original payload is now stale. Shopify exposes both the original and the current totals precisely because of this.
• Cancellations. A cancelled order keeps existing in the API. If you filter only on payment status you will happily keep counting it.
• Test and draft orders. Test orders carry a flag marking them as such. Nobody remembers to filter these until a QA run during a quiet week produces a suspicious spike.

The design consequence is simple to state and annoying to implement: your pipeline must be able to recompute any past day. Every storage and partitioning decision below follows from that one requirement.

Getting data out of Shopify: three paths, three trade-offs

Before anything else: new Shopify apps are built on the GraphQL Admin API. The REST Admin API has been designated a legacy API and new public apps must use GraphQL. If you’re starting fresh, start there. If you inherited a REST integration, it probably still runs, but you’re on borrowed time and you should plan the migration rather than discover the deadline.

Path 1: EventBridge partner event source

Shopify can deliver webhooks straight into an Amazon EventBridge partner event bus in your account. No public endpoint, no API Gateway, no HMAC verification code, because verification only applies to HTTPS deliveries. Shopify’s own docs confirm EventBridge and Pub/Sub deliveries skip it.

You create the source in the Shopify app configuration using your AWS account ID, region and a source name, then associate it with an event bus in the EventBridge console and write rules to route it. The address you register with Shopify is the partner event source ARN, not the event bus ARN. That distinction accounts for a large share of the “I set it up and nothing arrives” threads on the Shopify forums.

A rule matching everything from the Shopify partner source looks like this. Start broad, then narrow once you’ve seen the real shape of an event:

{
  "source": [ { "prefix": "aws.partner/shopify.com" } ]
}

Send that to an SQS queue with a dead-letter queue attached rather than straight to Lambda. Buffering gives you a replay buffer when a downstream deploy goes wrong, and the DLQ means a bad payload parks itself instead of poisoning the whole rule. This is the path I reach for first for anything already on AWS.

Path 2: HTTPS webhooks into API Gateway and Lambda

The conventional route, and the right one if you need webhook delivery outside AWS too, or you want the payloads to pass through something you fully control. The cost is that you now own an internet-facing endpoint and the HMAC verification on it.

Verify against the raw request body, before any JSON parsing. Re-serialising the payload changes byte-for-byte content and the signature will never match. Use a constant-time comparison so the check doesn’t leak timing information:

import base64, hashlib, hmac

def verify(raw_body: bytes, header_hmac: str, secret: str) -> bool:
    digest = hmac.new(secret.encode(), raw_body, hashlib.sha256).digest()
    computed = base64.b64encode(digest).decode()
    return hmac.compare_digest(computed, header_hmac)

Shopify sends the signature in the X-Shopify-Hmac-SHA256 header, base64-encoded, computed with your app’s client secret over the raw body. Store that secret in Secrets Manager or as an SSM SecureString parameter, not in a Lambda environment variable.

Acknowledge fast. Shopify’s timeout is short and it retries with backoff over a finite window, so a handler that does real work inline will generate a wall of duplicate deliveries during a flash sale, exactly when you can least afford it. Return 2xx immediately, do the work asynchronously.

Path 3: scheduled GraphQL bulk pull

Webhooks give you low latency. They do not give you completeness. Anything that fails past its retry window is gone, and Shopify will eventually remove a subscription that keeps failing. That’s a silent data loss mode with no local symptom at all.

So run a scheduled reconciliation pull alongside the stream. Shopify’s GraphQL bulk operations are built for this: you submit a query, it runs asynchronously, and you fetch a JSONL result file when it finishes. That’s the right tool for backfills and nightly catch-up, rather than paginating thousands of pages against a points-based rate limiter and getting throttled halfway through.

A nightly job that re-pulls the last seven to fourteen days and overwrites those partitions costs almost nothing and quietly fixes every category of drift described above. If you build one thing from this post, build that.

Decide what “revenue” means before you write a line of SQL

This is where most reconciliation arguments actually live, and it isn’t an engineering problem at all until you’ve had the conversation.

Shopify’s own sales reporting builds total sales from gross sales, minus discounts, minus returns, plus taxes and shipping. Gift card sales sit outside that in a separate finance report. If your dashboard sums order totals and calls it revenue, you have built a different metric with the same name, and it will disagree with the admin forever no matter how good your pipeline is.

Write the definition down. Put it in the dashboard as a tooltip. When someone challenges a number, you want the argument to be about the definition, not about whether your infrastructure works.

The currency trap

If the store sells in more than one currency, the money fields split in two. Shopify exposes totals as a set containing both shop_money and presentment_money: the amount in the store’s base currency, and the amount the customer actually saw and paid.

Sum the presentment amounts across a multi-currency store and you get a number with no meaning at all, euros and yen added together as if they were the same unit. For a single reporting figure you want the shop-currency side. Keep the presentment amount and its currency code in the table anyway, because the day someone asks “how much did we actually sell in Germany”, you’ll want it and it is painful to backfill.

One caveat worth knowing: orders created through the API rather than through checkout can behave differently from native multi-currency checkout orders. If your store takes orders from an ERP or a marketplace integration, spot-check a few of those specifically.

Storage layout: partition by order date, never by arrival date

Two layers in S3. Keep them separate and keep them honest about what they are.

1. Raw. Every payload exactly as received, partitioned by ingestion date. Append-only, never edited. This is your audit trail and your rebuild source. Lifecycle it to a colder storage class after a few months, don’t delete it.
2. Curated. One row per order representing current state, in Parquet, partitioned by order date. This is what the dashboard queries. It is derived, disposable and rewritable.

The partitioning choice on the curated layer is the load-bearing decision in the whole design. If you partition by arrival date, which is what Amazon Data Firehose does by default because it buckets on the moment it writes the file, then a refund that arrives three weeks late lands in today’s partition. Correcting Monday now means finding and rewriting fragments scattered across twenty other partitions. Partitioned by order date, correcting Monday means overwriting exactly one prefix.

Firehose can do this with dynamic partitioning, which routes records by keys inside the payload rather than by write time. If you’re not using Firehose, extract the order date in your Lambda and write the prefix yourself.

Use partition projection so Athena stops guessing

The default Glue Data Catalog approach means running a crawler or issuing MSCK REPAIR TABLE to register new partitions. Forget one and you get a query that silently returns nothing for recent days. Nobody notices until Monday.

Partition projection removes the metastore lookup entirely. You tell Athena the shape of the partition keys and it calculates the prefixes at query time:

CREATE EXTERNAL TABLE shop_orders (
  order_id             bigint,
  order_number         string,
  created_at           timestamp,
  financial_status     string,
  cancelled_at         timestamp,
  is_test              boolean,
  total_shop           decimal(12,2),
  shop_currency        string,
  total_presentment    decimal(12,2),
  presentment_currency string
)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://your-bucket/curated/orders/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.order_date.type' = 'date',
  'projection.order_date.format' = 'yyyy-MM-dd',
  'projection.order_date.range' = '2019-01-01,NOW',
  'projection.order_date.interval' = '1',
  'projection.order_date.interval.unit' = 'DAYS',
  'storage.location.template' =
    's3://your-bucket/curated/orders/order_date=${order_date}/'
);

Set the range start to your store’s actual first order month. Projection generates every prefix in the range, so a range starting a decade too early makes wide scans slower for no benefit.

Parquet matters here for the same reason. Athena bills on bytes scanned, so a columnar format with good compression cuts the bill directly, and a dashboard that only ever selects six columns from a forty-column table never touches the rest.

Net sales in one query

With refunds in their own table keyed by order and carrying their own date, attributing them back to the original order day is a left join and a subtraction:

SELECT
    o.order_date,
    SUM(o.total_shop)                                AS gross_shop,
    SUM(COALESCE(r.refunded_shop, 0))                AS refunded_shop,
    SUM(o.total_shop - COALESCE(r.refunded_shop, 0)) AS net_shop
FROM shop_orders o
LEFT JOIN (
    SELECT order_id, SUM(amount_shop) AS refunded_shop
    FROM shop_refunds
    GROUP BY order_id
) r ON r.order_id = o.order_id
WHERE o.order_date BETWEEN '2025-01-01' AND '2025-01-31'
  AND o.is_test = false
  AND o.cancelled_at IS NULL
GROUP BY o.order_date
ORDER BY o.order_date;

Note the two filters doing quiet work at the bottom. Those two lines are the difference between a number finance accepts and a number they don’t.

Choosing the dashboard layer

Once the data is correct, this part is genuinely a preference. All of these work.

• Amazon QuickSight, now delivered as part of Amazon Quick Suite, is the least-friction option if you’re already in AWS. Its in-memory SPICE layer means viewers aren’t firing an Athena query per chart interaction, which controls both latency and scan cost. Per-viewer pricing tends to be the deciding factor either way, so model it for your actual audience size before committing.
• Grafana with the Athena data source is a good fit if you’re already running Grafana for infrastructure and want commercial and operational panels on one screen. Grafana Cloud removes the hosting question if you’d rather not run it.
• Power BI makes sense when the finance team already lives in Microsoft 365 and models in DAX. The cross-cloud hop is real but manageable.
• Metabase or a self-hosted alternative on a small VPS from a provider like InterServer or Hetzner is the pragmatic answer for a handful of internal viewers, where per-seat BI licensing costs more than the entire pipeline.

The honest trade-off: managed BI costs more per month and saves you from becoming the person who patches the reporting server. Self-hosting inverts that. Neither is wrong, but pick deliberately rather than by inertia.

Troubleshooting: symptom to cause

Totals are consistently higher than the Shopify admin

Almost always refunds, cancellations or test orders. Check in that order. If the gap grows with the age of the reporting window, it’s refunds. If it’s a fixed offset on specific days, look for test orders or a QA run.

Totals are lower, and recent days are missing rows

Either partitions aren’t registered, which projection fixes permanently, or the webhook subscription has been dropped after repeated delivery failures. Check the subscription still exists before you go digging through Lambda logs. A nightly bulk pull would have masked this, which is another argument for having one.

Orders appear twice

Shopify’s delivery model is at-least-once, not exactly-once, and you may also have more than one subscription on the same topic. Deduplicate on the delivery ID header before you touch anything else, and make the write itself idempotent so a duplicate is a no-op rather than a second row.

Numbers are right on the daily view, wrong on the monthly

Timezone. Order timestamps carry an offset; your partition key is a date string. If you derive the date in UTC and the store reports in a local timezone, orders near midnight land on the wrong day. That averages out over a month, which is exactly why the discrepancy hides until month boundaries.

Athena costs jumped without more data

Someone built a dashboard with a filter that doesn’t hit the partition column, so every panel refresh scans the full table. Look at bytes scanned per query and check whether the BI tool is caching results or re-querying on every interaction.

Common mistakes

• Treating orders as immutable events and never revisiting a past day.
• Partitioning on arrival time because that’s the default, then discovering corrections are expensive.
• Verifying the HMAC against a re-serialised body instead of the raw bytes.
• Doing real work inside the webhook handler, generating duplicates under load.
• Summing presentment amounts across currencies.
• Registering the event bus ARN with Shopify instead of the partner event source ARN.
• Relying on webhooks alone with no scheduled reconciliation.
• Shipping a “revenue” number without ever defining what it includes.

Best practices

• Keep raw and curated layers separate. Raw is append-only; curated is rewritable.
• Make every partition idempotently rebuildable from raw. Test that path deliberately, before you need it.
• Buffer through SQS with a dead-letter queue. Free replay, free isolation of bad payloads.
• Run a nightly bulk pull over a rolling window and overwrite those partitions.
• Use partition projection. It removes an entire category of silent failure.
• Alarm on the absence of events, not just on errors. A CloudWatch alarm on zero orders processed in an hour during business hours catches broken subscriptions the same day.
• Store the API secret in Secrets Manager and scope the Lambda role to the exact prefixes it writes.
• Publish a reconciliation panel comparing your total to the admin’s for the same window. Surfacing the gap builds more trust than hiding it.

Frequently asked questions

Do I need a data warehouse, or is S3 and Athena enough?

For a single store’s order data, S3 with Athena is almost certainly enough, and it’s cheaper because you pay per query rather than for a running cluster. Redshift starts to earn its place when you’re joining Shopify data against several other large sources, or when concurrent query load makes Athena’s queue times noticeable.

How near-real-time can this be?

Events land within seconds. The practical floor is your buffering window, since writing one tiny file per order gives you a small-files problem that ruins query performance. A few minutes of buffering is the usual compromise. If you genuinely need sub-minute order counts, put a live counter in DynamoDB alongside the analytical pipeline rather than trying to make the data lake do both jobs.

Which webhook topics should I subscribe to?

At minimum, order creation, order update, order cancellation and refund creation. Update and refund topics are the ones people skip, and they’re exactly the ones carrying the corrections. Subscribe to fewer topics than you think you need and add rather than subscribing to everything, since every extra topic is volume you pay to store and process.

Can I skip AWS and use a connector tool?

Yes, and for many stores that’s the right answer. A managed connector into a hosted warehouse gets you a working dashboard in an afternoon. You’re paying a monthly fee to avoid owning any of this, and trading away control over the data model. Building it on AWS wins when you need Shopify data joined to systems the connector doesn’t cover, or when row-based connector pricing outgrows the infrastructure cost.

How do I backfill historical orders?

Use a GraphQL bulk operation rather than paginating the API. Submit the query, poll for completion, then stream the JSONL result into your raw bucket and run the same transformation your live pipeline uses. If backfill and live processing use different code paths, they will diverge, and you’ll spend an afternoon working out which one is lying.

What does a setup like this cost to run?

For a typical single store, the pipeline itself is small money: Lambda invocations, a few gigabytes in S3, and Athena billed on bytes scanned, which partitioning and Parquet keep low. The BI seats are usually the largest line item, which is why the dashboard layer decision deserves more thought than the ingestion one. Model it against current published rates rather than trusting any figure you read in a blog post, including this one.

The one thing to take away

A Shopify sales dashboard with AWS doesn’t fail because the pipeline breaks. It fails because the pipeline keeps working perfectly on data that has since changed underneath it.

Design for correction from the first commit. Partition by order date, keep the raw layer so you can always rebuild, run a scheduled pull to catch what the stream missed, and agree on what revenue means before anyone builds a chart. Do that and the Monday morning screenshot comparison becomes a non-event, which is the highest praise a reporting pipeline ever gets.

Need help with your Shopify data pipeline on AWS?

I design and build ecommerce data pipelines and reporting stacks on AWS. Typical engagements look like:

• Working out why an existing Shopify dashboard disagrees with the admin, and fixing the root cause rather than patching the query
• Building the ingestion layer end to end: EventBridge or API Gateway, Lambda, SQS with dead-letter handling, and a scheduled GraphQL bulk reconciliation job
• Designing the S3 layout, Glue schema and Athena tables so past days can be recomputed cheaply and partitions never go missing
• Migrating REST Admin API integrations to GraphQL before the deadline forces the issue
• Building the dashboard itself in QuickSight, Grafana or Metabase, including the metric definitions finance will actually sign off on
• Cutting Athena scan costs and BI licensing on a reporting stack that has grown more expensive than anyone planned

If you’re in the middle of one of these, send me the actual thing: the Athena query, the S3 prefix layout, the two totals that don’t match. It’s a much faster conversation than describing it in the abstract.

The post Shopify Sales Dashboard with AWS: Build One That Actually Reconciles appeared first on John Nessime.

That is usually the moment someone says “let’s just pull it into a warehouse.” A Jira analytics pipeline built on AWS Lambda, S3 and Athena is a reasonable answer to that, and it is genuinely cheap to run. It is also easy to build a version that looks correct for two weeks and then quietly reports numbers that are thirty percent wrong.

This post covers the extraction and modelling problems that actually cost you time: the search endpoint Atlassian removed, the change history that truncates without erroring, the incremental load pattern that duplicates rows across partitions, and the S3 layout decisions that decide whether Athena costs you pennies or hundreds. Code where it clarifies something, and honest notes on what I would skip.

Where Jira’s own reporting genuinely stops

Give the built-in tooling its due first. Jira’s velocity, burndown and control charts are fine for a single team inspecting its own recent work, they need no infrastructure, and they update instantly. Marketplace apps like eazyBI and Custom Charts cover a lot of ground without you writing a line of Python. If your question is “how did this sprint go,” you do not need a pipeline.

Where it stops is anything that crosses a boundary. Comparing lead time across projects that use different workflows. Joining ticket data to deploy events from your CI system or incident data from PagerDuty. Retaining a consistent view of history after someone renames a status or archives a board. Answering a question nobody anticipated when the board was configured. Those need the raw data somewhere you control, in a shape you decide.

The shape of the pipeline

The architecture is unremarkable, which is the point:

1. EventBridge Scheduler triggers a Lambda function on a schedule.
2. Lambda reads a Jira API token from Secrets Manager and pages through the Jira Cloud REST API.
3. It writes Parquet files to S3, partitioned by load date.
4. The Glue Data Catalog holds the table definitions, with partition projection so nothing has to crawl.
5. Athena queries S3 directly. Grafana, Power BI, Metabase or QuickSight sit on top of Athena.

No cluster, no always-on database, nothing to patch. The whole thing costs about as much as a small EC2 instance for a mid-sized Jira site, and most of that is S3 storage. Terraform or CloudFormation to define it, GitHub Actions to deploy it.

The complexity is not in the wiring. It is in four places, and they are worth taking in order.

Failure family 1: the search endpoint you were probably going to use is gone

Almost every Jira extraction tutorial and a good number of client libraries still reach for GET /rest/api/3/search. Atlassian removed it from Jira Cloud. It returns 410 Gone. If you are copying a script from a blog post that predates the change, this is the first thing that breaks, and at least it breaks loudly.

The replacement is /rest/api/3/search/jql, available as both GET and POST. Use POST for anything real, because JQL strings get long and you avoid URL encoding entirely. Three behavioural changes matter more than the URL:

Pagination is cursor-based, and there is no total

startAt is gone. You get a nextPageToken back and hand it to the next request. There is no total in the response, which means any progress bar, any “expected N rows” sanity check, and any loop that terminated on startAt >= total needs rewriting.

If you only need a count, there is a separate operation, POST /rest/api/3/search/approximate-count, which takes a JQL body and returns an approximate figure without paging through results. It is genuinely useful as a reconciliation check: run it before extraction, compare against the row count you actually wrote, and alert on a large gap.

There have been persistent community reports of isLast behaving unreliably on this endpoint, including tokens that chain without ever terminating. Do not trust isLast as your loop condition. Terminate on the absence of nextPageToken, and put a hard page cap in as a circuit breaker so a bad token cannot burn your entire Lambda budget in one invocation.

def search_issues(session, base_url, jql, fields, max_pages=2000):
    """Page through /search/jql. Terminates on missing nextPageToken,
    not on isLast, which has been reported as unreliable."""
    token = None
    for _ in range(max_pages):
        body = {"jql": jql, "fields": fields, "maxResults": 100}
        if token:
            body["nextPageToken"] = token
        r = session.post(f"{base_url}/rest/api/3/search/jql",
                         json=body, timeout=60)
        r.raise_for_status()
        page = r.json()
        for issue in page.get("issues", []):
            yield issue
        token = page.get("nextPageToken")
        if not token:
            return
    raise RuntimeError("page cap hit, refusing to loop further")

You have to ask for fields explicitly

The new endpoint does not hand you every field by default. Omit fields and you get essentially nothing back. This is the failure that looks like success: the pipeline runs, files land in S3, row counts look plausible, and every analytical column is null.

Be explicit and be narrow. Every field you request costs response size, and Jira sites accumulate hundreds of custom fields nobody uses. Name what you need:

FIELDS = [
    "summary", "status", "issuetype", "project", "priority",
    "assignee", "reporter", "created", "updated", "resolutiondate",
    "labels", "components", "parent",
    "customfield_10016",   # story points on this site, verify yours
]

Custom field IDs are per-site. Do not hardcode one you read in someone else’s blog post. Pull /rest/api/3/field once, find the field by name, and either store the mapping in config or resolve it at runtime and log what it resolved to. When a Jira admin rebuilds a field, an ID-based pipeline goes null and a name-resolving pipeline keeps working.

Rate limits and the fifteen-minute wall

Jira Cloud applies cost-based rate limiting and returns HTTP 429 when you exceed it, typically with a Retry-After header. Respect that header rather than inventing your own backoff. A naive retry loop that ignores it turns a brief throttle into a sustained one.

Lambda’s hard ceiling is fifteen minutes. A full historical backfill of a large Jira site will not finish in one invocation, and the ugly failure mode is a function that times out at minute fifteen having written half its data with no record of where it stopped. Two ways out:

• Shard the work. Fan out one Lambda invocation per project key, or per month of created date. Each one is small, independently retryable, and finishes well inside the limit.
• Checkpoint and continue. Persist the current nextPageToken to DynamoDB or S3 after each page. When the function is close to its deadline, stop cleanly and let Step Functions re-invoke it from the checkpoint.

Sharding is simpler and I reach for it first. Checkpointing is what you need when a single project is itself too large. Either way, watch Lambda’s ephemeral storage: the default /tmp allocation is 512 MB, and buffering a large Parquet write there will fail before your API calls do. Raise it or stream to S3 in chunks.

Failure family 2: the change history is the whole point, and it truncates silently

Current issue state answers almost none of the interesting questions. How long a ticket spent waiting for review, how many times it bounced back from QA, when it actually entered development rather than when someone remembered to drag the card, all of that lives in the changelog. Without it you have a list of tickets. With it you have a process.

Here is the trap. Requesting an issue with expand=changelog returns a capped number of history entries, commonly the first hundred, and it does not tell you it truncated. Well-worn tickets with lots of field edits blow past that easily. Your data does not error, it just quietly loses the later transitions, which are usually the ones near completion. Cycle time comes out looking better than reality.

Two correct approaches:

• GET /rest/api/3/issue/{issueIdOrKey}/changelog and page it properly. Correct, but it is one request per issue, which is brutal against rate limits on a large site.
• POST /rest/api/3/changelog/bulkfetch, which accepts issueIdsOrKeys and an optional fieldIds filter, with the same nextPageToken pagination. Far fewer round trips.

Bulk fetch is the one I would use, with a caveat: it has carried an experimental designation, so pin your expectations and keep the per-issue path available as a fallback. Filter fieldIds to status if status history is all you model. That cuts the response size enormously, because most changelog volume is description edits and label churn nobody analyses.

Store changelog as its own narrow table, one row per field change, not nested inside the issue record. Athena can handle nested structures, but flat is dramatically easier to reason about in SQL and much cheaper to scan.

Failure family 3: the incremental load that duplicates everything

This is the one that bites hardest, because nothing fails. Everything runs green for weeks and the numbers are wrong the entire time.

The obvious incremental design is a watermark: track the last successful run, then pull updated >= watermark on each run and append the results to S3. It is the standard pattern and it works for immutable event data.

Jira issues are not immutable. A ticket created in January and touched every week appears in every weekly extract. If you append each run into its own partition, that one issue now exists as a row in twenty partitions. Athena is doing exactly what you asked: SELECT count(*) FROM jira_issues WHERE status = 'Open' counts twenty things. Your open bug count is inflated, your throughput is inflated, and the inflation is proportional to how much a ticket gets edited, which correlates with how contentious it was. The busiest work is the most over-counted.

Nobody catches this from the dashboard, because the numbers move in the right direction and look approximately sane. It surfaces months later when someone reconciles against a JQL query in Jira and the two disagree.

Pick one of three fixes, deliberately

Full snapshot per load, dedupe at read time. Keep every version, partition by load_date, and always read through a view that takes the latest row per issue key. Storage is cheap, history is free, and you can answer “what did the board look like in March” without any extra machinery. The cost is that every query pays for the deduplication.

CREATE OR REPLACE VIEW jira_issues_current AS
SELECT * FROM (
  SELECT
    i.*,
    ROW_NUMBER() OVER (
      PARTITION BY issue_key
      ORDER BY load_date DESC, updated DESC
    ) AS rn
  FROM jira_issue_snapshot i
  WHERE load_date >= date_format(current_date - interval '7' day, '%Y-%m-%d')
) WHERE rn = 1;

The load_date filter inside the view matters. Without it the deduplication window scans the entire table on every query, which is the single most common way a cheap Athena setup becomes an expensive one.

Overwrite the affected partitions. Partition by something stable, usually issue created month, and rewrite whole partitions when any issue in them changes. Clean reads, no dedupe cost, but you now own read-modify-write logic in Lambda and a concurrency problem if two runs overlap.

Use an ACID table format. Apache Iceberg gives you real MERGE INTO semantics on S3, and Athena supports it natively. This is the right answer if you are already running Iceberg elsewhere or if the pipeline will grow to a dozen sources. It is not worth adopting solely to load one Jira site, because you inherit compaction and snapshot expiry as ongoing maintenance.

For a single Jira site feeding a handful of dashboards, snapshot plus a dedupe view is what I would build. It has the fewest moving parts and it gives you point-in-time history as a side effect, which you will want the first time someone asks a retrospective question.

Deletions and moves

A JQL watermark query never returns deleted issues, so they persist in your data forever. Same for issues moved out of scope or into an archived project. Periodically reconcile: pull the full set of issue keys with a minimal fields list, compare against what you hold, and mark the difference. Monthly is usually enough. Skip this and your historical counts drift upward permanently.

Failure family 4: the S3 layout that decides your Athena bill

Athena’s standard pricing model bills on bytes scanned, rounded up, with a small per-query minimum. DDL statements are free. That means your storage layout, not your SQL, is what determines cost.

Three things do almost all the work:

• Write Parquet, not JSON. Columnar storage lets Athena read only the columns your query touches. A dashboard selecting five columns from a forty-column table scans a small fraction of what the equivalent JSON would. Snappy or Zstd compression on top of that.
• Avoid tiny files. A Lambda that runs hourly and writes one small file per invocation produces thousands of objects. Athena spends more time opening files than reading them. Batch writes so files land in the low hundreds of megabytes, or run a periodic compaction job.
• Use partition projection. This is the one people skip and then wonder why queries have a fixed few-second overhead.

Partition projection lets Athena calculate partition locations from table properties instead of calling GetPartitions against the Glue Data Catalog. On a table with a couple of years of daily partitions, that lookup is real latency on every single query, and it grows as the table does. Projection removes it, and removes your need for a Glue crawler entirely, which is one less scheduled job and one less IAM role.

CREATE EXTERNAL TABLE jira_issue_snapshot (
  issue_id        string,
  issue_key       string,
  project_key     string,
  issue_type      string,
  status          string,
  status_category string,
  assignee_id     string,
  created         timestamp,
  updated         timestamp,
  resolutiondate  timestamp,
  story_points    double
)
PARTITIONED BY (load_date string)
STORED AS PARQUET
LOCATION 's3://example-jira-lake/issue_snapshot/'
TBLPROPERTIES (
  'projection.enabled'                = 'true',
  'projection.load_date.type'         = 'date',
  'projection.load_date.format'       = 'yyyy-MM-dd',
  'projection.load_date.range'        = 'NOW-3YEARS,NOW',
  'projection.load_date.interval'     = '1',
  'projection.load_date.interval.unit'= 'DAYS',
  'storage.location.template'         =
    's3://example-jira-lake/issue_snapshot/load_date=${load_date}'
);

One caution that surprises people: projection describes partitions Athena will look for, not partitions that exist. Set a range wider than your data and queries without a load_date filter will probe empty prefixes. Keep the range tight to what you actually hold.

Finally, set BytesScannedCutoffPerQuery on the Athena workgroup. It kills any query that exceeds a scan threshold. One analyst running SELECT * against three years of data in a BI tool’s preview pane is the classic surprise line item, and this stops it at the source.

Modelling time in status

Once status changelog rows are landed flat, the core metric is a window function. Each transition’s duration is the gap to the next transition on the same issue:

WITH transitions AS (
  SELECT
    issue_key,
    to_status,
    changed_at,
    LEAD(changed_at) OVER (
      PARTITION BY issue_key ORDER BY changed_at
    ) AS next_changed_at
  FROM jira_changelog
  WHERE field_id = 'status'
    AND load_date >= date_format(current_date - interval '90' day, '%Y-%m-%d')
)
SELECT
  issue_key,
  to_status,
  SUM(date_diff('second', changed_at,
                COALESCE(next_changed_at, current_timestamp))) / 3600.0
    AS hours_in_status
FROM transitions
GROUP BY issue_key, to_status;

The COALESCE handles the current status, which has no successor transition. Two modelling decisions to make consciously: whether to subtract non-working hours, and how to treat a ticket that moves backwards through the workflow. Both are business questions, not technical ones, and both should be settled in a documented view rather than reimplemented in each dashboard.

Map raw status names to a stable category early. Teams rename statuses constantly, and a metric keyed on the literal string breaks the moment someone changes “In Review” to “Peer Review”. A small mapping table joined at query time keeps history comparable.

Troubleshooting

• Every analytical column is null. You did not pass fields, or you passed a custom field ID from another site. Log the resolved field list on every run.
• Counts higher than the same JQL in Jira. Duplicate rows across partitions. Check whether your query goes through the dedupe view or straight at the base table.
• Extraction loops forever. Do not terminate on isLast. Terminate on missing nextPageToken and keep a page cap.
• HTTP 410 from the API. You are still calling the removed /rest/api/3/search. Check your client library version too, not just your own code.
• Cycle times suspiciously low. Truncated changelog. Move to the dedicated changelog endpoint or bulk fetch.
• Athena returns zero rows but the files are there. Partition projection range does not cover the partition, or the S3 prefix does not match storage.location.template. Compare a real object key against the template character by character.
• Queries slow down as the table grows. Either you have no projection and Glue lookups dominate, or you have accumulated small files.
• Authentication failures overnight with no deploy. API token expired or was revoked. Alert on the specific status code rather than on “run failed”.

Common mistakes

• Appending incremental extracts without a deduplication strategy.
• Building on current issue state and adding changelog later. Retrofitting history is far more work than including it from the start.
• Hardcoding custom field IDs copied from documentation.
• Storing the API token in a Lambda environment variable instead of Secrets Manager.
• Running an hourly schedule when the dashboard is read once a day. You pay in small files and API quota for freshness nobody uses.
• Letting BI tools query base tables directly instead of curated views.
• Never reconciling against Jira. A scheduled check comparing approximate count to your row count catches drift within a day.

Best practices for a Jira analytics pipeline that survives

• Land raw API responses to S3 before transforming. When your parsing is wrong, and it will be, you replay from raw rather than re-hammering the API.
• Make the load idempotent. Re-running for the same date should produce the same result, not a second copy.
• Define the pipeline in Terraform or CloudFormation and deploy it from CI. The IAM policy for Lambda and Athena is fiddly and you do not want to rebuild it from memory.
• Alert on a run that succeeds with zero rows, not just on runs that error. Silent empty loads are the more common failure.
• Scope the Jira token to a service account with read access to exactly the projects you need.
• Put an S3 lifecycle policy on the raw zone. It grows faster than you expect and nobody queries last year’s raw JSON.
• Expose curated views, not tables, to Grafana, Metabase, Power BI or QuickSight, and enable Athena’s query result reuse for repeated dashboard loads.

Frequently asked questions

Should I use Lambda or Glue for Jira extraction?

Lambda, for an API-driven pull. The work is mostly waiting on HTTP responses, which Spark’s distributed compute does nothing for, and you would be paying for a Glue job’s minimum billing on something that is idle. Glue earns its place downstream, if you have heavy joins across several sources. Athena CTAS often covers that too.

How often should the pipeline run?

Match the decision cadence, not the data cadence. Delivery metrics are reviewed weekly or in sprint ceremonies, so daily is almost always enough and produces far better file sizes. Reserve hourly for something with a real-time consumer, and know that you are buying that freshness with small-file overhead and API quota.

Can I use webhooks instead of polling?

You can, and for near-real-time reaction it is the right tool. For analytics it is a poor primary source, because a missed delivery leaves a permanent hole you have no way to detect. The pattern that works is webhooks for freshness plus a scheduled reconciliation pull as the source of truth. If you only build one, build the scheduled pull.

Does this work with Jira Data Center or Server?

The AWS half is identical. The extraction half is not. Cursor pagination and the /search/jql endpoint are Cloud-only changes, so self-hosted instances still use the older offset-based /rest/api/2/search. Write the extractor behind an interface if you need to support both, and expect the auth model to differ as well.

How much does an Athena-based Jira pipeline cost to run?

The mechanism matters more than any figure I could quote. You pay for S3 storage, Lambda invocation time, and Athena per byte scanned with a small per-query minimum. Jira issue data compresses extremely well as Parquet, so storage stays small. The variable is dashboard query volume, which is why partitioning and workgroup scan limits matter. Model it against current AWS rates and your own expected query count.

Why not just use eazyBI or a Jira reporting app?

Often you should. If your questions stay inside Jira, a Marketplace app gets you there in an afternoon with no infrastructure. The case for a pipeline is joining Jira to data that lives elsewhere, retaining history the app does not, or needing your data in a warehouse you already run. Build the pipeline when the app has actually failed you, not in anticipation.

Should I load into Redshift instead of querying S3?

Only if you have concurrent BI users hitting the same tables constantly and Athena’s per-query latency is a real complaint. Jira data volumes are small, the query pattern is bursty, and Athena’s zero-idle-cost model fits that far better. Redshift makes sense as a consolidation layer across many sources, not for one issue tracker.

The one thing to carry away

The hard part of a Jira analytics pipeline is not the AWS wiring. Lambda, S3, Glue and Athena will be working within a day. The hard part is that Jira issues are mutable, so an incremental load that appends is an incremental load that duplicates, and it does so without a single error in your logs.

Decide your deduplication strategy before you write the first extract, not after someone reconciles a dashboard against JQL and finds a gap. Snapshot with a read-time dedupe view, partition projection so queries stay fast as history accumulates, and a scheduled reconciliation check that alerts on drift. Get those three right and the rest is plumbing.

Need this built or fixed?

I design and run serverless data pipelines on AWS, and Jira extraction has more sharp edges than most sources. Things I can help with directly:

• Migrating an extractor off the removed /rest/api/3/search endpoint to cursor-based /search/jql without losing rows in the cutover
• Auditing an existing Jira pipeline for duplicate rows and reconciling your numbers back against JQL
• Building changelog-based cycle time and time-in-status models that survive workflow renames
• Restructuring an S3 layer with Parquet, partition projection and file compaction to cut Athena scan costs
• Packaging the whole thing as Terraform or CloudFormation with CI deployment and least-privilege IAM
• Connecting Athena to Grafana, Power BI, Metabase or QuickSight with curated views instead of raw tables

Send me your extractor code, a Glue table definition, or an Athena query that is scanning more than it should, and I will tell you what I would change.

The post Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything) appeared first on John Nessime.

Acme has nineteen. The assistant retrieved five chunks that happened to mention Acme, because five is roughly how many chunks a retrieval returns, and then it counted them. There is no bug. Nothing failed. Similarity search returned the most similar things and the model described what it was given.

That is the defining problem with preparing CRM data for Amazon Bedrock, and it is upstream of chunk sizes and embedding models. CRM data is structured. Retrieval-augmented generation is built for unstructured text. Feed a table of opportunities through an embedding pipeline and you get a system that is genuinely good at “what did we discuss with this account” and quietly terrible at “how many”, “how much” and “since when”.

This covers the split that fixes it, what a good document actually looks like, the permissions problem nobody notices until it is a problem, and how to know whether any of it works.

Route the question before you build anything

Your users will ask two kinds of question and they need two different mechanisms.

• Narrative questions. “What were the objections on the Acme renewal?” “Summarise our history with this account.” “What did the customer say about pricing?” These live in notes, emails, call summaries and descriptions. Semantic retrieval is exactly right for them.
• Analytical questions. “How many open deals over fifty thousand?” “What is total pipeline this quarter?” “Which accounts have had no activity in ninety days?” These need aggregation, filtering and joins. Retrieval cannot do them and will not tell you it cannot.

The useful part is that you do not have to build the second half yourself. Bedrock Knowledge Bases supports structured data retrieval: you point it at Amazon Redshift or a Glue Data Catalog, and it generates and executes SQL from natural language against your data in place, with no copying or embedding. The counting question goes to SQL, the narrative question goes to the vector store, and both answers are right for the right reasons.

So the first architectural decision is not which chunking strategy to use. It is: which of your fields are text worth embedding, and which are facts worth querying. Get that wrong and no amount of tuning downstream will save it.

One practical note if you take the structured route. Bedrock’s execution role authenticates to Redshift as an IAM identity, and you grant it access the same way you would any other reader. Point it at curated views rather than base tables, exactly as you would a BI tool:

-- Give the knowledge base the same narrow access you'd give a
-- reporting user. A view is a contract; a schema grant is not.
GRANT USAGE ON SCHEMA analytics
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

GRANT SELECT ON analytics.v_opportunities
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

Named columns and clear table names matter more here than anywhere else, because the model is reading your schema to write the SQL. A column called flag_2 will produce exactly the query quality it deserves.

What a good document looks like

For the narrative half, the instinct is to export records to CSV and let the chunker deal with it. That produces chunks that are half a row of one record and half a row of another, with no context about which is which.

Build documents deliberately instead. Three rules that do most of the work:

One record, one document. An opportunity and its notes become a single file. A chunk from it is then always about one thing.

Denormalise the context in. A note that says “they want a two-year term” is useless in isolation. Embed the account name, the opportunity name, the stage and the owner into the same document, so a chunk carries enough context to be retrieved and to be understood once retrieved.

Write it as prose, not as fields. Embedding models were trained on language. Stage__c: Negotiation embeds worse than “The opportunity is at the Negotiation stage.” This feels silly and it measurably improves retrieval.

On chunking itself: start with the default strategy and change it only when evaluation tells you to. If your documents are one record each and reasonably short, semantic or hierarchical chunking buys you less than people expect. The gains that matter come from document construction and metadata, not from chunk size.

The permissions problem

This is the one that turns a nice project into an incident, and it is easy to miss because nothing about it looks broken.

Salesforce has an elaborate sharing model: role hierarchies, sharing rules, territory management, field-level security. A rep sees their accounts and not everyone else’s. Your vector store has none of that. Once a record is embedded, it is a vector like any other, and anyone who can query the knowledge base can retrieve it.

So a rep asks about a competitor deal they were never on the team for, and the assistant helpfully summarises it. Or someone asks about compensation-adjacent records and gets an answer. No alert fires, because from the system’s perspective retrieval worked perfectly.

Three ways out, in increasing order of effort:

1. Only embed what everyone may see. Blunt, easy, and often correct for a first version. Exclude anything sensitive and be explicit that the assistant does not know about it.
2. Filter at retrieval time using metadata. Attach owner, team or region to every document, and pass a filter with each query derived from the caller’s identity. This is the practical answer for most teams.
3. Separate knowledge bases per audience. Heavier to operate, and the only option that gives you a hard boundary rather than a filter you have to remember to apply.

Whichever you pick, decide it before ingestion, because metadata has to be configured at ingest time. Retrofitting a filter you did not plan for means rebuilding the index.

Metadata is most of the value

Metadata filtering narrows the candidate set before similarity search runs. That improves relevance, reduces tokens in the prompt, and gives you the access control lever above. For S3 sources it lives in a sidecar file named after the object it describes:

// acme-renewal.txt.metadata.json, alongside acme-renewal.txt in S3
{
  "metadataAttributes": {
    "object_type": "Opportunity",
    "owner_id": "0051t00000XYZ",
    "account_id": "0011t00000ABC",
    "region": "EMEA",
    "is_closed": false,
    "last_modified_epoch": 1750000000
  }
}

Include, at minimum: what kind of record it is, who owns it, which account it belongs to, whether it is still open, and when it last changed. Those five turn “search everything” into “search the open opportunities this person owns, changed recently”, which is a different product.

Freshness, and the vectors that outlive the record

An embedding is a copy. When the source record changes, the copy is stale. When the source record is deleted, the copy is not.

The consequences are worse in a CRM than in a document store, because CRM records change constantly and confidently answering with last quarter’s stage is indistinguishable from answering correctly. An assistant that cites a deal which closed-lost in March, or a contact who left the company, is not obviously wrong to the person reading it.

• Sync on a schedule that matches how the data is used. Daily is fine for account summaries and useless for live pipeline questions, which should be going to SQL anyway.
• Handle deletes explicitly. Whatever removes a record from your lake must also remove its document from the source bucket, or the vector survives.
• Put the record’s last-modified timestamp in the metadata and surface it in the answer. “As of three days ago” is a small change that prevents a large class of misunderstanding.
• Reconcile document count against source record count periodically. Divergence means orphans.

What not to embed

CRM free-text fields are where people put things. Notes fields contain personal details, occasionally payment information, opinions about customers that were never meant to leave a private conversation, and years of accumulated noise.

Once embedded, all of it is retrievable by anyone with query access, and a vector is not something you can easily grep for a mistake.

• Run detection over free-text fields before ingestion and redact rather than hope.
• Exclude fields nobody asks questions about. Every extra field is noise competing for retrieval slots.
• Consider excluding records closed long ago; they dilute results and rarely answer anything.
• Add Bedrock Guardrails on the output side as a second layer, not as your only control.
• Keep the whole thing on private networking with VPC endpoints if the data warrants it.

There is a governance question here too, and it is worth raising before someone else does: your customers’ personal data is now in a vector store feeding a language model. Whether your privacy notice covers that is not an engineering decision, but it is an engineering responsibility to ask.

Evaluate it before you show anyone

A demo proves the pipeline runs. It does not tell you whether the answers are right, and this is a system whose failure mode is being wrong fluently.

Write thirty real questions from the people who will use it, with correct answers you have verified by hand. Include the analytical ones specifically, because those are where retrieval fails silently. Then run the set after every meaningful change to chunking, metadata or the document format, and compare.

Use the Retrieve API rather than RetrieveAndGenerate while you are tuning. Seeing which chunks came back tells you whether a bad answer is a retrieval problem or a generation problem, and those have completely different fixes.

Common mistakes

• Embedding structured records and expecting aggregation to work.
• Exporting to CSV and letting the chunker split rows arbitrarily.
• Field-value dumps instead of prose, so embeddings have little language to work with.
• No metadata, so every query searches everything and access control is impossible.
• Assuming Salesforce sharing rules follow the data into the vector store.
• Deciding on filtering after ingestion, then discovering it has to be configured at ingest.
• Syncing changes but never removing documents for deleted records.
• Embedding every field because it was easier than choosing.
• No PII detection over free-text notes.
• Tuning chunk size before fixing document construction.
• Judging the system on a demo instead of a scored question set.
• Granting the execution role a whole schema rather than specific views.

Best practices

• Route analytical questions to structured retrieval and narrative questions to the vector store.
• One record per document, with context denormalised in, written as prose.
• Metadata on every document: type, owner, account, open state, last modified.
• Decide the access model before ingestion, and filter at retrieval time by the caller’s identity.
• Redact free-text fields before embedding, and exclude fields nobody queries.
• Delete documents when source records are deleted, and reconcile the counts.
• Surface record age in answers so staleness is visible rather than implied.
• Expose curated views to the execution role, never whole schemas.
• Meaningful table and column names, because the model reads your schema to write SQL.
• A scored evaluation set, run on every change, including questions you expect it to fail.

FAQ

Why does it get counts wrong?

Because retrieval returns the most similar chunks, not all matching records, and the model counts what it was handed. Counting is a query, not a similarity search. Send those questions to structured retrieval over your warehouse instead.

Should I use the Salesforce connector or export to S3 first?

The connector is quicker to stand up. Going via S3 gives you control over document construction, metadata and redaction, which is where most of the quality lives. Start with the connector to learn what people ask, then move to S3 once you know what the documents should look like.

Which chunking strategy is best?

Usually the default, honestly. If each document is one record, chunking has less work to do. Change it in response to evaluation results rather than in advance, because chunking is the most over-discussed and least decisive variable in a CRM RAG setup.

How do I stop people seeing records they shouldn’t?

Metadata filters applied per query based on the caller’s identity, separate knowledge bases per audience, or simply not embedding sensitive records. There is no automatic inheritance of CRM sharing rules, and assuming otherwise is the most consequential mistake in this post.

How often should I re-sync?

Match the questions. Account histories and notes tolerate a daily sync comfortably. Anything needing current numbers should not be coming from embeddings at all. Whatever the interval, make sure deletions propagate.

Do I need a vector database at all?

Only for the narrative half. If every question your users ask is analytical, structured retrieval over Redshift answers all of them with no embedding pipeline, no vector store and no staleness. Plenty of CRM assistants would be better products if they had noticed that.

The one thing to remember

The model will answer either way. It will answer the narrative question well and the counting question badly, in the same tone, with the same citations, and nothing in the response distinguishes them.

So the preparation work is mostly triage: decide which questions are retrieval and which are queries, build documents that carry their own context, attach metadata before you ingest rather than after, and test with real questions whose answers you already know. Chunk size is the last thing to worry about and the first thing everybody argues about.

Building one of these?

The pipeline is usually the easy part; the decisions before it are where these projects succeed or quietly disappoint. Work I take on:

• Designing the split between semantic retrieval and structured querying, so counting questions get correct answers.
• Building the document preparation pipeline from CRM data: record-level documents, denormalised context, metadata, redaction.
• Access control design for knowledge bases, including per-caller metadata filtering and audience separation.
• Freshness and deletion handling so the index does not diverge from the source.
• Curated Redshift views and IAM grants for structured data retrieval.
• Evaluation harnesses with scored question sets, run on every change rather than once before launch.

Send me ten questions your users would actually ask, and I will tell you which of them retrieval can answer.

The post It Counted the Chunks: Preparing CRM Data for Amazon Bedrock appeared first on John Nessime.

Integrations very rarely fail on launch day. They fail in week six. That is roughly when the OAuth token first expires, when an admin adds a field nobody mentioned, when a second tool starts consuming the same API allowance, and when enough records have been deleted for the row counts to visibly diverge. None of those conditions exist in staging, which is why staging said yes.

So this is a Salesforce AWS integration checklist aimed at week six rather than at Tuesday. Every item has a pass test, because “we thought about that” is not the same as “we checked.”

1. Credentials and identity

• The integration has its own Salesforce user. Pass: the username is not a person’s, and nobody uses it interactively.
• That user is API-only and cannot log in through the UI. Pass: someone tried and was refused.
• Its profile grants only the objects and fields you actually read. Pass: you have the field-level permission list and can explain every entry.
• Credentials live in a secrets manager, not in environment variables or config files. Pass: the secret is retrievable by ARN and nothing in the repo contains it.
• You have rotated the credential at least once, in production, and the pipeline survived. Pass: there is a dated record of the rotation and no incident attached to it.
• You know what invalidates the refresh token. Pass: you can name the events that revoke it, including password resets and session policy changes on the integration user.

That last item is the classic week-four failure. A refresh token feels permanent right up until an admin does something routine to the user account it belongs to.

2. Network, versions and deadlines

• You know whether the compute needs to be in a VPC, and why. Pass: if it is, you can name the private resource it reaches.
• If it is in a VPC, S3 traffic goes through a gateway endpoint. Pass: the endpoint exists and the route table references it.
• The Salesforce API version in your endpoints is pinned and current. Pass: it is not a version on a retirement list.
• IAM permissions are scoped to the specific resources involved. Pass: no wildcard on S3 buckets or Secrets Manager paths.
• Login IP restrictions and connected app policies are configured deliberately. Pass: you know whether IP relaxation is on and made that choice on purpose.

The API version item has real dates attached, so it is worth stating plainly rather than leaving as a vague “keep current”. Salesforce retired versions 7.0 through 20.0 in Summer ’22 and versions 21.0 through 30.0 in Summer ’25, and has announced that 31.0 through 40.0 retire in June 2028, with the SOAP login() call retiring separately in June 2027. Retired versions return HTTP 410 on REST rather than degrading gracefully.

Usefully, Salesforce warns you first. Calls to legacy versions come back with a Warning header, so this is detectable today rather than on the morning it breaks:

# If this matches anything, you have a deadline you did not know about.
grep -i "Warning:" integration.log | sort -u

3. Correctness

• Deletes are handled, not just creates and updates. Pass: you can point at the code path and name the mechanism it uses.
• Merged records do not double-count. Pass: you have checked what happens to the losing record of a merge.
• Formula and computed fields are either recalculated downstream or documented as snapshots. Pass: there is a written list of which columns are derived.
• Re-running the same window produces the same result. Pass: you ran it twice and diffed the output.
• One real aggregate reconciles against Salesforce for a closed period. Pass: daily and monthly totals both match, not just daily.
• Timezone handling is explicit at the boundary. Pass: someone can say which timezone the date column is in without guessing.

If daily figures reconcile and monthly ones do not, that is a timezone problem at period boundaries, not missing data. Find it now, because finding it inside somebody’s board pack is a worse day.

4. Failure handling

• Retries are scoped to failures that deserve them. Pass: transient errors retry, data errors do not, and you can show which is which.
• Backoff is exponential with jitter. Pass: a fixed sleep does not appear anywhere in the retry path.
• A partial failure leaves recoverable state. Pass: the watermark advances only after data lands, not when the API call returns.
• You can replay an arbitrary historical window on demand. Pass: someone has done it, in production, with a command you could hand to a colleague.
• Rate limiting is treated as an expected condition. Pass: a 429 or a limit error produces a backoff, not an alert and a failed run.

5. Observability

• The pipeline emits a metric on successful completion. Pass: the metric exists and you can graph it.
• An alarm fires when that metric stops arriving. Pass: you disabled the schedule on purpose and the alarm went red.
• Data freshness is measured on the destination side. Pass: you can answer “how far behind are we right now” from a dashboard.
• Salesforce API allowance consumption is graphed. Pass: there is a percentage on a chart with a threshold line.
• Alerts describe symptoms and link to a runbook. Pass: the alarm description contains a URL somebody could follow at 3am.

The second item is the one people skip and it is the only one that matters. An alarm you have never seen fire is a belief, not a control.

6. Cost

• Every resource carries a cost allocation tag. Pass: Cost Explorer filtered to that tag returns a number.
• Log groups have a retention period. Pass: none of them say “Never expire”.
• The schedule matches how the output is actually used. Pass: somebody named the decision the data supports and the latency it tolerates.
• Average object size in the destination is sensible. Pass: you have looked, and it is not thousands of tiny files per partition.
• A budget alert exists on the pipeline’s tag. Pass: it has a threshold and a recipient who is not on holiday.

7. Operations and handover

• The integration has a named owner. Pass: a person, not a team inbox, and they know.
• A runbook exists covering the three most likely failures. Pass: someone who did not build it followed the runbook successfully.
• Infrastructure is defined as code. Pass: you could rebuild it in a fresh account from the repository.
• Schema drift produces a notification. Pass: adding a field in a sandbox generated an alert somewhere.
• Personal data has an access policy and a retention policy. Pass: both are written down and somebody outside the team has read them.
• You know what a Salesforce sandbox refresh does to it. Pass: you have been through one, or you know what will break when you do.

If you only do five

Nobody clears a thirty-item list before a deadline. These five catch most of what actually goes wrong:

1. Rotate the credential once before go-live. It proves the whole credential path works and it is the failure with the longest fuse.
2. Break it deliberately and confirm someone is told. Disable the schedule, watch the alarm, put it back.
3. Reconcile one real number against Salesforce for a closed period, daily and monthly.
4. Set log retention and one cost allocation tag. Five minutes, and it is the difference between a knowable bill and a mystery.
5. Write the runbook and name the owner. If it is nobody’s, it is nobody’s at 3am too.

FAQ

What actually breaks first in production?

Credentials, usually around the first expiry or the first time an admin touches the integration user. After that, schema drift and API allowance contention with a tool somebody else added.

How do I know if I’m on a retiring API version?

Look at the version string in your endpoint URLs, and search your logs for the Warning header Salesforce returns on legacy versions. Retired versions fail with an HTTP 410 rather than degrading, so there is no soft landing.

Is a staging environment enough to sign off?

No, because the conditions that break integrations barely exist there: real volume, real credential lifecycles, real deletions, and a shared API budget. Staging proves the code runs. It cannot prove the thing survives contact with an organisation.

How often should I revisit the list?

Quarterly for the alarm test and the reconciliation, annually for permissions, credentials and API versions. Put both in a calendar, because an integration that works is one nobody thinks about until it does not.

Does this apply to managed connectors too?

Most of it. You inherit retries and pagination from the vendor, but credentials, permissions, reconciliation, cost, alerting and ownership are all still yours. A managed connector removes code, not accountability.

The one thing to remember

“It works” is a statement about today. “It’s ready” is a claim about week six, and the only honest way to make it is to have tested the things that only happen later: rotate the credential, break the schedule, reconcile a number, follow the runbook.

If you cannot point at evidence for an item, it is not done. It is intended, which is a different thing, and the gap between the two is where you will spend your next quarter.

Want a second pair of eyes before go-live?

Readiness reviews are cheap compared with the incidents they prevent, and they go faster with someone who has seen the same six failures repeatedly. Work I take on:

• Pre-launch readiness review against this checklist, with evidence collected rather than assurances taken.
• Credential and permission audit: integration users, profiles, field-level security, secret storage and rotation.
• Reconciliation harness so row counts and aggregates are checked automatically rather than when someone complains.
• Alerting and runbooks, including a failure drill that proves the alerts reach a human.
• Legacy API version audits ahead of the announced retirement deadlines.
• Cost tagging, log retention and budget alerts so the first month’s bill is not a surprise.

Tell me which items on this list you have evidence for, and I will tell you which of the rest matter most for your setup.

The post It Works Isn’t Ready: A Production Checklist for Salesforce AWS Integrations appeared first on John Nessime.

Eighteen months later there are forty flows, several of them on five-minute schedules because someone wanted fresher dashboards, and AppFlow is a line item that gets mentioned in the monthly review. Nothing was done wrong. The spreadsheet modelled the situation at the moment of the decision and that situation changed.

This is what makes an honest AppFlow vs Lambda vs MuleSoft cost comparison hard: you are not comparing three prices, you are comparing three different shapes, and the cheapest one at two integrations is frequently not the cheapest one at forty. The crossover is the entire decision, and almost nobody models it.

A note on numbers before we start. I am not quoting rate cards. AWS pricing varies by region and changes, MuleSoft publishes no list prices at all, and every comparison article with a neat table of dollar figures is quietly out of date. What does not change is the mechanism each one bills on, and that is what determines whether your bill grows with volume, with integration count, or not at all.

Three shapes

• AppFlow is pure consumption with no floor. You pay per flow run and per gigabyte processed. Two flows cost almost nothing; the bill scales with flows multiplied by frequency.
• Lambda is consumption too, but the meter is nearly irrelevant. The real cost is engineering time to build and own it, and the surrounding AWS services it drags along.
• MuleSoft is a floor. An annual contract sized on capacity or usage entitlements, paid whether you use it or not, largely independent of how many integrations you run inside it.

Draw those as lines against integration count and you get the whole argument. One starts near zero and climbs steeply. One starts moderate and climbs gently. One starts high and is close to flat. Where they cross depends entirely on your situation, which is why a generic answer is worthless and a framework is not.

AppFlow: cheap until the schedule tightens

AppFlow bills per successful flow run plus data processed. The detail that catches people, and it is stated plainly in AWS’s own pricing documentation, is this: a flow run that checks for new data and finds none still counts. You pay for polling, not for movement.

That turns schedule frequency into the dominant variable, and it is the one nobody models because at decision time everything is daily:

Flow runs per month, per flow:

  every 5 minutes   12 x 24 x 30  =  8,640
  every 15 minutes   4 x 24 x 30  =  2,880
  hourly                 24 x 30  =    720
  daily                       30  =     30

Twenty flows, five-minute schedule:  172,800 runs
The same twenty flows, hourly:        14,400 runs

Identical data. Twelve times the flow run charge.

Two more things that are not on the AppFlow line item. Standard S3 request and storage charges apply to whatever it writes, and KMS charges apply to the keys it uses. And on some connectors, running additional concurrent processes to speed a transfer is billed as additional flows.

Where AppFlow wins: a small number of standard SaaS-to-AWS movements on relaxed schedules, where nobody wants to own code. It is genuinely excellent at that, it goes live in an afternoon, and there is no floor to justify.

Where it stops winning: many flows, tight schedules, or transformation logic it cannot express. Schema drift handling is basic, and there is a point where you are working around the tool more than using it.

Lambda: the invoice is not the cost

For CRM-sized workloads, Lambda’s compute charge is close to a rounding error. A function that runs hourly for thirty seconds costs so little that optimising it is a waste of an afternoon.

The costs are elsewhere, and two of them are large.

The surrounding AWS services. NAT gateway data processing if the function sits in a VPC, CloudWatch Logs ingestion, S3 requests from small-file writes, Athena scans downstream. On a real pipeline these routinely exceed the compute charge several times over, and none of them appear under “Lambda” in Cost Explorer.

Engineering time, forever. Not just the build. Pagination, retries, watermarks, delete handling, schema drift, credential rotation, alerting, and someone available when it breaks at an awkward hour. This is the line that gets costed at zero in every comparison and is usually the largest number in the exercise.

The interesting property, though, is that the marginal cost of integration number eleven is much lower than integration number one, if you built a framework rather than eleven scripts. Shared extraction, shared state, shared error handling, one deployment pipeline. That is what makes the Lambda line climb gently instead of steeply. Teams that build eleven bespoke functions get the worst of both worlds: consumption pricing with none of the amortisation.

Where Lambda wins: you already have AWS engineers, you need control over the output shape, and the integration count is going up. Also anywhere the requirement is genuinely unusual, because a managed connector cannot express what it does not have a checkbox for.

Where it does not: a team of two who would rather ship product, or an organisation where the integration needs to be maintainable by people who are not engineers.

MuleSoft: a floor, and what it buys

MuleSoft publishes no list prices. Everything is quote-based, annual, and negotiated, which means any figure you find online is somebody else’s contract and tells you little about yours. It is also worth knowing that the commercial model has been repackaged relatively recently, moving new customers away from the older capacity-based structure toward usage entitlements, so a pricing article from a couple of years ago may describe a product you cannot buy.

Three structural things matter more than the number.

It is a commitment, not a meter. You size it up front and pay for it regardless of use, which is bad if you are unsure and fine if you are certain. Undersizing means overages; oversizing means paying for headroom.

Separate meters do not offset each other. Integration capacity, API traffic and add-on modules are metered independently, so spare room in one does not cover a spike in another. Model each axis separately or the renewal will be a surprise.

Renewal escalation is normal. Enterprise contracts typically carry annual uplift clauses, so the three-year total is meaningfully above three times year one. Negotiate that at signature, because the leverage is gone afterwards.

Now the fair part, because a post that treats MuleSoft as merely expensive is not being honest. What the floor actually buys: API management and governance as a first-class capability rather than something you assemble; a connector catalogue covering legacy and on-premise systems that neither AWS option touches; integrations that a trained analyst can maintain without a software team; audit trails, support contracts and SLAs that satisfy people who ask for them in writing. At a hundred integrations across several business units, the fixed cost is spread thin and the governance is the point.

The cost that is easy to miss is skills. DataWeave and the Anypoint tooling are proprietary, so you are hiring for a specific platform and your switching cost rises with every integration you build. That is not an argument against it; it is an argument for deciding deliberately rather than drifting in.

Where MuleSoft wins: many integrations, many teams, on-premise or legacy systems in the mix, and governance requirements with a compliance function attached to them.

Where it does not: three integrations, one team, all cloud. The floor will never amortise and you will spend the first year justifying it.

Modelling the crossover

Build the comparison over three years rather than one month, and project the inputs rather than freezing them.

1. Count integrations at year three, not today. Ask the people requesting them. The answer is always higher than the current number and it is the variable the whole model turns on.
2. Write down the schedule for each one, and defend it. “Every five minutes” is usually a preference, not a requirement. For AppFlow this is the difference between a small bill and a large one.
3. For AppFlow, multiply flows by runs per month. Apply current rates yourself, then add S3 requests and KMS.
4. For Lambda, cost the engineering explicitly. Build days plus ongoing maintenance days per year, at a loaded rate. If that number embarrasses you, it is probably the accurate one.
5. For MuleSoft, get an actual quote, model each meter separately, and include renewal escalation across the term.
6. Add the cost of being wrong. How much does switching cost in each direction? Consumption services you can walk away from. An annual contract and a proprietary skill set you cannot.

Then plot all three. You are looking for where the lines cross and whether your projected integration count sits comfortably on one side of it or uncomfortably near it. Near a crossover, pick the option that is cheaper to reverse.

Common mistakes

• Comparing one month at today’s integration count.
• Costing engineering time at zero on the build-it-yourself option.
• Not modelling flow run frequency, which is the single largest AppFlow variable.
• Forgetting that AppFlow charges for polling runs that find nothing.
• Attributing only the Lambda invoice to the Lambda option, ignoring NAT, logs and S3 requests.
• Building eleven bespoke Lambdas instead of one framework, so nothing amortises.
• Using published MuleSoft prices from an article, when no list prices exist and the packaging has changed.
• Ignoring renewal escalation when comparing an annual contract against consumption pricing.
• Treating a hybrid as a failure of nerve rather than the usual right answer.
• Choosing for the architecture you want rather than the team you have.

How I’d decide

Start with AppFlow if the integrations are standard, few, and not urgent. It is the lowest-commitment option and you can leave whenever you like.

Move to Lambda when you have AWS engineers, the count is climbing, and you need control over the output. Build it once as a framework, not repeatedly as scripts, or you get the cost profile without the benefit.

Consider MuleSoft when the problem is organisational rather than technical: many teams, legacy systems, governance requirements, and integrations that need to outlive the person who built them.

And be relaxed about mixing them. Plenty of sensible estates run AppFlow for the boring standard movements and custom code for the two integrations that are genuinely specific. Insisting on one tool for everything is a preference, not an architecture.

FAQ

Which is cheapest overall?

None of them, universally. AppFlow is cheapest at low integration counts and relaxed schedules. Lambda is cheapest at scale if you already have the engineers and build a shared framework. MuleSoft is cheapest only when the fixed cost is spread across many integrations and teams. The question is which shape fits your trajectory.

Why did my AppFlow bill grow when data volume didn’t?

Because you are billed per flow run, including runs that check for new data and find none. Somebody tightened a schedule, or the flow count grew. Both multiply the run charge without moving a single extra record.

Is Lambda really cheaper than AppFlow?

On the invoice, usually yes. In total cost, only once the engineering amortises across several integrations. For one or two simple flows, AppFlow is almost always the better economic answer, and the “we’ll just write a Lambda” instinct is a false economy at that scale.

Why can’t I find MuleSoft’s prices?

Because there aren’t any published. It is sold through quotes with negotiated discounts, so figures you find online are other people’s contracts. Get a quote, model each meter separately, and negotiate the renewal escalation before you sign rather than after.

Can I start on one and move later?

Between the two consumption options, fairly easily, especially if your destination schema is defined by you rather than by the tool. Off an annual contract with proprietary tooling, much less easily, which is exactly why the switching cost belongs in the model rather than as a footnote.

The one thing to remember

You are not choosing the cheapest tool. You are choosing a cost shape, and committing to how it behaves as your integration count and your schedules change over the next three years.

So model the trajectory, not the snapshot. Put engineering time in the spreadsheet at a real rate. And when the lines are close together, take the option you can walk away from, because being wrong cheaply is worth more than being right narrowly.

Need the model built for your numbers?

This decision usually gets made under time pressure with incomplete inputs, and then lived with for years. Work I take on:

• Building the three-year cost model for your actual integration inventory, schedules and volumes, with engineering time costed honestly.
• Auditing an existing AppFlow estate for flow count, schedule frequency and the charges that sit outside the AppFlow line item.
• Designing a shared extraction framework so custom integrations amortise instead of multiplying.
• Migrating between approaches without downtime, including keeping the destination schema stable so the choice stays reversible.
• Reviewing an integration platform contract for meter structure, overage terms and renewal escalation before you sign.
• Hybrid architectures where managed connectors handle the standard movements and code handles the awkward ones.

Send me your integration list with schedules and rough volumes, and I will tell you which shape you are actually buying.

The post Cheapest at Two Flows, Worst at Forty: AppFlow vs Lambda vs MuleSoft on Cost appeared first on John Nessime.

Then you check the invocation count and it is also zero, and has been since the day somebody disabled an EventBridge rule while cleaning up a different stack. The job has not run in three weeks. It never errored because it never started.

This is the shape of almost every integration monitoring failure: a healthy graph and a stopped job produce identical output. Zero errors is what success looks like and it is also what absence looks like, and if the only thing you measure is failure, the two are indistinguishable.

This is about how to monitor Salesforce integrations so that stopping is as loud as breaking. Four signals worth emitting, how to get them into CloudWatch cheaply, the alarm configuration that actually fires, and what belongs on a Grafana dashboard once you have them.

Four signals, not one

Most integration monitoring stops at errors and duration, because those come free from Lambda or your container platform. Both are worth having and neither answers the question anybody actually asks, which is “is the data right”.

The four that do:

• Liveness. Did it run at all?
• Volume. Did it move a plausible amount of data?
• Freshness. How old is the newest record on the destination side?
• Budget. How much of Salesforce’s daily API allowance have you spent?

Errors are a fifth, and the least interesting, because errors are the failure mode that already announces itself.

Liveness: the alarm that has to fire on silence

Emit a metric on every successful completion. A single count, value 1. Then alarm when it stops arriving.

The trap is in the CloudWatch defaults. TreatMissingData has four settings, and the default is missing, which sends the alarm to INSUFFICIENT_DATA when nothing arrives. That state is not ALARM. Nothing pages. Your dashboard shows a grey alarm that most people read as “fine”.

So heartbeat alarms need breaching. That much is standard advice. Here is the part that is not: even with breaching set, a heartbeat alarm can still fail to fire. CloudWatch evaluates over a range wider than your evaluation periods, and if it finds any real data point in that wider range, those override the missing ones. On a job that runs hourly, a successful run from earlier can keep the alarm quiet through several missed runs.

The robust version uses metric math to turn absence into a real zero, so there is no missing data to interpret:

aws cloudwatch put-metric-alarm 
  --alarm-name "opportunity-sync-not-running" 
  --alarm-description "No completed run in the last 90 minutes" 
  --comparison-operator LessThanThreshold 
  --threshold 1 
  --evaluation-periods 1 
  --treat-missing-data breaching 
  --alarm-actions "$SNS_TOPIC_ARN" 
  --metrics '[
    {
      "Id": "runs",
      "MetricStat": {
        "Metric": {
          "Namespace": "SalesforceSync",
          "MetricName": "RunCompleted",
          "Dimensions": [{"Name": "Integration", "Value": "opportunity-sync"}]
        },
        "Period": 5400,
        "Stat": "Sum"
      },
      "ReturnData": false
    },
    {
      "Id": "filled",
      "Expression": "FILL(runs, 0)",
      "ReturnData": true
    }
  ]'

FILL(runs, 0) substitutes a zero wherever the metric has no data point, so the alarm always has something real to compare against the threshold. The window is deliberately longer than the schedule: an hourly job gets ninety minutes, so one late run does not wake anybody.

Then do the thing everyone skips: disable the schedule in a test account and confirm the alarm actually goes red. An untested alarm is a belief, not a control.

Volume and freshness: is the data actually moving

A job can complete successfully and process nothing. A credential with the wrong field-level permissions, a filter that silently matches nothing, a watermark that got written before the data landed: all of these produce a clean run and an empty result.

Volume is the count of records read and written per run. Alarming on it is harder than liveness because the right number varies: a quiet Sunday legitimately looks like a broken Tuesday. Two approaches that work. Use CloudWatch anomaly detection, which learns the daily and weekly shape and alarms on departures from it. Or set a crude floor that only catches the catastrophic case, which is usually zero, and accept that you will not catch a fifty percent drop.

I would start with the crude floor. It is five minutes of work and catches the failure that actually happens.

Freshness is the better metric and almost nobody emits it. At the end of each run, query the destination for the newest record’s modified timestamp, subtract it from now, and publish the difference in seconds. That single number answers the business question directly: how far behind Salesforce are we right now?

It also collapses several failure modes into one signal. A stopped job, a job that runs but writes nothing, a job stuck retrying, a job silently filtered down to zero rows: all of them show up as lag climbing. If you only add one metric from this post, add this one.

Budget: watch the Salesforce allowance

Your org has a daily API allowance shared across every integration touching it. Exceeding it does not just break your sync; it breaks marketing automation, support tooling, and whatever else somebody connected two years ago.

Enforcement is initially soft, and then it is not: past a protection threshold, calls come back as 403 with REQUEST_LIMIT_EXCEEDED until the rolling window drains. By that point you are in an incident that spans several teams.

The good news is that this costs nothing to observe. Salesforce returns your current consumption on ordinary REST responses in a header, so you get it on calls you were making anyway:

# Sforce-Limit-Info: api-usage=1212/15000
#
# Free: no extra API call, which matters when the thing you are
# measuring is an API budget. Add real error handling before
# shipping this; the header is not guaranteed on every response.
raw = response.headers.get("Sforce-Limit-Info", "")
used, allowed = (int(v) for v in raw.split("api-usage=")[1].split("/"))
emit("ApiUsagePercent", used / allowed * 100)

For a fuller picture, the /services/data/vXX.X/limits endpoint returns every allocation in the org, including DailyApiRequests with its max and remaining values. It needs the View Setup and Configuration permission and the numbers lag by a few minutes. Poll it on a schedule rather than per request, and alarm on percentage consumed rather than absolute calls, so the alarm survives a licence change.

Graph consumption by integration if you can attribute it. The conversation about which team is burning the allowance goes very differently when there is a chart.

Getting the metrics in without a bill shock

You can call PutMetricData directly, and it works, and it is a synchronous API call in the hot path of your job that can fail or add latency. Custom metrics are also charged per metric per month, and a metric is every unique combination of name and dimensions, so a dimension with high cardinality gets expensive quietly.

The better default is Embedded Metric Format: write structured JSON to stdout and CloudWatch extracts the metrics from your logs. No API call, no added latency, and the log line stays queryable in Logs Insights alongside the metric.

{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [{
      "Namespace": "SalesforceSync",
      "Dimensions": [["Integration"]],
      "Metrics": [
        { "Name": "RunCompleted",     "Unit": "Count"   },
        { "Name": "RecordsWritten",   "Unit": "Count"   },
        { "Name": "SourceLagSeconds", "Unit": "Seconds" },
        { "Name": "ApiUsagePercent",  "Unit": "Percent" }
      ]
    }]
  },
  "Integration": "opportunity-sync",
  "RunId": "a41c9f",
  "RunCompleted": 1,
  "RecordsWritten": 4127,
  "SourceLagSeconds": 312,
  "ApiUsagePercent": 8.1
}

Note what is a dimension and what is not. Integration is a dimension because it has a handful of values and you want to alarm per integration. RunId is a plain field: searchable in the logs, and not a dimension, because making it one would create a new metric on every run. That distinction is the whole cost story.

Grafana on top

CloudWatch dashboards are fine and Grafana is better for this, for three reasons: you can put Salesforce metrics next to your warehouse and application metrics on one screen, the alerting is more expressive, and non-engineers will actually open it.

Add CloudWatch as a data source using an IAM role rather than access keys, scoped to cloudwatch:GetMetricData, cloudwatch:ListMetrics and the Logs Insights permissions if you want log panels. One honest cost note: Grafana queries CloudWatch through the metric data API, which is billed per metric requested, so a busy dashboard on a short refresh interval is a real line item. Set a sane refresh, avoid auto-refresh on wall displays, and use the caching in Grafana’s CloudWatch data source.

What goes on the dashboard, in order down the page:

• Freshness per integration, as a stat panel with thresholds. This is the panel people look at.
• Time since last successful run, per integration.
• Records processed, over a window long enough to show the weekly shape.
• API allowance consumed, as a percentage with a threshold line.
• Errors and duration, at the bottom, where they belong.

One dashboard, one screen, no scrolling. A dashboard nobody can read at a glance during an incident is decoration.

Alerts people don’t ignore

Decide deliberately where alerting lives. CloudWatch alarms are more reliable, because they keep working when Grafana is down, and Grafana alerts are more flexible and can span data sources. My default is CloudWatch for the small number of alerts that page someone, and Grafana for everything informational.

Three things that separate a useful alert from noise. Alarm on the symptom, not the cause: “Opportunity data is more than two hours stale” is actionable in a way “Lambda errors greater than zero” is not. Put the runbook link in the alarm description, since that field ends up in the notification and is the only documentation anybody reads at midnight. And use composite alarms to suppress the cascade, so a Salesforce outage produces one page rather than nine.

Common mistakes

• Monitoring only errors, so a stopped job looks identical to a healthy one.
• Leaving TreatMissingData at its default on a heartbeat alarm.
• Setting it to breaching and assuming that is sufficient, without handling the evaluation range.
• Never testing that an alarm fires by actually breaking something.
• No freshness metric, so nobody can answer how far behind the data is.
• Ignoring API allowance until an integration you do not own breaks.
• High-cardinality dimensions such as record ID or run ID, and the bill that follows.
• Calling PutMetricData synchronously in the job’s critical path.
• Alerting on causes rather than on user-visible symptoms.
• A dashboard that requires scrolling and interpretation during an incident.

Best practices

• Emit liveness, volume, freshness and API budget from every integration, as a standard.
• Heartbeat alarms with breaching plus FILL(), and a window longer than the schedule.
• Freshness as the headline metric, because it maps to a question the business asks.
• Embedded Metric Format rather than direct API calls.
• Low-cardinality dimensions; everything else stays a log field.
• Alarm on percentage of the API allowance, not absolute calls.
• IAM roles for the Grafana data source, and a refresh interval you have costed.
• Runbook links in alarm descriptions.
• Composite alarms to collapse cascades into one page.
• A quarterly test that breaks each integration on purpose and confirms someone gets told.

FAQ

Why didn’t my CloudWatch alarm fire when the job stopped?

Almost certainly TreatMissingData. The default sends the alarm to INSUFFICIENT_DATA, which is not ALARM and pages nobody. Set it to breaching, and wrap the metric in FILL() so there is no missing data for CloudWatch to reinterpret.

What’s the single most useful metric to add?

Freshness: how old the newest record on the destination side is. It catches stopped jobs, empty runs, stuck retries and silent filtering with one number, and it is the only one of these metrics a non-engineer can interpret.

CloudWatch dashboards or Grafana?

Grafana if you already run it, because you can put Salesforce, warehouse and application metrics on one screen. CloudWatch if you do not, because a second system to operate is not free. Either way keep the paging alarms in CloudWatch so they survive Grafana being down.

Will custom metrics be expensive?

Only if you make them so. Cost scales with unique name-and-dimension combinations, so a handful of metrics dimensioned by integration name is negligible. Adding a run ID or record ID as a dimension is how the bill grows without anyone noticing.

How do I monitor a third-party connector I can’t add code to?

Monitor the destination instead. A scheduled job that queries the target for the newest record’s timestamp and emits it as a freshness metric works regardless of what wrote the data, and it is arguably a better test because it measures the outcome rather than the process.

The one thing to remember

Absence of failure is not evidence of success. An integration that stopped produces exactly the same error graph as one working perfectly, and every default in your monitoring stack is tuned to stay quiet when data stops arriving rather than to shout about it.

So measure the thing you actually care about. Not “did it error” but “how stale is the data right now”, alarmed in a way that fires on silence, and tested by deliberately breaking it. Everything else on the dashboard is supporting evidence.

Want this built properly?

Integration monitoring tends to get added after the first silent failure, which is one failure too late. Work I take on:

• Instrumenting Salesforce integrations with liveness, volume, freshness and API budget metrics via CloudWatch.
• Auditing existing alarms for the ones that cannot fire, and fixing the missing-data handling.
• Building the Grafana dashboard and data source, including cost-aware query and refresh configuration.
• Alert design: symptom-based alarms, composite alarms to suppress cascades, runbooks attached where people will read them.
• API allowance monitoring and attribution across multiple integrations sharing one org.
• Running a failure drill so you know the alerting works before you need it.

Tell me how you would currently find out that a sync stopped, and I will tell you how long it would take.

The post Zero Errors, Zero Records: Monitoring Salesforce Integrations with CloudWatch and Grafana appeared first on John Nessime.