You look at the CRM reporting tab. It cannot do that. So now you are picking a platform, and within about ten minutes someone will say “we should build a data lake,” and someone else will say “just put it in the warehouse,” and the deciding argument will be about how much data there is.

That argument is almost always the wrong one. This post covers data lake vs data warehouse specifically for CRM analytics: why volume rarely decides it, what actually does, a decision procedure you can run in an afternoon, and where zero-ETL has changed the maths recently enough that older advice is stale.

Start with the number, then stop using it

Go and count your rows. Not estimate. Count.

A mid-sized company’s entire Salesforce org is usually a few million rows across the objects anyone actually reports on: accounts, contacts, opportunities, activities, cases. Add HubSpot, Zendesk and a billing system and you are still typically in the low tens of gigabytes once it is written as compressed Parquet. Activity and event history is the one object that can genuinely run large, and it is worth checking separately.

Numbers at that scale do not stress either architecture. A data warehouse will not break a sweat. A lake will not break a sweat. The lake-versus-warehouse debate as it is usually framed, the one about petabytes and elasticity and separating storage from compute, is a debate about a scale problem you do not have.

So count the rows, confirm you are small, and then stop using volume as an argument. Everything below is about what decides it instead.

Data lake vs data warehouse: the three options, honestly

Warehouse only

Load CRM objects straight into Redshift, Snowflake, BigQuery or Postgres. Model with SQL. Point the BI tool at it.

Where it wins: it is the fastest path to an analyst being productive, and that is not a small thing. Query performance on joins is better than a lake gives you without tuning. Access control is mature and row-level. Every BI tool connects natively. The people who will maintain this already know SQL and probably do not know Spark.

Where it doesn’t: semi-structured data is awkward, and CRM payloads are full of it. Custom field bags, JSON blobs, email bodies, call transcripts. You can store JSON in a modern warehouse but you will not enjoy querying it at any depth. Storing many years of daily snapshots gets expensive on warehouse storage in a way it does not on object storage. And you are more locked in: moving between warehouses is a migration project, not a repoint.

Lake only

Land raw API responses to S3, catalogue them, query with Athena or Spark. Iceberg or Delta as the table format if you want anything resembling a database.

Where it wins: raw retention is nearly free, so keeping every version of every record for years is a storage line item rather than a decision. Schema evolution is genuinely easier: when a SaaS admin adds a field, the lake absorbs it. Unstructured content lives alongside structured tables, which matters more now that people want to run models over call notes and support tickets. And an open table format means the data is not hostage to one query engine.

Where it doesn’t: you have taken on work a warehouse does for you. Small-file compaction, partitioning, table maintenance, catalogue hygiene. Query latency on interactive dashboards is worse unless you invest in layout. Governance is a build, not a setting. And the per-query cost model punishes exactly the workload CRM analytics generates: lots of small, repeated, unpredictable queries from a BI tool. Athena bills on data scanned with a per-query minimum, which means a dashboard refreshing thirty tiles is thirty billable scans whether or not anyone is looking.

Lakehouse

Raw and historical data in object storage as Iceberg tables; curated marts in the warehouse or queried in place by a warehouse engine. This is where most serious setups end up, and it is what AWS, Databricks and Snowflake have all converged on.

Where it wins: you get cheap deep history and expensive fast queries in the right places. Raw payloads stay replayable when a vendor changes a field type. Analysts still get warehouse-shaped tables.

Where it doesn’t: it is two systems. Two cost models, two sets of permissions, two places a table can be defined and drift. For a five-person company with one CRM, this is architecture as procrastination. The complexity is real and it should be earning something.

What actually decides it

Four levers, in the order they usually matter for CRM data.

1. How much history you need, and whether deletes matter

This is the big one and it is the one people skip.

CRM systems mutate in place. An opportunity’s stage, amount and close date are overwritten as the deal moves. If you only ever sync current state, you can report on the pipeline as it is today and you cannot report on what it looked like last March. Every question about pipeline movement, forecast accuracy, stage velocity or “what changed since the QBR” needs point-in-time history, and the CRM will not give it to you retroactively. Deleted records are worse: they are simply gone.

If you need history, you need somewhere cheap to keep daily snapshots or a change feed. That pushes toward object storage, or toward a warehouse with a genuinely cheap storage tier. If a rolling twelve months of current-state data covers every question anyone asks, the warehouse handles it alone.

2. How fast the schema churns

Count the custom fields added to your main CRM object in the last six months. If it is two, schema churn is not your problem. If it is forty, and half of them are from a business unit you do not talk to, a rigid warehouse schema becomes a standing maintenance tax and the lake’s tolerance for drift starts paying for itself.

3. The shape of the queries

Scheduled dashboards hitting the same tables repeatedly favour a warehouse, because per-second compute amortises across many queries and results are cached. Redshift Serverless, for example, bills in processing-unit-hours per second with a short minimum and shuts down when idle, which suits bursty BI traffic.

Exploratory one-off analysis across wide, rarely-touched history favours a lake, because you only pay when someone asks. Most CRM analytics is overwhelmingly the first kind. That fact alone settles a lot of these arguments.

4. Who is going to maintain it

The least technical criterion and often the most decisive. If the answer is “a RevOps analyst who is very good at SQL and has no platform team behind them,” build the warehouse. A lake handed to someone who cannot maintain it becomes an unqueryable pile of Parquet in about eight months. That is not a hypothetical failure mode; it is the normal one.

A decision procedure you can run this afternoon

1. Count rows per object for everything anyone reports on. Note which objects are over about ten million rows. Usually it is activities and nothing else.
2. Write down the ten questions the business actually asks. Not the ones you imagine. Mark each one as needing current state or point-in-time history.
3. Count schema changes on your main objects over the last six months.
4. Estimate query volume: number of dashboards, tiles per dashboard, refresh frequency, plus rough ad-hoc queries per week. This is the number that drives cost, not storage.
5. Model both costs against those numbers. For the lake: bytes scanned per query times queries per month. For the warehouse: compute-hours per month plus storage. Do it on a napkin. At CRM scale the answer is usually obvious and usually favours the warehouse.
6. Name the maintainer. If you cannot name a person, pick the option that needs less of them.

Read together, most CRM-only shops land on: warehouse for the curated layer, cheap object storage for raw history, and no ambition beyond that until something forces it. That is a lakehouse, but the small honest kind, and it is a very different commitment from standing up a full lake platform.

Where zero-ETL changed the maths

A lot of writing on this topic predates a genuine shift, so it is worth stating plainly.

Managed zero-ETL integrations now replicate CRM and support platforms into AWS targets without you building a pipeline. AWS Glue supports SaaS sources including Salesforce, SAP, ServiceNow, Zendesk and Zoho CRM, landing into Amazon Redshift, the SageMaker lakehouse, or S3 Tables with Iceberg underneath. Salesforce has pushed the other direction with zero-copy federation, letting Data Cloud read Iceberg tables in Databricks, Snowflake, BigQuery or Redshift in place, and letting those platforms read Data Cloud objects back.

Two consequences for this decision:

• Ingestion is no longer the differentiator. “The lake is easier to load into” was a real argument when you were hand-rolling connectors. It is much weaker when the same managed integration targets either destination.
• Iceberg makes the choice less permanent. If your raw layer is Iceberg tables in object storage, adding a warehouse engine later is a configuration change rather than a migration. That is a genuine reason to prefer open table formats even if you never build a “lake” as such.

Two honest caveats. Zero-ETL integrations cover a curated list of sources and regions, so check yours before designing around them. And a managed replica is still current-state replication plus change capture, not a modelled history; you still have to decide how snapshots and slowly changing dimensions get built.

Arguments that don’t survive contact

• “We need a lake because we’ll do AI later.” Reasonable in spirit, wrong in sequence. Build the thing that answers this quarter’s questions, keep raw payloads in object storage so you have not destroyed your options, and revisit. Raw retention is the cheap insurance; a lake platform is not.
• “Storage is cheaper in a lake.” True and almost always irrelevant. At CRM volumes, storage is a rounding error next to compute and, more to the point, next to the salary of whoever maintains the thing.
• “The warehouse is a single point of vendor lock-in.” Fair, and it is why open table formats matter. But if you replicate raw data to object storage, the lock-in is on your transformation logic, not your data, and that is portable if you keep it in SQL and version control.
• “We’ll just query the CRM API directly and skip both.” Works until a dashboard fans out into rate limits and the whole thing becomes a data pipeline you did not design. If more than one person is asking questions, you need a copy.
• “Let’s do both properly from day one.” This is the expensive mistake. Two systems, two cost models, and tables that quietly disagree, in service of a workload that fits comfortably in one.

Frequently asked questions

Is a data lake overkill for one CRM?

For a single CRM with current-state reporting needs, usually yes. The point at which it stops being overkill is when you need years of point-in-time history, when unstructured content like call transcripts becomes part of the analysis, or when you are integrating enough systems that a common raw layer is cheaper than a dozen warehouse loaders.

How do I get historical CRM snapshots if I only have current state?

You mostly cannot get it retroactively, which is why this decision is urgent rather than academic. Start capturing now: either daily full snapshots to partitioned object storage, or a change feed if the platform offers one, then build slowly changing dimension tables from that. Field history tracking inside the CRM covers a limited number of fields and is not a substitute.

Does the lakehouse make this comparison obsolete?

It blurs the storage layer, not the operational reality. You still choose a query engine, a cost model, a governance model and a maintainer. The useful question has shifted from “lake or warehouse” to “how much of this do I want to operate myself,” which is a better question anyway.

Where does reverse ETL fit?

If you plan to push computed values back into the CRM, such as a lead score or account health field, that pushes toward the warehouse. Reverse ETL tools connect to warehouses far more readily than to query engines over object storage, and the write path wants a stable, modelled table rather than a raw one.

Can I start with one and move later?

Moving from lake to warehouse is straightforward: you are loading modelled tables into a new engine. Moving from warehouse to lake is harder, because the history only exists inside the warehouse and the transformations are written in its dialect. The asymmetric fix is to land raw data in object storage regardless of which you query from. That single habit preserves the option cheaply.

What about just using the CRM’s own analytics product?

Worth pricing seriously before building anything. It is genuinely the right answer when all your questions are about one system, and it removes an entire pipeline from your life. It stops being the right answer the moment a question spans CRM plus billing plus product usage, because you will be paying to ingest the other systems into a platform priced for CRM data.

The short version

In the data lake vs data warehouse question for CRM analytics, volume is the argument everyone reaches for and it is the one that matters least. Your CRM data is small. Both options handle it.

What decides it is how much history you need, how fast the schema moves, what shape the queries are, and who is going to keep it running. Answer those four and the architecture falls out on its own. And whatever you pick, land the raw payloads in object storage from day one, because that is the cheap decision that keeps every other decision reversible.

Need a second opinion on your CRM analytics stack?

I help teams make this call without over-building, and unpick it when it has already been over-built. Things I can help with:

• Running the sizing and cost model above against your real query volume, not a vendor calculator.
• Designing point-in-time history capture for CRM objects, including deletes and slowly changing dimensions.
• Setting up zero-ETL or connector-based ingestion from Salesforce, HubSpot, Zendesk or Zoho into Redshift, Snowflake or an Iceberg lake.
• Migrating an unmaintainable pile of Parquet into something an analyst can actually query.
• Cutting query costs on an existing setup: partitioning, table layout, dashboard refresh patterns.
• Building the curated CRM models that BI tools sit on, in SQL and in version control.

If you are mid-argument about this, send me your row counts and the ten questions the business wants answered. That is usually enough to settle it.

The post Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question appeared first on John Nessime.

A Salesforce admin renames a custom field on Tuesday afternoon. Your connector keeps running, your Glue job keeps succeeding, the crawler happily adds a new column and quietly stops populating the old one. Three weeks later someone asks why the pipeline report has a cliff in it, and you spend an afternoon reconstructing a change nobody in engineering made and nobody logged.

This post is about using AWS Glue Data Quality to catch that class of breakage. It is aimed at people pulling from SaaS sources into S3 or a lakehouse: Salesforce, HubSpot, Zendesk, Stripe, Shopify, Zoho, an internal partner API. I will cover where to run the checks, the flattening constraint that will bite you before anything else does, which DQDL rule actually catches which SaaS failure, when dynamic rules and anomaly detection quietly do nothing, and what should happen when a rule fails.

Why SaaS data breaks in ways your own database doesn’t

When you own the source database, the schema is under change control: someone writes a migration, it gets reviewed, and if it breaks a consumer there is a commit to point at. A SaaS source has none of that. Three consequences:

• The schema is edited by people who are not engineers. A sales ops admin adding a picklist value is a production schema change with no review, no notice and no version number.
• The payload is nested. SaaS APIs return structs and arrays: line items, related objects, custom field bags. That shape matters more than you would expect, for reasons in the next section but one.
• Absence is ambiguous. A field missing from a payload might mean “not set,” “not permitted for this user,” or “we removed it.” The connector usually cannot tell you which, and neither can your table.

Glue Data Quality is a rules engine built on Deequ that evaluates declarative checks written in DQDL, the Data Quality Definition Language, against a dataset. It is well suited to the first and third problems. The second one needs work before it is useful at all.

Decide where the checks run before you write a rule

There are two places to evaluate a ruleset, and they answer different questions.

Inside the Glue ETL job

The EvaluateDataQuality transform runs mid-pipeline against a DynamicFrame. It answers “is this batch safe to load?” and it can act on the answer: stop the job, or split failing rows into a quarantine prefix. This is also the only placement where row-level results exist, which matters because “which 400 rows are bad” is a far more useful answer than “the rule failed.”

Against a Data Catalog table

You can attach a ruleset to a catalog table and run it on a schedule, independent of any job. This answers “is the table currently trustworthy?” and surfaces a quality score analysts can see before they query. The trade-off is that it is after the fact: by the time a catalog evaluation fails, the bad batch is already queryable. Analyzers are also not supported for catalog evaluations, so some dynamic-rule tooling behaves differently there.

In practice I run both. The in-job ruleset is small and strict, and exists to stop garbage landing. The catalog ruleset is broader and looser, and exists so a table’s state is visible to people who will never open Glue Studio.

The flattening problem that comes first

Here is the constraint that catches most people on SaaS data, stated plainly in the AWS documentation and easy to skim past: DQDL cannot evaluate nested or list-type columns. No structs, no arrays.

Almost every interesting SaaS payload is nested. A Salesforce opportunity carries a related account struct, a Stripe invoice carries an array of line items, a Zendesk ticket carries a custom fields array of key-value pairs. Point a ruleset at the raw landed data and the fields you most want to check are the ones the engine cannot see.

So you flatten first, with Glue’s relationalize or an explicit projection, then evaluate. That works, but it changes what your rules mean:

• Your rules now describe the projection, not the payload. If your flattening step drops a field, no rule will notice, because from DQDL’s point of view that field was never there.
• Exploding an array changes the grain. One invoice with five line items becomes five rows. Every RowCount rule you write against that table is now counting line items, and a customer who starts itemising differently will move your row count without any data being wrong.
• Custom-field bags need pivoting, not flattening. A key-value array flattened naively gives you rows, not columns, and you cannot write a ColumnValues rule against a value whose meaning depends on a sibling key column. Pivot the ones you care about into real columns and check those.

The practical rule: put your quality checks immediately after the flattening step, and treat the flattening step itself as code that needs its own test. Glue Data Quality will not cover it for you.

Matching SaaS breakage to the rule that catches it

DQDL has a long rule list and most guides just enumerate it. More useful is the other direction: what actually goes wrong with SaaS sources, and what catches each one.

A field is renamed, removed, or added

This is the Tuesday-afternoon scenario. Two rules, doing two different jobs:

Rules = [
    ColumnExists "annual_revenue__c",
    ColumnCount = 47
]

ColumnExists protects the specific fields your downstream models depend on. ColumnCount protects against everything else: a field silently added, or one you forgot you needed. The first is precise and the second is a tripwire, and you want both because a rename shows up as one column gone and one column arrived, which leaves the count unchanged.

SchemaMatch compares against a reference dataset’s schema, which is stricter and useful if you maintain a golden reference table. Bear in mind a hard column count on a SaaS object fails every time an admin adds a field, so decide whether that blocks or just warns.

A field still exists but stopped being populated

Permissions change, a workflow gets disabled, an integration user loses field-level access. The column is still in the schema and is now all nulls.

Rules = [
    IsComplete "opportunity_id",
    Completeness "lead_source" > 0.85,
    Completeness "email" > 0.60 where "record_type = 'Contact'"
]

IsComplete is the absolute version: no nulls at all, right for keys. Completeness takes a threshold, which is what you want for optional business fields where a hard rule would fail constantly. The where clause is the one people miss: SaaS objects are usually polymorphic, and a completeness threshold that makes sense for contacts is nonsense across the whole table. Scope the rule to the record type.

The sync ran short

A paginated extract stops early, a rate limit gets absorbed by retries, an incremental cursor drifts. The job succeeds and delivers less data than it should have. A static RowCount > 1000 stops catching this the moment you cross a thousand rows. Dynamic rules compare against your own history instead:

Rules = [
    RowCount > min(last(3)),
    Sum "amount" > avg(last(5)) * 0.7,
    DetectAnomalies "RowCount"
]

Analyzers = [
    RowCount,
    DistinctValuesCount "owner_id"
]

last(k) pulls the last k recorded values of that metric for the same dataset; with k greater than one you need an aggregation like avg, min or max to reduce them to one number. DetectAnomalies goes further, using forecasting over stored history to flag deviations with no threshold from you at all. That is the right tool for anything seasonal: a B2B pipeline that dips every weekend trips a naive trailing-average rule every Monday, while anomaly detection learns the shape.

Analyzers deserve their own mention: they gather a metric without asserting anything, which builds the history dynamic rules and anomaly detection need. DistinctValuesCount on an owner or account column is a strong early signal for a partial sync, because row count can look fine while the number of distinct accounts represented quietly halves.

Records arrive twice, or the key stops being a key

Replays, overlapping incremental windows and connector retries all produce duplicates, and SaaS “IDs” are less unique than vendor docs imply once you have merged sandboxes or multiple accounts.

Rules = [
    IsPrimaryKey "id",
    Uniqueness "external_reference" > 0.99
]

IsPrimaryKey asserts unique and non-null together. Uniqueness with a threshold is the softer version for fields that are mostly-but-not-perfectly unique, which describes a lot of SaaS reference fields honestly.

Objects stop agreeing with each other

Multi-object extracts rot quietly. Opportunities reference accounts that were never synced; invoice totals stop matching line item sums because one object’s extract failed and the other’s didn’t.

ReferentialIntegrity checks that values in one dataset’s column exist in another’s. RowCountMatch and AggregateMatch compare counts and summary metrics across two datasets. These catch a broken join before an analyst does, and they need the multi-frame form of the transform with the second dataset supplied as an additional source.

The sync stopped and nobody noticed

The quietest failure of all: nothing is wrong with the data, there just isn’t any new data. DataFreshness checks a timestamp column against a window. It is deliberately simple and its comparison granularity is limited, so for anything more nuanced the custom SQL rule is the escape hatch, running a query against the dataset and evaluating the result against an expression. DQDL is case sensitive and that rule type’s exact capitalisation has appeared more than one way across the docs and the visual builder, so copy it from the current DQDL reference rather than from memory.

Static, dynamic and ML: when each one lies to you

All three check styles have a mode where they pass while the data is wrong. Knowing which is which is most of the skill.

• Static rules lie by going stale. A threshold that was correct when you wrote it stops being correct as the business grows, and a rule that can no longer fail is worse than no rule because it looks like coverage.
• Dynamic rules lie when they have no history. This one is genuinely nasty. last(k) reads previously recorded values for that metric, and if there aren’t any, it falls back to a default of zero. Add RowCount > avg(last(3)) to an existing job today and it will pass trivially for the first few runs, because almost any row count beats zero. It looks like a working check from the moment you deploy it. It isn’t one until it has accumulated runs.
• Anomaly detection lies while it is warming up, and after you retrain it badly. It needs a run of history before its forecasts mean anything, and if you leave a period of genuinely bad data in the training window, it learns that the bad shape is normal. You can exclude specific statistics from the calculation in the Data Quality tab, and you should, after any incident.

So when you add a dynamic or ML-based rule, write down the date and treat it as unarmed for the first couple of weeks. Keep a crude static rule alongside it in the meantime.

What should happen when a rule fails

The transform returns a collection. The two members you care about are the row-level outcomes, which give you the original data with an evaluation result column appended per row, and the rule outcomes, which give you one row per rule with its verdict.

from awsgluedq.transforms import EvaluateDataQuality

dq_results = EvaluateDataQuality().process_rows(
    frame=flattened_opportunities,
    ruleset=ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "salesforce_opportunity",
        "enableDataQualityCloudWatchMetrics": "true",
        "enableDataQualityResultsPublishing": "true",
    },
)

row_level = SelectFromCollection.apply(
    dfc=dq_results, key="rowLevelOutcomes"
)

dataQualityEvaluationContext is the namespace your CloudWatch metrics land under. Name it after the source object rather than the job, because you will eventually want to alert per object.

From there, three response patterns:

1. Fail fast. Stop the job on failure so nothing lands. Right for structural rules: if the primary key isn’t unique, downstream is going to be wrong no matter what you do next.
2. Quarantine. Filter the row-level outcomes on the evaluation result column, write passing rows to the target and failing rows to a separate prefix. Right for value-level rules where 2% bad records shouldn’t block 98% good ones. The important discipline is that somebody has to actually look at the quarantine prefix; an unmonitored quarantine is a delete with extra steps.
3. Warn and continue. Let everything land, publish the metrics, alert on the score. Right for new rules you don’t trust yet, and for the catalog-side ruleset.

Whichever you pick, get the failure out of Glue and into wherever your team actually looks. Results and anomaly predictions can be written back to catalog tables so they are queryable with plain SQL, and CloudWatch metrics plus EventBridge events give you the hook for paging or pausing a downstream pipeline. A Grafana or Datadog dashboard fed from those metrics beats a score in a console tab nobody opens.

Troubleshooting AWS Glue Data Quality on SaaS pipelines

“Input data does not include column X”

A rule references a column the engine cannot find. On SaaS data that means one of three things: the field disappeared from the source, your flattening step renamed it, or the Data Catalog is out of sync with the underlying table. Check the catalog schema against the landed files before touching the ruleset; rerunning the crawler resolves a surprising share of these.

A rule type isn’t recognised

DQDL is case sensitive and the rule set has grown over releases, so a rule copied from an older blog post can parse fine in your head and not at all in the engine. Check the rule name against the current DQDL rule type reference, and check that the feature is available on your Glue version. Anomaly detection in particular has a version floor.

A dynamic rule never fails

Almost always the empty-history problem described above. Look at the statistics for that metric in the Data Quality tab. If there are fewer recorded points than your last(k) asks for, the rule is not doing what you think. It will start working on its own; the mistake is trusting it in the meantime.

Evaluation is slow or the job runs out of memory

Each rule triggers analyzer computation over the dataset, and a large ruleset on a wide flattened SaaS table adds up fast. The transform exposes caching and observations-scope options to tune this. Before reaching for a bigger worker type, cut the ruleset down: most contain a dozen recommended rules nobody reads the results of.

A “column reference is ambiguous” error

Classic flattening artefact. Two nested paths collapsed to the same leaf name, so datechecked now matches two things. Fix it in the projection with explicit aliases rather than trying to disambiguate inside DQDL.

Common mistakes

• Accepting the recommended ruleset wholesale. Recommendations are generated from a sample of current data, so they encode today’s accidents as tomorrow’s requirements.
• Writing rules against nested data and assuming they cover the whole payload.
• Deploying a dynamic rule and treating it as armed on day one.
• Applying whole-table completeness thresholds to a polymorphic object instead of scoping with a where clause.
• Quarantining failed rows to a prefix nobody monitors.
• Writing RowCount rules against an exploded array without noticing the grain changed.
• Failing the job on every rule, including cosmetic ones, until the team starts rerunning with checks disabled.
• Keeping the ruleset in the console instead of in version control next to the job.

Best practices

• Split rules into blocking and reporting tiers, and be honest about which is which. A rule that always fails and never stops anything trains people to ignore the tab.
• Pair ColumnExists with ColumnCount. One protects the fields you named, the other catches the drift you did not anticipate.
• Add analyzers early, even for metrics you have no rule for yet. History has to accumulate before dynamic rules and anomaly detection are worth anything, and you cannot backfill it.
• Keep DQDL in Git alongside the job definition and deploy it with Terraform or CloudFormation. Rulesets are code and should move through the same review as the pipeline they guard.
• Namespace the evaluation context per source object so CloudWatch metrics and alerts can be routed per vendor.
• Exclude incident periods from the anomaly detection training window once the incident is resolved.
• Surface results where stakeholders already look. Writing outcomes back to catalog tables makes them queryable from Athena, which beats a score in a console tab.

Frequently asked questions

How much does AWS Glue Data Quality cost?

It bills on Glue DPU-hours like any other Glue workload, so cost is a function of evaluation time and DPUs used. The dimension that surprises people is anomaly detection, which consumes additional capacity per statistic analysed, so enabling it broadly across many columns and tables is where the bill moves. Model it against the current Glue pricing page; rates change and vary by region.

Can it check data before it lands in S3?

Yes, if the check runs inside the Glue job. The transform evaluates a DynamicFrame in memory, so you can validate after extraction and before the write. That is the placement that lets you stop a bad load rather than report on one.

Does it work on nested JSON from a SaaS API?

Not directly. DQDL does not evaluate nested or list-type columns, so you have to flatten the structure first and run the ruleset against the flattened frame. Plan the flattening deliberately, because your rules only ever see what the projection exposes.

How does it compare to Great Expectations, Soda or Monte Carlo?

Glue Data Quality’s real advantage is that it is already inside the runtime and the catalog: no extra infrastructure, no separate agent, one IAM story. The disadvantages are real too. DQDL is less expressive than Python-based assertions, the nested-data limitation is a hard wall, and it only covers what Glue can read. Great Expectations gives you more expressive checks at the cost of running it yourself; Monte Carlo and similar platforms add lineage and cross-warehouse coverage Glue does not attempt. Glue-centric stack, start here. Several engines, a standalone tool fits better.

Should I use the recommended rules?

As a starting draft, yes. As a ruleset, no. Recommendations are inferred from a sample of the data as it is today, which means any existing quality problem gets encoded as the expected baseline. Read every generated rule and delete the ones you would not have written yourself.

Can I reconcile row counts against the SaaS source itself?

Not directly; it only sees datasets it can read. The usual pattern is to have the extraction step record the count the API reported, land that alongside the data as a control table, then use RowCountMatch or AggregateMatch to compare the two. That turns “did we get everything” into a rule the engine can evaluate.

The one thing worth remembering

AWS Glue Data Quality is not really a data quality product. It is a way of writing down what you believe about a dataset so a machine can check it every run, and its value is entirely determined by whether those beliefs are written precisely enough to be wrong.

For SaaS sources that means two habits. Flatten deliberately and check immediately after, because your rules only see the projection. And treat every new dynamic or ML-based rule as unarmed until it has history, because a rule that cannot fail yet looks exactly like a rule that is passing.

Need help with data quality on your Glue pipelines?

I work with teams whose SaaS pipelines run green while the numbers drift. Things I can help with:

• Writing a DQDL ruleset for a specific SaaS object that catches schema drift instead of just restating the current schema.
• Designing the flattening and pivoting step for nested payloads and custom-field bags so the checks actually cover what matters.
• Setting up quarantine and fail-fast paths in Glue ETL, with the routing and alerting that makes a quarantine prefix something people act on.
• Cross-object reconciliation between SaaS extracts and control counts recorded at extraction time.
• Tuning anomaly detection and dynamic rules, including cleaning incident periods out of the training window.
• Getting data quality results out of the console and into Grafana, CloudWatch alarms or an Athena-queryable table.

If you have a ruleset that keeps passing while the data drifts, send me the DQDL and a schema dump. That is usually enough to see where the gap is.

The post AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed appeared first on John Nessime.

That is the shape of most API pipeline incidents. Not a crash. A success that isn’t one.

This post covers running Apache Airflow on AWS specifically for SaaS and API workloads: pulling from HubSpot, Salesforce, Stripe, Zendesk, Shopify, an internal partner API, whatever. It is organised by failure family rather than by feature, because the Airflow documentation already explains what an operator is and does a poor job of explaining which of these things will page you at 3am. I will cover choosing a deployment model, the concurrency trap that catches almost everyone, retry design, incremental state, secrets, and where the money actually goes.

Why SaaS and API sources break differently

When your source is a database you control, failure is loud: connection refused, deadlock, disk full. When it is somebody else’s SaaS API, three things change.

• You are a guest. The vendor decides your rate limit, and they can change it without telling you. Your pipeline’s correctness now depends on a number in someone else’s config file.
• Errors arrive as valid HTTP. A 429, a 200 with a truncated page, a 200 with an error object in the body. Your HTTP client is happy. Your data is not.
• Tasks spend most of their life waiting. API extraction is I/O bound almost end to end. That sounds harmless and is the root of the most expensive mistakes.

Pick the deployment model before you write a DAG

This decision constrains everything after it and is harder to reverse than people expect. Three realistic options.

Amazon MWAA, provisioned

AWS runs the scheduler, web server, workers, triggerer and metadata database on Fargate; you drop DAGs into an S3 bucket and they get picked up.

Where it wins: real Airflow, custom providers, custom plugins, full control over environment configuration. If your DAGs need arbitrary Python libraries, this option will not fight you.

Where it doesn’t: the environment bills by the hour whether or not anything is running. There is no scale to zero on the base environment. If you sync six APIs once a day and each run takes twenty minutes, you are paying for a mostly idle cluster around the clock. The mw1.micro class exists precisely for the small case, but it collapses the scheduler and worker into a single Fargate task and caps worker autoscale low, so treat it as a dev or isolation tier rather than a cheap production tier.

Amazon MWAA Serverless

You submit workflow definitions and AWS runs each task in its own Fargate container, billing per task duration with a one-minute minimum rather than per environment hour.

Where it wins: spiky or infrequent schedules. If the workload is “six syncs a day, nothing overnight,” the cost profile beats a permanently running environment by a wide margin. Each workflow also gets its own IAM execution role, which is a real security improvement over one shared role per environment.

Where it doesn’t: it leans on declarative YAML workflow definitions based on the DAG Factory format and a curated set of AWS operators. That is a deliberate trade: because the definition is declarative, the service can schedule tasks without executing your DAG code. It also means custom operators, exotic third-party providers and clever Python at parse time are not the sweet spot. It is also available in fewer regions than provisioned MWAA, so check your region before you design around it.

Self-managed on ECS, EKS or a VPS

On EKS with the Kubernetes executor you get per-task pods and tight cost control. On a single VPS from a provider like InterServer or Hetzner, a Docker Compose stack with a Postgres metadata database will run a modest set of API syncs for a fraction of any managed price.

Where it wins: cost at both extremes, and total control. Where it doesn’t: you now own metadata database upgrades, major version migrations, log retention and the 2am scheduler restart. Managed Airflow is a bet that your time is worth more than the hourly premium. For a solo engineer with three pipelines that bet often loses; for a data team of eight it usually wins. Astronomer is the main non-AWS managed option worth pricing alongside these.

Failure family one: the throttle that silently stops throttling

You start with a normal setup: an Airflow pool named crm_api with four slots, and every task that touches the vendor assigned to it. Four concurrent requests, comfortably under the vendor’s limit. This works.

Then you notice those tasks spend nearly all their runtime waiting on HTTP, burning worker slots to sit still. So you switch them to deferrable operators. A deferrable task suspends itself while waiting, releases its worker slot, and hands the waiting to the triggerer, which polls asynchronously. Worker pressure drops. Everything looks better.

And your rate limiting quietly stops working.

By default, a pool does not count tasks in the deferred state as occupying slots. That was deliberate, and the logic is sound in the abstract: a deferred task is not consuming a worker. But if you were using the pool to protect an external API rather than your own workers, it has just stopped doing the job you gave it. Every task can defer at once, and the vendor sees the full fan-out.

The fix is a per-pool flag, include_deferred, which tells the scheduler to count deferred tasks against the slot budget. It is off by default. You can set it when editing the pool in the Airflow UI, or through the API.

The failure signature is what makes this nasty. Nothing errors. Your DAG gets faster. The vendor starts returning 429s that your retry logic absorbs, and the only symptom is that runs take a little longer and occasionally a page goes missing. Weeks can pass. Two related traps in the same family:

• max_active_tasks at the DAG level has the same blind spot with deferred tasks, and there is no equivalent opt-in flag. If you need a hard external concurrency cap, use a pool with include_deferred enabled, not DAG-level concurrency.
• On MWAA, the triggerer runs alongside the scheduler on the same Fargate task, so scheduler count and triggerer capacity are linked. If you go heavily deferrable and your deferred tasks start stalling, scheduler capacity is the thing to look at.

Failure family two: retries that make the outage worse

The default instinct is to set retries high and move on. Against a rate-limited API, a fixed retry delay across many parallel tasks is just a slower version of the same stampede.

What you want is exponential backoff with a ceiling. The shape:

from datetime import timedelta

from airflow.sdk import dag, task

@dag(
    schedule="0 5 * * *",
    catchup=False,
    max_active_runs=1,          # never let two runs of this DAG overlap
    default_args={
        "retries": 5,
        "retry_delay": timedelta(seconds=30),
        "retry_exponential_backoff": True,   # 30s, 60s, 120s, 240s...
        "max_retry_delay": timedelta(minutes=15),  # stop doubling here
        "pool": "crm_api",      # shared budget across every task touching this vendor
    },
    tags=["crm", "extract"],
)
def crm_extract():

    @task(max_active_tis_per_dag=4)
    def fetch_page(page_token: str) -> str:
        ...

crm_extract()

The lines that matter:

• retry_exponential_backoff turns retry_delay into a base rather than a constant, so repeated failures spread out instead of hammering in lockstep.
• max_retry_delay caps the doubling. Without it, a task that fails five times can sit idle for hours and blow past the window you actually cared about.
• max_active_runs=1 is the one people skip. If a run overruns its schedule, the next one starts anyway, and now two runs are fetching the same pages from the same vendor with the same credentials. This is a common way to trigger a rate limit you have never hit before.
• max_active_tis_per_dag limits how many instances of that specific task run concurrently across DAG runs, which is the right knob for dynamically mapped extraction tasks.

One thing Airflow will not do for you: honour a Retry-After header. Airflow’s retry timing is computed from your config, not from the vendor’s response. If the API tells you exactly how long to wait, you have to catch that in your own code and sleep or reschedule accordingly. Ignoring a header the vendor bothered to send is a good way to get your API key throttled harder.

Failure family three: pagination, cursors and the empty page

Back to the Monday message. The specific bug behind most “the numbers are wrong but nothing failed” incidents is a loop that treats any non-error response as a terminating condition. Three rules prevent it:

1. Never infer “done” from an empty result. Terminate on the explicit signal the API gives you: a null next_cursor, a missing Link header, a page count. An empty array with a valid cursor still has more data behind it.
2. Assert the response shape before you use it. Check the status code explicitly and validate that the fields you depend on exist. A 200 carrying {"error": "..."} should raise, not return zero rows.
3. Land raw, transform later. Write the untouched API response to S3 first, then parse from S3. When the vendor changes a field type, you can replay from raw instead of re-extracting from an API that no longer serves that window.

Where to keep incremental state

The tempting pattern is to store the last-seen timestamp in an Airflow Variable and update it at the end of a run. Do not make that your source of truth. If a run dies midway, the Variable is in an undefined state, and clearing and re-running the DAG will not restore it. Airflow’s retry and backfill machinery has no idea it exists.

Better: make each run’s window a function of the run itself, and write output to a deterministic, run-scoped location such as s3://bucket/source=crm/dt=<logical-date>/. Re-running the same interval overwrites the same prefix. That is what makes a task idempotent, and idempotency is the difference between “clear the task and let it rerun” and a two-hour manual repair.

Then overlap your windows deliberately. Many SaaS APIs order results by modified time with eventual consistency, so a record edited at the boundary can appear after you have already moved on. Query a window slightly wider than your schedule interval and rely on an idempotent upsert downstream to absorb the duplicates. Late-arriving data is not an edge case with SaaS sources. It is the normal case.

Airflow’s asset-based scheduling is the clean way to trigger downstream DAGs from this: the extract DAG produces an asset, and the transform DAG runs when the asset updates, rather than being scheduled at a time you hope is late enough.

Failure family four: credentials

API tokens rotate, sometimes on the vendor’s schedule rather than yours. Storing an API key in an Airflow Connection through the UI works, and is the wrong long-term answer: the value lives in the metadata database and there is no rotation story. On AWS, point Airflow’s secrets backend at AWS Secrets Manager. On MWAA that is an environment configuration option:

secrets.backend
  airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

secrets.backend_kwargs
  {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}

With that in place, a connection lookup for crm_default resolves against the secret at airflow/connections/crm_default. Rotating the credential is a Secrets Manager operation with no Airflow deployment involved.

Two things to know before you turn it on. First, every connection and variable lookup becomes a Secrets Manager API call, and lookups fall through to the backend before hitting the metadata database, so a DAG that reads a Variable at parse time will generate a call on every parse cycle. Move those reads inside tasks. Second, the environment’s execution role needs explicit read permission on the relevant secret ARNs, and if you use a customer-managed KMS key, decrypt permission on that key too.

Worth knowing if you are on Airflow 3: task code can no longer reach the metadata database directly. All runtime interaction goes through the Task Execution API. If you inherited custom operators that open a session and query Airflow’s own tables, that is a migration blocker, not a warning.

Failure family five: the bill

Nobody is surprised by the environment line item. They are surprised by the other four.

• Idle time. A provisioned MWAA environment bills continuously. Compute the ratio of hours billed to hours doing work. If it is bad, that is the argument for MWAA Serverless or for consolidating several thin pipelines into one environment.
• NAT Gateway. This is the classic one. Private-subnet workers calling public SaaS APIs route through a NAT Gateway, which charges hourly and per gigabyte processed. A high-volume extraction pipeline can spend more on NAT than on Airflow. VPC endpoints remove that cost for AWS service traffic, but they do nothing for calls to a third-party API, which is exactly the traffic an API pipeline generates.
• CloudWatch Logs. Task logs go to CloudWatch, and ingestion is billed per gigabyte. Set the Airflow log level per component rather than globally at DEBUG, and set a retention policy on the log groups. The default is to keep logs forever.
• S3 requests. Landing raw API responses one small object per page generates a lot of PUTs. Batch pages into larger objects where you can.

Rates and dimensions change, so model your own workload against the current pricing page rather than trusting a number from a blog post. The point is knowing which four lines to look at.

Troubleshooting Apache Airflow on AWS when API pipelines misbehave

Tasks sit in “queued” and never start

Usually a slot problem, not a broken scheduler. Check, in order: is the pool full; has DAG-level max_active_tasks been hit; is worker autoscaling at its configured maximum. On MWAA, the container and queue utilisation metrics published to CloudWatch tell you which of the three it is far faster than reading scheduler logs.

DAG file changes don’t appear

On MWAA, DAGs sync from S3 on an interval; it is not instant. If a file has been there for several minutes and still hasn’t appeared, it almost always failed to parse. Check the DAG processing logs in CloudWatch rather than the scheduler logs, because a broken import raises there and never reaches the scheduler.

A new provider package won’t install

MWAA installs from your requirements.txt in the DAGs bucket, and from Airflow 2.7.2 onward that file must include a constraint line. Without one, MWAA picks a constraint for you, and pip is free to resolve a provider version that conflicts with the Airflow build in the image.

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<AIRFLOW_VERSION>/constraints-<PYTHON_VERSION>.txt"

apache-airflow-providers-http
apache-airflow-providers-salesforce

Substitute the literal Airflow version your environment runs and the Python version bundled with it. MWAA does not expand shell variables in that file. Test the requirements file against a local Airflow image before you upload it, because a failed install on MWAA surfaces as a partially working environment rather than a clean error.

A backfill is stuck and you need to clear it

You do not need a web login token for this. MWAA exposes the Airflow REST API through a signed AWS API call, so you can drive it from CI or a runbook with normal IAM credentials:

aws mwaa invoke-rest-api 
  --name MyMWAAEnvironment 
  --path "/dags/crm_extract/clearTaskInstances" 
  --method POST 
  --body '{"dry_run": true}'

Start with dry_run set to true so the response tells you which task instances would be cleared before you actually clear them. Note that the resource paths differ between Airflow 2 and Airflow 3 environments, so confirm against the API version your environment exposes.

Deferred tasks stall forever

If deferred tasks stop resuming while the environment reports healthy, suspect the triggerer rather than your DAG. A triggerer that has lost its ability to process triggers can keep heartbeating normally, so the scheduler sees nothing wrong while every deferred task drifts toward timeout. This class of bug has been fixed and re-fixed upstream, so check your Airflow version’s release notes before assuming it is your code.

Common mistakes

• Switching to deferrable operators without enabling include_deferred on the pools that were protecting the API.
• Leaving max_active_runs unset, so a slow run and the next scheduled run compete for the same rate limit budget.
• Treating an empty response page as the end of pagination.
• Storing the incremental watermark in an Airflow Variable and updating it mid-run.
• Calling an API or reading a Variable at DAG parse time, which executes on every parse cycle rather than once per run.
• Transforming during extraction, so a vendor schema change means re-pulling data the API may no longer serve.
• Sizing the environment for peak concurrency when the actual constraint is the vendor’s rate limit.

Best practices

• One pool per vendor, sized to their published limit with headroom, and include_deferred enabled on every one of them.
• Land raw responses to S3 before parsing. Extraction and transformation are separate tasks with separate failure modes.
• Make every task idempotent and window-scoped, so “clear and rerun” is always a safe repair.
• Overlap extraction windows and deduplicate downstream rather than trusting a vendor’s timestamps to be exact.
• Secrets Manager for credentials, with the execution role scoped to specific secret ARNs.
• Alert on row counts and freshness, not just task state. A green DAG that produced 40% of yesterday’s rows is the failure you actually care about. Shipping Airflow’s StatsD metrics into Prometheus, Grafana Cloud or Datadog makes that a dashboard rather than a discovery.
• Define the environment in Terraform or OpenTofu. Recreating an MWAA environment by hand after a bad configuration change is a bad afternoon.

Frequently asked questions

Is MWAA worth it compared to self-hosting Airflow on EC2?

It depends almost entirely on how many people share the platform. MWAA’s premium buys you managed metadata database upgrades, patched images and version migration support. If one engineer maintains three DAGs, self-hosting on a modest VPS is cheaper and the operational load is real but small. Once several teams depend on the scheduler being up, the premium is easy to justify.

Should I use Step Functions instead of Airflow for API pipelines?

Step Functions is genuinely better for event-driven, AWS-service-centric orchestration with modest branching, and it scales to zero. Airflow wins when you need scheduled batch semantics, backfills over historical windows, dependencies between many pipelines, and a UI that non-platform engineers can use to see why last Tuesday failed. Backfill is usually the deciding feature.

Do deferrable operators reduce my AWS bill?

On provisioned MWAA, they reduce worker pressure, which reduces autoscaling into additional worker instances. The base environment cost is unchanged. On a Kubernetes executor setup where each task is a pod, the saving is more direct. Either way, do not adopt them purely for cost without revisiting your pool configuration first.

How do I handle a vendor with no documented rate limit?

Start conservative, one or two concurrent requests, and instrument the response status codes. Raise the pool size gradually and watch for 429s or rising latency. Latency creeping up under load is often the earlier signal, because some vendors throttle by slowing you down before they start rejecting.

Can Airflow read a Retry-After header automatically?

No. Airflow computes retry timing from retry_delay and the backoff settings on the task. If a vendor sends Retry-After, you need to handle it in your own request code or in a custom operator.

What breaks when upgrading to Airflow 3?

The big one for API pipelines is that task code can no longer access the metadata database directly; everything goes through the Task Execution API. Imports also move to the airflow.sdk namespace, and several core operators now live in the standard provider package. Audit custom operators first, since that is where direct database access hides. MWAA requires you to be on the latest Airflow 2 minor version before a major upgrade, so plan two steps.

How many DAGs can one MWAA environment handle?

The binding constraint is usually the metadata database and scheduler CPU, not DAG count. Watch metadata database memory and scheduler CPU utilisation; when either saturates, you either move up an environment class or split into multiple environments. Splitting also gives you blast-radius isolation, which matters more than people expect.

Wrapping up

Running Apache Airflow on AWS for SaaS and API pipelines is mostly not an Airflow problem. The scheduler works. The operators work. What bites is the gap between “the task succeeded” and “the data is correct,” and that gap lives in concurrency settings, pagination logic and retry design rather than anywhere Airflow will warn you about.

If you take one thing away: a green DAG is not a signal that your data is complete. Enable include_deferred on the pools protecting your vendors, terminate pagination on an explicit signal instead of an empty page, make every task idempotent, and alert on row counts. Those four things prevent most of the incidents that never show up as a failed task.

Need help with your Airflow pipelines on AWS?

I work with teams running data and API pipelines on AWS, usually somewhere between “it works but nobody trusts it” and “we need to move off cron.” Things I can help with:

• Reviewing existing DAGs for silent data loss: pagination logic, retry behaviour, pool and concurrency configuration.
• Choosing between MWAA provisioned, MWAA Serverless and self-managed Airflow, with a cost model for your actual schedule rather than a generic comparison.
• Building SaaS extraction pipelines that are idempotent and safely re-runnable, landing raw to S3 with incremental windows that survive failure.
• Cutting MWAA cost: environment right-sizing, NAT Gateway traffic, CloudWatch log volume and dependency install time.
• Airflow 2 to 3 migration audits, focused on custom operators and direct metadata database access.
• Data freshness and volume alerting in Grafana or CloudWatch, so you learn about a partial sync before the business does.

If something specific is broken, send me the DAG file, the task log, or the CloudWatch metrics for the run that went wrong. It is usually faster to look at the real thing than to describe it.

The post Apache Airflow on AWS: Building SaaS and API Pipelines That Don’t Lie to You appeared first on John Nessime.

The frustrating part is that the pipeline is fine. Webhooks are arriving, Lambda is running clean, S3 has the files, Athena returns rows. Nothing is broken in the way monitoring understands “broken”. The pipeline is faithfully recording something that is no longer true.

This post is about building a Shopify sales dashboard with AWS that survives that conversation. Not the wiring, which is well documented and mostly straightforward, but the design decisions that determine whether your numbers still hold up six months in. I’ll cover the three ingestion paths and when each one is the right call, why append-only pipelines drift, how to lay out S3 and Athena so recomputation is cheap, and what to do when the totals are already wrong.

Why a Shopify sales dashboard with AWS drifts from the admin

Here’s the thing that catches almost everyone: a Shopify order is not an event, it’s a mutable record.

An event pipeline assumes facts are immutable once written. A payment happened. A shipment left. You append it, you never touch it again, and the sum of the log is the truth. That model is why streaming architectures are so clean, and it’s exactly wrong for order data.

An order created on Monday can be edited on Tuesday, partially refunded on Friday, and fully refunded three weeks later. Every one of those changes belongs, financially, to Monday. If your pipeline appends the orders/create payload and never revisits it, Monday’s revenue is frozen at the moment of checkout and it will only ever be too high.

This is the invisible failure. Nothing alerts. No queue backs up. Your dashboard is confidently wrong, and the gap widens roughly in proportion to your return rate. A store with a two percent return rate takes a long time to notice. A fashion store running thirty percent returns notices in about a month, usually via an angry accountant.

The four adjustments that move historical numbers

• Refunds. Full or partial. A refund carries its own created_at, which is when the money moved back. The order it belongs to has a different, earlier date. You need both, and which one you attribute to depends on whether finance wants cash-basis or order-basis reporting. Ask before you build.
• Order edits. A merchant adds a line item or adjusts a quantity after the fact. The original payload is now stale. Shopify exposes both the original and the current totals precisely because of this.
• Cancellations. A cancelled order keeps existing in the API. If you filter only on payment status you will happily keep counting it.
• Test and draft orders. Test orders carry a flag marking them as such. Nobody remembers to filter these until a QA run during a quiet week produces a suspicious spike.

The design consequence is simple to state and annoying to implement: your pipeline must be able to recompute any past day. Every storage and partitioning decision below follows from that one requirement.

Getting data out of Shopify: three paths, three trade-offs

Before anything else: new Shopify apps are built on the GraphQL Admin API. The REST Admin API has been designated a legacy API and new public apps must use GraphQL. If you’re starting fresh, start there. If you inherited a REST integration, it probably still runs, but you’re on borrowed time and you should plan the migration rather than discover the deadline.

Path 1: EventBridge partner event source

Shopify can deliver webhooks straight into an Amazon EventBridge partner event bus in your account. No public endpoint, no API Gateway, no HMAC verification code, because verification only applies to HTTPS deliveries. Shopify’s own docs confirm EventBridge and Pub/Sub deliveries skip it.

You create the source in the Shopify app configuration using your AWS account ID, region and a source name, then associate it with an event bus in the EventBridge console and write rules to route it. The address you register with Shopify is the partner event source ARN, not the event bus ARN. That distinction accounts for a large share of the “I set it up and nothing arrives” threads on the Shopify forums.

A rule matching everything from the Shopify partner source looks like this. Start broad, then narrow once you’ve seen the real shape of an event:

{
  "source": [ { "prefix": "aws.partner/shopify.com" } ]
}

Send that to an SQS queue with a dead-letter queue attached rather than straight to Lambda. Buffering gives you a replay buffer when a downstream deploy goes wrong, and the DLQ means a bad payload parks itself instead of poisoning the whole rule. This is the path I reach for first for anything already on AWS.

Path 2: HTTPS webhooks into API Gateway and Lambda

The conventional route, and the right one if you need webhook delivery outside AWS too, or you want the payloads to pass through something you fully control. The cost is that you now own an internet-facing endpoint and the HMAC verification on it.

Verify against the raw request body, before any JSON parsing. Re-serialising the payload changes byte-for-byte content and the signature will never match. Use a constant-time comparison so the check doesn’t leak timing information:

import base64, hashlib, hmac

def verify(raw_body: bytes, header_hmac: str, secret: str) -> bool:
    digest = hmac.new(secret.encode(), raw_body, hashlib.sha256).digest()
    computed = base64.b64encode(digest).decode()
    return hmac.compare_digest(computed, header_hmac)

Shopify sends the signature in the X-Shopify-Hmac-SHA256 header, base64-encoded, computed with your app’s client secret over the raw body. Store that secret in Secrets Manager or as an SSM SecureString parameter, not in a Lambda environment variable.

Acknowledge fast. Shopify’s timeout is short and it retries with backoff over a finite window, so a handler that does real work inline will generate a wall of duplicate deliveries during a flash sale, exactly when you can least afford it. Return 2xx immediately, do the work asynchronously.

Path 3: scheduled GraphQL bulk pull

Webhooks give you low latency. They do not give you completeness. Anything that fails past its retry window is gone, and Shopify will eventually remove a subscription that keeps failing. That’s a silent data loss mode with no local symptom at all.

So run a scheduled reconciliation pull alongside the stream. Shopify’s GraphQL bulk operations are built for this: you submit a query, it runs asynchronously, and you fetch a JSONL result file when it finishes. That’s the right tool for backfills and nightly catch-up, rather than paginating thousands of pages against a points-based rate limiter and getting throttled halfway through.

A nightly job that re-pulls the last seven to fourteen days and overwrites those partitions costs almost nothing and quietly fixes every category of drift described above. If you build one thing from this post, build that.

Decide what “revenue” means before you write a line of SQL

This is where most reconciliation arguments actually live, and it isn’t an engineering problem at all until you’ve had the conversation.

Shopify’s own sales reporting builds total sales from gross sales, minus discounts, minus returns, plus taxes and shipping. Gift card sales sit outside that in a separate finance report. If your dashboard sums order totals and calls it revenue, you have built a different metric with the same name, and it will disagree with the admin forever no matter how good your pipeline is.

Write the definition down. Put it in the dashboard as a tooltip. When someone challenges a number, you want the argument to be about the definition, not about whether your infrastructure works.

The currency trap

If the store sells in more than one currency, the money fields split in two. Shopify exposes totals as a set containing both shop_money and presentment_money: the amount in the store’s base currency, and the amount the customer actually saw and paid.

Sum the presentment amounts across a multi-currency store and you get a number with no meaning at all, euros and yen added together as if they were the same unit. For a single reporting figure you want the shop-currency side. Keep the presentment amount and its currency code in the table anyway, because the day someone asks “how much did we actually sell in Germany”, you’ll want it and it is painful to backfill.

One caveat worth knowing: orders created through the API rather than through checkout can behave differently from native multi-currency checkout orders. If your store takes orders from an ERP or a marketplace integration, spot-check a few of those specifically.

Storage layout: partition by order date, never by arrival date

Two layers in S3. Keep them separate and keep them honest about what they are.

1. Raw. Every payload exactly as received, partitioned by ingestion date. Append-only, never edited. This is your audit trail and your rebuild source. Lifecycle it to a colder storage class after a few months, don’t delete it.
2. Curated. One row per order representing current state, in Parquet, partitioned by order date. This is what the dashboard queries. It is derived, disposable and rewritable.

The partitioning choice on the curated layer is the load-bearing decision in the whole design. If you partition by arrival date, which is what Amazon Data Firehose does by default because it buckets on the moment it writes the file, then a refund that arrives three weeks late lands in today’s partition. Correcting Monday now means finding and rewriting fragments scattered across twenty other partitions. Partitioned by order date, correcting Monday means overwriting exactly one prefix.

Firehose can do this with dynamic partitioning, which routes records by keys inside the payload rather than by write time. If you’re not using Firehose, extract the order date in your Lambda and write the prefix yourself.

Use partition projection so Athena stops guessing

The default Glue Data Catalog approach means running a crawler or issuing MSCK REPAIR TABLE to register new partitions. Forget one and you get a query that silently returns nothing for recent days. Nobody notices until Monday.

Partition projection removes the metastore lookup entirely. You tell Athena the shape of the partition keys and it calculates the prefixes at query time:

CREATE EXTERNAL TABLE shop_orders (
  order_id             bigint,
  order_number         string,
  created_at           timestamp,
  financial_status     string,
  cancelled_at         timestamp,
  is_test              boolean,
  total_shop           decimal(12,2),
  shop_currency        string,
  total_presentment    decimal(12,2),
  presentment_currency string
)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://your-bucket/curated/orders/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.order_date.type' = 'date',
  'projection.order_date.format' = 'yyyy-MM-dd',
  'projection.order_date.range' = '2019-01-01,NOW',
  'projection.order_date.interval' = '1',
  'projection.order_date.interval.unit' = 'DAYS',
  'storage.location.template' =
    's3://your-bucket/curated/orders/order_date=${order_date}/'
);

Set the range start to your store’s actual first order month. Projection generates every prefix in the range, so a range starting a decade too early makes wide scans slower for no benefit.

Parquet matters here for the same reason. Athena bills on bytes scanned, so a columnar format with good compression cuts the bill directly, and a dashboard that only ever selects six columns from a forty-column table never touches the rest.

Net sales in one query

With refunds in their own table keyed by order and carrying their own date, attributing them back to the original order day is a left join and a subtraction:

SELECT
    o.order_date,
    SUM(o.total_shop)                                AS gross_shop,
    SUM(COALESCE(r.refunded_shop, 0))                AS refunded_shop,
    SUM(o.total_shop - COALESCE(r.refunded_shop, 0)) AS net_shop
FROM shop_orders o
LEFT JOIN (
    SELECT order_id, SUM(amount_shop) AS refunded_shop
    FROM shop_refunds
    GROUP BY order_id
) r ON r.order_id = o.order_id
WHERE o.order_date BETWEEN '2025-01-01' AND '2025-01-31'
  AND o.is_test = false
  AND o.cancelled_at IS NULL
GROUP BY o.order_date
ORDER BY o.order_date;

Note the two filters doing quiet work at the bottom. Those two lines are the difference between a number finance accepts and a number they don’t.

Choosing the dashboard layer

Once the data is correct, this part is genuinely a preference. All of these work.

• Amazon QuickSight, now delivered as part of Amazon Quick Suite, is the least-friction option if you’re already in AWS. Its in-memory SPICE layer means viewers aren’t firing an Athena query per chart interaction, which controls both latency and scan cost. Per-viewer pricing tends to be the deciding factor either way, so model it for your actual audience size before committing.
• Grafana with the Athena data source is a good fit if you’re already running Grafana for infrastructure and want commercial and operational panels on one screen. Grafana Cloud removes the hosting question if you’d rather not run it.
• Power BI makes sense when the finance team already lives in Microsoft 365 and models in DAX. The cross-cloud hop is real but manageable.
• Metabase or a self-hosted alternative on a small VPS from a provider like InterServer or Hetzner is the pragmatic answer for a handful of internal viewers, where per-seat BI licensing costs more than the entire pipeline.

The honest trade-off: managed BI costs more per month and saves you from becoming the person who patches the reporting server. Self-hosting inverts that. Neither is wrong, but pick deliberately rather than by inertia.

Troubleshooting: symptom to cause

Totals are consistently higher than the Shopify admin

Almost always refunds, cancellations or test orders. Check in that order. If the gap grows with the age of the reporting window, it’s refunds. If it’s a fixed offset on specific days, look for test orders or a QA run.

Totals are lower, and recent days are missing rows

Either partitions aren’t registered, which projection fixes permanently, or the webhook subscription has been dropped after repeated delivery failures. Check the subscription still exists before you go digging through Lambda logs. A nightly bulk pull would have masked this, which is another argument for having one.

Orders appear twice

Shopify’s delivery model is at-least-once, not exactly-once, and you may also have more than one subscription on the same topic. Deduplicate on the delivery ID header before you touch anything else, and make the write itself idempotent so a duplicate is a no-op rather than a second row.

Numbers are right on the daily view, wrong on the monthly

Timezone. Order timestamps carry an offset; your partition key is a date string. If you derive the date in UTC and the store reports in a local timezone, orders near midnight land on the wrong day. That averages out over a month, which is exactly why the discrepancy hides until month boundaries.

Athena costs jumped without more data

Someone built a dashboard with a filter that doesn’t hit the partition column, so every panel refresh scans the full table. Look at bytes scanned per query and check whether the BI tool is caching results or re-querying on every interaction.

Common mistakes

• Treating orders as immutable events and never revisiting a past day.
• Partitioning on arrival time because that’s the default, then discovering corrections are expensive.
• Verifying the HMAC against a re-serialised body instead of the raw bytes.
• Doing real work inside the webhook handler, generating duplicates under load.
• Summing presentment amounts across currencies.
• Registering the event bus ARN with Shopify instead of the partner event source ARN.
• Relying on webhooks alone with no scheduled reconciliation.
• Shipping a “revenue” number without ever defining what it includes.

Best practices

• Keep raw and curated layers separate. Raw is append-only; curated is rewritable.
• Make every partition idempotently rebuildable from raw. Test that path deliberately, before you need it.
• Buffer through SQS with a dead-letter queue. Free replay, free isolation of bad payloads.
• Run a nightly bulk pull over a rolling window and overwrite those partitions.
• Use partition projection. It removes an entire category of silent failure.
• Alarm on the absence of events, not just on errors. A CloudWatch alarm on zero orders processed in an hour during business hours catches broken subscriptions the same day.
• Store the API secret in Secrets Manager and scope the Lambda role to the exact prefixes it writes.
• Publish a reconciliation panel comparing your total to the admin’s for the same window. Surfacing the gap builds more trust than hiding it.

Frequently asked questions

Do I need a data warehouse, or is S3 and Athena enough?

For a single store’s order data, S3 with Athena is almost certainly enough, and it’s cheaper because you pay per query rather than for a running cluster. Redshift starts to earn its place when you’re joining Shopify data against several other large sources, or when concurrent query load makes Athena’s queue times noticeable.

How near-real-time can this be?

Events land within seconds. The practical floor is your buffering window, since writing one tiny file per order gives you a small-files problem that ruins query performance. A few minutes of buffering is the usual compromise. If you genuinely need sub-minute order counts, put a live counter in DynamoDB alongside the analytical pipeline rather than trying to make the data lake do both jobs.

Which webhook topics should I subscribe to?

At minimum, order creation, order update, order cancellation and refund creation. Update and refund topics are the ones people skip, and they’re exactly the ones carrying the corrections. Subscribe to fewer topics than you think you need and add rather than subscribing to everything, since every extra topic is volume you pay to store and process.

Can I skip AWS and use a connector tool?

Yes, and for many stores that’s the right answer. A managed connector into a hosted warehouse gets you a working dashboard in an afternoon. You’re paying a monthly fee to avoid owning any of this, and trading away control over the data model. Building it on AWS wins when you need Shopify data joined to systems the connector doesn’t cover, or when row-based connector pricing outgrows the infrastructure cost.

How do I backfill historical orders?

Use a GraphQL bulk operation rather than paginating the API. Submit the query, poll for completion, then stream the JSONL result into your raw bucket and run the same transformation your live pipeline uses. If backfill and live processing use different code paths, they will diverge, and you’ll spend an afternoon working out which one is lying.

What does a setup like this cost to run?

For a typical single store, the pipeline itself is small money: Lambda invocations, a few gigabytes in S3, and Athena billed on bytes scanned, which partitioning and Parquet keep low. The BI seats are usually the largest line item, which is why the dashboard layer decision deserves more thought than the ingestion one. Model it against current published rates rather than trusting any figure you read in a blog post, including this one.

The one thing to take away

A Shopify sales dashboard with AWS doesn’t fail because the pipeline breaks. It fails because the pipeline keeps working perfectly on data that has since changed underneath it.

Design for correction from the first commit. Partition by order date, keep the raw layer so you can always rebuild, run a scheduled pull to catch what the stream missed, and agree on what revenue means before anyone builds a chart. Do that and the Monday morning screenshot comparison becomes a non-event, which is the highest praise a reporting pipeline ever gets.

Need help with your Shopify data pipeline on AWS?

I design and build ecommerce data pipelines and reporting stacks on AWS. Typical engagements look like:

• Working out why an existing Shopify dashboard disagrees with the admin, and fixing the root cause rather than patching the query
• Building the ingestion layer end to end: EventBridge or API Gateway, Lambda, SQS with dead-letter handling, and a scheduled GraphQL bulk reconciliation job
• Designing the S3 layout, Glue schema and Athena tables so past days can be recomputed cheaply and partitions never go missing
• Migrating REST Admin API integrations to GraphQL before the deadline forces the issue
• Building the dashboard itself in QuickSight, Grafana or Metabase, including the metric definitions finance will actually sign off on
• Cutting Athena scan costs and BI licensing on a reporting stack that has grown more expensive than anyone planned

If you’re in the middle of one of these, send me the actual thing: the Athena query, the S3 prefix layout, the two totals that don’t match. It’s a much faster conversation than describing it in the abstract.

The post Shopify Sales Dashboard with AWS: Build One That Actually Reconciles appeared first on John Nessime.

That is usually the moment someone says “let’s just pull it into a warehouse.” A Jira analytics pipeline built on AWS Lambda, S3 and Athena is a reasonable answer to that, and it is genuinely cheap to run. It is also easy to build a version that looks correct for two weeks and then quietly reports numbers that are thirty percent wrong.

This post covers the extraction and modelling problems that actually cost you time: the search endpoint Atlassian removed, the change history that truncates without erroring, the incremental load pattern that duplicates rows across partitions, and the S3 layout decisions that decide whether Athena costs you pennies or hundreds. Code where it clarifies something, and honest notes on what I would skip.

Where Jira’s own reporting genuinely stops

Give the built-in tooling its due first. Jira’s velocity, burndown and control charts are fine for a single team inspecting its own recent work, they need no infrastructure, and they update instantly. Marketplace apps like eazyBI and Custom Charts cover a lot of ground without you writing a line of Python. If your question is “how did this sprint go,” you do not need a pipeline.

Where it stops is anything that crosses a boundary. Comparing lead time across projects that use different workflows. Joining ticket data to deploy events from your CI system or incident data from PagerDuty. Retaining a consistent view of history after someone renames a status or archives a board. Answering a question nobody anticipated when the board was configured. Those need the raw data somewhere you control, in a shape you decide.

The shape of the pipeline

The architecture is unremarkable, which is the point:

1. EventBridge Scheduler triggers a Lambda function on a schedule.
2. Lambda reads a Jira API token from Secrets Manager and pages through the Jira Cloud REST API.
3. It writes Parquet files to S3, partitioned by load date.
4. The Glue Data Catalog holds the table definitions, with partition projection so nothing has to crawl.
5. Athena queries S3 directly. Grafana, Power BI, Metabase or QuickSight sit on top of Athena.

No cluster, no always-on database, nothing to patch. The whole thing costs about as much as a small EC2 instance for a mid-sized Jira site, and most of that is S3 storage. Terraform or CloudFormation to define it, GitHub Actions to deploy it.

The complexity is not in the wiring. It is in four places, and they are worth taking in order.

Failure family 1: the search endpoint you were probably going to use is gone

Almost every Jira extraction tutorial and a good number of client libraries still reach for GET /rest/api/3/search. Atlassian removed it from Jira Cloud. It returns 410 Gone. If you are copying a script from a blog post that predates the change, this is the first thing that breaks, and at least it breaks loudly.

The replacement is /rest/api/3/search/jql, available as both GET and POST. Use POST for anything real, because JQL strings get long and you avoid URL encoding entirely. Three behavioural changes matter more than the URL:

Pagination is cursor-based, and there is no total

startAt is gone. You get a nextPageToken back and hand it to the next request. There is no total in the response, which means any progress bar, any “expected N rows” sanity check, and any loop that terminated on startAt >= total needs rewriting.

If you only need a count, there is a separate operation, POST /rest/api/3/search/approximate-count, which takes a JQL body and returns an approximate figure without paging through results. It is genuinely useful as a reconciliation check: run it before extraction, compare against the row count you actually wrote, and alert on a large gap.

There have been persistent community reports of isLast behaving unreliably on this endpoint, including tokens that chain without ever terminating. Do not trust isLast as your loop condition. Terminate on the absence of nextPageToken, and put a hard page cap in as a circuit breaker so a bad token cannot burn your entire Lambda budget in one invocation.

def search_issues(session, base_url, jql, fields, max_pages=2000):
    """Page through /search/jql. Terminates on missing nextPageToken,
    not on isLast, which has been reported as unreliable."""
    token = None
    for _ in range(max_pages):
        body = {"jql": jql, "fields": fields, "maxResults": 100}
        if token:
            body["nextPageToken"] = token
        r = session.post(f"{base_url}/rest/api/3/search/jql",
                         json=body, timeout=60)
        r.raise_for_status()
        page = r.json()
        for issue in page.get("issues", []):
            yield issue
        token = page.get("nextPageToken")
        if not token:
            return
    raise RuntimeError("page cap hit, refusing to loop further")

You have to ask for fields explicitly

The new endpoint does not hand you every field by default. Omit fields and you get essentially nothing back. This is the failure that looks like success: the pipeline runs, files land in S3, row counts look plausible, and every analytical column is null.

Be explicit and be narrow. Every field you request costs response size, and Jira sites accumulate hundreds of custom fields nobody uses. Name what you need:

FIELDS = [
    "summary", "status", "issuetype", "project", "priority",
    "assignee", "reporter", "created", "updated", "resolutiondate",
    "labels", "components", "parent",
    "customfield_10016",   # story points on this site, verify yours
]

Custom field IDs are per-site. Do not hardcode one you read in someone else’s blog post. Pull /rest/api/3/field once, find the field by name, and either store the mapping in config or resolve it at runtime and log what it resolved to. When a Jira admin rebuilds a field, an ID-based pipeline goes null and a name-resolving pipeline keeps working.

Rate limits and the fifteen-minute wall

Jira Cloud applies cost-based rate limiting and returns HTTP 429 when you exceed it, typically with a Retry-After header. Respect that header rather than inventing your own backoff. A naive retry loop that ignores it turns a brief throttle into a sustained one.

Lambda’s hard ceiling is fifteen minutes. A full historical backfill of a large Jira site will not finish in one invocation, and the ugly failure mode is a function that times out at minute fifteen having written half its data with no record of where it stopped. Two ways out:

• Shard the work. Fan out one Lambda invocation per project key, or per month of created date. Each one is small, independently retryable, and finishes well inside the limit.
• Checkpoint and continue. Persist the current nextPageToken to DynamoDB or S3 after each page. When the function is close to its deadline, stop cleanly and let Step Functions re-invoke it from the checkpoint.

Sharding is simpler and I reach for it first. Checkpointing is what you need when a single project is itself too large. Either way, watch Lambda’s ephemeral storage: the default /tmp allocation is 512 MB, and buffering a large Parquet write there will fail before your API calls do. Raise it or stream to S3 in chunks.

Failure family 2: the change history is the whole point, and it truncates silently

Current issue state answers almost none of the interesting questions. How long a ticket spent waiting for review, how many times it bounced back from QA, when it actually entered development rather than when someone remembered to drag the card, all of that lives in the changelog. Without it you have a list of tickets. With it you have a process.

Here is the trap. Requesting an issue with expand=changelog returns a capped number of history entries, commonly the first hundred, and it does not tell you it truncated. Well-worn tickets with lots of field edits blow past that easily. Your data does not error, it just quietly loses the later transitions, which are usually the ones near completion. Cycle time comes out looking better than reality.

Two correct approaches:

• GET /rest/api/3/issue/{issueIdOrKey}/changelog and page it properly. Correct, but it is one request per issue, which is brutal against rate limits on a large site.
• POST /rest/api/3/changelog/bulkfetch, which accepts issueIdsOrKeys and an optional fieldIds filter, with the same nextPageToken pagination. Far fewer round trips.

Bulk fetch is the one I would use, with a caveat: it has carried an experimental designation, so pin your expectations and keep the per-issue path available as a fallback. Filter fieldIds to status if status history is all you model. That cuts the response size enormously, because most changelog volume is description edits and label churn nobody analyses.

Store changelog as its own narrow table, one row per field change, not nested inside the issue record. Athena can handle nested structures, but flat is dramatically easier to reason about in SQL and much cheaper to scan.

Failure family 3: the incremental load that duplicates everything

This is the one that bites hardest, because nothing fails. Everything runs green for weeks and the numbers are wrong the entire time.

The obvious incremental design is a watermark: track the last successful run, then pull updated >= watermark on each run and append the results to S3. It is the standard pattern and it works for immutable event data.

Jira issues are not immutable. A ticket created in January and touched every week appears in every weekly extract. If you append each run into its own partition, that one issue now exists as a row in twenty partitions. Athena is doing exactly what you asked: SELECT count(*) FROM jira_issues WHERE status = 'Open' counts twenty things. Your open bug count is inflated, your throughput is inflated, and the inflation is proportional to how much a ticket gets edited, which correlates with how contentious it was. The busiest work is the most over-counted.

Nobody catches this from the dashboard, because the numbers move in the right direction and look approximately sane. It surfaces months later when someone reconciles against a JQL query in Jira and the two disagree.

Pick one of three fixes, deliberately

Full snapshot per load, dedupe at read time. Keep every version, partition by load_date, and always read through a view that takes the latest row per issue key. Storage is cheap, history is free, and you can answer “what did the board look like in March” without any extra machinery. The cost is that every query pays for the deduplication.

CREATE OR REPLACE VIEW jira_issues_current AS
SELECT * FROM (
  SELECT
    i.*,
    ROW_NUMBER() OVER (
      PARTITION BY issue_key
      ORDER BY load_date DESC, updated DESC
    ) AS rn
  FROM jira_issue_snapshot i
  WHERE load_date >= date_format(current_date - interval '7' day, '%Y-%m-%d')
) WHERE rn = 1;

The load_date filter inside the view matters. Without it the deduplication window scans the entire table on every query, which is the single most common way a cheap Athena setup becomes an expensive one.

Overwrite the affected partitions. Partition by something stable, usually issue created month, and rewrite whole partitions when any issue in them changes. Clean reads, no dedupe cost, but you now own read-modify-write logic in Lambda and a concurrency problem if two runs overlap.

Use an ACID table format. Apache Iceberg gives you real MERGE INTO semantics on S3, and Athena supports it natively. This is the right answer if you are already running Iceberg elsewhere or if the pipeline will grow to a dozen sources. It is not worth adopting solely to load one Jira site, because you inherit compaction and snapshot expiry as ongoing maintenance.

For a single Jira site feeding a handful of dashboards, snapshot plus a dedupe view is what I would build. It has the fewest moving parts and it gives you point-in-time history as a side effect, which you will want the first time someone asks a retrospective question.

Deletions and moves

A JQL watermark query never returns deleted issues, so they persist in your data forever. Same for issues moved out of scope or into an archived project. Periodically reconcile: pull the full set of issue keys with a minimal fields list, compare against what you hold, and mark the difference. Monthly is usually enough. Skip this and your historical counts drift upward permanently.

Failure family 4: the S3 layout that decides your Athena bill

Athena’s standard pricing model bills on bytes scanned, rounded up, with a small per-query minimum. DDL statements are free. That means your storage layout, not your SQL, is what determines cost.

Three things do almost all the work:

• Write Parquet, not JSON. Columnar storage lets Athena read only the columns your query touches. A dashboard selecting five columns from a forty-column table scans a small fraction of what the equivalent JSON would. Snappy or Zstd compression on top of that.
• Avoid tiny files. A Lambda that runs hourly and writes one small file per invocation produces thousands of objects. Athena spends more time opening files than reading them. Batch writes so files land in the low hundreds of megabytes, or run a periodic compaction job.
• Use partition projection. This is the one people skip and then wonder why queries have a fixed few-second overhead.

Partition projection lets Athena calculate partition locations from table properties instead of calling GetPartitions against the Glue Data Catalog. On a table with a couple of years of daily partitions, that lookup is real latency on every single query, and it grows as the table does. Projection removes it, and removes your need for a Glue crawler entirely, which is one less scheduled job and one less IAM role.

CREATE EXTERNAL TABLE jira_issue_snapshot (
  issue_id        string,
  issue_key       string,
  project_key     string,
  issue_type      string,
  status          string,
  status_category string,
  assignee_id     string,
  created         timestamp,
  updated         timestamp,
  resolutiondate  timestamp,
  story_points    double
)
PARTITIONED BY (load_date string)
STORED AS PARQUET
LOCATION 's3://example-jira-lake/issue_snapshot/'
TBLPROPERTIES (
  'projection.enabled'                = 'true',
  'projection.load_date.type'         = 'date',
  'projection.load_date.format'       = 'yyyy-MM-dd',
  'projection.load_date.range'        = 'NOW-3YEARS,NOW',
  'projection.load_date.interval'     = '1',
  'projection.load_date.interval.unit'= 'DAYS',
  'storage.location.template'         =
    's3://example-jira-lake/issue_snapshot/load_date=${load_date}'
);

One caution that surprises people: projection describes partitions Athena will look for, not partitions that exist. Set a range wider than your data and queries without a load_date filter will probe empty prefixes. Keep the range tight to what you actually hold.

Finally, set BytesScannedCutoffPerQuery on the Athena workgroup. It kills any query that exceeds a scan threshold. One analyst running SELECT * against three years of data in a BI tool’s preview pane is the classic surprise line item, and this stops it at the source.

Modelling time in status

Once status changelog rows are landed flat, the core metric is a window function. Each transition’s duration is the gap to the next transition on the same issue:

WITH transitions AS (
  SELECT
    issue_key,
    to_status,
    changed_at,
    LEAD(changed_at) OVER (
      PARTITION BY issue_key ORDER BY changed_at
    ) AS next_changed_at
  FROM jira_changelog
  WHERE field_id = 'status'
    AND load_date >= date_format(current_date - interval '90' day, '%Y-%m-%d')
)
SELECT
  issue_key,
  to_status,
  SUM(date_diff('second', changed_at,
                COALESCE(next_changed_at, current_timestamp))) / 3600.0
    AS hours_in_status
FROM transitions
GROUP BY issue_key, to_status;

The COALESCE handles the current status, which has no successor transition. Two modelling decisions to make consciously: whether to subtract non-working hours, and how to treat a ticket that moves backwards through the workflow. Both are business questions, not technical ones, and both should be settled in a documented view rather than reimplemented in each dashboard.

Map raw status names to a stable category early. Teams rename statuses constantly, and a metric keyed on the literal string breaks the moment someone changes “In Review” to “Peer Review”. A small mapping table joined at query time keeps history comparable.

Troubleshooting

• Every analytical column is null. You did not pass fields, or you passed a custom field ID from another site. Log the resolved field list on every run.
• Counts higher than the same JQL in Jira. Duplicate rows across partitions. Check whether your query goes through the dedupe view or straight at the base table.
• Extraction loops forever. Do not terminate on isLast. Terminate on missing nextPageToken and keep a page cap.
• HTTP 410 from the API. You are still calling the removed /rest/api/3/search. Check your client library version too, not just your own code.
• Cycle times suspiciously low. Truncated changelog. Move to the dedicated changelog endpoint or bulk fetch.
• Athena returns zero rows but the files are there. Partition projection range does not cover the partition, or the S3 prefix does not match storage.location.template. Compare a real object key against the template character by character.
• Queries slow down as the table grows. Either you have no projection and Glue lookups dominate, or you have accumulated small files.
• Authentication failures overnight with no deploy. API token expired or was revoked. Alert on the specific status code rather than on “run failed”.

Common mistakes

• Appending incremental extracts without a deduplication strategy.
• Building on current issue state and adding changelog later. Retrofitting history is far more work than including it from the start.
• Hardcoding custom field IDs copied from documentation.
• Storing the API token in a Lambda environment variable instead of Secrets Manager.
• Running an hourly schedule when the dashboard is read once a day. You pay in small files and API quota for freshness nobody uses.
• Letting BI tools query base tables directly instead of curated views.
• Never reconciling against Jira. A scheduled check comparing approximate count to your row count catches drift within a day.

Best practices for a Jira analytics pipeline that survives

• Land raw API responses to S3 before transforming. When your parsing is wrong, and it will be, you replay from raw rather than re-hammering the API.
• Make the load idempotent. Re-running for the same date should produce the same result, not a second copy.
• Define the pipeline in Terraform or CloudFormation and deploy it from CI. The IAM policy for Lambda and Athena is fiddly and you do not want to rebuild it from memory.
• Alert on a run that succeeds with zero rows, not just on runs that error. Silent empty loads are the more common failure.
• Scope the Jira token to a service account with read access to exactly the projects you need.
• Put an S3 lifecycle policy on the raw zone. It grows faster than you expect and nobody queries last year’s raw JSON.
• Expose curated views, not tables, to Grafana, Metabase, Power BI or QuickSight, and enable Athena’s query result reuse for repeated dashboard loads.

Frequently asked questions

Should I use Lambda or Glue for Jira extraction?

Lambda, for an API-driven pull. The work is mostly waiting on HTTP responses, which Spark’s distributed compute does nothing for, and you would be paying for a Glue job’s minimum billing on something that is idle. Glue earns its place downstream, if you have heavy joins across several sources. Athena CTAS often covers that too.

How often should the pipeline run?

Match the decision cadence, not the data cadence. Delivery metrics are reviewed weekly or in sprint ceremonies, so daily is almost always enough and produces far better file sizes. Reserve hourly for something with a real-time consumer, and know that you are buying that freshness with small-file overhead and API quota.

Can I use webhooks instead of polling?

You can, and for near-real-time reaction it is the right tool. For analytics it is a poor primary source, because a missed delivery leaves a permanent hole you have no way to detect. The pattern that works is webhooks for freshness plus a scheduled reconciliation pull as the source of truth. If you only build one, build the scheduled pull.

Does this work with Jira Data Center or Server?

The AWS half is identical. The extraction half is not. Cursor pagination and the /search/jql endpoint are Cloud-only changes, so self-hosted instances still use the older offset-based /rest/api/2/search. Write the extractor behind an interface if you need to support both, and expect the auth model to differ as well.

How much does an Athena-based Jira pipeline cost to run?

The mechanism matters more than any figure I could quote. You pay for S3 storage, Lambda invocation time, and Athena per byte scanned with a small per-query minimum. Jira issue data compresses extremely well as Parquet, so storage stays small. The variable is dashboard query volume, which is why partitioning and workgroup scan limits matter. Model it against current AWS rates and your own expected query count.

Why not just use eazyBI or a Jira reporting app?

Often you should. If your questions stay inside Jira, a Marketplace app gets you there in an afternoon with no infrastructure. The case for a pipeline is joining Jira to data that lives elsewhere, retaining history the app does not, or needing your data in a warehouse you already run. Build the pipeline when the app has actually failed you, not in anticipation.

Should I load into Redshift instead of querying S3?

Only if you have concurrent BI users hitting the same tables constantly and Athena’s per-query latency is a real complaint. Jira data volumes are small, the query pattern is bursty, and Athena’s zero-idle-cost model fits that far better. Redshift makes sense as a consolidation layer across many sources, not for one issue tracker.

The one thing to carry away

The hard part of a Jira analytics pipeline is not the AWS wiring. Lambda, S3, Glue and Athena will be working within a day. The hard part is that Jira issues are mutable, so an incremental load that appends is an incremental load that duplicates, and it does so without a single error in your logs.

Decide your deduplication strategy before you write the first extract, not after someone reconciles a dashboard against JQL and finds a gap. Snapshot with a read-time dedupe view, partition projection so queries stay fast as history accumulates, and a scheduled reconciliation check that alerts on drift. Get those three right and the rest is plumbing.

Need this built or fixed?

I design and run serverless data pipelines on AWS, and Jira extraction has more sharp edges than most sources. Things I can help with directly:

• Migrating an extractor off the removed /rest/api/3/search endpoint to cursor-based /search/jql without losing rows in the cutover
• Auditing an existing Jira pipeline for duplicate rows and reconciling your numbers back against JQL
• Building changelog-based cycle time and time-in-status models that survive workflow renames
• Restructuring an S3 layer with Parquet, partition projection and file compaction to cut Athena scan costs
• Packaging the whole thing as Terraform or CloudFormation with CI deployment and least-privilege IAM
• Connecting Athena to Grafana, Power BI, Metabase or QuickSight with curated views instead of raw tables

Send me your extractor code, a Glue table definition, or an Athena query that is scanning more than it should, and I will tell you what I would change.

The post Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything) appeared first on John Nessime.

That is where embedded analytics on AWS stops being a front-end task and becomes an architecture decision. The awkward part is that the choice you make first, how the viewer is identified, quietly decides which isolation mechanisms remain available to you afterwards. Get that order backwards and you rebuild the data layer, not the iframe.

This post covers the four decisions that determine whether the build holds: identity model, tenant isolation, query mode, and session economics. Then the embed handshake itself, the errors you will actually see in the browser console, and what to check first when it fails.

Before anything else: the product got renamed

Amazon QuickSight was folded into a broader platform called Amazon Quick Suite, and the BI product inside it is now called Amazon Quick Sight. AWS documentation has since moved again under an “Amazon Quick” umbrella. You will land on all three naming conventions depending on which search result you click, which makes finding the right doc page genuinely annoying.

The practical upshot: the APIs, SDKs and IAM action names did not change. You are still calling quicksight:GenerateEmbedUrlForRegisteredUser against ARNs in the quicksight namespace, and the JavaScript SDK is still published as amazon-quicksight-embedding-sdk. Nothing in your code breaks. Only your bookmarks do. I mention it because half the confusion in a first embedded build comes from following a doc page that describes a UI menu that has since been reorganised.

Decision one: registered users or anonymous sessions

Two API operations generate embed URLs. GenerateEmbedUrlForRegisteredUser issues a session for a user who exists inside the BI account. GenerateEmbedUrlForAnonymousUser issues a session for someone who does not, and never will.

This reads like a convenience choice. It is not. Row-level security using session tags, the mechanism most SaaS products want, is supported only for anonymous embedding. It does not work with GenerateEmbedUrlForRegisteredUser, it does not work with the older GetDashboardEmbedUrl operation, and it is not supported with the IAM identity type. That constraint is documented, easy to miss, and it is the single most expensive thing to discover late.

So the fork is really this. If you register every viewer, you get per-user features (bookmarks, threshold alerts, scheduled snapshots) and you enforce isolation with username or group rules on the dataset. You also inherit the job of provisioning, deprovisioning and reconciling a user directory that mirrors your own. If you go anonymous, you skip all of that and filter with session tags at embed time, but per-user features are off the table because there is no persistent user to hang them on.

The registered-user request body is small. Everything interesting is in ExperienceConfiguration:

POST /accounts/<aws-account-id>/embed-url/registered-user

{
  "UserArn": "arn:aws:quicksight:<region>:<account>:user/default/<user>",
  "SessionLifetimeInMinutes": 60,
  "AllowedDomains": ["https://app.example.com"],
  "ExperienceConfiguration": {
    "Dashboard": {
      "InitialDashboardId": "<dashboard-id>",
      "FeatureConfigurations": {
        "Bookmarks": { "Enabled": true }
      }
    }
  }
}

One trap on the anonymous path that deserves its own sentence. Anonymous sessions belong to a namespace, and any dashboard shared with that namespace is reachable by a session in it, whether or not you listed the dashboard in AuthorizedResourceArns. If you were treating that parameter as your allowlist, it is not. Namespace membership is the real boundary.

Decision two: where tenant isolation actually lives

There are three places you can put the filter, and only one of them scales.

• A dashboard per tenant. Works for five customers. Becomes a deployment problem at fifty and a change-management disaster at five hundred, because every visual fix is now a fan-out.
• A dataset per tenant, filtered in SQL. Better isolation guarantees, genuinely defensible in a compliance review, but you multiply refresh jobs and in-memory footprint by tenant count.
• One dashboard, one dataset, row-level security. The standard answer. One artifact to maintain, filtering applied per session.

With anonymous embedding, RLS is driven by tags. You declare tag keys against columns on the dataset, then supply values at embed time. The filter is evaluated server-side against the session, so a viewer poking at the iframe cannot lift it.

POST /accounts/<aws-account-id>/embed-url/anonymous-user

{
  "Namespace": "default",
  "SessionLifetimeInMinutes": 60,
  "AuthorizedResourceArns": [
    "arn:aws:quicksight:<region>:<account>:dashboard/<dashboard-id>"
  ],
  "SessionTags": [
    { "Key": "tenant_id", "Value": "acme-corp" },
    { "Key": "region",    "Value": "emea" }
  ],
  "AllowedDomains": ["https://app.example.com"],
  "ExperienceConfiguration": {
    "Dashboard": { "InitialDashboardId": "<dashboard-id>" }
  }
}

The value in SessionTags must come from your server-side session, never from a request parameter, a cookie your client can write, or a JWT claim you have not verified. This is the whole security boundary. Tag rules support combining conditions, so a manager who should see several sites is expressible without a second dashboard.

One quiet limit worth knowing before it bites: when RLS is applied to in-memory datasets, each field has a maximum length in Unicode characters, and fields exceeding it are truncated during ingestion rather than rejected. If your tenant identifiers are long opaque strings, test that a truncated value cannot collide with another tenant’s. Silent truncation plus a prefix collision is exactly the kind of bug that produces a cross-tenant data leak with no error anywhere in the logs.

Decision three: SPICE or direct query against your AWS data

Every dataset runs in one of two modes, and the difference shows up on a bill somewhere else in your account.

Direct query sends a live query to the source each time a visual renders. Against Amazon Athena that means an S3 scan per dashboard open, billed by bytes scanned. Against Amazon Redshift it means a concurrent query slot per viewer. Freshness is perfect. The failure mode is that dashboard load is now coupled to warehouse load, and your analytics traffic competes with everything else running there. Two hundred people opening a dashboard at 9am is two hundred queries, and Redshift concurrency is finite.

SPICE imports a snapshot into an in-memory engine and serves every viewer from it. One scan on refresh, then arbitrarily many reads. For an embedded product where the same aggregate is served to thousands of sessions, this is usually the right call, and the Athena cost difference between “scan once per refresh” and “scan once per pageview” is not subtle. What you give up is freshness, bounded by your refresh schedule, plus a capacity dimension to manage and incremental refresh to configure if the dataset is large.

The pattern I reach for first on a data-lake backend is a hybrid: recent partitions in SPICE with an incremental refresh on a look-back window, historical data left on direct query for the rare deep query. It costs more design effort up front and it is the thing most teams skip, but it is the only shape that keeps both the bill and the load time flat as history grows.

Whichever you pick, note that visual generation has a timeout, and data-source-specific timeouts apply on top of it. A query that is merely slow in a console tab renders as a broken visual in a customer’s browser. Model your worst partition, not your average one.

Decision four: what a session actually costs

I am not going to quote figures, because AWS changes them and you should read the current pricing page. The mechanism is what matters, and it is genuinely different from seat-based BI licensing.

• A reader session is a fixed 30-minute window. Not a pageview, not a query. Reopening the dashboard twenty minutes later is still the same session.
• Per-user pricing charges per session with a monthly cap per reader. Predictable when the same people return daily.
• Capacity pricing buys sessions in bulk with no user provisioning at all. This is the model built for embedding, and it is the one that pairs with anonymous sessions.
• Capacity pricing is also the prerequisite for programmatic dashboard refresh, so if near-real-time rendering is a product requirement, that decision is already made for you.
• Annual commitments to capacity unlock removing the “Powered by” attribution footer. If white-labelling is a contractual requirement, factor that in early rather than discovering it during a customer demo.
• Enabling certain Pro-tier and generative Q&A capabilities triggers an account-level monthly infrastructure fee that exists whether or not anyone uses the feature.

The cost failure mode nobody plans for is architectural rather than commercial. If you embed the dashboard on a tab that loads by default, you bill a session for every user who lands on that page and looks at something else. Lazy-load the iframe on explicit interaction. That one change is often the largest single lever on the bill, and it costs an afternoon.

The embed handshake, and the three things that break it

The flow is short. Your backend authenticates the user with your own identity system, calls the embed URL API with the right tags or user ARN, returns the URL to the browser, and the SDK mounts an iframe against it.

1. The generated URL carries a temporary bearer token valid for five minutes, and it is single use once redeemed. Generate it per page load from your backend. Never cache it, never put it in a build artifact, never log it.
2. Session lifetime is separate from URL validity, set with SessionLifetimeInMinutes, and ranges from fifteen minutes to ten hours with ten hours as the default. Ten hours is almost never what you want for a customer-facing product. Match it to your own session, or shorter.
3. Domains must be allowed explicitly. An administrator configures static domains in the admin menu, and AllowedDomains on the API call can override that with up to three domains or subdomains per request. Add an AllowedEmbeddingDomains condition to the IAM policy of the calling role, or any developer with that permission can list any domain on the internet.

On the browser side, the v2 SDK creates an embedding context (which appends its own zero-pixel iframe to body for message passing) and then mounts the experience:

import { createEmbeddingContext } from 'amazon-quicksight-embedding-sdk';

const context = await createEmbeddingContext();

await context.embedDashboard(
  {
    url: embedUrl,                        // fetched from your backend, just now
    container: '#analytics',
    height: '600px',                      // acts as loading height below
    resizeHeightOnSizeChangedEvent: true,
  },
  {
    toolbarOptions: { export: false, undoRedo: false, reset: false },
    attributionOptions: { overlayContent: true },
    onMessage: async (event) => {
      if (event.eventName === 'ERROR_OCCURRED') {
        console.error(event.message.errorCode);
      }
    },
  }
);

Two details in there earn their place. resizeHeightOnSizeChangedEvent turns the height value into a loading placeholder and lets the frame grow to fit content, which is what stops the dashboard rendering into a 600px letterbox with an inner scrollbar. And overlayContent tells the layout to overlay the attribution footer rather than reserve extra height at the bottom for it.

Troubleshooting embedded analytics on AWS

Almost every failure lands in one of these. Read the error code out of the ERROR_OCCURRED message before doing anything else.

• Forbidden means the URL’s authentication code expired. You held the URL longer than five minutes, or you served it from a cache, or a retry redeemed it twice. Fix the generation path, not the permissions.
• Unauthorized means the session obtained from that code expired. Different problem, different fix: your SessionLifetimeInMinutes is shorter than how long people keep the tab open. Handle it by re-fetching a fresh URL and re-mounting rather than letting the frame sit there dead.
• Frame never appears at all. Check the onChange handler for NO_CONTAINER or INVALID_CONTAINER, which usually means you mounted before your target element existed, and for INVALID_URL, which means the URL shape does not match the experience method you called.
• Frame appears, dashboard does not. Nine times out of ten this is the domain allowlist. The request domain has to match what was allowed, including scheme and any subdomain, and a staging hostname that nobody added is the usual culprit.
• Modals render off-screen. A known consequence of auto-resizing height: an export dialog can open above the visible viewport. Listen for MODAL_OPENED and scroll the parent page to the frame position.
• Toolbar features silently missing. Bookmarks, threshold alerts and scheduling require both the SDK toolbar flag and the matching entry under FeatureConfigurations in the embed URL request, and they only exist on the registered-user path. Setting the client flag alone does nothing.
• First render is slow, later ones are fine. Direct query against a cold warehouse. Compare the same query in Athena or Redshift directly to confirm before blaming the BI layer.

Common mistakes

• Choosing registered-user embedding for the identity story, then discovering session-tag RLS is unavailable on that path.
• Treating AuthorizedResourceArns as the security boundary instead of namespace membership.
• Deriving a session tag value from anything the client can influence.
• Generating the embed URL at build time, or caching it in a CDN, and then not understanding the Forbidden errors.
• Leaving session lifetime at the ten-hour default in a customer-facing app.
• Putting the dashboard on a default-loaded tab and paying for sessions nobody asked for.
• Building the first version on direct query against Athena because it is quicker to wire up, then meeting the scan bill.
• Forgetting that embedding and row-level security sit in the Enterprise tier, so a Standard-tier proof of concept proves nothing.

Best practices

• Decide the identity model before you build a single dataset. Everything downstream inherits it.
• Put the embed URL call behind one server-side endpoint that reads tenant scope from your own session and nowhere else. One function, one place to audit.
• Constrain the calling IAM role with an AllowedEmbeddingDomains condition and scope resources to specific namespaces rather than a wildcard.
• Write an automated test that requests tenant A’s embed URL and asserts tenant B’s rows are absent. Run it on every dataset change, because RLS breaks silently.
• Default to SPICE with a refresh schedule matched to a stated freshness SLA, and only reach for direct query where the SLA genuinely demands it.
• Track refresh failures as a first-class alert in CloudWatch or whatever you already run, whether that is Grafana, Datadog or something in-house. A stale dashboard that still renders is worse than one that errors, because nobody notices.
• Lazy-load the iframe on user intent, not on page mount.
• Keep the embedded surface read-only unless authoring is a real product requirement. Console embedding is a much larger permissions surface than dashboard embedding.

Is managed BI even the right call?

Worth asking honestly, because the answer is not always yes. The case for the AWS-native route is real: no connector layer to maintain against Athena, Redshift, S3 and Aurora, IAM you already understand, and a usage-based cost model that beats per-seat licensing when your viewers are bursty. If most of your data already sits in AWS, that adds up.

The case against is equally real. Visual customisation is limited compared to charting directly against your own API, the attribution footer needs a commitment to remove, and if you want full control of the front end you may be better served by Apache Superset or Metabase self-hosted, or by Grafana where the workload is closer to operational metrics than customer-facing BI. Those come with an operational burden you now own. That is the trade: you either run the BI layer or you rent it, and renting it means living inside its constraints.

Frequently asked questions

Do my users need AWS accounts to view an embedded dashboard?

No. With anonymous embedding they need no AWS account and no BI user record at all. Your application authenticates them however you already do, and your backend maps that identity to session tags when it requests the embed URL.

Can I use row-level security with registered-user embedding?

Yes, but only with username or group based rules, not with session tags. Tag-based RLS is restricted to the anonymous embedding path. If you need tags, you need anonymous sessions.

How long does an embed URL stay valid?

The URL itself carries a bearer token valid for five minutes and usable once. The session it opens is separate and lasts between fifteen minutes and ten hours depending on SessionLifetimeInMinutes, defaulting to ten hours.

Should I use SPICE or direct query for embedded analytics on AWS?

SPICE for anything with many viewers per refresh, which describes most embedded products. Direct query where the data must be current to the second, or where the dataset exceeds what you want to hold in memory. A hybrid split by data age is often the right answer and is under-used.

Why do I get a Forbidden error when the dashboard worked yesterday?

Forbidden points at the URL, not at permissions. The most common causes are caching the URL, generating it more than five minutes before use, or a client retry redeeming the same single-use token twice. If it is Unauthorized instead, the session expired and you need a fresh URL.

Can I white-label the embedded dashboard completely?

Largely. Themes control colours and typography, the SDK hides toolbar controls, and parameters let your own UI drive the dashboard. Removing the attribution footer entirely is tied to an annual capacity commitment, so confirm that against current terms before you promise it to a customer.

Does natural-language querying work in an embedded context?

Yes. The SDK exposes a generative Q&A experience alongside dashboards and visuals, driven by curated topics rather than raw tables. It is billed on its own capacity dimension and gates behind the Pro tiers, so treat it as a separate cost decision rather than a free addition.

The one thing worth remembering

Embedded analytics on AWS is not a rendering problem. The iframe is the last five percent. The part that decides whether the build survives contact with a second customer is the identity model, because it silently determines which isolation mechanism you are allowed to use, and that in turn shapes your dataset design, your refresh strategy and your bill.

Pick that first. Write the cross-tenant test before you write the dashboard. Everything else is recoverable in an afternoon.

Need help with an embedded analytics build?

This is the kind of work I do. Things I can help with directly:

• Reviewing an existing embed integration for cross-tenant leakage, including the session-tag path and the namespace boundary.
• Designing the identity and row-level security model before you commit to a dataset layout.
• Cutting Athena scan and Redshift concurrency cost by moving the right datasets into SPICE with incremental refresh.
• Building the backend embed-URL service with scoped IAM roles, domain conditions and sane session lifetimes.
• Setting up refresh failure alerting so a stale dashboard does not quietly serve last week’s numbers.
• Automated cross-tenant isolation tests wired into CI, so an RLS regression fails the build instead of the customer.

Send me the actual thing: your embed URL request payload with secrets stripped, the browser console error, or the dataset RLS rules. It is much faster to reason about a real payload than a description of one.

The post Embedded Analytics on AWS: The Four Decisions That Bite Later appeared first on John Nessime.

Acme has nineteen. The assistant retrieved five chunks that happened to mention Acme, because five is roughly how many chunks a retrieval returns, and then it counted them. There is no bug. Nothing failed. Similarity search returned the most similar things and the model described what it was given.

That is the defining problem with preparing CRM data for Amazon Bedrock, and it is upstream of chunk sizes and embedding models. CRM data is structured. Retrieval-augmented generation is built for unstructured text. Feed a table of opportunities through an embedding pipeline and you get a system that is genuinely good at “what did we discuss with this account” and quietly terrible at “how many”, “how much” and “since when”.

This covers the split that fixes it, what a good document actually looks like, the permissions problem nobody notices until it is a problem, and how to know whether any of it works.

Route the question before you build anything

Your users will ask two kinds of question and they need two different mechanisms.

• Narrative questions. “What were the objections on the Acme renewal?” “Summarise our history with this account.” “What did the customer say about pricing?” These live in notes, emails, call summaries and descriptions. Semantic retrieval is exactly right for them.
• Analytical questions. “How many open deals over fifty thousand?” “What is total pipeline this quarter?” “Which accounts have had no activity in ninety days?” These need aggregation, filtering and joins. Retrieval cannot do them and will not tell you it cannot.

The useful part is that you do not have to build the second half yourself. Bedrock Knowledge Bases supports structured data retrieval: you point it at Amazon Redshift or a Glue Data Catalog, and it generates and executes SQL from natural language against your data in place, with no copying or embedding. The counting question goes to SQL, the narrative question goes to the vector store, and both answers are right for the right reasons.

So the first architectural decision is not which chunking strategy to use. It is: which of your fields are text worth embedding, and which are facts worth querying. Get that wrong and no amount of tuning downstream will save it.

One practical note if you take the structured route. Bedrock’s execution role authenticates to Redshift as an IAM identity, and you grant it access the same way you would any other reader. Point it at curated views rather than base tables, exactly as you would a BI tool:

-- Give the knowledge base the same narrow access you'd give a
-- reporting user. A view is a contract; a schema grant is not.
GRANT USAGE ON SCHEMA analytics
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

GRANT SELECT ON analytics.v_opportunities
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

Named columns and clear table names matter more here than anywhere else, because the model is reading your schema to write the SQL. A column called flag_2 will produce exactly the query quality it deserves.

What a good document looks like

For the narrative half, the instinct is to export records to CSV and let the chunker deal with it. That produces chunks that are half a row of one record and half a row of another, with no context about which is which.

Build documents deliberately instead. Three rules that do most of the work:

One record, one document. An opportunity and its notes become a single file. A chunk from it is then always about one thing.

Denormalise the context in. A note that says “they want a two-year term” is useless in isolation. Embed the account name, the opportunity name, the stage and the owner into the same document, so a chunk carries enough context to be retrieved and to be understood once retrieved.

Write it as prose, not as fields. Embedding models were trained on language. Stage__c: Negotiation embeds worse than “The opportunity is at the Negotiation stage.” This feels silly and it measurably improves retrieval.

On chunking itself: start with the default strategy and change it only when evaluation tells you to. If your documents are one record each and reasonably short, semantic or hierarchical chunking buys you less than people expect. The gains that matter come from document construction and metadata, not from chunk size.

The permissions problem

This is the one that turns a nice project into an incident, and it is easy to miss because nothing about it looks broken.

Salesforce has an elaborate sharing model: role hierarchies, sharing rules, territory management, field-level security. A rep sees their accounts and not everyone else’s. Your vector store has none of that. Once a record is embedded, it is a vector like any other, and anyone who can query the knowledge base can retrieve it.

So a rep asks about a competitor deal they were never on the team for, and the assistant helpfully summarises it. Or someone asks about compensation-adjacent records and gets an answer. No alert fires, because from the system’s perspective retrieval worked perfectly.

Three ways out, in increasing order of effort:

1. Only embed what everyone may see. Blunt, easy, and often correct for a first version. Exclude anything sensitive and be explicit that the assistant does not know about it.
2. Filter at retrieval time using metadata. Attach owner, team or region to every document, and pass a filter with each query derived from the caller’s identity. This is the practical answer for most teams.
3. Separate knowledge bases per audience. Heavier to operate, and the only option that gives you a hard boundary rather than a filter you have to remember to apply.

Whichever you pick, decide it before ingestion, because metadata has to be configured at ingest time. Retrofitting a filter you did not plan for means rebuilding the index.

Metadata is most of the value

Metadata filtering narrows the candidate set before similarity search runs. That improves relevance, reduces tokens in the prompt, and gives you the access control lever above. For S3 sources it lives in a sidecar file named after the object it describes:

// acme-renewal.txt.metadata.json, alongside acme-renewal.txt in S3
{
  "metadataAttributes": {
    "object_type": "Opportunity",
    "owner_id": "0051t00000XYZ",
    "account_id": "0011t00000ABC",
    "region": "EMEA",
    "is_closed": false,
    "last_modified_epoch": 1750000000
  }
}

Include, at minimum: what kind of record it is, who owns it, which account it belongs to, whether it is still open, and when it last changed. Those five turn “search everything” into “search the open opportunities this person owns, changed recently”, which is a different product.

Freshness, and the vectors that outlive the record

An embedding is a copy. When the source record changes, the copy is stale. When the source record is deleted, the copy is not.

The consequences are worse in a CRM than in a document store, because CRM records change constantly and confidently answering with last quarter’s stage is indistinguishable from answering correctly. An assistant that cites a deal which closed-lost in March, or a contact who left the company, is not obviously wrong to the person reading it.

• Sync on a schedule that matches how the data is used. Daily is fine for account summaries and useless for live pipeline questions, which should be going to SQL anyway.
• Handle deletes explicitly. Whatever removes a record from your lake must also remove its document from the source bucket, or the vector survives.
• Put the record’s last-modified timestamp in the metadata and surface it in the answer. “As of three days ago” is a small change that prevents a large class of misunderstanding.
• Reconcile document count against source record count periodically. Divergence means orphans.

What not to embed

CRM free-text fields are where people put things. Notes fields contain personal details, occasionally payment information, opinions about customers that were never meant to leave a private conversation, and years of accumulated noise.

Once embedded, all of it is retrievable by anyone with query access, and a vector is not something you can easily grep for a mistake.

• Run detection over free-text fields before ingestion and redact rather than hope.
• Exclude fields nobody asks questions about. Every extra field is noise competing for retrieval slots.
• Consider excluding records closed long ago; they dilute results and rarely answer anything.
• Add Bedrock Guardrails on the output side as a second layer, not as your only control.
• Keep the whole thing on private networking with VPC endpoints if the data warrants it.

There is a governance question here too, and it is worth raising before someone else does: your customers’ personal data is now in a vector store feeding a language model. Whether your privacy notice covers that is not an engineering decision, but it is an engineering responsibility to ask.

Evaluate it before you show anyone

A demo proves the pipeline runs. It does not tell you whether the answers are right, and this is a system whose failure mode is being wrong fluently.

Write thirty real questions from the people who will use it, with correct answers you have verified by hand. Include the analytical ones specifically, because those are where retrieval fails silently. Then run the set after every meaningful change to chunking, metadata or the document format, and compare.

Use the Retrieve API rather than RetrieveAndGenerate while you are tuning. Seeing which chunks came back tells you whether a bad answer is a retrieval problem or a generation problem, and those have completely different fixes.

Common mistakes

• Embedding structured records and expecting aggregation to work.
• Exporting to CSV and letting the chunker split rows arbitrarily.
• Field-value dumps instead of prose, so embeddings have little language to work with.
• No metadata, so every query searches everything and access control is impossible.
• Assuming Salesforce sharing rules follow the data into the vector store.
• Deciding on filtering after ingestion, then discovering it has to be configured at ingest.
• Syncing changes but never removing documents for deleted records.
• Embedding every field because it was easier than choosing.
• No PII detection over free-text notes.
• Tuning chunk size before fixing document construction.
• Judging the system on a demo instead of a scored question set.
• Granting the execution role a whole schema rather than specific views.

Best practices

• Route analytical questions to structured retrieval and narrative questions to the vector store.
• One record per document, with context denormalised in, written as prose.
• Metadata on every document: type, owner, account, open state, last modified.
• Decide the access model before ingestion, and filter at retrieval time by the caller’s identity.
• Redact free-text fields before embedding, and exclude fields nobody queries.
• Delete documents when source records are deleted, and reconcile the counts.
• Surface record age in answers so staleness is visible rather than implied.
• Expose curated views to the execution role, never whole schemas.
• Meaningful table and column names, because the model reads your schema to write SQL.
• A scored evaluation set, run on every change, including questions you expect it to fail.

FAQ

Why does it get counts wrong?

Because retrieval returns the most similar chunks, not all matching records, and the model counts what it was handed. Counting is a query, not a similarity search. Send those questions to structured retrieval over your warehouse instead.

Should I use the Salesforce connector or export to S3 first?

The connector is quicker to stand up. Going via S3 gives you control over document construction, metadata and redaction, which is where most of the quality lives. Start with the connector to learn what people ask, then move to S3 once you know what the documents should look like.

Which chunking strategy is best?

Usually the default, honestly. If each document is one record, chunking has less work to do. Change it in response to evaluation results rather than in advance, because chunking is the most over-discussed and least decisive variable in a CRM RAG setup.

How do I stop people seeing records they shouldn’t?

Metadata filters applied per query based on the caller’s identity, separate knowledge bases per audience, or simply not embedding sensitive records. There is no automatic inheritance of CRM sharing rules, and assuming otherwise is the most consequential mistake in this post.

How often should I re-sync?

Match the questions. Account histories and notes tolerate a daily sync comfortably. Anything needing current numbers should not be coming from embeddings at all. Whatever the interval, make sure deletions propagate.

Do I need a vector database at all?

Only for the narrative half. If every question your users ask is analytical, structured retrieval over Redshift answers all of them with no embedding pipeline, no vector store and no staleness. Plenty of CRM assistants would be better products if they had noticed that.

The one thing to remember

The model will answer either way. It will answer the narrative question well and the counting question badly, in the same tone, with the same citations, and nothing in the response distinguishes them.

So the preparation work is mostly triage: decide which questions are retrieval and which are queries, build documents that carry their own context, attach metadata before you ingest rather than after, and test with real questions whose answers you already know. Chunk size is the last thing to worry about and the first thing everybody argues about.

Building one of these?

The pipeline is usually the easy part; the decisions before it are where these projects succeed or quietly disappoint. Work I take on:

• Designing the split between semantic retrieval and structured querying, so counting questions get correct answers.
• Building the document preparation pipeline from CRM data: record-level documents, denormalised context, metadata, redaction.
• Access control design for knowledge bases, including per-caller metadata filtering and audience separation.
• Freshness and deletion handling so the index does not diverge from the source.
• Curated Redshift views and IAM grants for structured data retrieval.
• Evaluation harnesses with scored question sets, run on every change rather than once before launch.

Send me ten questions your users would actually ask, and I will tell you which of them retrieval can answer.

The post It Counted the Chunks: Preparing CRM Data for Amazon Bedrock appeared first on John Nessime.

The answer is in there. It is one email, or one line in a site diary, sitting among forty thousand other files. Nobody can read forty thousand files, so the instinct is to point a language model at the pile and ask it. That instinct is right. The naive implementation of it is where the money goes.

Here is the failure mode that bites hardest, and it is invisible until an expert challenges you on it. You build retrieval over the document set, ask about notice of delay, and the system confidently returns a letter that discusses the delay event in great detail. It is a good letter. It is also dated eleven months after the event, written by the claims consultant during preparation of the claim itself. It scored highest precisely because it was written to argue the point. The contemporaneous notice, the thing you actually needed, was four badly typed lines in a routine progress email that mentioned the word “delay” once.

Semantic similarity has no concept of a deadline. That single gap is the difference between an AI construction claims platform that shortens a disclosure exercise and one that quietly manufactures a wrong answer with a citation attached to it.

This post covers how to build that platform on AWS: how to lay out ingestion, how to make retrieval respect the contractual clock, why schedule data must never go anywhere near your vector index, how to keep privileged material out of the same retrieval path as project records, and which AWS building blocks are actually the current ones now that several of the obvious candidates have been moved to maintenance mode.

What a claims platform actually has to answer

Before any architecture, be honest about the question shapes. They are not all the same problem and they do not all get solved by retrieval.

1. Chronology. What happened, in what order, and on what date was it recorded? This is a retrieval and metadata problem.
2. Entitlement. Which clause applies, and what did it require the parties to do? This is retrieval over the contract plus careful prompting.
3. Causation. Which event moved the critical path, and by how much? This is schedule data and date arithmetic. It is not a language problem at all.
4. Quantum. What did the disruption cost? This is cost and resource data, joined to the events above.

Treat all four as “ask the documents” and you will get fluent nonsense on two of them. The architecture below splits them deliberately.

Failure one: retrieval that finds the best match instead of the first one

Two corpora live in every dispute bundle and they look identical to an embedding model.

• Contemporaneous records. Site diaries, progress emails, minutes, early warnings, RFIs, instructions. Written while the project was running, by people with no idea a dispute was coming.
• Claim-era material. Narratives, expert reports, without-prejudice correspondence, internal analysis. Written afterwards, specifically to be persuasive about the same events.

Claim-era material wins on cosine similarity almost every time, because it is denser in exactly the terms you searched for. If your retriever cannot distinguish them, every answer is contaminated by the argument you were trying to test.

The fix is metadata, applied at ingestion, and it is cheap to get right and expensive to retrofit. Amazon Bedrock Knowledge Bases reads a sidecar file that sits next to each document in S3, named with the full original filename plus .metadata.json. So letter-0421.pdf gets letter-0421.pdf.metadata.json. The naming convention is the only link between them; there is no separate registration step.

{
  "metadataAttributes": {
    "doc_date": 20240314,
    "corpus": "contemporaneous",
    "doc_type": "site_correspondence",
    "matter_id": "matter-0007",
    "date_source": "email_header",
    "privileged": false
  }
}

Look closely at doc_date. It is an integer, not a string, and that is not a style choice. Bedrock Knowledge Bases metadata attributes support STRING, NUMBER, BOOLEAN and STRING_LIST. The range comparison operators, the ones you need to express “on or before the notice deadline”, only apply to NUMBER. Store the date as "2024-03-14" and your filter will not throw an error. It will just quietly match nothing, or match everything, depending on how you wrote it. You will find out weeks later when someone asks why a document they can see in the bundle never appears in results.

With the date as a sortable integer, a query filter can express the contractual window directly.

{
  "andAll": [
    { "equals":              { "key": "corpus",   "value": "contemporaneous" } },
    { "equals":              { "key": "matter_id","value": "matter-0007" } },
    { "greaterThanOrEquals": { "key": "doc_date", "value": 20240301 } },
    { "lessThanOrEquals":    { "key": "doc_date", "value": 20240329 } }
  ]
}

That is the whole trick. You are no longer asking “what is the most relevant document about this delay”. You are asking “what did the parties actually write during the window in which the contract required them to write it”. Those are different questions and only one of them is worth anything in a dispute.

Where the date comes from matters more than the date

Do not use the S3 object timestamp. It records when someone copied a folder, usually years after the fact and identical across ten thousand files. Derive the date from the document itself: the Date: header on an email, the printed date on a letter, the period covered by a diary entry.

Sometimes you cannot, because the scanned undated fax exists in every project archive. Record that honestly with a date_source attribute rather than guessing, and treat unknown-date documents as a separate review pile. An extension of time argument built on an inferred date is an argument you will lose.

Failure two: treating the programme like a document

This one is worse, because the output looks right.

Oracle Primavera P6 exports XER and PMXML files. Asta Powerproject and Microsoft Project have their own formats. XER in particular is a plain text dump of relational tables, so it goes through a text pipeline without complaint. Chunk it, embed it, and you now have vectors representing fragments of a table of activity codes with no relationships attached.

Ask that index how much float activity A1200 had at the March data date and you will get a number. It will be well formatted and it will be invented. Total float is the product of a forward and backward pass across the whole logic network under a specific calendar. It cannot be recovered from a retrieved fragment, and a language model asked to produce it will produce something plausible instead of admitting that.

Schedule data goes into a structured store, and the model queries it rather than reasoning about it.

1. Parse each programme file into tables. PyP6Xer handles XER from Python; MPXJ is a Java library that reads XER, PMXML, Asta Powerproject and MSPDI among others, which matters when the bundle contains four scheduling tools.
2. Load activities, logic links, calendars, resource assignments and WBS into Amazon Aurora PostgreSQL for interactive work, or into S3 with AWS Glue and Amazon Athena when you have hundreds of updates and want columnar scans.
3. Stamp every row with the data date of the update it came from. This is the single most important column in the whole platform. Without it you have a pile of schedules; with it you have a time series of the project’s own view of itself.
4. Run windows analysis, as-planned versus as-built comparison and float erosion in SQL or Python, deterministically, so the same inputs always give the same numbers.
5. Expose the results to the model as a tool it can call, or as generated SQL against a defined schema. The model turns a question into a query and narrates the result. It does not do the arithmetic.

A rough shape of the query that makes float erosion visible:

SELECT
    a.activity_id,
    a.data_date,
    a.total_float_days,
    a.total_float_days - LAG(a.total_float_days)
        OVER (PARTITION BY a.activity_id ORDER BY a.data_date)
      AS float_change
FROM   schedule_activities a
WHERE  a.project_id = 'PRJ-01'
  AND  a.data_date BETWEEN DATE '2024-01-01' AND DATE '2024-06-30'
ORDER BY a.activity_id, a.data_date;

Nothing clever there, and that is the point. Every number is traceable to a row that came from a named XER file. When an opposing expert asks where a figure came from, the answer is a file name and a query, not “the model said so”.

Be realistic about effort here. Programme parsing and normalisation across inconsistent updates is the hardest part of the build and the part clients always underestimate. Activity IDs get reused, calendars change mid-project, and someone will have re-baselined without telling anyone. Budget for it.

Failure three: an answer with no paper trail

In most RAG applications a citation is a nice touch. In dispute work it is the product. An answer that cannot be traced to a page of a disclosed document is not evidence, it is a rumour with good grammar.

Design for that from the ingestion layer, not the presentation layer.

• Keep page and position. Amazon Bedrock Data Automation returns confidence scores and bounding box data alongside extracted fields, and Amazon Textract returns geometry per block. Carry both through the pipeline so a citation resolves to a page and a region, not just a file.
• Route low confidence to humans. Handwritten site diaries and faxed variation orders will produce low-confidence extractions. Those should land in a review queue by default rather than silently entering the index.
• Reject ungrounded answers. Amazon Bedrock Guardrails includes contextual grounding checks that score whether a response is supported by the retrieved passages. It reduces confident invention. It does not eliminate it, and anyone who tells you otherwise is selling something.
• Keep an immutable evidential copy. S3 Versioning plus S3 Object Lock on the landing bucket means the file the platform indexed is provably the file that was disclosed.

One design rule underpins all of it: the platform shortlists evidence, it does not decide entitlement. Recognised frameworks for this work, the Society of Construction Law Delay and Disruption Protocol and AACE International’s Recommended Practice 29R-03 on forensic schedule analysis, both assume a named analyst applying a stated method and exercising judgement. A system that outputs “the contractor is entitled to 42 days” is not helping. A system that outputs “here are the eleven contemporaneous documents inside the notice window, here is the float movement across those updates, here is what is missing” is doing real work.

Failure four: one index for privileged and non-privileged material

Dispute bundles contain legal advice, counsel’s opinions, without-prejudice correspondence and internal settlement analysis. Those must not be retrievable through the same path as project records.

The tempting shortcut is a privileged: false metadata filter on every query. Do not rely on that as your boundary. A metadata filter is a query parameter. One missing filter in one code path, one debug endpoint, one caching layer that drops it, and privileged material surfaces in a general search. The blast radius of that mistake is not a bug report.

Separate the indexes physically and separate the IAM roles that can reach them. Amazon S3 Vectors makes this practical: you can set a dedicated customer-managed KMS key per vector index, and you get a large number of indexes per vector bucket, so per-matter and per-sensitivity separation does not become an operational burden. Keep the metadata flag as well, because defence in depth is free, but make the identity boundary the one you actually trust.

Amazon Macie is worth pointing at the landing bucket to find personal data you did not expect, particularly in HR records and accident reports that get swept into project archives.

Choosing the AWS building blocks, including what not to build on

A lot of published architectures for this kind of platform are now pointing at services AWS has stopped developing. Two matter here, and the dates are the point.

• Amazon Kendra entered maintenance mode on 30 June 2026 and stops accepting new customers on 30 July 2026. Existing customers keep support and security fixes but no new capability. AWS directs new enterprise search and RAG work to Amazon Bedrock Knowledge Bases. If a tutorial or a proposal you are reading starts with a Kendra index, it predates that change.
• Amazon Bedrock Agents moved to maintenance mode in the same round of service availability changes, with Amazon Bedrock AgentCore as the successor for agentic orchestration. Check the current AWS service availability page before you commit an orchestration layer.

For the retrieval layer itself, Bedrock Knowledge Bases now comes in two shapes and the choice is a real trade-off rather than a marketing tier.

Managed Knowledge Base

AWS manages the vector store, embeddings model, re-ranker and retrieval orchestration as a single primitive, with native connectors for Amazon S3, SharePoint, Confluence, Google Drive, OneDrive and a web crawler, plus automatic parsing strategy selection and a retriever that decomposes multi-step queries. The connectors pull source permissions along with content, which matters when the document set lives in the client’s SharePoint rather than a bucket you control.

Where it wins: you get a working retrieval layer in an afternoon instead of a fortnight, and the parsing tuning that normally eats the first weeks of a build is done for you. For a first matter, or a proof of value before a client commits budget, this is the one I would reach for.

Custom Knowledge Base

You bring your own vector store and control chunking, embedding model and index layout.

Where it wins: claims work has awkward chunking requirements. A two-page letter split mid-sentence at a page boundary produces a chunk where the notice sentence has lost its date and its addressee. Controlling chunk boundaries around document structure, and controlling which index a document lands in, are both easier when you own the store. Where it doesn’t: you now own embedding model upgrades, re-indexing, sync failures and capacity, which is real ongoing work for a small team.

Start managed, build a retrieval evaluation set of real questions with known correct documents, and only move to custom when that set demonstrates the problem is chunking. Most teams migrate on a hunch and discover the problem was metadata all along.

Where the vector storage bill actually comes from

Rates change, so learn the billing mechanism rather than a number. Amazon S3 Vectors charges on three axes: upload volume by logical gigabyte, storage by logical gigabyte, and queries by data processed, where data processed scales with the size of the index being searched. Note that filtering does not reduce the data processed by a query.

That shape suits claims work unusually well. A dispute archive is enormous and cold: millions of chunks, queried by a handful of analysts a few hundred times a day, so you pay mostly for storage, which is the cheap axis. Compare that against Amazon OpenSearch Serverless, which prices on provisioned compute units and therefore rewards high query volume against a smaller index, or Aurora PostgreSQL with pgvector when you already need Aurora for the schedule tables and would rather run one system than two.

The practical lever is to split indexes per matter. Query cost scales with index size, so one giant index across every dispute you have ever run makes every query more expensive than it needs to be, on top of being a bad idea for confidentiality.

A reference pipeline

1. Everything lands in S3 under a per-matter prefix, with Versioning and Object Lock enabled on the evidential copy.
2. S3 event notifications trigger AWS Step Functions. Use Step Functions rather than a chain of Lambdas so that a failed extraction on page 300 of a 400-page bundle is visible and resumable.
3. Classify and split. Scanned bundles arrive as one PDF containing forty separate documents. Splitting them correctly is a prerequisite for dating them correctly.
4. Extract text with Amazon Bedrock Data Automation or Amazon Textract, keeping confidence scores and geometry.
5. Derive the document date and write the .metadata.json sidecar. Anything undated goes to the review queue.
6. Route by type: correspondence to the knowledge base, programme files to the XER parser and the relational store, cost data to its own tables.
7. Sync the knowledge base, then run your retrieval evaluation set before anyone uses it. A sync that succeeds is not the same as an index that answers correctly.
8. Serve through an API that refuses to return an answer without citations, and log every query with the filters that were applied.

Define the whole thing in Terraform or OpenTofu from the start. Matters are per-client and short-lived, and standing one up should be a variable file, not an afternoon in the console. Point Amazon CloudWatch, or Grafana Cloud if you already run Grafana elsewhere, at the Step Functions execution metrics so a silently failing extraction stage does not go unnoticed for a week.

Troubleshooting

Date filters return nothing, and no error

Almost always the date was stored as a string. Range operators need NUMBER. Convert to an integer in YYYYMMDD form and re-sync the affected documents.

A document is in the bucket but never appears in results

Check the sidecar filename first. It must be the complete original filename with .metadata.json appended, extension included. report.pdf.metadata.json works; report.metadata.json is a file the ingestion job will happily ignore. After that, check whether a filter in the query path is excluding it.

Answers cite the right document but the wrong passage

Chunking split the document somewhere structurally meaningful. Look at the raw chunks for that file. If the notice sentence and its date are in different chunks, no amount of prompt tuning fixes it. That is the signal to take control of chunking.

Float figures do not match the client’s own analysis

Check calendars before you check logic. Different activity calendars, a changed default calendar, or an update where someone applied a progress override will move float without any logic change. Reconcile activity counts between your parsed tables and the source file before trusting anything downstream.

Query costs jumped without more usage

An index grew. With storage-side vector search, query cost tracks the size of the index being scanned, so ingesting a large new bundle raises the price of every subsequent query against that index. Split by matter.

Common mistakes

• Using the file’s storage timestamp as the document date. It records the migration, not the event.
• Indexing claim narratives and contemporaneous records into the same corpus with no way to tell them apart.
• Embedding programme exports because they happen to be text files.
• Treating a metadata filter as a privilege boundary instead of an optimisation.
• Letting the model state entitlement conclusions rather than assembling and citing evidence.
• Building on services that have moved to maintenance mode because the tutorial you followed predates the change.
• Shipping without a retrieval evaluation set, so you have no way to know whether a change made things better or worse.
• One index for every matter, which is both a cost problem and a confidentiality problem.

Best practices

• Make the document date a first-class, numeric, filterable attribute, and record where it came from.
• Keep an immutable evidential copy separate from the working copy the pipeline mutates.
• Separate structured schedule and cost data from unstructured documents, and let the model query the former rather than reason about it.
• Build a retrieval evaluation set from real questions with known correct documents before you tune anything.
• Enforce citations at the API layer, so an uncited answer is impossible rather than discouraged.
• Isolate privileged material by index and by IAM role, with metadata as a second layer.
• Log every query with its filters, so you can reconstruct how any given answer was reached.
• Define infrastructure as code so a new matter is a deployment, not a project.

FAQ

Can an AI construction claims platform replace a delay expert?

No, and building toward that goal produces something unusable. Established forensic frameworks assume a named analyst applying a stated method whose reasoning can be tested. The platform’s value is compressing weeks of document review into hours and making the schedule data queryable, so the expert spends their time on judgement rather than searching.

Should I use Amazon Kendra for the search layer?

Not for a new build. Kendra entered maintenance mode on 30 June 2026 and closed to new customers on 30 July 2026, with AWS pointing to Bedrock Knowledge Bases for equivalent and more current capability. Existing Kendra deployments continue to be supported, so this is a migration assessment rather than an emergency, but starting there now means starting on a service with no roadmap.

How do I stop the model inventing float and delay figures?

Do not give it the chance. Keep schedule data in a relational or columnar store and have the model generate queries against a defined schema, or call a tool that runs a fixed calculation. The arithmetic happens in SQL or Python where it is deterministic and reproducible; the model only turns questions into queries and results into sentences.

Which vector store should I choose for a claims archive?

Match the store to your query pattern. Large, cold archives queried by a few analysts favour storage-priced options like Amazon S3 Vectors, where you mostly pay to keep the data. Smaller indexes hit constantly favour compute-priced options like Amazon OpenSearch Serverless. If you already run Aurora PostgreSQL for schedule data, pgvector alongside it is a legitimate way to avoid operating a second system.

How do I handle scanned and handwritten site records?

Extract them with confidence scores retained, set a threshold, and route everything below it to human review before indexing. Handwritten diaries are frequently the most probative documents in a delay claim and also the least reliable to read automatically, so the review queue is not an edge case. Plan capacity for it.

Where do documents come from if they are not already in S3?

Most project records live in a common data environment such as Procore, Autodesk Construction Cloud, Aconex or a client SharePoint tenancy. Bedrock Managed Knowledge Base has native connectors for SharePoint, Confluence, Google Drive and OneDrive that ingest permissions alongside content. For platforms without a native connector, export to S3 and keep the export manifest as part of the disclosure record.

The one thing worth remembering

An AI construction claims platform lives or dies on whether it understands time. Every hard requirement in this build traces back to that: numeric dates so you can filter to a contractual window, a data date on every schedule row so float movement is measurable, a corpus flag so contemporaneous records are not drowned out by material written to argue about them, and citations that resolve to a page so any answer can be checked.

Get the temporal metadata right at ingestion and the rest of the architecture is ordinary AWS work. Get it wrong and you have built a very expensive way to retrieve the most persuasive document instead of the true one.

Need help building this on AWS?

I design and build document and data platforms on AWS, and this kind of system sits squarely in that work. Things I can help with:

• Designing the ingestion pipeline: S3 landing zones with Object Lock, Step Functions orchestration, splitting and classifying scanned bundles, and confidence-based routing to human review.
• Getting the temporal metadata model right, including date derivation, sidecar generation and filter design against Amazon Bedrock Knowledge Bases.
• Parsing Primavera P6 XER and PMXML exports into queryable tables in Aurora PostgreSQL or S3 with Glue and Athena, with a data date on every row.
• Choosing and sizing the vector layer across Amazon S3 Vectors, OpenSearch Serverless and pgvector, based on your actual query pattern rather than a benchmark.
• Building index and IAM separation for privileged material, plus KMS key strategy and Macie scanning of landing buckets.
• Setting up retrieval evaluation, citation enforcement, query audit logging and CloudWatch or Grafana dashboards over the pipeline so failures surface early.

If you are partway into something like this already, send me a sample metadata sidecar, a Step Functions execution history, or a query that returns the wrong document, and I will tell you what I think is going on.

The post Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny appeared first on John Nessime.

That is the failure mode worth planning for. Not a jailbreak, not a model saying something clinically wrong on stage. A user typing something perfectly reasonable into a box you built, and the data ending up somewhere you never drew on the diagram. This post walks through how to build a secure AI medical assistant on AWS, organised by the boundary the data actually crosses: the input box, your own logs, the Region the inference runs in, the retrieval index, and finally the answer itself. Commands are included where they explain something. Where a value depends on your account or your counsel, I say so instead of making one up.

Two different products hiding behind one request

“An AI assistant for medical students and doctors” is two builds with two risk profiles, and conflating them is the root of most of the trouble.

• The study tool. Question banks, guideline summaries, differential drills, spaced repetition. In theory it never touches patient data. Its real risk is confident wrongness and unattributed answers, not privacy.
• The clinical assistant. Note summarisation, coding support, chart question answering. It handles PHI by design, so the whole thing has to sit inside a HIPAA-designated account from day one.

The trap is that the study tool becomes the clinical assistant without anyone shipping a release. A student rehearses a case they saw on the ward. A doctor tries the study tool on a real chart because it is the one that is already open. The moment your input box accepts free text from someone with clinical access, you should assume PHI will arrive in it. Build accordingly, or put the study tool on infrastructure where PHI arriving is survivable.

My default is to run both in the same HIPAA-designated account with the same controls, and keep only the marketing site, the docs and the waitlist form somewhere ordinary and cheap like InterServer or any commodity host, entirely outside the AWS organisation. Small blast radius beats clever separation you have to explain to an auditor.

Boundary one: PHI arrives before you decide to accept it

Start with the thing that trips up almost every first build: HIPAA-eligible and HIPAA-compliant are not the same word. AWS designating a service as HIPAA-eligible means you are permitted to process PHI with it under an executed Business Associate Addendum. It says nothing about whether your deployment is compliant. That part is entirely yours.

Three conditions have to hold together before any PHI touches a service: the service is on the AWS HIPAA Eligible Services Reference, you have an executed BAA, and the account is designated for HIPAA use. Amazon Bedrock and Amazon Comprehend Medical both appear on that list, and the BAA is accepted through AWS Artifact rather than negotiated by email. Check the reference page yourself before you commit to an architecture, because the list changes and every component in your diagram has to be on it, not just the model.

Detecting PHI at the door

Amazon Comprehend Medical has an operation specifically for finding protected health information in unstructured clinical text. You hand it free text, it returns detected entities with a type, a confidence score and character offsets.

aws comprehendmedical detect-phi 
  --region us-east-1 
  --text "Patient seen for chest pain, MRN 004512, discharged Tuesday."

The offsets are the useful part. They let you redact or tokenise in place before the text goes anywhere else, rather than throwing the whole message away and telling the user to try again.

Now the caveat that matters more than the feature. AWS states plainly that Comprehend Medical may not identify PHI in all circumstances and that it does not, on its own, meet HIPAA’s requirements for de-identification. Read that as: it is a good filter and a terrible guarantee. If your compliance story is “we strip PHI before the model sees it, so we are outside HIPAA scope,” that story does not hold. Treat detection as defence in depth inside a compliant account, not as an escape hatch from needing one.

Boundary two: your own logs are the most likely leak

This is the invisible one. Bedrock model invocation logging is disabled by default and captures the full request data, response data and metadata for every call in the account, in that Region. You turn it on for a good reason, usually because someone in security asked who prompted what and when. Then it quietly becomes the largest concentration of raw clinical text you own, and it is nowhere on the architecture diagram because it is a checkbox rather than a component.

Check whether it is on before you assume anything:

aws bedrock get-model-invocation-logging-configuration --region us-east-1

The configuration is per account per Region, so run it in every Region where anyone has ever opened the Bedrock console. Setting it deliberately looks like this:

aws bedrock put-model-invocation-logging-configuration 
  --region us-east-1 
  --logging-config '{
    "s3Config": {
      "bucketName": "med-assistant-invocation-logs",
      "keyPrefix": "bedrock/"
    },
    "textDataDeliveryEnabled": true,
    "imageDataDeliveryEnabled": false,
    "embeddingDataDeliveryEnabled": false
  }'

Each delivery flag is a separate decision about a separate category of PHI. Text is the obvious one. Image delivery matters the moment anyone uploads a photographed chart or a scan, because burned-in identifiers travel with the pixels and no text filter will ever see them. Embedding delivery is the one people leave on without thinking; vectors derived from clinical text are not a safe artefact, and they are large.

The sharpest detail is buried in the Guardrails documentation: AWS notes you can disable invocation logs if you do not want blocked content appearing as plaintext in them. Read the implication. A guardrail can refuse a prompt, mask the identifiers, and stop the model ever seeing them, and the original text still lands in your log destination. The guardrail protects the model call. It does not protect the log.

• Encrypt the log destination with a customer-managed KMS key, and keep the key policy tight enough that “everyone with S3 read” is not also “everyone with chart access”.
• Set a retention period that reflects a legal decision someone actually made, not the CloudWatch default of never expiring.
• Send logs to a separate, tightly scoped account if your organisation is large enough to have people who need dashboards but not records.
• Remember your application logs too. A framework that logs request bodies on error will do this to you long before Bedrock does.

Boundary three: where the inference actually runs

Cross-Region inference in Bedrock exists because on-demand capacity is uneven and bursts happen. It routes your request to another Region to get it served. There are two flavours and the difference is not cosmetic.

• Geographic profiles keep routing inside a defined geography such as the US or the EU. A request that starts in the EU stays in EU Regions. This is the one built for residency requirements.
• Global profiles route to supported commercial Regions worldwide for maximum throughput. AWS documents that a request can be routed to a destination Region even if you never opted that Region into your account.

To be fair to global profiles: data is not stored in the destination Region, transfer happens encrypted across the AWS network, and your invocation logs, knowledge bases and configuration all stay in the source Region. For a workload with no geographic constraint it is a genuinely good default, and it typically carries a lower per-token rate than staying in-geography. For a clinical workload with a residency commitment in a contract, it is the wrong tool, and “the prompt left the geography but was not stored there” is a sentence you do not want to be constructing during an audit.

Pin it in policy rather than trusting a config value in a repo. A Service Control Policy denying the Bedrock API outside your approved Regions closes both doors at once, because invoking a cross-Region profile requires model access in the destination Regions as well as the source:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyBedrockOutsideApprovedRegions",
      "Effect": "Deny",
      "Action": "bedrock:*",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1", "us-west-2"]
        }
      }
    }
  ]
}

The trade-off is real and you should know it before you apply this. If you later adopt a geographic profile whose destination list includes a Region you denied, invocations fail with an access error that looks nothing like a routing problem. Keep the approved Region list and the profile’s destination list in the same review, not in two different tickets.

Boundary four: retrieval, and the grounding check that will not save you

A useful medical assistant is retrieval-augmented. The model alone is a fluent generalist; the value comes from grounding answers in a curated corpus, whether that is institutional guidelines, formulary rules or a licensed reference set. Bedrock Knowledge Bases will manage the ingestion and the vector store for you, or you can run your own index in Amazon OpenSearch Service.

Two things bite here.

Tenant isolation in the index. If you serve more than one hospital, department or study cohort, a shared index with a filter applied in application code is one refactor away from cross-tenant retrieval. Filters are easy to forget and impossible to notice, because a wrong answer that cites a real document looks exactly like a right one. Separate indexes per tenant, or fine-grained access control enforced below your application, cost more and fail safe.

The grounding check has a scope limit. Guardrails contextual grounding checks detect responses that are not supported by the source material or not relevant to the question. Excellent feature. The documentation states the supported use cases are summarisation, paraphrasing and question answering, and that conversational chatbot use cases are not supported. If your product is a chat interface with turn history, do not assume this check is covering you. There is also a streaming wrinkle: relevance is assessed per chunk, so an irrelevant response can reach the user before it is marked irrelevant at the end of the stream.

What you can rely on regardless of interface is applying a guardrail to arbitrary text directly, which is how you check an input before it enters your own pipeline:

import boto3

client = boto3.client("bedrock-runtime", region_name="us-east-1")

response = client.apply_guardrail(
    guardrailIdentifier=GUARDRAIL_ID,
    guardrailVersion="DRAFT",
    source="INPUT",
    content=[{"text": {"text": user_message}}],
)

if response["action"] == "GUARDRAIL_INTERVENED":
    # Stop here. Do not forward, and do not write the raw text anywhere.
    handle_blocked(response["assessments"])

Note what the comment is doing. The most common bug in this pattern is catching the intervention and then logging the offending input “for debugging”, which reintroduces exactly the leak the guardrail just prevented. Log the assessment, log a request identifier, never the text.

Sensitive information filters give you two handling modes: block, which rejects the request outright, and mask, which replaces detected entities with placeholder tags. For a study tool, mask is usually right, because a student who typed a name by reflex gets a useful answer instead of a wall. For a clinical assistant working over charts, blocking on unexpected identifiers in an input that should have arrived structured is a better signal that something upstream is wrong.

Boundary five: being confidently wrong

Everything above is about data leaving. This one is about a bad answer arriving, and for a medical audience it is the reputational failure that actually ends products.

• Cite or refuse. Return the retrieved passages alongside the answer. If retrieval returned nothing above your relevance threshold, say so rather than letting the model answer from parametric memory. A student cannot verify what they cannot see.
• Version the corpus, not just the model. When a guideline changes, you need to know which answers were generated against the old text. Store a corpus revision identifier with every logged response.
• Use denied topics deliberately. Dosing for a named patient, and anything that reads as a treatment directive rather than reference information, are reasonable things to route to a refusal with a clear explanation.
• Get regulatory advice early. Whether clinical decision support software is regulated as a medical device depends on your jurisdiction and, critically, on the claims you make about it. This is a legal question with engineering consequences, and it is much cheaper to answer before the interface exists.

A build order for a secure AI medical assistant on AWS

Sequence matters here more than in most builds, because several of these are painful to retrofit.

1. Accept the BAA through AWS Artifact and designate the account for HIPAA use. Do this before the first prototype, not before the first customer.
2. Pin Regions with a Service Control Policy, and decide the geographic-versus-global inference profile question in writing.
3. Configure invocation logging deliberately, with a customer-managed KMS key, an explicit retention period, and each data-type delivery flag chosen rather than defaulted.
4. Put the application in private subnets and reach Bedrock over VPC endpoints so PHI-bearing traffic does not traverse the public internet.
5. Build the guardrail before the prompt. Sensitive information filters, denied topics, and grounding checks where they apply.
6. Add Comprehend Medical detection in the ingestion path for anything you are storing, and at the input boundary for anything a user types.
7. Build retrieval with tenant isolation from the first index, not the second.
8. Only now write the assistant’s prompt and interface, and put a WAF such as Cloudflare or AWS WAF in front of the public endpoint.

Troubleshooting

• AccessDenied on a model that clearly works elsewhere. Usually one of three things: model access not requested in this Region, a cross-Region profile whose destination Regions your SCP denies, or an IAM policy that grants the model in the source Region only.
• No invocation logs appearing. The configuration is per Region and disabled by default. Confirm you queried the same Region the application calls, and that the delivery flag for the data type you expect is enabled.
• Guardrail passes text you expected it to catch. Sensitive information detection is probabilistic and context-dependent. Very short inputs give it little to work with. Test with realistic clinical phrasing, not single tokens, and add regex patterns for structured identifiers like MRNs that follow a house format.
• Grounding check appears to do nothing. Check your interface shape against the supported use cases before assuming it is misconfigured, and check whether streaming is masking the result until the response completes.
• Retrieval returns plausible but wrong documents. Look at chunking before you look at the model. Clinical guidelines chunked mid-table or mid-criteria retrieve badly no matter what embedding you use.

Common mistakes

• Prototyping in a personal or sandbox account and promising to migrate later. The prototype is where the first real chart gets pasted.
• Assuming the eligibility of Bedrock covers the whole stack. Every component that touches PHI needs to be eligible, including the vector store, the queue and the cache.
• Treating PHI detection as de-identification. AWS says explicitly that it is not.
• Leaving image and embedding log delivery enabled by copy-paste.
• Logging blocked prompts to debug the guardrail.
• Shipping a chat interface and citing contextual grounding as the hallucination control.
• Filtering tenants in application code over a shared index.

Best practices

• Write the data-flow diagram with logs, backups and the vector index drawn as first-class destinations. If PHI can land there, it is on the diagram.
• Define everything in Terraform or OpenTofu so the guardrail, the logging configuration and the SCP are reviewable artefacts rather than console state.
• Keep a small evaluation set of realistic clinical questions with known-good answers, and run it on every prompt or model change.
• Alarm on guardrail intervention rate. A sudden rise usually means a change upstream, not a change in users.
• Dashboard invocation counts, latency and intervention rates somewhere your on-call actually looks, whether that is CloudWatch, Grafana or Datadog.
• Scope IAM to specific model ARNs and specific guardrail identifiers. A wildcard on bedrock:InvokeModel means any model, including ones you never evaluated.
• Rehearse the breach path once. Knowing which bucket, which log group and which key you would need to reason about is worth an afternoon.

Frequently asked questions

Is Amazon Bedrock HIPAA compliant?

Bedrock is HIPAA-eligible, which is a different claim. Eligibility means AWS permits you to process PHI with the service under an executed BAA. Compliance is a property of your deployment: your account designation, encryption, access control, network isolation, logging and retention. A Bedrock workload with a signed BAA and public endpoints and unbounded plaintext logs is not compliant.

Do I need a BAA if the tool is only for medical students?

If the tool genuinely never receives PHI, HIPAA is not engaged. The practical question is whether you can guarantee that, given that your users have clinical access and a free-text box. If you cannot, get the BAA. It costs you a self-service acceptance in AWS Artifact and removes an entire category of incident.

Does AWS use my prompts to train the models?

AWS states that customer content submitted to Bedrock is not used to train the underlying foundation models or shared with model providers. That statement is the sort of thing a hospital security review will want quoted verbatim from the current AWS data protection documentation rather than from a blog, so pull the live wording when you write your assessment.

Is Comprehend Medical enough to de-identify clinical text?

No. AWS documents that it may not identify PHI in all circumstances and does not by itself meet HIPAA’s de-identification requirements. Use it as a detection layer and a redaction aid inside a compliant environment. Formal de-identification, whether by the Safe Harbor method or expert determination, is a separate exercise with its own review.

Where should the vector store live?

In the same account and Region as the rest of the workload, on a HIPAA-eligible service, encrypted with a customer-managed key, reachable only from private subnets. Embeddings derived from clinical text are not sanitised data and should not be treated as a lower-sensitivity artefact than the source.

Should I use a global or geographic inference profile?

Geographic if you have any residency commitment, contractual or regulatory. Global if you have none and want the throughput and the lower per-token rate. Decide once, document the reasoning, and enforce it with a Service Control Policy rather than a configuration constant.

Does an AI medical assistant count as a medical device?

It depends on your jurisdiction and on what you claim the software does. Software that surfaces reference information a clinician independently reviews has generally been treated differently from software that directs a clinical decision, but the boundary is fact-specific and moves. This is a question for regulatory counsel before launch, not a question for your architecture diagram.

The one thing worth remembering

A secure AI medical assistant on AWS is not mainly a model problem. Bedrock, Guardrails and Comprehend Medical are the easy part, and the documentation for them is good. The hard part is that PHI leaves through the paths you did not draw: an invocation log you enabled for good reasons, a global inference profile that routes wherever capacity exists, an index shared between tenants, a debug line added at two in the morning.

So build the boundary first and the assistant second. Get the BAA accepted, pin the Regions in policy, decide consciously what your logs are allowed to hold, and isolate retrieval per tenant before there is a second tenant. Every one of those is cheap on day one and expensive in month six.

Need a second pair of eyes on your build?

I work with teams building AI on AWS where the data is sensitive and the failure modes are quiet. Things I can help with on a project like this:

• Reviewing a Bedrock architecture against the boundaries above and telling you where PHI can actually land
• Setting up account separation, BAA scope, Region pinning with Service Control Policies, and VPC endpoint access to Bedrock
• Designing and tuning Guardrails policies, including custom regex for house identifier formats, and the block-versus-mask decision per surface
• Building the retrieval layer with per-tenant isolation, sensible clinical chunking, and citation-or-refuse behaviour
• Auditing invocation logging, KMS key policies, retention and application-level log hygiene for accidental PHI capture
• Putting the whole thing in Terraform or OpenTofu so your controls are reviewable instead of remembered

If any of that is on your plate, send me the piece you are least sure about. A redacted architecture diagram, a guardrail configuration, a logging policy, an AccessDenied you cannot explain. I would rather look at the real thing than talk in generalities.

The post Build a Secure AI Medical Assistant on AWS: The Boundaries That Actually Leak appeared first on John Nessime.

“EC2 – Other” is where AWS files NAT gateway charges, along with EBS, elastic IPs and assorted data transfer. Your Salesforce sync runs in a private subnet, so every byte it pulls from Salesforce and every byte it writes to S3 goes through a NAT gateway, which bills an hourly rate plus a charge per gigabyte processed. None of that appears under Lambda, or under S3, or under the tag you put on the pipeline.

That is the recurring theme when you look at the AWS cost of Salesforce data pipelines: the spend is almost never in the place you have been optimising. People tune Lambda memory settings while the bill is being driven by data movement, request counts, log ingestion and scanned bytes.

Five levers, in the order I would pull them.

First, find out where it actually goes

Every hour spent guessing is an hour not spent on the line item that matters. Tag everything the pipeline owns with a consistent cost allocation tag, activate it in the billing console, and wait for it to backfill.

Then group Cost Explorer by usage type rather than by service. Service-level grouping is what hides NAT charges inside a generic bucket; usage type is where the truth lives.

# Substitute a whole billing month. Grouping by USAGE_TYPE is the
# point: "EC2 - Other" tells you nothing, "NatGateway-Bytes" does.
aws ce get-cost-and-usage 
  --time-period Start=YYYY-MM-01,End=YYYY-MM-01 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["EC2 - Other"]}}'

Add S3 Storage Lens for the bucket. It tells you object count and average object size per prefix, which is the number that predicts your request bill and your Athena bill better than total storage does.

Lever 1: get the traffic off the NAT gateway

Usually the largest single saving available, and often a change you can make before lunch.

Start with the question nobody asks: does this function need to be in a VPC at all? A Lambda that calls the Salesforce REST API and writes to S3 touches nothing private. If it is in a VPC because that felt like the secure default, taking it out removes the NAT gateway from the path entirely and costs you nothing.

If it genuinely needs the VPC, because it also talks to RDS or something inside your network, add a gateway VPC endpoint for S3. AWS charges no hourly fee and no data processing fee for gateway endpoints, so S3 traffic simply stops flowing through NAT:

aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0

Two honest caveats. Gateway endpoints exist only for S3 and DynamoDB; everything else needs an interface endpoint, which has an hourly charge per availability zone plus a per-gigabyte charge. Still cheaper than NAT for AWS-service traffic, not free.

And the one specific to this workload: endpoints do nothing for your Salesforce traffic. Salesforce is not an AWS service, so those calls still traverse NAT. If you are pulling large volumes out of Salesforce from inside a VPC, the endpoint fixes the S3 half of the bill and leaves the other half untouched. That is the strongest argument for getting the function out of the VPC rather than papering over it.

Lever 2: fewer, bigger files

A sync running every five minutes across twelve Salesforce objects writes a lot of very small files. Storage for those is negligible. The costs are elsewhere and they compound.

• PUT requests are billed per request and cost meaningfully more than GETs. Millions of tiny writes is a request bill, not a storage bill.
• Athena opens every file it reads. Thousands of small objects per partition turn a fast scan into a slow, expensive one.
• Glue crawlers take longer over more objects, and they are billed by the second.

The fix is compaction: a scheduled job that rewrites each day’s partition into a small number of larger Parquet files and deletes the fragments. If you are on Iceberg, its compaction procedure does this for you and is the better reason to adopt it than any of the ones usually cited.

One trap worth flagging, because it is the opposite of what people expect. Lifecycle rules that move small objects to S3 Standard-IA can increase your bill: those classes have a minimum billable object size of 128 KB, so a 10 KB file is billed as 128 KB, plus a 30-day minimum duration. And S3 Intelligent-Tiering will not help either, since objects under 128 KB are never auto-tiered and stay at Frequent Access rates. Compact first, then think about storage classes. In the other order you are paying more for the privilege.

Lever 3: log less

CloudWatch Logs is billed primarily on ingestion volume, and on a high-frequency pipeline it is routinely larger than the compute it is logging. A sync that logs every record it processes, running every five minutes, generates an astonishing amount of data whose only reader is a retention policy.

• Set a retention period on every log group. The default is to keep logs forever, and log groups created automatically by Lambda inherit it. This is the single most common unclaimed saving on an AWS account.
• Log per run, not per record. One structured summary line with counts, durations and outcome tells you more during an incident than ten thousand lines of per-record chatter.
• Turn off debug logging in production, and make the level an environment variable so raising it is a deliberate act rather than a redeploy.

Check the log classes available before you settle. There is a cheaper class aimed at logs you rarely query, which suits pipeline output well, with fewer features in exchange.

Lever 4: scan less

Athena bills on bytes scanned, so its cost is a property of your data layout, not your SQL. Three things move it, in order.

Parquet rather than JSON or CSV. Columnar means a query touching four columns reads four columns. This is usually a large multiple, not a percentage.

Partitions that match how people filter. If every query has a date range, partition by date. If nobody ever filters by region, partitioning by region just creates more small files.

Partition projection instead of a crawler. Athena works partitions out from the path pattern rather than reading them from the catalog, which removes the crawler cost, removes the repair step, and means partitions are never stale:

ALTER TABLE salesforce.opportunity SET TBLPROPERTIES (
  'projection.enabled'             = 'true',
  'projection.extract_date.type'   = 'date',
  'projection.extract_date.range'  = '2024-01-01,NOW',
  'projection.extract_date.format' = 'yyyy-MM-dd',
  'storage.location.template' =
    's3://acme-lake/raw/salesforce/opportunity/extract_date=${extract_date}/'
);

Set a per-query and per-workgroup data scanned limit while you are in there. It converts a runaway SELECT * from a surprise on the bill into an error someone sees immediately.

Lever 5: run it less often

The cheapest work is work you do not do. A sync running every five minutes because five minutes felt responsive, feeding a dashboard people open once each morning, is paying for freshness nobody consumes.

Ask what decision the data supports and how quickly it needs to be right. Hourly is enough for most reporting, and dropping from five minutes to hourly cuts invocations, requests, log volume and small-file creation all at once. It is the only lever here that improves four line items with one change.

Two false economies while you are looking. Cutting Lambda memory usually costs more, not less, because the function runs proportionally longer and you also lose CPU. And moving from Athena to a provisioned engine to escape per-query pricing only pays off at genuinely high query volume; below that you have swapped a variable cost for a fixed one and called it a saving.

Common mistakes

• Optimising compute because it is the part you understand, while data movement drives the bill.
• Reading Cost Explorer grouped by service, so NAT charges stay hidden inside “EC2 – Other”.
• Running a Lambda in a VPC when nothing it touches is private.
• Adding a gateway endpoint and expecting it to help traffic bound for Salesforce.
• Millions of small objects, and treating it as a storage problem rather than a request problem.
• Lifecycle rules that move sub-128 KB objects to an infrequent access class and increase the bill.
• No retention policy on log groups.
• Logging per record on a five-minute schedule.
• JSON in the lake because it was easier to write.
• Running a Glue crawler on a schedule when partition projection would do it for nothing.
• No cost allocation tags, so nobody can attribute anything.
• Syncing far more frequently than anybody reads the output.

Best practices

• Tag every resource the pipeline owns, and group Cost Explorer by usage type.
• Keep functions out of VPCs unless they need private resources.
• Gateway endpoints for S3 and DynamoDB on every VPC that touches them.
• Compact partitions on a schedule; measure average object size, not just total bytes.
• Compact before choosing storage classes, never the other way round.
• Retention on every log group, set at creation, enforced in your IaC.
• One structured summary log line per run.
• Parquet, partitioned the way people actually filter, with partition projection.
• Per-query and per-workgroup scan limits in Athena.
• Match the schedule to the decision the data supports.
• A budget alert on the pipeline’s tag, so the next surprise arrives as a notification.

FAQ

Why is “EC2 – Other” large when I run no EC2?

It is a catch-all that includes NAT gateway hours and data processing, EBS, elastic IPs and data transfer. Filter Cost Explorer to that service and group by usage type; if you see NAT gateway bytes near the top, that is your pipeline paying to move its own data.

Should my sync Lambda be in a VPC?

Only if it needs to reach something private, such as a database in your own subnets. Calling an external API and writing to S3 does not require it, and putting it in a VPC adds NAT charges and cold start complexity for no security benefit you can articulate.

Will Intelligent-Tiering fix my small files?

No. Objects under 128 KB are never auto-tiered and stay at Frequent Access rates. Standard-IA is worse, because it has a 128 KB minimum billable size, so small objects are billed as if they were larger. Compaction is the fix; storage class is a decision for afterwards.

Glue or Lambda for the extraction?

Lambda for anything that fits in its execution limits, which covers most incremental CRM syncs, and it is cheaper for short frequent runs. Glue when you genuinely need Spark for volume or transformation. Glue billed by the second for a job that runs for ninety seconds is a poor trade.

What is the fastest saving to make?

Retention policies on log groups, then the S3 gateway endpoint. Both take minutes, neither carries risk, and between them they usually account for more than whatever you were planning to spend a week optimising.

The one thing to remember

Data pipeline bills are dominated by moving bytes, counting requests, ingesting logs and scanning storage. Compute is the part you think about and rarely the part you pay for.

So do the boring thing first: tag it, group Cost Explorer by usage type, and read the top five lines. The answer is almost always sitting there in a category you have been scrolling past, and it is usually cheaper to fix than to keep paying for.

Bill larger than it should be?

Pipeline cost work has a good ratio: a few days of investigation usually pays for itself in the first month. Work I take on:

• Cost breakdown of an existing Salesforce or CRM pipeline, by usage type, with a ranked list of what to fix and what it saves.
• Network path work: VPC endpoints, getting functions out of VPCs, and removing NAT from the data path.
• Storage layout: compaction jobs, partitioning strategy, Parquet or Iceberg conversion.
• Athena cost control: partition projection, workgroup scan limits, replacing scheduled crawlers.
• Logging and retention cleanup across accounts, which is usually the quickest win available.
• Cost allocation tagging and budget alerts so the next increase is noticed early.

Send me a month of Cost Explorer grouped by usage type and I will tell you where the money is going.

The post You Don’t Run Any EC2: Cutting the AWS Cost of Salesforce Data Pipelines appeared first on John Nessime.