You look at the CRM reporting tab. It cannot do that. So now you are picking a platform, and within about ten minutes someone will say “we should build a data lake,” and someone else will say “just put it in the warehouse,” and the deciding argument will be about how much data there is.

That argument is almost always the wrong one. This post covers data lake vs data warehouse specifically for CRM analytics: why volume rarely decides it, what actually does, a decision procedure you can run in an afternoon, and where zero-ETL has changed the maths recently enough that older advice is stale.

Start with the number, then stop using it

Go and count your rows. Not estimate. Count.

A mid-sized company’s entire Salesforce org is usually a few million rows across the objects anyone actually reports on: accounts, contacts, opportunities, activities, cases. Add HubSpot, Zendesk and a billing system and you are still typically in the low tens of gigabytes once it is written as compressed Parquet. Activity and event history is the one object that can genuinely run large, and it is worth checking separately.

Numbers at that scale do not stress either architecture. A data warehouse will not break a sweat. A lake will not break a sweat. The lake-versus-warehouse debate as it is usually framed, the one about petabytes and elasticity and separating storage from compute, is a debate about a scale problem you do not have.

So count the rows, confirm you are small, and then stop using volume as an argument. Everything below is about what decides it instead.

Data lake vs data warehouse: the three options, honestly

Warehouse only

Load CRM objects straight into Redshift, Snowflake, BigQuery or Postgres. Model with SQL. Point the BI tool at it.

Where it wins: it is the fastest path to an analyst being productive, and that is not a small thing. Query performance on joins is better than a lake gives you without tuning. Access control is mature and row-level. Every BI tool connects natively. The people who will maintain this already know SQL and probably do not know Spark.

Where it doesn’t: semi-structured data is awkward, and CRM payloads are full of it. Custom field bags, JSON blobs, email bodies, call transcripts. You can store JSON in a modern warehouse but you will not enjoy querying it at any depth. Storing many years of daily snapshots gets expensive on warehouse storage in a way it does not on object storage. And you are more locked in: moving between warehouses is a migration project, not a repoint.

Lake only

Land raw API responses to S3, catalogue them, query with Athena or Spark. Iceberg or Delta as the table format if you want anything resembling a database.

Where it wins: raw retention is nearly free, so keeping every version of every record for years is a storage line item rather than a decision. Schema evolution is genuinely easier: when a SaaS admin adds a field, the lake absorbs it. Unstructured content lives alongside structured tables, which matters more now that people want to run models over call notes and support tickets. And an open table format means the data is not hostage to one query engine.

Where it doesn’t: you have taken on work a warehouse does for you. Small-file compaction, partitioning, table maintenance, catalogue hygiene. Query latency on interactive dashboards is worse unless you invest in layout. Governance is a build, not a setting. And the per-query cost model punishes exactly the workload CRM analytics generates: lots of small, repeated, unpredictable queries from a BI tool. Athena bills on data scanned with a per-query minimum, which means a dashboard refreshing thirty tiles is thirty billable scans whether or not anyone is looking.

Lakehouse

Raw and historical data in object storage as Iceberg tables; curated marts in the warehouse or queried in place by a warehouse engine. This is where most serious setups end up, and it is what AWS, Databricks and Snowflake have all converged on.

Where it wins: you get cheap deep history and expensive fast queries in the right places. Raw payloads stay replayable when a vendor changes a field type. Analysts still get warehouse-shaped tables.

Where it doesn’t: it is two systems. Two cost models, two sets of permissions, two places a table can be defined and drift. For a five-person company with one CRM, this is architecture as procrastination. The complexity is real and it should be earning something.

What actually decides it

Four levers, in the order they usually matter for CRM data.

1. How much history you need, and whether deletes matter

This is the big one and it is the one people skip.

CRM systems mutate in place. An opportunity’s stage, amount and close date are overwritten as the deal moves. If you only ever sync current state, you can report on the pipeline as it is today and you cannot report on what it looked like last March. Every question about pipeline movement, forecast accuracy, stage velocity or “what changed since the QBR” needs point-in-time history, and the CRM will not give it to you retroactively. Deleted records are worse: they are simply gone.

If you need history, you need somewhere cheap to keep daily snapshots or a change feed. That pushes toward object storage, or toward a warehouse with a genuinely cheap storage tier. If a rolling twelve months of current-state data covers every question anyone asks, the warehouse handles it alone.

2. How fast the schema churns

Count the custom fields added to your main CRM object in the last six months. If it is two, schema churn is not your problem. If it is forty, and half of them are from a business unit you do not talk to, a rigid warehouse schema becomes a standing maintenance tax and the lake’s tolerance for drift starts paying for itself.

3. The shape of the queries

Scheduled dashboards hitting the same tables repeatedly favour a warehouse, because per-second compute amortises across many queries and results are cached. Redshift Serverless, for example, bills in processing-unit-hours per second with a short minimum and shuts down when idle, which suits bursty BI traffic.

Exploratory one-off analysis across wide, rarely-touched history favours a lake, because you only pay when someone asks. Most CRM analytics is overwhelmingly the first kind. That fact alone settles a lot of these arguments.

4. Who is going to maintain it

The least technical criterion and often the most decisive. If the answer is “a RevOps analyst who is very good at SQL and has no platform team behind them,” build the warehouse. A lake handed to someone who cannot maintain it becomes an unqueryable pile of Parquet in about eight months. That is not a hypothetical failure mode; it is the normal one.

A decision procedure you can run this afternoon

1. Count rows per object for everything anyone reports on. Note which objects are over about ten million rows. Usually it is activities and nothing else.
2. Write down the ten questions the business actually asks. Not the ones you imagine. Mark each one as needing current state or point-in-time history.
3. Count schema changes on your main objects over the last six months.
4. Estimate query volume: number of dashboards, tiles per dashboard, refresh frequency, plus rough ad-hoc queries per week. This is the number that drives cost, not storage.
5. Model both costs against those numbers. For the lake: bytes scanned per query times queries per month. For the warehouse: compute-hours per month plus storage. Do it on a napkin. At CRM scale the answer is usually obvious and usually favours the warehouse.
6. Name the maintainer. If you cannot name a person, pick the option that needs less of them.

Read together, most CRM-only shops land on: warehouse for the curated layer, cheap object storage for raw history, and no ambition beyond that until something forces it. That is a lakehouse, but the small honest kind, and it is a very different commitment from standing up a full lake platform.

Where zero-ETL changed the maths

A lot of writing on this topic predates a genuine shift, so it is worth stating plainly.

Managed zero-ETL integrations now replicate CRM and support platforms into AWS targets without you building a pipeline. AWS Glue supports SaaS sources including Salesforce, SAP, ServiceNow, Zendesk and Zoho CRM, landing into Amazon Redshift, the SageMaker lakehouse, or S3 Tables with Iceberg underneath. Salesforce has pushed the other direction with zero-copy federation, letting Data Cloud read Iceberg tables in Databricks, Snowflake, BigQuery or Redshift in place, and letting those platforms read Data Cloud objects back.

Two consequences for this decision:

• Ingestion is no longer the differentiator. “The lake is easier to load into” was a real argument when you were hand-rolling connectors. It is much weaker when the same managed integration targets either destination.
• Iceberg makes the choice less permanent. If your raw layer is Iceberg tables in object storage, adding a warehouse engine later is a configuration change rather than a migration. That is a genuine reason to prefer open table formats even if you never build a “lake” as such.

Two honest caveats. Zero-ETL integrations cover a curated list of sources and regions, so check yours before designing around them. And a managed replica is still current-state replication plus change capture, not a modelled history; you still have to decide how snapshots and slowly changing dimensions get built.

Arguments that don’t survive contact

• “We need a lake because we’ll do AI later.” Reasonable in spirit, wrong in sequence. Build the thing that answers this quarter’s questions, keep raw payloads in object storage so you have not destroyed your options, and revisit. Raw retention is the cheap insurance; a lake platform is not.
• “Storage is cheaper in a lake.” True and almost always irrelevant. At CRM volumes, storage is a rounding error next to compute and, more to the point, next to the salary of whoever maintains the thing.
• “The warehouse is a single point of vendor lock-in.” Fair, and it is why open table formats matter. But if you replicate raw data to object storage, the lock-in is on your transformation logic, not your data, and that is portable if you keep it in SQL and version control.
• “We’ll just query the CRM API directly and skip both.” Works until a dashboard fans out into rate limits and the whole thing becomes a data pipeline you did not design. If more than one person is asking questions, you need a copy.
• “Let’s do both properly from day one.” This is the expensive mistake. Two systems, two cost models, and tables that quietly disagree, in service of a workload that fits comfortably in one.

Frequently asked questions

Is a data lake overkill for one CRM?

For a single CRM with current-state reporting needs, usually yes. The point at which it stops being overkill is when you need years of point-in-time history, when unstructured content like call transcripts becomes part of the analysis, or when you are integrating enough systems that a common raw layer is cheaper than a dozen warehouse loaders.

How do I get historical CRM snapshots if I only have current state?

You mostly cannot get it retroactively, which is why this decision is urgent rather than academic. Start capturing now: either daily full snapshots to partitioned object storage, or a change feed if the platform offers one, then build slowly changing dimension tables from that. Field history tracking inside the CRM covers a limited number of fields and is not a substitute.

Does the lakehouse make this comparison obsolete?

It blurs the storage layer, not the operational reality. You still choose a query engine, a cost model, a governance model and a maintainer. The useful question has shifted from “lake or warehouse” to “how much of this do I want to operate myself,” which is a better question anyway.

Where does reverse ETL fit?

If you plan to push computed values back into the CRM, such as a lead score or account health field, that pushes toward the warehouse. Reverse ETL tools connect to warehouses far more readily than to query engines over object storage, and the write path wants a stable, modelled table rather than a raw one.

Can I start with one and move later?

Moving from lake to warehouse is straightforward: you are loading modelled tables into a new engine. Moving from warehouse to lake is harder, because the history only exists inside the warehouse and the transformations are written in its dialect. The asymmetric fix is to land raw data in object storage regardless of which you query from. That single habit preserves the option cheaply.

What about just using the CRM’s own analytics product?

Worth pricing seriously before building anything. It is genuinely the right answer when all your questions are about one system, and it removes an entire pipeline from your life. It stops being the right answer the moment a question spans CRM plus billing plus product usage, because you will be paying to ingest the other systems into a platform priced for CRM data.

The short version

In the data lake vs data warehouse question for CRM analytics, volume is the argument everyone reaches for and it is the one that matters least. Your CRM data is small. Both options handle it.

What decides it is how much history you need, how fast the schema moves, what shape the queries are, and who is going to keep it running. Answer those four and the architecture falls out on its own. And whatever you pick, land the raw payloads in object storage from day one, because that is the cheap decision that keeps every other decision reversible.

Need a second opinion on your CRM analytics stack?

I help teams make this call without over-building, and unpick it when it has already been over-built. Things I can help with:

• Running the sizing and cost model above against your real query volume, not a vendor calculator.
• Designing point-in-time history capture for CRM objects, including deletes and slowly changing dimensions.
• Setting up zero-ETL or connector-based ingestion from Salesforce, HubSpot, Zendesk or Zoho into Redshift, Snowflake or an Iceberg lake.
• Migrating an unmaintainable pile of Parquet into something an analyst can actually query.
• Cutting query costs on an existing setup: partitioning, table layout, dashboard refresh patterns.
• Building the curated CRM models that BI tools sit on, in SQL and in version control.

If you are mid-argument about this, send me your row counts and the ten questions the business wants answered. That is usually enough to settle it.

The post Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question appeared first on John Nessime.

A Salesforce admin renames a custom field on Tuesday afternoon. Your connector keeps running, your Glue job keeps succeeding, the crawler happily adds a new column and quietly stops populating the old one. Three weeks later someone asks why the pipeline report has a cliff in it, and you spend an afternoon reconstructing a change nobody in engineering made and nobody logged.

This post is about using AWS Glue Data Quality to catch that class of breakage. It is aimed at people pulling from SaaS sources into S3 or a lakehouse: Salesforce, HubSpot, Zendesk, Stripe, Shopify, Zoho, an internal partner API. I will cover where to run the checks, the flattening constraint that will bite you before anything else does, which DQDL rule actually catches which SaaS failure, when dynamic rules and anomaly detection quietly do nothing, and what should happen when a rule fails.

Why SaaS data breaks in ways your own database doesn’t

When you own the source database, the schema is under change control: someone writes a migration, it gets reviewed, and if it breaks a consumer there is a commit to point at. A SaaS source has none of that. Three consequences:

• The schema is edited by people who are not engineers. A sales ops admin adding a picklist value is a production schema change with no review, no notice and no version number.
• The payload is nested. SaaS APIs return structs and arrays: line items, related objects, custom field bags. That shape matters more than you would expect, for reasons in the next section but one.
• Absence is ambiguous. A field missing from a payload might mean “not set,” “not permitted for this user,” or “we removed it.” The connector usually cannot tell you which, and neither can your table.

Glue Data Quality is a rules engine built on Deequ that evaluates declarative checks written in DQDL, the Data Quality Definition Language, against a dataset. It is well suited to the first and third problems. The second one needs work before it is useful at all.

Decide where the checks run before you write a rule

There are two places to evaluate a ruleset, and they answer different questions.

Inside the Glue ETL job

The EvaluateDataQuality transform runs mid-pipeline against a DynamicFrame. It answers “is this batch safe to load?” and it can act on the answer: stop the job, or split failing rows into a quarantine prefix. This is also the only placement where row-level results exist, which matters because “which 400 rows are bad” is a far more useful answer than “the rule failed.”

Against a Data Catalog table

You can attach a ruleset to a catalog table and run it on a schedule, independent of any job. This answers “is the table currently trustworthy?” and surfaces a quality score analysts can see before they query. The trade-off is that it is after the fact: by the time a catalog evaluation fails, the bad batch is already queryable. Analyzers are also not supported for catalog evaluations, so some dynamic-rule tooling behaves differently there.

In practice I run both. The in-job ruleset is small and strict, and exists to stop garbage landing. The catalog ruleset is broader and looser, and exists so a table’s state is visible to people who will never open Glue Studio.

The flattening problem that comes first

Here is the constraint that catches most people on SaaS data, stated plainly in the AWS documentation and easy to skim past: DQDL cannot evaluate nested or list-type columns. No structs, no arrays.

Almost every interesting SaaS payload is nested. A Salesforce opportunity carries a related account struct, a Stripe invoice carries an array of line items, a Zendesk ticket carries a custom fields array of key-value pairs. Point a ruleset at the raw landed data and the fields you most want to check are the ones the engine cannot see.

So you flatten first, with Glue’s relationalize or an explicit projection, then evaluate. That works, but it changes what your rules mean:

• Your rules now describe the projection, not the payload. If your flattening step drops a field, no rule will notice, because from DQDL’s point of view that field was never there.
• Exploding an array changes the grain. One invoice with five line items becomes five rows. Every RowCount rule you write against that table is now counting line items, and a customer who starts itemising differently will move your row count without any data being wrong.
• Custom-field bags need pivoting, not flattening. A key-value array flattened naively gives you rows, not columns, and you cannot write a ColumnValues rule against a value whose meaning depends on a sibling key column. Pivot the ones you care about into real columns and check those.

The practical rule: put your quality checks immediately after the flattening step, and treat the flattening step itself as code that needs its own test. Glue Data Quality will not cover it for you.

Matching SaaS breakage to the rule that catches it

DQDL has a long rule list and most guides just enumerate it. More useful is the other direction: what actually goes wrong with SaaS sources, and what catches each one.

A field is renamed, removed, or added

This is the Tuesday-afternoon scenario. Two rules, doing two different jobs:

Rules = [
    ColumnExists "annual_revenue__c",
    ColumnCount = 47
]

ColumnExists protects the specific fields your downstream models depend on. ColumnCount protects against everything else: a field silently added, or one you forgot you needed. The first is precise and the second is a tripwire, and you want both because a rename shows up as one column gone and one column arrived, which leaves the count unchanged.

SchemaMatch compares against a reference dataset’s schema, which is stricter and useful if you maintain a golden reference table. Bear in mind a hard column count on a SaaS object fails every time an admin adds a field, so decide whether that blocks or just warns.

A field still exists but stopped being populated

Permissions change, a workflow gets disabled, an integration user loses field-level access. The column is still in the schema and is now all nulls.

Rules = [
    IsComplete "opportunity_id",
    Completeness "lead_source" > 0.85,
    Completeness "email" > 0.60 where "record_type = 'Contact'"
]

IsComplete is the absolute version: no nulls at all, right for keys. Completeness takes a threshold, which is what you want for optional business fields where a hard rule would fail constantly. The where clause is the one people miss: SaaS objects are usually polymorphic, and a completeness threshold that makes sense for contacts is nonsense across the whole table. Scope the rule to the record type.

The sync ran short

A paginated extract stops early, a rate limit gets absorbed by retries, an incremental cursor drifts. The job succeeds and delivers less data than it should have. A static RowCount > 1000 stops catching this the moment you cross a thousand rows. Dynamic rules compare against your own history instead:

Rules = [
    RowCount > min(last(3)),
    Sum "amount" > avg(last(5)) * 0.7,
    DetectAnomalies "RowCount"
]

Analyzers = [
    RowCount,
    DistinctValuesCount "owner_id"
]

last(k) pulls the last k recorded values of that metric for the same dataset; with k greater than one you need an aggregation like avg, min or max to reduce them to one number. DetectAnomalies goes further, using forecasting over stored history to flag deviations with no threshold from you at all. That is the right tool for anything seasonal: a B2B pipeline that dips every weekend trips a naive trailing-average rule every Monday, while anomaly detection learns the shape.

Analyzers deserve their own mention: they gather a metric without asserting anything, which builds the history dynamic rules and anomaly detection need. DistinctValuesCount on an owner or account column is a strong early signal for a partial sync, because row count can look fine while the number of distinct accounts represented quietly halves.

Records arrive twice, or the key stops being a key

Replays, overlapping incremental windows and connector retries all produce duplicates, and SaaS “IDs” are less unique than vendor docs imply once you have merged sandboxes or multiple accounts.

Rules = [
    IsPrimaryKey "id",
    Uniqueness "external_reference" > 0.99
]

IsPrimaryKey asserts unique and non-null together. Uniqueness with a threshold is the softer version for fields that are mostly-but-not-perfectly unique, which describes a lot of SaaS reference fields honestly.

Objects stop agreeing with each other

Multi-object extracts rot quietly. Opportunities reference accounts that were never synced; invoice totals stop matching line item sums because one object’s extract failed and the other’s didn’t.

ReferentialIntegrity checks that values in one dataset’s column exist in another’s. RowCountMatch and AggregateMatch compare counts and summary metrics across two datasets. These catch a broken join before an analyst does, and they need the multi-frame form of the transform with the second dataset supplied as an additional source.

The sync stopped and nobody noticed

The quietest failure of all: nothing is wrong with the data, there just isn’t any new data. DataFreshness checks a timestamp column against a window. It is deliberately simple and its comparison granularity is limited, so for anything more nuanced the custom SQL rule is the escape hatch, running a query against the dataset and evaluating the result against an expression. DQDL is case sensitive and that rule type’s exact capitalisation has appeared more than one way across the docs and the visual builder, so copy it from the current DQDL reference rather than from memory.

Static, dynamic and ML: when each one lies to you

All three check styles have a mode where they pass while the data is wrong. Knowing which is which is most of the skill.

• Static rules lie by going stale. A threshold that was correct when you wrote it stops being correct as the business grows, and a rule that can no longer fail is worse than no rule because it looks like coverage.
• Dynamic rules lie when they have no history. This one is genuinely nasty. last(k) reads previously recorded values for that metric, and if there aren’t any, it falls back to a default of zero. Add RowCount > avg(last(3)) to an existing job today and it will pass trivially for the first few runs, because almost any row count beats zero. It looks like a working check from the moment you deploy it. It isn’t one until it has accumulated runs.
• Anomaly detection lies while it is warming up, and after you retrain it badly. It needs a run of history before its forecasts mean anything, and if you leave a period of genuinely bad data in the training window, it learns that the bad shape is normal. You can exclude specific statistics from the calculation in the Data Quality tab, and you should, after any incident.

So when you add a dynamic or ML-based rule, write down the date and treat it as unarmed for the first couple of weeks. Keep a crude static rule alongside it in the meantime.

What should happen when a rule fails

The transform returns a collection. The two members you care about are the row-level outcomes, which give you the original data with an evaluation result column appended per row, and the rule outcomes, which give you one row per rule with its verdict.

from awsgluedq.transforms import EvaluateDataQuality

dq_results = EvaluateDataQuality().process_rows(
    frame=flattened_opportunities,
    ruleset=ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "salesforce_opportunity",
        "enableDataQualityCloudWatchMetrics": "true",
        "enableDataQualityResultsPublishing": "true",
    },
)

row_level = SelectFromCollection.apply(
    dfc=dq_results, key="rowLevelOutcomes"
)

dataQualityEvaluationContext is the namespace your CloudWatch metrics land under. Name it after the source object rather than the job, because you will eventually want to alert per object.

From there, three response patterns:

1. Fail fast. Stop the job on failure so nothing lands. Right for structural rules: if the primary key isn’t unique, downstream is going to be wrong no matter what you do next.
2. Quarantine. Filter the row-level outcomes on the evaluation result column, write passing rows to the target and failing rows to a separate prefix. Right for value-level rules where 2% bad records shouldn’t block 98% good ones. The important discipline is that somebody has to actually look at the quarantine prefix; an unmonitored quarantine is a delete with extra steps.
3. Warn and continue. Let everything land, publish the metrics, alert on the score. Right for new rules you don’t trust yet, and for the catalog-side ruleset.

Whichever you pick, get the failure out of Glue and into wherever your team actually looks. Results and anomaly predictions can be written back to catalog tables so they are queryable with plain SQL, and CloudWatch metrics plus EventBridge events give you the hook for paging or pausing a downstream pipeline. A Grafana or Datadog dashboard fed from those metrics beats a score in a console tab nobody opens.

Troubleshooting AWS Glue Data Quality on SaaS pipelines

“Input data does not include column X”

A rule references a column the engine cannot find. On SaaS data that means one of three things: the field disappeared from the source, your flattening step renamed it, or the Data Catalog is out of sync with the underlying table. Check the catalog schema against the landed files before touching the ruleset; rerunning the crawler resolves a surprising share of these.

A rule type isn’t recognised

DQDL is case sensitive and the rule set has grown over releases, so a rule copied from an older blog post can parse fine in your head and not at all in the engine. Check the rule name against the current DQDL rule type reference, and check that the feature is available on your Glue version. Anomaly detection in particular has a version floor.

A dynamic rule never fails

Almost always the empty-history problem described above. Look at the statistics for that metric in the Data Quality tab. If there are fewer recorded points than your last(k) asks for, the rule is not doing what you think. It will start working on its own; the mistake is trusting it in the meantime.

Evaluation is slow or the job runs out of memory

Each rule triggers analyzer computation over the dataset, and a large ruleset on a wide flattened SaaS table adds up fast. The transform exposes caching and observations-scope options to tune this. Before reaching for a bigger worker type, cut the ruleset down: most contain a dozen recommended rules nobody reads the results of.

A “column reference is ambiguous” error

Classic flattening artefact. Two nested paths collapsed to the same leaf name, so datechecked now matches two things. Fix it in the projection with explicit aliases rather than trying to disambiguate inside DQDL.

Common mistakes

• Accepting the recommended ruleset wholesale. Recommendations are generated from a sample of current data, so they encode today’s accidents as tomorrow’s requirements.
• Writing rules against nested data and assuming they cover the whole payload.
• Deploying a dynamic rule and treating it as armed on day one.
• Applying whole-table completeness thresholds to a polymorphic object instead of scoping with a where clause.
• Quarantining failed rows to a prefix nobody monitors.
• Writing RowCount rules against an exploded array without noticing the grain changed.
• Failing the job on every rule, including cosmetic ones, until the team starts rerunning with checks disabled.
• Keeping the ruleset in the console instead of in version control next to the job.

Best practices

• Split rules into blocking and reporting tiers, and be honest about which is which. A rule that always fails and never stops anything trains people to ignore the tab.
• Pair ColumnExists with ColumnCount. One protects the fields you named, the other catches the drift you did not anticipate.
• Add analyzers early, even for metrics you have no rule for yet. History has to accumulate before dynamic rules and anomaly detection are worth anything, and you cannot backfill it.
• Keep DQDL in Git alongside the job definition and deploy it with Terraform or CloudFormation. Rulesets are code and should move through the same review as the pipeline they guard.
• Namespace the evaluation context per source object so CloudWatch metrics and alerts can be routed per vendor.
• Exclude incident periods from the anomaly detection training window once the incident is resolved.
• Surface results where stakeholders already look. Writing outcomes back to catalog tables makes them queryable from Athena, which beats a score in a console tab.

Frequently asked questions

How much does AWS Glue Data Quality cost?

It bills on Glue DPU-hours like any other Glue workload, so cost is a function of evaluation time and DPUs used. The dimension that surprises people is anomaly detection, which consumes additional capacity per statistic analysed, so enabling it broadly across many columns and tables is where the bill moves. Model it against the current Glue pricing page; rates change and vary by region.

Can it check data before it lands in S3?

Yes, if the check runs inside the Glue job. The transform evaluates a DynamicFrame in memory, so you can validate after extraction and before the write. That is the placement that lets you stop a bad load rather than report on one.

Does it work on nested JSON from a SaaS API?

Not directly. DQDL does not evaluate nested or list-type columns, so you have to flatten the structure first and run the ruleset against the flattened frame. Plan the flattening deliberately, because your rules only ever see what the projection exposes.

How does it compare to Great Expectations, Soda or Monte Carlo?

Glue Data Quality’s real advantage is that it is already inside the runtime and the catalog: no extra infrastructure, no separate agent, one IAM story. The disadvantages are real too. DQDL is less expressive than Python-based assertions, the nested-data limitation is a hard wall, and it only covers what Glue can read. Great Expectations gives you more expressive checks at the cost of running it yourself; Monte Carlo and similar platforms add lineage and cross-warehouse coverage Glue does not attempt. Glue-centric stack, start here. Several engines, a standalone tool fits better.

Should I use the recommended rules?

As a starting draft, yes. As a ruleset, no. Recommendations are inferred from a sample of the data as it is today, which means any existing quality problem gets encoded as the expected baseline. Read every generated rule and delete the ones you would not have written yourself.

Can I reconcile row counts against the SaaS source itself?

Not directly; it only sees datasets it can read. The usual pattern is to have the extraction step record the count the API reported, land that alongside the data as a control table, then use RowCountMatch or AggregateMatch to compare the two. That turns “did we get everything” into a rule the engine can evaluate.

The one thing worth remembering

AWS Glue Data Quality is not really a data quality product. It is a way of writing down what you believe about a dataset so a machine can check it every run, and its value is entirely determined by whether those beliefs are written precisely enough to be wrong.

For SaaS sources that means two habits. Flatten deliberately and check immediately after, because your rules only see the projection. And treat every new dynamic or ML-based rule as unarmed until it has history, because a rule that cannot fail yet looks exactly like a rule that is passing.

Need help with data quality on your Glue pipelines?

I work with teams whose SaaS pipelines run green while the numbers drift. Things I can help with:

• Writing a DQDL ruleset for a specific SaaS object that catches schema drift instead of just restating the current schema.
• Designing the flattening and pivoting step for nested payloads and custom-field bags so the checks actually cover what matters.
• Setting up quarantine and fail-fast paths in Glue ETL, with the routing and alerting that makes a quarantine prefix something people act on.
• Cross-object reconciliation between SaaS extracts and control counts recorded at extraction time.
• Tuning anomaly detection and dynamic rules, including cleaning incident periods out of the training window.
• Getting data quality results out of the console and into Grafana, CloudWatch alarms or an Athena-queryable table.

If you have a ruleset that keeps passing while the data drifts, send me the DQDL and a schema dump. That is usually enough to see where the gap is.

The post AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed appeared first on John Nessime.

The frustrating part is that the pipeline is fine. Webhooks are arriving, Lambda is running clean, S3 has the files, Athena returns rows. Nothing is broken in the way monitoring understands “broken”. The pipeline is faithfully recording something that is no longer true.

This post is about building a Shopify sales dashboard with AWS that survives that conversation. Not the wiring, which is well documented and mostly straightforward, but the design decisions that determine whether your numbers still hold up six months in. I’ll cover the three ingestion paths and when each one is the right call, why append-only pipelines drift, how to lay out S3 and Athena so recomputation is cheap, and what to do when the totals are already wrong.

Why a Shopify sales dashboard with AWS drifts from the admin

Here’s the thing that catches almost everyone: a Shopify order is not an event, it’s a mutable record.

An event pipeline assumes facts are immutable once written. A payment happened. A shipment left. You append it, you never touch it again, and the sum of the log is the truth. That model is why streaming architectures are so clean, and it’s exactly wrong for order data.

An order created on Monday can be edited on Tuesday, partially refunded on Friday, and fully refunded three weeks later. Every one of those changes belongs, financially, to Monday. If your pipeline appends the orders/create payload and never revisits it, Monday’s revenue is frozen at the moment of checkout and it will only ever be too high.

This is the invisible failure. Nothing alerts. No queue backs up. Your dashboard is confidently wrong, and the gap widens roughly in proportion to your return rate. A store with a two percent return rate takes a long time to notice. A fashion store running thirty percent returns notices in about a month, usually via an angry accountant.

The four adjustments that move historical numbers

• Refunds. Full or partial. A refund carries its own created_at, which is when the money moved back. The order it belongs to has a different, earlier date. You need both, and which one you attribute to depends on whether finance wants cash-basis or order-basis reporting. Ask before you build.
• Order edits. A merchant adds a line item or adjusts a quantity after the fact. The original payload is now stale. Shopify exposes both the original and the current totals precisely because of this.
• Cancellations. A cancelled order keeps existing in the API. If you filter only on payment status you will happily keep counting it.
• Test and draft orders. Test orders carry a flag marking them as such. Nobody remembers to filter these until a QA run during a quiet week produces a suspicious spike.

The design consequence is simple to state and annoying to implement: your pipeline must be able to recompute any past day. Every storage and partitioning decision below follows from that one requirement.

Getting data out of Shopify: three paths, three trade-offs

Before anything else: new Shopify apps are built on the GraphQL Admin API. The REST Admin API has been designated a legacy API and new public apps must use GraphQL. If you’re starting fresh, start there. If you inherited a REST integration, it probably still runs, but you’re on borrowed time and you should plan the migration rather than discover the deadline.

Path 1: EventBridge partner event source

Shopify can deliver webhooks straight into an Amazon EventBridge partner event bus in your account. No public endpoint, no API Gateway, no HMAC verification code, because verification only applies to HTTPS deliveries. Shopify’s own docs confirm EventBridge and Pub/Sub deliveries skip it.

You create the source in the Shopify app configuration using your AWS account ID, region and a source name, then associate it with an event bus in the EventBridge console and write rules to route it. The address you register with Shopify is the partner event source ARN, not the event bus ARN. That distinction accounts for a large share of the “I set it up and nothing arrives” threads on the Shopify forums.

A rule matching everything from the Shopify partner source looks like this. Start broad, then narrow once you’ve seen the real shape of an event:

{
  "source": [ { "prefix": "aws.partner/shopify.com" } ]
}

Send that to an SQS queue with a dead-letter queue attached rather than straight to Lambda. Buffering gives you a replay buffer when a downstream deploy goes wrong, and the DLQ means a bad payload parks itself instead of poisoning the whole rule. This is the path I reach for first for anything already on AWS.

Path 2: HTTPS webhooks into API Gateway and Lambda

The conventional route, and the right one if you need webhook delivery outside AWS too, or you want the payloads to pass through something you fully control. The cost is that you now own an internet-facing endpoint and the HMAC verification on it.

Verify against the raw request body, before any JSON parsing. Re-serialising the payload changes byte-for-byte content and the signature will never match. Use a constant-time comparison so the check doesn’t leak timing information:

import base64, hashlib, hmac

def verify(raw_body: bytes, header_hmac: str, secret: str) -> bool:
    digest = hmac.new(secret.encode(), raw_body, hashlib.sha256).digest()
    computed = base64.b64encode(digest).decode()
    return hmac.compare_digest(computed, header_hmac)

Shopify sends the signature in the X-Shopify-Hmac-SHA256 header, base64-encoded, computed with your app’s client secret over the raw body. Store that secret in Secrets Manager or as an SSM SecureString parameter, not in a Lambda environment variable.

Acknowledge fast. Shopify’s timeout is short and it retries with backoff over a finite window, so a handler that does real work inline will generate a wall of duplicate deliveries during a flash sale, exactly when you can least afford it. Return 2xx immediately, do the work asynchronously.

Path 3: scheduled GraphQL bulk pull

Webhooks give you low latency. They do not give you completeness. Anything that fails past its retry window is gone, and Shopify will eventually remove a subscription that keeps failing. That’s a silent data loss mode with no local symptom at all.

So run a scheduled reconciliation pull alongside the stream. Shopify’s GraphQL bulk operations are built for this: you submit a query, it runs asynchronously, and you fetch a JSONL result file when it finishes. That’s the right tool for backfills and nightly catch-up, rather than paginating thousands of pages against a points-based rate limiter and getting throttled halfway through.

A nightly job that re-pulls the last seven to fourteen days and overwrites those partitions costs almost nothing and quietly fixes every category of drift described above. If you build one thing from this post, build that.

Decide what “revenue” means before you write a line of SQL

This is where most reconciliation arguments actually live, and it isn’t an engineering problem at all until you’ve had the conversation.

Shopify’s own sales reporting builds total sales from gross sales, minus discounts, minus returns, plus taxes and shipping. Gift card sales sit outside that in a separate finance report. If your dashboard sums order totals and calls it revenue, you have built a different metric with the same name, and it will disagree with the admin forever no matter how good your pipeline is.

Write the definition down. Put it in the dashboard as a tooltip. When someone challenges a number, you want the argument to be about the definition, not about whether your infrastructure works.

The currency trap

If the store sells in more than one currency, the money fields split in two. Shopify exposes totals as a set containing both shop_money and presentment_money: the amount in the store’s base currency, and the amount the customer actually saw and paid.

Sum the presentment amounts across a multi-currency store and you get a number with no meaning at all, euros and yen added together as if they were the same unit. For a single reporting figure you want the shop-currency side. Keep the presentment amount and its currency code in the table anyway, because the day someone asks “how much did we actually sell in Germany”, you’ll want it and it is painful to backfill.

One caveat worth knowing: orders created through the API rather than through checkout can behave differently from native multi-currency checkout orders. If your store takes orders from an ERP or a marketplace integration, spot-check a few of those specifically.

Storage layout: partition by order date, never by arrival date

Two layers in S3. Keep them separate and keep them honest about what they are.

1. Raw. Every payload exactly as received, partitioned by ingestion date. Append-only, never edited. This is your audit trail and your rebuild source. Lifecycle it to a colder storage class after a few months, don’t delete it.
2. Curated. One row per order representing current state, in Parquet, partitioned by order date. This is what the dashboard queries. It is derived, disposable and rewritable.

The partitioning choice on the curated layer is the load-bearing decision in the whole design. If you partition by arrival date, which is what Amazon Data Firehose does by default because it buckets on the moment it writes the file, then a refund that arrives three weeks late lands in today’s partition. Correcting Monday now means finding and rewriting fragments scattered across twenty other partitions. Partitioned by order date, correcting Monday means overwriting exactly one prefix.

Firehose can do this with dynamic partitioning, which routes records by keys inside the payload rather than by write time. If you’re not using Firehose, extract the order date in your Lambda and write the prefix yourself.

Use partition projection so Athena stops guessing

The default Glue Data Catalog approach means running a crawler or issuing MSCK REPAIR TABLE to register new partitions. Forget one and you get a query that silently returns nothing for recent days. Nobody notices until Monday.

Partition projection removes the metastore lookup entirely. You tell Athena the shape of the partition keys and it calculates the prefixes at query time:

CREATE EXTERNAL TABLE shop_orders (
  order_id             bigint,
  order_number         string,
  created_at           timestamp,
  financial_status     string,
  cancelled_at         timestamp,
  is_test              boolean,
  total_shop           decimal(12,2),
  shop_currency        string,
  total_presentment    decimal(12,2),
  presentment_currency string
)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://your-bucket/curated/orders/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.order_date.type' = 'date',
  'projection.order_date.format' = 'yyyy-MM-dd',
  'projection.order_date.range' = '2019-01-01,NOW',
  'projection.order_date.interval' = '1',
  'projection.order_date.interval.unit' = 'DAYS',
  'storage.location.template' =
    's3://your-bucket/curated/orders/order_date=${order_date}/'
);

Set the range start to your store’s actual first order month. Projection generates every prefix in the range, so a range starting a decade too early makes wide scans slower for no benefit.

Parquet matters here for the same reason. Athena bills on bytes scanned, so a columnar format with good compression cuts the bill directly, and a dashboard that only ever selects six columns from a forty-column table never touches the rest.

Net sales in one query

With refunds in their own table keyed by order and carrying their own date, attributing them back to the original order day is a left join and a subtraction:

SELECT
    o.order_date,
    SUM(o.total_shop)                                AS gross_shop,
    SUM(COALESCE(r.refunded_shop, 0))                AS refunded_shop,
    SUM(o.total_shop - COALESCE(r.refunded_shop, 0)) AS net_shop
FROM shop_orders o
LEFT JOIN (
    SELECT order_id, SUM(amount_shop) AS refunded_shop
    FROM shop_refunds
    GROUP BY order_id
) r ON r.order_id = o.order_id
WHERE o.order_date BETWEEN '2025-01-01' AND '2025-01-31'
  AND o.is_test = false
  AND o.cancelled_at IS NULL
GROUP BY o.order_date
ORDER BY o.order_date;

Note the two filters doing quiet work at the bottom. Those two lines are the difference between a number finance accepts and a number they don’t.

Choosing the dashboard layer

Once the data is correct, this part is genuinely a preference. All of these work.

• Amazon QuickSight, now delivered as part of Amazon Quick Suite, is the least-friction option if you’re already in AWS. Its in-memory SPICE layer means viewers aren’t firing an Athena query per chart interaction, which controls both latency and scan cost. Per-viewer pricing tends to be the deciding factor either way, so model it for your actual audience size before committing.
• Grafana with the Athena data source is a good fit if you’re already running Grafana for infrastructure and want commercial and operational panels on one screen. Grafana Cloud removes the hosting question if you’d rather not run it.
• Power BI makes sense when the finance team already lives in Microsoft 365 and models in DAX. The cross-cloud hop is real but manageable.
• Metabase or a self-hosted alternative on a small VPS from a provider like InterServer or Hetzner is the pragmatic answer for a handful of internal viewers, where per-seat BI licensing costs more than the entire pipeline.

The honest trade-off: managed BI costs more per month and saves you from becoming the person who patches the reporting server. Self-hosting inverts that. Neither is wrong, but pick deliberately rather than by inertia.

Troubleshooting: symptom to cause

Totals are consistently higher than the Shopify admin

Almost always refunds, cancellations or test orders. Check in that order. If the gap grows with the age of the reporting window, it’s refunds. If it’s a fixed offset on specific days, look for test orders or a QA run.

Totals are lower, and recent days are missing rows

Either partitions aren’t registered, which projection fixes permanently, or the webhook subscription has been dropped after repeated delivery failures. Check the subscription still exists before you go digging through Lambda logs. A nightly bulk pull would have masked this, which is another argument for having one.

Orders appear twice

Shopify’s delivery model is at-least-once, not exactly-once, and you may also have more than one subscription on the same topic. Deduplicate on the delivery ID header before you touch anything else, and make the write itself idempotent so a duplicate is a no-op rather than a second row.

Numbers are right on the daily view, wrong on the monthly

Timezone. Order timestamps carry an offset; your partition key is a date string. If you derive the date in UTC and the store reports in a local timezone, orders near midnight land on the wrong day. That averages out over a month, which is exactly why the discrepancy hides until month boundaries.

Athena costs jumped without more data

Someone built a dashboard with a filter that doesn’t hit the partition column, so every panel refresh scans the full table. Look at bytes scanned per query and check whether the BI tool is caching results or re-querying on every interaction.

Common mistakes

• Treating orders as immutable events and never revisiting a past day.
• Partitioning on arrival time because that’s the default, then discovering corrections are expensive.
• Verifying the HMAC against a re-serialised body instead of the raw bytes.
• Doing real work inside the webhook handler, generating duplicates under load.
• Summing presentment amounts across currencies.
• Registering the event bus ARN with Shopify instead of the partner event source ARN.
• Relying on webhooks alone with no scheduled reconciliation.
• Shipping a “revenue” number without ever defining what it includes.

Best practices

• Keep raw and curated layers separate. Raw is append-only; curated is rewritable.
• Make every partition idempotently rebuildable from raw. Test that path deliberately, before you need it.
• Buffer through SQS with a dead-letter queue. Free replay, free isolation of bad payloads.
• Run a nightly bulk pull over a rolling window and overwrite those partitions.
• Use partition projection. It removes an entire category of silent failure.
• Alarm on the absence of events, not just on errors. A CloudWatch alarm on zero orders processed in an hour during business hours catches broken subscriptions the same day.
• Store the API secret in Secrets Manager and scope the Lambda role to the exact prefixes it writes.
• Publish a reconciliation panel comparing your total to the admin’s for the same window. Surfacing the gap builds more trust than hiding it.

Frequently asked questions

Do I need a data warehouse, or is S3 and Athena enough?

For a single store’s order data, S3 with Athena is almost certainly enough, and it’s cheaper because you pay per query rather than for a running cluster. Redshift starts to earn its place when you’re joining Shopify data against several other large sources, or when concurrent query load makes Athena’s queue times noticeable.

How near-real-time can this be?

Events land within seconds. The practical floor is your buffering window, since writing one tiny file per order gives you a small-files problem that ruins query performance. A few minutes of buffering is the usual compromise. If you genuinely need sub-minute order counts, put a live counter in DynamoDB alongside the analytical pipeline rather than trying to make the data lake do both jobs.

Which webhook topics should I subscribe to?

At minimum, order creation, order update, order cancellation and refund creation. Update and refund topics are the ones people skip, and they’re exactly the ones carrying the corrections. Subscribe to fewer topics than you think you need and add rather than subscribing to everything, since every extra topic is volume you pay to store and process.

Can I skip AWS and use a connector tool?

Yes, and for many stores that’s the right answer. A managed connector into a hosted warehouse gets you a working dashboard in an afternoon. You’re paying a monthly fee to avoid owning any of this, and trading away control over the data model. Building it on AWS wins when you need Shopify data joined to systems the connector doesn’t cover, or when row-based connector pricing outgrows the infrastructure cost.

How do I backfill historical orders?

Use a GraphQL bulk operation rather than paginating the API. Submit the query, poll for completion, then stream the JSONL result into your raw bucket and run the same transformation your live pipeline uses. If backfill and live processing use different code paths, they will diverge, and you’ll spend an afternoon working out which one is lying.

What does a setup like this cost to run?

For a typical single store, the pipeline itself is small money: Lambda invocations, a few gigabytes in S3, and Athena billed on bytes scanned, which partitioning and Parquet keep low. The BI seats are usually the largest line item, which is why the dashboard layer decision deserves more thought than the ingestion one. Model it against current published rates rather than trusting any figure you read in a blog post, including this one.

The one thing to take away

A Shopify sales dashboard with AWS doesn’t fail because the pipeline breaks. It fails because the pipeline keeps working perfectly on data that has since changed underneath it.

Design for correction from the first commit. Partition by order date, keep the raw layer so you can always rebuild, run a scheduled pull to catch what the stream missed, and agree on what revenue means before anyone builds a chart. Do that and the Monday morning screenshot comparison becomes a non-event, which is the highest praise a reporting pipeline ever gets.

Need help with your Shopify data pipeline on AWS?

I design and build ecommerce data pipelines and reporting stacks on AWS. Typical engagements look like:

• Working out why an existing Shopify dashboard disagrees with the admin, and fixing the root cause rather than patching the query
• Building the ingestion layer end to end: EventBridge or API Gateway, Lambda, SQS with dead-letter handling, and a scheduled GraphQL bulk reconciliation job
• Designing the S3 layout, Glue schema and Athena tables so past days can be recomputed cheaply and partitions never go missing
• Migrating REST Admin API integrations to GraphQL before the deadline forces the issue
• Building the dashboard itself in QuickSight, Grafana or Metabase, including the metric definitions finance will actually sign off on
• Cutting Athena scan costs and BI licensing on a reporting stack that has grown more expensive than anyone planned

If you’re in the middle of one of these, send me the actual thing: the Athena query, the S3 prefix layout, the two totals that don’t match. It’s a much faster conversation than describing it in the abstract.

The post Shopify Sales Dashboard with AWS: Build One That Actually Reconciles appeared first on John Nessime.

That is usually the moment someone says “let’s just pull it into a warehouse.” A Jira analytics pipeline built on AWS Lambda, S3 and Athena is a reasonable answer to that, and it is genuinely cheap to run. It is also easy to build a version that looks correct for two weeks and then quietly reports numbers that are thirty percent wrong.

This post covers the extraction and modelling problems that actually cost you time: the search endpoint Atlassian removed, the change history that truncates without erroring, the incremental load pattern that duplicates rows across partitions, and the S3 layout decisions that decide whether Athena costs you pennies or hundreds. Code where it clarifies something, and honest notes on what I would skip.

Where Jira’s own reporting genuinely stops

Give the built-in tooling its due first. Jira’s velocity, burndown and control charts are fine for a single team inspecting its own recent work, they need no infrastructure, and they update instantly. Marketplace apps like eazyBI and Custom Charts cover a lot of ground without you writing a line of Python. If your question is “how did this sprint go,” you do not need a pipeline.

Where it stops is anything that crosses a boundary. Comparing lead time across projects that use different workflows. Joining ticket data to deploy events from your CI system or incident data from PagerDuty. Retaining a consistent view of history after someone renames a status or archives a board. Answering a question nobody anticipated when the board was configured. Those need the raw data somewhere you control, in a shape you decide.

The shape of the pipeline

The architecture is unremarkable, which is the point:

1. EventBridge Scheduler triggers a Lambda function on a schedule.
2. Lambda reads a Jira API token from Secrets Manager and pages through the Jira Cloud REST API.
3. It writes Parquet files to S3, partitioned by load date.
4. The Glue Data Catalog holds the table definitions, with partition projection so nothing has to crawl.
5. Athena queries S3 directly. Grafana, Power BI, Metabase or QuickSight sit on top of Athena.

No cluster, no always-on database, nothing to patch. The whole thing costs about as much as a small EC2 instance for a mid-sized Jira site, and most of that is S3 storage. Terraform or CloudFormation to define it, GitHub Actions to deploy it.

The complexity is not in the wiring. It is in four places, and they are worth taking in order.

Failure family 1: the search endpoint you were probably going to use is gone

Almost every Jira extraction tutorial and a good number of client libraries still reach for GET /rest/api/3/search. Atlassian removed it from Jira Cloud. It returns 410 Gone. If you are copying a script from a blog post that predates the change, this is the first thing that breaks, and at least it breaks loudly.

The replacement is /rest/api/3/search/jql, available as both GET and POST. Use POST for anything real, because JQL strings get long and you avoid URL encoding entirely. Three behavioural changes matter more than the URL:

Pagination is cursor-based, and there is no total

startAt is gone. You get a nextPageToken back and hand it to the next request. There is no total in the response, which means any progress bar, any “expected N rows” sanity check, and any loop that terminated on startAt >= total needs rewriting.

If you only need a count, there is a separate operation, POST /rest/api/3/search/approximate-count, which takes a JQL body and returns an approximate figure without paging through results. It is genuinely useful as a reconciliation check: run it before extraction, compare against the row count you actually wrote, and alert on a large gap.

There have been persistent community reports of isLast behaving unreliably on this endpoint, including tokens that chain without ever terminating. Do not trust isLast as your loop condition. Terminate on the absence of nextPageToken, and put a hard page cap in as a circuit breaker so a bad token cannot burn your entire Lambda budget in one invocation.

def search_issues(session, base_url, jql, fields, max_pages=2000):
    """Page through /search/jql. Terminates on missing nextPageToken,
    not on isLast, which has been reported as unreliable."""
    token = None
    for _ in range(max_pages):
        body = {"jql": jql, "fields": fields, "maxResults": 100}
        if token:
            body["nextPageToken"] = token
        r = session.post(f"{base_url}/rest/api/3/search/jql",
                         json=body, timeout=60)
        r.raise_for_status()
        page = r.json()
        for issue in page.get("issues", []):
            yield issue
        token = page.get("nextPageToken")
        if not token:
            return
    raise RuntimeError("page cap hit, refusing to loop further")

You have to ask for fields explicitly

The new endpoint does not hand you every field by default. Omit fields and you get essentially nothing back. This is the failure that looks like success: the pipeline runs, files land in S3, row counts look plausible, and every analytical column is null.

Be explicit and be narrow. Every field you request costs response size, and Jira sites accumulate hundreds of custom fields nobody uses. Name what you need:

FIELDS = [
    "summary", "status", "issuetype", "project", "priority",
    "assignee", "reporter", "created", "updated", "resolutiondate",
    "labels", "components", "parent",
    "customfield_10016",   # story points on this site, verify yours
]

Custom field IDs are per-site. Do not hardcode one you read in someone else’s blog post. Pull /rest/api/3/field once, find the field by name, and either store the mapping in config or resolve it at runtime and log what it resolved to. When a Jira admin rebuilds a field, an ID-based pipeline goes null and a name-resolving pipeline keeps working.

Rate limits and the fifteen-minute wall

Jira Cloud applies cost-based rate limiting and returns HTTP 429 when you exceed it, typically with a Retry-After header. Respect that header rather than inventing your own backoff. A naive retry loop that ignores it turns a brief throttle into a sustained one.

Lambda’s hard ceiling is fifteen minutes. A full historical backfill of a large Jira site will not finish in one invocation, and the ugly failure mode is a function that times out at minute fifteen having written half its data with no record of where it stopped. Two ways out:

• Shard the work. Fan out one Lambda invocation per project key, or per month of created date. Each one is small, independently retryable, and finishes well inside the limit.
• Checkpoint and continue. Persist the current nextPageToken to DynamoDB or S3 after each page. When the function is close to its deadline, stop cleanly and let Step Functions re-invoke it from the checkpoint.

Sharding is simpler and I reach for it first. Checkpointing is what you need when a single project is itself too large. Either way, watch Lambda’s ephemeral storage: the default /tmp allocation is 512 MB, and buffering a large Parquet write there will fail before your API calls do. Raise it or stream to S3 in chunks.

Failure family 2: the change history is the whole point, and it truncates silently

Current issue state answers almost none of the interesting questions. How long a ticket spent waiting for review, how many times it bounced back from QA, when it actually entered development rather than when someone remembered to drag the card, all of that lives in the changelog. Without it you have a list of tickets. With it you have a process.

Here is the trap. Requesting an issue with expand=changelog returns a capped number of history entries, commonly the first hundred, and it does not tell you it truncated. Well-worn tickets with lots of field edits blow past that easily. Your data does not error, it just quietly loses the later transitions, which are usually the ones near completion. Cycle time comes out looking better than reality.

Two correct approaches:

• GET /rest/api/3/issue/{issueIdOrKey}/changelog and page it properly. Correct, but it is one request per issue, which is brutal against rate limits on a large site.
• POST /rest/api/3/changelog/bulkfetch, which accepts issueIdsOrKeys and an optional fieldIds filter, with the same nextPageToken pagination. Far fewer round trips.

Bulk fetch is the one I would use, with a caveat: it has carried an experimental designation, so pin your expectations and keep the per-issue path available as a fallback. Filter fieldIds to status if status history is all you model. That cuts the response size enormously, because most changelog volume is description edits and label churn nobody analyses.

Store changelog as its own narrow table, one row per field change, not nested inside the issue record. Athena can handle nested structures, but flat is dramatically easier to reason about in SQL and much cheaper to scan.

Failure family 3: the incremental load that duplicates everything

This is the one that bites hardest, because nothing fails. Everything runs green for weeks and the numbers are wrong the entire time.

The obvious incremental design is a watermark: track the last successful run, then pull updated >= watermark on each run and append the results to S3. It is the standard pattern and it works for immutable event data.

Jira issues are not immutable. A ticket created in January and touched every week appears in every weekly extract. If you append each run into its own partition, that one issue now exists as a row in twenty partitions. Athena is doing exactly what you asked: SELECT count(*) FROM jira_issues WHERE status = 'Open' counts twenty things. Your open bug count is inflated, your throughput is inflated, and the inflation is proportional to how much a ticket gets edited, which correlates with how contentious it was. The busiest work is the most over-counted.

Nobody catches this from the dashboard, because the numbers move in the right direction and look approximately sane. It surfaces months later when someone reconciles against a JQL query in Jira and the two disagree.

Pick one of three fixes, deliberately

Full snapshot per load, dedupe at read time. Keep every version, partition by load_date, and always read through a view that takes the latest row per issue key. Storage is cheap, history is free, and you can answer “what did the board look like in March” without any extra machinery. The cost is that every query pays for the deduplication.

CREATE OR REPLACE VIEW jira_issues_current AS
SELECT * FROM (
  SELECT
    i.*,
    ROW_NUMBER() OVER (
      PARTITION BY issue_key
      ORDER BY load_date DESC, updated DESC
    ) AS rn
  FROM jira_issue_snapshot i
  WHERE load_date >= date_format(current_date - interval '7' day, '%Y-%m-%d')
) WHERE rn = 1;

The load_date filter inside the view matters. Without it the deduplication window scans the entire table on every query, which is the single most common way a cheap Athena setup becomes an expensive one.

Overwrite the affected partitions. Partition by something stable, usually issue created month, and rewrite whole partitions when any issue in them changes. Clean reads, no dedupe cost, but you now own read-modify-write logic in Lambda and a concurrency problem if two runs overlap.

Use an ACID table format. Apache Iceberg gives you real MERGE INTO semantics on S3, and Athena supports it natively. This is the right answer if you are already running Iceberg elsewhere or if the pipeline will grow to a dozen sources. It is not worth adopting solely to load one Jira site, because you inherit compaction and snapshot expiry as ongoing maintenance.

For a single Jira site feeding a handful of dashboards, snapshot plus a dedupe view is what I would build. It has the fewest moving parts and it gives you point-in-time history as a side effect, which you will want the first time someone asks a retrospective question.

Deletions and moves

A JQL watermark query never returns deleted issues, so they persist in your data forever. Same for issues moved out of scope or into an archived project. Periodically reconcile: pull the full set of issue keys with a minimal fields list, compare against what you hold, and mark the difference. Monthly is usually enough. Skip this and your historical counts drift upward permanently.

Failure family 4: the S3 layout that decides your Athena bill

Athena’s standard pricing model bills on bytes scanned, rounded up, with a small per-query minimum. DDL statements are free. That means your storage layout, not your SQL, is what determines cost.

Three things do almost all the work:

• Write Parquet, not JSON. Columnar storage lets Athena read only the columns your query touches. A dashboard selecting five columns from a forty-column table scans a small fraction of what the equivalent JSON would. Snappy or Zstd compression on top of that.
• Avoid tiny files. A Lambda that runs hourly and writes one small file per invocation produces thousands of objects. Athena spends more time opening files than reading them. Batch writes so files land in the low hundreds of megabytes, or run a periodic compaction job.
• Use partition projection. This is the one people skip and then wonder why queries have a fixed few-second overhead.

Partition projection lets Athena calculate partition locations from table properties instead of calling GetPartitions against the Glue Data Catalog. On a table with a couple of years of daily partitions, that lookup is real latency on every single query, and it grows as the table does. Projection removes it, and removes your need for a Glue crawler entirely, which is one less scheduled job and one less IAM role.

CREATE EXTERNAL TABLE jira_issue_snapshot (
  issue_id        string,
  issue_key       string,
  project_key     string,
  issue_type      string,
  status          string,
  status_category string,
  assignee_id     string,
  created         timestamp,
  updated         timestamp,
  resolutiondate  timestamp,
  story_points    double
)
PARTITIONED BY (load_date string)
STORED AS PARQUET
LOCATION 's3://example-jira-lake/issue_snapshot/'
TBLPROPERTIES (
  'projection.enabled'                = 'true',
  'projection.load_date.type'         = 'date',
  'projection.load_date.format'       = 'yyyy-MM-dd',
  'projection.load_date.range'        = 'NOW-3YEARS,NOW',
  'projection.load_date.interval'     = '1',
  'projection.load_date.interval.unit'= 'DAYS',
  'storage.location.template'         =
    's3://example-jira-lake/issue_snapshot/load_date=${load_date}'
);

One caution that surprises people: projection describes partitions Athena will look for, not partitions that exist. Set a range wider than your data and queries without a load_date filter will probe empty prefixes. Keep the range tight to what you actually hold.

Finally, set BytesScannedCutoffPerQuery on the Athena workgroup. It kills any query that exceeds a scan threshold. One analyst running SELECT * against three years of data in a BI tool’s preview pane is the classic surprise line item, and this stops it at the source.

Modelling time in status

Once status changelog rows are landed flat, the core metric is a window function. Each transition’s duration is the gap to the next transition on the same issue:

WITH transitions AS (
  SELECT
    issue_key,
    to_status,
    changed_at,
    LEAD(changed_at) OVER (
      PARTITION BY issue_key ORDER BY changed_at
    ) AS next_changed_at
  FROM jira_changelog
  WHERE field_id = 'status'
    AND load_date >= date_format(current_date - interval '90' day, '%Y-%m-%d')
)
SELECT
  issue_key,
  to_status,
  SUM(date_diff('second', changed_at,
                COALESCE(next_changed_at, current_timestamp))) / 3600.0
    AS hours_in_status
FROM transitions
GROUP BY issue_key, to_status;

The COALESCE handles the current status, which has no successor transition. Two modelling decisions to make consciously: whether to subtract non-working hours, and how to treat a ticket that moves backwards through the workflow. Both are business questions, not technical ones, and both should be settled in a documented view rather than reimplemented in each dashboard.

Map raw status names to a stable category early. Teams rename statuses constantly, and a metric keyed on the literal string breaks the moment someone changes “In Review” to “Peer Review”. A small mapping table joined at query time keeps history comparable.

Troubleshooting

• Every analytical column is null. You did not pass fields, or you passed a custom field ID from another site. Log the resolved field list on every run.
• Counts higher than the same JQL in Jira. Duplicate rows across partitions. Check whether your query goes through the dedupe view or straight at the base table.
• Extraction loops forever. Do not terminate on isLast. Terminate on missing nextPageToken and keep a page cap.
• HTTP 410 from the API. You are still calling the removed /rest/api/3/search. Check your client library version too, not just your own code.
• Cycle times suspiciously low. Truncated changelog. Move to the dedicated changelog endpoint or bulk fetch.
• Athena returns zero rows but the files are there. Partition projection range does not cover the partition, or the S3 prefix does not match storage.location.template. Compare a real object key against the template character by character.
• Queries slow down as the table grows. Either you have no projection and Glue lookups dominate, or you have accumulated small files.
• Authentication failures overnight with no deploy. API token expired or was revoked. Alert on the specific status code rather than on “run failed”.

Common mistakes

• Appending incremental extracts without a deduplication strategy.
• Building on current issue state and adding changelog later. Retrofitting history is far more work than including it from the start.
• Hardcoding custom field IDs copied from documentation.
• Storing the API token in a Lambda environment variable instead of Secrets Manager.
• Running an hourly schedule when the dashboard is read once a day. You pay in small files and API quota for freshness nobody uses.
• Letting BI tools query base tables directly instead of curated views.
• Never reconciling against Jira. A scheduled check comparing approximate count to your row count catches drift within a day.

Best practices for a Jira analytics pipeline that survives

• Land raw API responses to S3 before transforming. When your parsing is wrong, and it will be, you replay from raw rather than re-hammering the API.
• Make the load idempotent. Re-running for the same date should produce the same result, not a second copy.
• Define the pipeline in Terraform or CloudFormation and deploy it from CI. The IAM policy for Lambda and Athena is fiddly and you do not want to rebuild it from memory.
• Alert on a run that succeeds with zero rows, not just on runs that error. Silent empty loads are the more common failure.
• Scope the Jira token to a service account with read access to exactly the projects you need.
• Put an S3 lifecycle policy on the raw zone. It grows faster than you expect and nobody queries last year’s raw JSON.
• Expose curated views, not tables, to Grafana, Metabase, Power BI or QuickSight, and enable Athena’s query result reuse for repeated dashboard loads.

Frequently asked questions

Should I use Lambda or Glue for Jira extraction?

Lambda, for an API-driven pull. The work is mostly waiting on HTTP responses, which Spark’s distributed compute does nothing for, and you would be paying for a Glue job’s minimum billing on something that is idle. Glue earns its place downstream, if you have heavy joins across several sources. Athena CTAS often covers that too.

How often should the pipeline run?

Match the decision cadence, not the data cadence. Delivery metrics are reviewed weekly or in sprint ceremonies, so daily is almost always enough and produces far better file sizes. Reserve hourly for something with a real-time consumer, and know that you are buying that freshness with small-file overhead and API quota.

Can I use webhooks instead of polling?

You can, and for near-real-time reaction it is the right tool. For analytics it is a poor primary source, because a missed delivery leaves a permanent hole you have no way to detect. The pattern that works is webhooks for freshness plus a scheduled reconciliation pull as the source of truth. If you only build one, build the scheduled pull.

Does this work with Jira Data Center or Server?

The AWS half is identical. The extraction half is not. Cursor pagination and the /search/jql endpoint are Cloud-only changes, so self-hosted instances still use the older offset-based /rest/api/2/search. Write the extractor behind an interface if you need to support both, and expect the auth model to differ as well.

How much does an Athena-based Jira pipeline cost to run?

The mechanism matters more than any figure I could quote. You pay for S3 storage, Lambda invocation time, and Athena per byte scanned with a small per-query minimum. Jira issue data compresses extremely well as Parquet, so storage stays small. The variable is dashboard query volume, which is why partitioning and workgroup scan limits matter. Model it against current AWS rates and your own expected query count.

Why not just use eazyBI or a Jira reporting app?

Often you should. If your questions stay inside Jira, a Marketplace app gets you there in an afternoon with no infrastructure. The case for a pipeline is joining Jira to data that lives elsewhere, retaining history the app does not, or needing your data in a warehouse you already run. Build the pipeline when the app has actually failed you, not in anticipation.

Should I load into Redshift instead of querying S3?

Only if you have concurrent BI users hitting the same tables constantly and Athena’s per-query latency is a real complaint. Jira data volumes are small, the query pattern is bursty, and Athena’s zero-idle-cost model fits that far better. Redshift makes sense as a consolidation layer across many sources, not for one issue tracker.

The one thing to carry away

The hard part of a Jira analytics pipeline is not the AWS wiring. Lambda, S3, Glue and Athena will be working within a day. The hard part is that Jira issues are mutable, so an incremental load that appends is an incremental load that duplicates, and it does so without a single error in your logs.

Decide your deduplication strategy before you write the first extract, not after someone reconciles a dashboard against JQL and finds a gap. Snapshot with a read-time dedupe view, partition projection so queries stay fast as history accumulates, and a scheduled reconciliation check that alerts on drift. Get those three right and the rest is plumbing.

Need this built or fixed?

I design and run serverless data pipelines on AWS, and Jira extraction has more sharp edges than most sources. Things I can help with directly:

• Migrating an extractor off the removed /rest/api/3/search endpoint to cursor-based /search/jql without losing rows in the cutover
• Auditing an existing Jira pipeline for duplicate rows and reconciling your numbers back against JQL
• Building changelog-based cycle time and time-in-status models that survive workflow renames
• Restructuring an S3 layer with Parquet, partition projection and file compaction to cut Athena scan costs
• Packaging the whole thing as Terraform or CloudFormation with CI deployment and least-privilege IAM
• Connecting Athena to Grafana, Power BI, Metabase or QuickSight with curated views instead of raw tables

Send me your extractor code, a Glue table definition, or an Athena query that is scanning more than it should, and I will tell you what I would change.

The post Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything) appeared first on John Nessime.

That gap between what you think you are paying for and what you are actually paying for is what makes Redshift cost optimization awkward in a SaaS analytics product. You are not running one nightly batch against a warehouse that sleeps the rest of the day. You are serving hundreds of small, latency-sensitive queries that fire whenever a customer opens a dashboard, plus ingestion, plus whatever your BI layer and your connection pool are doing when nobody is watching.

This post covers the levers that genuinely move that number: how the meter works, why idle-looking connections still bill, how to work out which tenant is expensive, and which isolation choices cost more than they save. Where the popular advice is wrong for SaaS specifically, I will say so.

How Amazon Redshift actually charges you

Three buckets, and they behave very differently.

• Compute. On Redshift Serverless this is RPU-hours, metered per second. On provisioned clusters it is node-hours, plus separate line items for concurrency scaling and Spectrum.
• Storage. Redshift Managed Storage, billed by GB per month, independent of compute. Snapshots are storage too.
• Everything else. Cross-region data sharing and snapshot replication, machine learning, data transfer outside the usual in-region S3 paths.

In a SaaS analytics workload compute dominates, often overwhelmingly. And the important part: compute is a function of how long the warehouse is awake and at what capacity, not how many rows you touched. Two teams can scan identical data volumes and get bills that differ by a factor of five, purely because of how their queries arrive.

The billing mechanic that catches SaaS teams out

Read the serverless billing notes properly once and a lot of mysterious spend stops being mysterious. The parts that matter:

• The minimum charge is 60 seconds of resource usage, metered per second beyond that. This is a minimum for the warehouse, not for each individual query.
• Usage is recorded when a transaction completes, rolls back, or is stopped. A transaction that runs for hours shows up in your usage view only at the end.
• Cancel a query before it finishes and you still pay for the time it ran.
• Querying system tables is billed like any other query. Your monitoring loop is a workload.
• After a burst, capacity can stay elevated for a period after the load drops. Scale-down is not instant.

Put those together and you reach a conclusion that irritates most engineers: on serverless, ten small queries crammed into one minute are cheaper than the same ten queries spread across ten minutes. Every wake-up costs you a minimum billing window multiplied by your base capacity. That is the opposite of the instinct you have from tuning an OLTP service, where you smooth load out to protect tail latency.

Before you change anything, get the real numbers out of the warehouse rather than out of Cost Explorer, which lags and aggregates.

-- Daily billed RPU-seconds converted to RPU-hours.
-- Multiply by your region's on-demand RPU-hour rate for dollars.
SELECT trunc(start_time) AS day,
       sum(charged_seconds) / 3600::double precision AS rpu_hours
FROM   sys_serverless_usage
GROUP  BY 1
ORDER  BY 1 DESC;

charged_seconds is the column to build cost reporting on. compute_seconds is informative but it is not what the invoice is derived from, and the two can disagree within a given interval. Two constraints worth knowing before you wire this into a dashboard: the view holds roughly a week of history, and it is visible only to superusers. If you want month-over-month trends, UNLOAD it to S3 on a schedule and query the archive with Amazon Athena instead.

Lever one: connections that look idle and are not

This is the one that bites hardest and shows up last, because there is nothing to see. AWS documents it plainly: Redshift Serverless treats all incoming queries as billable user activity, including lightweight health-check queries sent by connection pools. It does not matter whether the statement came from your application, a JDBC driver, or a pooling framework doing its job.

So a pool that fires SELECT 1 every thirty seconds to validate connections is a warehouse that never gets to sleep. Your product has no users at 3am and you are still paying the minimum window, over and over, multiplied by base capacity. HikariCP, Apache Commons DBCP and PgBouncer all have some form of this behaviour, and the defaults are tuned for OLTP databases where a validation query costs nothing.

Open transactions are the same problem wearing a different hat. A BEGIN without a matching COMMIT or ROLLBACK keeps consuming RPUs until the session ends. Session timeouts exist precisely because this happens.

What I would check, in this order:

1. Disable the pool’s validation or heartbeat query entirely if the driver allows it. If it does not, stretch the interval as far as your failure tolerance permits.
2. Drop idle pool size to something honest. A pool sized for peak that stays warm overnight is pure waste on this pricing model.
3. Fix any code path that opens a transaction and returns early on error without ending it.
4. Set a session timeout per application role so a leaked connection cannot bill indefinitely.

-- Cap idle sessions for the application role.
-- Value is in seconds; the documented range is 60 to 1,728,000.
ALTER USER analytics_app SESSION TIMEOUT 1800;

-- Cap how many connections a single role can hold open at once.
ALTER USER analytics_app CONNECTION LIMIT 40;

-- What is connected right now, and with what timeout.
SELECT * FROM stv_sessions;

Session timeout changes apply to new sessions only, so recycle the pool afterwards or you will conclude the setting does nothing.

Lever two: base capacity, max capacity and usage limits are three different things

These get conflated constantly, and two of them will not save you a cent on their own.

• Base capacity (base RPU). The floor. It multiplies every billed second, including that 60-second minimum. Halving base capacity roughly halves the cost of a warehouse dominated by short queries. It also halves the compute those queries get, so watch p95 latency alongside the bill.
• Max capacity (MaxRPU). A ceiling on how far automatic scaling can go. It caps compute available to the workgroup, it does not stop queries and it does not interrupt anything running. Useful as a guard rail against a runaway scan, useless as a budget.
• Usage limits. An actual budget, expressed in RPU-hours over a daily, weekly or monthly period. The breach actions are: log to a system table, raise an SNS alert, or turn off user queries.

Only the third one can stop you spending money, and only the third one can take your product down at 2pm on a Tuesday. Set it to alert first, live with it for a full billing cycle so you learn the shape of a normal week, then decide whether you are genuinely willing to have queries turned off. In a customer-facing SaaS product the answer is usually no, and the limit stays as an alarm feeding PagerDuty or whatever you already page from.

There is also the price-performance target, the slider that hands scaling decisions to AWS in exchange for a stated cost or speed preference. AWS recommends it for mid-range base capacities and advises against it at the very bottom and very top of the RPU scale, so check the current guidance against your base setting before enabling it. It is worth trying on a staging workgroup with a replayed query mix; it is not worth switching on blind in production.

On provisioned clusters the equivalent controls are per-feature usage limits: concurrency scaling measured in time, Spectrum measured in data scanned, cross-region data sharing, and extra compute for automatic optimization. Each takes a breach action of log, emit a metric, or disable the feature. Concurrency scaling also earns free credits as the main cluster runs, which is why a moderately bursty provisioned cluster often shows no concurrency scaling charge at all until it suddenly does.

Lever three: Redshift cost optimization starts with knowing which tenant is expensive

Be clear-eyed about what is possible here. On serverless you cannot get an exact dollar figure per query, because billing happens at the warehouse level and the minimum charge is shared across whatever else was running in that window. What you can build is a defensible apportionment, and that is enough to find the customer whose scheduled export is quietly eating your margin.

Start by labelling every statement your API issues on a tenant’s behalf.

-- Set in the pool's per-checkout init SQL, or per request.
SET query_group TO 'tenant_4417';

SELECT metric_date, sum(events)
FROM   fact_events
WHERE  tenant_id = 4417
  AND  metric_date >= dateadd(day, -30, current_date)
GROUP  BY 1;

RESET query_group;

The label lands in the query log and surfaces as query_label in the SYS monitoring views. Keep it short: the older query log views truncate the label to 30 characters, so a tenant slug beats a UUID with prefixes bolted on.

-- Seven days of activity grouped by tenant label.
-- Note: time columns in the SYS views are microseconds;
-- confirm units before converting anything to money.
SELECT trim(query_label)   AS tenant,
       count(*)            AS queries,
       sum(execution_time) AS exec_time,
       sum(queue_time)     AS queue_time
FROM   sys_query_history
WHERE  start_time > dateadd(day, -7, sysdate)
  AND  query_label LIKE 'tenant_%'
GROUP  BY 1
ORDER  BY exec_time DESC;

Three columns in that view earn their keep beyond the obvious ones. result_cache_hit tells you which dashboard queries are already free, which is often a bigger share than people expect. The split between queue_time and execution_time tells you whether you have a tuning problem or a capacity problem, and those have opposite fixes. And user_query_hash groups repeated queries with different literals, which is exactly what an embedded dashboard produces, so it is the fastest way to find the one panel that fifty tenants are running badly.

From there, apportion the day’s charged_seconds by each tenant’s share of execution time. It is an approximation and you should label it as one when you show it to finance. It is still the difference between “Redshift costs us a lot” and “eleven percent of our warehouse spend is one customer pulling an unbounded date range every fifteen minutes.”

Lever four: the isolation model you picked is a cost decision

AWS’s SaaS guidance describes three partitioning models, and each one has a distinct cost signature on Redshift.

• Pool. All tenants share tables with a tenant identifier column. Cheapest by a wide margin, one warehouse to keep warm, one set of statistics. You pay for it in noisy-neighbour risk and in the access-control work you now have to do yourself.
• Bridge. Separate schemas or databases inside one cluster. Sounds like a compromise, behaves like neither. AWS’s own whitepaper is fairly blunt that the isolation profile does not usually justify it, since cluster-level access grants reach across the databases anyway.
• Silo. A warehouse per tenant. Clean boundaries and per-tenant cost visibility for free. On serverless it is also the most expensive thing you can do, because every workgroup carries its own base capacity floor and its own 60-second minimums. Twenty small tenants means twenty warehouses waking up independently.

Data sharing sits between these and is the pattern I reach for when workload interference is the real problem. One producer handles ingestion and transformation; consumers read the shared data without copying it, and a consumer’s load does not touch the producer. Genuinely useful for separating a heavy ETL window from customer-facing reads. But be honest about the arithmetic: every consumer is its own billable warehouse. Data sharing buys you performance isolation, not cheaper compute.

In a pooled model, the thing I set up first is a sort key that leads with the tenant identifier followed by the time column everyone filters on. That lets Redshift prune blocks before it reads them instead of scanning broadly and filtering afterwards. Combine it with row-level security so the tenant predicate cannot be forgotten by an application bug, and you have removed both the largest cost driver and the scariest failure mode in one change.

Lever five: scan less, refresh less

Classic warehouse hygiene still applies, it just pays differently here. Shorter queries mean fewer billed seconds at your base capacity.

• Sort keys that match your real predicates. Not the ones from the design doc. Pull the top twenty query hashes and read their WHERE clauses.
• Materialized views for the panels every tenant loads. Real savings on the read path, but refresh is compute you pay for. A view refreshed every five minutes and read twice an hour is a net loss.
• Let the result cache work. Identical query text against unchanged data is free. Anything your BI layer does that injects a timestamp or a random parameter into otherwise identical SQL is throwing that away. Worth checking in Amazon QuickSight, Metabase or whatever sits in front.
• Tune zero-ETL refresh intervals. The refresh interval on the target database is adjustable via ALTER DATABASE. Shorter is fresher and more expensive. For reporting and historical analysis, a longer interval is usually the right call and nobody notices.
• Keep cold history out of managed storage. Partitioned Parquet or Apache Iceberg tables in S3, catalogued in AWS Glue, queried through the lake. On serverless those queries bill at the same RPU rate rather than as a separate Spectrum line, so the win is in scan efficiency and storage cost, not in dodging a charge.

One reassuring detail: the automatic optimization work Redshift does in the background is not billed by default. It becomes billable only if you explicitly enable extra compute resources so those operations can run during busy periods. That is a deliberate trade, not an accident, and it is worth knowing before you turn it on.

Provisioned or serverless: how I would decide

Both have a genuine case and the honest answer depends on the shape of your load, not on which is newer.

Serverless wins when demand is spiky or concentrated in business hours, when you cannot forecast capacity, and for dev and test environments that sit idle most of the week. It also folds concurrency scaling and data-lake queries into a single rate, which removes two line items people routinely forget to model.

Provisioned RA3 wins when load is steady around the clock, because a reserved commitment on nodes can beat accumulated on-demand RPU-hours, and because you get the full workload management surface: queues, query priority, query monitoring rules with the complete set of controls. If you need to guarantee that a tenant’s export can never starve the interactive path, that machinery is more expressive than a price-performance slider.

Commitment discounts now exist on both sides, including reservations for serverless managed at the payer account level. Rates and terms change, so price it against your own numbers rather than a blog post.

The tell is simple. Pull a week of charged_seconds bucketed by hour and plot it. A flat line means you are paying serverless rates for provisioned behaviour. A sawtooth with long dead zones means the opposite.

Troubleshooting: the bill moved and nothing shipped

1. Get hourly billed seconds first. Aggregate charged_seconds by hour from the usage view. If the increase is spread evenly across all 24 hours, it is background activity: a pool, a monitor, a health check. If it is concentrated, it is a workload.
2. Check for anything running or queued right now. A single stuck statement explains a lot of otherwise inexplicable spend.
3. Look for transactions that never ended. A deploy that changed error handling can leave transactions open without a single failed request in your logs.
4. Compare query counts against query cost. Flat count with rising cost points at base capacity changes, scale-down lag, or data growth making the same queries slower.
5. Group by user_query_hash and diff against last week. New shapes appearing means a shipped change. Old shapes getting slower means data or statistics.
6. Only then look at storage. Managed storage grows quietly and it is rarely the cause of a sudden jump, but it is often the cause of a slow one.

-- Anything currently running or waiting.
SELECT user_id, query_id, transaction_id, session_id, status,
       trim(database_name) AS database_name,
       start_time, queue_time, execution_time
FROM   sys_query_history
WHERE  status IN ('running','queued')
ORDER  BY start_time;

Common mistakes

• Treating max capacity as a spending cap. It caps compute, not cost, and it will not stop a workload that simply runs for a long time.
• Optimising individual slow queries while ignoring a connection pool that wakes the warehouse every thirty seconds all night.
• Smoothing scheduled jobs out across the hour to be gentle on the warehouse. On serverless this is backwards; batching into fewer windows costs less.
• Building cost dashboards on compute_seconds instead of charged_seconds, then wondering why the totals never reconcile with the invoice.
• Giving every tenant their own workgroup for isolation, then discovering that base capacity floors and minimum charges multiply by tenant count.
• Setting a usage limit to “turn off user queries” on the first day, before anyone knows what a normal week looks like.
• Leaving the monitoring loop itself unbounded. Polling system views every few seconds is a workload that bills like any other.

Best practices worth the effort

• Label every tenant-originated query with query_group from day one. Retrofitting attribution is far more painful than adding a SET statement to your pool’s init SQL.
• UNLOAD the serverless usage view to S3 on a schedule. Seven days of retention is not enough to argue about a monthly invoice.
• Keep at least one usage limit configured as an alert, permanently, even if you never set a hard cap.
• Review base capacity quarterly against p95 latency, not just against cost. The right number moves as your workload changes.
• Put a hard date bound on every customer-facing query in the application layer. Unbounded ranges are the single most common source of surprise spend in embedded analytics.
• Model concurrency scaling and data-lake charges explicitly if you are on provisioned. They are the line items people forget until they appear.
• Tag workgroups and clusters consistently so cost tooling, whether that is AWS Cost Explorer or something like CloudZero or Vantage, can split spend by environment without guesswork.

FAQ

Does Redshift Serverless really charge me when nobody is using the product?

Idle time itself is not billed, but anything that sends a query is. AWS states explicitly that health-check queries from connection pools count as billable user activity. If your pool validates connections on a timer overnight, you are paying minimum billing windows all night. Check the pool before you conclude the pricing model is broken.

How do I calculate the cost of a single query?

You cannot, exactly. Serverless bills the warehouse, and the 60-second minimum is shared with whatever else ran in that window. The workable approach is apportionment: take charged_seconds for a period and divide it by each labelled tenant’s share of execution time from the query history view. Useful for finding outliers, not precise enough for per-customer invoicing.

Should I lower base capacity to save money?

Often yes, and it is the single highest-leverage change for a workload made of many short queries, because base capacity multiplies every billed second including the minimum. The catch is that it also reduces the compute each query gets. Change it in one step, watch p95 latency and queue time together for a full week, then decide whether to go further.

Is a warehouse per tenant a good idea?

Only when tenants are large enough to keep a warehouse genuinely busy, or when a contract requires that level of separation. For a long tail of small tenants it is the most expensive option available, since each warehouse carries its own capacity floor and its own minimum charges. Pooled tables with row-level security and a tenant-leading sort key gets you most of the isolation for a fraction of the compute.

Does concurrency scaling cost extra?

On Redshift Serverless, no, scaling is included in the RPU rate. On provisioned clusters it is a separate charge, offset by credits that accrue while the main cluster runs. That difference catches out teams migrating between the two, in both directions.

Will a usage limit take my product down?

It will if you configure the breach action to turn off user queries. The logging and alerting actions are safe and are what you want in a customer-facing system. Treat the hard stop as a deliberate business decision about which is worse, an unexpected invoice or an outage, rather than as a default setting.

Why does my cost report never match the AWS invoice?

Usually one of three things: using compute_seconds rather than charged_seconds, forgetting that usage is recorded only when a transaction completes so long transactions land in a later interval, or leaving storage and cross-region transfer out of the model entirely.

The one thing worth remembering

Redshift cost optimization for a SaaS analytics product is mostly not a query tuning exercise. It is a question of how often something wakes the warehouse up and at what capacity. Query tuning matters, sort keys matter, materialized views matter, but a connection pool with default settings will quietly outspend all of them combined.

So start at the meter. Pull hourly billed seconds, look at the overnight hours when your product has no users, and see whether the line goes to zero. If it does not, you have found your first and cheapest win before touching a single line of SQL.

Need help getting your Redshift bill under control?

I work with SaaS and data teams on exactly this problem, usually somewhere between the warehouse and the application that is hammering it. Things I can help with:

• Auditing an existing Redshift Serverless or RA3 workload and producing a ranked list of what is actually driving spend, with the numbers pulled from your own system views.
• Building per-tenant cost attribution: query labelling, a usage archive in S3, and a dashboard your product and finance teams can both read.
• Fixing the connection and session layer, including pool configuration, validation queries, session timeouts and transaction hygiene.
• Right-sizing base and max capacity against measured latency, and setting usage limits and alerts that warn without risking an outage.
• Reviewing multi-tenant data models: sort and distribution keys, row-level security, and whether data sharing or a pooled model fits your tenant mix.
• Deciding between provisioned and serverless with a workload profile behind the recommendation rather than a rule of thumb.

If you have a week of usage data, a suspicious hourly cost chart, or a pool configuration you are not sure about, send it over and I will tell you what I see in it.

The post Redshift Cost Optimization for SaaS Analytics: The Levers That Actually Move the Bill appeared first on John Nessime.

That is where embedded analytics on AWS stops being a front-end task and becomes an architecture decision. The awkward part is that the choice you make first, how the viewer is identified, quietly decides which isolation mechanisms remain available to you afterwards. Get that order backwards and you rebuild the data layer, not the iframe.

This post covers the four decisions that determine whether the build holds: identity model, tenant isolation, query mode, and session economics. Then the embed handshake itself, the errors you will actually see in the browser console, and what to check first when it fails.

Before anything else: the product got renamed

Amazon QuickSight was folded into a broader platform called Amazon Quick Suite, and the BI product inside it is now called Amazon Quick Sight. AWS documentation has since moved again under an “Amazon Quick” umbrella. You will land on all three naming conventions depending on which search result you click, which makes finding the right doc page genuinely annoying.

The practical upshot: the APIs, SDKs and IAM action names did not change. You are still calling quicksight:GenerateEmbedUrlForRegisteredUser against ARNs in the quicksight namespace, and the JavaScript SDK is still published as amazon-quicksight-embedding-sdk. Nothing in your code breaks. Only your bookmarks do. I mention it because half the confusion in a first embedded build comes from following a doc page that describes a UI menu that has since been reorganised.

Decision one: registered users or anonymous sessions

Two API operations generate embed URLs. GenerateEmbedUrlForRegisteredUser issues a session for a user who exists inside the BI account. GenerateEmbedUrlForAnonymousUser issues a session for someone who does not, and never will.

This reads like a convenience choice. It is not. Row-level security using session tags, the mechanism most SaaS products want, is supported only for anonymous embedding. It does not work with GenerateEmbedUrlForRegisteredUser, it does not work with the older GetDashboardEmbedUrl operation, and it is not supported with the IAM identity type. That constraint is documented, easy to miss, and it is the single most expensive thing to discover late.

So the fork is really this. If you register every viewer, you get per-user features (bookmarks, threshold alerts, scheduled snapshots) and you enforce isolation with username or group rules on the dataset. You also inherit the job of provisioning, deprovisioning and reconciling a user directory that mirrors your own. If you go anonymous, you skip all of that and filter with session tags at embed time, but per-user features are off the table because there is no persistent user to hang them on.

The registered-user request body is small. Everything interesting is in ExperienceConfiguration:

POST /accounts/<aws-account-id>/embed-url/registered-user

{
  "UserArn": "arn:aws:quicksight:<region>:<account>:user/default/<user>",
  "SessionLifetimeInMinutes": 60,
  "AllowedDomains": ["https://app.example.com"],
  "ExperienceConfiguration": {
    "Dashboard": {
      "InitialDashboardId": "<dashboard-id>",
      "FeatureConfigurations": {
        "Bookmarks": { "Enabled": true }
      }
    }
  }
}

One trap on the anonymous path that deserves its own sentence. Anonymous sessions belong to a namespace, and any dashboard shared with that namespace is reachable by a session in it, whether or not you listed the dashboard in AuthorizedResourceArns. If you were treating that parameter as your allowlist, it is not. Namespace membership is the real boundary.

Decision two: where tenant isolation actually lives

There are three places you can put the filter, and only one of them scales.

• A dashboard per tenant. Works for five customers. Becomes a deployment problem at fifty and a change-management disaster at five hundred, because every visual fix is now a fan-out.
• A dataset per tenant, filtered in SQL. Better isolation guarantees, genuinely defensible in a compliance review, but you multiply refresh jobs and in-memory footprint by tenant count.
• One dashboard, one dataset, row-level security. The standard answer. One artifact to maintain, filtering applied per session.

With anonymous embedding, RLS is driven by tags. You declare tag keys against columns on the dataset, then supply values at embed time. The filter is evaluated server-side against the session, so a viewer poking at the iframe cannot lift it.

POST /accounts/<aws-account-id>/embed-url/anonymous-user

{
  "Namespace": "default",
  "SessionLifetimeInMinutes": 60,
  "AuthorizedResourceArns": [
    "arn:aws:quicksight:<region>:<account>:dashboard/<dashboard-id>"
  ],
  "SessionTags": [
    { "Key": "tenant_id", "Value": "acme-corp" },
    { "Key": "region",    "Value": "emea" }
  ],
  "AllowedDomains": ["https://app.example.com"],
  "ExperienceConfiguration": {
    "Dashboard": { "InitialDashboardId": "<dashboard-id>" }
  }
}

The value in SessionTags must come from your server-side session, never from a request parameter, a cookie your client can write, or a JWT claim you have not verified. This is the whole security boundary. Tag rules support combining conditions, so a manager who should see several sites is expressible without a second dashboard.

One quiet limit worth knowing before it bites: when RLS is applied to in-memory datasets, each field has a maximum length in Unicode characters, and fields exceeding it are truncated during ingestion rather than rejected. If your tenant identifiers are long opaque strings, test that a truncated value cannot collide with another tenant’s. Silent truncation plus a prefix collision is exactly the kind of bug that produces a cross-tenant data leak with no error anywhere in the logs.

Decision three: SPICE or direct query against your AWS data

Every dataset runs in one of two modes, and the difference shows up on a bill somewhere else in your account.

Direct query sends a live query to the source each time a visual renders. Against Amazon Athena that means an S3 scan per dashboard open, billed by bytes scanned. Against Amazon Redshift it means a concurrent query slot per viewer. Freshness is perfect. The failure mode is that dashboard load is now coupled to warehouse load, and your analytics traffic competes with everything else running there. Two hundred people opening a dashboard at 9am is two hundred queries, and Redshift concurrency is finite.

SPICE imports a snapshot into an in-memory engine and serves every viewer from it. One scan on refresh, then arbitrarily many reads. For an embedded product where the same aggregate is served to thousands of sessions, this is usually the right call, and the Athena cost difference between “scan once per refresh” and “scan once per pageview” is not subtle. What you give up is freshness, bounded by your refresh schedule, plus a capacity dimension to manage and incremental refresh to configure if the dataset is large.

The pattern I reach for first on a data-lake backend is a hybrid: recent partitions in SPICE with an incremental refresh on a look-back window, historical data left on direct query for the rare deep query. It costs more design effort up front and it is the thing most teams skip, but it is the only shape that keeps both the bill and the load time flat as history grows.

Whichever you pick, note that visual generation has a timeout, and data-source-specific timeouts apply on top of it. A query that is merely slow in a console tab renders as a broken visual in a customer’s browser. Model your worst partition, not your average one.

Decision four: what a session actually costs

I am not going to quote figures, because AWS changes them and you should read the current pricing page. The mechanism is what matters, and it is genuinely different from seat-based BI licensing.

• A reader session is a fixed 30-minute window. Not a pageview, not a query. Reopening the dashboard twenty minutes later is still the same session.
• Per-user pricing charges per session with a monthly cap per reader. Predictable when the same people return daily.
• Capacity pricing buys sessions in bulk with no user provisioning at all. This is the model built for embedding, and it is the one that pairs with anonymous sessions.
• Capacity pricing is also the prerequisite for programmatic dashboard refresh, so if near-real-time rendering is a product requirement, that decision is already made for you.
• Annual commitments to capacity unlock removing the “Powered by” attribution footer. If white-labelling is a contractual requirement, factor that in early rather than discovering it during a customer demo.
• Enabling certain Pro-tier and generative Q&A capabilities triggers an account-level monthly infrastructure fee that exists whether or not anyone uses the feature.

The cost failure mode nobody plans for is architectural rather than commercial. If you embed the dashboard on a tab that loads by default, you bill a session for every user who lands on that page and looks at something else. Lazy-load the iframe on explicit interaction. That one change is often the largest single lever on the bill, and it costs an afternoon.

The embed handshake, and the three things that break it

The flow is short. Your backend authenticates the user with your own identity system, calls the embed URL API with the right tags or user ARN, returns the URL to the browser, and the SDK mounts an iframe against it.

1. The generated URL carries a temporary bearer token valid for five minutes, and it is single use once redeemed. Generate it per page load from your backend. Never cache it, never put it in a build artifact, never log it.
2. Session lifetime is separate from URL validity, set with SessionLifetimeInMinutes, and ranges from fifteen minutes to ten hours with ten hours as the default. Ten hours is almost never what you want for a customer-facing product. Match it to your own session, or shorter.
3. Domains must be allowed explicitly. An administrator configures static domains in the admin menu, and AllowedDomains on the API call can override that with up to three domains or subdomains per request. Add an AllowedEmbeddingDomains condition to the IAM policy of the calling role, or any developer with that permission can list any domain on the internet.

On the browser side, the v2 SDK creates an embedding context (which appends its own zero-pixel iframe to body for message passing) and then mounts the experience:

import { createEmbeddingContext } from 'amazon-quicksight-embedding-sdk';

const context = await createEmbeddingContext();

await context.embedDashboard(
  {
    url: embedUrl,                        // fetched from your backend, just now
    container: '#analytics',
    height: '600px',                      // acts as loading height below
    resizeHeightOnSizeChangedEvent: true,
  },
  {
    toolbarOptions: { export: false, undoRedo: false, reset: false },
    attributionOptions: { overlayContent: true },
    onMessage: async (event) => {
      if (event.eventName === 'ERROR_OCCURRED') {
        console.error(event.message.errorCode);
      }
    },
  }
);

Two details in there earn their place. resizeHeightOnSizeChangedEvent turns the height value into a loading placeholder and lets the frame grow to fit content, which is what stops the dashboard rendering into a 600px letterbox with an inner scrollbar. And overlayContent tells the layout to overlay the attribution footer rather than reserve extra height at the bottom for it.

Troubleshooting embedded analytics on AWS

Almost every failure lands in one of these. Read the error code out of the ERROR_OCCURRED message before doing anything else.

• Forbidden means the URL’s authentication code expired. You held the URL longer than five minutes, or you served it from a cache, or a retry redeemed it twice. Fix the generation path, not the permissions.
• Unauthorized means the session obtained from that code expired. Different problem, different fix: your SessionLifetimeInMinutes is shorter than how long people keep the tab open. Handle it by re-fetching a fresh URL and re-mounting rather than letting the frame sit there dead.
• Frame never appears at all. Check the onChange handler for NO_CONTAINER or INVALID_CONTAINER, which usually means you mounted before your target element existed, and for INVALID_URL, which means the URL shape does not match the experience method you called.
• Frame appears, dashboard does not. Nine times out of ten this is the domain allowlist. The request domain has to match what was allowed, including scheme and any subdomain, and a staging hostname that nobody added is the usual culprit.
• Modals render off-screen. A known consequence of auto-resizing height: an export dialog can open above the visible viewport. Listen for MODAL_OPENED and scroll the parent page to the frame position.
• Toolbar features silently missing. Bookmarks, threshold alerts and scheduling require both the SDK toolbar flag and the matching entry under FeatureConfigurations in the embed URL request, and they only exist on the registered-user path. Setting the client flag alone does nothing.
• First render is slow, later ones are fine. Direct query against a cold warehouse. Compare the same query in Athena or Redshift directly to confirm before blaming the BI layer.

Common mistakes

• Choosing registered-user embedding for the identity story, then discovering session-tag RLS is unavailable on that path.
• Treating AuthorizedResourceArns as the security boundary instead of namespace membership.
• Deriving a session tag value from anything the client can influence.
• Generating the embed URL at build time, or caching it in a CDN, and then not understanding the Forbidden errors.
• Leaving session lifetime at the ten-hour default in a customer-facing app.
• Putting the dashboard on a default-loaded tab and paying for sessions nobody asked for.
• Building the first version on direct query against Athena because it is quicker to wire up, then meeting the scan bill.
• Forgetting that embedding and row-level security sit in the Enterprise tier, so a Standard-tier proof of concept proves nothing.

Best practices

• Decide the identity model before you build a single dataset. Everything downstream inherits it.
• Put the embed URL call behind one server-side endpoint that reads tenant scope from your own session and nowhere else. One function, one place to audit.
• Constrain the calling IAM role with an AllowedEmbeddingDomains condition and scope resources to specific namespaces rather than a wildcard.
• Write an automated test that requests tenant A’s embed URL and asserts tenant B’s rows are absent. Run it on every dataset change, because RLS breaks silently.
• Default to SPICE with a refresh schedule matched to a stated freshness SLA, and only reach for direct query where the SLA genuinely demands it.
• Track refresh failures as a first-class alert in CloudWatch or whatever you already run, whether that is Grafana, Datadog or something in-house. A stale dashboard that still renders is worse than one that errors, because nobody notices.
• Lazy-load the iframe on user intent, not on page mount.
• Keep the embedded surface read-only unless authoring is a real product requirement. Console embedding is a much larger permissions surface than dashboard embedding.

Is managed BI even the right call?

Worth asking honestly, because the answer is not always yes. The case for the AWS-native route is real: no connector layer to maintain against Athena, Redshift, S3 and Aurora, IAM you already understand, and a usage-based cost model that beats per-seat licensing when your viewers are bursty. If most of your data already sits in AWS, that adds up.

The case against is equally real. Visual customisation is limited compared to charting directly against your own API, the attribution footer needs a commitment to remove, and if you want full control of the front end you may be better served by Apache Superset or Metabase self-hosted, or by Grafana where the workload is closer to operational metrics than customer-facing BI. Those come with an operational burden you now own. That is the trade: you either run the BI layer or you rent it, and renting it means living inside its constraints.

Frequently asked questions

Do my users need AWS accounts to view an embedded dashboard?

No. With anonymous embedding they need no AWS account and no BI user record at all. Your application authenticates them however you already do, and your backend maps that identity to session tags when it requests the embed URL.

Can I use row-level security with registered-user embedding?

Yes, but only with username or group based rules, not with session tags. Tag-based RLS is restricted to the anonymous embedding path. If you need tags, you need anonymous sessions.

How long does an embed URL stay valid?

The URL itself carries a bearer token valid for five minutes and usable once. The session it opens is separate and lasts between fifteen minutes and ten hours depending on SessionLifetimeInMinutes, defaulting to ten hours.

Should I use SPICE or direct query for embedded analytics on AWS?

SPICE for anything with many viewers per refresh, which describes most embedded products. Direct query where the data must be current to the second, or where the dataset exceeds what you want to hold in memory. A hybrid split by data age is often the right answer and is under-used.

Why do I get a Forbidden error when the dashboard worked yesterday?

Forbidden points at the URL, not at permissions. The most common causes are caching the URL, generating it more than five minutes before use, or a client retry redeeming the same single-use token twice. If it is Unauthorized instead, the session expired and you need a fresh URL.

Can I white-label the embedded dashboard completely?

Largely. Themes control colours and typography, the SDK hides toolbar controls, and parameters let your own UI drive the dashboard. Removing the attribution footer entirely is tied to an annual capacity commitment, so confirm that against current terms before you promise it to a customer.

Does natural-language querying work in an embedded context?

Yes. The SDK exposes a generative Q&A experience alongside dashboards and visuals, driven by curated topics rather than raw tables. It is billed on its own capacity dimension and gates behind the Pro tiers, so treat it as a separate cost decision rather than a free addition.

The one thing worth remembering

Embedded analytics on AWS is not a rendering problem. The iframe is the last five percent. The part that decides whether the build survives contact with a second customer is the identity model, because it silently determines which isolation mechanism you are allowed to use, and that in turn shapes your dataset design, your refresh strategy and your bill.

Pick that first. Write the cross-tenant test before you write the dashboard. Everything else is recoverable in an afternoon.

Need help with an embedded analytics build?

This is the kind of work I do. Things I can help with directly:

• Reviewing an existing embed integration for cross-tenant leakage, including the session-tag path and the namespace boundary.
• Designing the identity and row-level security model before you commit to a dataset layout.
• Cutting Athena scan and Redshift concurrency cost by moving the right datasets into SPICE with incremental refresh.
• Building the backend embed-URL service with scoped IAM roles, domain conditions and sane session lifetimes.
• Setting up refresh failure alerting so a stale dashboard does not quietly serve last week’s numbers.
• Automated cross-tenant isolation tests wired into CI, so an RLS regression fails the build instead of the customer.

Send me the actual thing: your embed URL request payload with secrets stripped, the browser console error, or the dataset RLS rules. It is much faster to reason about a real payload than a description of one.

The post Embedded Analytics on AWS: The Four Decisions That Bite Later appeared first on John Nessime.

The answer is in there. It is one email, or one line in a site diary, sitting among forty thousand other files. Nobody can read forty thousand files, so the instinct is to point a language model at the pile and ask it. That instinct is right. The naive implementation of it is where the money goes.

Here is the failure mode that bites hardest, and it is invisible until an expert challenges you on it. You build retrieval over the document set, ask about notice of delay, and the system confidently returns a letter that discusses the delay event in great detail. It is a good letter. It is also dated eleven months after the event, written by the claims consultant during preparation of the claim itself. It scored highest precisely because it was written to argue the point. The contemporaneous notice, the thing you actually needed, was four badly typed lines in a routine progress email that mentioned the word “delay” once.

Semantic similarity has no concept of a deadline. That single gap is the difference between an AI construction claims platform that shortens a disclosure exercise and one that quietly manufactures a wrong answer with a citation attached to it.

This post covers how to build that platform on AWS: how to lay out ingestion, how to make retrieval respect the contractual clock, why schedule data must never go anywhere near your vector index, how to keep privileged material out of the same retrieval path as project records, and which AWS building blocks are actually the current ones now that several of the obvious candidates have been moved to maintenance mode.

What a claims platform actually has to answer

Before any architecture, be honest about the question shapes. They are not all the same problem and they do not all get solved by retrieval.

1. Chronology. What happened, in what order, and on what date was it recorded? This is a retrieval and metadata problem.
2. Entitlement. Which clause applies, and what did it require the parties to do? This is retrieval over the contract plus careful prompting.
3. Causation. Which event moved the critical path, and by how much? This is schedule data and date arithmetic. It is not a language problem at all.
4. Quantum. What did the disruption cost? This is cost and resource data, joined to the events above.

Treat all four as “ask the documents” and you will get fluent nonsense on two of them. The architecture below splits them deliberately.

Failure one: retrieval that finds the best match instead of the first one

Two corpora live in every dispute bundle and they look identical to an embedding model.

• Contemporaneous records. Site diaries, progress emails, minutes, early warnings, RFIs, instructions. Written while the project was running, by people with no idea a dispute was coming.
• Claim-era material. Narratives, expert reports, without-prejudice correspondence, internal analysis. Written afterwards, specifically to be persuasive about the same events.

Claim-era material wins on cosine similarity almost every time, because it is denser in exactly the terms you searched for. If your retriever cannot distinguish them, every answer is contaminated by the argument you were trying to test.

The fix is metadata, applied at ingestion, and it is cheap to get right and expensive to retrofit. Amazon Bedrock Knowledge Bases reads a sidecar file that sits next to each document in S3, named with the full original filename plus .metadata.json. So letter-0421.pdf gets letter-0421.pdf.metadata.json. The naming convention is the only link between them; there is no separate registration step.

{
  "metadataAttributes": {
    "doc_date": 20240314,
    "corpus": "contemporaneous",
    "doc_type": "site_correspondence",
    "matter_id": "matter-0007",
    "date_source": "email_header",
    "privileged": false
  }
}

Look closely at doc_date. It is an integer, not a string, and that is not a style choice. Bedrock Knowledge Bases metadata attributes support STRING, NUMBER, BOOLEAN and STRING_LIST. The range comparison operators, the ones you need to express “on or before the notice deadline”, only apply to NUMBER. Store the date as "2024-03-14" and your filter will not throw an error. It will just quietly match nothing, or match everything, depending on how you wrote it. You will find out weeks later when someone asks why a document they can see in the bundle never appears in results.

With the date as a sortable integer, a query filter can express the contractual window directly.

{
  "andAll": [
    { "equals":              { "key": "corpus",   "value": "contemporaneous" } },
    { "equals":              { "key": "matter_id","value": "matter-0007" } },
    { "greaterThanOrEquals": { "key": "doc_date", "value": 20240301 } },
    { "lessThanOrEquals":    { "key": "doc_date", "value": 20240329 } }
  ]
}

That is the whole trick. You are no longer asking “what is the most relevant document about this delay”. You are asking “what did the parties actually write during the window in which the contract required them to write it”. Those are different questions and only one of them is worth anything in a dispute.

Where the date comes from matters more than the date

Do not use the S3 object timestamp. It records when someone copied a folder, usually years after the fact and identical across ten thousand files. Derive the date from the document itself: the Date: header on an email, the printed date on a letter, the period covered by a diary entry.

Sometimes you cannot, because the scanned undated fax exists in every project archive. Record that honestly with a date_source attribute rather than guessing, and treat unknown-date documents as a separate review pile. An extension of time argument built on an inferred date is an argument you will lose.

Failure two: treating the programme like a document

This one is worse, because the output looks right.

Oracle Primavera P6 exports XER and PMXML files. Asta Powerproject and Microsoft Project have their own formats. XER in particular is a plain text dump of relational tables, so it goes through a text pipeline without complaint. Chunk it, embed it, and you now have vectors representing fragments of a table of activity codes with no relationships attached.

Ask that index how much float activity A1200 had at the March data date and you will get a number. It will be well formatted and it will be invented. Total float is the product of a forward and backward pass across the whole logic network under a specific calendar. It cannot be recovered from a retrieved fragment, and a language model asked to produce it will produce something plausible instead of admitting that.

Schedule data goes into a structured store, and the model queries it rather than reasoning about it.

1. Parse each programme file into tables. PyP6Xer handles XER from Python; MPXJ is a Java library that reads XER, PMXML, Asta Powerproject and MSPDI among others, which matters when the bundle contains four scheduling tools.
2. Load activities, logic links, calendars, resource assignments and WBS into Amazon Aurora PostgreSQL for interactive work, or into S3 with AWS Glue and Amazon Athena when you have hundreds of updates and want columnar scans.
3. Stamp every row with the data date of the update it came from. This is the single most important column in the whole platform. Without it you have a pile of schedules; with it you have a time series of the project’s own view of itself.
4. Run windows analysis, as-planned versus as-built comparison and float erosion in SQL or Python, deterministically, so the same inputs always give the same numbers.
5. Expose the results to the model as a tool it can call, or as generated SQL against a defined schema. The model turns a question into a query and narrates the result. It does not do the arithmetic.

A rough shape of the query that makes float erosion visible:

SELECT
    a.activity_id,
    a.data_date,
    a.total_float_days,
    a.total_float_days - LAG(a.total_float_days)
        OVER (PARTITION BY a.activity_id ORDER BY a.data_date)
      AS float_change
FROM   schedule_activities a
WHERE  a.project_id = 'PRJ-01'
  AND  a.data_date BETWEEN DATE '2024-01-01' AND DATE '2024-06-30'
ORDER BY a.activity_id, a.data_date;

Nothing clever there, and that is the point. Every number is traceable to a row that came from a named XER file. When an opposing expert asks where a figure came from, the answer is a file name and a query, not “the model said so”.

Be realistic about effort here. Programme parsing and normalisation across inconsistent updates is the hardest part of the build and the part clients always underestimate. Activity IDs get reused, calendars change mid-project, and someone will have re-baselined without telling anyone. Budget for it.

Failure three: an answer with no paper trail

In most RAG applications a citation is a nice touch. In dispute work it is the product. An answer that cannot be traced to a page of a disclosed document is not evidence, it is a rumour with good grammar.

Design for that from the ingestion layer, not the presentation layer.

• Keep page and position. Amazon Bedrock Data Automation returns confidence scores and bounding box data alongside extracted fields, and Amazon Textract returns geometry per block. Carry both through the pipeline so a citation resolves to a page and a region, not just a file.
• Route low confidence to humans. Handwritten site diaries and faxed variation orders will produce low-confidence extractions. Those should land in a review queue by default rather than silently entering the index.
• Reject ungrounded answers. Amazon Bedrock Guardrails includes contextual grounding checks that score whether a response is supported by the retrieved passages. It reduces confident invention. It does not eliminate it, and anyone who tells you otherwise is selling something.
• Keep an immutable evidential copy. S3 Versioning plus S3 Object Lock on the landing bucket means the file the platform indexed is provably the file that was disclosed.

One design rule underpins all of it: the platform shortlists evidence, it does not decide entitlement. Recognised frameworks for this work, the Society of Construction Law Delay and Disruption Protocol and AACE International’s Recommended Practice 29R-03 on forensic schedule analysis, both assume a named analyst applying a stated method and exercising judgement. A system that outputs “the contractor is entitled to 42 days” is not helping. A system that outputs “here are the eleven contemporaneous documents inside the notice window, here is the float movement across those updates, here is what is missing” is doing real work.

Failure four: one index for privileged and non-privileged material

Dispute bundles contain legal advice, counsel’s opinions, without-prejudice correspondence and internal settlement analysis. Those must not be retrievable through the same path as project records.

The tempting shortcut is a privileged: false metadata filter on every query. Do not rely on that as your boundary. A metadata filter is a query parameter. One missing filter in one code path, one debug endpoint, one caching layer that drops it, and privileged material surfaces in a general search. The blast radius of that mistake is not a bug report.

Separate the indexes physically and separate the IAM roles that can reach them. Amazon S3 Vectors makes this practical: you can set a dedicated customer-managed KMS key per vector index, and you get a large number of indexes per vector bucket, so per-matter and per-sensitivity separation does not become an operational burden. Keep the metadata flag as well, because defence in depth is free, but make the identity boundary the one you actually trust.

Amazon Macie is worth pointing at the landing bucket to find personal data you did not expect, particularly in HR records and accident reports that get swept into project archives.

Choosing the AWS building blocks, including what not to build on

A lot of published architectures for this kind of platform are now pointing at services AWS has stopped developing. Two matter here, and the dates are the point.

• Amazon Kendra entered maintenance mode on 30 June 2026 and stops accepting new customers on 30 July 2026. Existing customers keep support and security fixes but no new capability. AWS directs new enterprise search and RAG work to Amazon Bedrock Knowledge Bases. If a tutorial or a proposal you are reading starts with a Kendra index, it predates that change.
• Amazon Bedrock Agents moved to maintenance mode in the same round of service availability changes, with Amazon Bedrock AgentCore as the successor for agentic orchestration. Check the current AWS service availability page before you commit an orchestration layer.

For the retrieval layer itself, Bedrock Knowledge Bases now comes in two shapes and the choice is a real trade-off rather than a marketing tier.

Managed Knowledge Base

AWS manages the vector store, embeddings model, re-ranker and retrieval orchestration as a single primitive, with native connectors for Amazon S3, SharePoint, Confluence, Google Drive, OneDrive and a web crawler, plus automatic parsing strategy selection and a retriever that decomposes multi-step queries. The connectors pull source permissions along with content, which matters when the document set lives in the client’s SharePoint rather than a bucket you control.

Where it wins: you get a working retrieval layer in an afternoon instead of a fortnight, and the parsing tuning that normally eats the first weeks of a build is done for you. For a first matter, or a proof of value before a client commits budget, this is the one I would reach for.

Custom Knowledge Base

You bring your own vector store and control chunking, embedding model and index layout.

Where it wins: claims work has awkward chunking requirements. A two-page letter split mid-sentence at a page boundary produces a chunk where the notice sentence has lost its date and its addressee. Controlling chunk boundaries around document structure, and controlling which index a document lands in, are both easier when you own the store. Where it doesn’t: you now own embedding model upgrades, re-indexing, sync failures and capacity, which is real ongoing work for a small team.

Start managed, build a retrieval evaluation set of real questions with known correct documents, and only move to custom when that set demonstrates the problem is chunking. Most teams migrate on a hunch and discover the problem was metadata all along.

Where the vector storage bill actually comes from

Rates change, so learn the billing mechanism rather than a number. Amazon S3 Vectors charges on three axes: upload volume by logical gigabyte, storage by logical gigabyte, and queries by data processed, where data processed scales with the size of the index being searched. Note that filtering does not reduce the data processed by a query.

That shape suits claims work unusually well. A dispute archive is enormous and cold: millions of chunks, queried by a handful of analysts a few hundred times a day, so you pay mostly for storage, which is the cheap axis. Compare that against Amazon OpenSearch Serverless, which prices on provisioned compute units and therefore rewards high query volume against a smaller index, or Aurora PostgreSQL with pgvector when you already need Aurora for the schedule tables and would rather run one system than two.

The practical lever is to split indexes per matter. Query cost scales with index size, so one giant index across every dispute you have ever run makes every query more expensive than it needs to be, on top of being a bad idea for confidentiality.

A reference pipeline

1. Everything lands in S3 under a per-matter prefix, with Versioning and Object Lock enabled on the evidential copy.
2. S3 event notifications trigger AWS Step Functions. Use Step Functions rather than a chain of Lambdas so that a failed extraction on page 300 of a 400-page bundle is visible and resumable.
3. Classify and split. Scanned bundles arrive as one PDF containing forty separate documents. Splitting them correctly is a prerequisite for dating them correctly.
4. Extract text with Amazon Bedrock Data Automation or Amazon Textract, keeping confidence scores and geometry.
5. Derive the document date and write the .metadata.json sidecar. Anything undated goes to the review queue.
6. Route by type: correspondence to the knowledge base, programme files to the XER parser and the relational store, cost data to its own tables.
7. Sync the knowledge base, then run your retrieval evaluation set before anyone uses it. A sync that succeeds is not the same as an index that answers correctly.
8. Serve through an API that refuses to return an answer without citations, and log every query with the filters that were applied.

Define the whole thing in Terraform or OpenTofu from the start. Matters are per-client and short-lived, and standing one up should be a variable file, not an afternoon in the console. Point Amazon CloudWatch, or Grafana Cloud if you already run Grafana elsewhere, at the Step Functions execution metrics so a silently failing extraction stage does not go unnoticed for a week.

Troubleshooting

Date filters return nothing, and no error

Almost always the date was stored as a string. Range operators need NUMBER. Convert to an integer in YYYYMMDD form and re-sync the affected documents.

A document is in the bucket but never appears in results

Check the sidecar filename first. It must be the complete original filename with .metadata.json appended, extension included. report.pdf.metadata.json works; report.metadata.json is a file the ingestion job will happily ignore. After that, check whether a filter in the query path is excluding it.

Answers cite the right document but the wrong passage

Chunking split the document somewhere structurally meaningful. Look at the raw chunks for that file. If the notice sentence and its date are in different chunks, no amount of prompt tuning fixes it. That is the signal to take control of chunking.

Float figures do not match the client’s own analysis

Check calendars before you check logic. Different activity calendars, a changed default calendar, or an update where someone applied a progress override will move float without any logic change. Reconcile activity counts between your parsed tables and the source file before trusting anything downstream.

Query costs jumped without more usage

An index grew. With storage-side vector search, query cost tracks the size of the index being scanned, so ingesting a large new bundle raises the price of every subsequent query against that index. Split by matter.

Common mistakes

• Using the file’s storage timestamp as the document date. It records the migration, not the event.
• Indexing claim narratives and contemporaneous records into the same corpus with no way to tell them apart.
• Embedding programme exports because they happen to be text files.
• Treating a metadata filter as a privilege boundary instead of an optimisation.
• Letting the model state entitlement conclusions rather than assembling and citing evidence.
• Building on services that have moved to maintenance mode because the tutorial you followed predates the change.
• Shipping without a retrieval evaluation set, so you have no way to know whether a change made things better or worse.
• One index for every matter, which is both a cost problem and a confidentiality problem.

Best practices

• Make the document date a first-class, numeric, filterable attribute, and record where it came from.
• Keep an immutable evidential copy separate from the working copy the pipeline mutates.
• Separate structured schedule and cost data from unstructured documents, and let the model query the former rather than reason about it.
• Build a retrieval evaluation set from real questions with known correct documents before you tune anything.
• Enforce citations at the API layer, so an uncited answer is impossible rather than discouraged.
• Isolate privileged material by index and by IAM role, with metadata as a second layer.
• Log every query with its filters, so you can reconstruct how any given answer was reached.
• Define infrastructure as code so a new matter is a deployment, not a project.

FAQ

Can an AI construction claims platform replace a delay expert?

No, and building toward that goal produces something unusable. Established forensic frameworks assume a named analyst applying a stated method whose reasoning can be tested. The platform’s value is compressing weeks of document review into hours and making the schedule data queryable, so the expert spends their time on judgement rather than searching.

Should I use Amazon Kendra for the search layer?

Not for a new build. Kendra entered maintenance mode on 30 June 2026 and closed to new customers on 30 July 2026, with AWS pointing to Bedrock Knowledge Bases for equivalent and more current capability. Existing Kendra deployments continue to be supported, so this is a migration assessment rather than an emergency, but starting there now means starting on a service with no roadmap.

How do I stop the model inventing float and delay figures?

Do not give it the chance. Keep schedule data in a relational or columnar store and have the model generate queries against a defined schema, or call a tool that runs a fixed calculation. The arithmetic happens in SQL or Python where it is deterministic and reproducible; the model only turns questions into queries and results into sentences.

Which vector store should I choose for a claims archive?

Match the store to your query pattern. Large, cold archives queried by a few analysts favour storage-priced options like Amazon S3 Vectors, where you mostly pay to keep the data. Smaller indexes hit constantly favour compute-priced options like Amazon OpenSearch Serverless. If you already run Aurora PostgreSQL for schedule data, pgvector alongside it is a legitimate way to avoid operating a second system.

How do I handle scanned and handwritten site records?

Extract them with confidence scores retained, set a threshold, and route everything below it to human review before indexing. Handwritten diaries are frequently the most probative documents in a delay claim and also the least reliable to read automatically, so the review queue is not an edge case. Plan capacity for it.

Where do documents come from if they are not already in S3?

Most project records live in a common data environment such as Procore, Autodesk Construction Cloud, Aconex or a client SharePoint tenancy. Bedrock Managed Knowledge Base has native connectors for SharePoint, Confluence, Google Drive and OneDrive that ingest permissions alongside content. For platforms without a native connector, export to S3 and keep the export manifest as part of the disclosure record.

The one thing worth remembering

An AI construction claims platform lives or dies on whether it understands time. Every hard requirement in this build traces back to that: numeric dates so you can filter to a contractual window, a data date on every schedule row so float movement is measurable, a corpus flag so contemporaneous records are not drowned out by material written to argue about them, and citations that resolve to a page so any answer can be checked.

Get the temporal metadata right at ingestion and the rest of the architecture is ordinary AWS work. Get it wrong and you have built a very expensive way to retrieve the most persuasive document instead of the true one.

Need help building this on AWS?

I design and build document and data platforms on AWS, and this kind of system sits squarely in that work. Things I can help with:

• Designing the ingestion pipeline: S3 landing zones with Object Lock, Step Functions orchestration, splitting and classifying scanned bundles, and confidence-based routing to human review.
• Getting the temporal metadata model right, including date derivation, sidecar generation and filter design against Amazon Bedrock Knowledge Bases.
• Parsing Primavera P6 XER and PMXML exports into queryable tables in Aurora PostgreSQL or S3 with Glue and Athena, with a data date on every row.
• Choosing and sizing the vector layer across Amazon S3 Vectors, OpenSearch Serverless and pgvector, based on your actual query pattern rather than a benchmark.
• Building index and IAM separation for privileged material, plus KMS key strategy and Macie scanning of landing buckets.
• Setting up retrieval evaluation, citation enforcement, query audit logging and CloudWatch or Grafana dashboards over the pipeline so failures surface early.

If you are partway into something like this already, send me a sample metadata sidecar, a Step Functions execution history, or a query that returns the wrong document, and I will tell you what I think is going on.

The post Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny appeared first on John Nessime.

You pull a sample of the extra rows and they all look fine. Real opportunity IDs, real amounts, real owners. You paste one into Salesforce and get nothing. The record was deleted eight months ago. So were the others.

Your incremental extract has never deleted a row in its life. It asks Salesforce for everything where SystemModstamp is greater than the last run, and a record that no longer exists cannot come back in that result set. So the lake only ever grows, the gap widens a little every week, and nothing anywhere reports an error.

That is the defining problem when you build a Salesforce data lake on AWS, and it is barely mentioned in the tutorials, which mostly stop once data lands in S3. Landing the data is the easy part. Keeping it equal to the source is the work.

This covers the ingestion options and their honest trade-offs, how to structure the lake, how to handle deletes properly (harder than it sounds), the fields that go stale without telling you, staying inside your API budget, and the query layer on top.

Why Salesforce is an awkward source

Four properties make it different from replicating a database, and every design decision below follows from them.

• Deletes are soft, then permanent. A deleted record moves to the Recycle Bin, where it is visible for a limited window and then gone. Your ability to detect a deletion expires.
• Some fields are computed at read time. Formula fields are not stored. Salesforce works them out when you query, which means what you extracted is a snapshot, not a value that stays true.
• The API is a metered resource. Your org has a daily request allowance shared with every other integration. A careless extract can starve the tools sales actually uses.
• The schema changes without warning. Admins add fields as part of their normal job. Nobody tells the data team.

Choosing how to get the data out

Four realistic paths, roughly in order of how much you have to operate yourself.

Amazon AppFlow is the native option: a managed connector that moves Salesforce objects into S3 on a schedule or on events, with no infrastructure to run. It handles both scheduled batch extracts and Salesforce Change Data Capture event flows, and it supports AWS PrivateLink to Salesforce, so the traffic does not have to cross the public internet. That last point matters more than it usually gets credit for.

Where it gets thin: schema drift handling is basic, and mapping large numbers of objects through the console gets tedious fast. Define the flows in Terraform or CloudFormation from the start rather than clicking them, or you will end up with production configuration nobody can reproduce.

A managed ELT vendor such as Fivetran or Airbyte handles schema drift, deletes and incremental logic for you, which is genuinely most of the hard work in this post. You pay per row or per connector, and on a large Salesforce org that number gets attention. Worth pricing against the engineering time before dismissing it.

Rolling your own on the Bulk API gives you total control and makes you responsible for everything: pagination, retries, API budget, delete detection, schema evolution. I would only choose this when an existing tool genuinely cannot express what you need, and I would expect it to be a real service with monitoring rather than a Lambda someone wrote in an afternoon.

Zero-copy sharing via Salesforce Data Cloud avoids replication entirely by querying Salesforce-held data from your warehouse. Attractive when compliance objects to copying CRM data, and it carries its own licensing and latency questions. Check what your Salesforce contract actually includes before designing around it.

Structure the lake so mistakes are cheap

Two layers, and the discipline to keep them separate.

Raw is append-only and never edited. Every extract lands as a new partition, in Parquet, exactly as Salesforce returned it. You do not deduplicate here, you do not fix types here, and you certainly do not delete here. Raw is your ability to rebuild everything downstream when you discover a logic bug six months in, which you will.

# Partition by extraction date, not by a business date. A bad run
# is then one partition to drop rather than a full-table repair.
s3://acme-lake/raw/salesforce/opportunity/extract_date=YYYY-MM-DD/

Curated is what people query: deduplicated to current state, typed properly, deletes applied. Build it as views over raw to begin with, and only materialise the ones that are genuinely too slow. Materialising early gives you a second copy of the truth to keep in sync, which is the problem you are already trying to solve.

-- Current state from an append-only raw layer: keep the most
-- recent version of each record and drop anything flagged deleted.
CREATE OR REPLACE VIEW curated.opportunity AS
SELECT *
FROM (
    SELECT o.*,
           ROW_NUMBER() OVER (
               PARTITION BY id ORDER BY systemmodstamp DESC
           ) AS rn
    FROM raw.opportunity o
)
WHERE rn = 1
  AND is_deleted = false;

Register both in the Glue Data Catalog so Athena and Redshift Spectrum see the same definitions. If you expect frequent updates and deletes rather than pure appends, an open table format like Apache Iceberg is worth the extra setup: row-level deletes, schema evolution and time travel are exactly the operations this workload needs, and they are painful to hand-roll on plain Parquet.

Handling deletes properly

Here is the section that matters. There are three mechanisms and you need more than one, because each has a hole.

1. Query the Recycle Bin

Standard SOQL hides deleted records. The REST API’s queryAll endpoint, the SOAP queryAll() call, or ALL ROWS in Apex all return them, and IsDeleted exists on virtually every object even though it does not appear in Setup.

-- Against /services/data/vXX.X/queryAll, not /query.
-- Substitute your own high-water mark for the placeholder.
SELECT Id, IsDeleted, SystemModstamp
FROM Opportunity
WHERE IsDeleted = true
  AND SystemModstamp > {last_run_utc}

The hole: records sit in the Recycle Bin for fifteen days by default. Salesforce Classic offers an extended retention setting that pushes it to thirty. But the bin also has a capacity tied to your org’s storage allocation, and when it fills, the oldest records are purged early to make room. No alert, no warning. So your detection window is not fifteen days. It is fifteen days or less, and you find out which during a mass-delete week.

2. Change Data Capture

Salesforce CDC publishes create, update, delete and undelete events as they happen, and AppFlow can consume them. This is the cleanest mechanism, because a delete arrives as an event rather than being inferred from an absence.

The hole: it is a streaming subscription, so if your consumer is down long enough, you miss events. Event replay windows are finite. CDC is a good primary mechanism and a bad only mechanism.

3. Periodic full ID reconciliation

This is the safety net, it catches everything the other two miss, and it is the one people skip because it feels crude. Pull nothing but the Id column for the whole object. One narrow field over the Bulk API is cheap even on millions of rows.

SELECT Id FROM Opportunity

Then diff it against the lake. Whatever is on your side and not on theirs no longer exists, whatever the reason:

-- Rows the lake still believes in and Salesforce does not.
SELECT l.id
FROM curated.opportunity l
LEFT JOIN staging.opportunity_ids s
       ON s.id = l.id
WHERE s.id IS NULL;

Run it weekly on your important objects. Soft-delete the differences in the curated layer rather than hard-deleting from raw, so you keep the audit trail.

One more case this catches that nothing else does cleanly: merges. When an admin merges two Accounts, Contacts or Leads, the losing record is deleted and gets a MasterRecordId pointing at the survivor. If your lake keeps the loser, you are double-counting a customer, and the arithmetic is wrong in a way that looks like a data quality problem rather than a pipeline problem.

The other thing that goes stale silently

Formula fields are not stored anywhere. Salesforce computes them when you ask. So the value that landed in your lake is a snapshot from extraction time, and Salesforce will happily return something different tomorrow without the record’s SystemModstamp moving at all.

Cross-object formulas are the worst version. A formula on Opportunity that references a field on its Account recalculates when the Account changes. The Opportunity did not change, so your incremental extract never picks it up. That column in your lake can be wrong indefinitely and nothing detects it.

Three ways out, in order of preference:

1. Do not extract them. Pull the underlying fields and reimplement the calculation in your transformation layer, where it is version-controlled and testable. More work up front, correct forever after.
2. Refresh them on a schedule. A narrow periodic extract of just the ID plus the formula columns, full-table. Cheap if the column list is short.
3. Accept it and document it. Fine for a formula nobody reports on. Not fine for anything in a revenue calculation.

Keep a list of which columns in your lake are formula-derived. Six months from now, when a number disagrees with Salesforce, that list is the first thing you will want and the last thing anybody wrote down.

Stay inside the API budget

Your Salesforce org has a daily API request allowance determined by edition and licence count, and it is shared. Exceed it and everything integrated with Salesforce stops working, not just your pipeline. That includes whatever marketing and support depend on, which is a conversation you would rather not have.

• Use the Bulk API for extracts, not the REST query API. Bulk is designed for volume and consumes the allowance very differently.
• Incremental by default, full refresh by exception. A nightly full extract of a large object is the single most common way to blow the budget.
• Give the pipeline its own integration user. That way the API usage reports attribute consumption to it, and you can see your own footprint instead of arguing about it.
• Watch consumption as a metric. Salesforce exposes API usage; graph it. Finding out by outage is expensive.

One thing that surprises people: field-level security on that integration user determines which fields come back. A field can exist in Salesforce, be populated, and arrive in your lake as consistently null because the integration user’s profile cannot see it. Check the profile before debugging the pipeline.

Query layer and access

Athena over the Glue catalog is the default answer and a good one: no cluster to run, pay per query, and it reads the curated views directly. Redshift Spectrum makes sense when you already run Redshift and want to join CRM data to warehouse tables.

Cost control on Athena is mostly about scan volume, which means partitioning and columnar storage rather than query tuning. Parquet plus sensible partitions does more than any amount of SQL cleverness.

On access: you have just replicated your customer database into object storage. Contacts and Leads are personal data, and Salesforce’s own field-level security does not follow it across. Encrypt the bucket, keep it private, use Lake Formation for column-level grants if different teams need different views, and decide the retention policy deliberately rather than by default. “We keep everything forever in S3 because it’s cheap” is a defensible engineering position and a poor compliance one.

Troubleshooting

Row counts drift upward over time

Deletes are not being applied. Run the full ID reconciliation and see how many rows come back. If it is a lot, also check for merged records via MasterRecordId before assuming the extract is at fault.

A column is always null in the lake, populated in Salesforce

Field-level security on the integration user’s profile. Log in as that user, or check the profile’s field permissions, before touching the pipeline.

A value disagrees with Salesforce but the record looks current

Almost certainly a formula field, especially a cross-object one. Check whether the column is formula-derived; if it is, the record’s modstamp never moved and your incremental extract had no reason to refetch it.

Extract fails partway on a large object

Query timeouts or governor limits. Chunk by date range or by ID range rather than pulling the object in one go, and make each chunk independently retryable so a failure costs you one slice rather than the whole run.

New Salesforce fields never appear

Most connectors map fields explicitly at configuration time and do not add new ones on their own. Poll the object’s describe metadata on a schedule and alert when the field list changes, so schema drift is a notification rather than a discovery.

Athena queries are slow or expensive

You are scanning too much. Check partitioning first, then whether the files are Parquet rather than JSON or CSV, then whether many tiny files are being read per query. Small-file proliferation is the usual culprit on frequently-run extracts, and compaction fixes it.

Common mistakes

• Building incremental sync on SystemModstamp alone and never handling deletes.
• Assuming the Recycle Bin gives you a guaranteed fifteen-day detection window.
• Treating CDC as sufficient on its own, with no reconciliation behind it.
• Ignoring merges, so a merged customer is counted twice forever.
• Extracting formula fields and treating them as durable values.
• Nightly full extracts of large objects, and the API outage that follows.
• Sharing an integration user with other tools, so nobody can attribute API consumption.
• Mutating the raw layer, which destroys your ability to rebuild.
• Clicking AppFlow flows together in the console with no infrastructure as code.
• Landing JSON or CSV instead of Parquet, then paying for it on every Athena query.
• No alerting on schema drift, so new fields are found by a user asking.
• Replicating Contacts and Leads into S3 with no retention policy or access controls.

Best practices

• Append-only raw layer, curated views on top, and never edit raw.
• Use at least two delete mechanisms, one of which is periodic full ID reconciliation.
• Soft-delete in curated rather than hard-deleting, so the audit trail survives.
• Reimplement formula logic downstream instead of trusting extracted formula values.
• Bulk API for extracts, incremental by default.
• A dedicated integration user with a documented, deliberately scoped profile.
• Partition by extraction date and store Parquet, or use Iceberg if you need row-level updates.
• Define flows, catalogs and permissions as code.
• Monitor API consumption and row-count delta against Salesforce as first-class metrics.
• Alert on schema changes rather than discovering them.
• Encrypt, restrict and set a retention policy on personal data the day you land it, not later.

FAQ

How do I handle deleted Salesforce records in a data lake?

Combine mechanisms. Query the Recycle Bin with queryAll and IsDeleted for recent deletions, consume Change Data Capture events for real-time coverage, and run a periodic full ID reconciliation as the backstop. No single one of the three is complete on its own.

Should I use AppFlow or a third-party ELT tool?

AppFlow if you want to stay inside AWS, value PrivateLink connectivity, and are willing to handle schema drift and deletes yourself. A managed vendor if you would rather buy those solved and can live with per-row pricing. Price both against the engineering time honestly; the build-it-yourself option is usually costed at zero and is not.

Parquet or Iceberg?

Parquet with date partitions is fine for append-only raw. Iceberg earns its complexity in the curated layer, where you want row-level updates and deletes, schema evolution and time travel. Salesforce data changes constantly, so that need is real rather than theoretical.

How often should I sync?

Match the decisions people actually make with the data. Hourly incrementals suit most reporting; sub-minute freshness usually means CDC and a real streaming consumer, which is a much larger commitment. Schedule the ID reconciliation separately and less often, weekly is normally enough.

Will this use up my Salesforce API limit?

It can, and the failure is shared with every other integration on the org. Use the Bulk API, stay incremental, give the pipeline its own user so consumption is attributable, and graph usage. Do not find out from an outage.

Do I need Salesforce Data Cloud?

Not to build a lake on AWS. It becomes interesting when compliance objects to copying CRM data at all, since zero-copy sharing avoids replication. Check what your existing Salesforce agreement covers before designing around it, because the licensing is a real factor.

Why do my numbers differ from Salesforce reports?

Check in this order: missing deletes, merged records counted twice, stale formula fields, fields the integration user cannot see, and only then your own transformation logic. The first two account for most of it, and the last is where people look first.

The one thing to remember

Getting Salesforce data into S3 is a configuration exercise you can finish in an afternoon. Keeping that data equal to Salesforce is an ongoing engineering problem, and the parts that break do not raise errors. They produce numbers that are slightly wrong, in a consistent direction, for months.

So build the reconciliation before you build the dashboards. A weekly job that pulls every ID and diffs it against the lake is unglamorous, cheap, and the only thing that will tell you the truth when someone asks why the two systems disagree.

Building or fixing one of these?

Most of the Salesforce lakes I get asked to look at work correctly for the first quarter and then quietly diverge. Work I take on:

• Building a Salesforce to S3 pipeline end to end: AppFlow or Bulk API ingestion, Glue catalog, Athena or Redshift Spectrum query layer.
• Auditing an existing lake for drift, and reporting exactly how many rows disagree with Salesforce and why.
• Implementing proper delete handling: CDC consumption, Recycle Bin queries and scheduled ID reconciliation.
• Reworking formula-derived columns into version-controlled transformations that stay correct.
• API budget work: moving extracts to Bulk, splitting integration users, and monitoring consumption before it causes an outage.
• Access and retention design for CRM data in S3, including encryption, Lake Formation grants and deletion policies.

Tell me which objects you replicate and how you currently detect deletes, and I will tell you where the drift is coming from.

The post The Row Count That Only Goes Up: Building a Salesforce Data Lake on AWS appeared first on John Nessime.

Nothing failed. Sync Design ran exactly as designed. What it did not do is fetch any new column information, because there is an unresolved mismatch sitting in the connection, on a different table, from a rename somebody did two months ago that nobody noticed. Until that is cleared, design sync will keep running and keep declining to pick up anything new.

That behaviour is documented, not a bug, and it is the single most useful thing to know about a Zoho Analytics Redshift schema mismatch. Mismatches are not independent little problems you can leave lying around. One of them jams the mechanism for all of them.

This walks through the three families these problems come in, names, types and time, the fix for each, and the structural change that stops them recurring: stop pointing the BI tool at your base tables.

First: work out which mode you are actually in

Zoho Analytics connects to Redshift two completely different ways, and half the confusion in this area comes from people reading advice written for the other one. Whoever set the connection up may have left, so check rather than assume.

Data Import copies the data into Zoho Analytics on a schedule. Reports are fast because they run against Zoho’s own storage. In this mode:

• Column additions and deletions are synchronised automatically.
• You can change a column’s data type inside Zoho Analytics, but the type has to stay compatible with the Redshift column or subsequent syncs fail. Zoho’s own guidance is to change it in both places, which is worth taking literally.
• You can create query tables, and import a filtered subset using a custom query.

Live Connect keeps nothing locally and queries Redshift when a report loads. It is available on the paid tiers only. In this mode:

• Column additions, deletions and renames are not synchronised automatically. You have to trigger Sync Design from the Edit Redshift Settings page.
• You cannot change a column’s data type in Zoho Analytics at all. Whatever Redshift says, that is what you get.
• No query tables, and you cannot pull other data sources into that workspace.
• Foreign keys defined in Redshift become lookup relationships automatically, which is a genuine advantage over Data Import, where you build those by hand.
• Report loading time is your Redshift cluster’s problem now.

The practical consequence: in Live Connect, every schema fix has to happen in Redshift. There is no BI-side escape hatch. That constraint sounds annoying and is actually the thing that pushes you toward the right architecture, which is the last section of this post.

Family 1: names

Zoho keeps its own copy of the table and column names it expects. When Redshift’s names drift away from that copy, the difference shows up in the Mismatch tab of the connection settings, and the two most common causes are both silent.

Case folding

Redshift lowercases unquoted identifiers. Somebody writes what looks like a camel-case column name, Redshift stores something else, and the BI tool is now looking for a column that does not exist under that name.

-- These two do NOT create the same column.
ALTER TABLE analytics.orders ADD COLUMN DiscountAmount DECIMAL(12,2);
-- ...stored as: discountamount

ALTER TABLE analytics.orders ADD COLUMN "DiscountAmount" DECIMAL(12,2);
-- ...stored as: DiscountAmount

Pick one convention, lowercase with underscores, and enforce it. Mixed quoting across a schema means some columns are case-sensitive and some are not, and you will spend an afternoon working out which.

See what Redshift actually has

Before touching anything in Zoho, get the ground truth. Use SVV_COLUMNS rather than PG_TABLE_DEF, because the latter only returns rows for schemas that happen to be in your search_path and silently returns nothing otherwise, which has wasted a lot of people’s time.

SELECT table_name,
       column_name,
       ordinal_position,
       data_type,
       character_maximum_length,
       numeric_precision,
       numeric_scale
FROM svv_columns
WHERE table_schema = 'analytics'
ORDER BY table_name, ordinal_position;

Export that, put it next to the Mismatch tab, and work down the list. Guessing from memory is how you resolve four mismatches and leave the fifth.

Renames and drops

A rename upstream reads to Zoho as one column disappearing and an unrelated one appearing. If a report or formula referenced the old name, you will also see the alert about a view that cannot be accessed because of changes made to the table. The fix there is to re-synchronise the table from the connection settings, but re-syncing will not help while the Mismatch tab still has entries.

So the order is fixed: clear every mismatch first, then Sync Design, then fix reports. Doing it in any other order produces the “I clicked sync and nothing happened” experience.

Family 2: types

Redshift has a rich type system. A BI tool has maybe a dozen column types. The mapping is lossy in places, and the losses are quiet.

TEXT and BPCHAR are not what they look like

This one catches people migrating from PostgreSQL, where TEXT is unbounded. In Redshift it is an alias that becomes VARCHAR(256), and BPCHAR becomes CHAR(256). Longer values get rejected or truncated depending on how they arrive, and the column reaching Zoho is a 256-character string rather than the free text you thought you had.

-- Looks unbounded. Is not.
CREATE TABLE staging.notes (body TEXT);        -- VARCHAR(256)

-- Say what you mean. 65535 bytes is the VARCHAR maximum.
CREATE TABLE staging.notes (body VARCHAR(65535));

VARCHAR length is measured in bytes

Not characters. An accented Latin character costs two bytes, most CJK characters three, an emoji four. A VARCHAR(50) holds fifty English letters or twelve emoji. Names, addresses and free-text fields with international data hit this constantly, and the symptom in the dashboard is a truncated string rather than an error.

-- LENGTH counts characters, OCTET_LENGTH counts bytes.
-- The second number is the one that has to fit.
SELECT MAX(LENGTH(customer_name))       AS max_chars,
       MAX(OCTET_LENGTH(customer_name)) AS max_bytes
FROM analytics.customers;

Widening a VARCHAR is one of the few in-place alterations Redshift allows. Narrowing one, or changing a column’s type outright, generally means rebuilding the table, so size these deliberately at creation rather than planning to fix them later.

Numbers and precision

A DECIMAL(38,10) is a perfectly reasonable warehouse column and an awkward BI column. Currency stored as a float is worse, because you get rounding that appears only in the total row and only sometimes, which is a genuinely unpleasant thing to debug in front of a finance team.

Cast money to a fixed scale before it leaves Redshift. Two decimal places, DECIMAL not FLOAT, decided once in the warehouse rather than per-report in the BI tool.

SUPER, and anything else with no BI equivalent

Semi-structured SUPER columns, VARBYTE, GEOMETRY, HLLSKETCH: there is no sensible column type on the other side. Do not expose them. Flatten what you need into typed scalar columns in a view and leave the rest in the warehouse.

If you are in Data Import mode

You have the option of overriding a column’s type on the Zoho side. Use it sparingly. Zoho’s requirement is that the type stays compatible with Redshift’s, and “compatible” is doing quiet work in that sentence: an override that works today breaks the next sync when a value arrives that the Zoho type cannot hold. Changing it in both places, as Zoho recommends, is the version that keeps working.

Family 3: time

This one does not appear as a mismatch anywhere. It appears as a reconciliation problem, which is worse, because you spend the first hour looking for missing rows.

Redshift has TIMESTAMP, which carries no timezone and means whatever the writer intended, and TIMESTAMPTZ, which is stored in UTC. A BI tool has a timezone setting of its own. Between those, a row written at 23:40 local time can be counted on a different day at each end.

The tell is specific and worth memorising: daily totals match, monthly totals do not. Nothing is missing. A few hours’ worth of rows at each month boundary are being attributed to the neighbouring month.

-- Run the dashboard's aggregate directly against Redshift and
-- compare. Relative bounds so this keeps working next month.
SELECT DATE_TRUNC('day', created_at) AS day,
       COUNT(*)                      AS orders,
       SUM(total_amount)             AS revenue
FROM analytics.orders
WHERE created_at >= DATEADD(month, -1, DATE_TRUNC('month', GETDATE()))
  AND created_at <  DATE_TRUNC('month', GETDATE())
GROUP BY 1
ORDER BY 1;

The fix is to stop making the BI tool guess. Convert in Redshift, expose both the UTC instant and a pre-computed local date, and build every report on the pre-computed one:

-- CONVERT_TIMEZONE is the Redshift idiom. Doing this once here
-- beats doing it in every report and getting it right in most.
SELECT
    created_at                                            AS created_at_utc,
    CONVERT_TIMEZONE('UTC', 'Europe/London', created_at)  AS created_at_local,
    CAST(CONVERT_TIMEZONE('UTC', 'Europe/London', created_at) AS DATE)
                                                          AS order_date_local
FROM analytics.orders;

Named zones rather than fixed offsets, so daylight saving is handled for you. A hardcoded offset is correct for roughly half the year.

The structural fix: give Zoho a contract, not your tables

Everything above is treatment. This is prevention, and it is the part worth doing even if nothing is currently broken.

Pointing a BI tool at base tables means every upstream change is a potential BI incident. Someone widening a column, renaming a field, or adding a SUPER column for a new feature has no idea a dashboard depends on it. Put a view layer in between and that stops being true: the view is the interface, the tables underneath are free to change, and you decide when the interface changes.

CREATE OR REPLACE VIEW analytics.v_orders_bi AS
SELECT
    -- Explicit casts pin the types Zoho will see, so an upstream
    -- change cannot quietly alter the shape of the report.
    CAST(o.order_id      AS BIGINT)         AS order_id,
    CAST(o.order_status  AS VARCHAR(64))    AS order_status,
    CAST(o.total_amount  AS DECIMAL(18,2))  AS total_amount,

    o.created_at                            AS created_at_utc,
    CAST(CONVERT_TIMEZONE('UTC','Europe/London', o.created_at) AS DATE)
                                            AS order_date_local,

    -- SUPER flattened to something a BI column can hold.
    CAST(o.attributes.channel AS VARCHAR(64)) AS channel
FROM analytics.orders o
WITH NO SCHEMA BINDING;

WITH NO SCHEMA BINDING creates a late-binding view: it does not hold a dependency on the underlying table, so you can drop and recreate analytics.orders without Redshift refusing or the view vanishing. For a warehouse where tables get rebuilt by a nightly load, that is the difference between a maintenance window and a broken dashboard.

List columns explicitly. Never SELECT * in a view a BI tool depends on, because then any upstream column addition changes the contract without anybody deciding to.

Give the connection its own read-only Redshift user, granted access to the views and nothing else. That also means the credentials in the BI tool cannot read tables you did not intend to publish:

CREATE USER zoho_reader PASSWORD 'use-a-generated-one';
GRANT USAGE ON SCHEMA analytics TO zoho_reader;
GRANT SELECT ON analytics.v_orders_bi TO zoho_reader;

The honest cost: a view layer is a thing to maintain, and adding a column now means editing the view as well as the table. That is the point. The friction is the control. If your views are getting numerous, managing them with dbt or an equivalent gives you version control and review on what is otherwise a pile of undocumented SQL.

A repeatable resolution procedure

1. Get ground truth from Redshift. Run the SVV_COLUMNS query and save the output.
2. Open the Mismatch tab in the connection settings and list every entry, including ones on tables nobody reports on.
3. Resolve every mismatch. All of them. A single leftover blocks design sync for everything else.
4. Trigger Sync Design and confirm the new columns actually appear before moving on.
5. Check types, not just names. A column can sync successfully and still be the wrong type. Spot-check the ones carrying money and dates.
6. Reconcile a known number. Run the same aggregate in both places for a closed period. Daily and monthly. If daily matches and monthly does not, go back to the timezone section.
7. Fix broken reports last, once the data underneath them is right.

Troubleshooting

Sync Design runs but the new column never appears

An unresolved mismatch is blocking it, almost certainly on a table you were not looking at. Clear the Mismatch tab completely, then sync again.

“This view cannot be accessed due to some changes made in the table”

Something the report depends on was renamed or deleted in Redshift. Re-synchronise that table from the connection settings. If it recurs after every deployment, that is the argument for the view layer.

The connection failed entirely

Check three things in order: whether the Redshift database was renamed or dropped, whether Zoho’s IP addresses are still allowlisted in your security group, and whether the credentials still work. A renamed database needs the connection edited; a dropped one means starting over.

Reports show old data after a schema fix

In Live Connect, caching can be enabled per workspace with its own refresh interval, and it applies to reports rather than tables. If the numbers look stale after you fixed something, check that setting before you conclude the fix did not work.

Text is truncated in the dashboard but complete in Redshift

Byte-length truncation on multi-byte characters, or a TEXT column that quietly became VARCHAR(256). Compare LENGTH against OCTET_LENGTH and widen the column.

Totals are close but not equal

Timezone if the gap sits at period boundaries. Precision if it is a consistent tiny drift across everything. Filters that differ between the report and your reconciliation query if it is neither. Check them in that order, because the first two are far more common than a genuinely missing row.

Common mistakes

• Fixing one mismatch, running Sync Design, and assuming the rest can wait.
• Not knowing whether the connection is Data Import or Live Connect, and applying advice for the wrong one.
• Overriding a column type in Zoho without changing it in Redshift, so the next sync fails.
• Using TEXT in Redshift and expecting PostgreSQL behaviour.
• Sizing VARCHAR by character count when the limit is in bytes.
• Storing currency as a float.
• Mixing quoted and unquoted identifiers, so some column names are case-sensitive and some are not.
• Letting reports use raw timestamps and setting the timezone per report.
• Using a fixed UTC offset instead of a named timezone.
• Pointing the BI tool at base tables that a nightly job rebuilds.
• SELECT * in a view that a dashboard depends on.
• Connecting with an admin-level Redshift user because it was quicker.
• Declaring the fix done without reconciling a number against the warehouse.

Best practices

• Expose late-binding views to the BI tool, never base tables.
• Cast every column explicitly in the view so the types are decided, not inferred.
• Do timezone conversion in Redshift and publish a pre-computed local date.
• Lowercase, underscore-separated identifiers everywhere, unquoted.
• Fixed-scale DECIMAL for money, never floating point.
• Size VARCHAR against OCTET_LENGTH of real data, with headroom.
• A dedicated read-only Redshift user granted access only to the reporting views.
• Treat the Mismatch tab as a queue to empty, not a list to triage.
• Reconcile at least one aggregate against the warehouse after every schema change.
• Version-control the view definitions, with dbt or just a repository of SQL files.
• Tell whoever owns the upstream tables that a view depends on them.

FAQ

What exactly is a mismatch in Zoho Analytics?

A disagreement between the table and column names Zoho Analytics expects and the ones Redshift currently has. They are listed in the Mismatch tab of the Redshift connection settings. The important property is that leaving one unresolved stops Sync Design from fetching new column information at all.

Why does Sync Design not pick up my new column?

Because there is at least one mismatch outstanding. Clear the Mismatch tab entirely and run it again. It is not a caching issue and re-running it more times will not help.

Should I use Data Import or Live Connect?

Data Import when you want fast dashboards, query tables, and the ability to blend data, and can accept the data being as fresh as the last sync. Live Connect when the numbers must be current and you would rather not duplicate the data, accepting that report speed becomes a Redshift performance question and every schema change needs a manual sync.

Can I change a column’s data type in Zoho Analytics?

In Data Import, yes, provided it stays compatible with the Redshift type. In Live Connect, no. Either way the durable fix is to cast the column correctly in a view on the Redshift side, so both ends agree without anyone having to remember an override exists.

Why do my dashboard totals not match the warehouse?

If daily figures agree and monthly ones do not, it is timezone handling at period boundaries. If everything is off by a tiny consistent amount, it is numeric precision. Genuinely missing rows are the least likely of the three and the one people check first.

How do I handle SUPER columns?

Do not expose them. Extract the specific fields you report on, cast them to scalar types in a view, and let the rest stay in the warehouse. A BI tool has nowhere to put a nested document.

Will a view layer slow down Live Connect reports?

A little, since the casts and conversions run per query. In practice the dominant cost is how much data the query scans, so sort keys and distribution keys on the underlying tables matter far more than the view. Measure before optimising, and if a particular view is genuinely expensive, materialise it as a table refreshed by your load job.

The one thing to remember

A Zoho Analytics Redshift schema mismatch is not a small isolated problem you can leave in the queue. One unresolved entry stops new columns arriving at all, which is why the symptom people report is almost never “there’s a mismatch” and almost always “I added a column and nothing happened”.

Clear them all, then sync, then reconcile a real number rather than trusting that it worked. And once it is working, spend the afternoon putting a view layer in between, because the alternative is having this conversation again the next time someone upstream renames a field they had no idea you were reading.

Need this sorted out properly?

Warehouse-to-BI connections tend to be set up once, by someone who has since moved on, and then quietly degrade. Work I take on:

• Auditing an existing Redshift to Zoho Analytics connection and clearing the mismatch backlog properly.
• Building a reporting view layer in Redshift with explicit casts, timezone handling and late binding, so upstream changes stop breaking dashboards.
• Reconciling dashboard figures against the warehouse and finding where the difference comes from.
• Redshift schema work: type corrections, column sizing, flattening SUPER data into reportable columns.
• Least-privilege database users and network access for BI tools, including security group and allowlist configuration.
• Putting the view definitions under version control with dbt so schema changes get reviewed instead of discovered.

Send me the output of the SVV_COLUMNS query above and a screenshot of your Mismatch tab, and I will tell you what is actually wrong.

The post The Column That Never Shows Up: Fixing Redshift and Zoho Analytics Schema Mismatches appeared first on John Nessime.

What just happened is that your data warehouse now has an endpoint on the public internet, reachable on port 5439, and a third-party SaaS platform holds credentials to it. Usually those credentials belong to whatever user was to hand, which on a lot of clusters means the admin account that can read every schema you have.

That is not an argument against doing it. It is an argument for doing it on purpose. Most guides on how to connect Amazon Redshift to Zoho Analytics walk you through six screenshots and never mention the two decisions that actually matter: which user this connection runs as, and how much of your network it needs.

This covers the whole setup in order, with those decisions made deliberately: the mode you pick and why it is hard to change later, a read-only user scoped to almost nothing, network access done narrowly, and what to actually expose once it works.

What the connection actually requires

Strip away the interface and the mechanics are simple. Zoho Analytics runs on Zoho’s infrastructure. To reach your cluster it opens an inbound TCP connection from Zoho’s own IP addresses to your Redshift endpoint, on the Redshift port. That is it.

Which means three things have to be true, and they are the three places setup fails:

• The endpoint has to be reachable from outside your VPC. A cluster in a private subnet with no public endpoint cannot be reached by a SaaS tool, full stop.
• The security group has to allow it. Zoho publishes the IP addresses it connects from, and those need an inbound rule on the Redshift port. The default port is 5439, though it can be changed at cluster creation, so check yours rather than assuming.
• The credentials have to work and the user has to be able to see the schemas you want to report on.

Zoho keeps the current allowlist on a dedicated help page. Do not copy a list of addresses out of a blog post, including this one. They change, and a stale allowlist produces an intermittent failure that looks like everything except what it is.

Decide the mode before you start

Zoho offers two fundamentally different connection types, and switching later means rebuilding the workspace. Get this one right the first time.

Data Import copies your data into Zoho Analytics on a schedule and reports run against their storage. Dashboards are fast, you can create query tables, and you can import a filtered subset using a custom query. Column additions and deletions sync automatically. The trade is freshness: your numbers are as current as the last sync.

Live Connect stores nothing and queries Redshift each time a report loads. Numbers are always current and your data does not leave AWS. It is available on the paid tiers only. The trade is bigger than it looks: report speed becomes a Redshift performance problem, schema changes need a manual sync each time, you cannot change column types on the Zoho side at all, and no query tables.

My default for most teams is Data Import, because the operational surface is smaller and a scheduled sync is usually fresh enough for a dashboard people look at once a day. Live Connect earns its cost when the numbers genuinely need to be current, or when duplicating the data into a third-party system is a compliance problem.

Step 1: create a user that cannot do much

Do this before you open any network access, so that when the connection works you already know its blast radius.

-- A dedicated user for this connection and nothing else.
CREATE USER zoho_reader PASSWORD 'generate-a-long-random-one';

-- Reach the schema, then read the objects in it.
-- Both grants are needed; USAGE alone gets you nothing.
GRANT USAGE ON SCHEMA analytics TO zoho_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO zoho_reader;

-- The grant above covers what exists today. This covers what
-- gets created tomorrow, which is the part people forget and
-- then debug as a mysterious missing table three months later.
ALTER DEFAULT PRIVILEGES IN SCHEMA analytics
  GRANT SELECT ON TABLES TO zoho_reader;

Then check what you actually granted, rather than trusting that the statements did what you meant:

SELECT table_schema, table_name, privilege_type
FROM information_schema.table_privileges
WHERE grantee = 'zoho_reader'
ORDER BY table_schema, table_name;

If that returns rows from schemas you did not intend to publish, fix it now. Once the connection is live, someone will build a report on whatever is visible and removing access becomes a conversation instead of a command.

The password goes into a SaaS platform, so treat it accordingly: long, random, generated, stored in your password manager, and rotated when people leave.

Step 2: open exactly as much network as you need

First find out what your cluster currently looks like, because the console tells you less at a glance than this does:

aws redshift describe-clusters 
  --cluster-identifier my-warehouse 
  --query 'Clusters[0].{Endpoint:Endpoint.Address,Port:Endpoint.Port,Public:PubliclyAccessible,SecurityGroups:VpcSecurityGroups}'

That gives you the hostname Zoho needs, the real port, whether the cluster is currently public, and which security groups govern it. Redshift Serverless has an equivalent setting on the workgroup rather than a cluster, but the same reasoning applies.

Then add one narrow inbound rule per Zoho range. Not 0.0.0.0/0, which is what happens when someone is in a hurry and it never gets tightened afterwards:

aws ec2 authorize-security-group-ingress 
  --group-id sg-0123456789abcdef0 
  --protocol tcp 
  --port 5439 
  --cidr 203.0.113.0/24

Put a description on each rule in the console so that in a year somebody can tell which ranges belong to which vendor. An undocumented allow rule is one nobody will ever dare remove.

While you are in the cluster parameter group, turn on require_ssl so connections that do not use TLS are refused rather than merely discouraged. Parameter group changes need a cluster reboot, so do it in a window rather than mid-afternoon.

Test from outside before you touch Zoho

# Does anything answer on the port, from a machine outside your VPC?
nc -zv my-warehouse.abc123.eu-west-1.redshift.amazonaws.com 5439

# Then prove the credentials work, over TLS, as the new user.
psql "host=my-warehouse.abc123.eu-west-1.redshift.amazonaws.com 
      port=5439 dbname=analytics user=zoho_reader sslmode=require"

If both of those work from your laptop, the remaining variable is Zoho’s IP ranges rather than anything structural. If the first one hangs, it is the security group or public accessibility. If it connects but authentication fails, it is the user. Separating those two takes thirty seconds and saves an hour of guessing inside a web form that reports one generic error for both.

If you cannot make the cluster public

Plenty of organisations have a policy against it, and that policy is not unreasonable. Zoho’s answer for databases behind a firewall is Zoho Databridge, a lightweight agent you install on a machine inside your own network. It opens an outbound connection on 443 to Zoho and waits for requests, so nothing inbound has to be allowed at all. It runs on Windows, macOS and Linux, and it is designed for exactly this situation.

Worth checking before you commit to it: Databridge is documented against a long list of relational databases, and Redshift is not always named explicitly in that list. Since Redshift speaks the PostgreSQL wire protocol and has a JDBC driver, it is usually workable, but confirm with Zoho support for your plan rather than taking my word for it. If it fits, an outbound-only agent is a meaningfully better architecture than a public endpoint, and almost nobody setting this up knows the option exists.

Step 3: make the connection

With the groundwork done, this part is genuinely a form. In Zoho Analytics, create a workspace, choose to import from or connect to Amazon Redshift, and supply the endpoint hostname, the port, the database name, and the zoho_reader credentials.

Two expectations worth setting so you do not diagnose a non-problem:

• The first fetch takes a while, depending on volume and how fast your cluster responds. Zoho emails you when it finishes.
• An empty workspace before that fetch completes is normal. If you open it early it will show nothing, which looks exactly like a failure and is not one.

Step 4: choose what to expose, carefully

This is the screen people click through fastest and the one with the longest consequences. Whatever tables you select here become the interface between your warehouse and your dashboards, and every future upstream change becomes a potential BI incident.

Select views, not base tables. A reporting view with explicit casts, timezone conversion done in Redshift, and semi-structured columns already flattened gives you a stable contract. The tables underneath stay free to change.

CREATE OR REPLACE VIEW analytics.v_orders_bi AS
SELECT
    CAST(order_id     AS BIGINT)        AS order_id,
    CAST(order_status AS VARCHAR(64))   AS order_status,
    CAST(total_amount AS DECIMAL(18,2)) AS total_amount,
    created_at                          AS created_at_utc,
    CAST(CONVERT_TIMEZONE('UTC','Europe/London', created_at) AS DATE)
                                        AS order_date_local
FROM analytics.orders
WITH NO SCHEMA BINDING;

WITH NO SCHEMA BINDING makes it a late-binding view, so a nightly job can drop and rebuild the underlying table without Redshift refusing or the view disappearing. On a warehouse with a rebuild-style load, that alone prevents a recurring class of broken-dashboard morning.

Grant your reader access to the views specifically, and skip the blanket grant on the whole schema if you can. Fewer objects visible means fewer things somebody can accidentally build a report on.

Step 5: schedule it, then verify a real number

In Data Import mode, set the sync schedule to match how the data is actually produced. Syncing hourly when the warehouse loads once at 03:00 just means twenty-three pointless queries a day against your cluster, and Redshift bills by uptime and workload.

Then do the step everyone skips. Pick a closed period, run the same aggregate in both places, and compare:

SELECT DATE_TRUNC('day', created_at) AS day,
       COUNT(*)                      AS orders,
       SUM(total_amount)             AS revenue
FROM analytics.v_orders_bi
WHERE created_at_utc >= DATEADD(month, -1, DATE_TRUNC('month', GETDATE()))
  AND created_at_utc <  DATE_TRUNC('month', GETDATE())
GROUP BY 1
ORDER BY 1;

A connection that returns data is not the same as a connection that returns correct data. If daily figures agree but monthly ones do not, you have a timezone problem rather than a connection problem, and it is far easier to find now than six weeks into someone’s board pack.

Troubleshooting

Connection times out

Network, not credentials. A timeout means nothing answered. Check public accessibility, then the security group rule, then whether Zoho’s allowlist has changed since you configured it. A refused connection rather than a timeout usually means you reached the right host on the wrong port.

Authentication failed

The network is fine, which is genuine progress. Test the same credentials with psql from your own machine. Also check the database name: Redshift clusters often have both a default database and the one you actually use, and connecting to the wrong one authenticates fine and then shows you nothing.

Connected, but no tables listed

A permissions gap. USAGE on the schema and SELECT on the objects are separate grants and you need both. Run the information_schema.table_privileges query as a check rather than re-running the grants and hoping.

Worked on setup, fails intermittently later

Classic symptom of a partially stale IP allowlist: some of Zoho’s ranges reach you and some do not, so syncs succeed or fail depending on which host tries. Re-check the published list against your security group rules.

Sync succeeds but a new column never appears

In Live Connect, schema changes need a manual Sync Design, and an unresolved mismatch anywhere in the connection stops it fetching new column information at all. Clear the Mismatch tab completely, then sync again.

Reports are slow

In Live Connect that is your cluster, not Zoho. Look at the queries hitting Redshift and at sort and distribution keys on the underlying tables. Zoho also offers per-workspace caching for Live Connect with a configurable refresh interval, which trades freshness for speed.

Common mistakes

• Connecting with the cluster admin user because it was already in the password manager.
• Opening the Redshift port to 0.0.0.0/0 to make the test pass, then never tightening it.
• Copying an IP allowlist out of a blog post instead of Zoho’s own page.
• Granting USAGE without SELECT, or the reverse, and concluding the connector is broken.
• Forgetting ALTER DEFAULT PRIVILEGES, so tables created later are invisible.
• Pointing the connection at base tables rather than reporting views.
• Picking Live Connect for the freshness without accounting for the manual sync on every schema change.
• Assuming the setup failed because the workspace is empty during the initial fetch.
• Scheduling syncs far more often than the warehouse is actually loaded.
• Not enforcing TLS on the cluster.
• Declaring it done without reconciling a single number against Redshift.
• Leaving no note anywhere about which security group rules belong to which vendor.

Best practices

• A dedicated read-only user per integration, never a shared or admin account.
• Grant access to reporting views only, not whole schemas, where you can.
• One narrow security group rule per vendor range, described and dated.
• Enforce TLS with require_ssl rather than trusting the client to ask for it.
• Prefer an outbound-only agent over a public endpoint if your plan and setup support it.
• Decide Data Import versus Live Connect deliberately, because changing it means rebuilding.
• Expose late-binding views with explicit casts and timezone conversion done in Redshift.
• Match the sync schedule to your load schedule, not to how fresh you wish the data were.
• Reconcile at least one aggregate before anyone builds a dashboard on it.
• Rotate the connection password when people leave, and document where it lives.
• Test connectivity and credentials separately, from outside the VPC, before blaming the connector.

FAQ

Does Redshift have to be publicly accessible?

For a direct connection, yes: Zoho reaches your cluster inbound from its own IP addresses, so there has to be an endpoint it can resolve and reach. The alternative is Zoho Databridge, an agent inside your network that connects outbound on 443, which removes the inbound requirement entirely. Confirm Redshift support for it with Zoho before planning around it.

Which port does Zoho need open?

Whichever port your cluster listens on. 5439 is the Redshift default, but it can be set to something else when the cluster is created, so read it from describe-clusters rather than assuming.

Should I use Data Import or Live Connect?

Data Import for most cases: faster dashboards, query tables, automatic column syncing, and a smaller operational surface. Live Connect when the numbers must be current or copying data into a third party is a compliance issue, accepting slower reports and manual schema syncs. Decide before you build, because switching means starting the workspace over.

What permissions does the Zoho user need?

USAGE on the schema and SELECT on the objects you want reported on. Nothing else. Add ALTER DEFAULT PRIVILEGES so objects created later are covered without anybody having to remember.

Why is the workspace empty after I finish setup?

The initial fetch has not finished. Zoho emails you when it completes, and the workspace shows nothing until then. Give it time before you start pulling the configuration apart.

Can I limit which tables Zoho can see?

Yes, and you should. Grant the reader access only to a set of reporting views in a dedicated schema. That is a stronger control than the table picker in the interface, because it holds even if somebody later edits the connection.

Will this increase my Redshift bill?

Live Connect can, since every report load is a query. Data Import costs you one scheduled query run per sync. If cost matters, Data Import on a schedule matched to your load window is the cheaper shape by a wide margin.

The one thing to remember

Getting Zoho Analytics to talk to Redshift takes about twenty minutes. Getting it to talk to Redshift without handing a SaaS platform a superuser account and an open port takes about an hour, and that hour is the entire difference between a connection you can defend in an audit and one you quietly hope nobody asks about.

So do it in this order: least-privilege user first, narrow network access second, connection third, views rather than tables fourth, and a reconciled number before anyone builds a dashboard on it. Every step after the first is easier when the first one is already done.

Want this set up properly the first time?

Most of these connections get built under time pressure and inherit whatever shortcuts made the test pass. Work I take on:

• Setting up a Redshift to Zoho Analytics connection end to end, with least-privilege credentials and scoped network access.
• Reviewing an existing connection and reporting what it can actually reach, then narrowing it without breaking reports.
• Building the reporting view layer in Redshift: explicit casts, timezone handling, flattened semi-structured columns, late binding.
• Network architecture for BI access, including outbound-agent options where a public endpoint is not acceptable.
• Choosing between Data Import and Live Connect based on your actual freshness, cost and compliance constraints.
• Reconciliation checks so dashboard numbers are verified against the warehouse rather than assumed.

Tell me whether your cluster is currently public and which user the connection runs as, and I will tell you what I would change first.

The post Connect Amazon Redshift to Zoho Analytics Without Putting Your Warehouse on the Internet appeared first on John Nessime.