A Salesforce admin renames a custom field on Tuesday afternoon. Your connector keeps running, your Glue job keeps succeeding, the crawler happily adds a new column and quietly stops populating the old one. Three weeks later someone asks why the pipeline report has a cliff in it, and you spend an afternoon reconstructing a change nobody in engineering made and nobody logged.

This post is about using AWS Glue Data Quality to catch that class of breakage. It is aimed at people pulling from SaaS sources into S3 or a lakehouse: Salesforce, HubSpot, Zendesk, Stripe, Shopify, Zoho, an internal partner API. I will cover where to run the checks, the flattening constraint that will bite you before anything else does, which DQDL rule actually catches which SaaS failure, when dynamic rules and anomaly detection quietly do nothing, and what should happen when a rule fails.

Why SaaS data breaks in ways your own database doesn’t

When you own the source database, the schema is under change control: someone writes a migration, it gets reviewed, and if it breaks a consumer there is a commit to point at. A SaaS source has none of that. Three consequences:

• The schema is edited by people who are not engineers. A sales ops admin adding a picklist value is a production schema change with no review, no notice and no version number.
• The payload is nested. SaaS APIs return structs and arrays: line items, related objects, custom field bags. That shape matters more than you would expect, for reasons in the next section but one.
• Absence is ambiguous. A field missing from a payload might mean “not set,” “not permitted for this user,” or “we removed it.” The connector usually cannot tell you which, and neither can your table.

Glue Data Quality is a rules engine built on Deequ that evaluates declarative checks written in DQDL, the Data Quality Definition Language, against a dataset. It is well suited to the first and third problems. The second one needs work before it is useful at all.

Decide where the checks run before you write a rule

There are two places to evaluate a ruleset, and they answer different questions.

Inside the Glue ETL job

The EvaluateDataQuality transform runs mid-pipeline against a DynamicFrame. It answers “is this batch safe to load?” and it can act on the answer: stop the job, or split failing rows into a quarantine prefix. This is also the only placement where row-level results exist, which matters because “which 400 rows are bad” is a far more useful answer than “the rule failed.”

Against a Data Catalog table

You can attach a ruleset to a catalog table and run it on a schedule, independent of any job. This answers “is the table currently trustworthy?” and surfaces a quality score analysts can see before they query. The trade-off is that it is after the fact: by the time a catalog evaluation fails, the bad batch is already queryable. Analyzers are also not supported for catalog evaluations, so some dynamic-rule tooling behaves differently there.

In practice I run both. The in-job ruleset is small and strict, and exists to stop garbage landing. The catalog ruleset is broader and looser, and exists so a table’s state is visible to people who will never open Glue Studio.

The flattening problem that comes first

Here is the constraint that catches most people on SaaS data, stated plainly in the AWS documentation and easy to skim past: DQDL cannot evaluate nested or list-type columns. No structs, no arrays.

Almost every interesting SaaS payload is nested. A Salesforce opportunity carries a related account struct, a Stripe invoice carries an array of line items, a Zendesk ticket carries a custom fields array of key-value pairs. Point a ruleset at the raw landed data and the fields you most want to check are the ones the engine cannot see.

So you flatten first, with Glue’s relationalize or an explicit projection, then evaluate. That works, but it changes what your rules mean:

• Your rules now describe the projection, not the payload. If your flattening step drops a field, no rule will notice, because from DQDL’s point of view that field was never there.
• Exploding an array changes the grain. One invoice with five line items becomes five rows. Every RowCount rule you write against that table is now counting line items, and a customer who starts itemising differently will move your row count without any data being wrong.
• Custom-field bags need pivoting, not flattening. A key-value array flattened naively gives you rows, not columns, and you cannot write a ColumnValues rule against a value whose meaning depends on a sibling key column. Pivot the ones you care about into real columns and check those.

The practical rule: put your quality checks immediately after the flattening step, and treat the flattening step itself as code that needs its own test. Glue Data Quality will not cover it for you.

Matching SaaS breakage to the rule that catches it

DQDL has a long rule list and most guides just enumerate it. More useful is the other direction: what actually goes wrong with SaaS sources, and what catches each one.

A field is renamed, removed, or added

This is the Tuesday-afternoon scenario. Two rules, doing two different jobs:

Rules = [
    ColumnExists "annual_revenue__c",
    ColumnCount = 47
]

ColumnExists protects the specific fields your downstream models depend on. ColumnCount protects against everything else: a field silently added, or one you forgot you needed. The first is precise and the second is a tripwire, and you want both because a rename shows up as one column gone and one column arrived, which leaves the count unchanged.

SchemaMatch compares against a reference dataset’s schema, which is stricter and useful if you maintain a golden reference table. Bear in mind a hard column count on a SaaS object fails every time an admin adds a field, so decide whether that blocks or just warns.

A field still exists but stopped being populated

Permissions change, a workflow gets disabled, an integration user loses field-level access. The column is still in the schema and is now all nulls.

Rules = [
    IsComplete "opportunity_id",
    Completeness "lead_source" > 0.85,
    Completeness "email" > 0.60 where "record_type = 'Contact'"
]

IsComplete is the absolute version: no nulls at all, right for keys. Completeness takes a threshold, which is what you want for optional business fields where a hard rule would fail constantly. The where clause is the one people miss: SaaS objects are usually polymorphic, and a completeness threshold that makes sense for contacts is nonsense across the whole table. Scope the rule to the record type.

The sync ran short

A paginated extract stops early, a rate limit gets absorbed by retries, an incremental cursor drifts. The job succeeds and delivers less data than it should have. A static RowCount > 1000 stops catching this the moment you cross a thousand rows. Dynamic rules compare against your own history instead:

Rules = [
    RowCount > min(last(3)),
    Sum "amount" > avg(last(5)) * 0.7,
    DetectAnomalies "RowCount"
]

Analyzers = [
    RowCount,
    DistinctValuesCount "owner_id"
]

last(k) pulls the last k recorded values of that metric for the same dataset; with k greater than one you need an aggregation like avg, min or max to reduce them to one number. DetectAnomalies goes further, using forecasting over stored history to flag deviations with no threshold from you at all. That is the right tool for anything seasonal: a B2B pipeline that dips every weekend trips a naive trailing-average rule every Monday, while anomaly detection learns the shape.

Analyzers deserve their own mention: they gather a metric without asserting anything, which builds the history dynamic rules and anomaly detection need. DistinctValuesCount on an owner or account column is a strong early signal for a partial sync, because row count can look fine while the number of distinct accounts represented quietly halves.

Records arrive twice, or the key stops being a key

Replays, overlapping incremental windows and connector retries all produce duplicates, and SaaS “IDs” are less unique than vendor docs imply once you have merged sandboxes or multiple accounts.

Rules = [
    IsPrimaryKey "id",
    Uniqueness "external_reference" > 0.99
]

IsPrimaryKey asserts unique and non-null together. Uniqueness with a threshold is the softer version for fields that are mostly-but-not-perfectly unique, which describes a lot of SaaS reference fields honestly.

Objects stop agreeing with each other

Multi-object extracts rot quietly. Opportunities reference accounts that were never synced; invoice totals stop matching line item sums because one object’s extract failed and the other’s didn’t.

ReferentialIntegrity checks that values in one dataset’s column exist in another’s. RowCountMatch and AggregateMatch compare counts and summary metrics across two datasets. These catch a broken join before an analyst does, and they need the multi-frame form of the transform with the second dataset supplied as an additional source.

The sync stopped and nobody noticed

The quietest failure of all: nothing is wrong with the data, there just isn’t any new data. DataFreshness checks a timestamp column against a window. It is deliberately simple and its comparison granularity is limited, so for anything more nuanced the custom SQL rule is the escape hatch, running a query against the dataset and evaluating the result against an expression. DQDL is case sensitive and that rule type’s exact capitalisation has appeared more than one way across the docs and the visual builder, so copy it from the current DQDL reference rather than from memory.

Static, dynamic and ML: when each one lies to you

All three check styles have a mode where they pass while the data is wrong. Knowing which is which is most of the skill.

• Static rules lie by going stale. A threshold that was correct when you wrote it stops being correct as the business grows, and a rule that can no longer fail is worse than no rule because it looks like coverage.
• Dynamic rules lie when they have no history. This one is genuinely nasty. last(k) reads previously recorded values for that metric, and if there aren’t any, it falls back to a default of zero. Add RowCount > avg(last(3)) to an existing job today and it will pass trivially for the first few runs, because almost any row count beats zero. It looks like a working check from the moment you deploy it. It isn’t one until it has accumulated runs.
• Anomaly detection lies while it is warming up, and after you retrain it badly. It needs a run of history before its forecasts mean anything, and if you leave a period of genuinely bad data in the training window, it learns that the bad shape is normal. You can exclude specific statistics from the calculation in the Data Quality tab, and you should, after any incident.

So when you add a dynamic or ML-based rule, write down the date and treat it as unarmed for the first couple of weeks. Keep a crude static rule alongside it in the meantime.

What should happen when a rule fails

The transform returns a collection. The two members you care about are the row-level outcomes, which give you the original data with an evaluation result column appended per row, and the rule outcomes, which give you one row per rule with its verdict.

from awsgluedq.transforms import EvaluateDataQuality

dq_results = EvaluateDataQuality().process_rows(
    frame=flattened_opportunities,
    ruleset=ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "salesforce_opportunity",
        "enableDataQualityCloudWatchMetrics": "true",
        "enableDataQualityResultsPublishing": "true",
    },
)

row_level = SelectFromCollection.apply(
    dfc=dq_results, key="rowLevelOutcomes"
)

dataQualityEvaluationContext is the namespace your CloudWatch metrics land under. Name it after the source object rather than the job, because you will eventually want to alert per object.

From there, three response patterns:

1. Fail fast. Stop the job on failure so nothing lands. Right for structural rules: if the primary key isn’t unique, downstream is going to be wrong no matter what you do next.
2. Quarantine. Filter the row-level outcomes on the evaluation result column, write passing rows to the target and failing rows to a separate prefix. Right for value-level rules where 2% bad records shouldn’t block 98% good ones. The important discipline is that somebody has to actually look at the quarantine prefix; an unmonitored quarantine is a delete with extra steps.
3. Warn and continue. Let everything land, publish the metrics, alert on the score. Right for new rules you don’t trust yet, and for the catalog-side ruleset.

Whichever you pick, get the failure out of Glue and into wherever your team actually looks. Results and anomaly predictions can be written back to catalog tables so they are queryable with plain SQL, and CloudWatch metrics plus EventBridge events give you the hook for paging or pausing a downstream pipeline. A Grafana or Datadog dashboard fed from those metrics beats a score in a console tab nobody opens.

Troubleshooting AWS Glue Data Quality on SaaS pipelines

“Input data does not include column X”

A rule references a column the engine cannot find. On SaaS data that means one of three things: the field disappeared from the source, your flattening step renamed it, or the Data Catalog is out of sync with the underlying table. Check the catalog schema against the landed files before touching the ruleset; rerunning the crawler resolves a surprising share of these.

A rule type isn’t recognised

DQDL is case sensitive and the rule set has grown over releases, so a rule copied from an older blog post can parse fine in your head and not at all in the engine. Check the rule name against the current DQDL rule type reference, and check that the feature is available on your Glue version. Anomaly detection in particular has a version floor.

A dynamic rule never fails

Almost always the empty-history problem described above. Look at the statistics for that metric in the Data Quality tab. If there are fewer recorded points than your last(k) asks for, the rule is not doing what you think. It will start working on its own; the mistake is trusting it in the meantime.

Evaluation is slow or the job runs out of memory

Each rule triggers analyzer computation over the dataset, and a large ruleset on a wide flattened SaaS table adds up fast. The transform exposes caching and observations-scope options to tune this. Before reaching for a bigger worker type, cut the ruleset down: most contain a dozen recommended rules nobody reads the results of.

A “column reference is ambiguous” error

Classic flattening artefact. Two nested paths collapsed to the same leaf name, so datechecked now matches two things. Fix it in the projection with explicit aliases rather than trying to disambiguate inside DQDL.

Common mistakes

• Accepting the recommended ruleset wholesale. Recommendations are generated from a sample of current data, so they encode today’s accidents as tomorrow’s requirements.
• Writing rules against nested data and assuming they cover the whole payload.
• Deploying a dynamic rule and treating it as armed on day one.
• Applying whole-table completeness thresholds to a polymorphic object instead of scoping with a where clause.
• Quarantining failed rows to a prefix nobody monitors.
• Writing RowCount rules against an exploded array without noticing the grain changed.
• Failing the job on every rule, including cosmetic ones, until the team starts rerunning with checks disabled.
• Keeping the ruleset in the console instead of in version control next to the job.

Best practices

• Split rules into blocking and reporting tiers, and be honest about which is which. A rule that always fails and never stops anything trains people to ignore the tab.
• Pair ColumnExists with ColumnCount. One protects the fields you named, the other catches the drift you did not anticipate.
• Add analyzers early, even for metrics you have no rule for yet. History has to accumulate before dynamic rules and anomaly detection are worth anything, and you cannot backfill it.
• Keep DQDL in Git alongside the job definition and deploy it with Terraform or CloudFormation. Rulesets are code and should move through the same review as the pipeline they guard.
• Namespace the evaluation context per source object so CloudWatch metrics and alerts can be routed per vendor.
• Exclude incident periods from the anomaly detection training window once the incident is resolved.
• Surface results where stakeholders already look. Writing outcomes back to catalog tables makes them queryable from Athena, which beats a score in a console tab.

Frequently asked questions

How much does AWS Glue Data Quality cost?

It bills on Glue DPU-hours like any other Glue workload, so cost is a function of evaluation time and DPUs used. The dimension that surprises people is anomaly detection, which consumes additional capacity per statistic analysed, so enabling it broadly across many columns and tables is where the bill moves. Model it against the current Glue pricing page; rates change and vary by region.

Can it check data before it lands in S3?

Yes, if the check runs inside the Glue job. The transform evaluates a DynamicFrame in memory, so you can validate after extraction and before the write. That is the placement that lets you stop a bad load rather than report on one.

Does it work on nested JSON from a SaaS API?

Not directly. DQDL does not evaluate nested or list-type columns, so you have to flatten the structure first and run the ruleset against the flattened frame. Plan the flattening deliberately, because your rules only ever see what the projection exposes.

How does it compare to Great Expectations, Soda or Monte Carlo?

Glue Data Quality’s real advantage is that it is already inside the runtime and the catalog: no extra infrastructure, no separate agent, one IAM story. The disadvantages are real too. DQDL is less expressive than Python-based assertions, the nested-data limitation is a hard wall, and it only covers what Glue can read. Great Expectations gives you more expressive checks at the cost of running it yourself; Monte Carlo and similar platforms add lineage and cross-warehouse coverage Glue does not attempt. Glue-centric stack, start here. Several engines, a standalone tool fits better.

Should I use the recommended rules?

As a starting draft, yes. As a ruleset, no. Recommendations are inferred from a sample of the data as it is today, which means any existing quality problem gets encoded as the expected baseline. Read every generated rule and delete the ones you would not have written yourself.

Can I reconcile row counts against the SaaS source itself?

Not directly; it only sees datasets it can read. The usual pattern is to have the extraction step record the count the API reported, land that alongside the data as a control table, then use RowCountMatch or AggregateMatch to compare the two. That turns “did we get everything” into a rule the engine can evaluate.

The one thing worth remembering

AWS Glue Data Quality is not really a data quality product. It is a way of writing down what you believe about a dataset so a machine can check it every run, and its value is entirely determined by whether those beliefs are written precisely enough to be wrong.

For SaaS sources that means two habits. Flatten deliberately and check immediately after, because your rules only see the projection. And treat every new dynamic or ML-based rule as unarmed until it has history, because a rule that cannot fail yet looks exactly like a rule that is passing.

Need help with data quality on your Glue pipelines?

I work with teams whose SaaS pipelines run green while the numbers drift. Things I can help with:

• Writing a DQDL ruleset for a specific SaaS object that catches schema drift instead of just restating the current schema.
• Designing the flattening and pivoting step for nested payloads and custom-field bags so the checks actually cover what matters.
• Setting up quarantine and fail-fast paths in Glue ETL, with the routing and alerting that makes a quarantine prefix something people act on.
• Cross-object reconciliation between SaaS extracts and control counts recorded at extraction time.
• Tuning anomaly detection and dynamic rules, including cleaning incident periods out of the training window.
• Getting data quality results out of the console and into Grafana, CloudWatch alarms or an Athena-queryable table.

If you have a ruleset that keeps passing while the data drifts, send me the DQDL and a schema dump. That is usually enough to see where the gap is.

The post AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed appeared first on John Nessime.