<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Anomaly Detection | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/anomaly-detection/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/anomaly-detection/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 10:29:42 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Anomaly Detection | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/anomaly-detection/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>CloudWatch Alarms That Are Actually Useful (and the Ones to Delete)</title>
		<link>https://john-nessime.com/blog/devops/useful-cloudwatch-alarms/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 26 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alarm Fatigue]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Anomaly Detection]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Composite Alarms]]></category>
		<category><![CDATA[Incident Response]]></category>
		<category><![CDATA[Metric Math]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[On-Call]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Terraform]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=266</guid>

					<description><![CDATA[<p>Most AWS accounts have dozens of CloudWatch alarms and about three anyone trusts. A practical guide to missing-data settings, M-out-of-N evaluation, metric math, composite alarms and mute rules, and how to test the notification path before you need it.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/useful-cloudwatch-alarms/">CloudWatch Alarms That Are Actually Useful (and the Ones to Delete)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The CPU alarm has been green for six weeks. That is not because the instance is healthy. It is because the agent stopped publishing the metric, the alarm quietly slid into INSUFFICIENT_DATA, and nobody ever wired that state to anything. Green on the dashboard, deaf in production.</p>



<p class="wp-block-paragraph">That is the failure mode nobody catches during a review, and it is why most AWS accounts have dozens of alarms and about three anyone trusts. The rest either fire constantly and get muted, or never fire at all and get mistaken for coverage.</p>



<p class="wp-block-paragraph">This post is about building useful CloudWatch alarms: which signals deserve a page, which settings decide whether an alarm fires correctly, how to collapse alarm storms without going blind, and how to prove the notification path works before you need it.</p>



<h2 class="wp-block-heading">The test an alarm has to pass before it exists</h2>



<p class="wp-block-paragraph">Before the threshold, before the metric, one question: if this fires at 3am, what does the person on call do?</p>



<p class="wp-block-paragraph">If the honest answer is &#8220;look at it and go back to bed&#8221;, it is not an alarm. It is a dashboard panel. That single filter removes most of the CPU, memory and disk alarms people inherit. High CPU is not an incident. High CPU while latency climbs and the queue backs up is an incident. Every alarm that fires without requiring action trains the on-call to ignore the channel, and that training is permanent.</p>



<ul class="wp-block-list">
<li><strong>Page</strong> on symptoms a customer can feel: error rate, latency at a high percentile, queue age, failed jobs, expiring credentials.</li>

<li><strong>Ticket</strong> on causes that bite next week: disk trending toward full, approaching a service quota, a renewal that is not happening.</li>

<li><strong>Graph only</strong> on everything else. Resource utilisation is context during an incident, not the incident.</li>
</ul>



<p class="wp-block-paragraph">Encode that split in the alarm&#8217;s action, not in your head. A page goes to an SNS topic wired to PagerDuty or Opsgenie. A ticket goes to a different topic that opens a Jira issue or a Systems Manager OpsItem. Same engine, different consequences.</p>



<h2 class="wp-block-heading">The alarm that goes quiet: missing data is the invisible failure</h2>



<p class="wp-block-paragraph">Every metric alarm has a <code>TreatMissingData</code> setting, and if you never touched it, it is <code>missing</code>. That default means the alarm ignores gaps entirely when deciding state. It keeps whatever state it had, or sits in INSUFFICIENT_DATA if there is nothing at all. For a metric that always reports, that is a silent hole: the instance dies, the metric stops, and the alarm does not go red. It just stops having an opinion.</p>



<ul class="wp-block-list">
<li><code>breaching</code> for metrics that report continuously. If the data stops, something is wrong. Heartbeats, health checks, CPU on a box that should always be up.</li>

<li><code>notBreaching</code> for metrics that only exist when something bad happens. Error counts, throttled requests, dead letter queue depth. No data means no errors.</li>

<li><code>ignore</code> keeps the current state through gaps, so an alarm stays red until genuinely resolved rather than flapping to OK on a hole in the data.</li>

<li><code>missing</code> is the default, and defensible only when you decided it was right.</li>
</ul>



<p class="wp-block-paragraph">There is a second-order effect that surprises people. When an alarm evaluates, CloudWatch pulls a wider range of data points than the evaluation periods you configured. AWS calls that the evaluation range. If some points in that range are missing but enough real ones exist, CloudWatch evaluates against the real points and your missing-data setting never applies to the gaps.</p>



<p class="wp-block-paragraph">The consequence: an alarm set to eight breaching points out of ten can fire on one breaching point followed by nine gaps, because the older breach is still inside the evaluation range and everything newer is absent. Teams hit this with load balancer 5XX alarms constantly, and it looks exactly like an unexplained false positive.</p>



<p class="wp-block-paragraph">Setting <code>notBreaching</code> on sparse error metrics is usually the fix. Filling gaps with metric math is the other, and it is more explicit:</p>



<pre class="wp-block-code"><code># Treat absent error data points as zero rather than as gaps.
# m1 is the raw error metric; e1 is what the alarm evaluates.
e1 = FILL(m1, 0)</code></pre>



<p class="wp-block-paragraph">One exception: alarms on metrics in the <code>AWS/DynamoDB</code> namespace always ignore missing data regardless of what you set. If you need a DynamoDB alarm that reacts to silence, build the fill into metric math.</p>



<p class="wp-block-paragraph">Then find the ones already stuck. This is the fastest audit you can run on an inherited account:</p>



<pre class="wp-block-code"><code>aws cloudwatch describe-alarms 
  --state-value INSUFFICIENT_DATA 
  --query 'MetricAlarms[].[AlarmName,MetricName,StateReason]' 
  --output table</code></pre>



<p class="wp-block-paragraph">Anything on that list older than a deploy cycle is either watching something that no longer exists, or watching something that stopped reporting. Both deserve thirty seconds.</p>



<h2 class="wp-block-heading">The alarm that cries wolf: periods, data points and windows</h2>



<p class="wp-block-paragraph">Three settings decide whether an alarm is jumpy: <code>Period</code> (how long each data point covers), <code>EvaluationPeriods</code> (how many recent points to look at), and <code>DatapointsToAlarm</code> (how many of those must breach).</p>



<h3 class="wp-block-heading">M out of N beats consecutive breaches</h3>



<p class="wp-block-paragraph">Most people set evaluation periods to 3 and leave data points to alarm unset, which makes it 3 as well. That demands three consecutive breaching points. Real degradation is rarely that tidy; it flickers, one clean point resets your patience, and the alarm never fires.</p>



<p class="wp-block-paragraph">Set them separately. Three out of five catches intermittent problems while still filtering single-point spikes. The breaching points do not need to be consecutive, only inside the last N.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name "api-5xx-elevated" 
  --namespace "AWS/ApplicationELB" 
  --metric-name HTTPCode_ELB_5XX_Count 
  --dimensions Name=LoadBalancer,Value=app/my-alb/0123456789abcdef 
  --statistic Sum 
  --period 60 
  --evaluation-periods 5 
  --datapoints-to-alarm 3 
  --threshold 25 
  --comparison-operator GreaterThanThreshold 
  --treat-missing-data notBreaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page</code></pre>



<p class="wp-block-paragraph">Read that as: over the last five minutes, if three or more individual minutes each saw more than 25 backend 5XX responses, page. Minutes with no data count as fine, because a load balancer with no errors publishes nothing.</p>



<h3 class="wp-block-heading">Percentiles need a sample-count decision</h3>



<p class="wp-block-paragraph">Average latency hides everything interesting, so alarm on <code>p95</code> or <code>p99</code> using the extended statistic rather than <code>--statistic Average</code>.</p>



<p class="wp-block-paragraph">The catch is low traffic. A p99 computed from four requests is noise, and by default CloudWatch still evaluates it. The <code>EvaluateLowSampleCountPercentile</code> setting controls that: set it to <code>ignore</code> and the alarm holds its current state when the sample count is too small to mean anything. On an overnight window or a quiet endpoint, that one setting removes a whole class of 4am pages.</p>



<h3 class="wp-block-heading">Sliding versus wall clock windows</h3>



<p class="wp-block-paragraph">By default the evaluation window slides: it advances a minute at a time and is not aligned to the clock. That is what you want for latency and errors, because you want to know now.</p>



<p class="wp-block-paragraph">CloudWatch also supports a wall clock window aligned to real boundaries in a time zone you pick: top of the hour, midnight, start of the calendar week. That is the right choice for anything measured per calendar period, like a daily batch job or a monthly threshold, because the alarm considers a completed period rather than a rolling one. Note that the wider evaluation range described earlier applies only to sliding windows, so wall clock alarms are easier to reason about but have no safety net for late-arriving data.</p>



<h2 class="wp-block-heading">Alarming on the wrong number</h2>



<p class="wp-block-paragraph">A large share of useless alarms are correctly configured alarms watching a meaningless quantity.</p>



<h3 class="wp-block-heading">Rates, not counts</h3>



<p class="wp-block-paragraph">An alarm on &#8220;more than 50 errors&#8221; is wrong twice. During a traffic peak, 50 errors out of half a million requests is nothing. At 3am, 50 errors out of 60 requests is a total outage that never crosses the threshold.</p>



<pre class="wp-block-code"><code># m1 = Errors (Sum), m2 = Invocations (Sum)
# Guard the zero case so the alarm gets a data point at idle
# instead of a gap it has to interpret.
e1 = IF(m2 &gt; 0, m1 / m2 * 100, 0)</code></pre>



<p class="wp-block-paragraph">Only the final expression should return data to the alarm; the underlying metrics are inputs. A rate alarm alone is still incomplete, though, because at very low volume one error is 100 percent. Pair it with a minimum-volume condition in a composite alarm, or fold the request count into the expression so tiny denominators cannot trip it.</p>



<h3 class="wp-block-heading">Quotas and saturation, before they bite</h3>



<p class="wp-block-paragraph">Service quotas are the outage nobody sees coming, because everything is healthy right up until it is not. For services that publish usage metrics, the <code>SERVICE_QUOTA</code> metric math function pulls your current limit from Service Quotas, so you can alarm on percentage consumed instead of a number you hardcoded and forgot:</p>



<pre class="wp-block-code"><code># m1 is a usage metric under the Usage namespace.
# Alarm on consumption as a share of the live quota.
e1 = m1 / SERVICE_QUOTA(m1) * 100</code></pre>



<p class="wp-block-paragraph">This is the definitive ticket-not-page alarm. Nobody needs waking for concurrency at 80 percent, but somebody needs to file a limit increase before the deploy that takes it to 100. Same category: queue age rather than queue depth. A deep queue draining fast is fine; a shallow queue whose oldest message is twenty minutes old means your consumers are stuck.</p>



<h2 class="wp-block-heading">Cutting the storm: composite alarms, suppression and mute rules</h2>



<p class="wp-block-paragraph">One bad deploy in an Auto Scaling group with per-instance alarms produces forty notifications describing one event. A composite alarm evaluates a boolean rule over other alarms. The metric alarms stay exactly as they are and keep their own state, but stop being wired to your paging topic. The composite becomes the only thing that pages.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-composite-alarm 
  --alarm-name "checkout-degraded" 
  --alarm-rule 'ALARM("checkout-latency-p99") AND ALARM("checkout-error-rate")' 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page</code></pre>



<p class="wp-block-paragraph">Latency alone might be a slow dependency. Errors alone might be a bad client. Both together is a real problem, and that is what earns a page.</p>



<p class="wp-block-paragraph">The second half is suppression. A composite alarm accepts a suppressor alarm: while that alarm is in ALARM, the composite still evaluates but takes no actions. Point it at a deployment-in-progress alarm and you stop paging for known-bad windows.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-composite-alarm 
  --alarm-name "checkout-degraded" 
  --alarm-rule 'ALARM("checkout-latency-p99") AND ALARM("checkout-error-rate")' 
  --actions-suppressor "deployment-in-progress" 
  --actions-suppressor-wait-period 60 
  --actions-suppressor-extension-period 60 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page</code></pre>



<p class="wp-block-paragraph">Those two periods are the part people skip and then get burned by. The wait period gives the suppressor time to reach ALARM before the composite decides to fire; the extension period gives the composite time to settle back to OK after the suppressor clears. AWS suggests 60 seconds for each, on the reasoning that metric alarms evaluate once a minute. Leave them at zero and you get paged in the gap between the deploy starting and the deploy alarm noticing.</p>



<p class="wp-block-paragraph">CloudWatch has since added alarm mute rules, which handle the scheduled case directly. A mute rule targets specific alarms by name and mutes their actions during a one-time or recurring window. The alarms keep evaluating and keep changing state; only the actions are muted. That removes the EventBridge and Lambda contraption teams used to build for maintenance windows. Two things to keep straight: mute rules take precedence over composite action suppression when both apply, and a mute rule targets alarms by name, so a rename in your Terraform or OpenTofu module silently orphans the rule.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Suppression is for known-bad windows you chose. If you are muting an alarm because it is annoying rather than because you scheduled the disruption, the alarm is wrong and muting it just hides that.</p>
</blockquote>



<h2 class="wp-block-heading">Test the path, not the threshold</h2>



<p class="wp-block-paragraph">An alarm you have never seen fire is a hypothesis. The threshold is the easy part. The fragile part is everything downstream: SNS subscriptions that were never confirmed, a Lambda whose role lost a permission, a chat webhook rotated three months ago, a topic policy that quietly rejects CloudWatch.</p>



<p class="wp-block-paragraph">You can force a state transition without touching the metric:</p>



<pre class="wp-block-code"><code>aws cloudwatch set-alarm-state 
  --alarm-name "api-5xx-elevated" 
  --state-value ALARM 
  --state-reason "Routing test, ticket OPS-1234"</code></pre>



<p class="wp-block-paragraph">This drives the real actions, so the page really goes out. Tell the on-call first. The forced state is temporary; the next real evaluation overwrites it. Do this for every new alarm before you call it done, and again after any change to the notification stack. On a mixed estate where some workloads sit on a VPS at a provider like Contabo or InterServer alongside your AWS footprint, test each path separately, because a shared SNS topic creates the illusion that one confirmed subscription covers everything.</p>



<p class="wp-block-paragraph">Then confirm the transition and the action both happened:</p>



<pre class="wp-block-code"><code>aws cloudwatch describe-alarm-history 
  --alarm-name "api-5xx-elevated" 
  --history-item-type Action 
  --max-records 10</code></pre>



<h2 class="wp-block-heading">What useful CloudWatch alarms cost you</h2>



<p class="wp-block-paragraph">Alarms bill per alarm metric per month, so the mental model that matters is this: a standard metric alarm counts as one, and a metric math alarm counts as one per metric referenced in the expression. An error-rate alarm built from errors and invocations bills as two, not one. Anomaly detection alarms carry their own multiplier because the band is evaluated alongside the metric. High-resolution alarms, with periods of 10, 20 or 30 seconds, are charged at a higher rate than standard ones, which makes them worth it for a handful of latency-critical paths and wasteful everywhere else.</p>



<p class="wp-block-paragraph">Check current rates on the CloudWatch pricing page rather than trusting a number in a blog post, including this one. And note the bigger cost is usually not the alarm at all. It is the custom metrics feeding it, especially high-cardinality ones published per request or per container.</p>



<h2 class="wp-block-heading">Troubleshooting an alarm that misbehaves</h2>



<ol class="wp-block-list">
<li><strong>Stuck in INSUFFICIENT_DATA.</strong> Either the metric is not being published, or the alarm&#8217;s dimensions do not match the metric&#8217;s dimensions exactly. Dimension mismatch is the common one and it is silent: an alarm on a dimension set that never existed looks identical to an alarm on a dead resource.</li>

<li><strong>Fired but nobody heard.</strong> Check <code>describe-alarm-history</code> filtered to Action items. Transition present but action missing points at the SNS topic policy or the subscription. Both present points downstream of SNS.</li>

<li><strong>Fires with no visible breach on the graph.</strong> Almost always missing data plus the wider evaluation range. The state reason names how many points were breaching and how many were unknown.</li>

<li><strong>Flapping between OK and ALARM.</strong> The period is too short for the metric&#8217;s natural variance, or data points to alarm equals evaluation periods. Widen the window before raising the threshold; raising the threshold loses sensitivity you may want later.</li>

<li><strong>Composite alarm never fires.</strong> Confirm every child alarm named in the rule exists and is spelled correctly, then confirm no suppressor or mute rule is active. The console shows a mute indicator on affected alarms.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Leaving <code>TreatMissingData</code> at the default because you did not know it existed.</li>

<li>Picking 80 percent CPU because it sounds like a threshold, on a service that normally runs at 75.</li>

<li>Alarming per instance in an Auto Scaling group instead of on the aggregate a customer experiences.</li>

<li>Wiring OK actions to the same paging channel, so every recovery is also an interruption.</li>

<li>Alarm descriptions that restate the alarm name. The description is the only context the responder gets at 3am, so put the runbook link there.</li>

<li>Anomaly detection on a metric with no stable pattern. The band is only as good as the history it learned from.</li>
</ul>



<h2 class="wp-block-heading">Best practices for useful CloudWatch alarms</h2>



<ul class="wp-block-list">
<li>Set <code>TreatMissingData</code> explicitly on every alarm, and write down why in the description.</li>

<li>Separate <code>DatapointsToAlarm</code> from <code>EvaluationPeriods</code>. M out of N is almost always better than N consecutive.</li>

<li>Observe a metric across a full weekly cycle before fixing its threshold. Weekend traffic is a different distribution.</li>

<li>Define alarms in code, in whatever you already use: Terraform, OpenTofu, CDK, CloudFormation. Console-created alarms drift and disappear.</li>

<li>Use anomaly detection for the signal and composite alarms for the logic. An anomalous shape plus real user impact is a page; an anomalous shape alone is a graph.</li>

<li>Review what fired last quarter. Any alarm that fired and was closed without action is a candidate for deletion. Send it to a Grafana panel instead.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How many CloudWatch alarms should a service have?</h3>



<p class="wp-block-paragraph">Fewer than you think. A handful of symptom alarms that page, plus a set of cause alarms that open tickets, covers most services. The number that matters is not how many exist but how many fired last quarter and led to somebody doing something.</p>



<h3 class="wp-block-heading">Should I use anomaly detection or static thresholds?</h3>



<p class="wp-block-paragraph">Static thresholds where a real limit exists: a disk that must not fill, a latency budget you promised. Anomaly detection where the metric has a genuine daily or weekly shape and no fixed correct value, like request volume. Anomaly detection needs history before it is useful and struggles when a deploy changes the pattern, so it is a poor fit for anything new.</p>



<h3 class="wp-block-heading">Why did my alarm fire when the graph looks fine?</h3>



<p class="wp-block-paragraph">Usually missing data. CloudWatch pulls a wider evaluation range than the periods you configured, so an older breaching point can still count while newer points are simply absent. Read the state reason, which spells out how many points were breaching and how many were unknown, then reconsider the missing-data setting.</p>



<h3 class="wp-block-heading">Can an alarm watch a metric in another AWS account?</h3>



<p class="wp-block-paragraph">Cross-account observability lets a monitoring account view and alarm on metrics from linked source accounts, which is the usual pattern for an AWS Organizations estate. The alternative is running alarms in each account and aggregating at the SNS or paging layer. That is more alarms to manage but keeps the blast radius per account smaller.</p>



<h3 class="wp-block-heading">How do I stop alarms firing during a deployment?</h3>



<p class="wp-block-paragraph">Two options. An alarm mute rule on a schedule, if your deploys are predictable. Or a composite alarm with a deployment-in-progress alarm as its actions suppressor, if they are not. The suppressor approach suits continuous delivery better because it reacts to the actual deploy rather than a calendar guess.</p>



<h3 class="wp-block-heading">Do CloudWatch alarms work on logs?</h3>



<p class="wp-block-paragraph">Indirectly. A metric filter on a log group turns matching log lines into a numeric metric, and you alarm on that metric like any other. Metric filters only emit data points when they match, so <code>notBreaching</code> is almost always the right missing-data setting for them.</p>



<h3 class="wp-block-heading">Is CloudWatch enough, or do I need Datadog or Grafana?</h3>



<p class="wp-block-paragraph">For a single-account AWS workload, CloudWatch alarms plus SNS handle alerting fine, and staying inside AWS avoids another integration to maintain. Third-party platforms earn their keep when you need correlation across non-AWS systems, richer on-call routing, or long metric retention with fast queries. The honest test is whether you are already spending engineering time rebuilding those features on top of CloudWatch.</p>



<h2 class="wp-block-heading">The one thing to keep</h2>



<p class="wp-block-paragraph">Useful CloudWatch alarms are not the ones with the cleverest thresholds. They are the ones where somebody decided, on purpose, what silence means.</p>



<p class="wp-block-paragraph">Every alarm in your account is making a claim about the absence of data, whether you configured that claim or inherited it. Go through them, set <code>TreatMissingData</code> deliberately, split data points from evaluation periods, delete the ones nobody acts on, and force a state transition on the ones you keep so you know the page lands. Six alarms that fire correctly beat sixty that nobody reads.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your alerting?</h2>



<p class="wp-block-paragraph">I work with teams on AWS monitoring that people actually trust. Typically that looks like:</p>



<ul class="wp-block-list">
<li>Auditing existing CloudWatch alarms and separating the ones that page, the ones that should open tickets, and the ones to delete.</li>

<li>Fixing missing-data and evaluation settings so alarms stop firing on gaps and stop going silent on dead metrics.</li>

<li>Building composite alarms, suppression and mute rules so one incident produces one page instead of forty.</li>

<li>Writing metric math for error rates, queue age and quota headroom instead of raw counts.</li>

<li>Codifying the whole alarm set in Terraform or OpenTofu so it survives the next person.</li>

<li>Wiring CloudWatch into Grafana dashboards and testing every notification path end to end.</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me the output of <code>describe-alarms</code> for one account and I will tell you which alarms are load-bearing and which are decoration.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/useful-cloudwatch-alarms/">CloudWatch Alarms That Are Actually Useful (and the Ones to Delete)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</title>
		<link>https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 15 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Anomaly Detection]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=225</guid>

					<description><![CDATA[<p>Your SaaS pipeline will fail far more often by succeeding at nothing than by throwing an exception, and every CloudWatch default treats an absent metric as a non-event. Here are the four signals worth alarming on: liveness, volume, freshness and shape, plus the missing-data traps that leave alarms permanently green.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/">CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said the Salesforce numbers looked wrong on the executive dashboard. I opened the pipeline. Every run for the past several days was green: the state machine succeeded, no failed states, no Lambda errors, no alarms in the account. The pipeline had been running flawlessly and writing an empty file every single time.</p>



<p class="wp-block-paragraph">That gap is the whole problem with CloudWatch data pipeline monitoring, and it is why most teams find out about broken SaaS integrations from a business user rather than from a page. A web server that breaks returns 500s, and your error-rate alarm fires. A pipeline that breaks frequently returns exit code 0, writes a zero-byte object, and reports success. The connected app got revoked, a scope was dropped during an admin cleanup, or a filter clause quietly started matching nothing. None of the AWS-native metrics have an opinion about any of that.</p>



<p class="wp-block-paragraph">This post is about what to measure instead: the four signals worth alarming on, the CloudWatch defaults that leave those alarms silently in the wrong state, how to emit custom metrics without turning your bill into a line item someone asks about, and what to check when a metric exists but the alarm still refuses to fire.</p>



<h2 class="wp-block-heading">&#8220;Did the job fail?&#8221; is the weakest question you can ask</h2>



<p class="wp-block-paragraph">Job status is a proxy for pipeline health, and a bad one. It tells you the orchestrator finished its instructions. It says nothing about whether those instructions produced data.</p>



<p class="wp-block-paragraph">SaaS sources make this worse than usual. A REST API returning HTTP 200 with an empty result array is, to your extraction code, a completely successful call. Pagination that stops on the first page because a cursor was reset is successful. An OAuth refresh that returns a token with fewer scopes than yesterday is successful right up until one object starts coming back empty. Every one of those is a green run.</p>



<p class="wp-block-paragraph">Build the monitoring around four questions the job outcome cannot answer:</p>



<ol class="wp-block-list"><li><strong>Liveness</strong> &#8211; did the run happen at all?</li><li><strong>Volume</strong> &#8211; did it move a plausible amount of data?</li><li><strong>Freshness</strong> &#8211; how old is the newest record that actually landed?</li><li><strong>Shape</strong> &#8211; are the fields and values still what downstream expects?</li></ol>



<p class="wp-block-paragraph">Each needs a different kind of metric, and each fails silently in its own way.</p>



<h2 class="wp-block-heading">Signal 1: liveness, and the missing-data trap that hides it</h2>



<p class="wp-block-paragraph">Step Functions publishes execution metrics to the <code>AWS/States</code> namespace, including <code>ExecutionsStarted</code>, <code>ExecutionsSucceeded</code>, <code>ExecutionsFailed</code>, <code>ExecutionsTimedOut</code>, <code>ExecutionsAborted</code> and <code>ExecutionTime</code>. The obvious move is an alarm on <code>ExecutionsFailed</code>. Most teams stop there, and that alarm is close to useless on its own.</p>



<p class="wp-block-paragraph">Here is why. Those metrics are published as a result of executions. If somebody disables the EventBridge schedule rule during an incident and forgets to re-enable it, no execution starts, so no data point is published for that dimension at all. Your alarm has nothing to evaluate.</p>



<p class="wp-block-paragraph">What happens next depends entirely on a parameter most people never set. CloudWatch alarms accept a <code>TreatMissingData</code> setting with four values: <code>breaching</code>, <code>notBreaching</code>, <code>ignore</code> and <code>missing</code>. The default is <code>missing</code>, which means the alarm keeps its current state when there is nothing to evaluate. An alarm that was sitting in OK stays in OK. Forever. It is green because it is blind, and there is no visual difference between the two.</p>



<p class="wp-block-paragraph">The fix is not to change <code>TreatMissingData</code> on the failure alarm. It is to add a separate heartbeat metric that your pipeline emits on completion, and alarm on the <em>absence</em> of that heartbeat:</p>



<pre class="wp-block-code"><code># Emitted at the very end of a successful run, per source system.
aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --value 1 
  --unit Count 
  --dimensions Source=salesforce,Stage=extract</code></pre>



<p class="wp-block-paragraph">Then the alarm that actually catches a dead scheduler:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name "ingest-salesforce-no-heartbeat" 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --dimensions Name=Source,Value=salesforce Name=Stage,Value=extract 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --datapoints-to-alarm 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts</code></pre>



<p class="wp-block-paragraph">The two flags doing the real work are <code>--comparison-operator LessThanThreshold</code> and <code>--treat-missing-data breaching</code>. The first catches a run that completes but emits nothing. The second catches a run that never happens at all. Without the second flag, silence reads as health.</p>



<p class="wp-block-paragraph">Set <code>--period</code> comfortably longer than your schedule interval so one slow run does not page anyone, and use <code>--datapoints-to-alarm</code> to require consecutive breaches. One caveat: <code>breaching</code> is right for a heartbeat you control, but a poor blanket default across the account, because plenty of AWS metrics are only emitted when non-zero. Throttling counters are legitimately sparse, and forcing them to breach on silence just makes noise.</p>



<h2 class="wp-block-heading">Signal 2: volume, and why a static threshold beats anomaly detection at first</h2>



<p class="wp-block-paragraph">A heartbeat proves the run happened. It does not prove the run did anything. Emit a row count, per source, per object:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RecordsWritten" 
  --value 14238 
  --unit Count 
  --dimensions Source=salesforce,Object=opportunity</code></pre>



<p class="wp-block-paragraph">Alarm on that being unexpectedly low rather than on it being zero. Zero-row alarms feel precise, but the more common failure is partial: a cursor that only advances through the first page, a rate limit that truncates the pull, an incremental sync that silently narrows its window. Those deliver a fraction of the expected rows, not none of them.</p>



<p class="wp-block-paragraph">There is a subtlety here that trips people. If a run fails before emitting <code>RecordsWritten</code> at all, a low-volume alarm on that metric sees missing data, not a zero. Metric math fixes it cleanly:</p>



<pre class="wp-block-code"><code>FILL(m1, 0)</code></pre>



<p class="wp-block-paragraph">Wrapping the metric in <code>FILL</code> with a value of 0 substitutes zero for every empty period, so an absent run and an empty run both breach the same alarm. You get one alert instead of two competing ones, and it means what you think it means.</p>



<p class="wp-block-paragraph">CloudWatch anomaly detection is the obvious upgrade for seasonal volume. It models expected behaviour and alarms on deviation from the band, which handles a Monday-morning spike that a static threshold cannot. I still would not reach for it first: it needs a stretch of representative history before the band is trustworthy, and during that stretch it will be confidently wrong. Start with a static floor well below your observed minimum, and move to anomaly detection once you know what normal looks like and can see the fixed number fighting you.</p>



<h2 class="wp-block-heading">Signal 3: freshness, the metric almost nobody emits</h2>



<p class="wp-block-paragraph">This is the one that separates a pipeline you can trust from one you merely watch. Liveness and volume can both look perfect while the data is stale, because a job that re-reads the same window every run produces rows, produces a heartbeat, and produces nothing new.</p>



<p class="wp-block-paragraph">The metric to emit is lag: the difference in seconds between now and the maximum source-side modification timestamp in the batch you just landed. Not your own processing timestamp, which will always look current. The source system&#8217;s timestamp, the one that tells you how far behind the truth you are.</p>



<pre class="wp-block-code"><code>import time
import boto3

cw = boto3.client("cloudwatch")

def emit_freshness(source, obj, max_source_ts_epoch):
    """max_source_ts_epoch: max LastModifiedDate in this batch, as epoch seconds."""
    lag_seconds = int(time.time()) - int(max_source_ts_epoch)
    cw.put_metric_data(
        Namespace="Pipelines/Ingest",
        MetricData=[{
            "MetricName": "SourceLagSeconds",
            "Value": lag_seconds,
            "Unit": "Seconds",
            "Dimensions": [
                {"Name": "Source", "Value": source},
                {"Name": "Object", "Value": obj},
            ],
        }],
    )</code></pre>



<p class="wp-block-paragraph">Alarm with <code>GreaterThanThreshold</code>, and set the threshold to roughly two schedule intervals plus your worst observed run duration. An hourly pipeline that takes twelve minutes on a bad day gets a threshold somewhere around two and a half hours. Tighter than that and you page on ordinary jitter.</p>



<p class="wp-block-paragraph">Freshness is also the one metric worth putting in front of non-engineers. &#8220;Your Salesforce data is currently 40 minutes behind&#8221; is a sentence a stakeholder can act on; &#8220;ExecutionsSucceeded is 1&#8221; is not. If you surface dashboards to a business audience through Grafana Cloud or QuickSight over the CloudWatch data source, lag is the panel that earns its space.</p>



<h2 class="wp-block-heading">Signal 4: shape, using logs instead of counters</h2>



<p class="wp-block-paragraph">Shape problems are schema drift and value drift. A SaaS admin renames a custom field, a picklist gains a value your mapping does not handle, a required field starts arriving null for a subset of records. The rows keep flowing. The row count looks fine. Downstream joins quietly drop records.</p>



<p class="wp-block-paragraph">Counters alone are the wrong shape here, because what you need at 2am is not &#8220;the null rate rose&#8221;, it is &#8220;which records, and what changed&#8221;. Embedded Metric Format is built for exactly that split: you write one structured JSON log line, CloudWatch extracts the numbers as real metrics you can alarm on, and the full event stays queryable in Logs Insights for the investigation.</p>



<pre class="wp-block-code"><code>{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "Pipelines/Quality",
        "Dimensions": [["Source", "Object"]],
        "Metrics": [
          { "Name": "NullRequiredFields", "Unit": "Count" },
          { "Name": "UnmappedFields", "Unit": "Count" }
        ]
      }
    ]
  },
  "Source": "salesforce",
  "Object": "opportunity",
  "NullRequiredFields": 412,
  "UnmappedFields": 2,
  "unmapped_field_names": ["Renewal_Tier__c", "Partner_Code__c"],
  "batch_id": "2f8c1a9e"
}</code></pre>



<p class="wp-block-paragraph">Two things to get right. Every name listed in <code>Dimensions</code> must also exist as a top-level property in the same JSON object, or extraction silently produces nothing. And note that <code>unmapped_field_names</code> and <code>batch_id</code> are properties, not dimensions: they ride along in the log event for querying but never become metric dimensions, which is exactly what you want.</p>



<p class="wp-block-paragraph">When the alarm fires, the investigation is a Logs Insights query against the same log group:</p>



<pre class="wp-block-code"><code>fields @timestamp, Object, UnmappedFields, unmapped_field_names
| filter Source = "salesforce" and UnmappedFields &gt; 0
| sort @timestamp desc
| limit 50</code></pre>



<p class="wp-block-paragraph">That hands you the new field names in seconds, which is the difference between a five-minute fix and an hour of guessing.</p>



<h2 class="wp-block-heading">Keeping CloudWatch data pipeline monitoring off the top of your bill</h2>



<p class="wp-block-paragraph">Custom metrics and log ingestion are consistently the two largest CloudWatch line items for data teams. Neither has a flat fee, so the cost is a direct function of choices you make in code. Three matter far more than the rest.</p>



<h3 class="wp-block-heading">Cardinality is the cost lever</h3>



<p class="wp-block-paragraph">A custom metric is billed per unique combination of namespace, metric name and dimension values. <code>Source</code> with five values and <code>Object</code> with twenty gives you a hundred metrics per metric name. Add <code>Tenant</code> with two hundred values and you have twenty thousand. Multiply by four metric names and the number stops being a rounding error.</p>



<p class="wp-block-paragraph">Never put a run ID, batch ID, record ID or raw error string in a dimension. Those belong in the log event. Dimensions are for things you would actually build an alarm on, and you are not going to build an alarm per batch.</p>



<h3 class="wp-block-heading">EMF over PutMetricData for anything high-frequency</h3>



<p class="wp-block-paragraph"><code>PutMetricData</code> is a synchronous API call on your critical path. In Lambda that means duration you pay for, plus a request that can throttle and needs handling. EMF avoids both: you write JSON to stdout, CloudWatch Logs extracts the metrics asynchronously, your function returns. For a per-run summary either is fine. For anything per batch or per record, EMF is the one I reach for.</p>



<h3 class="wp-block-heading">The Infrequent Access log class will break this silently</h3>



<p class="wp-block-paragraph">This one has caught competent people. CloudWatch Logs offers a Standard and an Infrequent Access log class, and IA has a lower ingestion price, which makes it an obvious target during a cost review. According to the AWS documentation, the IA class does not support embedded metric format, metric filters or subscription filters. Logs Insights queries still work.</p>



<p class="wp-block-paragraph">So if somebody moves your pipeline log group to IA to save money, every EMF-derived metric stops being extracted. The alarms built on those metrics go to <code>INSUFFICIENT_DATA</code>, or worse, sit in OK if they use the default missing-data behaviour. The logs still arrive. The dashboard still exists. The metrics are just gone.</p>



<p class="wp-block-paragraph">Log class cannot be changed after a log group is created, so recovery means a new Standard log group and redirecting output to it. Keep metric-backing log groups on Standard and point IA at the genuinely archival stuff: verbose debug output, raw payload dumps, anything you only read during a forensic investigation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Routing: one page, not four</h2>



<p class="wp-block-paragraph">Build all four signals and one broken pipeline fires the heartbeat, volume and freshness alarms within minutes of each other. Three pages, one problem. That is how alert fatigue starts.</p>



<p class="wp-block-paragraph">Composite alarms fix it. Build one composite alarm per pipeline whose rule combines the child alarms with OR, put the notification action there, and leave the child alarms with no actions at all. They become diagnostic detail you read after the page rather than three separate interruptions.</p>



<p class="wp-block-paragraph">SNS to email is fine for one person and poor past that, since it has no deduplication, escalation or acknowledgement. With more than one person on call, route through something with a schedule such as PagerDuty, Opsgenie or Better Stack. If your team already lives in Grafana Cloud or Datadog for application monitoring, pulling CloudWatch alarm state in through their AWS integration beats maintaining two alerting surfaces.</p>



<h2 class="wp-block-heading">Troubleshooting: the metric exists but the alarm never fires</h2>



<p class="wp-block-paragraph">Roughly in order of how often I have seen each one.</p>



<ul class="wp-block-list"><li><strong>Dimension mismatch.</strong> The alarm&#8217;s dimension set must match the published metric&#8217;s dimension set exactly, all of them, in the same values. A metric published with <code>Source</code> and <code>Object</code> is a different metric from one published with only <code>Source</code>. Run <code>aws cloudwatch list-metrics --namespace "Pipelines/Ingest"</code> and compare the output against your alarm definition character by character.</li><li><strong>Alarm stuck in INSUFFICIENT_DATA.</strong> The metric is not being published at all for that dimension set, or the period is shorter than the publishing interval so most periods are empty. Lengthen the period to at least the emission interval.</li><li><strong>Alarm stuck in OK during a real outage.</strong> Classic default <code>missing</code> behaviour. Any alarm whose purpose is to detect absence needs <code>--treat-missing-data breaching</code>.</li><li><strong>EMF metrics never appear.</strong> Check the log group&#8217;s class first, then confirm every dimension name in the <code>Dimensions</code> array exists as a top-level property in the same JSON object, then confirm the JSON is a single line with no wrapping prefix from your logger.</li><li><strong>Freshness spikes on a schedule.</strong> Almost always a timezone mismatch between how the SaaS API reports modification timestamps and how you convert them. Normalise everything to UTC epoch seconds at the boundary and stop converting anywhere else.</li><li><strong>Row counts double intermittently.</strong> Retries emitting the metric twice for one logical run. Emit volume metrics once, after the write is confirmed, not inside the retried block.</li></ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Alarming only on job failure, which misses every silent-success failure mode.</li><li>Leaving <code>TreatMissingData</code> at its default on alarms designed to detect absence.</li><li>Alarming on zero rows instead of on unexpectedly low rows, so partial extractions slip through.</li><li>Emitting a processing timestamp as the freshness metric, which is always current and therefore always useless.</li><li>Putting high-cardinality identifiers into metric dimensions and then being surprised by the bill.</li><li>Moving pipeline log groups to the Infrequent Access class without checking what depends on metric extraction.</li><li>Wiring notification actions to every child alarm instead of to one composite alarm per pipeline.</li><li>Building dashboards nobody looks at instead of alarms that reach a person.</li></ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list"><li>Emit liveness, volume, freshness and a shape metric for every source. Four metrics per source is a small amount of code and covers most of what actually breaks.</li><li>Define alarms in Terraform or CloudFormation alongside the pipeline, so a new source cannot ship without monitoring.</li><li>Use consistent dimension names across every pipeline. <code>Source</code> and <code>Object</code> everywhere makes search expressions and dashboards trivial; a mix of <code>source</code>, <code>system</code> and <code>connector</code> makes them impossible.</li><li>Set alarm descriptions to a one-line runbook. The person paged at 3am should not have to guess what the alarm means.</li><li>Test the absence case deliberately. Disable the schedule in a non-production account and confirm the heartbeat alarm fires. An untested absence alarm is an assumption.</li><li>Set explicit log retention on every pipeline log group. Never-expire is the default and it compounds quietly.</li><li>Keep an eye on the source side too, not just yours. Most SaaS APIs publish their own rate limit or API usage counters, and pulling those into a metric is often the earliest possible warning.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is CloudWatch enough on its own for data pipeline monitoring?</h3>



<p class="wp-block-paragraph">For alerting, yes, if your workloads are on AWS. It has metrics, alarms, composite alarms and log querying, and it is already collecting service metrics you would otherwise have to ship somewhere. Where it gets weak is exploratory analysis and cross-source correlation, which is where teams add Grafana, Datadog or a dedicated data-observability tool on top. Run CloudWatch for the paging path first and add the analysis layer when you have an actual gap.</p>



<h3 class="wp-block-heading">How do I monitor a pipeline that runs on a managed service like Glue or AppFlow?</h3>



<p class="wp-block-paragraph">The same four signals apply; only the emission point changes. Managed services publish their own metrics, but those describe the service&#8217;s execution, not your data. Where you cannot add code inside the job, hook the completion event through EventBridge to a small Lambda function that inspects what landed and emits the volume and freshness metrics. It is a handful of lines and it is the only way to get data-level signals out of a job you do not control.</p>



<h3 class="wp-block-heading">Should each tenant get its own metric in a multi-tenant pipeline?</h3>



<p class="wp-block-paragraph">Usually not as a dimension. Tenant counts grow, and cardinality is billed per unique dimension combination. Emit aggregate metrics with <code>Source</code> and <code>Object</code> dimensions, put the tenant identifier in the log event as a property, and use Logs Insights to slice by tenant when you need to. Break out a dimension per tenant only for the small number of accounts where a contractual SLA makes it worth the cost.</p>



<h3 class="wp-block-heading">What is a sensible freshness threshold?</h3>



<p class="wp-block-paragraph">Start at two schedule intervals plus your worst observed run duration, then adjust after watching real data for a couple of weeks. The threshold that matters is the one the business would notice, so if a stakeholder can tolerate half a day of staleness on a nightly export, do not page at four hours. Match the alarm to the actual consequence.</p>



<h3 class="wp-block-heading">Why does my alarm sit in INSUFFICIENT_DATA instead of ALARM?</h3>



<p class="wp-block-paragraph">CloudWatch has no data points to evaluate for that metric and dimension set. Either nothing is publishing, the dimensions do not match what is published, or the alarm period is shorter than the interval at which you emit. Confirm the metric exists with <code>list-metrics</code> first, and only then look at the alarm configuration.</p>



<h3 class="wp-block-heading">How much does this cost to run?</h3>



<p class="wp-block-paragraph">It depends on three things you control: the number of unique custom metrics, the volume of log data ingested, and how much data your Logs Insights queries scan. Alarms and dashboards are minor by comparison. Keep dimension cardinality low, set retention deliberately, and query narrow time windows, and the monitoring stays well below the cost of the pipeline it protects. Check current rates on the AWS pricing page rather than trusting any number in a blog post, including this one.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Effective CloudWatch data pipeline monitoring is built around absence, not around errors. Your SaaS integrations will fail far more often by succeeding at nothing than by throwing an exception, and every default in CloudWatch treats an absent metric as a non-event.</p>



<p class="wp-block-paragraph">If you only change one thing after reading this, add a heartbeat metric with <code>--treat-missing-data breaching</code> to your most important pipeline, then go and disable its schedule in a test account and confirm the alarm actually fires. If it does not, you have just found out on your own terms instead of from a stakeholder asking why the numbers look wrong.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your pipeline monitoring?</h2>



<p class="wp-block-paragraph">I work with teams running SaaS-to-warehouse pipelines on AWS, usually at the point where something broke quietly and nobody wants that to happen twice. Things I can help with here:</p>



<ul class="wp-block-list"><li>Auditing existing CloudWatch alarms for silent-failure gaps, particularly missing-data configuration and dimension mismatches that leave alarms permanently green.</li><li>Instrumenting liveness, volume, freshness and data-quality metrics into pipelines built on Lambda, Glue, Step Functions or self-hosted schedulers.</li><li>Designing EMF logging and metric namespaces that stay useful as source and object counts grow, without cardinality running away.</li><li>Building composite alarms and routing so one broken pipeline produces one actionable page rather than a cluster of them.</li><li>Cutting CloudWatch spend on log ingestion and custom metrics without losing the signals you actually rely on.</li><li>Writing Terraform modules that ship monitoring alongside every new source connector, so coverage is structural rather than remembered.</li></ul>



<p class="wp-block-paragraph">If you have a pipeline that has gone quiet, send me the alarm configuration and a Logs Insights output from the run in question and I will tell you what it is missing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/">CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</title>
		<link>https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 05 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Anomaly Detection]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[DQDL]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=148</guid>

					<description><![CDATA[<p>A SaaS admin changes a field and your pipeline stays green while the numbers drift. A practical guide to AWS Glue Data Quality for SaaS sources: where to run the checks, why nested payloads need flattening before DQDL can see them, which rule catches which failure, and the dynamic rules that pass silently because they have no history yet.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/">AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Nobody deployed anything. That is the part that makes it hard to explain.</p>



<p class="wp-block-paragraph">A Salesforce admin renames a custom field on Tuesday afternoon. Your connector keeps running, your Glue job keeps succeeding, the crawler happily adds a new column and quietly stops populating the old one. Three weeks later someone asks why the pipeline report has a cliff in it, and you spend an afternoon reconstructing a change nobody in engineering made and nobody logged.</p>



<p class="wp-block-paragraph">This post is about using AWS Glue Data Quality to catch that class of breakage. It is aimed at people pulling from SaaS sources into S3 or a lakehouse: Salesforce, HubSpot, Zendesk, Stripe, Shopify, Zoho, an internal partner API. I will cover where to run the checks, the flattening constraint that will bite you before anything else does, which DQDL rule actually catches which SaaS failure, when dynamic rules and anomaly detection quietly do nothing, and what should happen when a rule fails.</p>



<h2 class="wp-block-heading">Why SaaS data breaks in ways your own database doesn&#8217;t</h2>



<p class="wp-block-paragraph">When you own the source database, the schema is under change control: someone writes a migration, it gets reviewed, and if it breaks a consumer there is a commit to point at. A SaaS source has none of that. Three consequences:</p>



<ul class="wp-block-list">
<li><strong>The schema is edited by people who are not engineers.</strong> A sales ops admin adding a picklist value is a production schema change with no review, no notice and no version number.</li>

<li><strong>The payload is nested.</strong> SaaS APIs return structs and arrays: line items, related objects, custom field bags. That shape matters more than you would expect, for reasons in the next section but one.</li>

<li><strong>Absence is ambiguous.</strong> A field missing from a payload might mean &#8220;not set,&#8221; &#8220;not permitted for this user,&#8221; or &#8220;we removed it.&#8221; The connector usually cannot tell you which, and neither can your table.</li>
</ul>



<p class="wp-block-paragraph">Glue Data Quality is a rules engine built on Deequ that evaluates declarative checks written in DQDL, the Data Quality Definition Language, against a dataset. It is well suited to the first and third problems. The second one needs work before it is useful at all.</p>



<h2 class="wp-block-heading">Decide where the checks run before you write a rule</h2>



<p class="wp-block-paragraph">There are two places to evaluate a ruleset, and they answer different questions.</p>



<h3 class="wp-block-heading">Inside the Glue ETL job</h3>



<p class="wp-block-paragraph">The <code>EvaluateDataQuality</code> transform runs mid-pipeline against a DynamicFrame. It answers &#8220;is this batch safe to load?&#8221; and it can act on the answer: stop the job, or split failing rows into a quarantine prefix. This is also the only placement where row-level results exist, which matters because &#8220;which 400 rows are bad&#8221; is a far more useful answer than &#8220;the rule failed.&#8221;</p>



<h3 class="wp-block-heading">Against a Data Catalog table</h3>



<p class="wp-block-paragraph">You can attach a ruleset to a catalog table and run it on a schedule, independent of any job. This answers &#8220;is the table currently trustworthy?&#8221; and surfaces a quality score analysts can see before they query. The trade-off is that it is after the fact: by the time a catalog evaluation fails, the bad batch is already queryable. Analyzers are also not supported for catalog evaluations, so some dynamic-rule tooling behaves differently there.</p>



<p class="wp-block-paragraph">In practice I run both. The in-job ruleset is small and strict, and exists to stop garbage landing. The catalog ruleset is broader and looser, and exists so a table&#8217;s state is visible to people who will never open Glue Studio.</p>



<h2 class="wp-block-heading">The flattening problem that comes first</h2>



<p class="wp-block-paragraph">Here is the constraint that catches most people on SaaS data, stated plainly in the AWS documentation and easy to skim past: <strong>DQDL cannot evaluate nested or list-type columns.</strong> No structs, no arrays.</p>



<p class="wp-block-paragraph">Almost every interesting SaaS payload is nested. A Salesforce opportunity carries a related account struct, a Stripe invoice carries an array of line items, a Zendesk ticket carries a custom fields array of key-value pairs. Point a ruleset at the raw landed data and the fields you most want to check are the ones the engine cannot see.</p>



<p class="wp-block-paragraph">So you flatten first, with Glue&#8217;s relationalize or an explicit projection, then evaluate. That works, but it changes what your rules mean:</p>



<ul class="wp-block-list">
<li><strong>Your rules now describe the projection, not the payload.</strong> If your flattening step drops a field, no rule will notice, because from DQDL&#8217;s point of view that field was never there.</li>

<li><strong>Exploding an array changes the grain.</strong> One invoice with five line items becomes five rows. Every <code>RowCount</code> rule you write against that table is now counting line items, and a customer who starts itemising differently will move your row count without any data being wrong.</li>

<li><strong>Custom-field bags need pivoting, not flattening.</strong> A key-value array flattened naively gives you rows, not columns, and you cannot write a <code>ColumnValues</code> rule against a value whose meaning depends on a sibling key column. Pivot the ones you care about into real columns and check those.</li>
</ul>



<p class="wp-block-paragraph">The practical rule: put your quality checks immediately after the flattening step, and treat the flattening step itself as code that needs its own test. Glue Data Quality will not cover it for you.</p>



<h2 class="wp-block-heading">Matching SaaS breakage to the rule that catches it</h2>



<p class="wp-block-paragraph">DQDL has a long rule list and most guides just enumerate it. More useful is the other direction: what actually goes wrong with SaaS sources, and what catches each one.</p>



<h3 class="wp-block-heading">A field is renamed, removed, or added</h3>



<p class="wp-block-paragraph">This is the Tuesday-afternoon scenario. Two rules, doing two different jobs:</p>



<pre class="wp-block-code"><code>Rules = [
    ColumnExists "annual_revenue__c",
    ColumnCount = 47
]</code></pre>



<p class="wp-block-paragraph"><code>ColumnExists</code> protects the specific fields your downstream models depend on. <code>ColumnCount</code> protects against everything else: a field silently added, or one you forgot you needed. The first is precise and the second is a tripwire, and you want both because a rename shows up as one column gone and one column arrived, which leaves the count unchanged.</p>



<p class="wp-block-paragraph"><code>SchemaMatch</code> compares against a reference dataset&#8217;s schema, which is stricter and useful if you maintain a golden reference table. Bear in mind a hard column count on a SaaS object fails every time an admin adds a field, so decide whether that blocks or just warns.</p>



<h3 class="wp-block-heading">A field still exists but stopped being populated</h3>



<p class="wp-block-paragraph">Permissions change, a workflow gets disabled, an integration user loses field-level access. The column is still in the schema and is now all nulls.</p>



<pre class="wp-block-code"><code>Rules = [
    IsComplete "opportunity_id",
    Completeness "lead_source" &gt; 0.85,
    Completeness "email" &gt; 0.60 where "record_type = 'Contact'"
]</code></pre>



<p class="wp-block-paragraph"><code>IsComplete</code> is the absolute version: no nulls at all, right for keys. <code>Completeness</code> takes a threshold, which is what you want for optional business fields where a hard rule would fail constantly. The <code>where</code> clause is the one people miss: SaaS objects are usually polymorphic, and a completeness threshold that makes sense for contacts is nonsense across the whole table. Scope the rule to the record type.</p>



<h3 class="wp-block-heading">The sync ran short</h3>



<p class="wp-block-paragraph">A paginated extract stops early, a rate limit gets absorbed by retries, an incremental cursor drifts. The job succeeds and delivers less data than it should have. A static <code>RowCount &gt; 1000</code> stops catching this the moment you cross a thousand rows. Dynamic rules compare against your own history instead:</p>



<pre class="wp-block-code"><code>Rules = [
    RowCount &gt; min(last(3)),
    Sum "amount" &gt; avg(last(5)) * 0.7,
    DetectAnomalies "RowCount"
]

Analyzers = [
    RowCount,
    DistinctValuesCount "owner_id"
]</code></pre>



<p class="wp-block-paragraph"><code>last(k)</code> pulls the last k recorded values of that metric for the same dataset; with k greater than one you need an aggregation like <code>avg</code>, <code>min</code> or <code>max</code> to reduce them to one number. <code>DetectAnomalies</code> goes further, using forecasting over stored history to flag deviations with no threshold from you at all. That is the right tool for anything seasonal: a B2B pipeline that dips every weekend trips a naive trailing-average rule every Monday, while anomaly detection learns the shape.</p>



<p class="wp-block-paragraph">Analyzers deserve their own mention: they gather a metric without asserting anything, which builds the history dynamic rules and anomaly detection need. <code>DistinctValuesCount</code> on an owner or account column is a strong early signal for a partial sync, because row count can look fine while the number of distinct accounts represented quietly halves.</p>



<h3 class="wp-block-heading">Records arrive twice, or the key stops being a key</h3>



<p class="wp-block-paragraph">Replays, overlapping incremental windows and connector retries all produce duplicates, and SaaS &#8220;IDs&#8221; are less unique than vendor docs imply once you have merged sandboxes or multiple accounts.</p>



<pre class="wp-block-code"><code>Rules = [
    IsPrimaryKey "id",
    Uniqueness "external_reference" &gt; 0.99
]</code></pre>



<p class="wp-block-paragraph"><code>IsPrimaryKey</code> asserts unique and non-null together. <code>Uniqueness</code> with a threshold is the softer version for fields that are mostly-but-not-perfectly unique, which describes a lot of SaaS reference fields honestly.</p>



<h3 class="wp-block-heading">Objects stop agreeing with each other</h3>



<p class="wp-block-paragraph">Multi-object extracts rot quietly. Opportunities reference accounts that were never synced; invoice totals stop matching line item sums because one object&#8217;s extract failed and the other&#8217;s didn&#8217;t.</p>



<p class="wp-block-paragraph"><code>ReferentialIntegrity</code> checks that values in one dataset&#8217;s column exist in another&#8217;s. <code>RowCountMatch</code> and <code>AggregateMatch</code> compare counts and summary metrics across two datasets. These catch a broken join before an analyst does, and they need the multi-frame form of the transform with the second dataset supplied as an additional source.</p>



<h3 class="wp-block-heading">The sync stopped and nobody noticed</h3>



<p class="wp-block-paragraph">The quietest failure of all: nothing is wrong with the data, there just isn&#8217;t any new data. <code>DataFreshness</code> checks a timestamp column against a window. It is deliberately simple and its comparison granularity is limited, so for anything more nuanced the custom SQL rule is the escape hatch, running a query against the dataset and evaluating the result against an expression. DQDL is case sensitive and that rule type&#8217;s exact capitalisation has appeared more than one way across the docs and the visual builder, so copy it from the current DQDL reference rather than from memory.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Static, dynamic and ML: when each one lies to you</h2>



<p class="wp-block-paragraph">All three check styles have a mode where they pass while the data is wrong. Knowing which is which is most of the skill.</p>



<ul class="wp-block-list">
<li><strong>Static rules lie by going stale.</strong> A threshold that was correct when you wrote it stops being correct as the business grows, and a rule that can no longer fail is worse than no rule because it looks like coverage.</li>

<li><strong>Dynamic rules lie when they have no history.</strong> This one is genuinely nasty. <code>last(k)</code> reads previously recorded values for that metric, and if there aren&#8217;t any, it falls back to a default of zero. Add <code>RowCount &gt; avg(last(3))</code> to an existing job today and it will pass trivially for the first few runs, because almost any row count beats zero. It looks like a working check from the moment you deploy it. It isn&#8217;t one until it has accumulated runs.</li>

<li><strong>Anomaly detection lies while it is warming up, and after you retrain it badly.</strong> It needs a run of history before its forecasts mean anything, and if you leave a period of genuinely bad data in the training window, it learns that the bad shape is normal. You can exclude specific statistics from the calculation in the Data Quality tab, and you should, after any incident.</li>
</ul>



<p class="wp-block-paragraph">So when you add a dynamic or ML-based rule, write down the date and treat it as unarmed for the first couple of weeks. Keep a crude static rule alongside it in the meantime.</p>



<h2 class="wp-block-heading">What should happen when a rule fails</h2>



<p class="wp-block-paragraph">The transform returns a collection. The two members you care about are the row-level outcomes, which give you the original data with an evaluation result column appended per row, and the rule outcomes, which give you one row per rule with its verdict.</p>



<pre class="wp-block-code"><code>from awsgluedq.transforms import EvaluateDataQuality

dq_results = EvaluateDataQuality().process_rows(
    frame=flattened_opportunities,
    ruleset=ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "salesforce_opportunity",
        "enableDataQualityCloudWatchMetrics": "true",
        "enableDataQualityResultsPublishing": "true",
    },
)

row_level = SelectFromCollection.apply(
    dfc=dq_results, key="rowLevelOutcomes"
)</code></pre>



<p class="wp-block-paragraph"><code>dataQualityEvaluationContext</code> is the namespace your CloudWatch metrics land under. Name it after the source object rather than the job, because you will eventually want to alert per object.</p>



<p class="wp-block-paragraph">From there, three response patterns:</p>



<ol class="wp-block-list">
<li><strong>Fail fast.</strong> Stop the job on failure so nothing lands. Right for structural rules: if the primary key isn&#8217;t unique, downstream is going to be wrong no matter what you do next.</li>

<li><strong>Quarantine.</strong> Filter the row-level outcomes on the evaluation result column, write passing rows to the target and failing rows to a separate prefix. Right for value-level rules where 2% bad records shouldn&#8217;t block 98% good ones. The important discipline is that somebody has to actually look at the quarantine prefix; an unmonitored quarantine is a delete with extra steps.</li>

<li><strong>Warn and continue.</strong> Let everything land, publish the metrics, alert on the score. Right for new rules you don&#8217;t trust yet, and for the catalog-side ruleset.</li>
</ol>



<p class="wp-block-paragraph">Whichever you pick, get the failure out of Glue and into wherever your team actually looks. Results and anomaly predictions can be written back to catalog tables so they are queryable with plain SQL, and CloudWatch metrics plus EventBridge events give you the hook for paging or pausing a downstream pipeline. A Grafana or Datadog dashboard fed from those metrics beats a score in a console tab nobody opens.</p>



<h2 class="wp-block-heading">Troubleshooting AWS Glue Data Quality on SaaS pipelines</h2>



<h3 class="wp-block-heading">&#8220;Input data does not include column X&#8221;</h3>



<p class="wp-block-paragraph">A rule references a column the engine cannot find. On SaaS data that means one of three things: the field disappeared from the source, your flattening step renamed it, or the Data Catalog is out of sync with the underlying table. Check the catalog schema against the landed files before touching the ruleset; rerunning the crawler resolves a surprising share of these.</p>



<h3 class="wp-block-heading">A rule type isn&#8217;t recognised</h3>



<p class="wp-block-paragraph">DQDL is case sensitive and the rule set has grown over releases, so a rule copied from an older blog post can parse fine in your head and not at all in the engine. Check the rule name against the current DQDL rule type reference, and check that the feature is available on your Glue version. Anomaly detection in particular has a version floor.</p>



<h3 class="wp-block-heading">A dynamic rule never fails</h3>



<p class="wp-block-paragraph">Almost always the empty-history problem described above. Look at the statistics for that metric in the Data Quality tab. If there are fewer recorded points than your <code>last(k)</code> asks for, the rule is not doing what you think. It will start working on its own; the mistake is trusting it in the meantime.</p>



<h3 class="wp-block-heading">Evaluation is slow or the job runs out of memory</h3>



<p class="wp-block-paragraph">Each rule triggers analyzer computation over the dataset, and a large ruleset on a wide flattened SaaS table adds up fast. The transform exposes caching and observations-scope options to tune this. Before reaching for a bigger worker type, cut the ruleset down: most contain a dozen recommended rules nobody reads the results of.</p>



<h3 class="wp-block-heading">A &#8220;column reference is ambiguous&#8221; error</h3>



<p class="wp-block-paragraph">Classic flattening artefact. Two nested paths collapsed to the same leaf name, so <code>datechecked</code> now matches two things. Fix it in the projection with explicit aliases rather than trying to disambiguate inside DQDL.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Accepting the recommended ruleset wholesale. Recommendations are generated from a sample of current data, so they encode today&#8217;s accidents as tomorrow&#8217;s requirements.</li>

<li>Writing rules against nested data and assuming they cover the whole payload.</li>

<li>Deploying a dynamic rule and treating it as armed on day one.</li>

<li>Applying whole-table completeness thresholds to a polymorphic object instead of scoping with a <code>where</code> clause.</li>

<li>Quarantining failed rows to a prefix nobody monitors.</li>

<li>Writing <code>RowCount</code> rules against an exploded array without noticing the grain changed.</li>

<li>Failing the job on every rule, including cosmetic ones, until the team starts rerunning with checks disabled.</li>

<li>Keeping the ruleset in the console instead of in version control next to the job.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Split rules into blocking and reporting tiers, and be honest about which is which. A rule that always fails and never stops anything trains people to ignore the tab.</li>

<li>Pair <code>ColumnExists</code> with <code>ColumnCount</code>. One protects the fields you named, the other catches the drift you did not anticipate.</li>

<li>Add analyzers early, even for metrics you have no rule for yet. History has to accumulate before dynamic rules and anomaly detection are worth anything, and you cannot backfill it.</li>

<li>Keep DQDL in Git alongside the job definition and deploy it with Terraform or CloudFormation. Rulesets are code and should move through the same review as the pipeline they guard.</li>

<li>Namespace the evaluation context per source object so CloudWatch metrics and alerts can be routed per vendor.</li>

<li>Exclude incident periods from the anomaly detection training window once the incident is resolved.</li>

<li>Surface results where stakeholders already look. Writing outcomes back to catalog tables makes them queryable from Athena, which beats a score in a console tab.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much does AWS Glue Data Quality cost?</h3>



<p class="wp-block-paragraph">It bills on Glue DPU-hours like any other Glue workload, so cost is a function of evaluation time and DPUs used. The dimension that surprises people is anomaly detection, which consumes additional capacity per statistic analysed, so enabling it broadly across many columns and tables is where the bill moves. Model it against the current Glue pricing page; rates change and vary by region.</p>



<h3 class="wp-block-heading">Can it check data before it lands in S3?</h3>



<p class="wp-block-paragraph">Yes, if the check runs inside the Glue job. The transform evaluates a DynamicFrame in memory, so you can validate after extraction and before the write. That is the placement that lets you stop a bad load rather than report on one.</p>



<h3 class="wp-block-heading">Does it work on nested JSON from a SaaS API?</h3>



<p class="wp-block-paragraph">Not directly. DQDL does not evaluate nested or list-type columns, so you have to flatten the structure first and run the ruleset against the flattened frame. Plan the flattening deliberately, because your rules only ever see what the projection exposes.</p>



<h3 class="wp-block-heading">How does it compare to Great Expectations, Soda or Monte Carlo?</h3>



<p class="wp-block-paragraph">Glue Data Quality&#8217;s real advantage is that it is already inside the runtime and the catalog: no extra infrastructure, no separate agent, one IAM story. The disadvantages are real too. DQDL is less expressive than Python-based assertions, the nested-data limitation is a hard wall, and it only covers what Glue can read. Great Expectations gives you more expressive checks at the cost of running it yourself; Monte Carlo and similar platforms add lineage and cross-warehouse coverage Glue does not attempt. Glue-centric stack, start here. Several engines, a standalone tool fits better.</p>



<h3 class="wp-block-heading">Should I use the recommended rules?</h3>



<p class="wp-block-paragraph">As a starting draft, yes. As a ruleset, no. Recommendations are inferred from a sample of the data as it is today, which means any existing quality problem gets encoded as the expected baseline. Read every generated rule and delete the ones you would not have written yourself.</p>



<h3 class="wp-block-heading">Can I reconcile row counts against the SaaS source itself?</h3>



<p class="wp-block-paragraph">Not directly; it only sees datasets it can read. The usual pattern is to have the extraction step record the count the API reported, land that alongside the data as a control table, then use <code>RowCountMatch</code> or <code>AggregateMatch</code> to compare the two. That turns &#8220;did we get everything&#8221; into a rule the engine can evaluate.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">AWS Glue Data Quality is not really a data quality product. It is a way of writing down what you believe about a dataset so a machine can check it every run, and its value is entirely determined by whether those beliefs are written precisely enough to be wrong.</p>



<p class="wp-block-paragraph">For SaaS sources that means two habits. Flatten deliberately and check immediately after, because your rules only see the projection. And treat every new dynamic or ML-based rule as unarmed until it has history, because a rule that cannot fail yet looks exactly like a rule that is passing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with data quality on your Glue pipelines?</h2>



<p class="wp-block-paragraph">I work with teams whose SaaS pipelines run green while the numbers drift. Things I can help with:</p>



<ul class="wp-block-list">
<li>Writing a DQDL ruleset for a specific SaaS object that catches schema drift instead of just restating the current schema.</li>

<li>Designing the flattening and pivoting step for nested payloads and custom-field bags so the checks actually cover what matters.</li>

<li>Setting up quarantine and fail-fast paths in Glue ETL, with the routing and alerting that makes a quarantine prefix something people act on.</li>

<li>Cross-object reconciliation between SaaS extracts and control counts recorded at extraction time.</li>

<li>Tuning anomaly detection and dynamic rules, including cleaning incident periods out of the training window.</li>

<li>Getting data quality results out of the console and into Grafana, CloudWatch alarms or an Athena-queryable table.</li>
</ul>



<p class="wp-block-paragraph">If you have a ruleset that keeps passing while the data drifts, send me the DQDL and a schema dump. That is usually enough to see where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/">AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
