<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Data Observability | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/data-observability/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/data-observability/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Mon, 14 Sep 2026 13:13:17 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Data Observability | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/data-observability/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates</title>
		<link>https://john-nessime.com/blog/devops/data-pipeline-freshness-monitoring/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 08 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[Deadman Alerting]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Kafka]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[PromQL]]></category>
		<category><![CDATA[Pushgateway]]></category>
		<category><![CDATA[SLO]]></category>
		<category><![CDATA[SRE]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=407</guid>

					<description><![CDATA[<p>Every task green, zero errors, and the dashboard still shows yesterday's numbers. Run status describes your code, not your data. Here is how to instrument freshness, latency and failure rate so your pipeline tells you before an analyst does.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-freshness-monitoring/">Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message lands at 8:40 in the morning. &#8220;Are the revenue numbers right? They look like yesterday&#8217;s.&#8221; You open the orchestrator and every task is green. The run finished in eleven minutes, comfortably inside its window. Zero errors. Zero retries. The pipeline did exactly what it was told to do, and the number on the dashboard is a day old.</p>



<p class="wp-block-paragraph">That gap is the entire problem. Run status tells you the code executed. It tells you nothing about whether data arrived. Data pipeline freshness monitoring is the signal that closes the gap, and most teams add it only after an analyst finds the stale table first.</p>



<p class="wp-block-paragraph">This post covers the three signals worth instrumenting for a batch or streaming pipeline: how current the data is, how long it took to get there, and how often the thing breaks. For each one I&#8217;ll cover what to measure, where to measure it, the queries and config that produce it, and the specific ways the measurement lies to you.</p>



<h2 class="wp-block-heading">The signal that lies to you first: run status</h2>



<p class="wp-block-paragraph">Job success is a statement about your code path, not about your data. A pipeline that reads an empty S3 prefix, transforms zero rows, writes zero rows and exits cleanly has succeeded. So has one whose upstream API silently started returning an empty page after a token rotation. So has one whose incremental watermark got stuck and now re-reads the same already-loaded slice on every run.</p>



<p class="wp-block-paragraph">All three are green. All three are producing stale data. This is why the ordering matters: freshness is the signal that fires first in a real incident, and run status is usually the last one to notice anything at all.</p>



<p class="wp-block-paragraph">So keep run status. It&#8217;s cheap and it catches crashes. Just stop treating it as your top-line health indicator.</p>



<h2 class="wp-block-heading">Freshness: measure the table, not the job</h2>



<p class="wp-block-paragraph">Freshness is the age of the newest record in a dataset. It&#8217;s measured against the dataset itself, not against the process that filled it, and that distinction is what makes it useful. If your job disappears entirely, freshness keeps climbing and keeps alerting. If your job succeeds while doing nothing, freshness keeps climbing and keeps alerting. It&#8217;s the one signal that survives both failure modes.</p>



<h3 class="wp-block-heading">Two clocks, and you need both</h3>



<p class="wp-block-paragraph">Every row usually carries two timestamps: when the event happened in the source system, and when your pipeline wrote it. Track the maximum of each.</p>



<ul class="wp-block-list">
<li><strong>Max event time</strong> tells you whether the upstream system is still producing. If this stops moving, the problem is upstream of you.</li>



<li><strong>Max load time</strong> tells you whether your pipeline is still writing. If this stops moving while event time is fine, the problem is yours.</li>
</ul>



<p class="wp-block-paragraph">Watching only one of them means every stale-data page starts with twenty minutes of figuring out which side of the boundary the fault sits on. Watching both answers that in the alert body.</p>



<p class="wp-block-paragraph">A freshness probe is a small scheduled query. This one is PostgreSQL syntax, and it returns both clocks plus a volume check in a single round trip:</p>



<pre class="wp-block-code"><code>select
  'orders' as dataset,
  extract(epoch from max(event_ts))  as max_event_ts,
  extract(epoch from max(loaded_at)) as max_loaded_at,
  count(*) filter (where loaded_at &gt;= now() - interval '1 hour') as rows_last_hour
from analytics.orders;</code></pre>



<p class="wp-block-paragraph">The row count matters. A pipeline can advance its load timestamp while writing almost nothing, which is what a partially broken source looks like. Freshness alone will not catch that; freshness plus volume will.</p>



<p class="wp-block-paragraph">Run that probe on a schedule that&#8217;s independent of the pipeline, push the results as gauges, and alert on age in PromQL:</p>



<pre class="wp-block-code"><code># Newest row is older than 90 minutes
time() - max by (dataset) (dataset_max_loaded_timestamp_seconds) &gt; 5400

# The probe itself has stopped reporting: a deadman check
absent(dataset_max_loaded_timestamp_seconds{dataset="orders"})</code></pre>



<p class="wp-block-paragraph">That second rule is the one people forget. A freshness metric that disappears looks identical to a healthy silence on a graph. <code>absent()</code> is what turns &#8220;no data&#8221; into a page.</p>



<h3 class="wp-block-heading">Where dbt fits</h3>



<p class="wp-block-paragraph">If you&#8217;re already running dbt, source freshness is built in and worth using before you write anything custom. You declare thresholds per source, and dbt queries the maximum of your timestamp column and compares it to now.</p>



<pre class="wp-block-code"><code>sources:
  - name: raw_shop
    schema: raw
    config:
      loaded_at_field: _loaded_at
      freshness:
        warn_after: {count: 2, period: hour}
        error_after: {count: 6, period: hour}
    tables:
      - name: orders
      - name: refunds
        config:
          freshness:
            warn_after: {count: 12, period: hour}
            error_after: {count: 24, period: hour}</code></pre>



<p class="wp-block-paragraph">Three details that trip people up. First, <code>dbt source freshness</code> is a separate command; <code>dbt build</code> does not run it, so a green build says nothing about source staleness. Second, it exits non-zero when a source hits its <code>error_after</code> threshold, which makes it a natural gate at the top of a job: fail fast rather than building models on stale input. Third, results land in <code>target/sources.json</code>, which is the artifact you parse if you want to distinguish warn from error, or ship the numbers into Prometheus rather than just failing the run.</p>



<pre class="wp-block-code"><code>dbt source freshness --select source:raw_shop --output target/freshness.json</code></pre>



<p class="wp-block-paragraph">One caveat on the YAML above: dbt moved these keys under a <code>config:</code> block in recent releases, and <code>loaded_at_field</code> followed later. Older projects nest them directly under the source. Check what your project&#8217;s version expects before copying, because a misplaced key fails quietly by simply not calculating freshness at all.</p>



<h2 class="wp-block-heading">Latency: name the clock before you name the number</h2>



<p class="wp-block-paragraph">&#8220;Our pipeline latency is twelve minutes&#8221; is meaningless until you say which two points you measured between. There are at least three plausible definitions, and teams routinely argue past each other because they&#8217;re each using a different one.</p>



<ul class="wp-block-list">
<li><strong>Run duration.</strong> Start to end of the job. Easy, and mostly useless for anyone downstream.</li>



<li><strong>Ingestion latency.</strong> Source extract to target load. This is what you control.</li>



<li><strong>End-to-end latency.</strong> Event time to the moment the row is queryable. This is what the business actually feels, and it includes queue wait, scheduler delay, and every upstream hop you don&#8217;t own.</li>
</ul>



<p class="wp-block-paragraph">Publish end-to-end as the headline and keep the others as breakdown. If you only track run duration, a scheduler backlog that delays every run by forty minutes is completely invisible to you: each individual run still takes eleven minutes.</p>



<p class="wp-block-paragraph">Instrument per stage with a histogram so you can ask percentile questions later without re-instrumenting:</p>



<pre class="wp-block-code"><code>histogram_quantile(
  0.95,
  sum by (le, stage) (rate(pipeline_stage_duration_seconds_bucket[6h]))
)</code></pre>



<p class="wp-block-paragraph">Use the median for capacity planning and p95 or p99 for the SLO. Averages hide the exact tail that generates the complaints, and on a pipeline that runs a few dozen times a day, a single pathological run is a real fraction of your day.</p>



<p class="wp-block-paragraph">For streaming, the equivalent is consumer lag expressed in time rather than offsets. Offset lag of fifty thousand messages means nothing without a rate; two hundred seconds of lag means something to everyone. Kafka Lag Exporter popularised this by interpolating a time estimate from observed offset and timestamp samples, exposing <code>kafka_consumergroup_group_max_lag_seconds</code> alongside the offset-based <code>kafka_consumergroup_group_lag</code>. Worth knowing: that project&#8217;s repository has been archived and is read-only, so if you&#8217;re starting fresh, check whether your broker vendor or a maintained fork covers it before you deploy something unmaintained into the critical path.</p>



<h2 class="wp-block-heading">Failure rate: decide what counts as a failure</h2>



<p class="wp-block-paragraph">Failure rate is trivially easy to compute and surprisingly easy to compute wrongly. The denominator and the definition both need a decision.</p>



<p class="wp-block-paragraph">Count runs, not tasks. A DAG with sixty tasks where one flaps on a transient network error looks catastrophic at task level and fine at run level. The run is the unit the consumer cares about.</p>



<p class="wp-block-paragraph">Count a run that succeeded on its third retry as a success for availability and a failure for a separate reliability metric. Both are true and they answer different questions. If your only metric folds retries into success, you will never see the slow degradation of an upstream API until it stops responding entirely.</p>



<pre class="wp-block-code"><code>sum by (pipeline) (rate(pipeline_runs_total{result="failure"}[6h]))
/
sum by (pipeline) (rate(pipeline_runs_total[6h]))</code></pre>



<p class="wp-block-paragraph">And add a category label for the failure reason at push time: source unavailable, schema mismatch, permission denied, timeout, validation failed. The rate tells you something is wrong. The category tells you who to wake up. Without it, every failure alert costs you a log dive before you can even route the incident.</p>



<h2 class="wp-block-heading">Getting metrics out of a job that exits</h2>



<p class="wp-block-paragraph">Prometheus scrapes. Batch jobs finish and vanish. Pushgateway bridges that: the job pushes before exiting, and Pushgateway holds the values for Prometheus to scrape on its own schedule.</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
set -euo pipefail

JOB="orders_load"
PGW="http://pushgateway.internal:9091"

start=$(date +%s)
if python /opt/pipelines/load_orders.py; then
  result=0
else
  result=1
fi
end=$(date +%s)

{
  cat &lt;&lt;EOF
# TYPE pipeline_run_duration_seconds gauge
pipeline_run_duration_seconds $((end - start))
# TYPE pipeline_last_run_timestamp_seconds gauge
pipeline_last_run_timestamp_seconds $end
# TYPE pipeline_last_run_success gauge
pipeline_last_run_success $((1 - result))
EOF
  if [ "$result" -eq 0 ]; then
    cat &lt;&lt;EOF
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $end
EOF
  fi
} | curl --fail --data-binary @- "$PGW/metrics/job/$JOB"

exit $result</code></pre>



<p class="wp-block-paragraph">The conditional block is doing real work. curl&#8217;s <code>--data-binary</code> issues a POST, and a POST to Pushgateway replaces only the metrics whose names appear in the payload, leaving the rest of the group intact. So a failing run updates the run timestamp and the success flag while leaving the previous <code>pipeline_last_success_timestamp_seconds</code> exactly where it was. That&#8217;s what lets <code>time() - pipeline_last_success_timestamp_seconds</code> keep climbing across consecutive failures. Send a PUT instead and you replace the whole group, wiping the value you needed.</p>



<p class="wp-block-paragraph">On the Prometheus side, one setting is not optional:</p>



<pre class="wp-block-code"><code>scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway.internal:9091']</code></pre>



<p class="wp-block-paragraph">Without <code>honor_labels: true</code>, Prometheus overwrites the <code>job</code> label your pipeline pushed with the scrape job&#8217;s own name, and every pipeline in your estate collapses into one indistinguishable series called <code>pushgateway</code>.</p>



<p class="wp-block-paragraph">The trap worth internalising: Pushgateway never forgets. Metrics persist until something explicitly deletes them or the process restarts. A pipeline you decommissioned last quarter is still cheerfully reporting a success timestamp and a duration, and it looks alive on every dashboard. Treat Pushgateway as a cache of past executions, not a picture of current state, and use the <code>push_time_seconds</code> gauge that Pushgateway attaches to each group to tell the difference between a fresh push and a fossil.</p>



<h3 class="wp-block-heading">If you&#8217;re on Airflow, the SLA feature is gone</h3>



<p class="wp-block-paragraph">This one will bite anyone upgrading. The <code>sla</code> and <code>sla_miss_callback</code> parameters were removed in Airflow 3.0, and the replacement, Deadline Alerts, arrived in 3.1. DAGs carrying the old configuration need manual migration; they don&#8217;t quietly keep working.</p>



<pre class="wp-block-code"><code>from datetime import timedelta

from airflow.sdk import AsyncCallback, DAG, DeadlineAlert, DeadlineReference
from airflow.providers.slack.notifications.slack_webhook import SlackWebhookNotifier

with DAG(
    dag_id="orders_load",
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_QUEUED_AT,
        interval=timedelta(minutes=45),
        callback=AsyncCallback(
            SlackWebhookNotifier,
            kwargs={"text": "orders_load has not finished 45 minutes after queuing."},
        ),
    ),
):
    ...</code></pre>



<p class="wp-block-paragraph">Note the reference point. Measuring from when the run was queued, rather than from its logical date, means scheduler backlog counts against the deadline. That&#8217;s usually what you want, because a run that sat in a queue for an hour is late to its consumers regardless of how fast it executed once it started.</p>



<p class="wp-block-paragraph">There&#8217;s a structural weakness here worth naming: any alert that lives inside the orchestrator dies with the orchestrator. If the scheduler is down, nothing evaluates your deadline and nothing notifies anyone. The freshness probe from earlier is the answer, and it needs to run somewhere else. A small VPS from a provider like Contabo or InterServer running Prometheus and Alertmanager, or a hosted option like Grafana Cloud, gives you a watcher outside the blast radius of the thing being watched. This is the single highest-value piece of monitoring most data teams are missing.</p>



<h2 class="wp-block-heading">Alerting without burning your on-call</h2>



<p class="wp-block-paragraph">The fastest way to make all of this worthless is to alert on every threshold crossing. Three rules keep it survivable.</p>



<ol class="wp-block-list">
<li><strong>Set thresholds from observed behaviour, not from wishes.</strong> Look at a month of actual freshness values and set <code>warn_after</code> a comfortable margin above the normal worst case. A source that habitually passes at eleven hours against a twelve-hour threshold is not healthy, it&#8217;s one upstream hiccup from paging you.</li>



<li><strong>Page on consumer impact, not internal events.</strong> A task retry is not an incident. A dataset breaching the freshness commitment its consumers rely on is. Route everything else to a channel someone reads in the morning.</li>



<li><strong>Use burn rate over multiple windows for SLOs.</strong> A short window catches fast breakage, a long window catches slow erosion, and requiring both to fire filters out the transient spikes that generate most false pages.</li>
</ol>



<p class="wp-block-paragraph">One more: alert on the absence of your own telemetry. A freshness gauge that stops updating is indistinguishable from one that&#8217;s fine, right up until someone asks about the numbers.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>Freshness alert fires but the data looks current.</strong> Almost always a timezone problem. Your <code>loaded_at</code> column is in local time, <code>now()</code> is in UTC, and the offset shows up as a constant bias in the age. Store load timestamps in UTC and cast explicitly at read time.</p>



<p class="wp-block-paragraph"><strong>Freshness flaps in and out of breach.</strong> Your check runs too close to the expected arrival. If loads land around six and your probe runs at five past, a fifteen-minute upstream delay produces an intermittent failure that trains everyone to ignore the alert. Move the probe later or widen the threshold.</p>



<p class="wp-block-paragraph"><strong>Every pipeline reports as one series.</strong> <code>honor_labels: true</code> is missing from the Pushgateway scrape config.</p>



<p class="wp-block-paragraph"><strong>A decommissioned pipeline still shows healthy.</strong> Stale group in Pushgateway. Delete the group and add a check on <code>push_time_seconds</code> so the next one surfaces on its own.</p>



<p class="wp-block-paragraph"><strong>Latency looks fine but consumers say data is late.</strong> You&#8217;re measuring run duration and they&#8217;re feeling end-to-end. Add queue wait and event-time-to-load and the gap will be obvious.</p>



<p class="wp-block-paragraph"><strong>Freshness passes, row count is near zero.</strong> The load timestamp advanced without meaningful data. This is a broken source or a stuck watermark, and it&#8217;s the reason the volume check belongs in the same probe.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating job success as the health signal and discovering staleness through a human.</li>



<li>Running the freshness check inside the same pipeline it&#8217;s meant to police.</li>



<li>Tracking freshness without volume, so a zero-row load reads as healthy.</li>



<li>Reporting latency without saying which two timestamps it spans.</li>



<li>Putting run IDs, batch IDs or timestamps into Prometheus labels, which multiplies your series count without bound. Those belong in traces or logs.</li>



<li>Alerting on task-level failures instead of run-level outcomes, then muting the whole channel a week later.</li>



<li>Assuming Pushgateway reflects current state rather than the last thing anyone pushed.</li>
</ul>



<h2 class="wp-block-heading">Best practices for data pipeline freshness monitoring</h2>



<ul class="wp-block-list">
<li>Start with your three most-used tables. Full coverage is a project; three tables is an afternoon and catches most of the pain.</li>



<li>Write down the freshness commitment per dataset in plain language, then encode it. &#8220;Yesterday&#8217;s orders are complete by 07:00&#8221; converts directly into a threshold.</li>



<li>Name metrics consistently across pipelines. A shared prefix and a stable label set is what makes one dashboard work for all of them.</li>



<li>Keep labels low-cardinality: pipeline, dataset, stage, environment, result. Nothing unbounded.</li>



<li>Emit a failure category alongside every failure so alerts route themselves.</li>



<li>Run the deadman check on infrastructure that doesn&#8217;t share a failure domain with the pipeline.</li>



<li>Link every alert to a runbook that names the owner and the first three things to check.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the difference between pipeline latency and data freshness?</h3>



<p class="wp-block-paragraph">Latency measures how long a specific batch of data took to travel from source to destination. Freshness measures how old the newest available record is right now, regardless of whether anything is currently running. A pipeline can have excellent latency and terrible freshness if it stopped being triggered.</p>



<h3 class="wp-block-heading">How often should freshness checks run?</h3>



<p class="wp-block-paragraph">Frequently enough that you find out before your consumers do. A useful rule is roughly a quarter of your tolerance window: if data may be up to four hours old, check hourly. Freshness probes are cheap single-aggregate queries, so the limiting factor is usually warehouse billing rather than load.</p>



<h3 class="wp-block-heading">Do I need a data observability platform for this?</h3>



<p class="wp-block-paragraph">Not to start. Freshness, latency and failure rate for a handful of critical datasets is a scheduled query, a push, and a few alert rules. Commercial platforms earn their cost when you need automatic column-level lineage, anomaly detection across hundreds of tables, or coverage of assets nobody has explicitly instrumented. That&#8217;s a real problem at scale, and a genuinely expensive one to build yourself. It&#8217;s just not the problem you have on day one.</p>



<h3 class="wp-block-heading">How do I monitor freshness for a table that only updates weekly?</h3>



<p class="wp-block-paragraph">Set the threshold from the schedule plus a delivery margin, and add a deadman rule so a missing metric alerts on its own. For genuinely static reference tables, disable freshness explicitly rather than leaving a check that always warns; in dbt that means setting freshness to null for the table.</p>



<h3 class="wp-block-heading">Should freshness checks fail the pipeline or just warn?</h3>



<p class="wp-block-paragraph">Both, in different places. Checking source freshness at the start of a job and failing hard prevents you building models on stale input, which is the cheapest bug to prevent and the most expensive to unwind. Checking output freshness after the fact should alert rather than fail, because the run is already over.</p>



<h3 class="wp-block-heading">What percentile should I use for a latency SLO?</h3>



<p class="wp-block-paragraph">p95 for most internal analytics pipelines, p99 where downstream systems make automated decisions on the data. Track the median separately for capacity planning. Never use the average as the SLO number, because it hides exactly the tail that produces complaints.</p>



<h3 class="wp-block-heading">Does this work for streaming pipelines too?</h3>



<p class="wp-block-paragraph">Yes, with different plumbing. Freshness becomes consumer lag measured in time, latency becomes event-time to availability-time, and failure rate becomes connector and task state plus dead-letter volume. The reasoning is identical; only the source of the numbers changes.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you remember one thing, make it this: measure the data, not the job. Run status, task counts and duration all describe your code. Only freshness describes what your consumers actually receive, and it&#8217;s the one signal that stays honest when the pipeline succeeds at doing nothing.</p>



<p class="wp-block-paragraph">Good data pipeline freshness monitoring is not a platform purchase. It&#8217;s a scheduled query that reports the age and volume of your most important tables, pushed somewhere durable, with a deadman rule so that silence is treated as a failure rather than as health. Add latency broken down by stage and failure rate categorised by reason, and you can answer &#8220;is the data good right now&#8221; without opening a single log file.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help instrumenting your pipelines?</h2>



<p class="wp-block-paragraph">Most of my consulting work in this area is retrofitting observability onto pipelines that already exist and can&#8217;t be paused. Specifically:</p>



<ul class="wp-block-list">
<li>Defining freshness and latency commitments per dataset, then translating them into thresholds and alert rules that hold up on-call.</li>



<li>Adding freshness and volume probes to existing warehouses without touching the pipelines themselves.</li>



<li>Wiring batch jobs into Prometheus through Pushgateway, including the grouping-key and stale-metric problems that bite six months later.</li>



<li>Migrating Airflow DAGs off the removed SLA feature onto Deadline Alerts, or onto external checks that survive a scheduler outage.</li>



<li>Building the Grafana dashboard that answers &#8220;is the data good right now&#8221; in one screen, with drill-down by stage and failure category.</li>



<li>Cutting alert noise on pipelines where the channel has already been muted, by moving from task-level events to consumer-impact SLOs.</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me a DAG file, a scrape config, or a screenshot of the dashboard you don&#8217;t trust, and I&#8217;ll tell you what I&#8217;d instrument first and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-freshness-monitoring/">Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</title>
		<link>https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 21 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Amazon Managed Grafana]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Metric Math]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[MWAA]]></category>
		<category><![CDATA[Notification Policies]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=260</guid>

					<description><![CDATA[<p>A Glue job that stops running emits no metrics, so the dashboard stays green and the alert quietly resolves itself. Here is why Grafana monitoring for AWS data pipelines misses that failure, and the heartbeat, metric math and no-data configuration that closes the gap, along with the CloudWatch query costs and IAM boundaries nobody warns you about.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/">Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on a Monday. An analyst pings you: &#8220;the dashboard is showing last week&#8217;s numbers.&#8221; You open Grafana, look at the pipeline board, and every panel is calm. No red. No spikes. Error counts flat at zero.</p>



<p class="wp-block-paragraph">Then you check the Glue job history and the last successful run was six days ago. The job has not failed. It has not run at all. Because it never ran, it never emitted a metric, so CloudWatch recorded nothing and Grafana drew nothing. The panel is not green because the pipeline is healthy. It is green because the pipeline is silent.</p>



<p class="wp-block-paragraph">That gap is the most common problem I see with <strong>Grafana monitoring for AWS data pipelines</strong>, and it is not a Grafana bug. It follows from how AWS batch services emit metrics and how alert engines read an empty result set. This post covers that failure and the three that follow close behind: wrong signal, cost, and access. Then the dashboard layout, the alerting rules, and a troubleshooting list for empty panels.</p>



<h2 class="wp-block-heading">Failure family one: the pipeline stopped and nothing turned red</h2>



<p class="wp-block-paragraph">Application metrics are continuous. A web server emits request counts every minute forever, so a gap in the line genuinely means something is wrong.</p>



<p class="wp-block-paragraph">Pipeline metrics are not like that. Glue reports job metrics only while a job executes, and only if job metrics are enabled and the script initialises a GlueContext. Step Functions publishes execution metrics when executions happen. MWAA publishes DAG metrics when DAGs run. Between runs the metric does not exist. A daily job produces a few minutes of data and twenty-three hours of nothing.</p>



<p class="wp-block-paragraph">So the question that decides whether your monitoring works is: what does your alert engine do with an empty result?</p>



<h3 class="wp-block-heading">No Data and MissingSeries are not the same thing</h3>



<ul class="wp-block-list">
<li><strong>No Data</strong> means the query returned no data points at all. Grafana handles this with a per-rule setting and by default raises a separate alert named <code>DatasourceNoData</code>. That alert does not inherit your rule&#8217;s labels and it fires immediately, skipping the pending period. If your notification policy routes on a team label, it will not match.</li>



<li><strong>MissingSeries</strong> means the query still returns data but one series has disappeared. That is the dead-job case exactly: a query for failed tasks across all Glue jobs still returns rows for the jobs that ran, and the dead one just stops existing. Grafana holds the last state for a couple of evaluation cycles, then evicts the instance with a transition to Normal, annotated with <code>grafana_state_reason</code> set to <code>MissingSeries</code>.</li>
</ul>



<p class="wp-block-paragraph">Read that again, because it is the whole problem. A job that stops running produces an alert that resolves itself and sends a cheerful &#8220;back to normal&#8221; notification. Nobody investigates a resolution.</p>



<h3 class="wp-block-heading">The fix: stop inferring health from absence</h3>



<p class="wp-block-paragraph">Do not build the &#8220;did it run&#8221; signal out of metrics that only exist when it runs. Publish a heartbeat: a custom metric your pipeline writes once, deliberately, at the point you consider the run complete.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-data 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions PipelineName=orders_daily 
  --value 1 
  --unit Count</code></pre>



<p class="wp-block-paragraph">Put that at the end of the Glue script, in the final Step Functions state, or in the last task of the DAG. Now &#8220;did it run&#8221; has a positive signal instead of an inferred one.</p>



<p class="wp-block-paragraph">The heartbeat is still sparse, so you also need to tell the evaluator that absence means failure. In a CloudWatch alarm, that is the missing-data setting: the options are <code>missing</code>, <code>ignore</code>, <code>breaching</code> and <code>notBreaching</code>, and for a heartbeat you want <code>breaching</code>.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name "orders_daily-heartbeat-missing" 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions Name=PipelineName,Value=orders_daily 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 26 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts</code></pre>



<p class="wp-block-paragraph">One subtlety here bites people. CloudWatch looks back over an evaluation range wider than your evaluation periods, and if it finds enough real data points in that wider range, the missing-data setting is ignored entirely. For a once-a-day metric that behaviour can suppress the alarm you just built. The reliable way around it is metric math: fill the gaps before the alarm evaluates anything.</p>



<pre class="wp-block-code"><code>FILL(m1, 0)</code></pre>



<p class="wp-block-paragraph">Now every period has a value, zero means &#8220;no run completed&#8221;, and a plain threshold behaves the way you expected. <code>FILL(m1, REPEAT)</code> and <code>FILL(m1, LINEAR)</code> carry the last known value or interpolate instead, which suits gauges better than heartbeats.</p>



<p class="wp-block-paragraph">On the Grafana side, set the no-data state explicitly on each rule and build a notification policy that catches <code>alertname=DatasourceNoData</code>, otherwise those alerts go nowhere. The heartbeat still matters even with that in place, because it turns a disappearing series into a present series with a bad value, and every rule engine handles that correctly.</p>



<h2 class="wp-block-heading">Failure family two: you are monitoring the compute, not the data</h2>



<p class="wp-block-paragraph">The second failure is subtler. The job runs, succeeds, takes its usual time, and writes nothing, because an upstream API returned an empty page or a partition filter matched no files. Executor CPU and heap will not tell you that. Two signals will.</p>



<p class="wp-block-paragraph"><strong>Freshness</strong> is time since the last successful completion, per pipeline. It maps directly onto &#8220;can I trust the report on my screen&#8221;, which is the only question your stakeholders are asking. Derive it from the heartbeat.</p>



<p class="wp-block-paragraph"><strong>Volume</strong> is rows or bytes written per run, compared against the same run last week. A drop to zero with a successful exit code is the classic empty-load bug. For Spark jobs on Glue the built-in counters give you a usable start:</p>



<ul class="wp-block-list">
<li><code>glue.driver.aggregate.recordsRead</code> and <code>glue.driver.aggregate.bytesRead</code> for input volume</li>



<li><code>glue.driver.aggregate.numFailedTasks</code> for partial failures that do not fail the job</li>



<li><code>glue.driver.s3.filesystem.write_bytes</code> and the <code>glue.ALL</code> equivalents for output across executors</li>
</ul>



<p class="wp-block-paragraph">Glue reports these roughly every thirty seconds as deltas from the previous report, so Sum over the run window is usually what you want rather than Average. Beyond those, the signals I put on a board first are <code>ExecutionsStarted</code> minus <code>ExecutionsSucceeded</code> from the <code>AWS/States</code> namespace, since aborted and timed-out executions land in separate counters and failure count alone misses them; DAG metrics from the <code>AmazonMWAA</code> namespace, remembering that <code>metrics.metrics_allow_list</code> and <code>metrics.metrics_block_list</code> control what reaches CloudWatch at all; consumer lag rather than throughput on streaming ingestion, because a stream will happily accept records nobody is reading; and message age on dead letter queues, which is a near-perfect proxy for &#8220;something is failing repeatedly and nobody noticed&#8221;.</p>



<h2 class="wp-block-heading">Failure family three: the dashboard costs more than the pipeline</h2>



<p class="wp-block-paragraph">Grafana&#8217;s CloudWatch data source uses <code>GetMetricData</code> to fetch samples and <code>ListMetrics</code> when you pick dimensions in the editor. <code>GetMetricData</code> does not qualify for the CloudWatch API free tier, and it is billed per metric requested rather than per API call. One request pulling five hundred metrics is five hundred billable metric requests.</p>



<p class="wp-block-paragraph">Now multiply: panels, times series per panel, times statistics per series, times refresh rate, times every user with the board open, times every alert rule on its own schedule. A wildcard query matching every Glue job in the account looks tidy in the editor and quietly becomes the largest line item in your monitoring bill. Asking for more than five statistics on one metric counts as an extra metric request too.</p>



<p class="wp-block-paragraph">The levers, roughly in order of return:</p>



<ol class="wp-block-list">
<li>Set a sane refresh interval. Metrics that arrive every thirty seconds do not need a five-second refresh, and a daily batch board does not need auto-refresh at all.</li>



<li>Replace wildcard dimension matching with explicit resource lists on always-open boards. Keep wildcards for ad hoc investigation.</li>



<li>Query only the regions you run in. It is easy to leave a data source pointed at a region you migrated out of.</li>



<li>Use template variables instead of one duplicated board per team.</li>



<li>If you pull a large metric catalogue continuously, evaluate CloudWatch metric streams. Streams push through Amazon Data Firehose to a destination you control, and AWS recommends that path for third-party tools with heavy <code>GetMetricData</code> usage. The trade-off is genuine: you lose metric math and Metrics Insights against CloudWatch directly, and you pick up storage and Firehose ingestion costs instead.</li>
</ol>



<p class="wp-block-paragraph">To find which client is driving the spend, enable CloudTrail data events on <code>GetMetricData</code>. It surfaces the calling IAM principal and source address, which is usually enough to identify the forgotten dashboard.</p>



<h2 class="wp-block-heading">Failure family four: access, accounts and regions</h2>



<p class="wp-block-paragraph">Grafana ships CloudWatch support natively, so there is no plugin to install. What there is, is an IAM policy people habitually make too broad or too narrow. This is the read-only set the data source needs:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarms",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetMetricStatistics"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingRegions",
      "Effect": "Allow",
      "Action": "ec2:DescribeRegions",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Prefer an assumed role over long-lived access keys. If Grafana runs on EC2 or ECS in the same account, the instance or task role is cleanest and there is no secret to rotate.</p>



<p class="wp-block-paragraph">Two boundaries to plan for. CloudWatch quotas are defined per account and per region, so a multi-account estate can hit limits in one account while the rest are fine, and each needs its own quota increase. And metrics do not cross regions: a pipeline that reads in one region and writes in another needs both on the board, and the missing half is a common reason a panel looks empty for no obvious cause.</p>



<h2 class="wp-block-heading">Where to run Grafana for this</h2>



<p class="wp-block-paragraph">Three options, and the choice is mostly about viewer count and who carries the operational load.</p>



<ul class="wp-block-list">
<li><strong>Amazon Managed Grafana</strong> bills per active user license per workspace, split into editor and viewer tiers, with at least one editor license required per workspace even if nobody logs in. An active user is anyone who logged in or made an API request during the billing cycle. Excellent for a small engineering team, expensive if you hand read-only boards to a large business audience.</li>



<li><strong>Grafana Cloud</strong> from Grafana Labs bills on usage rather than seats, which flips that calculation. It also brings the AWS observability integrations and Grafana Alloy if you want CloudWatch metrics in a Prometheus-compatible store.</li>



<li><strong>Self-hosted</strong> is cheapest per seat and most work. A small VPS from a provider like Contabo or InterServer runs Grafana for a handful of pipelines comfortably, and you own upgrades, TLS renewal and backups. One licensing note: Grafana&#8217;s open source core moved from Apache 2.0 to AGPLv3, while plugins and agents stayed Apache-licensed. If your legal team has a policy on AGPL, have that conversation before deployment.</li>
</ul>



<h2 class="wp-block-heading">Laying out a board people actually read</h2>



<p class="wp-block-paragraph">Organise by question, not by service. Whoever opens this board on a Monday morning has one question, and the answer should be visible without scrolling.</p>



<ol class="wp-block-list">
<li><strong>Freshness table.</strong> One row per pipeline, hours since last successful completion, threshold colour tied to that pipeline&#8217;s actual SLA. Most viewers read only this row.</li>



<li><strong>Volume against baseline.</strong> Records or bytes written per run, overlaid with the same period a week ago.</li>



<li><strong>Failure counts and durations</strong>, grouped by stage so a slow extract is distinguishable from a slow load.</li>



<li><strong>Resource detail at the bottom.</strong> Executor memory, DPU usage, throttling. Useful during an incident, noise the rest of the time.</li>
</ol>



<p class="wp-block-paragraph">Use a template variable for pipeline name rather than one board per pipeline. It keeps the query count down and new pipelines appear automatically. For cross-pipeline aggregation without fanning out into individual series, CloudWatch Metrics Insights runs a SQL-style query inside CloudWatch and the Grafana data source supports it:</p>



<pre class="wp-block-code"><code>SELECT SUM(ExecutionsFailed)
FROM SCHEMA("AWS/States", StateMachineArn)
GROUP BY StateMachineArn</code></pre>



<p class="wp-block-paragraph">Each <code>GetMetricData</code> operation can carry only one Metrics Insights query, so treat it as a way to consolidate one fan-out, not to run several at once.</p>



<h2 class="wp-block-heading">Troubleshooting: why the panel is empty</h2>



<ol class="wp-block-list">
<li><strong>Check the region on the query, not the data source.</strong> Grafana allows a per-query region override and it is easy to leave one on default.</li>



<li><strong>Widen the time range to seven days.</strong> If data appears, the metric is sparse and you have a heartbeat problem, not a query problem.</li>



<li><strong>Check the statistic.</strong> Sum and Average behave very differently on delta counters, and empty-looking panels are often Average over a window that is mostly gaps.</li>



<li><strong>Check dimensions match exactly.</strong> With Match Exact on, every dimension must be specified, so a metric published with two dimensions returns nothing when queried with one. Turning Match Exact off makes Grafana generate a search expression instead, which is the usual fix.</li>



<li><strong>Check job metrics are enabled.</strong> For Glue you need both the job metrics option and a GlueContext in the script. Without the GlueContext, nothing is emitted regardless of the console setting.</li>



<li><strong>Check IAM.</strong> A role with <code>ListMetrics</code> but not <code>GetMetricData</code> populates every dropdown perfectly and returns no data, which is confusing until you have seen it once.</li>



<li><strong>Check for throttling.</strong> Panels that are intermittently empty under load rather than consistently empty usually mean a CloudWatch API quota for that account and region.</li>
</ol>



<p class="wp-block-paragraph">For alert rules behaving oddly rather than panels, treat the Error state separately from No Data. Grafana raises <code>DatasourceError</code> on evaluation timeouts or repeated query failures, governed by the <code>evaluation_timeout</code> and <code>max_attempts</code> settings. Slow CloudWatch queries under quota pressure trip this, and the resulting alert looks nothing like the one you wrote.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Alerting on failure counts only. A job that never starts cannot fail, so failure-count alerting is blind to the most common outage.</li>



<li>Leaving no-data handling on the default and never routing <code>DatasourceNoData</code>. The alert fires into the void.</li>



<li>Treating a MissingSeries resolution as good news. If alerts resolve without anyone acting, check the reason annotation.</li>



<li>Importing a large community dashboard pack wholesale. Most of those panels use wildcards, and you have just subscribed to a permanent <code>GetMetricData</code> bill for metrics nobody reads.</li>



<li>Monitoring the orchestrator but not the destination. Step Functions reports a clean success while the load that ran inside it wrote zero rows.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Emit a heartbeat from every scheduled pipeline using one consistent namespace and dimension scheme. Consistency is what lets a single dashboard cover everything.</li>



<li>Write down a freshness SLA per pipeline, then set the threshold from it. &#8220;Alert if stale&#8221; only means something once someone says how stale is too stale.</li>



<li>Provision dashboards and alert rules as code in the same repository as the pipeline. A pipeline that ships without its monitoring is not finished.</li>



<li>Route by severity, not by service. Freshness breaches page; executor memory warnings go to a channel.</li>



<li>Break something on purpose to test the alert. Disable the schedule on a non-production pipeline and confirm you get paged. An untested alert is a guess.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a plugin to connect Grafana to CloudWatch?</h3>



<p class="wp-block-paragraph">No. Grafana includes native support for the Amazon CloudWatch data source. Configure credentials or an assumed role, set a default region, and query. The same data source also reaches CloudWatch Logs and X-Ray.</p>



<h3 class="wp-block-heading">Why does my Grafana alert resolve itself when the pipeline is still broken?</h3>



<p class="wp-block-paragraph">Almost certainly MissingSeries eviction. When one series in a multi-series query disappears, Grafana holds the last state briefly and then transitions the instance to Normal with <code>grafana_state_reason</code> set to <code>MissingSeries</code>. It is not treated as No Data. A heartbeat metric with gaps filled to zero stops the series disappearing in the first place.</p>



<h3 class="wp-block-heading">CloudWatch dashboards or Grafana for AWS data pipelines?</h3>



<p class="wp-block-paragraph">CloudWatch dashboards are the shorter path if everything sits in one account and the visualisation options suit you. Grafana earns its place when you need pipeline metrics correlated with a warehouse query, a database, a trace or a business KPI on one screen, and when alerting has to route to several destinations. Most data platforms end up wanting the correlation.</p>



<h3 class="wp-block-heading">How do I monitor a Glue job that succeeds but writes no data?</h3>



<p class="wp-block-paragraph">Alert on output volume rather than exit status. Use the Glue write byte counters, or better, emit your own row-count metric after the write so it reflects what your code believes it wrote. Compare against the equivalent run a week earlier, because fixed thresholds age badly as data grows.</p>



<h3 class="wp-block-heading">Does Grafana querying CloudWatch cost money?</h3>



<p class="wp-block-paragraph">Yes. Grafana uses <code>GetMetricData</code>, which is billed per metric requested and excluded from the CloudWatch API free tier. Cost scales with panels multiplied by series multiplied by refresh frequency multiplied by viewers, so wildcards on an auto-refreshing board are the expensive pattern. Check current rates on the CloudWatch pricing page and do the arithmetic for your own board.</p>



<h3 class="wp-block-heading">Amazon Managed Grafana or self-hosted?</h3>



<p class="wp-block-paragraph">It comes down to viewer count. Managed Grafana bills per active user per workspace, which suits a small engineering team and gets costly when read-only dashboards go out widely. Self-hosting removes the per-seat cost and adds upgrade, TLS and backup work. Grafana Cloud sits between them on usage-based billing. Count likely viewers first, then choose.</p>



<h3 class="wp-block-heading">What is the minimum useful monitoring for a new pipeline?</h3>



<p class="wp-block-paragraph">A heartbeat proving the run completed, a row or byte count proving it moved something, and an alert that fires when the heartbeat is missing for longer than the agreed freshness window. Everything else is refinement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Effective <strong>Grafana monitoring for AWS data pipelines</strong> is not about drawing more metrics. It is about refusing to infer health from absence. Batch pipelines are silent most of the time, and every monitoring system in existence reads silence as calm unless you tell it otherwise.</p>



<p class="wp-block-paragraph">Publish a heartbeat. Fill the gaps with zero. Set your no-data behaviour deliberately. Then alert on freshness and volume, because those are the two things people downstream will notice before you do. Do that, and a green panel means something.</p>



<h2 class="wp-block-heading">Need help building this on your AWS estate?</h2>



<p class="wp-block-paragraph">I design and build monitoring for AWS data platforms. This is the kind of work I take on:</p>



<ul class="wp-block-list">
<li>Heartbeat and freshness instrumentation across Glue, Step Functions, MWAA and Lambda pipelines, with one consistent namespace scheme so a single dashboard covers everything</li>



<li>Grafana dashboards built around freshness and volume rather than executor charts nobody reads</li>



<li>Alert rules and notification policies that handle No Data and MissingSeries correctly, routed to Slack, Teams, Telegram or email by severity</li>



<li>CloudWatch query cost reviews, tracing <code>GetMetricData</code> spend back to the dashboard or rule causing it</li>



<li>Cross-account and multi-region data source setup with least-privilege IAM roles instead of long-lived keys</li>



<li>Grafana deployment and hardening on Amazon Managed Grafana, Grafana Cloud or a self-hosted instance behind TLS</li>
</ul>



<p class="wp-block-paragraph">If you have a pipeline that failed quietly, send me the job definition, the alert rule, or a screenshot of the panel that stayed green. I would rather look at the actual thing than talk in generalities.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/">Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</title>
		<link>https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 15 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Anomaly Detection]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=225</guid>

					<description><![CDATA[<p>Your SaaS pipeline will fail far more often by succeeding at nothing than by throwing an exception, and every CloudWatch default treats an absent metric as a non-event. Here are the four signals worth alarming on: liveness, volume, freshness and shape, plus the missing-data traps that leave alarms permanently green.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/">CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said the Salesforce numbers looked wrong on the executive dashboard. I opened the pipeline. Every run for the past several days was green: the state machine succeeded, no failed states, no Lambda errors, no alarms in the account. The pipeline had been running flawlessly and writing an empty file every single time.</p>



<p class="wp-block-paragraph">That gap is the whole problem with CloudWatch data pipeline monitoring, and it is why most teams find out about broken SaaS integrations from a business user rather than from a page. A web server that breaks returns 500s, and your error-rate alarm fires. A pipeline that breaks frequently returns exit code 0, writes a zero-byte object, and reports success. The connected app got revoked, a scope was dropped during an admin cleanup, or a filter clause quietly started matching nothing. None of the AWS-native metrics have an opinion about any of that.</p>



<p class="wp-block-paragraph">This post is about what to measure instead: the four signals worth alarming on, the CloudWatch defaults that leave those alarms silently in the wrong state, how to emit custom metrics without turning your bill into a line item someone asks about, and what to check when a metric exists but the alarm still refuses to fire.</p>



<h2 class="wp-block-heading">&#8220;Did the job fail?&#8221; is the weakest question you can ask</h2>



<p class="wp-block-paragraph">Job status is a proxy for pipeline health, and a bad one. It tells you the orchestrator finished its instructions. It says nothing about whether those instructions produced data.</p>



<p class="wp-block-paragraph">SaaS sources make this worse than usual. A REST API returning HTTP 200 with an empty result array is, to your extraction code, a completely successful call. Pagination that stops on the first page because a cursor was reset is successful. An OAuth refresh that returns a token with fewer scopes than yesterday is successful right up until one object starts coming back empty. Every one of those is a green run.</p>



<p class="wp-block-paragraph">Build the monitoring around four questions the job outcome cannot answer:</p>



<ol class="wp-block-list"><li><strong>Liveness</strong> &#8211; did the run happen at all?</li><li><strong>Volume</strong> &#8211; did it move a plausible amount of data?</li><li><strong>Freshness</strong> &#8211; how old is the newest record that actually landed?</li><li><strong>Shape</strong> &#8211; are the fields and values still what downstream expects?</li></ol>



<p class="wp-block-paragraph">Each needs a different kind of metric, and each fails silently in its own way.</p>



<h2 class="wp-block-heading">Signal 1: liveness, and the missing-data trap that hides it</h2>



<p class="wp-block-paragraph">Step Functions publishes execution metrics to the <code>AWS/States</code> namespace, including <code>ExecutionsStarted</code>, <code>ExecutionsSucceeded</code>, <code>ExecutionsFailed</code>, <code>ExecutionsTimedOut</code>, <code>ExecutionsAborted</code> and <code>ExecutionTime</code>. The obvious move is an alarm on <code>ExecutionsFailed</code>. Most teams stop there, and that alarm is close to useless on its own.</p>



<p class="wp-block-paragraph">Here is why. Those metrics are published as a result of executions. If somebody disables the EventBridge schedule rule during an incident and forgets to re-enable it, no execution starts, so no data point is published for that dimension at all. Your alarm has nothing to evaluate.</p>



<p class="wp-block-paragraph">What happens next depends entirely on a parameter most people never set. CloudWatch alarms accept a <code>TreatMissingData</code> setting with four values: <code>breaching</code>, <code>notBreaching</code>, <code>ignore</code> and <code>missing</code>. The default is <code>missing</code>, which means the alarm keeps its current state when there is nothing to evaluate. An alarm that was sitting in OK stays in OK. Forever. It is green because it is blind, and there is no visual difference between the two.</p>



<p class="wp-block-paragraph">The fix is not to change <code>TreatMissingData</code> on the failure alarm. It is to add a separate heartbeat metric that your pipeline emits on completion, and alarm on the <em>absence</em> of that heartbeat:</p>



<pre class="wp-block-code"><code># Emitted at the very end of a successful run, per source system.
aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --value 1 
  --unit Count 
  --dimensions Source=salesforce,Stage=extract</code></pre>



<p class="wp-block-paragraph">Then the alarm that actually catches a dead scheduler:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name "ingest-salesforce-no-heartbeat" 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --dimensions Name=Source,Value=salesforce Name=Stage,Value=extract 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --datapoints-to-alarm 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts</code></pre>



<p class="wp-block-paragraph">The two flags doing the real work are <code>--comparison-operator LessThanThreshold</code> and <code>--treat-missing-data breaching</code>. The first catches a run that completes but emits nothing. The second catches a run that never happens at all. Without the second flag, silence reads as health.</p>



<p class="wp-block-paragraph">Set <code>--period</code> comfortably longer than your schedule interval so one slow run does not page anyone, and use <code>--datapoints-to-alarm</code> to require consecutive breaches. One caveat: <code>breaching</code> is right for a heartbeat you control, but a poor blanket default across the account, because plenty of AWS metrics are only emitted when non-zero. Throttling counters are legitimately sparse, and forcing them to breach on silence just makes noise.</p>



<h2 class="wp-block-heading">Signal 2: volume, and why a static threshold beats anomaly detection at first</h2>



<p class="wp-block-paragraph">A heartbeat proves the run happened. It does not prove the run did anything. Emit a row count, per source, per object:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RecordsWritten" 
  --value 14238 
  --unit Count 
  --dimensions Source=salesforce,Object=opportunity</code></pre>



<p class="wp-block-paragraph">Alarm on that being unexpectedly low rather than on it being zero. Zero-row alarms feel precise, but the more common failure is partial: a cursor that only advances through the first page, a rate limit that truncates the pull, an incremental sync that silently narrows its window. Those deliver a fraction of the expected rows, not none of them.</p>



<p class="wp-block-paragraph">There is a subtlety here that trips people. If a run fails before emitting <code>RecordsWritten</code> at all, a low-volume alarm on that metric sees missing data, not a zero. Metric math fixes it cleanly:</p>



<pre class="wp-block-code"><code>FILL(m1, 0)</code></pre>



<p class="wp-block-paragraph">Wrapping the metric in <code>FILL</code> with a value of 0 substitutes zero for every empty period, so an absent run and an empty run both breach the same alarm. You get one alert instead of two competing ones, and it means what you think it means.</p>



<p class="wp-block-paragraph">CloudWatch anomaly detection is the obvious upgrade for seasonal volume. It models expected behaviour and alarms on deviation from the band, which handles a Monday-morning spike that a static threshold cannot. I still would not reach for it first: it needs a stretch of representative history before the band is trustworthy, and during that stretch it will be confidently wrong. Start with a static floor well below your observed minimum, and move to anomaly detection once you know what normal looks like and can see the fixed number fighting you.</p>



<h2 class="wp-block-heading">Signal 3: freshness, the metric almost nobody emits</h2>



<p class="wp-block-paragraph">This is the one that separates a pipeline you can trust from one you merely watch. Liveness and volume can both look perfect while the data is stale, because a job that re-reads the same window every run produces rows, produces a heartbeat, and produces nothing new.</p>



<p class="wp-block-paragraph">The metric to emit is lag: the difference in seconds between now and the maximum source-side modification timestamp in the batch you just landed. Not your own processing timestamp, which will always look current. The source system&#8217;s timestamp, the one that tells you how far behind the truth you are.</p>



<pre class="wp-block-code"><code>import time
import boto3

cw = boto3.client("cloudwatch")

def emit_freshness(source, obj, max_source_ts_epoch):
    """max_source_ts_epoch: max LastModifiedDate in this batch, as epoch seconds."""
    lag_seconds = int(time.time()) - int(max_source_ts_epoch)
    cw.put_metric_data(
        Namespace="Pipelines/Ingest",
        MetricData=[{
            "MetricName": "SourceLagSeconds",
            "Value": lag_seconds,
            "Unit": "Seconds",
            "Dimensions": [
                {"Name": "Source", "Value": source},
                {"Name": "Object", "Value": obj},
            ],
        }],
    )</code></pre>



<p class="wp-block-paragraph">Alarm with <code>GreaterThanThreshold</code>, and set the threshold to roughly two schedule intervals plus your worst observed run duration. An hourly pipeline that takes twelve minutes on a bad day gets a threshold somewhere around two and a half hours. Tighter than that and you page on ordinary jitter.</p>



<p class="wp-block-paragraph">Freshness is also the one metric worth putting in front of non-engineers. &#8220;Your Salesforce data is currently 40 minutes behind&#8221; is a sentence a stakeholder can act on; &#8220;ExecutionsSucceeded is 1&#8221; is not. If you surface dashboards to a business audience through Grafana Cloud or QuickSight over the CloudWatch data source, lag is the panel that earns its space.</p>



<h2 class="wp-block-heading">Signal 4: shape, using logs instead of counters</h2>



<p class="wp-block-paragraph">Shape problems are schema drift and value drift. A SaaS admin renames a custom field, a picklist gains a value your mapping does not handle, a required field starts arriving null for a subset of records. The rows keep flowing. The row count looks fine. Downstream joins quietly drop records.</p>



<p class="wp-block-paragraph">Counters alone are the wrong shape here, because what you need at 2am is not &#8220;the null rate rose&#8221;, it is &#8220;which records, and what changed&#8221;. Embedded Metric Format is built for exactly that split: you write one structured JSON log line, CloudWatch extracts the numbers as real metrics you can alarm on, and the full event stays queryable in Logs Insights for the investigation.</p>



<pre class="wp-block-code"><code>{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "Pipelines/Quality",
        "Dimensions": [["Source", "Object"]],
        "Metrics": [
          { "Name": "NullRequiredFields", "Unit": "Count" },
          { "Name": "UnmappedFields", "Unit": "Count" }
        ]
      }
    ]
  },
  "Source": "salesforce",
  "Object": "opportunity",
  "NullRequiredFields": 412,
  "UnmappedFields": 2,
  "unmapped_field_names": ["Renewal_Tier__c", "Partner_Code__c"],
  "batch_id": "2f8c1a9e"
}</code></pre>



<p class="wp-block-paragraph">Two things to get right. Every name listed in <code>Dimensions</code> must also exist as a top-level property in the same JSON object, or extraction silently produces nothing. And note that <code>unmapped_field_names</code> and <code>batch_id</code> are properties, not dimensions: they ride along in the log event for querying but never become metric dimensions, which is exactly what you want.</p>



<p class="wp-block-paragraph">When the alarm fires, the investigation is a Logs Insights query against the same log group:</p>



<pre class="wp-block-code"><code>fields @timestamp, Object, UnmappedFields, unmapped_field_names
| filter Source = "salesforce" and UnmappedFields &gt; 0
| sort @timestamp desc
| limit 50</code></pre>



<p class="wp-block-paragraph">That hands you the new field names in seconds, which is the difference between a five-minute fix and an hour of guessing.</p>



<h2 class="wp-block-heading">Keeping CloudWatch data pipeline monitoring off the top of your bill</h2>



<p class="wp-block-paragraph">Custom metrics and log ingestion are consistently the two largest CloudWatch line items for data teams. Neither has a flat fee, so the cost is a direct function of choices you make in code. Three matter far more than the rest.</p>



<h3 class="wp-block-heading">Cardinality is the cost lever</h3>



<p class="wp-block-paragraph">A custom metric is billed per unique combination of namespace, metric name and dimension values. <code>Source</code> with five values and <code>Object</code> with twenty gives you a hundred metrics per metric name. Add <code>Tenant</code> with two hundred values and you have twenty thousand. Multiply by four metric names and the number stops being a rounding error.</p>



<p class="wp-block-paragraph">Never put a run ID, batch ID, record ID or raw error string in a dimension. Those belong in the log event. Dimensions are for things you would actually build an alarm on, and you are not going to build an alarm per batch.</p>



<h3 class="wp-block-heading">EMF over PutMetricData for anything high-frequency</h3>



<p class="wp-block-paragraph"><code>PutMetricData</code> is a synchronous API call on your critical path. In Lambda that means duration you pay for, plus a request that can throttle and needs handling. EMF avoids both: you write JSON to stdout, CloudWatch Logs extracts the metrics asynchronously, your function returns. For a per-run summary either is fine. For anything per batch or per record, EMF is the one I reach for.</p>



<h3 class="wp-block-heading">The Infrequent Access log class will break this silently</h3>



<p class="wp-block-paragraph">This one has caught competent people. CloudWatch Logs offers a Standard and an Infrequent Access log class, and IA has a lower ingestion price, which makes it an obvious target during a cost review. According to the AWS documentation, the IA class does not support embedded metric format, metric filters or subscription filters. Logs Insights queries still work.</p>



<p class="wp-block-paragraph">So if somebody moves your pipeline log group to IA to save money, every EMF-derived metric stops being extracted. The alarms built on those metrics go to <code>INSUFFICIENT_DATA</code>, or worse, sit in OK if they use the default missing-data behaviour. The logs still arrive. The dashboard still exists. The metrics are just gone.</p>



<p class="wp-block-paragraph">Log class cannot be changed after a log group is created, so recovery means a new Standard log group and redirecting output to it. Keep metric-backing log groups on Standard and point IA at the genuinely archival stuff: verbose debug output, raw payload dumps, anything you only read during a forensic investigation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Routing: one page, not four</h2>



<p class="wp-block-paragraph">Build all four signals and one broken pipeline fires the heartbeat, volume and freshness alarms within minutes of each other. Three pages, one problem. That is how alert fatigue starts.</p>



<p class="wp-block-paragraph">Composite alarms fix it. Build one composite alarm per pipeline whose rule combines the child alarms with OR, put the notification action there, and leave the child alarms with no actions at all. They become diagnostic detail you read after the page rather than three separate interruptions.</p>



<p class="wp-block-paragraph">SNS to email is fine for one person and poor past that, since it has no deduplication, escalation or acknowledgement. With more than one person on call, route through something with a schedule such as PagerDuty, Opsgenie or Better Stack. If your team already lives in Grafana Cloud or Datadog for application monitoring, pulling CloudWatch alarm state in through their AWS integration beats maintaining two alerting surfaces.</p>



<h2 class="wp-block-heading">Troubleshooting: the metric exists but the alarm never fires</h2>



<p class="wp-block-paragraph">Roughly in order of how often I have seen each one.</p>



<ul class="wp-block-list"><li><strong>Dimension mismatch.</strong> The alarm&#8217;s dimension set must match the published metric&#8217;s dimension set exactly, all of them, in the same values. A metric published with <code>Source</code> and <code>Object</code> is a different metric from one published with only <code>Source</code>. Run <code>aws cloudwatch list-metrics --namespace "Pipelines/Ingest"</code> and compare the output against your alarm definition character by character.</li><li><strong>Alarm stuck in INSUFFICIENT_DATA.</strong> The metric is not being published at all for that dimension set, or the period is shorter than the publishing interval so most periods are empty. Lengthen the period to at least the emission interval.</li><li><strong>Alarm stuck in OK during a real outage.</strong> Classic default <code>missing</code> behaviour. Any alarm whose purpose is to detect absence needs <code>--treat-missing-data breaching</code>.</li><li><strong>EMF metrics never appear.</strong> Check the log group&#8217;s class first, then confirm every dimension name in the <code>Dimensions</code> array exists as a top-level property in the same JSON object, then confirm the JSON is a single line with no wrapping prefix from your logger.</li><li><strong>Freshness spikes on a schedule.</strong> Almost always a timezone mismatch between how the SaaS API reports modification timestamps and how you convert them. Normalise everything to UTC epoch seconds at the boundary and stop converting anywhere else.</li><li><strong>Row counts double intermittently.</strong> Retries emitting the metric twice for one logical run. Emit volume metrics once, after the write is confirmed, not inside the retried block.</li></ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Alarming only on job failure, which misses every silent-success failure mode.</li><li>Leaving <code>TreatMissingData</code> at its default on alarms designed to detect absence.</li><li>Alarming on zero rows instead of on unexpectedly low rows, so partial extractions slip through.</li><li>Emitting a processing timestamp as the freshness metric, which is always current and therefore always useless.</li><li>Putting high-cardinality identifiers into metric dimensions and then being surprised by the bill.</li><li>Moving pipeline log groups to the Infrequent Access class without checking what depends on metric extraction.</li><li>Wiring notification actions to every child alarm instead of to one composite alarm per pipeline.</li><li>Building dashboards nobody looks at instead of alarms that reach a person.</li></ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list"><li>Emit liveness, volume, freshness and a shape metric for every source. Four metrics per source is a small amount of code and covers most of what actually breaks.</li><li>Define alarms in Terraform or CloudFormation alongside the pipeline, so a new source cannot ship without monitoring.</li><li>Use consistent dimension names across every pipeline. <code>Source</code> and <code>Object</code> everywhere makes search expressions and dashboards trivial; a mix of <code>source</code>, <code>system</code> and <code>connector</code> makes them impossible.</li><li>Set alarm descriptions to a one-line runbook. The person paged at 3am should not have to guess what the alarm means.</li><li>Test the absence case deliberately. Disable the schedule in a non-production account and confirm the heartbeat alarm fires. An untested absence alarm is an assumption.</li><li>Set explicit log retention on every pipeline log group. Never-expire is the default and it compounds quietly.</li><li>Keep an eye on the source side too, not just yours. Most SaaS APIs publish their own rate limit or API usage counters, and pulling those into a metric is often the earliest possible warning.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is CloudWatch enough on its own for data pipeline monitoring?</h3>



<p class="wp-block-paragraph">For alerting, yes, if your workloads are on AWS. It has metrics, alarms, composite alarms and log querying, and it is already collecting service metrics you would otherwise have to ship somewhere. Where it gets weak is exploratory analysis and cross-source correlation, which is where teams add Grafana, Datadog or a dedicated data-observability tool on top. Run CloudWatch for the paging path first and add the analysis layer when you have an actual gap.</p>



<h3 class="wp-block-heading">How do I monitor a pipeline that runs on a managed service like Glue or AppFlow?</h3>



<p class="wp-block-paragraph">The same four signals apply; only the emission point changes. Managed services publish their own metrics, but those describe the service&#8217;s execution, not your data. Where you cannot add code inside the job, hook the completion event through EventBridge to a small Lambda function that inspects what landed and emits the volume and freshness metrics. It is a handful of lines and it is the only way to get data-level signals out of a job you do not control.</p>



<h3 class="wp-block-heading">Should each tenant get its own metric in a multi-tenant pipeline?</h3>



<p class="wp-block-paragraph">Usually not as a dimension. Tenant counts grow, and cardinality is billed per unique dimension combination. Emit aggregate metrics with <code>Source</code> and <code>Object</code> dimensions, put the tenant identifier in the log event as a property, and use Logs Insights to slice by tenant when you need to. Break out a dimension per tenant only for the small number of accounts where a contractual SLA makes it worth the cost.</p>



<h3 class="wp-block-heading">What is a sensible freshness threshold?</h3>



<p class="wp-block-paragraph">Start at two schedule intervals plus your worst observed run duration, then adjust after watching real data for a couple of weeks. The threshold that matters is the one the business would notice, so if a stakeholder can tolerate half a day of staleness on a nightly export, do not page at four hours. Match the alarm to the actual consequence.</p>



<h3 class="wp-block-heading">Why does my alarm sit in INSUFFICIENT_DATA instead of ALARM?</h3>



<p class="wp-block-paragraph">CloudWatch has no data points to evaluate for that metric and dimension set. Either nothing is publishing, the dimensions do not match what is published, or the alarm period is shorter than the interval at which you emit. Confirm the metric exists with <code>list-metrics</code> first, and only then look at the alarm configuration.</p>



<h3 class="wp-block-heading">How much does this cost to run?</h3>



<p class="wp-block-paragraph">It depends on three things you control: the number of unique custom metrics, the volume of log data ingested, and how much data your Logs Insights queries scan. Alarms and dashboards are minor by comparison. Keep dimension cardinality low, set retention deliberately, and query narrow time windows, and the monitoring stays well below the cost of the pipeline it protects. Check current rates on the AWS pricing page rather than trusting any number in a blog post, including this one.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Effective CloudWatch data pipeline monitoring is built around absence, not around errors. Your SaaS integrations will fail far more often by succeeding at nothing than by throwing an exception, and every default in CloudWatch treats an absent metric as a non-event.</p>



<p class="wp-block-paragraph">If you only change one thing after reading this, add a heartbeat metric with <code>--treat-missing-data breaching</code> to your most important pipeline, then go and disable its schedule in a test account and confirm the alarm actually fires. If it does not, you have just found out on your own terms instead of from a stakeholder asking why the numbers look wrong.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your pipeline monitoring?</h2>



<p class="wp-block-paragraph">I work with teams running SaaS-to-warehouse pipelines on AWS, usually at the point where something broke quietly and nobody wants that to happen twice. Things I can help with here:</p>



<ul class="wp-block-list"><li>Auditing existing CloudWatch alarms for silent-failure gaps, particularly missing-data configuration and dimension mismatches that leave alarms permanently green.</li><li>Instrumenting liveness, volume, freshness and data-quality metrics into pipelines built on Lambda, Glue, Step Functions or self-hosted schedulers.</li><li>Designing EMF logging and metric namespaces that stay useful as source and object counts grow, without cardinality running away.</li><li>Building composite alarms and routing so one broken pipeline produces one actionable page rather than a cluster of them.</li><li>Cutting CloudWatch spend on log ingestion and custom metrics without losing the signals you actually rely on.</li><li>Writing Terraform modules that ship monitoring alongside every new source connector, so coverage is structural rather than remembered.</li></ul>



<p class="wp-block-paragraph">If you have a pipeline that has gone quiet, send me the alarm configuration and a Logs Insights output from the run in question and I will tell you what it is missing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/">CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Data Pipeline Operations Dashboard That Catches Silent Failures</title>
		<link>https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 15 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[OpenTelemetry]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[PromQL]]></category>
		<category><![CDATA[Pushgateway]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[StatsD]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=227</guid>

					<description><![CDATA[<p>Green DAGs and wrong numbers are the same morning. This is how to build a data pipeline operations dashboard around the four questions that actually matter, where each signal comes from, and how to turn the panels into alerts that fire before your users notice.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/">Building a Data Pipeline Operations Dashboard That Catches Silent Failures</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually lands around nine in the morning, from someone in finance or ops: &#8220;Is the revenue dashboard broken, or is it just late?&#8221;</p>



<p class="wp-block-paragraph">So you open the orchestrator. Every task is green. The run started on schedule, finished inside its usual window, exit code zero across the board. Nothing paged. And the numbers are still wrong.</p>



<p class="wp-block-paragraph">That gap is the whole reason to build a data pipeline operations dashboard. Not to watch DAGs go green, which the orchestrator UI already does perfectly well, but to answer the questions the orchestrator structurally cannot answer. This post covers the four signals worth putting on a screen, where each one actually comes from, how to wire them into Prometheus and Grafana without inventing a metrics platform, and how to turn the panels into alerts that fire before a human notices.</p>



<h2 class="wp-block-heading">Why green pipelines still ship bad data</h2>



<p class="wp-block-paragraph">An orchestrator tracks process outcomes. Did the Python exit cleanly, did the SQL statement return without an error, did the container terminate zero. That is a real signal and you should keep it. It is just answering a different question than the one your users care about.</p>



<p class="wp-block-paragraph">Consider what a perfectly successful run looks like when the upstream source has quietly changed:</p>



<ul class="wp-block-list">
<li>An API starts paginating differently and your extractor pulls the first page only. It ran fine. It loaded 500 rows instead of 500,000.</li>
<li>A source column gets renamed and your transform selects it with a <code>coalesce</code> default. It ran fine. Every row now says &#8220;unknown&#8221;.</li>
<li>A partition arrives late, so the incremental filter finds nothing to process. It ran fine. It did nothing.</li>
<li>An upstream team backfills, and your dedupe key no longer holds. It ran fine. Everything is doubled.</li>
</ul>



<p class="wp-block-paragraph">None of those raise an exception. All four are visible in under a second if you are tracking row counts and freshness at the table level. This is the failure family that eats the most incident time, because the clock does not start when the pipeline breaks. It starts when a human notices, which can be days later, and by then the bad data has been copied into reports, cached in a BI extract, and quoted in a meeting.</p>



<h2 class="wp-block-heading">The four questions a data pipeline operations dashboard has to answer</h2>



<p class="wp-block-paragraph">Resist the instinct to start from what your tools can emit. Start from the questions you ask during an incident, in the order you ask them. Everything that does not answer one of these is a second-page panel.</p>



<h3 class="wp-block-heading">Did it run at all?</h3>



<p class="wp-block-paragraph">Sounds trivial, and it is the one most homegrown dashboards get wrong. A panel showing &#8220;last run: success&#8221; does not distinguish between a pipeline that succeeded ten minutes ago and one that succeeded eleven days ago and has not been scheduled since. You want time since last successful completion, as a number that grows, not a status pill that sits there looking calm.</p>



<h3 class="wp-block-heading">Did it finish in time?</h3>



<p class="wp-block-paragraph">Duration matters less than duration against a deadline. A load that takes forty minutes is fine if the report is read at 9am and the run starts at 6am. The same forty minutes is an incident if someone moved the schedule to 8:40. Put the deadline on the panel as a threshold line, not in a runbook. Trending duration also catches slow decay: a stage that creeps from five minutes to fifty usually means data growth, a bad query plan, or resource contention, and it will breach your window eventually.</p>



<h3 class="wp-block-heading">Did the right amount of data arrive?</h3>



<p class="wp-block-paragraph">Rows in versus rows out, per stage. This is the highest value panel per unit of effort on the entire dashboard. A drop from a million rows to twenty thousand is not subtle once you plot it, but it is completely invisible in an exit code. Compare against the same weekday from the previous week rather than against yesterday, because most business data has a weekly shape and Monday-versus-Sunday comparisons generate noise you will learn to ignore.</p>



<h3 class="wp-block-heading">Is the data still the shape you expect?</h3>



<p class="wp-block-paragraph">Null rates, distinct counts on key columns, and schema changes. This is the panel almost nobody builds, and it is where the expensive failures live. A gradual rise in null rate on a joining key means an upstream system is degrading and your joins are silently dropping rows. You do not need statistical anomaly detection to catch that. You need the number plotted over time next to the others.</p>



<p class="wp-block-paragraph">There is a fifth question that belongs on a different surface: what breaks downstream when this table is wrong. That is lineage, and it answers blast radius rather than health. Keep it one click away, not on the operations screen.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where each signal actually comes from</h2>



<p class="wp-block-paragraph">Three sources, three different collection mechanisms. This is the part that takes an afternoon rather than a sprint, provided you do not try to build a unified agent.</p>



<h3 class="wp-block-heading">The orchestrator, for run and timing signals</h3>



<p class="wp-block-paragraph">Airflow has emitted StatsD metrics for a long time, and that path is still the one most deployments run. Airflow speaks StatsD, so you put a <code>statsd_exporter</code> in front of it to translate into Prometheus format. Enable it in the metrics section of your config, or with the equivalent <code>AIRFLOW__METRICS__</code> environment variables if you are on containers:</p>



<pre class="wp-block-code"><code>[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 9125
statsd_prefix = airflow
metrics_allow_list = scheduler,executor,dagrun,pool,triggerer</code></pre>



<p class="wp-block-paragraph">Two things worth knowing before you roll this out. First, <code>metrics_allow_list</code> is not optional at any real scale. Airflow emits per-DAG and per-task metric names, so without a prefix filter your cardinality grows with every DAG anyone adds, and you find out when the time series database starts struggling. Second, Airflow expects the StatsD destination to be resolvable at startup, so a scheduler that boots before the exporter can fail to come up. Order your dependencies accordingly.</p>



<p class="wp-block-paragraph">StatsD has no concept of labels, so the exporter has to reconstruct them from the dotted metric name. That mapping file is the fiddly part:</p>



<pre class="wp-block-code"><code>mappings:
  - match: "*.dag.*.*.duration"
    match_metric_type: observer
    name: "airflow_task_duration"
    labels:
      airflow_id: "$1"
      dag_id: "$2"
      task_id: "$3"</code></pre>



<p class="wp-block-paragraph">The wildcards map positionally onto <code>$1</code>, <code>$2</code>, <code>$3</code>, and <code>match_metric_type</code> has to match what Airflow actually sends, timer versus counter versus gauge. Get that wrong and the metric silently never appears at the exporter, which is a genuinely annoying half hour of debugging. Check the exporter&#8217;s own metrics endpoint before you go looking at Prometheus, because if it is not there it never left Airflow.</p>



<p class="wp-block-paragraph">Airflow also has a native OpenTelemetry path for both metrics and traces, which skips the StatsD translation entirely and gives you real labels rather than reconstructed ones:</p>



<pre class="wp-block-code"><code>[metrics]
otel_on = True
otel_host = otel-collector
otel_port = 4318
otel_prefix = airflow
otel_interval_milliseconds = 30000</code></pre>



<p class="wp-block-paragraph">The port there is the collector&#8217;s OTLP HTTP receiver, and you need an OpenTelemetry Collector or compatible endpoint in the path. If you are standing up a new stack, this is the one I would reach for, because the label problem disappears and traces give you per-task spans under a DAG run parent span, which is worth a lot during a slow-pipeline investigation. If you already have working StatsD mappings and dashboards built on them, migrating is real work for a modest gain, and the honest answer is that it can wait. Metric names do shift between Airflow major versions either way, so pin your dashboard queries to the metrics reference for the version you actually run.</p>



<p class="wp-block-paragraph">Managed Airflow changes the plumbing but not the shape. On Amazon MWAA the metrics land in CloudWatch instead, and Google Cloud Composer and Astronomer each have their own export path. The four questions do not change.</p>



<h3 class="wp-block-heading">The transformation layer, for data-level signals</h3>



<p class="wp-block-paragraph">If you use dbt, most of what you need is already being written to disk and thrown away. Every invocation produces <code>run_results.json</code> in the target directory, containing per-node status and timing, and <code>dbt source freshness</code> produces its own <code>sources.json</code>. The <code>manifest.json</code> holds the dependency graph, which is your lineage.</p>



<p class="wp-block-paragraph">The cheapest useful thing you can do is parse the failures out at the end of the run:</p>



<pre class="wp-block-code"><code>jq -r '.results[]
  | select(.status != "success")
  | [.unique_id, .status, (.execution_time | tostring)]
  | @tsv' target/run_results.json</code></pre>



<p class="wp-block-paragraph">One trap if you run dbt in ephemeral Kubernetes pods: the target directory dies with the pod. Upload the artifacts to object storage as the last step inside the same task that ran dbt, not as a downstream task, or you will find your history is empty exactly when you want it.</p>



<p class="wp-block-paragraph">Elementary is the low-friction option here. It installs as a dbt package with an on-run-end hook that loads the artifacts into your warehouse as ordinary tables, which means your dashboard can query test results and model timings in SQL alongside everything else. Great Expectations and Soda cover the same validation ground with more expressive checks and more setup. The commercial platforms, Monte Carlo and the vendors around it, add automated anomaly detection and column-level lineage across the whole stack, and the genuine case for them is that they find the checks you did not think to write. The trade-off is that you are paying per monitored table for statistical inference, and on a stack of twenty tables that inference is not telling you much that a row-count panel and a null-rate panel would not.</p>



<p class="wp-block-paragraph">Without dbt, you write the check yourself and it is still short. One query per critical table, run on a schedule:</p>



<pre class="wp-block-code"><code>SELECT
  COUNT(*)                                        AS row_count,
  MAX(updated_at)                                 AS last_event,
  SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) AS null_keys
FROM analytics.orders
WHERE loaded_at &gt;= CURRENT_DATE - 1;</code></pre>



<h3 class="wp-block-heading">Batch jobs, and the Pushgateway caveat</h3>



<p class="wp-block-paragraph">Prometheus pulls. Batch jobs finish and vanish long before a scrape interval comes round, so there is nothing to pull from. The Pushgateway exists for exactly this: short-lived service-level jobs push their results to it, and Prometheus scrapes the gateway instead.</p>



<pre class="wp-block-code"><code>cat &lt;&lt;EOF | curl --data-binary @- 
  http://pushgateway:9091/metrics/job/orders_sync
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $(date +%s)
# TYPE pipeline_rows_loaded gauge
pipeline_rows_loaded ${ROWS}
EOF</code></pre>



<p class="wp-block-paragraph">Now the caveat, which is the single most common way a pipeline dashboard lies to you. The Pushgateway is a cache, not an aggregator, and it never forgets. Push a success metric, then have the job stop running entirely, and that success sits there being scraped forever. Your dashboard stays green because the last known value is green, and the fact that nothing has updated it in six days is not something a status panel can see.</p>



<p class="wp-block-paragraph">Two defences, and you want both. First, always push a timestamp gauge rather than a boolean status, so staleness is arithmetic rather than inference. Second, alert on absence as well as on value, because the Pushgateway also exposes <code>push_time_seconds</code> per group, which tells you when anything last arrived regardless of what it said. Prometheus&#8217;s usual safety net, the <code>up</code> metric going to zero when a target disappears, does not apply here: Prometheus is scraping the gateway successfully, and the gateway is happily serving stale data.</p>



<p class="wp-block-paragraph">Also keep instance identifiers out of the grouping labels. Push per service, not per worker, or you accumulate a permanent time series for every container that ever ran.</p>



<h2 class="wp-block-heading">Laying out the dashboard so it works during an incident</h2>



<p class="wp-block-paragraph">The layout constraint is that someone is reading this at 7am on a phone with a stakeholder waiting. Three rows, top to bottom, in the order you need them:</p>



<ol class="wp-block-list">
<li><strong>Freshness across everything.</strong> One panel, one row per critical table, showing time since last successful load against its threshold. If this row is clean, the pipelines are fine and the problem is elsewhere. That is a genuinely useful thing to establish in five seconds.</li>
<li><strong>Volume and duration per pipeline.</strong> Row counts with a week-over-week comparison, and stage duration with the deadline drawn on. This is where you localise a problem to a specific pipeline and stage.</li>
<li><strong>Data quality detail.</strong> Null rates, failed test counts, schema change events. This is where you work out what actually went wrong once you know where.</li>
</ol>



<p class="wp-block-paragraph">The freshness query in PromQL, given the timestamp gauge above:</p>



<pre class="wp-block-code"><code>time() - max by (job) (pipeline_last_success_timestamp_seconds)</code></pre>



<p class="wp-block-paragraph">Grafana renders that nicely as a table with per-row thresholds, which is what makes the top row scannable. If you are self-hosting, Prometheus plus Grafana on a modest VPS from Hetzner, InterServer or DigitalOcean handles a surprising amount of pipeline metrics, because pipeline telemetry is low-volume compared to application metrics. Grafana Cloud&#8217;s free tier is a reasonable alternative if you do not want to own the storage, and Datadog or New Relic make sense if your organisation is already paying for one of them and you would rather have one pane than a better one.</p>



<p class="wp-block-paragraph">Keep it to one screen. Every panel you add past that dilutes the ones that matter, and a dashboard nobody can read at a glance is a dashboard nobody opens.</p>



<h2 class="wp-block-heading">Turning panels into alerts</h2>



<p class="wp-block-paragraph">A dashboard that requires someone to look at it is a dashboard that finds problems the morning after. The freshness panel converts into an alert almost directly:</p>



<pre class="wp-block-code"><code>groups:
  - name: pipelines
    rules:
      - alert: PipelineStale
        expr: time() - max by (job) (pipeline_last_success_timestamp_seconds) &gt; 5400
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} has not completed successfully in over 90 minutes"

      - alert: PipelineMetricsMissing
        expr: absent(pipeline_last_success_timestamp_seconds{job="orders_sync"})
        for: 30m
        labels:
          severity: page
        annotations:
          summary: "No metrics received from orders_sync"</code></pre>



<p class="wp-block-paragraph">The first catches a pipeline that is late or broken. The second catches the one that stopped reporting entirely, which the first cannot see, because an expression over a series that no longer exists returns nothing and an empty result does not fire. That pairing is the whole trick, and it applies to every push-based metric you collect.</p>



<p class="wp-block-paragraph">Set the threshold from the deadline the data has to meet, not from how long the job usually takes. Ninety minutes is not &#8220;the run takes an hour so give it some room&#8221;. It should be the point past which someone downstream is materially affected.</p>



<h2 class="wp-block-heading">Troubleshooting a dashboard that shows nothing useful</h2>



<p class="wp-block-paragraph"><strong>The metric never appears in Prometheus.</strong> Work backwards along the path, one hop at a time. Curl the statsd_exporter&#8217;s own metrics endpoint first. If the metric is not there, it never left the orchestrator, so the problem is the config or an allow-list that filtered it out. If it is there but not in Prometheus, the problem is your scrape config. Do not start in Grafana.</p>



<p class="wp-block-paragraph"><strong>The metric appears with no labels, or with the DAG name baked into the metric name.</strong> Your statsd_exporter mapping did not match. Check the wildcard count against the actual dotted name, and check <code>match_metric_type</code>, which is the usual culprit.</p>



<p class="wp-block-paragraph"><strong>Everything is green but the data is old.</strong> Classic Pushgateway staleness. Query <code>push_time_seconds</code> for the affected group and compare it to now. If it has not moved, the job is not running and you are looking at a cached value.</p>



<p class="wp-block-paragraph"><strong>Volume alerts fire every Monday.</strong> You are comparing against the previous day on data with a weekly cycle. Compare against the same day last week using an offset, or exclude weekends from the baseline.</p>



<p class="wp-block-paragraph"><strong>Gaps in the duration graph.</strong> If the metric is a timer sent only on completion, a run that is still going or that failed early sends nothing. That is correct behaviour, and it is why liveness needs its own panel rather than being inferred from the duration series.</p>



<p class="wp-block-paragraph"><strong>Prometheus memory climbing after rollout.</strong> Cardinality. Per-DAG and per-task metrics multiply, and any label carrying a run identifier, timestamp or UUID creates a permanent new series. Tighten the allow list and drop the offending labels.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating job success as data correctness. This is the entire problem, and every other mistake is a variation of it.</li>
<li>Pushing a boolean status instead of a timestamp, so staleness becomes invisible.</li>
<li>Alerting on threshold breaches but never on metric absence.</li>
<li>Building the dashboard before agreeing what &#8220;healthy&#8221; means for each table with the people who consume it.</li>
<li>Instrumenting every pipeline at once instead of the three that matter, then abandoning the project when the noise gets unmanageable.</li>
<li>Putting instance or pod identifiers into Pushgateway grouping labels and accumulating dead series forever.</li>
<li>Alerting on things nobody will act on at 3am. An alert that gets acknowledged and ignored twice has trained your team to ignore it permanently.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Baseline before you threshold. Collect for two or three weeks and look at the actual shape of the data before deciding what abnormal means.</li>
<li>Start with freshness on your three most critical tables. It catches more real problems per line of code than anything else you can instrument.</li>
<li>Define freshness thresholds from the consumer&#8217;s deadline, and write the consumer&#8217;s name next to the threshold so the number has an owner.</li>
<li>Version dashboards and alert rules in Git alongside the pipelines. A dashboard edited live in the UI is a dashboard that will be silently different next quarter.</li>
<li>Emit row counts at stage boundaries as a habit, not as a special case. In versus out at each hop is what turns &#8220;the numbers are wrong&#8221; into &#8220;stage three drops rows&#8221;.</li>
<li>Review alerts monthly and delete the ones that never fired usefully. Alert rules are code and they rot the same way.</li>
<li>Keep lineage accessible but off the main screen. It answers a different question and it competes for attention with the ones you need first.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the difference between pipeline monitoring and data observability?</h3>



<p class="wp-block-paragraph">Monitoring watches the process: did the job run, did it succeed, how long did it take. Observability watches the data the process produced: is it fresh, is there the right amount, has its shape changed. You need both, and the second is the one most teams are missing when they say their dashboards did not catch something.</p>



<h3 class="wp-block-heading">Do I need a commercial data observability platform?</h3>



<p class="wp-block-paragraph">Not to start. Freshness, volume and null-rate checks on your critical tables cover most real incidents and cost an afternoon. Commercial platforms earn their keep when you have hundreds of tables, several teams changing them independently, and nobody with time to hand-write checks. The honest signal that you have outgrown the homegrown version is when maintaining the checks becomes its own recurring task.</p>



<h3 class="wp-block-heading">Should I use StatsD or OpenTelemetry for Airflow metrics?</h3>



<p class="wp-block-paragraph">New stack, OpenTelemetry: you get real labels instead of reconstructing them from dotted names, and traces come along with it. Existing stack with working StatsD mappings, stay put until you have another reason to touch it. Metric coverage and naming differ between the two paths and between Airflow versions, so verify against the metrics reference for your version rather than copying a dashboard from a blog post.</p>



<h3 class="wp-block-heading">How do I monitor a pipeline that only runs weekly?</h3>



<p class="wp-block-paragraph">Same timestamp gauge, much larger threshold, and lean harder on the absence alert. Infrequent pipelines are where Pushgateway staleness does the most damage, because the last successful push can sit there for a fortnight looking entirely healthy. Check <code>push_time_seconds</code> for those groups specifically.</p>



<h3 class="wp-block-heading">What should a data pipeline operations dashboard show first?</h3>



<p class="wp-block-paragraph">Time since last successful load, per critical table, with its threshold visible. That one panel answers the question you will be asked most often, and when it is clean it rules out the entire pipeline layer in a glance.</p>



<h3 class="wp-block-heading">How many alerts is the right number?</h3>



<p class="wp-block-paragraph">Few enough that every page gets investigated. If someone can name an alert they routinely dismiss without looking, you have too many, and the cost is not the noise itself but the habit it builds. Start with freshness and absence per critical pipeline, add volume, and only go further when a real incident shows you a gap.</p>



<h3 class="wp-block-heading">Can I build this without an orchestrator like Airflow?</h3>



<p class="wp-block-paragraph">Yes. Cron jobs, Step Functions, GitHub Actions and shell scripts can all push a timestamp and a row count to a Pushgateway in three lines. The orchestrator gives you the run and timing signals for free, but it is the data-level signals that do the heavy lifting, and those come from querying the warehouse regardless of what triggered the load.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A data pipeline operations dashboard is not a nicer view of your orchestrator. It exists because the orchestrator can only tell you that a process finished, and the question you are actually being asked is whether the data is right.</p>



<p class="wp-block-paragraph">If you build one panel this week, build time since last successful load, sourced from a timestamp gauge, alerted on both threshold and absence. It costs an afternoon, it catches the failure mode that hides longest, and it turns &#8220;is the dashboard broken or just late?&#8221; into a question you can answer before anyone has to ask it.</p>



<h2 class="wp-block-heading">Need help building or fixing this?</h2>



<p class="wp-block-paragraph">I work with teams on pipeline observability and the plumbing underneath it. Things I regularly help with:</p>



<ul class="wp-block-list">
<li>Designing freshness, volume and quality checks for your critical tables, with thresholds derived from real consumer deadlines rather than guesses</li>
<li>Wiring Airflow into Prometheus and Grafana, including the statsd_exporter mappings and the cardinality controls that keep it affordable</li>
<li>Migrating an existing StatsD setup to the OpenTelemetry path without losing your dashboard history</li>
<li>Getting dbt artifacts out of ephemeral pods and into somewhere you can query them, with Elementary or with plain object storage</li>
<li>Auditing an existing dashboard and alert set to find what is stale, what will never fire, and what is quietly training your team to ignore pages</li>
<li>Standing up a self-hosted Prometheus and Grafana stack on a VPS, or sizing a managed alternative if owning the storage is not worth it to you</li>
</ul>



<p class="wp-block-paragraph">If you have a dashboard that went green through an incident, send me the panel queries and the alert rules and I will tell you where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/">Building a Data Pipeline Operations Dashboard That Catches Silent Failures</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</title>
		<link>https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 05 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Anomaly Detection]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[DQDL]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=148</guid>

					<description><![CDATA[<p>A SaaS admin changes a field and your pipeline stays green while the numbers drift. A practical guide to AWS Glue Data Quality for SaaS sources: where to run the checks, why nested payloads need flattening before DQDL can see them, which rule catches which failure, and the dynamic rules that pass silently because they have no history yet.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/">AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Nobody deployed anything. That is the part that makes it hard to explain.</p>



<p class="wp-block-paragraph">A Salesforce admin renames a custom field on Tuesday afternoon. Your connector keeps running, your Glue job keeps succeeding, the crawler happily adds a new column and quietly stops populating the old one. Three weeks later someone asks why the pipeline report has a cliff in it, and you spend an afternoon reconstructing a change nobody in engineering made and nobody logged.</p>



<p class="wp-block-paragraph">This post is about using AWS Glue Data Quality to catch that class of breakage. It is aimed at people pulling from SaaS sources into S3 or a lakehouse: Salesforce, HubSpot, Zendesk, Stripe, Shopify, Zoho, an internal partner API. I will cover where to run the checks, the flattening constraint that will bite you before anything else does, which DQDL rule actually catches which SaaS failure, when dynamic rules and anomaly detection quietly do nothing, and what should happen when a rule fails.</p>



<h2 class="wp-block-heading">Why SaaS data breaks in ways your own database doesn&#8217;t</h2>



<p class="wp-block-paragraph">When you own the source database, the schema is under change control: someone writes a migration, it gets reviewed, and if it breaks a consumer there is a commit to point at. A SaaS source has none of that. Three consequences:</p>



<ul class="wp-block-list">
<li><strong>The schema is edited by people who are not engineers.</strong> A sales ops admin adding a picklist value is a production schema change with no review, no notice and no version number.</li>

<li><strong>The payload is nested.</strong> SaaS APIs return structs and arrays: line items, related objects, custom field bags. That shape matters more than you would expect, for reasons in the next section but one.</li>

<li><strong>Absence is ambiguous.</strong> A field missing from a payload might mean &#8220;not set,&#8221; &#8220;not permitted for this user,&#8221; or &#8220;we removed it.&#8221; The connector usually cannot tell you which, and neither can your table.</li>
</ul>



<p class="wp-block-paragraph">Glue Data Quality is a rules engine built on Deequ that evaluates declarative checks written in DQDL, the Data Quality Definition Language, against a dataset. It is well suited to the first and third problems. The second one needs work before it is useful at all.</p>



<h2 class="wp-block-heading">Decide where the checks run before you write a rule</h2>



<p class="wp-block-paragraph">There are two places to evaluate a ruleset, and they answer different questions.</p>



<h3 class="wp-block-heading">Inside the Glue ETL job</h3>



<p class="wp-block-paragraph">The <code>EvaluateDataQuality</code> transform runs mid-pipeline against a DynamicFrame. It answers &#8220;is this batch safe to load?&#8221; and it can act on the answer: stop the job, or split failing rows into a quarantine prefix. This is also the only placement where row-level results exist, which matters because &#8220;which 400 rows are bad&#8221; is a far more useful answer than &#8220;the rule failed.&#8221;</p>



<h3 class="wp-block-heading">Against a Data Catalog table</h3>



<p class="wp-block-paragraph">You can attach a ruleset to a catalog table and run it on a schedule, independent of any job. This answers &#8220;is the table currently trustworthy?&#8221; and surfaces a quality score analysts can see before they query. The trade-off is that it is after the fact: by the time a catalog evaluation fails, the bad batch is already queryable. Analyzers are also not supported for catalog evaluations, so some dynamic-rule tooling behaves differently there.</p>



<p class="wp-block-paragraph">In practice I run both. The in-job ruleset is small and strict, and exists to stop garbage landing. The catalog ruleset is broader and looser, and exists so a table&#8217;s state is visible to people who will never open Glue Studio.</p>



<h2 class="wp-block-heading">The flattening problem that comes first</h2>



<p class="wp-block-paragraph">Here is the constraint that catches most people on SaaS data, stated plainly in the AWS documentation and easy to skim past: <strong>DQDL cannot evaluate nested or list-type columns.</strong> No structs, no arrays.</p>



<p class="wp-block-paragraph">Almost every interesting SaaS payload is nested. A Salesforce opportunity carries a related account struct, a Stripe invoice carries an array of line items, a Zendesk ticket carries a custom fields array of key-value pairs. Point a ruleset at the raw landed data and the fields you most want to check are the ones the engine cannot see.</p>



<p class="wp-block-paragraph">So you flatten first, with Glue&#8217;s relationalize or an explicit projection, then evaluate. That works, but it changes what your rules mean:</p>



<ul class="wp-block-list">
<li><strong>Your rules now describe the projection, not the payload.</strong> If your flattening step drops a field, no rule will notice, because from DQDL&#8217;s point of view that field was never there.</li>

<li><strong>Exploding an array changes the grain.</strong> One invoice with five line items becomes five rows. Every <code>RowCount</code> rule you write against that table is now counting line items, and a customer who starts itemising differently will move your row count without any data being wrong.</li>

<li><strong>Custom-field bags need pivoting, not flattening.</strong> A key-value array flattened naively gives you rows, not columns, and you cannot write a <code>ColumnValues</code> rule against a value whose meaning depends on a sibling key column. Pivot the ones you care about into real columns and check those.</li>
</ul>



<p class="wp-block-paragraph">The practical rule: put your quality checks immediately after the flattening step, and treat the flattening step itself as code that needs its own test. Glue Data Quality will not cover it for you.</p>



<h2 class="wp-block-heading">Matching SaaS breakage to the rule that catches it</h2>



<p class="wp-block-paragraph">DQDL has a long rule list and most guides just enumerate it. More useful is the other direction: what actually goes wrong with SaaS sources, and what catches each one.</p>



<h3 class="wp-block-heading">A field is renamed, removed, or added</h3>



<p class="wp-block-paragraph">This is the Tuesday-afternoon scenario. Two rules, doing two different jobs:</p>



<pre class="wp-block-code"><code>Rules = [
    ColumnExists "annual_revenue__c",
    ColumnCount = 47
]</code></pre>



<p class="wp-block-paragraph"><code>ColumnExists</code> protects the specific fields your downstream models depend on. <code>ColumnCount</code> protects against everything else: a field silently added, or one you forgot you needed. The first is precise and the second is a tripwire, and you want both because a rename shows up as one column gone and one column arrived, which leaves the count unchanged.</p>



<p class="wp-block-paragraph"><code>SchemaMatch</code> compares against a reference dataset&#8217;s schema, which is stricter and useful if you maintain a golden reference table. Bear in mind a hard column count on a SaaS object fails every time an admin adds a field, so decide whether that blocks or just warns.</p>



<h3 class="wp-block-heading">A field still exists but stopped being populated</h3>



<p class="wp-block-paragraph">Permissions change, a workflow gets disabled, an integration user loses field-level access. The column is still in the schema and is now all nulls.</p>



<pre class="wp-block-code"><code>Rules = [
    IsComplete "opportunity_id",
    Completeness "lead_source" &gt; 0.85,
    Completeness "email" &gt; 0.60 where "record_type = 'Contact'"
]</code></pre>



<p class="wp-block-paragraph"><code>IsComplete</code> is the absolute version: no nulls at all, right for keys. <code>Completeness</code> takes a threshold, which is what you want for optional business fields where a hard rule would fail constantly. The <code>where</code> clause is the one people miss: SaaS objects are usually polymorphic, and a completeness threshold that makes sense for contacts is nonsense across the whole table. Scope the rule to the record type.</p>



<h3 class="wp-block-heading">The sync ran short</h3>



<p class="wp-block-paragraph">A paginated extract stops early, a rate limit gets absorbed by retries, an incremental cursor drifts. The job succeeds and delivers less data than it should have. A static <code>RowCount &gt; 1000</code> stops catching this the moment you cross a thousand rows. Dynamic rules compare against your own history instead:</p>



<pre class="wp-block-code"><code>Rules = [
    RowCount &gt; min(last(3)),
    Sum "amount" &gt; avg(last(5)) * 0.7,
    DetectAnomalies "RowCount"
]

Analyzers = [
    RowCount,
    DistinctValuesCount "owner_id"
]</code></pre>



<p class="wp-block-paragraph"><code>last(k)</code> pulls the last k recorded values of that metric for the same dataset; with k greater than one you need an aggregation like <code>avg</code>, <code>min</code> or <code>max</code> to reduce them to one number. <code>DetectAnomalies</code> goes further, using forecasting over stored history to flag deviations with no threshold from you at all. That is the right tool for anything seasonal: a B2B pipeline that dips every weekend trips a naive trailing-average rule every Monday, while anomaly detection learns the shape.</p>



<p class="wp-block-paragraph">Analyzers deserve their own mention: they gather a metric without asserting anything, which builds the history dynamic rules and anomaly detection need. <code>DistinctValuesCount</code> on an owner or account column is a strong early signal for a partial sync, because row count can look fine while the number of distinct accounts represented quietly halves.</p>



<h3 class="wp-block-heading">Records arrive twice, or the key stops being a key</h3>



<p class="wp-block-paragraph">Replays, overlapping incremental windows and connector retries all produce duplicates, and SaaS &#8220;IDs&#8221; are less unique than vendor docs imply once you have merged sandboxes or multiple accounts.</p>



<pre class="wp-block-code"><code>Rules = [
    IsPrimaryKey "id",
    Uniqueness "external_reference" &gt; 0.99
]</code></pre>



<p class="wp-block-paragraph"><code>IsPrimaryKey</code> asserts unique and non-null together. <code>Uniqueness</code> with a threshold is the softer version for fields that are mostly-but-not-perfectly unique, which describes a lot of SaaS reference fields honestly.</p>



<h3 class="wp-block-heading">Objects stop agreeing with each other</h3>



<p class="wp-block-paragraph">Multi-object extracts rot quietly. Opportunities reference accounts that were never synced; invoice totals stop matching line item sums because one object&#8217;s extract failed and the other&#8217;s didn&#8217;t.</p>



<p class="wp-block-paragraph"><code>ReferentialIntegrity</code> checks that values in one dataset&#8217;s column exist in another&#8217;s. <code>RowCountMatch</code> and <code>AggregateMatch</code> compare counts and summary metrics across two datasets. These catch a broken join before an analyst does, and they need the multi-frame form of the transform with the second dataset supplied as an additional source.</p>



<h3 class="wp-block-heading">The sync stopped and nobody noticed</h3>



<p class="wp-block-paragraph">The quietest failure of all: nothing is wrong with the data, there just isn&#8217;t any new data. <code>DataFreshness</code> checks a timestamp column against a window. It is deliberately simple and its comparison granularity is limited, so for anything more nuanced the custom SQL rule is the escape hatch, running a query against the dataset and evaluating the result against an expression. DQDL is case sensitive and that rule type&#8217;s exact capitalisation has appeared more than one way across the docs and the visual builder, so copy it from the current DQDL reference rather than from memory.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Static, dynamic and ML: when each one lies to you</h2>



<p class="wp-block-paragraph">All three check styles have a mode where they pass while the data is wrong. Knowing which is which is most of the skill.</p>



<ul class="wp-block-list">
<li><strong>Static rules lie by going stale.</strong> A threshold that was correct when you wrote it stops being correct as the business grows, and a rule that can no longer fail is worse than no rule because it looks like coverage.</li>

<li><strong>Dynamic rules lie when they have no history.</strong> This one is genuinely nasty. <code>last(k)</code> reads previously recorded values for that metric, and if there aren&#8217;t any, it falls back to a default of zero. Add <code>RowCount &gt; avg(last(3))</code> to an existing job today and it will pass trivially for the first few runs, because almost any row count beats zero. It looks like a working check from the moment you deploy it. It isn&#8217;t one until it has accumulated runs.</li>

<li><strong>Anomaly detection lies while it is warming up, and after you retrain it badly.</strong> It needs a run of history before its forecasts mean anything, and if you leave a period of genuinely bad data in the training window, it learns that the bad shape is normal. You can exclude specific statistics from the calculation in the Data Quality tab, and you should, after any incident.</li>
</ul>



<p class="wp-block-paragraph">So when you add a dynamic or ML-based rule, write down the date and treat it as unarmed for the first couple of weeks. Keep a crude static rule alongside it in the meantime.</p>



<h2 class="wp-block-heading">What should happen when a rule fails</h2>



<p class="wp-block-paragraph">The transform returns a collection. The two members you care about are the row-level outcomes, which give you the original data with an evaluation result column appended per row, and the rule outcomes, which give you one row per rule with its verdict.</p>



<pre class="wp-block-code"><code>from awsgluedq.transforms import EvaluateDataQuality

dq_results = EvaluateDataQuality().process_rows(
    frame=flattened_opportunities,
    ruleset=ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "salesforce_opportunity",
        "enableDataQualityCloudWatchMetrics": "true",
        "enableDataQualityResultsPublishing": "true",
    },
)

row_level = SelectFromCollection.apply(
    dfc=dq_results, key="rowLevelOutcomes"
)</code></pre>



<p class="wp-block-paragraph"><code>dataQualityEvaluationContext</code> is the namespace your CloudWatch metrics land under. Name it after the source object rather than the job, because you will eventually want to alert per object.</p>



<p class="wp-block-paragraph">From there, three response patterns:</p>



<ol class="wp-block-list">
<li><strong>Fail fast.</strong> Stop the job on failure so nothing lands. Right for structural rules: if the primary key isn&#8217;t unique, downstream is going to be wrong no matter what you do next.</li>

<li><strong>Quarantine.</strong> Filter the row-level outcomes on the evaluation result column, write passing rows to the target and failing rows to a separate prefix. Right for value-level rules where 2% bad records shouldn&#8217;t block 98% good ones. The important discipline is that somebody has to actually look at the quarantine prefix; an unmonitored quarantine is a delete with extra steps.</li>

<li><strong>Warn and continue.</strong> Let everything land, publish the metrics, alert on the score. Right for new rules you don&#8217;t trust yet, and for the catalog-side ruleset.</li>
</ol>



<p class="wp-block-paragraph">Whichever you pick, get the failure out of Glue and into wherever your team actually looks. Results and anomaly predictions can be written back to catalog tables so they are queryable with plain SQL, and CloudWatch metrics plus EventBridge events give you the hook for paging or pausing a downstream pipeline. A Grafana or Datadog dashboard fed from those metrics beats a score in a console tab nobody opens.</p>



<h2 class="wp-block-heading">Troubleshooting AWS Glue Data Quality on SaaS pipelines</h2>



<h3 class="wp-block-heading">&#8220;Input data does not include column X&#8221;</h3>



<p class="wp-block-paragraph">A rule references a column the engine cannot find. On SaaS data that means one of three things: the field disappeared from the source, your flattening step renamed it, or the Data Catalog is out of sync with the underlying table. Check the catalog schema against the landed files before touching the ruleset; rerunning the crawler resolves a surprising share of these.</p>



<h3 class="wp-block-heading">A rule type isn&#8217;t recognised</h3>



<p class="wp-block-paragraph">DQDL is case sensitive and the rule set has grown over releases, so a rule copied from an older blog post can parse fine in your head and not at all in the engine. Check the rule name against the current DQDL rule type reference, and check that the feature is available on your Glue version. Anomaly detection in particular has a version floor.</p>



<h3 class="wp-block-heading">A dynamic rule never fails</h3>



<p class="wp-block-paragraph">Almost always the empty-history problem described above. Look at the statistics for that metric in the Data Quality tab. If there are fewer recorded points than your <code>last(k)</code> asks for, the rule is not doing what you think. It will start working on its own; the mistake is trusting it in the meantime.</p>



<h3 class="wp-block-heading">Evaluation is slow or the job runs out of memory</h3>



<p class="wp-block-paragraph">Each rule triggers analyzer computation over the dataset, and a large ruleset on a wide flattened SaaS table adds up fast. The transform exposes caching and observations-scope options to tune this. Before reaching for a bigger worker type, cut the ruleset down: most contain a dozen recommended rules nobody reads the results of.</p>



<h3 class="wp-block-heading">A &#8220;column reference is ambiguous&#8221; error</h3>



<p class="wp-block-paragraph">Classic flattening artefact. Two nested paths collapsed to the same leaf name, so <code>datechecked</code> now matches two things. Fix it in the projection with explicit aliases rather than trying to disambiguate inside DQDL.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Accepting the recommended ruleset wholesale. Recommendations are generated from a sample of current data, so they encode today&#8217;s accidents as tomorrow&#8217;s requirements.</li>

<li>Writing rules against nested data and assuming they cover the whole payload.</li>

<li>Deploying a dynamic rule and treating it as armed on day one.</li>

<li>Applying whole-table completeness thresholds to a polymorphic object instead of scoping with a <code>where</code> clause.</li>

<li>Quarantining failed rows to a prefix nobody monitors.</li>

<li>Writing <code>RowCount</code> rules against an exploded array without noticing the grain changed.</li>

<li>Failing the job on every rule, including cosmetic ones, until the team starts rerunning with checks disabled.</li>

<li>Keeping the ruleset in the console instead of in version control next to the job.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Split rules into blocking and reporting tiers, and be honest about which is which. A rule that always fails and never stops anything trains people to ignore the tab.</li>

<li>Pair <code>ColumnExists</code> with <code>ColumnCount</code>. One protects the fields you named, the other catches the drift you did not anticipate.</li>

<li>Add analyzers early, even for metrics you have no rule for yet. History has to accumulate before dynamic rules and anomaly detection are worth anything, and you cannot backfill it.</li>

<li>Keep DQDL in Git alongside the job definition and deploy it with Terraform or CloudFormation. Rulesets are code and should move through the same review as the pipeline they guard.</li>

<li>Namespace the evaluation context per source object so CloudWatch metrics and alerts can be routed per vendor.</li>

<li>Exclude incident periods from the anomaly detection training window once the incident is resolved.</li>

<li>Surface results where stakeholders already look. Writing outcomes back to catalog tables makes them queryable from Athena, which beats a score in a console tab.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much does AWS Glue Data Quality cost?</h3>



<p class="wp-block-paragraph">It bills on Glue DPU-hours like any other Glue workload, so cost is a function of evaluation time and DPUs used. The dimension that surprises people is anomaly detection, which consumes additional capacity per statistic analysed, so enabling it broadly across many columns and tables is where the bill moves. Model it against the current Glue pricing page; rates change and vary by region.</p>



<h3 class="wp-block-heading">Can it check data before it lands in S3?</h3>



<p class="wp-block-paragraph">Yes, if the check runs inside the Glue job. The transform evaluates a DynamicFrame in memory, so you can validate after extraction and before the write. That is the placement that lets you stop a bad load rather than report on one.</p>



<h3 class="wp-block-heading">Does it work on nested JSON from a SaaS API?</h3>



<p class="wp-block-paragraph">Not directly. DQDL does not evaluate nested or list-type columns, so you have to flatten the structure first and run the ruleset against the flattened frame. Plan the flattening deliberately, because your rules only ever see what the projection exposes.</p>



<h3 class="wp-block-heading">How does it compare to Great Expectations, Soda or Monte Carlo?</h3>



<p class="wp-block-paragraph">Glue Data Quality&#8217;s real advantage is that it is already inside the runtime and the catalog: no extra infrastructure, no separate agent, one IAM story. The disadvantages are real too. DQDL is less expressive than Python-based assertions, the nested-data limitation is a hard wall, and it only covers what Glue can read. Great Expectations gives you more expressive checks at the cost of running it yourself; Monte Carlo and similar platforms add lineage and cross-warehouse coverage Glue does not attempt. Glue-centric stack, start here. Several engines, a standalone tool fits better.</p>



<h3 class="wp-block-heading">Should I use the recommended rules?</h3>



<p class="wp-block-paragraph">As a starting draft, yes. As a ruleset, no. Recommendations are inferred from a sample of the data as it is today, which means any existing quality problem gets encoded as the expected baseline. Read every generated rule and delete the ones you would not have written yourself.</p>



<h3 class="wp-block-heading">Can I reconcile row counts against the SaaS source itself?</h3>



<p class="wp-block-paragraph">Not directly; it only sees datasets it can read. The usual pattern is to have the extraction step record the count the API reported, land that alongside the data as a control table, then use <code>RowCountMatch</code> or <code>AggregateMatch</code> to compare the two. That turns &#8220;did we get everything&#8221; into a rule the engine can evaluate.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">AWS Glue Data Quality is not really a data quality product. It is a way of writing down what you believe about a dataset so a machine can check it every run, and its value is entirely determined by whether those beliefs are written precisely enough to be wrong.</p>



<p class="wp-block-paragraph">For SaaS sources that means two habits. Flatten deliberately and check immediately after, because your rules only see the projection. And treat every new dynamic or ML-based rule as unarmed until it has history, because a rule that cannot fail yet looks exactly like a rule that is passing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with data quality on your Glue pipelines?</h2>



<p class="wp-block-paragraph">I work with teams whose SaaS pipelines run green while the numbers drift. Things I can help with:</p>



<ul class="wp-block-list">
<li>Writing a DQDL ruleset for a specific SaaS object that catches schema drift instead of just restating the current schema.</li>

<li>Designing the flattening and pivoting step for nested payloads and custom-field bags so the checks actually cover what matters.</li>

<li>Setting up quarantine and fail-fast paths in Glue ETL, with the routing and alerting that makes a quarantine prefix something people act on.</li>

<li>Cross-object reconciliation between SaaS extracts and control counts recorded at extraction time.</li>

<li>Tuning anomaly detection and dynamic rules, including cleaning incident periods out of the training window.</li>

<li>Getting data quality results out of the console and into Grafana, CloudWatch alarms or an Athena-queryable table.</li>
</ul>



<p class="wp-block-paragraph">If you have a ruleset that keeps passing while the data drifts, send me the DQDL and a schema dump. That is usually enough to see where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/">AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
