<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Apache Airflow | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/apache-airflow/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/apache-airflow/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 12:01:15 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Apache Airflow | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/apache-airflow/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates</title>
		<link>https://john-nessime.com/blog/devops/data-pipeline-freshness-monitoring/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 08 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[Deadman Alerting]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Kafka]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[PromQL]]></category>
		<category><![CDATA[Pushgateway]]></category>
		<category><![CDATA[SLO]]></category>
		<category><![CDATA[SRE]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=407</guid>

					<description><![CDATA[<p>Every task green, zero errors, and the dashboard still shows yesterday's numbers. Run status describes your code, not your data. Here is how to instrument freshness, latency and failure rate so your pipeline tells you before an analyst does.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-freshness-monitoring/">Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message lands at 8:40 in the morning. &#8220;Are the revenue numbers right? They look like yesterday&#8217;s.&#8221; You open the orchestrator and every task is green. The run finished in eleven minutes, comfortably inside its window. Zero errors. Zero retries. The pipeline did exactly what it was told to do, and the number on the dashboard is a day old.</p>



<p class="wp-block-paragraph">That gap is the entire problem. Run status tells you the code executed. It tells you nothing about whether data arrived. Data pipeline freshness monitoring is the signal that closes the gap, and most teams add it only after an analyst finds the stale table first.</p>



<p class="wp-block-paragraph">This post covers the three signals worth instrumenting for a batch or streaming pipeline: how current the data is, how long it took to get there, and how often the thing breaks. For each one I&#8217;ll cover what to measure, where to measure it, the queries and config that produce it, and the specific ways the measurement lies to you.</p>



<h2 class="wp-block-heading">The signal that lies to you first: run status</h2>



<p class="wp-block-paragraph">Job success is a statement about your code path, not about your data. A pipeline that reads an empty S3 prefix, transforms zero rows, writes zero rows and exits cleanly has succeeded. So has one whose upstream API silently started returning an empty page after a token rotation. So has one whose incremental watermark got stuck and now re-reads the same already-loaded slice on every run.</p>



<p class="wp-block-paragraph">All three are green. All three are producing stale data. This is why the ordering matters: freshness is the signal that fires first in a real incident, and run status is usually the last one to notice anything at all.</p>



<p class="wp-block-paragraph">So keep run status. It&#8217;s cheap and it catches crashes. Just stop treating it as your top-line health indicator.</p>



<h2 class="wp-block-heading">Freshness: measure the table, not the job</h2>



<p class="wp-block-paragraph">Freshness is the age of the newest record in a dataset. It&#8217;s measured against the dataset itself, not against the process that filled it, and that distinction is what makes it useful. If your job disappears entirely, freshness keeps climbing and keeps alerting. If your job succeeds while doing nothing, freshness keeps climbing and keeps alerting. It&#8217;s the one signal that survives both failure modes.</p>



<h3 class="wp-block-heading">Two clocks, and you need both</h3>



<p class="wp-block-paragraph">Every row usually carries two timestamps: when the event happened in the source system, and when your pipeline wrote it. Track the maximum of each.</p>



<ul class="wp-block-list">
<li><strong>Max event time</strong> tells you whether the upstream system is still producing. If this stops moving, the problem is upstream of you.</li>



<li><strong>Max load time</strong> tells you whether your pipeline is still writing. If this stops moving while event time is fine, the problem is yours.</li>
</ul>



<p class="wp-block-paragraph">Watching only one of them means every stale-data page starts with twenty minutes of figuring out which side of the boundary the fault sits on. Watching both answers that in the alert body.</p>



<p class="wp-block-paragraph">A freshness probe is a small scheduled query. This one is PostgreSQL syntax, and it returns both clocks plus a volume check in a single round trip:</p>



<pre class="wp-block-code"><code>select
  'orders' as dataset,
  extract(epoch from max(event_ts))  as max_event_ts,
  extract(epoch from max(loaded_at)) as max_loaded_at,
  count(*) filter (where loaded_at &gt;= now() - interval '1 hour') as rows_last_hour
from analytics.orders;</code></pre>



<p class="wp-block-paragraph">The row count matters. A pipeline can advance its load timestamp while writing almost nothing, which is what a partially broken source looks like. Freshness alone will not catch that; freshness plus volume will.</p>



<p class="wp-block-paragraph">Run that probe on a schedule that&#8217;s independent of the pipeline, push the results as gauges, and alert on age in PromQL:</p>



<pre class="wp-block-code"><code># Newest row is older than 90 minutes
time() - max by (dataset) (dataset_max_loaded_timestamp_seconds) &gt; 5400

# The probe itself has stopped reporting: a deadman check
absent(dataset_max_loaded_timestamp_seconds{dataset="orders"})</code></pre>



<p class="wp-block-paragraph">That second rule is the one people forget. A freshness metric that disappears looks identical to a healthy silence on a graph. <code>absent()</code> is what turns &#8220;no data&#8221; into a page.</p>



<h3 class="wp-block-heading">Where dbt fits</h3>



<p class="wp-block-paragraph">If you&#8217;re already running dbt, source freshness is built in and worth using before you write anything custom. You declare thresholds per source, and dbt queries the maximum of your timestamp column and compares it to now.</p>



<pre class="wp-block-code"><code>sources:
  - name: raw_shop
    schema: raw
    config:
      loaded_at_field: _loaded_at
      freshness:
        warn_after: {count: 2, period: hour}
        error_after: {count: 6, period: hour}
    tables:
      - name: orders
      - name: refunds
        config:
          freshness:
            warn_after: {count: 12, period: hour}
            error_after: {count: 24, period: hour}</code></pre>



<p class="wp-block-paragraph">Three details that trip people up. First, <code>dbt source freshness</code> is a separate command; <code>dbt build</code> does not run it, so a green build says nothing about source staleness. Second, it exits non-zero when a source hits its <code>error_after</code> threshold, which makes it a natural gate at the top of a job: fail fast rather than building models on stale input. Third, results land in <code>target/sources.json</code>, which is the artifact you parse if you want to distinguish warn from error, or ship the numbers into Prometheus rather than just failing the run.</p>



<pre class="wp-block-code"><code>dbt source freshness --select source:raw_shop --output target/freshness.json</code></pre>



<p class="wp-block-paragraph">One caveat on the YAML above: dbt moved these keys under a <code>config:</code> block in recent releases, and <code>loaded_at_field</code> followed later. Older projects nest them directly under the source. Check what your project&#8217;s version expects before copying, because a misplaced key fails quietly by simply not calculating freshness at all.</p>



<h2 class="wp-block-heading">Latency: name the clock before you name the number</h2>



<p class="wp-block-paragraph">&#8220;Our pipeline latency is twelve minutes&#8221; is meaningless until you say which two points you measured between. There are at least three plausible definitions, and teams routinely argue past each other because they&#8217;re each using a different one.</p>



<ul class="wp-block-list">
<li><strong>Run duration.</strong> Start to end of the job. Easy, and mostly useless for anyone downstream.</li>



<li><strong>Ingestion latency.</strong> Source extract to target load. This is what you control.</li>



<li><strong>End-to-end latency.</strong> Event time to the moment the row is queryable. This is what the business actually feels, and it includes queue wait, scheduler delay, and every upstream hop you don&#8217;t own.</li>
</ul>



<p class="wp-block-paragraph">Publish end-to-end as the headline and keep the others as breakdown. If you only track run duration, a scheduler backlog that delays every run by forty minutes is completely invisible to you: each individual run still takes eleven minutes.</p>



<p class="wp-block-paragraph">Instrument per stage with a histogram so you can ask percentile questions later without re-instrumenting:</p>



<pre class="wp-block-code"><code>histogram_quantile(
  0.95,
  sum by (le, stage) (rate(pipeline_stage_duration_seconds_bucket[6h]))
)</code></pre>



<p class="wp-block-paragraph">Use the median for capacity planning and p95 or p99 for the SLO. Averages hide the exact tail that generates the complaints, and on a pipeline that runs a few dozen times a day, a single pathological run is a real fraction of your day.</p>



<p class="wp-block-paragraph">For streaming, the equivalent is consumer lag expressed in time rather than offsets. Offset lag of fifty thousand messages means nothing without a rate; two hundred seconds of lag means something to everyone. Kafka Lag Exporter popularised this by interpolating a time estimate from observed offset and timestamp samples, exposing <code>kafka_consumergroup_group_max_lag_seconds</code> alongside the offset-based <code>kafka_consumergroup_group_lag</code>. Worth knowing: that project&#8217;s repository has been archived and is read-only, so if you&#8217;re starting fresh, check whether your broker vendor or a maintained fork covers it before you deploy something unmaintained into the critical path.</p>



<h2 class="wp-block-heading">Failure rate: decide what counts as a failure</h2>



<p class="wp-block-paragraph">Failure rate is trivially easy to compute and surprisingly easy to compute wrongly. The denominator and the definition both need a decision.</p>



<p class="wp-block-paragraph">Count runs, not tasks. A DAG with sixty tasks where one flaps on a transient network error looks catastrophic at task level and fine at run level. The run is the unit the consumer cares about.</p>



<p class="wp-block-paragraph">Count a run that succeeded on its third retry as a success for availability and a failure for a separate reliability metric. Both are true and they answer different questions. If your only metric folds retries into success, you will never see the slow degradation of an upstream API until it stops responding entirely.</p>



<pre class="wp-block-code"><code>sum by (pipeline) (rate(pipeline_runs_total{result="failure"}[6h]))
/
sum by (pipeline) (rate(pipeline_runs_total[6h]))</code></pre>



<p class="wp-block-paragraph">And add a category label for the failure reason at push time: source unavailable, schema mismatch, permission denied, timeout, validation failed. The rate tells you something is wrong. The category tells you who to wake up. Without it, every failure alert costs you a log dive before you can even route the incident.</p>



<h2 class="wp-block-heading">Getting metrics out of a job that exits</h2>



<p class="wp-block-paragraph">Prometheus scrapes. Batch jobs finish and vanish. Pushgateway bridges that: the job pushes before exiting, and Pushgateway holds the values for Prometheus to scrape on its own schedule.</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
set -euo pipefail

JOB="orders_load"
PGW="http://pushgateway.internal:9091"

start=$(date +%s)
if python /opt/pipelines/load_orders.py; then
  result=0
else
  result=1
fi
end=$(date +%s)

{
  cat &lt;&lt;EOF
# TYPE pipeline_run_duration_seconds gauge
pipeline_run_duration_seconds $((end - start))
# TYPE pipeline_last_run_timestamp_seconds gauge
pipeline_last_run_timestamp_seconds $end
# TYPE pipeline_last_run_success gauge
pipeline_last_run_success $((1 - result))
EOF
  if [ "$result" -eq 0 ]; then
    cat &lt;&lt;EOF
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $end
EOF
  fi
} | curl --fail --data-binary @- "$PGW/metrics/job/$JOB"

exit $result</code></pre>



<p class="wp-block-paragraph">The conditional block is doing real work. curl&#8217;s <code>--data-binary</code> issues a POST, and a POST to Pushgateway replaces only the metrics whose names appear in the payload, leaving the rest of the group intact. So a failing run updates the run timestamp and the success flag while leaving the previous <code>pipeline_last_success_timestamp_seconds</code> exactly where it was. That&#8217;s what lets <code>time() - pipeline_last_success_timestamp_seconds</code> keep climbing across consecutive failures. Send a PUT instead and you replace the whole group, wiping the value you needed.</p>



<p class="wp-block-paragraph">On the Prometheus side, one setting is not optional:</p>



<pre class="wp-block-code"><code>scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway.internal:9091']</code></pre>



<p class="wp-block-paragraph">Without <code>honor_labels: true</code>, Prometheus overwrites the <code>job</code> label your pipeline pushed with the scrape job&#8217;s own name, and every pipeline in your estate collapses into one indistinguishable series called <code>pushgateway</code>.</p>



<p class="wp-block-paragraph">The trap worth internalising: Pushgateway never forgets. Metrics persist until something explicitly deletes them or the process restarts. A pipeline you decommissioned last quarter is still cheerfully reporting a success timestamp and a duration, and it looks alive on every dashboard. Treat Pushgateway as a cache of past executions, not a picture of current state, and use the <code>push_time_seconds</code> gauge that Pushgateway attaches to each group to tell the difference between a fresh push and a fossil.</p>



<h3 class="wp-block-heading">If you&#8217;re on Airflow, the SLA feature is gone</h3>



<p class="wp-block-paragraph">This one will bite anyone upgrading. The <code>sla</code> and <code>sla_miss_callback</code> parameters were removed in Airflow 3.0, and the replacement, Deadline Alerts, arrived in 3.1. DAGs carrying the old configuration need manual migration; they don&#8217;t quietly keep working.</p>



<pre class="wp-block-code"><code>from datetime import timedelta

from airflow.sdk import AsyncCallback, DAG, DeadlineAlert, DeadlineReference
from airflow.providers.slack.notifications.slack_webhook import SlackWebhookNotifier

with DAG(
    dag_id="orders_load",
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_QUEUED_AT,
        interval=timedelta(minutes=45),
        callback=AsyncCallback(
            SlackWebhookNotifier,
            kwargs={"text": "orders_load has not finished 45 minutes after queuing."},
        ),
    ),
):
    ...</code></pre>



<p class="wp-block-paragraph">Note the reference point. Measuring from when the run was queued, rather than from its logical date, means scheduler backlog counts against the deadline. That&#8217;s usually what you want, because a run that sat in a queue for an hour is late to its consumers regardless of how fast it executed once it started.</p>



<p class="wp-block-paragraph">There&#8217;s a structural weakness here worth naming: any alert that lives inside the orchestrator dies with the orchestrator. If the scheduler is down, nothing evaluates your deadline and nothing notifies anyone. The freshness probe from earlier is the answer, and it needs to run somewhere else. A small VPS from a provider like Contabo or InterServer running Prometheus and Alertmanager, or a hosted option like Grafana Cloud, gives you a watcher outside the blast radius of the thing being watched. This is the single highest-value piece of monitoring most data teams are missing.</p>



<h2 class="wp-block-heading">Alerting without burning your on-call</h2>



<p class="wp-block-paragraph">The fastest way to make all of this worthless is to alert on every threshold crossing. Three rules keep it survivable.</p>



<ol class="wp-block-list">
<li><strong>Set thresholds from observed behaviour, not from wishes.</strong> Look at a month of actual freshness values and set <code>warn_after</code> a comfortable margin above the normal worst case. A source that habitually passes at eleven hours against a twelve-hour threshold is not healthy, it&#8217;s one upstream hiccup from paging you.</li>



<li><strong>Page on consumer impact, not internal events.</strong> A task retry is not an incident. A dataset breaching the freshness commitment its consumers rely on is. Route everything else to a channel someone reads in the morning.</li>



<li><strong>Use burn rate over multiple windows for SLOs.</strong> A short window catches fast breakage, a long window catches slow erosion, and requiring both to fire filters out the transient spikes that generate most false pages.</li>
</ol>



<p class="wp-block-paragraph">One more: alert on the absence of your own telemetry. A freshness gauge that stops updating is indistinguishable from one that&#8217;s fine, right up until someone asks about the numbers.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>Freshness alert fires but the data looks current.</strong> Almost always a timezone problem. Your <code>loaded_at</code> column is in local time, <code>now()</code> is in UTC, and the offset shows up as a constant bias in the age. Store load timestamps in UTC and cast explicitly at read time.</p>



<p class="wp-block-paragraph"><strong>Freshness flaps in and out of breach.</strong> Your check runs too close to the expected arrival. If loads land around six and your probe runs at five past, a fifteen-minute upstream delay produces an intermittent failure that trains everyone to ignore the alert. Move the probe later or widen the threshold.</p>



<p class="wp-block-paragraph"><strong>Every pipeline reports as one series.</strong> <code>honor_labels: true</code> is missing from the Pushgateway scrape config.</p>



<p class="wp-block-paragraph"><strong>A decommissioned pipeline still shows healthy.</strong> Stale group in Pushgateway. Delete the group and add a check on <code>push_time_seconds</code> so the next one surfaces on its own.</p>



<p class="wp-block-paragraph"><strong>Latency looks fine but consumers say data is late.</strong> You&#8217;re measuring run duration and they&#8217;re feeling end-to-end. Add queue wait and event-time-to-load and the gap will be obvious.</p>



<p class="wp-block-paragraph"><strong>Freshness passes, row count is near zero.</strong> The load timestamp advanced without meaningful data. This is a broken source or a stuck watermark, and it&#8217;s the reason the volume check belongs in the same probe.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating job success as the health signal and discovering staleness through a human.</li>



<li>Running the freshness check inside the same pipeline it&#8217;s meant to police.</li>



<li>Tracking freshness without volume, so a zero-row load reads as healthy.</li>



<li>Reporting latency without saying which two timestamps it spans.</li>



<li>Putting run IDs, batch IDs or timestamps into Prometheus labels, which multiplies your series count without bound. Those belong in traces or logs.</li>



<li>Alerting on task-level failures instead of run-level outcomes, then muting the whole channel a week later.</li>



<li>Assuming Pushgateway reflects current state rather than the last thing anyone pushed.</li>
</ul>



<h2 class="wp-block-heading">Best practices for data pipeline freshness monitoring</h2>



<ul class="wp-block-list">
<li>Start with your three most-used tables. Full coverage is a project; three tables is an afternoon and catches most of the pain.</li>



<li>Write down the freshness commitment per dataset in plain language, then encode it. &#8220;Yesterday&#8217;s orders are complete by 07:00&#8221; converts directly into a threshold.</li>



<li>Name metrics consistently across pipelines. A shared prefix and a stable label set is what makes one dashboard work for all of them.</li>



<li>Keep labels low-cardinality: pipeline, dataset, stage, environment, result. Nothing unbounded.</li>



<li>Emit a failure category alongside every failure so alerts route themselves.</li>



<li>Run the deadman check on infrastructure that doesn&#8217;t share a failure domain with the pipeline.</li>



<li>Link every alert to a runbook that names the owner and the first three things to check.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the difference between pipeline latency and data freshness?</h3>



<p class="wp-block-paragraph">Latency measures how long a specific batch of data took to travel from source to destination. Freshness measures how old the newest available record is right now, regardless of whether anything is currently running. A pipeline can have excellent latency and terrible freshness if it stopped being triggered.</p>



<h3 class="wp-block-heading">How often should freshness checks run?</h3>



<p class="wp-block-paragraph">Frequently enough that you find out before your consumers do. A useful rule is roughly a quarter of your tolerance window: if data may be up to four hours old, check hourly. Freshness probes are cheap single-aggregate queries, so the limiting factor is usually warehouse billing rather than load.</p>



<h3 class="wp-block-heading">Do I need a data observability platform for this?</h3>



<p class="wp-block-paragraph">Not to start. Freshness, latency and failure rate for a handful of critical datasets is a scheduled query, a push, and a few alert rules. Commercial platforms earn their cost when you need automatic column-level lineage, anomaly detection across hundreds of tables, or coverage of assets nobody has explicitly instrumented. That&#8217;s a real problem at scale, and a genuinely expensive one to build yourself. It&#8217;s just not the problem you have on day one.</p>



<h3 class="wp-block-heading">How do I monitor freshness for a table that only updates weekly?</h3>



<p class="wp-block-paragraph">Set the threshold from the schedule plus a delivery margin, and add a deadman rule so a missing metric alerts on its own. For genuinely static reference tables, disable freshness explicitly rather than leaving a check that always warns; in dbt that means setting freshness to null for the table.</p>



<h3 class="wp-block-heading">Should freshness checks fail the pipeline or just warn?</h3>



<p class="wp-block-paragraph">Both, in different places. Checking source freshness at the start of a job and failing hard prevents you building models on stale input, which is the cheapest bug to prevent and the most expensive to unwind. Checking output freshness after the fact should alert rather than fail, because the run is already over.</p>



<h3 class="wp-block-heading">What percentile should I use for a latency SLO?</h3>



<p class="wp-block-paragraph">p95 for most internal analytics pipelines, p99 where downstream systems make automated decisions on the data. Track the median separately for capacity planning. Never use the average as the SLO number, because it hides exactly the tail that produces complaints.</p>



<h3 class="wp-block-heading">Does this work for streaming pipelines too?</h3>



<p class="wp-block-paragraph">Yes, with different plumbing. Freshness becomes consumer lag measured in time, latency becomes event-time to availability-time, and failure rate becomes connector and task state plus dead-letter volume. The reasoning is identical; only the source of the numbers changes.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you remember one thing, make it this: measure the data, not the job. Run status, task counts and duration all describe your code. Only freshness describes what your consumers actually receive, and it&#8217;s the one signal that stays honest when the pipeline succeeds at doing nothing.</p>



<p class="wp-block-paragraph">Good data pipeline freshness monitoring is not a platform purchase. It&#8217;s a scheduled query that reports the age and volume of your most important tables, pushed somewhere durable, with a deadman rule so that silence is treated as a failure rather than as health. Add latency broken down by stage and failure rate categorised by reason, and you can answer &#8220;is the data good right now&#8221; without opening a single log file.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help instrumenting your pipelines?</h2>



<p class="wp-block-paragraph">Most of my consulting work in this area is retrofitting observability onto pipelines that already exist and can&#8217;t be paused. Specifically:</p>



<ul class="wp-block-list">
<li>Defining freshness and latency commitments per dataset, then translating them into thresholds and alert rules that hold up on-call.</li>



<li>Adding freshness and volume probes to existing warehouses without touching the pipelines themselves.</li>



<li>Wiring batch jobs into Prometheus through Pushgateway, including the grouping-key and stale-metric problems that bite six months later.</li>



<li>Migrating Airflow DAGs off the removed SLA feature onto Deadline Alerts, or onto external checks that survive a scheduler outage.</li>



<li>Building the Grafana dashboard that answers &#8220;is the data good right now&#8221; in one screen, with drill-down by stage and failure category.</li>



<li>Cutting alert noise on pipelines where the channel has already been muted, by moving from task-level events to consumer-impact SLOs.</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me a DAG file, a scrape config, or a screenshot of the dashboard you don&#8217;t trust, and I&#8217;ll tell you what I&#8217;d instrument first and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-freshness-monitoring/">Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Data Pipeline Operations Dashboard That Catches Silent Failures</title>
		<link>https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 15 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[OpenTelemetry]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[PromQL]]></category>
		<category><![CDATA[Pushgateway]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[StatsD]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=227</guid>

					<description><![CDATA[<p>Green DAGs and wrong numbers are the same morning. This is how to build a data pipeline operations dashboard around the four questions that actually matter, where each signal comes from, and how to turn the panels into alerts that fire before your users notice.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/">Building a Data Pipeline Operations Dashboard That Catches Silent Failures</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually lands around nine in the morning, from someone in finance or ops: &#8220;Is the revenue dashboard broken, or is it just late?&#8221;</p>



<p class="wp-block-paragraph">So you open the orchestrator. Every task is green. The run started on schedule, finished inside its usual window, exit code zero across the board. Nothing paged. And the numbers are still wrong.</p>



<p class="wp-block-paragraph">That gap is the whole reason to build a data pipeline operations dashboard. Not to watch DAGs go green, which the orchestrator UI already does perfectly well, but to answer the questions the orchestrator structurally cannot answer. This post covers the four signals worth putting on a screen, where each one actually comes from, how to wire them into Prometheus and Grafana without inventing a metrics platform, and how to turn the panels into alerts that fire before a human notices.</p>



<h2 class="wp-block-heading">Why green pipelines still ship bad data</h2>



<p class="wp-block-paragraph">An orchestrator tracks process outcomes. Did the Python exit cleanly, did the SQL statement return without an error, did the container terminate zero. That is a real signal and you should keep it. It is just answering a different question than the one your users care about.</p>



<p class="wp-block-paragraph">Consider what a perfectly successful run looks like when the upstream source has quietly changed:</p>



<ul class="wp-block-list">
<li>An API starts paginating differently and your extractor pulls the first page only. It ran fine. It loaded 500 rows instead of 500,000.</li>
<li>A source column gets renamed and your transform selects it with a <code>coalesce</code> default. It ran fine. Every row now says &#8220;unknown&#8221;.</li>
<li>A partition arrives late, so the incremental filter finds nothing to process. It ran fine. It did nothing.</li>
<li>An upstream team backfills, and your dedupe key no longer holds. It ran fine. Everything is doubled.</li>
</ul>



<p class="wp-block-paragraph">None of those raise an exception. All four are visible in under a second if you are tracking row counts and freshness at the table level. This is the failure family that eats the most incident time, because the clock does not start when the pipeline breaks. It starts when a human notices, which can be days later, and by then the bad data has been copied into reports, cached in a BI extract, and quoted in a meeting.</p>



<h2 class="wp-block-heading">The four questions a data pipeline operations dashboard has to answer</h2>



<p class="wp-block-paragraph">Resist the instinct to start from what your tools can emit. Start from the questions you ask during an incident, in the order you ask them. Everything that does not answer one of these is a second-page panel.</p>



<h3 class="wp-block-heading">Did it run at all?</h3>



<p class="wp-block-paragraph">Sounds trivial, and it is the one most homegrown dashboards get wrong. A panel showing &#8220;last run: success&#8221; does not distinguish between a pipeline that succeeded ten minutes ago and one that succeeded eleven days ago and has not been scheduled since. You want time since last successful completion, as a number that grows, not a status pill that sits there looking calm.</p>



<h3 class="wp-block-heading">Did it finish in time?</h3>



<p class="wp-block-paragraph">Duration matters less than duration against a deadline. A load that takes forty minutes is fine if the report is read at 9am and the run starts at 6am. The same forty minutes is an incident if someone moved the schedule to 8:40. Put the deadline on the panel as a threshold line, not in a runbook. Trending duration also catches slow decay: a stage that creeps from five minutes to fifty usually means data growth, a bad query plan, or resource contention, and it will breach your window eventually.</p>



<h3 class="wp-block-heading">Did the right amount of data arrive?</h3>



<p class="wp-block-paragraph">Rows in versus rows out, per stage. This is the highest value panel per unit of effort on the entire dashboard. A drop from a million rows to twenty thousand is not subtle once you plot it, but it is completely invisible in an exit code. Compare against the same weekday from the previous week rather than against yesterday, because most business data has a weekly shape and Monday-versus-Sunday comparisons generate noise you will learn to ignore.</p>



<h3 class="wp-block-heading">Is the data still the shape you expect?</h3>



<p class="wp-block-paragraph">Null rates, distinct counts on key columns, and schema changes. This is the panel almost nobody builds, and it is where the expensive failures live. A gradual rise in null rate on a joining key means an upstream system is degrading and your joins are silently dropping rows. You do not need statistical anomaly detection to catch that. You need the number plotted over time next to the others.</p>



<p class="wp-block-paragraph">There is a fifth question that belongs on a different surface: what breaks downstream when this table is wrong. That is lineage, and it answers blast radius rather than health. Keep it one click away, not on the operations screen.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where each signal actually comes from</h2>



<p class="wp-block-paragraph">Three sources, three different collection mechanisms. This is the part that takes an afternoon rather than a sprint, provided you do not try to build a unified agent.</p>



<h3 class="wp-block-heading">The orchestrator, for run and timing signals</h3>



<p class="wp-block-paragraph">Airflow has emitted StatsD metrics for a long time, and that path is still the one most deployments run. Airflow speaks StatsD, so you put a <code>statsd_exporter</code> in front of it to translate into Prometheus format. Enable it in the metrics section of your config, or with the equivalent <code>AIRFLOW__METRICS__</code> environment variables if you are on containers:</p>



<pre class="wp-block-code"><code>[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 9125
statsd_prefix = airflow
metrics_allow_list = scheduler,executor,dagrun,pool,triggerer</code></pre>



<p class="wp-block-paragraph">Two things worth knowing before you roll this out. First, <code>metrics_allow_list</code> is not optional at any real scale. Airflow emits per-DAG and per-task metric names, so without a prefix filter your cardinality grows with every DAG anyone adds, and you find out when the time series database starts struggling. Second, Airflow expects the StatsD destination to be resolvable at startup, so a scheduler that boots before the exporter can fail to come up. Order your dependencies accordingly.</p>



<p class="wp-block-paragraph">StatsD has no concept of labels, so the exporter has to reconstruct them from the dotted metric name. That mapping file is the fiddly part:</p>



<pre class="wp-block-code"><code>mappings:
  - match: "*.dag.*.*.duration"
    match_metric_type: observer
    name: "airflow_task_duration"
    labels:
      airflow_id: "$1"
      dag_id: "$2"
      task_id: "$3"</code></pre>



<p class="wp-block-paragraph">The wildcards map positionally onto <code>$1</code>, <code>$2</code>, <code>$3</code>, and <code>match_metric_type</code> has to match what Airflow actually sends, timer versus counter versus gauge. Get that wrong and the metric silently never appears at the exporter, which is a genuinely annoying half hour of debugging. Check the exporter&#8217;s own metrics endpoint before you go looking at Prometheus, because if it is not there it never left Airflow.</p>



<p class="wp-block-paragraph">Airflow also has a native OpenTelemetry path for both metrics and traces, which skips the StatsD translation entirely and gives you real labels rather than reconstructed ones:</p>



<pre class="wp-block-code"><code>[metrics]
otel_on = True
otel_host = otel-collector
otel_port = 4318
otel_prefix = airflow
otel_interval_milliseconds = 30000</code></pre>



<p class="wp-block-paragraph">The port there is the collector&#8217;s OTLP HTTP receiver, and you need an OpenTelemetry Collector or compatible endpoint in the path. If you are standing up a new stack, this is the one I would reach for, because the label problem disappears and traces give you per-task spans under a DAG run parent span, which is worth a lot during a slow-pipeline investigation. If you already have working StatsD mappings and dashboards built on them, migrating is real work for a modest gain, and the honest answer is that it can wait. Metric names do shift between Airflow major versions either way, so pin your dashboard queries to the metrics reference for the version you actually run.</p>



<p class="wp-block-paragraph">Managed Airflow changes the plumbing but not the shape. On Amazon MWAA the metrics land in CloudWatch instead, and Google Cloud Composer and Astronomer each have their own export path. The four questions do not change.</p>



<h3 class="wp-block-heading">The transformation layer, for data-level signals</h3>



<p class="wp-block-paragraph">If you use dbt, most of what you need is already being written to disk and thrown away. Every invocation produces <code>run_results.json</code> in the target directory, containing per-node status and timing, and <code>dbt source freshness</code> produces its own <code>sources.json</code>. The <code>manifest.json</code> holds the dependency graph, which is your lineage.</p>



<p class="wp-block-paragraph">The cheapest useful thing you can do is parse the failures out at the end of the run:</p>



<pre class="wp-block-code"><code>jq -r '.results[]
  | select(.status != "success")
  | [.unique_id, .status, (.execution_time | tostring)]
  | @tsv' target/run_results.json</code></pre>



<p class="wp-block-paragraph">One trap if you run dbt in ephemeral Kubernetes pods: the target directory dies with the pod. Upload the artifacts to object storage as the last step inside the same task that ran dbt, not as a downstream task, or you will find your history is empty exactly when you want it.</p>



<p class="wp-block-paragraph">Elementary is the low-friction option here. It installs as a dbt package with an on-run-end hook that loads the artifacts into your warehouse as ordinary tables, which means your dashboard can query test results and model timings in SQL alongside everything else. Great Expectations and Soda cover the same validation ground with more expressive checks and more setup. The commercial platforms, Monte Carlo and the vendors around it, add automated anomaly detection and column-level lineage across the whole stack, and the genuine case for them is that they find the checks you did not think to write. The trade-off is that you are paying per monitored table for statistical inference, and on a stack of twenty tables that inference is not telling you much that a row-count panel and a null-rate panel would not.</p>



<p class="wp-block-paragraph">Without dbt, you write the check yourself and it is still short. One query per critical table, run on a schedule:</p>



<pre class="wp-block-code"><code>SELECT
  COUNT(*)                                        AS row_count,
  MAX(updated_at)                                 AS last_event,
  SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) AS null_keys
FROM analytics.orders
WHERE loaded_at &gt;= CURRENT_DATE - 1;</code></pre>



<h3 class="wp-block-heading">Batch jobs, and the Pushgateway caveat</h3>



<p class="wp-block-paragraph">Prometheus pulls. Batch jobs finish and vanish long before a scrape interval comes round, so there is nothing to pull from. The Pushgateway exists for exactly this: short-lived service-level jobs push their results to it, and Prometheus scrapes the gateway instead.</p>



<pre class="wp-block-code"><code>cat &lt;&lt;EOF | curl --data-binary @- 
  http://pushgateway:9091/metrics/job/orders_sync
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $(date +%s)
# TYPE pipeline_rows_loaded gauge
pipeline_rows_loaded ${ROWS}
EOF</code></pre>



<p class="wp-block-paragraph">Now the caveat, which is the single most common way a pipeline dashboard lies to you. The Pushgateway is a cache, not an aggregator, and it never forgets. Push a success metric, then have the job stop running entirely, and that success sits there being scraped forever. Your dashboard stays green because the last known value is green, and the fact that nothing has updated it in six days is not something a status panel can see.</p>



<p class="wp-block-paragraph">Two defences, and you want both. First, always push a timestamp gauge rather than a boolean status, so staleness is arithmetic rather than inference. Second, alert on absence as well as on value, because the Pushgateway also exposes <code>push_time_seconds</code> per group, which tells you when anything last arrived regardless of what it said. Prometheus&#8217;s usual safety net, the <code>up</code> metric going to zero when a target disappears, does not apply here: Prometheus is scraping the gateway successfully, and the gateway is happily serving stale data.</p>



<p class="wp-block-paragraph">Also keep instance identifiers out of the grouping labels. Push per service, not per worker, or you accumulate a permanent time series for every container that ever ran.</p>



<h2 class="wp-block-heading">Laying out the dashboard so it works during an incident</h2>



<p class="wp-block-paragraph">The layout constraint is that someone is reading this at 7am on a phone with a stakeholder waiting. Three rows, top to bottom, in the order you need them:</p>



<ol class="wp-block-list">
<li><strong>Freshness across everything.</strong> One panel, one row per critical table, showing time since last successful load against its threshold. If this row is clean, the pipelines are fine and the problem is elsewhere. That is a genuinely useful thing to establish in five seconds.</li>
<li><strong>Volume and duration per pipeline.</strong> Row counts with a week-over-week comparison, and stage duration with the deadline drawn on. This is where you localise a problem to a specific pipeline and stage.</li>
<li><strong>Data quality detail.</strong> Null rates, failed test counts, schema change events. This is where you work out what actually went wrong once you know where.</li>
</ol>



<p class="wp-block-paragraph">The freshness query in PromQL, given the timestamp gauge above:</p>



<pre class="wp-block-code"><code>time() - max by (job) (pipeline_last_success_timestamp_seconds)</code></pre>



<p class="wp-block-paragraph">Grafana renders that nicely as a table with per-row thresholds, which is what makes the top row scannable. If you are self-hosting, Prometheus plus Grafana on a modest VPS from Hetzner, InterServer or DigitalOcean handles a surprising amount of pipeline metrics, because pipeline telemetry is low-volume compared to application metrics. Grafana Cloud&#8217;s free tier is a reasonable alternative if you do not want to own the storage, and Datadog or New Relic make sense if your organisation is already paying for one of them and you would rather have one pane than a better one.</p>



<p class="wp-block-paragraph">Keep it to one screen. Every panel you add past that dilutes the ones that matter, and a dashboard nobody can read at a glance is a dashboard nobody opens.</p>



<h2 class="wp-block-heading">Turning panels into alerts</h2>



<p class="wp-block-paragraph">A dashboard that requires someone to look at it is a dashboard that finds problems the morning after. The freshness panel converts into an alert almost directly:</p>



<pre class="wp-block-code"><code>groups:
  - name: pipelines
    rules:
      - alert: PipelineStale
        expr: time() - max by (job) (pipeline_last_success_timestamp_seconds) &gt; 5400
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} has not completed successfully in over 90 minutes"

      - alert: PipelineMetricsMissing
        expr: absent(pipeline_last_success_timestamp_seconds{job="orders_sync"})
        for: 30m
        labels:
          severity: page
        annotations:
          summary: "No metrics received from orders_sync"</code></pre>



<p class="wp-block-paragraph">The first catches a pipeline that is late or broken. The second catches the one that stopped reporting entirely, which the first cannot see, because an expression over a series that no longer exists returns nothing and an empty result does not fire. That pairing is the whole trick, and it applies to every push-based metric you collect.</p>



<p class="wp-block-paragraph">Set the threshold from the deadline the data has to meet, not from how long the job usually takes. Ninety minutes is not &#8220;the run takes an hour so give it some room&#8221;. It should be the point past which someone downstream is materially affected.</p>



<h2 class="wp-block-heading">Troubleshooting a dashboard that shows nothing useful</h2>



<p class="wp-block-paragraph"><strong>The metric never appears in Prometheus.</strong> Work backwards along the path, one hop at a time. Curl the statsd_exporter&#8217;s own metrics endpoint first. If the metric is not there, it never left the orchestrator, so the problem is the config or an allow-list that filtered it out. If it is there but not in Prometheus, the problem is your scrape config. Do not start in Grafana.</p>



<p class="wp-block-paragraph"><strong>The metric appears with no labels, or with the DAG name baked into the metric name.</strong> Your statsd_exporter mapping did not match. Check the wildcard count against the actual dotted name, and check <code>match_metric_type</code>, which is the usual culprit.</p>



<p class="wp-block-paragraph"><strong>Everything is green but the data is old.</strong> Classic Pushgateway staleness. Query <code>push_time_seconds</code> for the affected group and compare it to now. If it has not moved, the job is not running and you are looking at a cached value.</p>



<p class="wp-block-paragraph"><strong>Volume alerts fire every Monday.</strong> You are comparing against the previous day on data with a weekly cycle. Compare against the same day last week using an offset, or exclude weekends from the baseline.</p>



<p class="wp-block-paragraph"><strong>Gaps in the duration graph.</strong> If the metric is a timer sent only on completion, a run that is still going or that failed early sends nothing. That is correct behaviour, and it is why liveness needs its own panel rather than being inferred from the duration series.</p>



<p class="wp-block-paragraph"><strong>Prometheus memory climbing after rollout.</strong> Cardinality. Per-DAG and per-task metrics multiply, and any label carrying a run identifier, timestamp or UUID creates a permanent new series. Tighten the allow list and drop the offending labels.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating job success as data correctness. This is the entire problem, and every other mistake is a variation of it.</li>
<li>Pushing a boolean status instead of a timestamp, so staleness becomes invisible.</li>
<li>Alerting on threshold breaches but never on metric absence.</li>
<li>Building the dashboard before agreeing what &#8220;healthy&#8221; means for each table with the people who consume it.</li>
<li>Instrumenting every pipeline at once instead of the three that matter, then abandoning the project when the noise gets unmanageable.</li>
<li>Putting instance or pod identifiers into Pushgateway grouping labels and accumulating dead series forever.</li>
<li>Alerting on things nobody will act on at 3am. An alert that gets acknowledged and ignored twice has trained your team to ignore it permanently.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Baseline before you threshold. Collect for two or three weeks and look at the actual shape of the data before deciding what abnormal means.</li>
<li>Start with freshness on your three most critical tables. It catches more real problems per line of code than anything else you can instrument.</li>
<li>Define freshness thresholds from the consumer&#8217;s deadline, and write the consumer&#8217;s name next to the threshold so the number has an owner.</li>
<li>Version dashboards and alert rules in Git alongside the pipelines. A dashboard edited live in the UI is a dashboard that will be silently different next quarter.</li>
<li>Emit row counts at stage boundaries as a habit, not as a special case. In versus out at each hop is what turns &#8220;the numbers are wrong&#8221; into &#8220;stage three drops rows&#8221;.</li>
<li>Review alerts monthly and delete the ones that never fired usefully. Alert rules are code and they rot the same way.</li>
<li>Keep lineage accessible but off the main screen. It answers a different question and it competes for attention with the ones you need first.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the difference between pipeline monitoring and data observability?</h3>



<p class="wp-block-paragraph">Monitoring watches the process: did the job run, did it succeed, how long did it take. Observability watches the data the process produced: is it fresh, is there the right amount, has its shape changed. You need both, and the second is the one most teams are missing when they say their dashboards did not catch something.</p>



<h3 class="wp-block-heading">Do I need a commercial data observability platform?</h3>



<p class="wp-block-paragraph">Not to start. Freshness, volume and null-rate checks on your critical tables cover most real incidents and cost an afternoon. Commercial platforms earn their keep when you have hundreds of tables, several teams changing them independently, and nobody with time to hand-write checks. The honest signal that you have outgrown the homegrown version is when maintaining the checks becomes its own recurring task.</p>



<h3 class="wp-block-heading">Should I use StatsD or OpenTelemetry for Airflow metrics?</h3>



<p class="wp-block-paragraph">New stack, OpenTelemetry: you get real labels instead of reconstructing them from dotted names, and traces come along with it. Existing stack with working StatsD mappings, stay put until you have another reason to touch it. Metric coverage and naming differ between the two paths and between Airflow versions, so verify against the metrics reference for your version rather than copying a dashboard from a blog post.</p>



<h3 class="wp-block-heading">How do I monitor a pipeline that only runs weekly?</h3>



<p class="wp-block-paragraph">Same timestamp gauge, much larger threshold, and lean harder on the absence alert. Infrequent pipelines are where Pushgateway staleness does the most damage, because the last successful push can sit there for a fortnight looking entirely healthy. Check <code>push_time_seconds</code> for those groups specifically.</p>



<h3 class="wp-block-heading">What should a data pipeline operations dashboard show first?</h3>



<p class="wp-block-paragraph">Time since last successful load, per critical table, with its threshold visible. That one panel answers the question you will be asked most often, and when it is clean it rules out the entire pipeline layer in a glance.</p>



<h3 class="wp-block-heading">How many alerts is the right number?</h3>



<p class="wp-block-paragraph">Few enough that every page gets investigated. If someone can name an alert they routinely dismiss without looking, you have too many, and the cost is not the noise itself but the habit it builds. Start with freshness and absence per critical pipeline, add volume, and only go further when a real incident shows you a gap.</p>



<h3 class="wp-block-heading">Can I build this without an orchestrator like Airflow?</h3>



<p class="wp-block-paragraph">Yes. Cron jobs, Step Functions, GitHub Actions and shell scripts can all push a timestamp and a row count to a Pushgateway in three lines. The orchestrator gives you the run and timing signals for free, but it is the data-level signals that do the heavy lifting, and those come from querying the warehouse regardless of what triggered the load.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A data pipeline operations dashboard is not a nicer view of your orchestrator. It exists because the orchestrator can only tell you that a process finished, and the question you are actually being asked is whether the data is right.</p>



<p class="wp-block-paragraph">If you build one panel this week, build time since last successful load, sourced from a timestamp gauge, alerted on both threshold and absence. It costs an afternoon, it catches the failure mode that hides longest, and it turns &#8220;is the dashboard broken or just late?&#8221; into a question you can answer before anyone has to ask it.</p>



<h2 class="wp-block-heading">Need help building or fixing this?</h2>



<p class="wp-block-paragraph">I work with teams on pipeline observability and the plumbing underneath it. Things I regularly help with:</p>



<ul class="wp-block-list">
<li>Designing freshness, volume and quality checks for your critical tables, with thresholds derived from real consumer deadlines rather than guesses</li>
<li>Wiring Airflow into Prometheus and Grafana, including the statsd_exporter mappings and the cardinality controls that keep it affordable</li>
<li>Migrating an existing StatsD setup to the OpenTelemetry path without losing your dashboard history</li>
<li>Getting dbt artifacts out of ephemeral pods and into somewhere you can query them, with Elementary or with plain object storage</li>
<li>Auditing an existing dashboard and alert set to find what is stale, what will never fire, and what is quietly training your team to ignore pages</li>
<li>Standing up a self-hosted Prometheus and Grafana stack on a VPS, or sizing a managed alternative if owning the storage is not worth it to you</li>
</ul>



<p class="wp-block-paragraph">If you have a dashboard that went green through an incident, send me the panel queries and the alert rules and I will tell you where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/">Building a Data Pipeline Operations Dashboard That Catches Silent Failures</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 05 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[MWAA]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=145</guid>

					<description><![CDATA[<p>Most API pipeline failures are green DAGs producing incomplete data. A practical guide to running Apache Airflow on AWS for SaaS and API extraction: choosing between MWAA provisioned, MWAA Serverless and self-managed, the pool setting that silently stops throttling when you go deferrable, retry and pagination design, secrets handling, and the four cost lines that actually move.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually lands on a Monday: &#8220;the CRM numbers look wrong again.&#8221; Not missing. Wrong. The dashboard populated, every DAG run is green, and somewhere in the middle of last week&#8217;s data there is a hole where a paginated API returned a 429 and the task treated the empty body as a legitimate final page.</p>



<p class="wp-block-paragraph">That is the shape of most API pipeline incidents. Not a crash. A success that isn&#8217;t one.</p>



<p class="wp-block-paragraph">This post covers running Apache Airflow on AWS specifically for SaaS and API workloads: pulling from HubSpot, Salesforce, Stripe, Zendesk, Shopify, an internal partner API, whatever. It is organised by failure family rather than by feature, because the Airflow documentation already explains what an operator is and does a poor job of explaining which of these things will page you at 3am. I will cover choosing a deployment model, the concurrency trap that catches almost everyone, retry design, incremental state, secrets, and where the money actually goes.</p>



<h2 class="wp-block-heading">Why SaaS and API sources break differently</h2>



<p class="wp-block-paragraph">When your source is a database you control, failure is loud: connection refused, deadlock, disk full. When it is somebody else&#8217;s SaaS API, three things change.</p>



<ul class="wp-block-list">
<li><strong>You are a guest.</strong> The vendor decides your rate limit, and they can change it without telling you. Your pipeline&#8217;s correctness now depends on a number in someone else&#8217;s config file.</li>

<li><strong>Errors arrive as valid HTTP.</strong> A 429, a 200 with a truncated page, a 200 with an error object in the body. Your HTTP client is happy. Your data is not.</li>

<li><strong>Tasks spend most of their life waiting.</strong> API extraction is I/O bound almost end to end. That sounds harmless and is the root of the most expensive mistakes.</li>
</ul>



<h2 class="wp-block-heading">Pick the deployment model before you write a DAG</h2>



<p class="wp-block-paragraph">This decision constrains everything after it and is harder to reverse than people expect. Three realistic options.</p>



<h3 class="wp-block-heading">Amazon MWAA, provisioned</h3>



<p class="wp-block-paragraph">AWS runs the scheduler, web server, workers, triggerer and metadata database on Fargate; you drop DAGs into an S3 bucket and they get picked up.</p>



<p class="wp-block-paragraph">Where it wins: real Airflow, custom providers, custom plugins, full control over environment configuration. If your DAGs need arbitrary Python libraries, this option will not fight you.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: the environment bills by the hour whether or not anything is running. There is no scale to zero on the base environment. If you sync six APIs once a day and each run takes twenty minutes, you are paying for a mostly idle cluster around the clock. The <code>mw1.micro</code> class exists precisely for the small case, but it collapses the scheduler and worker into a single Fargate task and caps worker autoscale low, so treat it as a dev or isolation tier rather than a cheap production tier.</p>



<h3 class="wp-block-heading">Amazon MWAA Serverless</h3>



<p class="wp-block-paragraph">You submit workflow definitions and AWS runs each task in its own Fargate container, billing per task duration with a one-minute minimum rather than per environment hour.</p>



<p class="wp-block-paragraph">Where it wins: spiky or infrequent schedules. If the workload is &#8220;six syncs a day, nothing overnight,&#8221; the cost profile beats a permanently running environment by a wide margin. Each workflow also gets its own IAM execution role, which is a real security improvement over one shared role per environment.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: it leans on declarative YAML workflow definitions based on the DAG Factory format and a curated set of AWS operators. That is a deliberate trade: because the definition is declarative, the service can schedule tasks without executing your DAG code. It also means custom operators, exotic third-party providers and clever Python at parse time are not the sweet spot. It is also available in fewer regions than provisioned MWAA, so check your region before you design around it.</p>



<h3 class="wp-block-heading">Self-managed on ECS, EKS or a VPS</h3>



<p class="wp-block-paragraph">On EKS with the Kubernetes executor you get per-task pods and tight cost control. On a single VPS from a provider like InterServer or Hetzner, a Docker Compose stack with a Postgres metadata database will run a modest set of API syncs for a fraction of any managed price.</p>



<p class="wp-block-paragraph">Where it wins: cost at both extremes, and total control. Where it doesn&#8217;t: you now own metadata database upgrades, major version migrations, log retention and the 2am scheduler restart. Managed Airflow is a bet that your time is worth more than the hourly premium. For a solo engineer with three pipelines that bet often loses; for a data team of eight it usually wins. Astronomer is the main non-AWS managed option worth pricing alongside these.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the throttle that silently stops throttling</h2>



<p class="wp-block-paragraph">You start with a normal setup: an Airflow pool named <code>crm_api</code> with four slots, and every task that touches the vendor assigned to it. Four concurrent requests, comfortably under the vendor&#8217;s limit. This works.</p>



<p class="wp-block-paragraph">Then you notice those tasks spend nearly all their runtime waiting on HTTP, burning worker slots to sit still. So you switch them to deferrable operators. A deferrable task suspends itself while waiting, releases its worker slot, and hands the waiting to the triggerer, which polls asynchronously. Worker pressure drops. Everything looks better.</p>



<p class="wp-block-paragraph">And your rate limiting quietly stops working.</p>



<p class="wp-block-paragraph">By default, a pool does not count tasks in the <em>deferred</em> state as occupying slots. That was deliberate, and the logic is sound in the abstract: a deferred task is not consuming a worker. But if you were using the pool to protect an external API rather than your own workers, it has just stopped doing the job you gave it. Every task can defer at once, and the vendor sees the full fan-out.</p>



<p class="wp-block-paragraph">The fix is a per-pool flag, <code>include_deferred</code>, which tells the scheduler to count deferred tasks against the slot budget. It is off by default. You can set it when editing the pool in the Airflow UI, or through the API.</p>



<p class="wp-block-paragraph">The failure signature is what makes this nasty. Nothing errors. Your DAG gets faster. The vendor starts returning 429s that your retry logic absorbs, and the only symptom is that runs take a little longer and occasionally a page goes missing. Weeks can pass. Two related traps in the same family:</p>



<ul class="wp-block-list">
<li><code>max_active_tasks</code> at the DAG level has the same blind spot with deferred tasks, and there is no equivalent opt-in flag. If you need a hard external concurrency cap, use a pool with <code>include_deferred</code> enabled, not DAG-level concurrency.</li>

<li>On MWAA, the triggerer runs alongside the scheduler on the same Fargate task, so scheduler count and triggerer capacity are linked. If you go heavily deferrable and your deferred tasks start stalling, scheduler capacity is the thing to look at.</li>
</ul>



<h2 class="wp-block-heading">Failure family two: retries that make the outage worse</h2>



<p class="wp-block-paragraph">The default instinct is to set <code>retries</code> high and move on. Against a rate-limited API, a fixed retry delay across many parallel tasks is just a slower version of the same stampede.</p>



<p class="wp-block-paragraph">What you want is exponential backoff with a ceiling. The shape:</p>



<pre class="wp-block-code"><code>from datetime import timedelta

from airflow.sdk import dag, task

@dag(
    schedule="0 5 * * *",
    catchup=False,
    max_active_runs=1,          # never let two runs of this DAG overlap
    default_args={
        "retries": 5,
        "retry_delay": timedelta(seconds=30),
        "retry_exponential_backoff": True,   # 30s, 60s, 120s, 240s...
        "max_retry_delay": timedelta(minutes=15),  # stop doubling here
        "pool": "crm_api",      # shared budget across every task touching this vendor
    },
    tags=["crm", "extract"],
)
def crm_extract():

    @task(max_active_tis_per_dag=4)
    def fetch_page(page_token: str) -&gt; str:
        ...

crm_extract()</code></pre>



<p class="wp-block-paragraph">The lines that matter:</p>



<ul class="wp-block-list">
<li><code>retry_exponential_backoff</code> turns <code>retry_delay</code> into a base rather than a constant, so repeated failures spread out instead of hammering in lockstep.</li>

<li><code>max_retry_delay</code> caps the doubling. Without it, a task that fails five times can sit idle for hours and blow past the window you actually cared about.</li>

<li><code>max_active_runs=1</code> is the one people skip. If a run overruns its schedule, the next one starts anyway, and now two runs are fetching the same pages from the same vendor with the same credentials. This is a common way to trigger a rate limit you have never hit before.</li>

<li><code>max_active_tis_per_dag</code> limits how many instances of that specific task run concurrently across DAG runs, which is the right knob for dynamically mapped extraction tasks.</li>
</ul>



<p class="wp-block-paragraph">One thing Airflow will not do for you: honour a <code>Retry-After</code> header. Airflow&#8217;s retry timing is computed from your config, not from the vendor&#8217;s response. If the API tells you exactly how long to wait, you have to catch that in your own code and sleep or reschedule accordingly. Ignoring a header the vendor bothered to send is a good way to get your API key throttled harder.</p>



<h2 class="wp-block-heading">Failure family three: pagination, cursors and the empty page</h2>



<p class="wp-block-paragraph">Back to the Monday message. The specific bug behind most &#8220;the numbers are wrong but nothing failed&#8221; incidents is a loop that treats any non-error response as a terminating condition. Three rules prevent it:</p>



<ol class="wp-block-list">
<li><strong>Never infer &#8220;done&#8221; from an empty result.</strong> Terminate on the explicit signal the API gives you: a null <code>next_cursor</code>, a missing <code>Link</code> header, a page count. An empty array with a valid cursor still has more data behind it.</li>

<li><strong>Assert the response shape before you use it.</strong> Check the status code explicitly and validate that the fields you depend on exist. A 200 carrying <code>{"error": "..."}</code> should raise, not return zero rows.</li>

<li><strong>Land raw, transform later.</strong> Write the untouched API response to S3 first, then parse from S3. When the vendor changes a field type, you can replay from raw instead of re-extracting from an API that no longer serves that window.</li>
</ol>



<h3 class="wp-block-heading">Where to keep incremental state</h3>



<p class="wp-block-paragraph">The tempting pattern is to store the last-seen timestamp in an Airflow Variable and update it at the end of a run. Do not make that your source of truth. If a run dies midway, the Variable is in an undefined state, and clearing and re-running the DAG will not restore it. Airflow&#8217;s retry and backfill machinery has no idea it exists.</p>



<p class="wp-block-paragraph">Better: make each run&#8217;s window a function of the run itself, and write output to a deterministic, run-scoped location such as <code>s3://bucket/source=crm/dt=&lt;logical-date&gt;/</code>. Re-running the same interval overwrites the same prefix. That is what makes a task idempotent, and idempotency is the difference between &#8220;clear the task and let it rerun&#8221; and a two-hour manual repair.</p>



<p class="wp-block-paragraph">Then overlap your windows deliberately. Many SaaS APIs order results by <em>modified</em> time with eventual consistency, so a record edited at the boundary can appear after you have already moved on. Query a window slightly wider than your schedule interval and rely on an idempotent upsert downstream to absorb the duplicates. Late-arriving data is not an edge case with SaaS sources. It is the normal case.</p>



<p class="wp-block-paragraph">Airflow&#8217;s asset-based scheduling is the clean way to trigger downstream DAGs from this: the extract DAG produces an asset, and the transform DAG runs when the asset updates, rather than being scheduled at a time you hope is late enough.</p>



<h2 class="wp-block-heading">Failure family four: credentials</h2>



<p class="wp-block-paragraph">API tokens rotate, sometimes on the vendor&#8217;s schedule rather than yours. Storing an API key in an Airflow Connection through the UI works, and is the wrong long-term answer: the value lives in the metadata database and there is no rotation story. On AWS, point Airflow&#8217;s secrets backend at AWS Secrets Manager. On MWAA that is an environment configuration option:</p>



<pre class="wp-block-code"><code>secrets.backend
  airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

secrets.backend_kwargs
  {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}</code></pre>



<p class="wp-block-paragraph">With that in place, a connection lookup for <code>crm_default</code> resolves against the secret at <code>airflow/connections/crm_default</code>. Rotating the credential is a Secrets Manager operation with no Airflow deployment involved.</p>



<p class="wp-block-paragraph">Two things to know before you turn it on. First, every connection and variable lookup becomes a Secrets Manager API call, and lookups fall through to the backend before hitting the metadata database, so a DAG that reads a Variable at parse time will generate a call on every parse cycle. Move those reads inside tasks. Second, the environment&#8217;s execution role needs explicit read permission on the relevant secret ARNs, and if you use a customer-managed KMS key, decrypt permission on that key too.</p>



<p class="wp-block-paragraph">Worth knowing if you are on Airflow 3: task code can no longer reach the metadata database directly. All runtime interaction goes through the Task Execution API. If you inherited custom operators that open a session and query Airflow&#8217;s own tables, that is a migration blocker, not a warning.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">Nobody is surprised by the environment line item. They are surprised by the other four.</p>



<ul class="wp-block-list">
<li><strong>Idle time.</strong> A provisioned MWAA environment bills continuously. Compute the ratio of hours billed to hours doing work. If it is bad, that is the argument for MWAA Serverless or for consolidating several thin pipelines into one environment.</li>

<li><strong>NAT Gateway.</strong> This is the classic one. Private-subnet workers calling public SaaS APIs route through a NAT Gateway, which charges hourly <em>and</em> per gigabyte processed. A high-volume extraction pipeline can spend more on NAT than on Airflow. VPC endpoints remove that cost for AWS service traffic, but they do nothing for calls to a third-party API, which is exactly the traffic an API pipeline generates.</li>

<li><strong>CloudWatch Logs.</strong> Task logs go to CloudWatch, and ingestion is billed per gigabyte. Set the Airflow log level per component rather than globally at DEBUG, and set a retention policy on the log groups. The default is to keep logs forever.</li>

<li><strong>S3 requests.</strong> Landing raw API responses one small object per page generates a lot of PUTs. Batch pages into larger objects where you can.</li>
</ul>



<p class="wp-block-paragraph">Rates and dimensions change, so model your own workload against the current pricing page rather than trusting a number from a blog post. The point is knowing which four lines to look at.</p>



<h2 class="wp-block-heading">Troubleshooting Apache Airflow on AWS when API pipelines misbehave</h2>



<h3 class="wp-block-heading">Tasks sit in &#8220;queued&#8221; and never start</h3>



<p class="wp-block-paragraph">Usually a slot problem, not a broken scheduler. Check, in order: is the pool full; has DAG-level <code>max_active_tasks</code> been hit; is worker autoscaling at its configured maximum. On MWAA, the container and queue utilisation metrics published to CloudWatch tell you which of the three it is far faster than reading scheduler logs.</p>



<h3 class="wp-block-heading">DAG file changes don&#8217;t appear</h3>



<p class="wp-block-paragraph">On MWAA, DAGs sync from S3 on an interval; it is not instant. If a file has been there for several minutes and still hasn&#8217;t appeared, it almost always failed to parse. Check the DAG processing logs in CloudWatch rather than the scheduler logs, because a broken import raises there and never reaches the scheduler.</p>



<h3 class="wp-block-heading">A new provider package won&#8217;t install</h3>



<p class="wp-block-paragraph">MWAA installs from your <code>requirements.txt</code> in the DAGs bucket, and from Airflow 2.7.2 onward that file must include a constraint line. Without one, MWAA picks a constraint for you, and pip is free to resolve a provider version that conflicts with the Airflow build in the image.</p>



<pre class="wp-block-code"><code>--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-&lt;AIRFLOW_VERSION&gt;/constraints-&lt;PYTHON_VERSION&gt;.txt"

apache-airflow-providers-http
apache-airflow-providers-salesforce</code></pre>



<p class="wp-block-paragraph">Substitute the literal Airflow version your environment runs and the Python version bundled with it. MWAA does not expand shell variables in that file. Test the requirements file against a local Airflow image before you upload it, because a failed install on MWAA surfaces as a partially working environment rather than a clean error.</p>



<h3 class="wp-block-heading">A backfill is stuck and you need to clear it</h3>



<p class="wp-block-paragraph">You do not need a web login token for this. MWAA exposes the Airflow REST API through a signed AWS API call, so you can drive it from CI or a runbook with normal IAM credentials:</p>



<pre class="wp-block-code"><code>aws mwaa invoke-rest-api 
  --name MyMWAAEnvironment 
  --path "/dags/crm_extract/clearTaskInstances" 
  --method POST 
  --body '{"dry_run": true}'</code></pre>



<p class="wp-block-paragraph">Start with <code>dry_run</code> set to true so the response tells you which task instances would be cleared before you actually clear them. Note that the resource paths differ between Airflow 2 and Airflow 3 environments, so confirm against the API version your environment exposes.</p>



<h3 class="wp-block-heading">Deferred tasks stall forever</h3>



<p class="wp-block-paragraph">If deferred tasks stop resuming while the environment reports healthy, suspect the triggerer rather than your DAG. A triggerer that has lost its ability to process triggers can keep heartbeating normally, so the scheduler sees nothing wrong while every deferred task drifts toward timeout. This class of bug has been fixed and re-fixed upstream, so check your Airflow version&#8217;s release notes before assuming it is your code.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Switching to deferrable operators without enabling <code>include_deferred</code> on the pools that were protecting the API.</li>

<li>Leaving <code>max_active_runs</code> unset, so a slow run and the next scheduled run compete for the same rate limit budget.</li>

<li>Treating an empty response page as the end of pagination.</li>

<li>Storing the incremental watermark in an Airflow Variable and updating it mid-run.</li>

<li>Calling an API or reading a Variable at DAG parse time, which executes on every parse cycle rather than once per run.</li>

<li>Transforming during extraction, so a vendor schema change means re-pulling data the API may no longer serve.</li>

<li>Sizing the environment for peak concurrency when the actual constraint is the vendor&#8217;s rate limit.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>One pool per vendor, sized to their published limit with headroom, and <code>include_deferred</code> enabled on every one of them.</li>

<li>Land raw responses to S3 before parsing. Extraction and transformation are separate tasks with separate failure modes.</li>

<li>Make every task idempotent and window-scoped, so &#8220;clear and rerun&#8221; is always a safe repair.</li>

<li>Overlap extraction windows and deduplicate downstream rather than trusting a vendor&#8217;s timestamps to be exact.</li>

<li>Secrets Manager for credentials, with the execution role scoped to specific secret ARNs.</li>

<li>Alert on row counts and freshness, not just task state. A green DAG that produced 40% of yesterday&#8217;s rows is the failure you actually care about. Shipping Airflow&#8217;s StatsD metrics into Prometheus, Grafana Cloud or Datadog makes that a dashboard rather than a discovery.</li>

<li>Define the environment in Terraform or OpenTofu. Recreating an MWAA environment by hand after a bad configuration change is a bad afternoon.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is MWAA worth it compared to self-hosting Airflow on EC2?</h3>



<p class="wp-block-paragraph">It depends almost entirely on how many people share the platform. MWAA&#8217;s premium buys you managed metadata database upgrades, patched images and version migration support. If one engineer maintains three DAGs, self-hosting on a modest VPS is cheaper and the operational load is real but small. Once several teams depend on the scheduler being up, the premium is easy to justify.</p>



<h3 class="wp-block-heading">Should I use Step Functions instead of Airflow for API pipelines?</h3>



<p class="wp-block-paragraph">Step Functions is genuinely better for event-driven, AWS-service-centric orchestration with modest branching, and it scales to zero. Airflow wins when you need scheduled batch semantics, backfills over historical windows, dependencies between many pipelines, and a UI that non-platform engineers can use to see why last Tuesday failed. Backfill is usually the deciding feature.</p>



<h3 class="wp-block-heading">Do deferrable operators reduce my AWS bill?</h3>



<p class="wp-block-paragraph">On provisioned MWAA, they reduce worker <em>pressure</em>, which reduces autoscaling into additional worker instances. The base environment cost is unchanged. On a Kubernetes executor setup where each task is a pod, the saving is more direct. Either way, do not adopt them purely for cost without revisiting your pool configuration first.</p>



<h3 class="wp-block-heading">How do I handle a vendor with no documented rate limit?</h3>



<p class="wp-block-paragraph">Start conservative, one or two concurrent requests, and instrument the response status codes. Raise the pool size gradually and watch for 429s or rising latency. Latency creeping up under load is often the earlier signal, because some vendors throttle by slowing you down before they start rejecting.</p>



<h3 class="wp-block-heading">Can Airflow read a Retry-After header automatically?</h3>



<p class="wp-block-paragraph">No. Airflow computes retry timing from <code>retry_delay</code> and the backoff settings on the task. If a vendor sends <code>Retry-After</code>, you need to handle it in your own request code or in a custom operator.</p>



<h3 class="wp-block-heading">What breaks when upgrading to Airflow 3?</h3>



<p class="wp-block-paragraph">The big one for API pipelines is that task code can no longer access the metadata database directly; everything goes through the Task Execution API. Imports also move to the <code>airflow.sdk</code> namespace, and several core operators now live in the standard provider package. Audit custom operators first, since that is where direct database access hides. MWAA requires you to be on the latest Airflow 2 minor version before a major upgrade, so plan two steps.</p>



<h3 class="wp-block-heading">How many DAGs can one MWAA environment handle?</h3>



<p class="wp-block-paragraph">The binding constraint is usually the metadata database and scheduler CPU, not DAG count. Watch metadata database memory and scheduler CPU utilisation; when either saturates, you either move up an environment class or split into multiple environments. Splitting also gives you blast-radius isolation, which matters more than people expect.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">Running Apache Airflow on AWS for SaaS and API pipelines is mostly not an Airflow problem. The scheduler works. The operators work. What bites is the gap between &#8220;the task succeeded&#8221; and &#8220;the data is correct,&#8221; and that gap lives in concurrency settings, pagination logic and retry design rather than anywhere Airflow will warn you about.</p>



<p class="wp-block-paragraph">If you take one thing away: <strong>a green DAG is not a signal that your data is complete.</strong> Enable <code>include_deferred</code> on the pools protecting your vendors, terminate pagination on an explicit signal instead of an empty page, make every task idempotent, and alert on row counts. Those four things prevent most of the incidents that never show up as a failed task.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your Airflow pipelines on AWS?</h2>



<p class="wp-block-paragraph">I work with teams running data and API pipelines on AWS, usually somewhere between &#8220;it works but nobody trusts it&#8221; and &#8220;we need to move off cron.&#8221; Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing existing DAGs for silent data loss: pagination logic, retry behaviour, pool and concurrency configuration.</li>

<li>Choosing between MWAA provisioned, MWAA Serverless and self-managed Airflow, with a cost model for your actual schedule rather than a generic comparison.</li>

<li>Building SaaS extraction pipelines that are idempotent and safely re-runnable, landing raw to S3 with incremental windows that survive failure.</li>

<li>Cutting MWAA cost: environment right-sizing, NAT Gateway traffic, CloudWatch log volume and dependency install time.</li>

<li>Airflow 2 to 3 migration audits, focused on custom operators and direct metadata database access.</li>

<li>Data freshness and volume alerting in Grafana or CloudWatch, so you learn about a partial sync before the business does.</li>
</ul>



<p class="wp-block-paragraph">If something specific is broken, send me the DAG file, the task log, or the CloudWatch metrics for the run that went wrong. It is usually faster to look at the real thing than to describe it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
