<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>MWAA | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/mwaa/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/mwaa/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 12:01:15 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>MWAA | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/mwaa/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</title>
		<link>https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 21 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Amazon Managed Grafana]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Metric Math]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[MWAA]]></category>
		<category><![CDATA[Notification Policies]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=260</guid>

					<description><![CDATA[<p>A Glue job that stops running emits no metrics, so the dashboard stays green and the alert quietly resolves itself. Here is why Grafana monitoring for AWS data pipelines misses that failure, and the heartbeat, metric math and no-data configuration that closes the gap, along with the CloudWatch query costs and IAM boundaries nobody warns you about.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/">Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on a Monday. An analyst pings you: &#8220;the dashboard is showing last week&#8217;s numbers.&#8221; You open Grafana, look at the pipeline board, and every panel is calm. No red. No spikes. Error counts flat at zero.</p>



<p class="wp-block-paragraph">Then you check the Glue job history and the last successful run was six days ago. The job has not failed. It has not run at all. Because it never ran, it never emitted a metric, so CloudWatch recorded nothing and Grafana drew nothing. The panel is not green because the pipeline is healthy. It is green because the pipeline is silent.</p>



<p class="wp-block-paragraph">That gap is the most common problem I see with <strong>Grafana monitoring for AWS data pipelines</strong>, and it is not a Grafana bug. It follows from how AWS batch services emit metrics and how alert engines read an empty result set. This post covers that failure and the three that follow close behind: wrong signal, cost, and access. Then the dashboard layout, the alerting rules, and a troubleshooting list for empty panels.</p>



<h2 class="wp-block-heading">Failure family one: the pipeline stopped and nothing turned red</h2>



<p class="wp-block-paragraph">Application metrics are continuous. A web server emits request counts every minute forever, so a gap in the line genuinely means something is wrong.</p>



<p class="wp-block-paragraph">Pipeline metrics are not like that. Glue reports job metrics only while a job executes, and only if job metrics are enabled and the script initialises a GlueContext. Step Functions publishes execution metrics when executions happen. MWAA publishes DAG metrics when DAGs run. Between runs the metric does not exist. A daily job produces a few minutes of data and twenty-three hours of nothing.</p>



<p class="wp-block-paragraph">So the question that decides whether your monitoring works is: what does your alert engine do with an empty result?</p>



<h3 class="wp-block-heading">No Data and MissingSeries are not the same thing</h3>



<ul class="wp-block-list">
<li><strong>No Data</strong> means the query returned no data points at all. Grafana handles this with a per-rule setting and by default raises a separate alert named <code>DatasourceNoData</code>. That alert does not inherit your rule&#8217;s labels and it fires immediately, skipping the pending period. If your notification policy routes on a team label, it will not match.</li>



<li><strong>MissingSeries</strong> means the query still returns data but one series has disappeared. That is the dead-job case exactly: a query for failed tasks across all Glue jobs still returns rows for the jobs that ran, and the dead one just stops existing. Grafana holds the last state for a couple of evaluation cycles, then evicts the instance with a transition to Normal, annotated with <code>grafana_state_reason</code> set to <code>MissingSeries</code>.</li>
</ul>



<p class="wp-block-paragraph">Read that again, because it is the whole problem. A job that stops running produces an alert that resolves itself and sends a cheerful &#8220;back to normal&#8221; notification. Nobody investigates a resolution.</p>



<h3 class="wp-block-heading">The fix: stop inferring health from absence</h3>



<p class="wp-block-paragraph">Do not build the &#8220;did it run&#8221; signal out of metrics that only exist when it runs. Publish a heartbeat: a custom metric your pipeline writes once, deliberately, at the point you consider the run complete.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-data 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions PipelineName=orders_daily 
  --value 1 
  --unit Count</code></pre>



<p class="wp-block-paragraph">Put that at the end of the Glue script, in the final Step Functions state, or in the last task of the DAG. Now &#8220;did it run&#8221; has a positive signal instead of an inferred one.</p>



<p class="wp-block-paragraph">The heartbeat is still sparse, so you also need to tell the evaluator that absence means failure. In a CloudWatch alarm, that is the missing-data setting: the options are <code>missing</code>, <code>ignore</code>, <code>breaching</code> and <code>notBreaching</code>, and for a heartbeat you want <code>breaching</code>.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name "orders_daily-heartbeat-missing" 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions Name=PipelineName,Value=orders_daily 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 26 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts</code></pre>



<p class="wp-block-paragraph">One subtlety here bites people. CloudWatch looks back over an evaluation range wider than your evaluation periods, and if it finds enough real data points in that wider range, the missing-data setting is ignored entirely. For a once-a-day metric that behaviour can suppress the alarm you just built. The reliable way around it is metric math: fill the gaps before the alarm evaluates anything.</p>



<pre class="wp-block-code"><code>FILL(m1, 0)</code></pre>



<p class="wp-block-paragraph">Now every period has a value, zero means &#8220;no run completed&#8221;, and a plain threshold behaves the way you expected. <code>FILL(m1, REPEAT)</code> and <code>FILL(m1, LINEAR)</code> carry the last known value or interpolate instead, which suits gauges better than heartbeats.</p>



<p class="wp-block-paragraph">On the Grafana side, set the no-data state explicitly on each rule and build a notification policy that catches <code>alertname=DatasourceNoData</code>, otherwise those alerts go nowhere. The heartbeat still matters even with that in place, because it turns a disappearing series into a present series with a bad value, and every rule engine handles that correctly.</p>



<h2 class="wp-block-heading">Failure family two: you are monitoring the compute, not the data</h2>



<p class="wp-block-paragraph">The second failure is subtler. The job runs, succeeds, takes its usual time, and writes nothing, because an upstream API returned an empty page or a partition filter matched no files. Executor CPU and heap will not tell you that. Two signals will.</p>



<p class="wp-block-paragraph"><strong>Freshness</strong> is time since the last successful completion, per pipeline. It maps directly onto &#8220;can I trust the report on my screen&#8221;, which is the only question your stakeholders are asking. Derive it from the heartbeat.</p>



<p class="wp-block-paragraph"><strong>Volume</strong> is rows or bytes written per run, compared against the same run last week. A drop to zero with a successful exit code is the classic empty-load bug. For Spark jobs on Glue the built-in counters give you a usable start:</p>



<ul class="wp-block-list">
<li><code>glue.driver.aggregate.recordsRead</code> and <code>glue.driver.aggregate.bytesRead</code> for input volume</li>



<li><code>glue.driver.aggregate.numFailedTasks</code> for partial failures that do not fail the job</li>



<li><code>glue.driver.s3.filesystem.write_bytes</code> and the <code>glue.ALL</code> equivalents for output across executors</li>
</ul>



<p class="wp-block-paragraph">Glue reports these roughly every thirty seconds as deltas from the previous report, so Sum over the run window is usually what you want rather than Average. Beyond those, the signals I put on a board first are <code>ExecutionsStarted</code> minus <code>ExecutionsSucceeded</code> from the <code>AWS/States</code> namespace, since aborted and timed-out executions land in separate counters and failure count alone misses them; DAG metrics from the <code>AmazonMWAA</code> namespace, remembering that <code>metrics.metrics_allow_list</code> and <code>metrics.metrics_block_list</code> control what reaches CloudWatch at all; consumer lag rather than throughput on streaming ingestion, because a stream will happily accept records nobody is reading; and message age on dead letter queues, which is a near-perfect proxy for &#8220;something is failing repeatedly and nobody noticed&#8221;.</p>



<h2 class="wp-block-heading">Failure family three: the dashboard costs more than the pipeline</h2>



<p class="wp-block-paragraph">Grafana&#8217;s CloudWatch data source uses <code>GetMetricData</code> to fetch samples and <code>ListMetrics</code> when you pick dimensions in the editor. <code>GetMetricData</code> does not qualify for the CloudWatch API free tier, and it is billed per metric requested rather than per API call. One request pulling five hundred metrics is five hundred billable metric requests.</p>



<p class="wp-block-paragraph">Now multiply: panels, times series per panel, times statistics per series, times refresh rate, times every user with the board open, times every alert rule on its own schedule. A wildcard query matching every Glue job in the account looks tidy in the editor and quietly becomes the largest line item in your monitoring bill. Asking for more than five statistics on one metric counts as an extra metric request too.</p>



<p class="wp-block-paragraph">The levers, roughly in order of return:</p>



<ol class="wp-block-list">
<li>Set a sane refresh interval. Metrics that arrive every thirty seconds do not need a five-second refresh, and a daily batch board does not need auto-refresh at all.</li>



<li>Replace wildcard dimension matching with explicit resource lists on always-open boards. Keep wildcards for ad hoc investigation.</li>



<li>Query only the regions you run in. It is easy to leave a data source pointed at a region you migrated out of.</li>



<li>Use template variables instead of one duplicated board per team.</li>



<li>If you pull a large metric catalogue continuously, evaluate CloudWatch metric streams. Streams push through Amazon Data Firehose to a destination you control, and AWS recommends that path for third-party tools with heavy <code>GetMetricData</code> usage. The trade-off is genuine: you lose metric math and Metrics Insights against CloudWatch directly, and you pick up storage and Firehose ingestion costs instead.</li>
</ol>



<p class="wp-block-paragraph">To find which client is driving the spend, enable CloudTrail data events on <code>GetMetricData</code>. It surfaces the calling IAM principal and source address, which is usually enough to identify the forgotten dashboard.</p>



<h2 class="wp-block-heading">Failure family four: access, accounts and regions</h2>



<p class="wp-block-paragraph">Grafana ships CloudWatch support natively, so there is no plugin to install. What there is, is an IAM policy people habitually make too broad or too narrow. This is the read-only set the data source needs:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarms",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetMetricStatistics"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingRegions",
      "Effect": "Allow",
      "Action": "ec2:DescribeRegions",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Prefer an assumed role over long-lived access keys. If Grafana runs on EC2 or ECS in the same account, the instance or task role is cleanest and there is no secret to rotate.</p>



<p class="wp-block-paragraph">Two boundaries to plan for. CloudWatch quotas are defined per account and per region, so a multi-account estate can hit limits in one account while the rest are fine, and each needs its own quota increase. And metrics do not cross regions: a pipeline that reads in one region and writes in another needs both on the board, and the missing half is a common reason a panel looks empty for no obvious cause.</p>



<h2 class="wp-block-heading">Where to run Grafana for this</h2>



<p class="wp-block-paragraph">Three options, and the choice is mostly about viewer count and who carries the operational load.</p>



<ul class="wp-block-list">
<li><strong>Amazon Managed Grafana</strong> bills per active user license per workspace, split into editor and viewer tiers, with at least one editor license required per workspace even if nobody logs in. An active user is anyone who logged in or made an API request during the billing cycle. Excellent for a small engineering team, expensive if you hand read-only boards to a large business audience.</li>



<li><strong>Grafana Cloud</strong> from Grafana Labs bills on usage rather than seats, which flips that calculation. It also brings the AWS observability integrations and Grafana Alloy if you want CloudWatch metrics in a Prometheus-compatible store.</li>



<li><strong>Self-hosted</strong> is cheapest per seat and most work. A small VPS from a provider like Contabo or InterServer runs Grafana for a handful of pipelines comfortably, and you own upgrades, TLS renewal and backups. One licensing note: Grafana&#8217;s open source core moved from Apache 2.0 to AGPLv3, while plugins and agents stayed Apache-licensed. If your legal team has a policy on AGPL, have that conversation before deployment.</li>
</ul>



<h2 class="wp-block-heading">Laying out a board people actually read</h2>



<p class="wp-block-paragraph">Organise by question, not by service. Whoever opens this board on a Monday morning has one question, and the answer should be visible without scrolling.</p>



<ol class="wp-block-list">
<li><strong>Freshness table.</strong> One row per pipeline, hours since last successful completion, threshold colour tied to that pipeline&#8217;s actual SLA. Most viewers read only this row.</li>



<li><strong>Volume against baseline.</strong> Records or bytes written per run, overlaid with the same period a week ago.</li>



<li><strong>Failure counts and durations</strong>, grouped by stage so a slow extract is distinguishable from a slow load.</li>



<li><strong>Resource detail at the bottom.</strong> Executor memory, DPU usage, throttling. Useful during an incident, noise the rest of the time.</li>
</ol>



<p class="wp-block-paragraph">Use a template variable for pipeline name rather than one board per pipeline. It keeps the query count down and new pipelines appear automatically. For cross-pipeline aggregation without fanning out into individual series, CloudWatch Metrics Insights runs a SQL-style query inside CloudWatch and the Grafana data source supports it:</p>



<pre class="wp-block-code"><code>SELECT SUM(ExecutionsFailed)
FROM SCHEMA("AWS/States", StateMachineArn)
GROUP BY StateMachineArn</code></pre>



<p class="wp-block-paragraph">Each <code>GetMetricData</code> operation can carry only one Metrics Insights query, so treat it as a way to consolidate one fan-out, not to run several at once.</p>



<h2 class="wp-block-heading">Troubleshooting: why the panel is empty</h2>



<ol class="wp-block-list">
<li><strong>Check the region on the query, not the data source.</strong> Grafana allows a per-query region override and it is easy to leave one on default.</li>



<li><strong>Widen the time range to seven days.</strong> If data appears, the metric is sparse and you have a heartbeat problem, not a query problem.</li>



<li><strong>Check the statistic.</strong> Sum and Average behave very differently on delta counters, and empty-looking panels are often Average over a window that is mostly gaps.</li>



<li><strong>Check dimensions match exactly.</strong> With Match Exact on, every dimension must be specified, so a metric published with two dimensions returns nothing when queried with one. Turning Match Exact off makes Grafana generate a search expression instead, which is the usual fix.</li>



<li><strong>Check job metrics are enabled.</strong> For Glue you need both the job metrics option and a GlueContext in the script. Without the GlueContext, nothing is emitted regardless of the console setting.</li>



<li><strong>Check IAM.</strong> A role with <code>ListMetrics</code> but not <code>GetMetricData</code> populates every dropdown perfectly and returns no data, which is confusing until you have seen it once.</li>



<li><strong>Check for throttling.</strong> Panels that are intermittently empty under load rather than consistently empty usually mean a CloudWatch API quota for that account and region.</li>
</ol>



<p class="wp-block-paragraph">For alert rules behaving oddly rather than panels, treat the Error state separately from No Data. Grafana raises <code>DatasourceError</code> on evaluation timeouts or repeated query failures, governed by the <code>evaluation_timeout</code> and <code>max_attempts</code> settings. Slow CloudWatch queries under quota pressure trip this, and the resulting alert looks nothing like the one you wrote.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Alerting on failure counts only. A job that never starts cannot fail, so failure-count alerting is blind to the most common outage.</li>



<li>Leaving no-data handling on the default and never routing <code>DatasourceNoData</code>. The alert fires into the void.</li>



<li>Treating a MissingSeries resolution as good news. If alerts resolve without anyone acting, check the reason annotation.</li>



<li>Importing a large community dashboard pack wholesale. Most of those panels use wildcards, and you have just subscribed to a permanent <code>GetMetricData</code> bill for metrics nobody reads.</li>



<li>Monitoring the orchestrator but not the destination. Step Functions reports a clean success while the load that ran inside it wrote zero rows.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Emit a heartbeat from every scheduled pipeline using one consistent namespace and dimension scheme. Consistency is what lets a single dashboard cover everything.</li>



<li>Write down a freshness SLA per pipeline, then set the threshold from it. &#8220;Alert if stale&#8221; only means something once someone says how stale is too stale.</li>



<li>Provision dashboards and alert rules as code in the same repository as the pipeline. A pipeline that ships without its monitoring is not finished.</li>



<li>Route by severity, not by service. Freshness breaches page; executor memory warnings go to a channel.</li>



<li>Break something on purpose to test the alert. Disable the schedule on a non-production pipeline and confirm you get paged. An untested alert is a guess.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a plugin to connect Grafana to CloudWatch?</h3>



<p class="wp-block-paragraph">No. Grafana includes native support for the Amazon CloudWatch data source. Configure credentials or an assumed role, set a default region, and query. The same data source also reaches CloudWatch Logs and X-Ray.</p>



<h3 class="wp-block-heading">Why does my Grafana alert resolve itself when the pipeline is still broken?</h3>



<p class="wp-block-paragraph">Almost certainly MissingSeries eviction. When one series in a multi-series query disappears, Grafana holds the last state briefly and then transitions the instance to Normal with <code>grafana_state_reason</code> set to <code>MissingSeries</code>. It is not treated as No Data. A heartbeat metric with gaps filled to zero stops the series disappearing in the first place.</p>



<h3 class="wp-block-heading">CloudWatch dashboards or Grafana for AWS data pipelines?</h3>



<p class="wp-block-paragraph">CloudWatch dashboards are the shorter path if everything sits in one account and the visualisation options suit you. Grafana earns its place when you need pipeline metrics correlated with a warehouse query, a database, a trace or a business KPI on one screen, and when alerting has to route to several destinations. Most data platforms end up wanting the correlation.</p>



<h3 class="wp-block-heading">How do I monitor a Glue job that succeeds but writes no data?</h3>



<p class="wp-block-paragraph">Alert on output volume rather than exit status. Use the Glue write byte counters, or better, emit your own row-count metric after the write so it reflects what your code believes it wrote. Compare against the equivalent run a week earlier, because fixed thresholds age badly as data grows.</p>



<h3 class="wp-block-heading">Does Grafana querying CloudWatch cost money?</h3>



<p class="wp-block-paragraph">Yes. Grafana uses <code>GetMetricData</code>, which is billed per metric requested and excluded from the CloudWatch API free tier. Cost scales with panels multiplied by series multiplied by refresh frequency multiplied by viewers, so wildcards on an auto-refreshing board are the expensive pattern. Check current rates on the CloudWatch pricing page and do the arithmetic for your own board.</p>



<h3 class="wp-block-heading">Amazon Managed Grafana or self-hosted?</h3>



<p class="wp-block-paragraph">It comes down to viewer count. Managed Grafana bills per active user per workspace, which suits a small engineering team and gets costly when read-only dashboards go out widely. Self-hosting removes the per-seat cost and adds upgrade, TLS and backup work. Grafana Cloud sits between them on usage-based billing. Count likely viewers first, then choose.</p>



<h3 class="wp-block-heading">What is the minimum useful monitoring for a new pipeline?</h3>



<p class="wp-block-paragraph">A heartbeat proving the run completed, a row or byte count proving it moved something, and an alert that fires when the heartbeat is missing for longer than the agreed freshness window. Everything else is refinement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Effective <strong>Grafana monitoring for AWS data pipelines</strong> is not about drawing more metrics. It is about refusing to infer health from absence. Batch pipelines are silent most of the time, and every monitoring system in existence reads silence as calm unless you tell it otherwise.</p>



<p class="wp-block-paragraph">Publish a heartbeat. Fill the gaps with zero. Set your no-data behaviour deliberately. Then alert on freshness and volume, because those are the two things people downstream will notice before you do. Do that, and a green panel means something.</p>



<h2 class="wp-block-heading">Need help building this on your AWS estate?</h2>



<p class="wp-block-paragraph">I design and build monitoring for AWS data platforms. This is the kind of work I take on:</p>



<ul class="wp-block-list">
<li>Heartbeat and freshness instrumentation across Glue, Step Functions, MWAA and Lambda pipelines, with one consistent namespace scheme so a single dashboard covers everything</li>



<li>Grafana dashboards built around freshness and volume rather than executor charts nobody reads</li>



<li>Alert rules and notification policies that handle No Data and MissingSeries correctly, routed to Slack, Teams, Telegram or email by severity</li>



<li>CloudWatch query cost reviews, tracing <code>GetMetricData</code> spend back to the dashboard or rule causing it</li>



<li>Cross-account and multi-region data source setup with least-privilege IAM roles instead of long-lived keys</li>



<li>Grafana deployment and hardening on Amazon Managed Grafana, Grafana Cloud or a self-hosted instance behind TLS</li>
</ul>



<p class="wp-block-paragraph">If you have a pipeline that failed quietly, send me the job definition, the alert rule, or a screenshot of the panel that stayed green. I would rather look at the actual thing than talk in generalities.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/">Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 05 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[MWAA]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=145</guid>

					<description><![CDATA[<p>Most API pipeline failures are green DAGs producing incomplete data. A practical guide to running Apache Airflow on AWS for SaaS and API extraction: choosing between MWAA provisioned, MWAA Serverless and self-managed, the pool setting that silently stops throttling when you go deferrable, retry and pagination design, secrets handling, and the four cost lines that actually move.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually lands on a Monday: &#8220;the CRM numbers look wrong again.&#8221; Not missing. Wrong. The dashboard populated, every DAG run is green, and somewhere in the middle of last week&#8217;s data there is a hole where a paginated API returned a 429 and the task treated the empty body as a legitimate final page.</p>



<p class="wp-block-paragraph">That is the shape of most API pipeline incidents. Not a crash. A success that isn&#8217;t one.</p>



<p class="wp-block-paragraph">This post covers running Apache Airflow on AWS specifically for SaaS and API workloads: pulling from HubSpot, Salesforce, Stripe, Zendesk, Shopify, an internal partner API, whatever. It is organised by failure family rather than by feature, because the Airflow documentation already explains what an operator is and does a poor job of explaining which of these things will page you at 3am. I will cover choosing a deployment model, the concurrency trap that catches almost everyone, retry design, incremental state, secrets, and where the money actually goes.</p>



<h2 class="wp-block-heading">Why SaaS and API sources break differently</h2>



<p class="wp-block-paragraph">When your source is a database you control, failure is loud: connection refused, deadlock, disk full. When it is somebody else&#8217;s SaaS API, three things change.</p>



<ul class="wp-block-list">
<li><strong>You are a guest.</strong> The vendor decides your rate limit, and they can change it without telling you. Your pipeline&#8217;s correctness now depends on a number in someone else&#8217;s config file.</li>

<li><strong>Errors arrive as valid HTTP.</strong> A 429, a 200 with a truncated page, a 200 with an error object in the body. Your HTTP client is happy. Your data is not.</li>

<li><strong>Tasks spend most of their life waiting.</strong> API extraction is I/O bound almost end to end. That sounds harmless and is the root of the most expensive mistakes.</li>
</ul>



<h2 class="wp-block-heading">Pick the deployment model before you write a DAG</h2>



<p class="wp-block-paragraph">This decision constrains everything after it and is harder to reverse than people expect. Three realistic options.</p>



<h3 class="wp-block-heading">Amazon MWAA, provisioned</h3>



<p class="wp-block-paragraph">AWS runs the scheduler, web server, workers, triggerer and metadata database on Fargate; you drop DAGs into an S3 bucket and they get picked up.</p>



<p class="wp-block-paragraph">Where it wins: real Airflow, custom providers, custom plugins, full control over environment configuration. If your DAGs need arbitrary Python libraries, this option will not fight you.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: the environment bills by the hour whether or not anything is running. There is no scale to zero on the base environment. If you sync six APIs once a day and each run takes twenty minutes, you are paying for a mostly idle cluster around the clock. The <code>mw1.micro</code> class exists precisely for the small case, but it collapses the scheduler and worker into a single Fargate task and caps worker autoscale low, so treat it as a dev or isolation tier rather than a cheap production tier.</p>



<h3 class="wp-block-heading">Amazon MWAA Serverless</h3>



<p class="wp-block-paragraph">You submit workflow definitions and AWS runs each task in its own Fargate container, billing per task duration with a one-minute minimum rather than per environment hour.</p>



<p class="wp-block-paragraph">Where it wins: spiky or infrequent schedules. If the workload is &#8220;six syncs a day, nothing overnight,&#8221; the cost profile beats a permanently running environment by a wide margin. Each workflow also gets its own IAM execution role, which is a real security improvement over one shared role per environment.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: it leans on declarative YAML workflow definitions based on the DAG Factory format and a curated set of AWS operators. That is a deliberate trade: because the definition is declarative, the service can schedule tasks without executing your DAG code. It also means custom operators, exotic third-party providers and clever Python at parse time are not the sweet spot. It is also available in fewer regions than provisioned MWAA, so check your region before you design around it.</p>



<h3 class="wp-block-heading">Self-managed on ECS, EKS or a VPS</h3>



<p class="wp-block-paragraph">On EKS with the Kubernetes executor you get per-task pods and tight cost control. On a single VPS from a provider like InterServer or Hetzner, a Docker Compose stack with a Postgres metadata database will run a modest set of API syncs for a fraction of any managed price.</p>



<p class="wp-block-paragraph">Where it wins: cost at both extremes, and total control. Where it doesn&#8217;t: you now own metadata database upgrades, major version migrations, log retention and the 2am scheduler restart. Managed Airflow is a bet that your time is worth more than the hourly premium. For a solo engineer with three pipelines that bet often loses; for a data team of eight it usually wins. Astronomer is the main non-AWS managed option worth pricing alongside these.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the throttle that silently stops throttling</h2>



<p class="wp-block-paragraph">You start with a normal setup: an Airflow pool named <code>crm_api</code> with four slots, and every task that touches the vendor assigned to it. Four concurrent requests, comfortably under the vendor&#8217;s limit. This works.</p>



<p class="wp-block-paragraph">Then you notice those tasks spend nearly all their runtime waiting on HTTP, burning worker slots to sit still. So you switch them to deferrable operators. A deferrable task suspends itself while waiting, releases its worker slot, and hands the waiting to the triggerer, which polls asynchronously. Worker pressure drops. Everything looks better.</p>



<p class="wp-block-paragraph">And your rate limiting quietly stops working.</p>



<p class="wp-block-paragraph">By default, a pool does not count tasks in the <em>deferred</em> state as occupying slots. That was deliberate, and the logic is sound in the abstract: a deferred task is not consuming a worker. But if you were using the pool to protect an external API rather than your own workers, it has just stopped doing the job you gave it. Every task can defer at once, and the vendor sees the full fan-out.</p>



<p class="wp-block-paragraph">The fix is a per-pool flag, <code>include_deferred</code>, which tells the scheduler to count deferred tasks against the slot budget. It is off by default. You can set it when editing the pool in the Airflow UI, or through the API.</p>



<p class="wp-block-paragraph">The failure signature is what makes this nasty. Nothing errors. Your DAG gets faster. The vendor starts returning 429s that your retry logic absorbs, and the only symptom is that runs take a little longer and occasionally a page goes missing. Weeks can pass. Two related traps in the same family:</p>



<ul class="wp-block-list">
<li><code>max_active_tasks</code> at the DAG level has the same blind spot with deferred tasks, and there is no equivalent opt-in flag. If you need a hard external concurrency cap, use a pool with <code>include_deferred</code> enabled, not DAG-level concurrency.</li>

<li>On MWAA, the triggerer runs alongside the scheduler on the same Fargate task, so scheduler count and triggerer capacity are linked. If you go heavily deferrable and your deferred tasks start stalling, scheduler capacity is the thing to look at.</li>
</ul>



<h2 class="wp-block-heading">Failure family two: retries that make the outage worse</h2>



<p class="wp-block-paragraph">The default instinct is to set <code>retries</code> high and move on. Against a rate-limited API, a fixed retry delay across many parallel tasks is just a slower version of the same stampede.</p>



<p class="wp-block-paragraph">What you want is exponential backoff with a ceiling. The shape:</p>



<pre class="wp-block-code"><code>from datetime import timedelta

from airflow.sdk import dag, task

@dag(
    schedule="0 5 * * *",
    catchup=False,
    max_active_runs=1,          # never let two runs of this DAG overlap
    default_args={
        "retries": 5,
        "retry_delay": timedelta(seconds=30),
        "retry_exponential_backoff": True,   # 30s, 60s, 120s, 240s...
        "max_retry_delay": timedelta(minutes=15),  # stop doubling here
        "pool": "crm_api",      # shared budget across every task touching this vendor
    },
    tags=["crm", "extract"],
)
def crm_extract():

    @task(max_active_tis_per_dag=4)
    def fetch_page(page_token: str) -&gt; str:
        ...

crm_extract()</code></pre>



<p class="wp-block-paragraph">The lines that matter:</p>



<ul class="wp-block-list">
<li><code>retry_exponential_backoff</code> turns <code>retry_delay</code> into a base rather than a constant, so repeated failures spread out instead of hammering in lockstep.</li>

<li><code>max_retry_delay</code> caps the doubling. Without it, a task that fails five times can sit idle for hours and blow past the window you actually cared about.</li>

<li><code>max_active_runs=1</code> is the one people skip. If a run overruns its schedule, the next one starts anyway, and now two runs are fetching the same pages from the same vendor with the same credentials. This is a common way to trigger a rate limit you have never hit before.</li>

<li><code>max_active_tis_per_dag</code> limits how many instances of that specific task run concurrently across DAG runs, which is the right knob for dynamically mapped extraction tasks.</li>
</ul>



<p class="wp-block-paragraph">One thing Airflow will not do for you: honour a <code>Retry-After</code> header. Airflow&#8217;s retry timing is computed from your config, not from the vendor&#8217;s response. If the API tells you exactly how long to wait, you have to catch that in your own code and sleep or reschedule accordingly. Ignoring a header the vendor bothered to send is a good way to get your API key throttled harder.</p>



<h2 class="wp-block-heading">Failure family three: pagination, cursors and the empty page</h2>



<p class="wp-block-paragraph">Back to the Monday message. The specific bug behind most &#8220;the numbers are wrong but nothing failed&#8221; incidents is a loop that treats any non-error response as a terminating condition. Three rules prevent it:</p>



<ol class="wp-block-list">
<li><strong>Never infer &#8220;done&#8221; from an empty result.</strong> Terminate on the explicit signal the API gives you: a null <code>next_cursor</code>, a missing <code>Link</code> header, a page count. An empty array with a valid cursor still has more data behind it.</li>

<li><strong>Assert the response shape before you use it.</strong> Check the status code explicitly and validate that the fields you depend on exist. A 200 carrying <code>{"error": "..."}</code> should raise, not return zero rows.</li>

<li><strong>Land raw, transform later.</strong> Write the untouched API response to S3 first, then parse from S3. When the vendor changes a field type, you can replay from raw instead of re-extracting from an API that no longer serves that window.</li>
</ol>



<h3 class="wp-block-heading">Where to keep incremental state</h3>



<p class="wp-block-paragraph">The tempting pattern is to store the last-seen timestamp in an Airflow Variable and update it at the end of a run. Do not make that your source of truth. If a run dies midway, the Variable is in an undefined state, and clearing and re-running the DAG will not restore it. Airflow&#8217;s retry and backfill machinery has no idea it exists.</p>



<p class="wp-block-paragraph">Better: make each run&#8217;s window a function of the run itself, and write output to a deterministic, run-scoped location such as <code>s3://bucket/source=crm/dt=&lt;logical-date&gt;/</code>. Re-running the same interval overwrites the same prefix. That is what makes a task idempotent, and idempotency is the difference between &#8220;clear the task and let it rerun&#8221; and a two-hour manual repair.</p>



<p class="wp-block-paragraph">Then overlap your windows deliberately. Many SaaS APIs order results by <em>modified</em> time with eventual consistency, so a record edited at the boundary can appear after you have already moved on. Query a window slightly wider than your schedule interval and rely on an idempotent upsert downstream to absorb the duplicates. Late-arriving data is not an edge case with SaaS sources. It is the normal case.</p>



<p class="wp-block-paragraph">Airflow&#8217;s asset-based scheduling is the clean way to trigger downstream DAGs from this: the extract DAG produces an asset, and the transform DAG runs when the asset updates, rather than being scheduled at a time you hope is late enough.</p>



<h2 class="wp-block-heading">Failure family four: credentials</h2>



<p class="wp-block-paragraph">API tokens rotate, sometimes on the vendor&#8217;s schedule rather than yours. Storing an API key in an Airflow Connection through the UI works, and is the wrong long-term answer: the value lives in the metadata database and there is no rotation story. On AWS, point Airflow&#8217;s secrets backend at AWS Secrets Manager. On MWAA that is an environment configuration option:</p>



<pre class="wp-block-code"><code>secrets.backend
  airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

secrets.backend_kwargs
  {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}</code></pre>



<p class="wp-block-paragraph">With that in place, a connection lookup for <code>crm_default</code> resolves against the secret at <code>airflow/connections/crm_default</code>. Rotating the credential is a Secrets Manager operation with no Airflow deployment involved.</p>



<p class="wp-block-paragraph">Two things to know before you turn it on. First, every connection and variable lookup becomes a Secrets Manager API call, and lookups fall through to the backend before hitting the metadata database, so a DAG that reads a Variable at parse time will generate a call on every parse cycle. Move those reads inside tasks. Second, the environment&#8217;s execution role needs explicit read permission on the relevant secret ARNs, and if you use a customer-managed KMS key, decrypt permission on that key too.</p>



<p class="wp-block-paragraph">Worth knowing if you are on Airflow 3: task code can no longer reach the metadata database directly. All runtime interaction goes through the Task Execution API. If you inherited custom operators that open a session and query Airflow&#8217;s own tables, that is a migration blocker, not a warning.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">Nobody is surprised by the environment line item. They are surprised by the other four.</p>



<ul class="wp-block-list">
<li><strong>Idle time.</strong> A provisioned MWAA environment bills continuously. Compute the ratio of hours billed to hours doing work. If it is bad, that is the argument for MWAA Serverless or for consolidating several thin pipelines into one environment.</li>

<li><strong>NAT Gateway.</strong> This is the classic one. Private-subnet workers calling public SaaS APIs route through a NAT Gateway, which charges hourly <em>and</em> per gigabyte processed. A high-volume extraction pipeline can spend more on NAT than on Airflow. VPC endpoints remove that cost for AWS service traffic, but they do nothing for calls to a third-party API, which is exactly the traffic an API pipeline generates.</li>

<li><strong>CloudWatch Logs.</strong> Task logs go to CloudWatch, and ingestion is billed per gigabyte. Set the Airflow log level per component rather than globally at DEBUG, and set a retention policy on the log groups. The default is to keep logs forever.</li>

<li><strong>S3 requests.</strong> Landing raw API responses one small object per page generates a lot of PUTs. Batch pages into larger objects where you can.</li>
</ul>



<p class="wp-block-paragraph">Rates and dimensions change, so model your own workload against the current pricing page rather than trusting a number from a blog post. The point is knowing which four lines to look at.</p>



<h2 class="wp-block-heading">Troubleshooting Apache Airflow on AWS when API pipelines misbehave</h2>



<h3 class="wp-block-heading">Tasks sit in &#8220;queued&#8221; and never start</h3>



<p class="wp-block-paragraph">Usually a slot problem, not a broken scheduler. Check, in order: is the pool full; has DAG-level <code>max_active_tasks</code> been hit; is worker autoscaling at its configured maximum. On MWAA, the container and queue utilisation metrics published to CloudWatch tell you which of the three it is far faster than reading scheduler logs.</p>



<h3 class="wp-block-heading">DAG file changes don&#8217;t appear</h3>



<p class="wp-block-paragraph">On MWAA, DAGs sync from S3 on an interval; it is not instant. If a file has been there for several minutes and still hasn&#8217;t appeared, it almost always failed to parse. Check the DAG processing logs in CloudWatch rather than the scheduler logs, because a broken import raises there and never reaches the scheduler.</p>



<h3 class="wp-block-heading">A new provider package won&#8217;t install</h3>



<p class="wp-block-paragraph">MWAA installs from your <code>requirements.txt</code> in the DAGs bucket, and from Airflow 2.7.2 onward that file must include a constraint line. Without one, MWAA picks a constraint for you, and pip is free to resolve a provider version that conflicts with the Airflow build in the image.</p>



<pre class="wp-block-code"><code>--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-&lt;AIRFLOW_VERSION&gt;/constraints-&lt;PYTHON_VERSION&gt;.txt"

apache-airflow-providers-http
apache-airflow-providers-salesforce</code></pre>



<p class="wp-block-paragraph">Substitute the literal Airflow version your environment runs and the Python version bundled with it. MWAA does not expand shell variables in that file. Test the requirements file against a local Airflow image before you upload it, because a failed install on MWAA surfaces as a partially working environment rather than a clean error.</p>



<h3 class="wp-block-heading">A backfill is stuck and you need to clear it</h3>



<p class="wp-block-paragraph">You do not need a web login token for this. MWAA exposes the Airflow REST API through a signed AWS API call, so you can drive it from CI or a runbook with normal IAM credentials:</p>



<pre class="wp-block-code"><code>aws mwaa invoke-rest-api 
  --name MyMWAAEnvironment 
  --path "/dags/crm_extract/clearTaskInstances" 
  --method POST 
  --body '{"dry_run": true}'</code></pre>



<p class="wp-block-paragraph">Start with <code>dry_run</code> set to true so the response tells you which task instances would be cleared before you actually clear them. Note that the resource paths differ between Airflow 2 and Airflow 3 environments, so confirm against the API version your environment exposes.</p>



<h3 class="wp-block-heading">Deferred tasks stall forever</h3>



<p class="wp-block-paragraph">If deferred tasks stop resuming while the environment reports healthy, suspect the triggerer rather than your DAG. A triggerer that has lost its ability to process triggers can keep heartbeating normally, so the scheduler sees nothing wrong while every deferred task drifts toward timeout. This class of bug has been fixed and re-fixed upstream, so check your Airflow version&#8217;s release notes before assuming it is your code.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Switching to deferrable operators without enabling <code>include_deferred</code> on the pools that were protecting the API.</li>

<li>Leaving <code>max_active_runs</code> unset, so a slow run and the next scheduled run compete for the same rate limit budget.</li>

<li>Treating an empty response page as the end of pagination.</li>

<li>Storing the incremental watermark in an Airflow Variable and updating it mid-run.</li>

<li>Calling an API or reading a Variable at DAG parse time, which executes on every parse cycle rather than once per run.</li>

<li>Transforming during extraction, so a vendor schema change means re-pulling data the API may no longer serve.</li>

<li>Sizing the environment for peak concurrency when the actual constraint is the vendor&#8217;s rate limit.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>One pool per vendor, sized to their published limit with headroom, and <code>include_deferred</code> enabled on every one of them.</li>

<li>Land raw responses to S3 before parsing. Extraction and transformation are separate tasks with separate failure modes.</li>

<li>Make every task idempotent and window-scoped, so &#8220;clear and rerun&#8221; is always a safe repair.</li>

<li>Overlap extraction windows and deduplicate downstream rather than trusting a vendor&#8217;s timestamps to be exact.</li>

<li>Secrets Manager for credentials, with the execution role scoped to specific secret ARNs.</li>

<li>Alert on row counts and freshness, not just task state. A green DAG that produced 40% of yesterday&#8217;s rows is the failure you actually care about. Shipping Airflow&#8217;s StatsD metrics into Prometheus, Grafana Cloud or Datadog makes that a dashboard rather than a discovery.</li>

<li>Define the environment in Terraform or OpenTofu. Recreating an MWAA environment by hand after a bad configuration change is a bad afternoon.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is MWAA worth it compared to self-hosting Airflow on EC2?</h3>



<p class="wp-block-paragraph">It depends almost entirely on how many people share the platform. MWAA&#8217;s premium buys you managed metadata database upgrades, patched images and version migration support. If one engineer maintains three DAGs, self-hosting on a modest VPS is cheaper and the operational load is real but small. Once several teams depend on the scheduler being up, the premium is easy to justify.</p>



<h3 class="wp-block-heading">Should I use Step Functions instead of Airflow for API pipelines?</h3>



<p class="wp-block-paragraph">Step Functions is genuinely better for event-driven, AWS-service-centric orchestration with modest branching, and it scales to zero. Airflow wins when you need scheduled batch semantics, backfills over historical windows, dependencies between many pipelines, and a UI that non-platform engineers can use to see why last Tuesday failed. Backfill is usually the deciding feature.</p>



<h3 class="wp-block-heading">Do deferrable operators reduce my AWS bill?</h3>



<p class="wp-block-paragraph">On provisioned MWAA, they reduce worker <em>pressure</em>, which reduces autoscaling into additional worker instances. The base environment cost is unchanged. On a Kubernetes executor setup where each task is a pod, the saving is more direct. Either way, do not adopt them purely for cost without revisiting your pool configuration first.</p>



<h3 class="wp-block-heading">How do I handle a vendor with no documented rate limit?</h3>



<p class="wp-block-paragraph">Start conservative, one or two concurrent requests, and instrument the response status codes. Raise the pool size gradually and watch for 429s or rising latency. Latency creeping up under load is often the earlier signal, because some vendors throttle by slowing you down before they start rejecting.</p>



<h3 class="wp-block-heading">Can Airflow read a Retry-After header automatically?</h3>



<p class="wp-block-paragraph">No. Airflow computes retry timing from <code>retry_delay</code> and the backoff settings on the task. If a vendor sends <code>Retry-After</code>, you need to handle it in your own request code or in a custom operator.</p>



<h3 class="wp-block-heading">What breaks when upgrading to Airflow 3?</h3>



<p class="wp-block-paragraph">The big one for API pipelines is that task code can no longer access the metadata database directly; everything goes through the Task Execution API. Imports also move to the <code>airflow.sdk</code> namespace, and several core operators now live in the standard provider package. Audit custom operators first, since that is where direct database access hides. MWAA requires you to be on the latest Airflow 2 minor version before a major upgrade, so plan two steps.</p>



<h3 class="wp-block-heading">How many DAGs can one MWAA environment handle?</h3>



<p class="wp-block-paragraph">The binding constraint is usually the metadata database and scheduler CPU, not DAG count. Watch metadata database memory and scheduler CPU utilisation; when either saturates, you either move up an environment class or split into multiple environments. Splitting also gives you blast-radius isolation, which matters more than people expect.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">Running Apache Airflow on AWS for SaaS and API pipelines is mostly not an Airflow problem. The scheduler works. The operators work. What bites is the gap between &#8220;the task succeeded&#8221; and &#8220;the data is correct,&#8221; and that gap lives in concurrency settings, pagination logic and retry design rather than anywhere Airflow will warn you about.</p>



<p class="wp-block-paragraph">If you take one thing away: <strong>a green DAG is not a signal that your data is complete.</strong> Enable <code>include_deferred</code> on the pools protecting your vendors, terminate pagination on an explicit signal instead of an empty page, make every task idempotent, and alert on row counts. Those four things prevent most of the incidents that never show up as a failed task.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your Airflow pipelines on AWS?</h2>



<p class="wp-block-paragraph">I work with teams running data and API pipelines on AWS, usually somewhere between &#8220;it works but nobody trusts it&#8221; and &#8220;we need to move off cron.&#8221; Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing existing DAGs for silent data loss: pagination logic, retry behaviour, pool and concurrency configuration.</li>

<li>Choosing between MWAA provisioned, MWAA Serverless and self-managed Airflow, with a cost model for your actual schedule rather than a generic comparison.</li>

<li>Building SaaS extraction pipelines that are idempotent and safely re-runnable, landing raw to S3 with incremental windows that survive failure.</li>

<li>Cutting MWAA cost: environment right-sizing, NAT Gateway traffic, CloudWatch log volume and dependency install time.</li>

<li>Airflow 2 to 3 migration audits, focused on custom operators and direct metadata database access.</li>

<li>Data freshness and volume alerting in Grafana or CloudWatch, so you learn about a partial sync before the business does.</li>
</ul>



<p class="wp-block-paragraph">If something specific is broken, send me the DAG file, the task log, or the CloudWatch metrics for the run that went wrong. It is usually faster to look at the real thing than to describe it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
