<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Orchestration | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/orchestration/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/orchestration/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 12:01:15 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Orchestration | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/orchestration/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=215</guid>

					<description><![CDATA[<p>A GraphQL source can hand you a 200 OK, a populated data block, and a quietly broken column in the same response. Here is how to build a GraphQL data ingestion pipeline on AWS that catches partial errors, respects cost-based rate limits, resumes cleanly from a cursor, and notices when the schema moves under you.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The row counts looked fine. That was the problem.</p>



<p class="wp-block-paragraph">A nightly sync from a partner&#8217;s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.</p>



<p class="wp-block-paragraph">The API had started returning a field-level error on that one field. Status code 200. A populated <code>data</code> block. An <code>errors</code> array sitting right next to it. The fetcher parsed <code>data</code>, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.</p>



<p class="wp-block-paragraph">This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.</p>



<h2 class="wp-block-heading">Why GraphQL ingestion breaks the habits REST taught you</h2>



<p class="wp-block-paragraph">Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server&#8217;s business, and rate limits are counted in requests. GraphQL invalidates all three.</p>



<ul class="wp-block-list">
<li><strong>The status code is not the contract.</strong> The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.</li>

<li><strong>You own the response shape.</strong> The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.</li>

<li><strong>Cost is query-shaped, not endpoint-shaped.</strong> One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.</li>
</ul>



<p class="wp-block-paragraph">Each of those becomes a distinct failure family. Let&#8217;s take them in the order they bite.</p>



<h2 class="wp-block-heading">Failure one: the 200 OK that quietly ate a column</h2>



<p class="wp-block-paragraph">This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.</p>



<ul class="wp-block-list">
<li><strong>Request errors</strong> happen before execution starts: syntax errors, validation failures, bad variable coercion. No <code>data</code> key is returned at all. Nothing executed.</li>

<li><strong>Field errors</strong> happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get <code>data</code> and <code>errors</code> together.</li>
</ul>



<p class="wp-block-paragraph">Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL&#8217;s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.</p>



<p class="wp-block-paragraph">The fix is to make the response body the contract. Classify before you commit anything:</p>



<pre class="wp-block-code"><code>import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]</code></pre>



<p class="wp-block-paragraph">Two things worth explaining. The <code>Accept</code> header asks for <code>application/graphql-response+json</code>, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don&#8217;t support it fall back to <code>application/json</code>, so including both is free.</p>



<p class="wp-block-paragraph">The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.</p>



<h3 class="wp-block-heading">Deciding what &#8220;partial&#8221; means for your data</h3>



<p class="wp-block-paragraph">Failing the whole page on any error is the safe default and the one I reach for first. But it&#8217;s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.</p>



<p class="wp-block-paragraph">The workable middle ground is to classify by path. Every error object carries a <code>path</code> array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.</p>



<h2 class="wp-block-heading">Failure two: rate limits that never send a 429</h2>



<p class="wp-block-paragraph">Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.</p>



<p class="wp-block-paragraph">GitHub&#8217;s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and <code>x-ratelimit-remaining</code> set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.</p>



<p class="wp-block-paragraph">The other half of the problem is that request counting is the wrong unit. Shopify&#8217;s GraphQL Admin API and several others price queries by <em>calculated cost</em>, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case <em>requested</em> cost, executes, then refunds the difference between requested and <em>actual</em> cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.</p>



<p class="wp-block-paragraph">The good news is that these APIs tell you the state of the bucket on every response, under the <code>extensions</code> key:</p>



<pre class="wp-block-code"><code>{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant&#8217;s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against <code>currentlyAvailable</code>, and if you&#8217;re short, sleep for the shortfall divided by <code>restoreRate</code>.</p>



<p class="wp-block-paragraph">On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:</p>



<pre class="wp-block-code"><code>query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}</code></pre>



<p class="wp-block-paragraph">Two more things about concurrency. First, if you fan out across Lambda invocations, each worker&#8217;s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it&#8217;s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can&#8217;t go wrong.</p>



<p class="wp-block-paragraph">Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.</p>



<h2 class="wp-block-heading">Failure three: cursors that don&#8217;t survive a restart</h2>



<p class="wp-block-paragraph">Most production GraphQL schemas follow the Relay cursor connections convention: <code>first</code> and <code>after</code> arguments, a <code>pageInfo</code> object with <code>hasNextPage</code> and <code>endCursor</code>, and edges carrying opaque per-item cursors.</p>



<pre class="wp-block-code"><code>query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.</p>



<p class="wp-block-paragraph">The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.</p>



<p class="wp-block-paragraph">Three habits make restarts safe:</p>



<ol class="wp-block-list">
<li><strong>Checkpoint after the write, not after the fetch.</strong> A cursor recorded before the page lands in S3 is a promise you can&#8217;t keep.</li>

<li><strong>Store a high watermark alongside the cursor.</strong> The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.</li>

<li><strong>Overlap the watermark deliberately.</strong> Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.</li>
</ol>



<p class="wp-block-paragraph">A DynamoDB item per source stream is enough state for all of this:</p>



<pre class="wp-block-code"><code>{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "&lt;updatedAt of newest committed record&gt;",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}</code></pre>



<p class="wp-block-paragraph">Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest <code>updatedAt</code>. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you&#8217;ll have traded silent loss for silent inflation.</p>



<h2 class="wp-block-heading">Failure four: the schema moved and nobody told you</h2>



<p class="wp-block-paragraph">GraphQL&#8217;s deprecation story is genuinely better than REST&#8217;s. Fields carry a <code>@deprecated</code> directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.</p>



<p class="wp-block-paragraph">The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.</p>



<p class="wp-block-paragraph">Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.</p>



<p class="wp-block-paragraph">The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring the GraphQL data ingestion pipeline on AWS</h2>



<p class="wp-block-paragraph">With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.</p>



<h3 class="wp-block-heading">Where the fetch loop runs</h3>



<p class="wp-block-paragraph">Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:</p>



<pre class="wp-block-code"><code>"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}</code></pre>



<p class="wp-block-paragraph">The execution role needs four permissions: <code>states:InvokeHTTPEndpoint</code>, <code>events:RetrieveConnectionCredentials</code>, and both <code>secretsmanager:GetSecretValue</code> and <code>secretsmanager:DescribeSecret</code> scoped to the EventBridge-managed secret.</p>



<p class="wp-block-paragraph">It&#8217;s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.</p>



<p class="wp-block-paragraph">The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow&#8217;s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.</p>



<p class="wp-block-paragraph">Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda&#8217;s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.</p>



<p class="wp-block-paragraph">One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.</p>



<h3 class="wp-block-heading">The landing zone</h3>



<p class="wp-block-paragraph">Land raw responses first, unmodified, including the <code>errors</code> and <code>extensions</code> keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.</p>



<ul class="wp-block-list">
<li><strong>Partition by ingest time, not event time.</strong> Event-time partitioning requires reading the record before you know where to put it, which means you can&#8217;t write until you&#8217;ve parsed. Ingest-time keys let the raw write be dumb and fast.</li>

<li><strong>Put the run identifier in the key.</strong> When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.</li>

<li><strong>Define Glue tables explicitly rather than crawling.</strong> You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.</li>

<li><strong>Convert to columnar in the curated layer.</strong> Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.</li>
</ul>



<p class="wp-block-paragraph">If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don&#8217;t need.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Runs succeed but a column is all null.</strong> Field errors. Pull a raw response from the landing zone and look at the <code>errors</code> array. The <code>path</code> tells you exactly which field.</li>

<li><strong>Runs succeed but ingest zero rows.</strong> Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.</li>

<li><strong>Intermittent failures partway through pagination.</strong> Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.</li>

<li><strong>Query fails immediately with no <code>data</code> key.</strong> Request error. Validate against current introspection; something in the schema moved.</li>

<li><strong>Throttled despite staying under the documented limit.</strong> Either concurrent workers are sharing a bucket they each think they own, or you&#8217;re being charged requested cost on near-empty pages.</li>

<li><strong>HTTP Task fails on large pages.</strong> The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.</li>

<li><strong>Row counts drift upward over time.</strong> Overlap without deduplication. Check that the curated layer actually resolves on the primary key.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a 200 as success and never reading the body.</li>

<li>Retrying request errors. They are deterministic; retrying just spends budget to fail identically.</li>

<li>Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.</li>

<li>Checkpointing the cursor before the data is durable.</li>

<li>Requesting the maximum page size on every call while being charged the worst-case cost for it.</li>

<li>Selecting entire objects because it was easier during development.</li>

<li>Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.</li>

<li>Overlapping restart windows with no downstream deduplication.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify every response into clean, partial, or request-failed before anything downstream sees it.</li>

<li>Persist raw responses with <code>errors</code> and <code>extensions</code> intact, then parse from the raw layer.</li>

<li>Alarm on row-count deltas and partial-error rate, not just on execution status.</li>

<li>Read the throttle state the API hands you and back off before you hit the wall, not after.</li>

<li>Keep both a cursor and a watermark, and know which one is authoritative on restart.</li>

<li>Diff the schema on a schedule so deprecations reach you before they reach your data.</li>

<li>Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.</li>

<li>Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you&#8217;re correlating child executions across a fan-out.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Step Functions call a GraphQL API without a Lambda function?</h3>



<p class="wp-block-paragraph">Yes. An HTTP Task using the <code>arn:aws:states:::http:invoke</code> resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.</p>



<h3 class="wp-block-heading">How do I detect GraphQL errors when the status code is always 200?</h3>



<p class="wp-block-paragraph">Inspect the body. If there&#8217;s no <code>data</code> key, nothing executed and you have a request error. If <code>data</code> is present alongside a non-empty <code>errors</code> array, you have a partial response and the data is incomplete. Only a response with <code>data</code> and no errors is unambiguously clean.</p>



<h3 class="wp-block-heading">Does AWS AppSync fit into this?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it&#8217;s what you&#8217;d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else&#8217;s GraphQL API, AppSync isn&#8217;t part of the picture.</p>



<h3 class="wp-block-heading">Should I store raw GraphQL responses or parsed records in S3?</h3>



<p class="wp-block-paragraph">Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they&#8217;re the only way to answer &#8220;was this field null in the source, or did my parser drop it?&#8221; weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.</p>



<h3 class="wp-block-heading">How do I run an incremental sync from a GraphQL API?</h3>



<p class="wp-block-paragraph">Find the schema&#8217;s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you&#8217;re limited to full pulls or webhook-driven change capture.</p>



<h3 class="wp-block-heading">Why am I throttled when I&#8217;m making very few requests?</h3>



<p class="wp-block-paragraph">Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.</p>



<h3 class="wp-block-heading">Is GraphQL a worse ingestion source than REST?</h3>



<p class="wp-block-paragraph">Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.</p>



<p class="wp-block-paragraph">So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with a GraphQL ingestion pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. Specifically:</p>



<ul class="wp-block-list">
<li>Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected</li>

<li>Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts</li>

<li>Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination</li>

<li>Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap</li>

<li>Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard</li>

<li>Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green</li>
</ul>



<p class="wp-block-paragraph">Send me a run log, a state machine definition, or a raw response with an <code>errors</code> array in it, and I&#8217;ll tell you what it&#8217;s costing you.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Data Pipeline Operations Dashboard That Catches Silent Failures</title>
		<link>https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 15 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[OpenTelemetry]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[PromQL]]></category>
		<category><![CDATA[Pushgateway]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[StatsD]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=227</guid>

					<description><![CDATA[<p>Green DAGs and wrong numbers are the same morning. This is how to build a data pipeline operations dashboard around the four questions that actually matter, where each signal comes from, and how to turn the panels into alerts that fire before your users notice.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/">Building a Data Pipeline Operations Dashboard That Catches Silent Failures</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually lands around nine in the morning, from someone in finance or ops: &#8220;Is the revenue dashboard broken, or is it just late?&#8221;</p>



<p class="wp-block-paragraph">So you open the orchestrator. Every task is green. The run started on schedule, finished inside its usual window, exit code zero across the board. Nothing paged. And the numbers are still wrong.</p>



<p class="wp-block-paragraph">That gap is the whole reason to build a data pipeline operations dashboard. Not to watch DAGs go green, which the orchestrator UI already does perfectly well, but to answer the questions the orchestrator structurally cannot answer. This post covers the four signals worth putting on a screen, where each one actually comes from, how to wire them into Prometheus and Grafana without inventing a metrics platform, and how to turn the panels into alerts that fire before a human notices.</p>



<h2 class="wp-block-heading">Why green pipelines still ship bad data</h2>



<p class="wp-block-paragraph">An orchestrator tracks process outcomes. Did the Python exit cleanly, did the SQL statement return without an error, did the container terminate zero. That is a real signal and you should keep it. It is just answering a different question than the one your users care about.</p>



<p class="wp-block-paragraph">Consider what a perfectly successful run looks like when the upstream source has quietly changed:</p>



<ul class="wp-block-list">
<li>An API starts paginating differently and your extractor pulls the first page only. It ran fine. It loaded 500 rows instead of 500,000.</li>
<li>A source column gets renamed and your transform selects it with a <code>coalesce</code> default. It ran fine. Every row now says &#8220;unknown&#8221;.</li>
<li>A partition arrives late, so the incremental filter finds nothing to process. It ran fine. It did nothing.</li>
<li>An upstream team backfills, and your dedupe key no longer holds. It ran fine. Everything is doubled.</li>
</ul>



<p class="wp-block-paragraph">None of those raise an exception. All four are visible in under a second if you are tracking row counts and freshness at the table level. This is the failure family that eats the most incident time, because the clock does not start when the pipeline breaks. It starts when a human notices, which can be days later, and by then the bad data has been copied into reports, cached in a BI extract, and quoted in a meeting.</p>



<h2 class="wp-block-heading">The four questions a data pipeline operations dashboard has to answer</h2>



<p class="wp-block-paragraph">Resist the instinct to start from what your tools can emit. Start from the questions you ask during an incident, in the order you ask them. Everything that does not answer one of these is a second-page panel.</p>



<h3 class="wp-block-heading">Did it run at all?</h3>



<p class="wp-block-paragraph">Sounds trivial, and it is the one most homegrown dashboards get wrong. A panel showing &#8220;last run: success&#8221; does not distinguish between a pipeline that succeeded ten minutes ago and one that succeeded eleven days ago and has not been scheduled since. You want time since last successful completion, as a number that grows, not a status pill that sits there looking calm.</p>



<h3 class="wp-block-heading">Did it finish in time?</h3>



<p class="wp-block-paragraph">Duration matters less than duration against a deadline. A load that takes forty minutes is fine if the report is read at 9am and the run starts at 6am. The same forty minutes is an incident if someone moved the schedule to 8:40. Put the deadline on the panel as a threshold line, not in a runbook. Trending duration also catches slow decay: a stage that creeps from five minutes to fifty usually means data growth, a bad query plan, or resource contention, and it will breach your window eventually.</p>



<h3 class="wp-block-heading">Did the right amount of data arrive?</h3>



<p class="wp-block-paragraph">Rows in versus rows out, per stage. This is the highest value panel per unit of effort on the entire dashboard. A drop from a million rows to twenty thousand is not subtle once you plot it, but it is completely invisible in an exit code. Compare against the same weekday from the previous week rather than against yesterday, because most business data has a weekly shape and Monday-versus-Sunday comparisons generate noise you will learn to ignore.</p>



<h3 class="wp-block-heading">Is the data still the shape you expect?</h3>



<p class="wp-block-paragraph">Null rates, distinct counts on key columns, and schema changes. This is the panel almost nobody builds, and it is where the expensive failures live. A gradual rise in null rate on a joining key means an upstream system is degrading and your joins are silently dropping rows. You do not need statistical anomaly detection to catch that. You need the number plotted over time next to the others.</p>



<p class="wp-block-paragraph">There is a fifth question that belongs on a different surface: what breaks downstream when this table is wrong. That is lineage, and it answers blast radius rather than health. Keep it one click away, not on the operations screen.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where each signal actually comes from</h2>



<p class="wp-block-paragraph">Three sources, three different collection mechanisms. This is the part that takes an afternoon rather than a sprint, provided you do not try to build a unified agent.</p>



<h3 class="wp-block-heading">The orchestrator, for run and timing signals</h3>



<p class="wp-block-paragraph">Airflow has emitted StatsD metrics for a long time, and that path is still the one most deployments run. Airflow speaks StatsD, so you put a <code>statsd_exporter</code> in front of it to translate into Prometheus format. Enable it in the metrics section of your config, or with the equivalent <code>AIRFLOW__METRICS__</code> environment variables if you are on containers:</p>



<pre class="wp-block-code"><code>[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 9125
statsd_prefix = airflow
metrics_allow_list = scheduler,executor,dagrun,pool,triggerer</code></pre>



<p class="wp-block-paragraph">Two things worth knowing before you roll this out. First, <code>metrics_allow_list</code> is not optional at any real scale. Airflow emits per-DAG and per-task metric names, so without a prefix filter your cardinality grows with every DAG anyone adds, and you find out when the time series database starts struggling. Second, Airflow expects the StatsD destination to be resolvable at startup, so a scheduler that boots before the exporter can fail to come up. Order your dependencies accordingly.</p>



<p class="wp-block-paragraph">StatsD has no concept of labels, so the exporter has to reconstruct them from the dotted metric name. That mapping file is the fiddly part:</p>



<pre class="wp-block-code"><code>mappings:
  - match: "*.dag.*.*.duration"
    match_metric_type: observer
    name: "airflow_task_duration"
    labels:
      airflow_id: "$1"
      dag_id: "$2"
      task_id: "$3"</code></pre>



<p class="wp-block-paragraph">The wildcards map positionally onto <code>$1</code>, <code>$2</code>, <code>$3</code>, and <code>match_metric_type</code> has to match what Airflow actually sends, timer versus counter versus gauge. Get that wrong and the metric silently never appears at the exporter, which is a genuinely annoying half hour of debugging. Check the exporter&#8217;s own metrics endpoint before you go looking at Prometheus, because if it is not there it never left Airflow.</p>



<p class="wp-block-paragraph">Airflow also has a native OpenTelemetry path for both metrics and traces, which skips the StatsD translation entirely and gives you real labels rather than reconstructed ones:</p>



<pre class="wp-block-code"><code>[metrics]
otel_on = True
otel_host = otel-collector
otel_port = 4318
otel_prefix = airflow
otel_interval_milliseconds = 30000</code></pre>



<p class="wp-block-paragraph">The port there is the collector&#8217;s OTLP HTTP receiver, and you need an OpenTelemetry Collector or compatible endpoint in the path. If you are standing up a new stack, this is the one I would reach for, because the label problem disappears and traces give you per-task spans under a DAG run parent span, which is worth a lot during a slow-pipeline investigation. If you already have working StatsD mappings and dashboards built on them, migrating is real work for a modest gain, and the honest answer is that it can wait. Metric names do shift between Airflow major versions either way, so pin your dashboard queries to the metrics reference for the version you actually run.</p>



<p class="wp-block-paragraph">Managed Airflow changes the plumbing but not the shape. On Amazon MWAA the metrics land in CloudWatch instead, and Google Cloud Composer and Astronomer each have their own export path. The four questions do not change.</p>



<h3 class="wp-block-heading">The transformation layer, for data-level signals</h3>



<p class="wp-block-paragraph">If you use dbt, most of what you need is already being written to disk and thrown away. Every invocation produces <code>run_results.json</code> in the target directory, containing per-node status and timing, and <code>dbt source freshness</code> produces its own <code>sources.json</code>. The <code>manifest.json</code> holds the dependency graph, which is your lineage.</p>



<p class="wp-block-paragraph">The cheapest useful thing you can do is parse the failures out at the end of the run:</p>



<pre class="wp-block-code"><code>jq -r '.results[]
  | select(.status != "success")
  | [.unique_id, .status, (.execution_time | tostring)]
  | @tsv' target/run_results.json</code></pre>



<p class="wp-block-paragraph">One trap if you run dbt in ephemeral Kubernetes pods: the target directory dies with the pod. Upload the artifacts to object storage as the last step inside the same task that ran dbt, not as a downstream task, or you will find your history is empty exactly when you want it.</p>



<p class="wp-block-paragraph">Elementary is the low-friction option here. It installs as a dbt package with an on-run-end hook that loads the artifacts into your warehouse as ordinary tables, which means your dashboard can query test results and model timings in SQL alongside everything else. Great Expectations and Soda cover the same validation ground with more expressive checks and more setup. The commercial platforms, Monte Carlo and the vendors around it, add automated anomaly detection and column-level lineage across the whole stack, and the genuine case for them is that they find the checks you did not think to write. The trade-off is that you are paying per monitored table for statistical inference, and on a stack of twenty tables that inference is not telling you much that a row-count panel and a null-rate panel would not.</p>



<p class="wp-block-paragraph">Without dbt, you write the check yourself and it is still short. One query per critical table, run on a schedule:</p>



<pre class="wp-block-code"><code>SELECT
  COUNT(*)                                        AS row_count,
  MAX(updated_at)                                 AS last_event,
  SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) AS null_keys
FROM analytics.orders
WHERE loaded_at &gt;= CURRENT_DATE - 1;</code></pre>



<h3 class="wp-block-heading">Batch jobs, and the Pushgateway caveat</h3>



<p class="wp-block-paragraph">Prometheus pulls. Batch jobs finish and vanish long before a scrape interval comes round, so there is nothing to pull from. The Pushgateway exists for exactly this: short-lived service-level jobs push their results to it, and Prometheus scrapes the gateway instead.</p>



<pre class="wp-block-code"><code>cat &lt;&lt;EOF | curl --data-binary @- 
  http://pushgateway:9091/metrics/job/orders_sync
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $(date +%s)
# TYPE pipeline_rows_loaded gauge
pipeline_rows_loaded ${ROWS}
EOF</code></pre>



<p class="wp-block-paragraph">Now the caveat, which is the single most common way a pipeline dashboard lies to you. The Pushgateway is a cache, not an aggregator, and it never forgets. Push a success metric, then have the job stop running entirely, and that success sits there being scraped forever. Your dashboard stays green because the last known value is green, and the fact that nothing has updated it in six days is not something a status panel can see.</p>



<p class="wp-block-paragraph">Two defences, and you want both. First, always push a timestamp gauge rather than a boolean status, so staleness is arithmetic rather than inference. Second, alert on absence as well as on value, because the Pushgateway also exposes <code>push_time_seconds</code> per group, which tells you when anything last arrived regardless of what it said. Prometheus&#8217;s usual safety net, the <code>up</code> metric going to zero when a target disappears, does not apply here: Prometheus is scraping the gateway successfully, and the gateway is happily serving stale data.</p>



<p class="wp-block-paragraph">Also keep instance identifiers out of the grouping labels. Push per service, not per worker, or you accumulate a permanent time series for every container that ever ran.</p>



<h2 class="wp-block-heading">Laying out the dashboard so it works during an incident</h2>



<p class="wp-block-paragraph">The layout constraint is that someone is reading this at 7am on a phone with a stakeholder waiting. Three rows, top to bottom, in the order you need them:</p>



<ol class="wp-block-list">
<li><strong>Freshness across everything.</strong> One panel, one row per critical table, showing time since last successful load against its threshold. If this row is clean, the pipelines are fine and the problem is elsewhere. That is a genuinely useful thing to establish in five seconds.</li>
<li><strong>Volume and duration per pipeline.</strong> Row counts with a week-over-week comparison, and stage duration with the deadline drawn on. This is where you localise a problem to a specific pipeline and stage.</li>
<li><strong>Data quality detail.</strong> Null rates, failed test counts, schema change events. This is where you work out what actually went wrong once you know where.</li>
</ol>



<p class="wp-block-paragraph">The freshness query in PromQL, given the timestamp gauge above:</p>



<pre class="wp-block-code"><code>time() - max by (job) (pipeline_last_success_timestamp_seconds)</code></pre>



<p class="wp-block-paragraph">Grafana renders that nicely as a table with per-row thresholds, which is what makes the top row scannable. If you are self-hosting, Prometheus plus Grafana on a modest VPS from Hetzner, InterServer or DigitalOcean handles a surprising amount of pipeline metrics, because pipeline telemetry is low-volume compared to application metrics. Grafana Cloud&#8217;s free tier is a reasonable alternative if you do not want to own the storage, and Datadog or New Relic make sense if your organisation is already paying for one of them and you would rather have one pane than a better one.</p>



<p class="wp-block-paragraph">Keep it to one screen. Every panel you add past that dilutes the ones that matter, and a dashboard nobody can read at a glance is a dashboard nobody opens.</p>



<h2 class="wp-block-heading">Turning panels into alerts</h2>



<p class="wp-block-paragraph">A dashboard that requires someone to look at it is a dashboard that finds problems the morning after. The freshness panel converts into an alert almost directly:</p>



<pre class="wp-block-code"><code>groups:
  - name: pipelines
    rules:
      - alert: PipelineStale
        expr: time() - max by (job) (pipeline_last_success_timestamp_seconds) &gt; 5400
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} has not completed successfully in over 90 minutes"

      - alert: PipelineMetricsMissing
        expr: absent(pipeline_last_success_timestamp_seconds{job="orders_sync"})
        for: 30m
        labels:
          severity: page
        annotations:
          summary: "No metrics received from orders_sync"</code></pre>



<p class="wp-block-paragraph">The first catches a pipeline that is late or broken. The second catches the one that stopped reporting entirely, which the first cannot see, because an expression over a series that no longer exists returns nothing and an empty result does not fire. That pairing is the whole trick, and it applies to every push-based metric you collect.</p>



<p class="wp-block-paragraph">Set the threshold from the deadline the data has to meet, not from how long the job usually takes. Ninety minutes is not &#8220;the run takes an hour so give it some room&#8221;. It should be the point past which someone downstream is materially affected.</p>



<h2 class="wp-block-heading">Troubleshooting a dashboard that shows nothing useful</h2>



<p class="wp-block-paragraph"><strong>The metric never appears in Prometheus.</strong> Work backwards along the path, one hop at a time. Curl the statsd_exporter&#8217;s own metrics endpoint first. If the metric is not there, it never left the orchestrator, so the problem is the config or an allow-list that filtered it out. If it is there but not in Prometheus, the problem is your scrape config. Do not start in Grafana.</p>



<p class="wp-block-paragraph"><strong>The metric appears with no labels, or with the DAG name baked into the metric name.</strong> Your statsd_exporter mapping did not match. Check the wildcard count against the actual dotted name, and check <code>match_metric_type</code>, which is the usual culprit.</p>



<p class="wp-block-paragraph"><strong>Everything is green but the data is old.</strong> Classic Pushgateway staleness. Query <code>push_time_seconds</code> for the affected group and compare it to now. If it has not moved, the job is not running and you are looking at a cached value.</p>



<p class="wp-block-paragraph"><strong>Volume alerts fire every Monday.</strong> You are comparing against the previous day on data with a weekly cycle. Compare against the same day last week using an offset, or exclude weekends from the baseline.</p>



<p class="wp-block-paragraph"><strong>Gaps in the duration graph.</strong> If the metric is a timer sent only on completion, a run that is still going or that failed early sends nothing. That is correct behaviour, and it is why liveness needs its own panel rather than being inferred from the duration series.</p>



<p class="wp-block-paragraph"><strong>Prometheus memory climbing after rollout.</strong> Cardinality. Per-DAG and per-task metrics multiply, and any label carrying a run identifier, timestamp or UUID creates a permanent new series. Tighten the allow list and drop the offending labels.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating job success as data correctness. This is the entire problem, and every other mistake is a variation of it.</li>
<li>Pushing a boolean status instead of a timestamp, so staleness becomes invisible.</li>
<li>Alerting on threshold breaches but never on metric absence.</li>
<li>Building the dashboard before agreeing what &#8220;healthy&#8221; means for each table with the people who consume it.</li>
<li>Instrumenting every pipeline at once instead of the three that matter, then abandoning the project when the noise gets unmanageable.</li>
<li>Putting instance or pod identifiers into Pushgateway grouping labels and accumulating dead series forever.</li>
<li>Alerting on things nobody will act on at 3am. An alert that gets acknowledged and ignored twice has trained your team to ignore it permanently.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Baseline before you threshold. Collect for two or three weeks and look at the actual shape of the data before deciding what abnormal means.</li>
<li>Start with freshness on your three most critical tables. It catches more real problems per line of code than anything else you can instrument.</li>
<li>Define freshness thresholds from the consumer&#8217;s deadline, and write the consumer&#8217;s name next to the threshold so the number has an owner.</li>
<li>Version dashboards and alert rules in Git alongside the pipelines. A dashboard edited live in the UI is a dashboard that will be silently different next quarter.</li>
<li>Emit row counts at stage boundaries as a habit, not as a special case. In versus out at each hop is what turns &#8220;the numbers are wrong&#8221; into &#8220;stage three drops rows&#8221;.</li>
<li>Review alerts monthly and delete the ones that never fired usefully. Alert rules are code and they rot the same way.</li>
<li>Keep lineage accessible but off the main screen. It answers a different question and it competes for attention with the ones you need first.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the difference between pipeline monitoring and data observability?</h3>



<p class="wp-block-paragraph">Monitoring watches the process: did the job run, did it succeed, how long did it take. Observability watches the data the process produced: is it fresh, is there the right amount, has its shape changed. You need both, and the second is the one most teams are missing when they say their dashboards did not catch something.</p>



<h3 class="wp-block-heading">Do I need a commercial data observability platform?</h3>



<p class="wp-block-paragraph">Not to start. Freshness, volume and null-rate checks on your critical tables cover most real incidents and cost an afternoon. Commercial platforms earn their keep when you have hundreds of tables, several teams changing them independently, and nobody with time to hand-write checks. The honest signal that you have outgrown the homegrown version is when maintaining the checks becomes its own recurring task.</p>



<h3 class="wp-block-heading">Should I use StatsD or OpenTelemetry for Airflow metrics?</h3>



<p class="wp-block-paragraph">New stack, OpenTelemetry: you get real labels instead of reconstructing them from dotted names, and traces come along with it. Existing stack with working StatsD mappings, stay put until you have another reason to touch it. Metric coverage and naming differ between the two paths and between Airflow versions, so verify against the metrics reference for your version rather than copying a dashboard from a blog post.</p>



<h3 class="wp-block-heading">How do I monitor a pipeline that only runs weekly?</h3>



<p class="wp-block-paragraph">Same timestamp gauge, much larger threshold, and lean harder on the absence alert. Infrequent pipelines are where Pushgateway staleness does the most damage, because the last successful push can sit there for a fortnight looking entirely healthy. Check <code>push_time_seconds</code> for those groups specifically.</p>



<h3 class="wp-block-heading">What should a data pipeline operations dashboard show first?</h3>



<p class="wp-block-paragraph">Time since last successful load, per critical table, with its threshold visible. That one panel answers the question you will be asked most often, and when it is clean it rules out the entire pipeline layer in a glance.</p>



<h3 class="wp-block-heading">How many alerts is the right number?</h3>



<p class="wp-block-paragraph">Few enough that every page gets investigated. If someone can name an alert they routinely dismiss without looking, you have too many, and the cost is not the noise itself but the habit it builds. Start with freshness and absence per critical pipeline, add volume, and only go further when a real incident shows you a gap.</p>



<h3 class="wp-block-heading">Can I build this without an orchestrator like Airflow?</h3>



<p class="wp-block-paragraph">Yes. Cron jobs, Step Functions, GitHub Actions and shell scripts can all push a timestamp and a row count to a Pushgateway in three lines. The orchestrator gives you the run and timing signals for free, but it is the data-level signals that do the heavy lifting, and those come from querying the warehouse regardless of what triggered the load.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A data pipeline operations dashboard is not a nicer view of your orchestrator. It exists because the orchestrator can only tell you that a process finished, and the question you are actually being asked is whether the data is right.</p>



<p class="wp-block-paragraph">If you build one panel this week, build time since last successful load, sourced from a timestamp gauge, alerted on both threshold and absence. It costs an afternoon, it catches the failure mode that hides longest, and it turns &#8220;is the dashboard broken or just late?&#8221; into a question you can answer before anyone has to ask it.</p>



<h2 class="wp-block-heading">Need help building or fixing this?</h2>



<p class="wp-block-paragraph">I work with teams on pipeline observability and the plumbing underneath it. Things I regularly help with:</p>



<ul class="wp-block-list">
<li>Designing freshness, volume and quality checks for your critical tables, with thresholds derived from real consumer deadlines rather than guesses</li>
<li>Wiring Airflow into Prometheus and Grafana, including the statsd_exporter mappings and the cardinality controls that keep it affordable</li>
<li>Migrating an existing StatsD setup to the OpenTelemetry path without losing your dashboard history</li>
<li>Getting dbt artifacts out of ephemeral pods and into somewhere you can query them, with Elementary or with plain object storage</li>
<li>Auditing an existing dashboard and alert set to find what is stale, what will never fire, and what is quietly training your team to ignore pages</li>
<li>Standing up a self-hosted Prometheus and Grafana stack on a VPS, or sizing a managed alternative if owning the storage is not worth it to you</li>
</ul>



<p class="wp-block-paragraph">If you have a dashboard that went green through an incident, send me the panel queries and the alert rules and I will tell you where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/">Building a Data Pipeline Operations Dashboard That Catches Silent Failures</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 05 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[MWAA]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=145</guid>

					<description><![CDATA[<p>Most API pipeline failures are green DAGs producing incomplete data. A practical guide to running Apache Airflow on AWS for SaaS and API extraction: choosing between MWAA provisioned, MWAA Serverless and self-managed, the pool setting that silently stops throttling when you go deferrable, retry and pagination design, secrets handling, and the four cost lines that actually move.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually lands on a Monday: &#8220;the CRM numbers look wrong again.&#8221; Not missing. Wrong. The dashboard populated, every DAG run is green, and somewhere in the middle of last week&#8217;s data there is a hole where a paginated API returned a 429 and the task treated the empty body as a legitimate final page.</p>



<p class="wp-block-paragraph">That is the shape of most API pipeline incidents. Not a crash. A success that isn&#8217;t one.</p>



<p class="wp-block-paragraph">This post covers running Apache Airflow on AWS specifically for SaaS and API workloads: pulling from HubSpot, Salesforce, Stripe, Zendesk, Shopify, an internal partner API, whatever. It is organised by failure family rather than by feature, because the Airflow documentation already explains what an operator is and does a poor job of explaining which of these things will page you at 3am. I will cover choosing a deployment model, the concurrency trap that catches almost everyone, retry design, incremental state, secrets, and where the money actually goes.</p>



<h2 class="wp-block-heading">Why SaaS and API sources break differently</h2>



<p class="wp-block-paragraph">When your source is a database you control, failure is loud: connection refused, deadlock, disk full. When it is somebody else&#8217;s SaaS API, three things change.</p>



<ul class="wp-block-list">
<li><strong>You are a guest.</strong> The vendor decides your rate limit, and they can change it without telling you. Your pipeline&#8217;s correctness now depends on a number in someone else&#8217;s config file.</li>

<li><strong>Errors arrive as valid HTTP.</strong> A 429, a 200 with a truncated page, a 200 with an error object in the body. Your HTTP client is happy. Your data is not.</li>

<li><strong>Tasks spend most of their life waiting.</strong> API extraction is I/O bound almost end to end. That sounds harmless and is the root of the most expensive mistakes.</li>
</ul>



<h2 class="wp-block-heading">Pick the deployment model before you write a DAG</h2>



<p class="wp-block-paragraph">This decision constrains everything after it and is harder to reverse than people expect. Three realistic options.</p>



<h3 class="wp-block-heading">Amazon MWAA, provisioned</h3>



<p class="wp-block-paragraph">AWS runs the scheduler, web server, workers, triggerer and metadata database on Fargate; you drop DAGs into an S3 bucket and they get picked up.</p>



<p class="wp-block-paragraph">Where it wins: real Airflow, custom providers, custom plugins, full control over environment configuration. If your DAGs need arbitrary Python libraries, this option will not fight you.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: the environment bills by the hour whether or not anything is running. There is no scale to zero on the base environment. If you sync six APIs once a day and each run takes twenty minutes, you are paying for a mostly idle cluster around the clock. The <code>mw1.micro</code> class exists precisely for the small case, but it collapses the scheduler and worker into a single Fargate task and caps worker autoscale low, so treat it as a dev or isolation tier rather than a cheap production tier.</p>



<h3 class="wp-block-heading">Amazon MWAA Serverless</h3>



<p class="wp-block-paragraph">You submit workflow definitions and AWS runs each task in its own Fargate container, billing per task duration with a one-minute minimum rather than per environment hour.</p>



<p class="wp-block-paragraph">Where it wins: spiky or infrequent schedules. If the workload is &#8220;six syncs a day, nothing overnight,&#8221; the cost profile beats a permanently running environment by a wide margin. Each workflow also gets its own IAM execution role, which is a real security improvement over one shared role per environment.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: it leans on declarative YAML workflow definitions based on the DAG Factory format and a curated set of AWS operators. That is a deliberate trade: because the definition is declarative, the service can schedule tasks without executing your DAG code. It also means custom operators, exotic third-party providers and clever Python at parse time are not the sweet spot. It is also available in fewer regions than provisioned MWAA, so check your region before you design around it.</p>



<h3 class="wp-block-heading">Self-managed on ECS, EKS or a VPS</h3>



<p class="wp-block-paragraph">On EKS with the Kubernetes executor you get per-task pods and tight cost control. On a single VPS from a provider like InterServer or Hetzner, a Docker Compose stack with a Postgres metadata database will run a modest set of API syncs for a fraction of any managed price.</p>



<p class="wp-block-paragraph">Where it wins: cost at both extremes, and total control. Where it doesn&#8217;t: you now own metadata database upgrades, major version migrations, log retention and the 2am scheduler restart. Managed Airflow is a bet that your time is worth more than the hourly premium. For a solo engineer with three pipelines that bet often loses; for a data team of eight it usually wins. Astronomer is the main non-AWS managed option worth pricing alongside these.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the throttle that silently stops throttling</h2>



<p class="wp-block-paragraph">You start with a normal setup: an Airflow pool named <code>crm_api</code> with four slots, and every task that touches the vendor assigned to it. Four concurrent requests, comfortably under the vendor&#8217;s limit. This works.</p>



<p class="wp-block-paragraph">Then you notice those tasks spend nearly all their runtime waiting on HTTP, burning worker slots to sit still. So you switch them to deferrable operators. A deferrable task suspends itself while waiting, releases its worker slot, and hands the waiting to the triggerer, which polls asynchronously. Worker pressure drops. Everything looks better.</p>



<p class="wp-block-paragraph">And your rate limiting quietly stops working.</p>



<p class="wp-block-paragraph">By default, a pool does not count tasks in the <em>deferred</em> state as occupying slots. That was deliberate, and the logic is sound in the abstract: a deferred task is not consuming a worker. But if you were using the pool to protect an external API rather than your own workers, it has just stopped doing the job you gave it. Every task can defer at once, and the vendor sees the full fan-out.</p>



<p class="wp-block-paragraph">The fix is a per-pool flag, <code>include_deferred</code>, which tells the scheduler to count deferred tasks against the slot budget. It is off by default. You can set it when editing the pool in the Airflow UI, or through the API.</p>



<p class="wp-block-paragraph">The failure signature is what makes this nasty. Nothing errors. Your DAG gets faster. The vendor starts returning 429s that your retry logic absorbs, and the only symptom is that runs take a little longer and occasionally a page goes missing. Weeks can pass. Two related traps in the same family:</p>



<ul class="wp-block-list">
<li><code>max_active_tasks</code> at the DAG level has the same blind spot with deferred tasks, and there is no equivalent opt-in flag. If you need a hard external concurrency cap, use a pool with <code>include_deferred</code> enabled, not DAG-level concurrency.</li>

<li>On MWAA, the triggerer runs alongside the scheduler on the same Fargate task, so scheduler count and triggerer capacity are linked. If you go heavily deferrable and your deferred tasks start stalling, scheduler capacity is the thing to look at.</li>
</ul>



<h2 class="wp-block-heading">Failure family two: retries that make the outage worse</h2>



<p class="wp-block-paragraph">The default instinct is to set <code>retries</code> high and move on. Against a rate-limited API, a fixed retry delay across many parallel tasks is just a slower version of the same stampede.</p>



<p class="wp-block-paragraph">What you want is exponential backoff with a ceiling. The shape:</p>



<pre class="wp-block-code"><code>from datetime import timedelta

from airflow.sdk import dag, task

@dag(
    schedule="0 5 * * *",
    catchup=False,
    max_active_runs=1,          # never let two runs of this DAG overlap
    default_args={
        "retries": 5,
        "retry_delay": timedelta(seconds=30),
        "retry_exponential_backoff": True,   # 30s, 60s, 120s, 240s...
        "max_retry_delay": timedelta(minutes=15),  # stop doubling here
        "pool": "crm_api",      # shared budget across every task touching this vendor
    },
    tags=["crm", "extract"],
)
def crm_extract():

    @task(max_active_tis_per_dag=4)
    def fetch_page(page_token: str) -&gt; str:
        ...

crm_extract()</code></pre>



<p class="wp-block-paragraph">The lines that matter:</p>



<ul class="wp-block-list">
<li><code>retry_exponential_backoff</code> turns <code>retry_delay</code> into a base rather than a constant, so repeated failures spread out instead of hammering in lockstep.</li>

<li><code>max_retry_delay</code> caps the doubling. Without it, a task that fails five times can sit idle for hours and blow past the window you actually cared about.</li>

<li><code>max_active_runs=1</code> is the one people skip. If a run overruns its schedule, the next one starts anyway, and now two runs are fetching the same pages from the same vendor with the same credentials. This is a common way to trigger a rate limit you have never hit before.</li>

<li><code>max_active_tis_per_dag</code> limits how many instances of that specific task run concurrently across DAG runs, which is the right knob for dynamically mapped extraction tasks.</li>
</ul>



<p class="wp-block-paragraph">One thing Airflow will not do for you: honour a <code>Retry-After</code> header. Airflow&#8217;s retry timing is computed from your config, not from the vendor&#8217;s response. If the API tells you exactly how long to wait, you have to catch that in your own code and sleep or reschedule accordingly. Ignoring a header the vendor bothered to send is a good way to get your API key throttled harder.</p>



<h2 class="wp-block-heading">Failure family three: pagination, cursors and the empty page</h2>



<p class="wp-block-paragraph">Back to the Monday message. The specific bug behind most &#8220;the numbers are wrong but nothing failed&#8221; incidents is a loop that treats any non-error response as a terminating condition. Three rules prevent it:</p>



<ol class="wp-block-list">
<li><strong>Never infer &#8220;done&#8221; from an empty result.</strong> Terminate on the explicit signal the API gives you: a null <code>next_cursor</code>, a missing <code>Link</code> header, a page count. An empty array with a valid cursor still has more data behind it.</li>

<li><strong>Assert the response shape before you use it.</strong> Check the status code explicitly and validate that the fields you depend on exist. A 200 carrying <code>{"error": "..."}</code> should raise, not return zero rows.</li>

<li><strong>Land raw, transform later.</strong> Write the untouched API response to S3 first, then parse from S3. When the vendor changes a field type, you can replay from raw instead of re-extracting from an API that no longer serves that window.</li>
</ol>



<h3 class="wp-block-heading">Where to keep incremental state</h3>



<p class="wp-block-paragraph">The tempting pattern is to store the last-seen timestamp in an Airflow Variable and update it at the end of a run. Do not make that your source of truth. If a run dies midway, the Variable is in an undefined state, and clearing and re-running the DAG will not restore it. Airflow&#8217;s retry and backfill machinery has no idea it exists.</p>



<p class="wp-block-paragraph">Better: make each run&#8217;s window a function of the run itself, and write output to a deterministic, run-scoped location such as <code>s3://bucket/source=crm/dt=&lt;logical-date&gt;/</code>. Re-running the same interval overwrites the same prefix. That is what makes a task idempotent, and idempotency is the difference between &#8220;clear the task and let it rerun&#8221; and a two-hour manual repair.</p>



<p class="wp-block-paragraph">Then overlap your windows deliberately. Many SaaS APIs order results by <em>modified</em> time with eventual consistency, so a record edited at the boundary can appear after you have already moved on. Query a window slightly wider than your schedule interval and rely on an idempotent upsert downstream to absorb the duplicates. Late-arriving data is not an edge case with SaaS sources. It is the normal case.</p>



<p class="wp-block-paragraph">Airflow&#8217;s asset-based scheduling is the clean way to trigger downstream DAGs from this: the extract DAG produces an asset, and the transform DAG runs when the asset updates, rather than being scheduled at a time you hope is late enough.</p>



<h2 class="wp-block-heading">Failure family four: credentials</h2>



<p class="wp-block-paragraph">API tokens rotate, sometimes on the vendor&#8217;s schedule rather than yours. Storing an API key in an Airflow Connection through the UI works, and is the wrong long-term answer: the value lives in the metadata database and there is no rotation story. On AWS, point Airflow&#8217;s secrets backend at AWS Secrets Manager. On MWAA that is an environment configuration option:</p>



<pre class="wp-block-code"><code>secrets.backend
  airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

secrets.backend_kwargs
  {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}</code></pre>



<p class="wp-block-paragraph">With that in place, a connection lookup for <code>crm_default</code> resolves against the secret at <code>airflow/connections/crm_default</code>. Rotating the credential is a Secrets Manager operation with no Airflow deployment involved.</p>



<p class="wp-block-paragraph">Two things to know before you turn it on. First, every connection and variable lookup becomes a Secrets Manager API call, and lookups fall through to the backend before hitting the metadata database, so a DAG that reads a Variable at parse time will generate a call on every parse cycle. Move those reads inside tasks. Second, the environment&#8217;s execution role needs explicit read permission on the relevant secret ARNs, and if you use a customer-managed KMS key, decrypt permission on that key too.</p>



<p class="wp-block-paragraph">Worth knowing if you are on Airflow 3: task code can no longer reach the metadata database directly. All runtime interaction goes through the Task Execution API. If you inherited custom operators that open a session and query Airflow&#8217;s own tables, that is a migration blocker, not a warning.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">Nobody is surprised by the environment line item. They are surprised by the other four.</p>



<ul class="wp-block-list">
<li><strong>Idle time.</strong> A provisioned MWAA environment bills continuously. Compute the ratio of hours billed to hours doing work. If it is bad, that is the argument for MWAA Serverless or for consolidating several thin pipelines into one environment.</li>

<li><strong>NAT Gateway.</strong> This is the classic one. Private-subnet workers calling public SaaS APIs route through a NAT Gateway, which charges hourly <em>and</em> per gigabyte processed. A high-volume extraction pipeline can spend more on NAT than on Airflow. VPC endpoints remove that cost for AWS service traffic, but they do nothing for calls to a third-party API, which is exactly the traffic an API pipeline generates.</li>

<li><strong>CloudWatch Logs.</strong> Task logs go to CloudWatch, and ingestion is billed per gigabyte. Set the Airflow log level per component rather than globally at DEBUG, and set a retention policy on the log groups. The default is to keep logs forever.</li>

<li><strong>S3 requests.</strong> Landing raw API responses one small object per page generates a lot of PUTs. Batch pages into larger objects where you can.</li>
</ul>



<p class="wp-block-paragraph">Rates and dimensions change, so model your own workload against the current pricing page rather than trusting a number from a blog post. The point is knowing which four lines to look at.</p>



<h2 class="wp-block-heading">Troubleshooting Apache Airflow on AWS when API pipelines misbehave</h2>



<h3 class="wp-block-heading">Tasks sit in &#8220;queued&#8221; and never start</h3>



<p class="wp-block-paragraph">Usually a slot problem, not a broken scheduler. Check, in order: is the pool full; has DAG-level <code>max_active_tasks</code> been hit; is worker autoscaling at its configured maximum. On MWAA, the container and queue utilisation metrics published to CloudWatch tell you which of the three it is far faster than reading scheduler logs.</p>



<h3 class="wp-block-heading">DAG file changes don&#8217;t appear</h3>



<p class="wp-block-paragraph">On MWAA, DAGs sync from S3 on an interval; it is not instant. If a file has been there for several minutes and still hasn&#8217;t appeared, it almost always failed to parse. Check the DAG processing logs in CloudWatch rather than the scheduler logs, because a broken import raises there and never reaches the scheduler.</p>



<h3 class="wp-block-heading">A new provider package won&#8217;t install</h3>



<p class="wp-block-paragraph">MWAA installs from your <code>requirements.txt</code> in the DAGs bucket, and from Airflow 2.7.2 onward that file must include a constraint line. Without one, MWAA picks a constraint for you, and pip is free to resolve a provider version that conflicts with the Airflow build in the image.</p>



<pre class="wp-block-code"><code>--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-&lt;AIRFLOW_VERSION&gt;/constraints-&lt;PYTHON_VERSION&gt;.txt"

apache-airflow-providers-http
apache-airflow-providers-salesforce</code></pre>



<p class="wp-block-paragraph">Substitute the literal Airflow version your environment runs and the Python version bundled with it. MWAA does not expand shell variables in that file. Test the requirements file against a local Airflow image before you upload it, because a failed install on MWAA surfaces as a partially working environment rather than a clean error.</p>



<h3 class="wp-block-heading">A backfill is stuck and you need to clear it</h3>



<p class="wp-block-paragraph">You do not need a web login token for this. MWAA exposes the Airflow REST API through a signed AWS API call, so you can drive it from CI or a runbook with normal IAM credentials:</p>



<pre class="wp-block-code"><code>aws mwaa invoke-rest-api 
  --name MyMWAAEnvironment 
  --path "/dags/crm_extract/clearTaskInstances" 
  --method POST 
  --body '{"dry_run": true}'</code></pre>



<p class="wp-block-paragraph">Start with <code>dry_run</code> set to true so the response tells you which task instances would be cleared before you actually clear them. Note that the resource paths differ between Airflow 2 and Airflow 3 environments, so confirm against the API version your environment exposes.</p>



<h3 class="wp-block-heading">Deferred tasks stall forever</h3>



<p class="wp-block-paragraph">If deferred tasks stop resuming while the environment reports healthy, suspect the triggerer rather than your DAG. A triggerer that has lost its ability to process triggers can keep heartbeating normally, so the scheduler sees nothing wrong while every deferred task drifts toward timeout. This class of bug has been fixed and re-fixed upstream, so check your Airflow version&#8217;s release notes before assuming it is your code.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Switching to deferrable operators without enabling <code>include_deferred</code> on the pools that were protecting the API.</li>

<li>Leaving <code>max_active_runs</code> unset, so a slow run and the next scheduled run compete for the same rate limit budget.</li>

<li>Treating an empty response page as the end of pagination.</li>

<li>Storing the incremental watermark in an Airflow Variable and updating it mid-run.</li>

<li>Calling an API or reading a Variable at DAG parse time, which executes on every parse cycle rather than once per run.</li>

<li>Transforming during extraction, so a vendor schema change means re-pulling data the API may no longer serve.</li>

<li>Sizing the environment for peak concurrency when the actual constraint is the vendor&#8217;s rate limit.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>One pool per vendor, sized to their published limit with headroom, and <code>include_deferred</code> enabled on every one of them.</li>

<li>Land raw responses to S3 before parsing. Extraction and transformation are separate tasks with separate failure modes.</li>

<li>Make every task idempotent and window-scoped, so &#8220;clear and rerun&#8221; is always a safe repair.</li>

<li>Overlap extraction windows and deduplicate downstream rather than trusting a vendor&#8217;s timestamps to be exact.</li>

<li>Secrets Manager for credentials, with the execution role scoped to specific secret ARNs.</li>

<li>Alert on row counts and freshness, not just task state. A green DAG that produced 40% of yesterday&#8217;s rows is the failure you actually care about. Shipping Airflow&#8217;s StatsD metrics into Prometheus, Grafana Cloud or Datadog makes that a dashboard rather than a discovery.</li>

<li>Define the environment in Terraform or OpenTofu. Recreating an MWAA environment by hand after a bad configuration change is a bad afternoon.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is MWAA worth it compared to self-hosting Airflow on EC2?</h3>



<p class="wp-block-paragraph">It depends almost entirely on how many people share the platform. MWAA&#8217;s premium buys you managed metadata database upgrades, patched images and version migration support. If one engineer maintains three DAGs, self-hosting on a modest VPS is cheaper and the operational load is real but small. Once several teams depend on the scheduler being up, the premium is easy to justify.</p>



<h3 class="wp-block-heading">Should I use Step Functions instead of Airflow for API pipelines?</h3>



<p class="wp-block-paragraph">Step Functions is genuinely better for event-driven, AWS-service-centric orchestration with modest branching, and it scales to zero. Airflow wins when you need scheduled batch semantics, backfills over historical windows, dependencies between many pipelines, and a UI that non-platform engineers can use to see why last Tuesday failed. Backfill is usually the deciding feature.</p>



<h3 class="wp-block-heading">Do deferrable operators reduce my AWS bill?</h3>



<p class="wp-block-paragraph">On provisioned MWAA, they reduce worker <em>pressure</em>, which reduces autoscaling into additional worker instances. The base environment cost is unchanged. On a Kubernetes executor setup where each task is a pod, the saving is more direct. Either way, do not adopt them purely for cost without revisiting your pool configuration first.</p>



<h3 class="wp-block-heading">How do I handle a vendor with no documented rate limit?</h3>



<p class="wp-block-paragraph">Start conservative, one or two concurrent requests, and instrument the response status codes. Raise the pool size gradually and watch for 429s or rising latency. Latency creeping up under load is often the earlier signal, because some vendors throttle by slowing you down before they start rejecting.</p>



<h3 class="wp-block-heading">Can Airflow read a Retry-After header automatically?</h3>



<p class="wp-block-paragraph">No. Airflow computes retry timing from <code>retry_delay</code> and the backoff settings on the task. If a vendor sends <code>Retry-After</code>, you need to handle it in your own request code or in a custom operator.</p>



<h3 class="wp-block-heading">What breaks when upgrading to Airflow 3?</h3>



<p class="wp-block-paragraph">The big one for API pipelines is that task code can no longer access the metadata database directly; everything goes through the Task Execution API. Imports also move to the <code>airflow.sdk</code> namespace, and several core operators now live in the standard provider package. Audit custom operators first, since that is where direct database access hides. MWAA requires you to be on the latest Airflow 2 minor version before a major upgrade, so plan two steps.</p>



<h3 class="wp-block-heading">How many DAGs can one MWAA environment handle?</h3>



<p class="wp-block-paragraph">The binding constraint is usually the metadata database and scheduler CPU, not DAG count. Watch metadata database memory and scheduler CPU utilisation; when either saturates, you either move up an environment class or split into multiple environments. Splitting also gives you blast-radius isolation, which matters more than people expect.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">Running Apache Airflow on AWS for SaaS and API pipelines is mostly not an Airflow problem. The scheduler works. The operators work. What bites is the gap between &#8220;the task succeeded&#8221; and &#8220;the data is correct,&#8221; and that gap lives in concurrency settings, pagination logic and retry design rather than anywhere Airflow will warn you about.</p>



<p class="wp-block-paragraph">If you take one thing away: <strong>a green DAG is not a signal that your data is complete.</strong> Enable <code>include_deferred</code> on the pools protecting your vendors, terminate pagination on an explicit signal instead of an empty page, make every task idempotent, and alert on row counts. Those four things prevent most of the incidents that never show up as a failed task.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your Airflow pipelines on AWS?</h2>



<p class="wp-block-paragraph">I work with teams running data and API pipelines on AWS, usually somewhere between &#8220;it works but nobody trusts it&#8221; and &#8220;we need to move off cron.&#8221; Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing existing DAGs for silent data loss: pagination logic, retry behaviour, pool and concurrency configuration.</li>

<li>Choosing between MWAA provisioned, MWAA Serverless and self-managed Airflow, with a cost model for your actual schedule rather than a generic comparison.</li>

<li>Building SaaS extraction pipelines that are idempotent and safely re-runnable, landing raw to S3 with incremental windows that survive failure.</li>

<li>Cutting MWAA cost: environment right-sizing, NAT Gateway traffic, CloudWatch log volume and dependency install time.</li>

<li>Airflow 2 to 3 migration audits, focused on custom operators and direct metadata database access.</li>

<li>Data freshness and volume alerting in Grafana or CloudWatch, so you learn about a partial sync before the business does.</li>
</ul>



<p class="wp-block-paragraph">If something specific is broken, send me the DAG file, the task log, or the CloudWatch metrics for the run that went wrong. It is usually faster to look at the real thing than to describe it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
