<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Schema Drift | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/schema-drift/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/schema-drift/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 15 Sep 2026 06:47:24 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Schema Drift | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/schema-drift/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Turning External Documents Into Structured Data With Amazon Textract</title>
		<link>https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 22 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=390</guid>

					<description><![CDATA[<p>Amazon Textract almost never fails loudly. A working guide to turning external PDFs and scans into structured data: picking the right operation per document family, parsing the block graph, routing on per-field confidence, and catching the limits that silently truncate your records.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/">Turning External Documents Into Structured Data With Amazon Textract</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A vendor sends over a year of delivery notes. It arrives as one PDF, several hundred pages, scanned on a machine that was clearly overdue for service. Someone asks how long it will take to get the reference numbers into the warehouse table, and the honest answer is that the OCR call is the easy part.</p>



<p class="wp-block-paragraph">Document pipelines are not hard because optical character recognition is inaccurate. Amazon Textract reads text well. They are hard because everything after the API call belongs to you, and because Textract almost never fails loudly. You get a 200 response, a list of blocks, a confidence score, and a record with an empty field where the purchase order number should have been. Nothing alerts. The row lands in the warehouse with a null, and someone in finance notices three weeks later while reconciling.</p>



<p class="wp-block-paragraph">That silent gap is the whole problem. Getting reliable Amazon Textract structured data out of files you did not design has less to do with OCR accuracy than with everything around it: which operation to call for which document family, why the response is a graph rather than a record, how to route on confidence instead of trusting it, and the limits that quietly truncate your data. All of it assumes documents you cannot ask anyone to change.</p>



<h2 class="wp-block-heading">Why external documents break pipelines that passed testing</h2>



<p class="wp-block-paragraph">Internal documents are generated by a template you control. External documents vary along three axes at once, and each one has its own failure mode:</p>



<ul class="wp-block-list">
<li><strong>Container.</strong> One logical document per file, or forty stapled into one PDF, or a single page split across three JPEGs.</li>

<li><strong>Text origin.</strong> Born-digital PDFs with a real text layer, flatbed scans, and photographs taken at an angle in bad light. Same file extension, wildly different accuracy.</li>

<li><strong>Layout stability.</strong> The vendor redesigns their invoice template and moves the PO number from the header to a footer block. Nobody tells you.</li>
</ul>



<p class="wp-block-paragraph">You cannot fix any of that upstream. You can only detect it at ingest and route accordingly. The sections below are organized by the failure family each one produces.</p>



<h2 class="wp-block-heading">Failure family one: the file is out of spec before Textract sees it</h2>



<p class="wp-block-paragraph">Textract accepts JPEG, PNG, PDF, and TIFF. It does not support XFA-based PDFs, and it will not open a password-protected PDF. Images are accepted up to 10,000 pixels on any side. For PDFs, the maximum page dimensions are 40 inches and 9,000 points.</p>



<p class="wp-block-paragraph">Two more constraints catch people out. Text detection covers English, French, German, Italian, Portuguese, and Spanish, and several capabilities are narrower still: handwriting, invoices and receipts, identity documents, and Queries are English only. Vertical text, common in Japanese and Chinese documents, is not supported at all. Rotation is fine, including heavy in-plane rotation. Vertical writing direction is not.</p>



<p class="wp-block-paragraph">Build an ingest gate that checks encryption, page count, byte size, and dimensions before it spends an API call, and give each rejection a reason code a human can act on. Then catch exceptions properly instead of wrapping the call in a bare retry. <code>BadDocumentException</code> means Textract could not read the file, and retrying fails identically. <code>DocumentTooLargeException</code> means route to the asynchronous path or split the file. <code>InvalidS3ObjectException</code> is usually IAM or a bucket-region mismatch, not a document problem. Only <code>InternalServerErrorException</code> deserves a retry with backoff.</p>



<h2 class="wp-block-heading">Failure family two: synchronous versus asynchronous is a correctness gate</h2>



<p class="wp-block-paragraph">This is the one that silently eats data, so it goes early.</p>



<p class="wp-block-paragraph">The synchronous operations, <code>DetectDocumentText</code> and <code>AnalyzeDocument</code>, cap at 10 MB in memory. For PDF and TIFF input they also cap at <strong>one page</strong>. Not one page per call with automatic paging. One page, full stop. Hand a synchronous call a 40-page PDF and you get page one, and a pipeline that reports success.</p>



<p class="wp-block-paragraph">The asynchronous operations are a different shape. PDF and TIFF go up to 500 MB and 3,000 pages, while JPEG and PNG stay at 10 MB. You start a job, Textract works in the background, and you collect results by job ID.</p>



<p class="wp-block-paragraph">A synchronous call looks like this. The document lives in S3, and you name the features you want:</p>



<pre class="wp-block-code"><code>aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["FORMS","TABLES"]' 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">The asynchronous version returns a job identifier immediately and publishes completion to an SNS topic. That is what you want in production, because polling a job inside a Lambda function means paying for a function that sleeps:</p>



<pre class="wp-block-code"><code>import boto3

textract = boto3.client("textract")

start = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "my-doc-bucket", "Name": "inbound/batch-0042.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:us-east-1:111122223333:textract-jobs",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublishRole",
    },
)

job_id = start["JobId"]</code></pre>



<p class="wp-block-paragraph">Results come back paginated, and forgetting the continuation token is a second, quieter way to lose most of a file:</p>



<pre class="wp-block-code"><code>blocks = []
next_token = None

while True:
    kwargs = {"JobId": job_id}
    if next_token:
        kwargs["NextToken"] = next_token

    page = textract.get_document_analysis(**kwargs)
    blocks.extend(page["Blocks"])

    next_token = page.get("NextToken")
    if not next_token:
        break</code></pre>



<p class="wp-block-paragraph">Check <code>JobStatus</code> before you trust <code>Blocks</code>. A job can come back partially successful, which means some pages processed and some did not. If you treat that as success you will publish a document that is missing its middle.</p>



<h2 class="wp-block-heading">Failure family three: the response is a graph, not a record</h2>



<p class="wp-block-paragraph">Textract does not return fields. It returns a flat list of <code>Block</code> objects, each with an ID and relationships pointing at other IDs. Reassembling those into a row is your code, and it is where most of the real work lives.</p>



<p class="wp-block-paragraph">For form data, the shape is specific. A key-value pair is two blocks of type <code>KEY_VALUE_SET</code>, distinguished by their <code>EntityTypes</code> array containing either <code>KEY</code> or <code>VALUE</code>. The key block carries a relationship of type <code>VALUE</code> holding the ID of its value block, and a relationship of type <code>CHILD</code> holding the IDs of the <code>WORD</code> blocks that spell out the label. The value block carries only <code>CHILD</code> relationships pointing at its own words. So reading one field means three hops.</p>



<pre class="wp-block-code"><code>blocks_by_id = {b["Id"]: b for b in blocks}

def child_text(block):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for cid in rel["Ids"]:
            child = blocks_by_id[cid]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)

fields = {}
for b in blocks:
    if b["BlockType"] != "KEY_VALUE_SET":
        continue
    if "KEY" not in b.get("EntityTypes", []):
        continue

    value_text = ""
    for rel in b.get("Relationships", []):
        if rel["Type"] == "VALUE":
            for vid in rel["Ids"]:
                value_text = child_text(blocks_by_id[vid])

    fields[child_text(b)] = (value_text, b["Confidence"])</code></pre>



<p class="wp-block-paragraph">Two things about that snippet. The <code>SELECTION_ELEMENT</code> branch matters because checkboxes and radio buttons are values too, and their content is a selection status rather than text. And the confidence on the key block is not a separate key confidence: Textract returns the same value for both halves of a pair. If you planned to compare label confidence against value confidence to catch mismatches, that signal does not exist.</p>



<p class="wp-block-paragraph">Tables follow the same pattern with different block types, and cells carry row and column indices plus span values for merged cells. Layout adds another set, including title, header, footer, section header, page number, list, and figure blocks, and it sequences text into human reading order. That last property is why Layout is worth requesting when the output is headed for a language model rather than a database: chunking on reading order beats chunking on raw line order.</p>



<p class="wp-block-paragraph">Before hand-rolling any of this, look at the AWS-maintained parsers. The <code>amazon-textract-response-parser</code> package and the higher-level <code>amazon-textract-textractor</code> library both turn the block graph into objects with sensible properties. Writing your own traversal is a good way to understand the model and a poor way to spend two sprints.</p>



<h2 class="wp-block-heading">Getting Amazon Textract structured data out of the right API</h2>



<p class="wp-block-paragraph">Textract is a family of operations, not one endpoint, and the biggest single accuracy win available is usually picking a narrower one. Features are billed per page and they stack, so requesting <code>TABLES</code> on a corpus with no tables costs money and returns noise.</p>



<ul class="wp-block-list">
<li><strong>DetectDocumentText.</strong> Raw text and geometry. Use it when you only need a search index or a text layer.</li>

<li><strong>AnalyzeDocument with FORMS.</strong> Key-value pairs discovered from labels on the page. Works when the label is physically present and stable.</li>

<li><strong>AnalyzeDocument with TABLES.</strong> Grid reconstruction with row and column indices. The right call for line-item grids and statements.</li>

<li><strong>AnalyzeDocument with QUERIES.</strong> You ask natural language questions and get answers back. This is the fix for layout drift.</li>

<li><strong>AnalyzeDocument with LAYOUT and SIGNATURES.</strong> Reading order and structural elements; signature locations for compliance checks.</li>

<li><strong>AnalyzeExpense.</strong> Purpose-built for invoices and receipts, with normalized field types.</li>

<li><strong>AnalyzeID.</strong> Purpose-built for identity documents.</li>
</ul>



<h3 class="wp-block-heading">Why Queries beats FORMS on documents you don&#8217;t control</h3>



<p class="wp-block-paragraph">FORMS gives you whatever labels the document happens to contain. That means your downstream schema is hostage to a vendor&#8217;s wording. When they change &#8220;P.O. No.&#8221; to &#8220;Order Reference&#8221;, your parser looks for a key that no longer exists and writes a null.</p>



<p class="wp-block-paragraph">Queries inverts this. You supply the question and an alias, and the alias comes back attached to the answer, so your output key is fixed by you rather than by the document:</p>



<pre class="wp-block-code"><code>aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["QUERIES"]' 
  --queries-config '{"Queries":[
      {"Text":"What is the purchase order number?","Alias":"PO_NUMBER"},
      {"Text":"What is the delivery date?","Alias":"DELIVERY_DATE"}
  ]}' 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">The alias is the entire point. It is a contract between your schema and the extraction layer that survives a vendor redesign. There are ceilings: 15 queries per page synchronously, 30 per page asynchronously, English only. Phrase questions using words that actually appear on the page, and add positional hints when you know the layout, because the model uses them. If Queries underperforms on your corpus, adapters let you tune the feature against your own examples and pass an adapter ID and version at call time.</p>



<h3 class="wp-block-heading">Invoices are their own problem</h3>



<p class="wp-block-paragraph">For invoices and receipts, reach for AnalyzeExpense first. Its response is shaped around the domain rather than around the page: each <code>ExpenseDocument</code> holds <code>SummaryFields</code> for header-level data and <code>LineItemGroups</code> containing the line items. Fields carry a normalized <code>Type</code> such as <code>VENDOR_NAME</code>, plus a <code>ValueDetection</code> and an optional <code>LabelDetection</code>.</p>



<p class="wp-block-paragraph">That optional label is the thing to notice. AnalyzeExpense can identify a vendor name printed only inside a logo, and pull line items out of a grid with no column headers. FORMS structurally cannot do either, because FORMS needs a key to exist. On receipts that is not a small accuracy delta, it is the difference between a usable field and an empty one.</p>



<h2 class="wp-block-heading">Failure family four: confidence is per field and it is not your risk model</h2>



<p class="wp-block-paragraph">Every extracted element carries a confidence score. Two mistakes follow from that.</p>



<p class="wp-block-paragraph">The first is averaging into a document-level score. A document where every field is strong except the invoice total is not a mostly-good document. It is a document with one field that must not be trusted, surrounded by noise. Score fields, not documents.</p>



<p class="wp-block-paragraph">The second is treating one threshold as universal. A misread address on a delivery note is an annoyance. A misread bank account number is an incident. Set thresholds per field from the cost of being wrong, then route into three lanes:</p>



<ol class="wp-block-list">
<li><strong>Auto-accept.</strong> Above threshold and passing a format check. Straight into the target table.</li>

<li><strong>Review.</strong> Below threshold, or above it but failing a validation rule. A human sees the value and the cropped region it came from.</li>

<li><strong>Reject.</strong> Field absent entirely, or the document failed the ingest gate. Back to the sender with a reason.</li>
</ol>



<p class="wp-block-paragraph">Validation rules do more work than thresholds here. A total that does not equal the sum of its line items is wrong at any confidence. Check-digit rules on account and tax identifiers catch character-level errors the model was perfectly confident about. Confidence tells you how sure the model is. Validation tells you whether the answer is even possible.</p>



<p class="wp-block-paragraph">Textract can hand low-confidence results to Amazon Augmented AI through a human loop configuration on the API call, which saves building a review queue. Whether that beats a small internal tool depends on how much domain context your reviewers need on screen. Either way, capture the corrections. Reviewer edits are the only labeled data you get for free, and they are how you calibrate thresholds against your own corpus instead of guessing.</p>



<h2 class="wp-block-heading">Failure family five: reprocessing you didn&#8217;t budget for</h2>



<p class="wp-block-paragraph">Textract bills per page, per feature. Your parser, meanwhile, will change repeatedly in the first months of a pipeline&#8217;s life, because that is what parsers do.</p>



<p class="wp-block-paragraph">So separate extraction from parsing, permanently. Write the raw response JSON to S3 keyed by a hash of the document bytes plus the operation and feature set requested, then parse from that stored artifact. Reprocessing becomes free, deterministic, and diffable: you can replay a parser change across the whole back catalogue and compare outputs before promoting it. Without that cache, every parser bug fix is a bill.</p>



<pre class="wp-block-code"><code>import hashlib

digest = hashlib.sha256(doc_bytes).hexdigest()
features = "-".join(sorted(["FORMS", "TABLES"]))
key = f"textract-raw/{digest}/analyze-document/{features}.json"</code></pre>



<p class="wp-block-paragraph">Hashing content rather than filename also deduplicates the same scan arriving twice under different names, which happens constantly with email-driven intake. For the asynchronous operations, a client request token gives idempotency at the job level; reusing a token with different parameters raises a mismatch error rather than quietly starting a second job.</p>



<p class="wp-block-paragraph">One caveat: source documents often contain personal data, so a cache is a retention decision. Set S3 lifecycle rules on both the originals and the raw JSON, and if working copies land on a workstation or a build box, delete them properly. Tools like O&amp;O SafeErase exist for exactly that gap between &#8220;moved to trash&#8221; and &#8220;actually gone&#8221;.</p>



<h2 class="wp-block-heading">Where Textract stops being the right tool</h2>



<p class="wp-block-paragraph">Being fair about this matters, because AWS now offers overlapping paths to the same outcome. Amazon Bedrock Data Automation is a managed document processing service that handles classification and extraction behind one API, priced per document rather than per page and per feature. AWS positions it as the default starting point for new builds, and for mixed document types with no dominant format that recommendation is sound. You write far less orchestration.</p>



<p class="wp-block-paragraph">Textract keeps the advantage where the corpus is high volume and standardized, where geometry and per-field confidence are needed for auditability, where regional availability constrains you, or where you want extraction to be deterministic and separately testable. Bounding boxes and confidence scores are not a nice-to-have in a regulated workflow, they are the evidence trail.</p>



<p class="wp-block-paragraph">The hybrid is common and sensible: Textract for extraction, a Bedrock model for reasoning over the extracted text. Just do not send a multimodal model a full-page image because OCR felt inconvenient. It costs considerably more per page and gives you no geometry.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Only the first page came through.</strong> A synchronous call on a multi-page PDF or TIFF. Move to the asynchronous operations, and assert the page count Textract reports against the count you expected.</li>

<li><strong>Output truncates partway through a long document.</strong> Missing continuation token handling on the get operation. Loop until the token is absent.</li>

<li><strong>Fields visible on the page never appear in output.</strong> The label is implied rather than printed, or sits inside a logo. Switch that field to Queries, or to AnalyzeExpense if it is an invoice.</li>

<li><strong>Quality collapsed for one sender only.</strong> Template change. Diff the current layout against a stored sample, which is the argument for keeping raw responses.</li>

<li><strong>Throttling during a backfill.</strong> Transactions per second and concurrent job counts are per-account, per-region quotas. Queue submissions rather than fanning out, and request an increase before the migration.</li>

<li><strong>Table cells land in the wrong columns.</strong> Usually a skewed scan. Deskew and raise contrast before submitting, because image preparation is cheaper than post-processing repairs.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Testing on three clean sample documents supplied by the vendor rather than a random sample of real intake.</li>

<li>Requesting every feature type on every page because it is simpler than routing, then being surprised by the bill.</li>

<li>Using FORMS for invoices when AnalyzeExpense exists and understands unlabeled fields.</li>

<li>Treating a confidence score as an accuracy guarantee instead of pairing it with validation rules.</li>

<li>Discarding the raw JSON after parsing, which turns every parser fix into a re-extraction charge.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify before you extract. Route each document to the narrowest API that covers it.</li>

<li>Use Queries with explicit aliases for fields your schema depends on, so vendor wording changes cannot break your column names.</li>

<li>Store raw responses immutably, keyed by content hash, and parse from storage.</li>

<li>Set per-field confidence thresholds from the cost of being wrong, and pair every threshold with a format or arithmetic validation.</li>

<li>Instrument it like any other pipeline: extraction latency, straight-through rate, review queue depth, per-field null rate. A slow rise in nulls on one field is the earliest signal of a template change. Prometheus with Grafana, or Grafana Cloud if you would rather not run it yourself, covers this well.</li>

<li>Keep documents and pipeline in the same region, use VPC endpoints for sensitive data, and apply lifecycle rules to originals and extracted output alike.</li>

<li>Keep a golden set of real documents with known correct answers, and run it on every parser change.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Textract process multi-page PDFs?</h3>



<p class="wp-block-paragraph">Yes, but only through the asynchronous operations. Synchronous calls accept a single page for PDF and TIFF input. Asynchronous processing handles PDF and TIFF up to 500 MB and 3,000 pages.</p>



<h3 class="wp-block-heading">What is the difference between FORMS and Queries?</h3>



<p class="wp-block-paragraph">FORMS returns whatever key-value pairs it finds from labels printed on the page, so your output keys follow the document. Queries lets you ask for values by natural language question and attach your own alias to each answer, so output keys stay fixed when the layout changes. For a schema you must keep stable, Queries is the safer contract.</p>



<h3 class="wp-block-heading">How much does Amazon Textract cost?</h3>



<p class="wp-block-paragraph">Billing is per page and varies by operation and feature, and requesting several features on one page adds their costs together. Check the current rate card for your region and multiply by pages, not documents. The practical lever is routing: send each document to the narrowest operation that answers your question, and cache raw responses so parser iteration is free.</p>



<h3 class="wp-block-heading">Does Textract handle handwriting?</h3>



<p class="wp-block-paragraph">It detects handwritten text, in English only. Treat handwritten fields as a separate confidence tier with lower auto-accept thresholds, because the variance across writers is much wider than for printed text.</p>



<h3 class="wp-block-heading">Which languages does Amazon Textract support?</h3>



<p class="wp-block-paragraph">Text detection covers English, French, German, Italian, Portuguese, and Spanish. Handwriting, invoices and receipts, identity documents, and Queries are English only. Vertically written text is not supported.</p>



<h3 class="wp-block-heading">Should I use Textract or Bedrock Data Automation?</h3>



<p class="wp-block-paragraph">Bedrock Data Automation is the lower-effort default for mixed document types, with classification and extraction behind a single managed API. Textract fits better at high volume on standardized documents, where per-field confidence and geometry matter for audit, or where extraction needs to be deterministic and independently testable. Combining them is normal.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If one thing survives this post, make it this: Amazon Textract structured data extraction fails quietly, not loudly. The API returns success while your record is missing its most important field, because a synchronous call read one page of forty, or because a vendor renamed a label, or because a value came back at low confidence and nothing was watching.</p>



<p class="wp-block-paragraph">So build for the silence. Gate at ingest, use asynchronous operations by default, pick the narrowest API per document family, pin your schema with query aliases, store the raw response, and route on per-field confidence backed by validation rules that know what a possible answer looks like. Do that and the OCR really does become the easy part.</p>



<h2 class="wp-block-heading">Need help turning documents into a pipeline you can trust?</h2>



<p class="wp-block-paragraph">Most of the difficulty in document extraction shows up after the proof of concept, once real intake replaces the sample files. That is the part I work on:</p>



<ul class="wp-block-list">
<li>Designing the ingest gate and classification step so each document family reaches the right Textract operation instead of one call doing everything.</li>

<li>Building the asynchronous path properly: S3 events, SNS completion, paginated retrieval, dead-letter handling, idempotent reprocessing.</li>

<li>Replacing brittle FORMS parsing with query aliases or AnalyzeExpense, and validating the change against a golden set of your own documents.</li>

<li>Setting per-field confidence thresholds and validation rules, then wiring a review queue whose corrections feed back into tuning.</li>

<li>Adding observability that catches template drift early: per-field null rates, straight-through rate, and review queue depth on a dashboard.</li>

<li>Getting the cost curve under control through routing, raw-response caching, and lifecycle rules on stored documents and JSON.</li>
</ul>



<p class="wp-block-paragraph">If you have a pipeline misbehaving, send me a sample document with the fields you need and the raw Textract JSON it produced. That is usually enough to see where it is going wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/">Turning External Documents Into Structured Data With Amazon Textract</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Zoho Analytics Import vs Live Connect: The Trade-Offs Nobody Mentions</title>
		<link>https://john-nessime.com/blog/technical-guides/zoho-analytics-import-vs-live-connect/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 30 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Caching]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Live Connect]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Query Timeout]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho Analytics]]></category>
		<category><![CDATA[Zoho Databridge]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=346</guid>

					<description><![CDATA[<p>Import mode fails quietly and Live Connect fails loudly. A practical breakdown of what each Zoho Analytics connection mode actually does, where each one wins, the four levers that should drive the choice, and a decision procedure you can run before you build the workspace.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zoho-analytics-import-vs-live-connect/">Zoho Analytics Import vs Live Connect: The Trade-Offs Nobody Mentions</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on a Monday. Someone in finance opens the revenue dashboard, compares it against the operational system, and asks why one says 41 and the other says 47. Nothing is broken. No sync failed. No alert fired. The dashboard is just showing you a snapshot from six hours ago, and absolutely nothing on the screen tells you that.</p>



<p class="wp-block-paragraph">That is the real cost of picking the wrong data connection mode, and it is why the choice between Zoho Analytics import vs Live Connect deserves more than five minutes during setup. Both modes work. Both are documented. But they fail in completely different ways, and one of those failure modes is silent.</p>



<p class="wp-block-paragraph">This post covers what each mode actually does under the hood, where each one genuinely wins, the four levers that should drive the decision, a short procedure for running that decision yourself, and the arguments people repeat that don&#8217;t hold up once you have a workspace in production.</p>



<h2 class="wp-block-heading">What the two modes actually do</h2>



<p class="wp-block-paragraph">Strip away the marketing language and the difference is where the data lives when someone opens a report.</p>



<p class="wp-block-paragraph"><strong>Data Import</strong> copies rows out of your source and stores them inside Zoho Analytics. Reports query that stored copy. You schedule refreshes to pull new data on an interval, and between refreshes your dashboard is a photograph, not a window.</p>



<p class="wp-block-paragraph"><strong>Live Connect</strong> does not copy anything. Zoho pulls metadata about your tables and columns, then generates SQL against your source database every single time a report or dashboard is opened. Your database does the work. Zoho does the rendering.</p>



<p class="wp-block-paragraph">That one architectural difference cascades into everything else: what you get billed for, which features you can use, who owns schema changes, and what happens when a query gets slow.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Import mode: where it wins and where it hurts</h2>



<h3 class="wp-block-heading">Where import mode wins</h3>



<p class="wp-block-paragraph">Import mode is the default for a reason, and the reasons are good ones.</p>



<ul class="wp-block-list">
<li><strong>Report speed is yours to control.</strong> Because the data sits in Zoho&#8217;s storage, load time doesn&#8217;t depend on how busy your production database is at 9am on a Monday.</li>



<li><strong>You get the full feature surface.</strong> Query Tables, custom import queries, data type changes on imported columns, and blending several unrelated sources into one workspace all work in import mode.</li>



<li><strong>Your production database is insulated.</strong> An analyst who builds an accidental many-to-many join hits a copy, not the system taking customer orders.</li>



<li><strong>Incremental fetch keeps sync cheap.</strong> Rather than re-pulling a whole table on every schedule, you nominate a column that identifies new rows and pull only what changed.</li>



<li><strong>It works on every plan.</strong> There is no tier gate on importing data, which matters if you are prototyping before anyone has approved a budget.</li>
</ul>



<h3 class="wp-block-heading">Where import mode hurts</h3>



<p class="wp-block-paragraph">The first problem is the one from the opening: <strong>staleness is invisible</strong>. A Zoho Analytics dashboard built on imported data looks exactly the same whether it refreshed two minutes ago or two days ago. There is no built-in &#8220;as of&#8221; banner on the chart. If your refresh schedule is daily and someone is making a call at 4pm, they are working from this morning&#8217;s world.</p>



<p class="wp-block-paragraph">The second is <strong>billing shape</strong>. Zoho Analytics prices on stored rows, counted as the sum of all rows across every table in every workspace in the account. Import mode is therefore the mode that grows your bill. Every table you sync &#8220;just in case&#8221; is a line item. Rates change, so check the current pricing page rather than trusting any figure you read in a blog post, but understand the mechanism: rows stored equals money.</p>



<p class="wp-block-paragraph">Third, <strong>the copy drifts and nobody owns it</strong>. Six months in, a table has a scheduled import that has been silently failing, or someone changed a transformation in the source and the imported copy still reflects the old logic. Import mode creates a second version of the truth, and second versions need maintenance.</p>



<p class="wp-block-paragraph">Fourth, <strong>refresh frequency is plan-gated</strong>. How often you can schedule a sync depends on your subscription tier. If your reporting requirement is &#8220;within the hour&#8221; and your plan permits fewer refreshes per day than that implies, import mode cannot meet the requirement no matter how you configure it.</p>



<h2 class="wp-block-heading">Live Connect: where it wins and where it hurts</h2>



<h3 class="wp-block-heading">Where Live Connect wins</h3>



<ul class="wp-block-list">
<li><strong>There is exactly one copy of the data.</strong> No reconciliation conversation, because there is nothing to reconcile. What the report shows is what the database holds at that moment.</li>



<li><strong>Sensitive data stays where it already is.</strong> For teams with a data residency position, or a security team that objects to bulk-copying customer records into a BI vendor&#8217;s storage, this is often the deciding argument on its own.</li>



<li><strong>Foreign keys carry over.</strong> Tables linked by foreign keys in the source get lookup relationships created automatically in Zoho, which is a genuine time saver compared to wiring lookup columns by hand.</li>



<li><strong>Row storage isn&#8217;t the cost driver.</strong> You&#8217;re not paying to warehouse a duplicate of a table you already pay to host on Amazon RDS, Snowflake, or a Postgres instance on your own VPS.</li>
</ul>



<h3 class="wp-block-heading">Where Live Connect hurts</h3>



<p class="wp-block-paragraph">Here is the one that catches people, and it is worth reading twice: <strong>Live Connect does not automatically track schema changes</strong>. Add a column, rename a column, drop a table, change a data type in the source, and Zoho will not follow along. Reports built on the affected objects land in a Mismatch state and you have to reconcile them manually through the Sync Design option in the data source settings.</p>



<p class="wp-block-paragraph">This is the inverse of what most people assume. &#8220;Live&#8221; sounds like &#8220;always current in every respect.&#8221; It means live <em>data</em>, not live <em>structure</em>. A routine migration that renames <code>customer_id</code> to <code>account_id</code> will break dashboards that nobody looked at that week, and you&#8217;ll find out from a user rather than from a monitor.</p>



<p class="wp-block-paragraph">The rest of the trade-offs:</p>



<ul class="wp-block-list">
<li><strong>Query timeout.</strong> Zoho documents a default query timeout of 2.5 minutes on Live Connect. Anything slower fails. On a source that hasn&#8217;t been tuned for analytical queries, that ceiling arrives faster than you expect.</li>



<li><strong>No Custom Query at import time.</strong> You can&#8217;t hand Zoho a filtered SELECT to narrow what it sees. The documented workaround is to create a view in the source database and connect to that instead.</li>



<li><strong>No Query Tables.</strong> The SQL-over-your-workspace feature depends on data being stored locally. Zoho notes Query Table support for selected databases is enabled on request, so it is worth asking, but do not plan around it.</li>



<li><strong>No data type changes in Zoho.</strong> If a column is text in the source and should be a date, you fix it in the source or in a view. There is no in-Zoho override.</li>



<li><strong>Lookups only within the connection.</strong> You can define lookup rules between tables in the same live connection, not across to other data sources. Reports from different sources can still be combined on a dashboard, but that is a presentation-layer join, not a data-layer one.</li>



<li><strong>It is gated to higher plans.</strong> Zoho&#8217;s documentation places Live Connect on the Premium and Enterprise tiers. If you&#8217;re on a lower plan, the decision has already been made for you.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The four levers behind Zoho Analytics import vs Live Connect</h2>



<p class="wp-block-paragraph">Feature checklists make this look like a close call. It usually isn&#8217;t, once you work out which of these four you actually care about.</p>



<p class="wp-block-paragraph"><strong>1. Your freshness contract.</strong> Not &#8220;how fresh would be nice.&#8221; What decision gets made from this dashboard, and how wrong can the data be before that decision goes bad? A weekly pipeline review tolerates a nightly sync comfortably. A dispatcher deciding which order to fulfil next does not. Write the tolerance down as a number of minutes before you look at any feature table.</p>



<p class="wp-block-paragraph"><strong>2. Where the cost lands.</strong> Import mode moves cost onto your Zoho row count. Live Connect moves it onto your database, in query load and in whatever you pay for the compute that serves it. Neither is free. A warehouse that bills per query scanned is a different economic proposition under Live Connect than a fixed-price MySQL instance on Contabo or InterServer, where extra query load costs nothing until you outgrow the box.</p>



<p class="wp-block-paragraph"><strong>3. Where transformation happens.</strong> If your model needs joins, unions, derived columns, and cleanup, import mode gives you Query Tables to do that inside Zoho. Live Connect pushes all of it down into the source, which means you need database write access and someone comfortable maintaining views. That&#8217;s a team-capability question as much as a technical one.</p>



<p class="wp-block-paragraph"><strong>4. Who owns schema change.</strong> Under Live Connect, every source migration is potentially a reporting incident. That is fine if the same team owns both, and painful if the application team ships migrations without telling anyone. Import mode is more forgiving here, since Zoho can automatically pick up column additions and deletions on synced tables.</p>



<h2 class="wp-block-heading">A decision procedure you can run in ten minutes</h2>



<ol class="wp-block-list">
<li><strong>Check the plan gate first.</strong> If you&#8217;re not on a tier that includes Live Connect, stop. Import mode, tuned well, is the answer.</li>



<li><strong>Write down the freshness tolerance in minutes.</strong> If the number is larger than your plan&#8217;s shortest available sync interval, import mode satisfies it. If it&#8217;s smaller, Live Connect is the only mode that can meet it.</li>



<li><strong>Time the heaviest query.</strong> Take the ugliest aggregation your dashboard will need and run it against the source with realistic filters. If it takes anywhere near 2.5 minutes, Live Connect will fail under real usage. Tune the source or choose import.</li>



<li><strong>List the transformations you need.</strong> If any of them require a Query Table, joining across unrelated sources, or changing a column&#8217;s data type in Zoho, import mode is doing that work.</li>



<li><strong>Ask who ships schema migrations.</strong> If it isn&#8217;t the people who own the dashboards, budget for Sync Design work every release, or use import mode.</li>



<li><strong>Check the security position last, because it can override everything.</strong> If the answer to &#8220;may customer records be stored in a third-party BI platform&#8221; is no, Live Connect is the only option and you build around its constraints.</li>
</ol>



<p class="wp-block-paragraph">In practice, mixed estates are common and reasonable. Live Connect for the operational workspace where freshness is the point, import mode for the historical and blended reporting where transformation and speed matter more.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<p class="wp-block-paragraph"><em>&#8220;Live Connect is real-time, so it&#8217;s more accurate.&#8221;</em> It&#8217;s more current. Accuracy is a modelling problem, not a transport problem. A live connection to a table with duplicate lookup keys produces wrong numbers instantly instead of wrong numbers on a schedule.</p>



<p class="wp-block-paragraph"><em>&#8220;Live Connect is cheaper because you don&#8217;t pay for rows.&#8221;</em> Sometimes. You&#8217;ve moved the cost, not removed it. Every dashboard open is a set of queries against your source. On per-query or per-compute-second billing, an enthusiastic team with auto-refreshing dashboards can generate a bill that makes row storage look like a bargain.</p>



<p class="wp-block-paragraph"><em>&#8220;Just enable the cache and Live Connect gets fast.&#8221;</em> Zoho does offer report caching on Live Connect workspaces, with a refresh interval you set in minutes. It genuinely helps. But note what you&#8217;ve just built: a copy of the data with a staleness window, which is the thing you chose Live Connect to avoid. Caching is a legitimate tuning tool, not a free upgrade.</p>



<p class="wp-block-paragraph"><em>&#8220;We&#8217;ll switch modes later if it doesn&#8217;t work out.&#8221;</em> Switching is not a toggle. Zoho does not permit importing data into an existing Live Connect workspace, and the two modes support different feature sets, so a migration means rebuilding the workspace and its reports. Choose deliberately the first time.</p>



<h2 class="wp-block-heading">If you go Live Connect, tune the source first</h2>



<p class="wp-block-paragraph">Live Connect performance is your database&#8217;s performance. Zoho&#8217;s own guidance is blunt about this, and it matches what you&#8217;d do for any reporting workload.</p>



<p class="wp-block-paragraph">Index the columns that lookups and joins run through. Without them, every dashboard open is a table scan:</p>



<pre class="wp-block-code"><code>-- Index the join key that Zoho will generate queries against
CREATE INDEX idx_orders_customer_id ON orders (customer_id);

-- Confirm the planner actually uses it (PostgreSQL)
EXPLAIN ANALYZE
SELECT customer_id, SUM(total)
FROM orders
WHERE order_date &gt;= CURRENT_DATE - INTERVAL '90 days'
GROUP BY customer_id;</code></pre>



<p class="wp-block-paragraph">Because Custom Query isn&#8217;t available in Live Connect, a view is how you narrow and shape what Zoho sees. Connect to the view rather than the raw table:</p>



<pre class="wp-block-code"><code>CREATE VIEW reporting_orders AS
SELECT
    o.id,
    o.customer_id,
    o.order_date,
    o.total,
    CASE WHEN o.status = 'complete' THEN 1 ELSE 0 END AS is_complete
FROM orders o
WHERE o.order_date &gt;= CURRENT_DATE - INTERVAL '2 years';</code></pre>



<p class="wp-block-paragraph">For calculations that get repeated across many reports, a materialized view precomputes the work so the 2.5 minute ceiling stops being a threat. You trade a refresh schedule for query speed, which is a much smaller staleness window than a full import.</p>



<p class="wp-block-paragraph">The rest of the list is short and worth following:</p>



<ul class="wp-block-list">
<li>Keep formula columns simple, and prefer CASE WHEN over deeply nested IF statements.</li>



<li>Make sure lookup columns hold unique values. Duplicate keys degrade query performance badly and quietly inflate aggregates.</li>



<li>Avoid many-to-many joins and circular join paths in your model.</li>



<li>Give Zoho a database user with SELECT plus metadata read on exactly the tables it needs, and nothing more. Missing metadata permission is a common cause of a table that connects but never loads.</li>



<li>Decide your connectivity path deliberately: allowlisting Zoho&#8217;s published IP ranges for a cloud database, or Zoho Databridge as an agent when the database sits in a private network. SSL and SSH tunneling are both supported on the connection.</li>



<li>Put a change-notification step in your migration checklist. If a release renames or drops a column, someone should run Sync Design before users find the Mismatch.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I switch a workspace from import mode to Live Connect?</h3>



<p class="wp-block-paragraph">Not as an in-place setting change. Zoho does not permit importing data into an existing Live Connect workspace, and the feature sets differ enough that reports built one way won&#8217;t simply carry over. Treat a mode change as a rebuild and plan the reporting downtime accordingly.</p>



<h3 class="wp-block-heading">Does Live Connect count against my row limit?</h3>



<p class="wp-block-paragraph">Zoho&#8217;s row-based pricing counts rows stored across your workspace tables, and Live Connect doesn&#8217;t store your source data in Zoho. The practical implication is that Live Connect shifts your constraint from storage to source query capacity. Confirm the exact treatment for your account with Zoho before you build a cost model on it, since add-ons and plan definitions change.</p>



<h3 class="wp-block-heading">Why do my Live Connect reports suddenly show a view-cannot-be-accessed error?</h3>



<p class="wp-block-paragraph">Almost always because a table or column the report depends on was renamed or dropped in the source. Live Connect doesn&#8217;t track structural changes automatically. Open the data source settings, check the Mismatch tab, and use Sync Design to remap. Insufficient SELECT or metadata permissions on the source produce a similar symptom.</p>



<h3 class="wp-block-heading">What is incremental fetch, and does it apply to both modes?</h3>



<p class="wp-block-paragraph">Incremental fetch is an import-mode feature. Instead of re-pulling every row on each schedule, you nominate a column that identifies new records, such as an auto-incrementing ID or a timestamp, and choose whether new rows are appended or matched and updated. It cuts sync time dramatically on large transactional tables. It has no meaning under Live Connect, where nothing is fetched on a schedule in the first place.</p>



<h3 class="wp-block-heading">Can I combine Live Connect tables with data from other sources?</h3>



<p class="wp-block-paragraph">Not at the data layer. Lookup rules only work between tables within the same live connection. You can place reports from a Live Connect workspace and reports from other sources side by side on a dashboard, but you cannot join them into a single dataset the way you can with imported tables in one workspace.</p>



<h3 class="wp-block-heading">Is Live Connect available for on-premise and private databases?</h3>



<p class="wp-block-paragraph">Yes, through Zoho Databridge, the agent that bridges a database in a private network to Zoho&#8217;s servers. You choose the connection type during setup, so the same agent supports either mode. If your database is publicly reachable instead, you allowlist Zoho&#8217;s published IP ranges and connect directly.</p>



<h3 class="wp-block-heading">Which mode is better for a Zoho CRM or Zoho Desk workspace?</h3>



<p class="wp-block-paragraph">Neither, in the sense that the question doesn&#8217;t apply. Zoho&#8217;s own application connectors sync data into Analytics on a schedule, which behaves like import mode. Live Connect is for external databases and data warehouses. If you need faster refreshes on Zoho app data, the lever is the connector&#8217;s sync schedule and your plan tier, not the connection mode.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The Zoho Analytics import vs Live Connect decision is not a performance question. It&#8217;s a question about which failure you&#8217;d rather have.</p>



<p class="wp-block-paragraph">Import mode fails quietly. Numbers stay confidently wrong until someone reconciles them by hand, and the bill grows with every table you sync out of habit. Live Connect fails loudly. Someone renames a column and a dashboard breaks in front of a user, or a heavy query hits the timeout during the Monday review.</p>



<p class="wp-block-paragraph">Loud failures are usually easier to live with, because you find out about them. But they only work if you have the source database, the access to tune it, and a change process that includes reporting. If you don&#8217;t have those three, import mode with an honest refresh schedule and a documented staleness window will serve you far better than a live connection you can&#8217;t maintain.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your Zoho Analytics setup?</h2>



<p class="wp-block-paragraph">Most of the Zoho Analytics work I get called into is somebody discovering the mode choice was made in five minutes eighteen months ago. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing an existing workspace and telling you honestly whether the connection mode still fits what you&#8217;re using it for</li>



<li>Profiling and tuning the source database behind a Live Connect workspace so dashboards stop hitting the query timeout</li>



<li>Designing reporting views and materialized views that push transformation into the database where Live Connect can use it</li>



<li>Setting up Zoho Databridge for databases in private networks, including SSL and SSH tunneling and the firewall rules around it</li>



<li>Configuring incremental fetch on import-mode tables to cut sync times and stop row counts drifting upward</li>



<li>Building a schema-change process so source migrations don&#8217;t turn into reporting incidents</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me the slow query, the Mismatch screenshot, or the EXPLAIN output from the aggregation that&#8217;s timing out. That&#8217;s usually enough to tell where the actual problem is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zoho-analytics-import-vs-live-connect/">Zoho Analytics Import vs Live Connect: The Trade-Offs Nobody Mentions</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=215</guid>

					<description><![CDATA[<p>A GraphQL source can hand you a 200 OK, a populated data block, and a quietly broken column in the same response. Here is how to build a GraphQL data ingestion pipeline on AWS that catches partial errors, respects cost-based rate limits, resumes cleanly from a cursor, and notices when the schema moves under you.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The row counts looked fine. That was the problem.</p>



<p class="wp-block-paragraph">A nightly sync from a partner&#8217;s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.</p>



<p class="wp-block-paragraph">The API had started returning a field-level error on that one field. Status code 200. A populated <code>data</code> block. An <code>errors</code> array sitting right next to it. The fetcher parsed <code>data</code>, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.</p>



<p class="wp-block-paragraph">This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.</p>



<h2 class="wp-block-heading">Why GraphQL ingestion breaks the habits REST taught you</h2>



<p class="wp-block-paragraph">Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server&#8217;s business, and rate limits are counted in requests. GraphQL invalidates all three.</p>



<ul class="wp-block-list">
<li><strong>The status code is not the contract.</strong> The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.</li>

<li><strong>You own the response shape.</strong> The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.</li>

<li><strong>Cost is query-shaped, not endpoint-shaped.</strong> One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.</li>
</ul>



<p class="wp-block-paragraph">Each of those becomes a distinct failure family. Let&#8217;s take them in the order they bite.</p>



<h2 class="wp-block-heading">Failure one: the 200 OK that quietly ate a column</h2>



<p class="wp-block-paragraph">This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.</p>



<ul class="wp-block-list">
<li><strong>Request errors</strong> happen before execution starts: syntax errors, validation failures, bad variable coercion. No <code>data</code> key is returned at all. Nothing executed.</li>

<li><strong>Field errors</strong> happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get <code>data</code> and <code>errors</code> together.</li>
</ul>



<p class="wp-block-paragraph">Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL&#8217;s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.</p>



<p class="wp-block-paragraph">The fix is to make the response body the contract. Classify before you commit anything:</p>



<pre class="wp-block-code"><code>import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]</code></pre>



<p class="wp-block-paragraph">Two things worth explaining. The <code>Accept</code> header asks for <code>application/graphql-response+json</code>, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don&#8217;t support it fall back to <code>application/json</code>, so including both is free.</p>



<p class="wp-block-paragraph">The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.</p>



<h3 class="wp-block-heading">Deciding what &#8220;partial&#8221; means for your data</h3>



<p class="wp-block-paragraph">Failing the whole page on any error is the safe default and the one I reach for first. But it&#8217;s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.</p>



<p class="wp-block-paragraph">The workable middle ground is to classify by path. Every error object carries a <code>path</code> array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.</p>



<h2 class="wp-block-heading">Failure two: rate limits that never send a 429</h2>



<p class="wp-block-paragraph">Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.</p>



<p class="wp-block-paragraph">GitHub&#8217;s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and <code>x-ratelimit-remaining</code> set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.</p>



<p class="wp-block-paragraph">The other half of the problem is that request counting is the wrong unit. Shopify&#8217;s GraphQL Admin API and several others price queries by <em>calculated cost</em>, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case <em>requested</em> cost, executes, then refunds the difference between requested and <em>actual</em> cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.</p>



<p class="wp-block-paragraph">The good news is that these APIs tell you the state of the bucket on every response, under the <code>extensions</code> key:</p>



<pre class="wp-block-code"><code>{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant&#8217;s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against <code>currentlyAvailable</code>, and if you&#8217;re short, sleep for the shortfall divided by <code>restoreRate</code>.</p>



<p class="wp-block-paragraph">On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:</p>



<pre class="wp-block-code"><code>query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}</code></pre>



<p class="wp-block-paragraph">Two more things about concurrency. First, if you fan out across Lambda invocations, each worker&#8217;s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it&#8217;s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can&#8217;t go wrong.</p>



<p class="wp-block-paragraph">Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.</p>



<h2 class="wp-block-heading">Failure three: cursors that don&#8217;t survive a restart</h2>



<p class="wp-block-paragraph">Most production GraphQL schemas follow the Relay cursor connections convention: <code>first</code> and <code>after</code> arguments, a <code>pageInfo</code> object with <code>hasNextPage</code> and <code>endCursor</code>, and edges carrying opaque per-item cursors.</p>



<pre class="wp-block-code"><code>query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.</p>



<p class="wp-block-paragraph">The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.</p>



<p class="wp-block-paragraph">Three habits make restarts safe:</p>



<ol class="wp-block-list">
<li><strong>Checkpoint after the write, not after the fetch.</strong> A cursor recorded before the page lands in S3 is a promise you can&#8217;t keep.</li>

<li><strong>Store a high watermark alongside the cursor.</strong> The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.</li>

<li><strong>Overlap the watermark deliberately.</strong> Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.</li>
</ol>



<p class="wp-block-paragraph">A DynamoDB item per source stream is enough state for all of this:</p>



<pre class="wp-block-code"><code>{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "&lt;updatedAt of newest committed record&gt;",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}</code></pre>



<p class="wp-block-paragraph">Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest <code>updatedAt</code>. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you&#8217;ll have traded silent loss for silent inflation.</p>



<h2 class="wp-block-heading">Failure four: the schema moved and nobody told you</h2>



<p class="wp-block-paragraph">GraphQL&#8217;s deprecation story is genuinely better than REST&#8217;s. Fields carry a <code>@deprecated</code> directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.</p>



<p class="wp-block-paragraph">The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.</p>



<p class="wp-block-paragraph">Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.</p>



<p class="wp-block-paragraph">The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring the GraphQL data ingestion pipeline on AWS</h2>



<p class="wp-block-paragraph">With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.</p>



<h3 class="wp-block-heading">Where the fetch loop runs</h3>



<p class="wp-block-paragraph">Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:</p>



<pre class="wp-block-code"><code>"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}</code></pre>



<p class="wp-block-paragraph">The execution role needs four permissions: <code>states:InvokeHTTPEndpoint</code>, <code>events:RetrieveConnectionCredentials</code>, and both <code>secretsmanager:GetSecretValue</code> and <code>secretsmanager:DescribeSecret</code> scoped to the EventBridge-managed secret.</p>



<p class="wp-block-paragraph">It&#8217;s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.</p>



<p class="wp-block-paragraph">The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow&#8217;s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.</p>



<p class="wp-block-paragraph">Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda&#8217;s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.</p>



<p class="wp-block-paragraph">One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.</p>



<h3 class="wp-block-heading">The landing zone</h3>



<p class="wp-block-paragraph">Land raw responses first, unmodified, including the <code>errors</code> and <code>extensions</code> keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.</p>



<ul class="wp-block-list">
<li><strong>Partition by ingest time, not event time.</strong> Event-time partitioning requires reading the record before you know where to put it, which means you can&#8217;t write until you&#8217;ve parsed. Ingest-time keys let the raw write be dumb and fast.</li>

<li><strong>Put the run identifier in the key.</strong> When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.</li>

<li><strong>Define Glue tables explicitly rather than crawling.</strong> You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.</li>

<li><strong>Convert to columnar in the curated layer.</strong> Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.</li>
</ul>



<p class="wp-block-paragraph">If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don&#8217;t need.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Runs succeed but a column is all null.</strong> Field errors. Pull a raw response from the landing zone and look at the <code>errors</code> array. The <code>path</code> tells you exactly which field.</li>

<li><strong>Runs succeed but ingest zero rows.</strong> Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.</li>

<li><strong>Intermittent failures partway through pagination.</strong> Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.</li>

<li><strong>Query fails immediately with no <code>data</code> key.</strong> Request error. Validate against current introspection; something in the schema moved.</li>

<li><strong>Throttled despite staying under the documented limit.</strong> Either concurrent workers are sharing a bucket they each think they own, or you&#8217;re being charged requested cost on near-empty pages.</li>

<li><strong>HTTP Task fails on large pages.</strong> The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.</li>

<li><strong>Row counts drift upward over time.</strong> Overlap without deduplication. Check that the curated layer actually resolves on the primary key.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a 200 as success and never reading the body.</li>

<li>Retrying request errors. They are deterministic; retrying just spends budget to fail identically.</li>

<li>Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.</li>

<li>Checkpointing the cursor before the data is durable.</li>

<li>Requesting the maximum page size on every call while being charged the worst-case cost for it.</li>

<li>Selecting entire objects because it was easier during development.</li>

<li>Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.</li>

<li>Overlapping restart windows with no downstream deduplication.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify every response into clean, partial, or request-failed before anything downstream sees it.</li>

<li>Persist raw responses with <code>errors</code> and <code>extensions</code> intact, then parse from the raw layer.</li>

<li>Alarm on row-count deltas and partial-error rate, not just on execution status.</li>

<li>Read the throttle state the API hands you and back off before you hit the wall, not after.</li>

<li>Keep both a cursor and a watermark, and know which one is authoritative on restart.</li>

<li>Diff the schema on a schedule so deprecations reach you before they reach your data.</li>

<li>Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.</li>

<li>Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you&#8217;re correlating child executions across a fan-out.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Step Functions call a GraphQL API without a Lambda function?</h3>



<p class="wp-block-paragraph">Yes. An HTTP Task using the <code>arn:aws:states:::http:invoke</code> resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.</p>



<h3 class="wp-block-heading">How do I detect GraphQL errors when the status code is always 200?</h3>



<p class="wp-block-paragraph">Inspect the body. If there&#8217;s no <code>data</code> key, nothing executed and you have a request error. If <code>data</code> is present alongside a non-empty <code>errors</code> array, you have a partial response and the data is incomplete. Only a response with <code>data</code> and no errors is unambiguously clean.</p>



<h3 class="wp-block-heading">Does AWS AppSync fit into this?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it&#8217;s what you&#8217;d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else&#8217;s GraphQL API, AppSync isn&#8217;t part of the picture.</p>



<h3 class="wp-block-heading">Should I store raw GraphQL responses or parsed records in S3?</h3>



<p class="wp-block-paragraph">Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they&#8217;re the only way to answer &#8220;was this field null in the source, or did my parser drop it?&#8221; weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.</p>



<h3 class="wp-block-heading">How do I run an incremental sync from a GraphQL API?</h3>



<p class="wp-block-paragraph">Find the schema&#8217;s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you&#8217;re limited to full pulls or webhook-driven change capture.</p>



<h3 class="wp-block-heading">Why am I throttled when I&#8217;m making very few requests?</h3>



<p class="wp-block-paragraph">Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.</p>



<h3 class="wp-block-heading">Is GraphQL a worse ingestion source than REST?</h3>



<p class="wp-block-paragraph">Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.</p>



<p class="wp-block-paragraph">So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with a GraphQL ingestion pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. Specifically:</p>



<ul class="wp-block-list">
<li>Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected</li>

<li>Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts</li>

<li>Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination</li>

<li>Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap</li>

<li>Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard</li>

<li>Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green</li>
</ul>



<p class="wp-block-paragraph">Send me a run log, a state machine definition, or a raw response with an <code>errors</code> array in it, and I&#8217;ll tell you what it&#8217;s costing you.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</title>
		<link>https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 15 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Anomaly Detection]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=225</guid>

					<description><![CDATA[<p>Your SaaS pipeline will fail far more often by succeeding at nothing than by throwing an exception, and every CloudWatch default treats an absent metric as a non-event. Here are the four signals worth alarming on: liveness, volume, freshness and shape, plus the missing-data traps that leave alarms permanently green.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/">CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said the Salesforce numbers looked wrong on the executive dashboard. I opened the pipeline. Every run for the past several days was green: the state machine succeeded, no failed states, no Lambda errors, no alarms in the account. The pipeline had been running flawlessly and writing an empty file every single time.</p>



<p class="wp-block-paragraph">That gap is the whole problem with CloudWatch data pipeline monitoring, and it is why most teams find out about broken SaaS integrations from a business user rather than from a page. A web server that breaks returns 500s, and your error-rate alarm fires. A pipeline that breaks frequently returns exit code 0, writes a zero-byte object, and reports success. The connected app got revoked, a scope was dropped during an admin cleanup, or a filter clause quietly started matching nothing. None of the AWS-native metrics have an opinion about any of that.</p>



<p class="wp-block-paragraph">This post is about what to measure instead: the four signals worth alarming on, the CloudWatch defaults that leave those alarms silently in the wrong state, how to emit custom metrics without turning your bill into a line item someone asks about, and what to check when a metric exists but the alarm still refuses to fire.</p>



<h2 class="wp-block-heading">&#8220;Did the job fail?&#8221; is the weakest question you can ask</h2>



<p class="wp-block-paragraph">Job status is a proxy for pipeline health, and a bad one. It tells you the orchestrator finished its instructions. It says nothing about whether those instructions produced data.</p>



<p class="wp-block-paragraph">SaaS sources make this worse than usual. A REST API returning HTTP 200 with an empty result array is, to your extraction code, a completely successful call. Pagination that stops on the first page because a cursor was reset is successful. An OAuth refresh that returns a token with fewer scopes than yesterday is successful right up until one object starts coming back empty. Every one of those is a green run.</p>



<p class="wp-block-paragraph">Build the monitoring around four questions the job outcome cannot answer:</p>



<ol class="wp-block-list"><li><strong>Liveness</strong> &#8211; did the run happen at all?</li><li><strong>Volume</strong> &#8211; did it move a plausible amount of data?</li><li><strong>Freshness</strong> &#8211; how old is the newest record that actually landed?</li><li><strong>Shape</strong> &#8211; are the fields and values still what downstream expects?</li></ol>



<p class="wp-block-paragraph">Each needs a different kind of metric, and each fails silently in its own way.</p>



<h2 class="wp-block-heading">Signal 1: liveness, and the missing-data trap that hides it</h2>



<p class="wp-block-paragraph">Step Functions publishes execution metrics to the <code>AWS/States</code> namespace, including <code>ExecutionsStarted</code>, <code>ExecutionsSucceeded</code>, <code>ExecutionsFailed</code>, <code>ExecutionsTimedOut</code>, <code>ExecutionsAborted</code> and <code>ExecutionTime</code>. The obvious move is an alarm on <code>ExecutionsFailed</code>. Most teams stop there, and that alarm is close to useless on its own.</p>



<p class="wp-block-paragraph">Here is why. Those metrics are published as a result of executions. If somebody disables the EventBridge schedule rule during an incident and forgets to re-enable it, no execution starts, so no data point is published for that dimension at all. Your alarm has nothing to evaluate.</p>



<p class="wp-block-paragraph">What happens next depends entirely on a parameter most people never set. CloudWatch alarms accept a <code>TreatMissingData</code> setting with four values: <code>breaching</code>, <code>notBreaching</code>, <code>ignore</code> and <code>missing</code>. The default is <code>missing</code>, which means the alarm keeps its current state when there is nothing to evaluate. An alarm that was sitting in OK stays in OK. Forever. It is green because it is blind, and there is no visual difference between the two.</p>



<p class="wp-block-paragraph">The fix is not to change <code>TreatMissingData</code> on the failure alarm. It is to add a separate heartbeat metric that your pipeline emits on completion, and alarm on the <em>absence</em> of that heartbeat:</p>



<pre class="wp-block-code"><code># Emitted at the very end of a successful run, per source system.
aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --value 1 
  --unit Count 
  --dimensions Source=salesforce,Stage=extract</code></pre>



<p class="wp-block-paragraph">Then the alarm that actually catches a dead scheduler:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name "ingest-salesforce-no-heartbeat" 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --dimensions Name=Source,Value=salesforce Name=Stage,Value=extract 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --datapoints-to-alarm 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts</code></pre>



<p class="wp-block-paragraph">The two flags doing the real work are <code>--comparison-operator LessThanThreshold</code> and <code>--treat-missing-data breaching</code>. The first catches a run that completes but emits nothing. The second catches a run that never happens at all. Without the second flag, silence reads as health.</p>



<p class="wp-block-paragraph">Set <code>--period</code> comfortably longer than your schedule interval so one slow run does not page anyone, and use <code>--datapoints-to-alarm</code> to require consecutive breaches. One caveat: <code>breaching</code> is right for a heartbeat you control, but a poor blanket default across the account, because plenty of AWS metrics are only emitted when non-zero. Throttling counters are legitimately sparse, and forcing them to breach on silence just makes noise.</p>



<h2 class="wp-block-heading">Signal 2: volume, and why a static threshold beats anomaly detection at first</h2>



<p class="wp-block-paragraph">A heartbeat proves the run happened. It does not prove the run did anything. Emit a row count, per source, per object:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RecordsWritten" 
  --value 14238 
  --unit Count 
  --dimensions Source=salesforce,Object=opportunity</code></pre>



<p class="wp-block-paragraph">Alarm on that being unexpectedly low rather than on it being zero. Zero-row alarms feel precise, but the more common failure is partial: a cursor that only advances through the first page, a rate limit that truncates the pull, an incremental sync that silently narrows its window. Those deliver a fraction of the expected rows, not none of them.</p>



<p class="wp-block-paragraph">There is a subtlety here that trips people. If a run fails before emitting <code>RecordsWritten</code> at all, a low-volume alarm on that metric sees missing data, not a zero. Metric math fixes it cleanly:</p>



<pre class="wp-block-code"><code>FILL(m1, 0)</code></pre>



<p class="wp-block-paragraph">Wrapping the metric in <code>FILL</code> with a value of 0 substitutes zero for every empty period, so an absent run and an empty run both breach the same alarm. You get one alert instead of two competing ones, and it means what you think it means.</p>



<p class="wp-block-paragraph">CloudWatch anomaly detection is the obvious upgrade for seasonal volume. It models expected behaviour and alarms on deviation from the band, which handles a Monday-morning spike that a static threshold cannot. I still would not reach for it first: it needs a stretch of representative history before the band is trustworthy, and during that stretch it will be confidently wrong. Start with a static floor well below your observed minimum, and move to anomaly detection once you know what normal looks like and can see the fixed number fighting you.</p>



<h2 class="wp-block-heading">Signal 3: freshness, the metric almost nobody emits</h2>



<p class="wp-block-paragraph">This is the one that separates a pipeline you can trust from one you merely watch. Liveness and volume can both look perfect while the data is stale, because a job that re-reads the same window every run produces rows, produces a heartbeat, and produces nothing new.</p>



<p class="wp-block-paragraph">The metric to emit is lag: the difference in seconds between now and the maximum source-side modification timestamp in the batch you just landed. Not your own processing timestamp, which will always look current. The source system&#8217;s timestamp, the one that tells you how far behind the truth you are.</p>



<pre class="wp-block-code"><code>import time
import boto3

cw = boto3.client("cloudwatch")

def emit_freshness(source, obj, max_source_ts_epoch):
    """max_source_ts_epoch: max LastModifiedDate in this batch, as epoch seconds."""
    lag_seconds = int(time.time()) - int(max_source_ts_epoch)
    cw.put_metric_data(
        Namespace="Pipelines/Ingest",
        MetricData=[{
            "MetricName": "SourceLagSeconds",
            "Value": lag_seconds,
            "Unit": "Seconds",
            "Dimensions": [
                {"Name": "Source", "Value": source},
                {"Name": "Object", "Value": obj},
            ],
        }],
    )</code></pre>



<p class="wp-block-paragraph">Alarm with <code>GreaterThanThreshold</code>, and set the threshold to roughly two schedule intervals plus your worst observed run duration. An hourly pipeline that takes twelve minutes on a bad day gets a threshold somewhere around two and a half hours. Tighter than that and you page on ordinary jitter.</p>



<p class="wp-block-paragraph">Freshness is also the one metric worth putting in front of non-engineers. &#8220;Your Salesforce data is currently 40 minutes behind&#8221; is a sentence a stakeholder can act on; &#8220;ExecutionsSucceeded is 1&#8221; is not. If you surface dashboards to a business audience through Grafana Cloud or QuickSight over the CloudWatch data source, lag is the panel that earns its space.</p>



<h2 class="wp-block-heading">Signal 4: shape, using logs instead of counters</h2>



<p class="wp-block-paragraph">Shape problems are schema drift and value drift. A SaaS admin renames a custom field, a picklist gains a value your mapping does not handle, a required field starts arriving null for a subset of records. The rows keep flowing. The row count looks fine. Downstream joins quietly drop records.</p>



<p class="wp-block-paragraph">Counters alone are the wrong shape here, because what you need at 2am is not &#8220;the null rate rose&#8221;, it is &#8220;which records, and what changed&#8221;. Embedded Metric Format is built for exactly that split: you write one structured JSON log line, CloudWatch extracts the numbers as real metrics you can alarm on, and the full event stays queryable in Logs Insights for the investigation.</p>



<pre class="wp-block-code"><code>{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "Pipelines/Quality",
        "Dimensions": [["Source", "Object"]],
        "Metrics": [
          { "Name": "NullRequiredFields", "Unit": "Count" },
          { "Name": "UnmappedFields", "Unit": "Count" }
        ]
      }
    ]
  },
  "Source": "salesforce",
  "Object": "opportunity",
  "NullRequiredFields": 412,
  "UnmappedFields": 2,
  "unmapped_field_names": ["Renewal_Tier__c", "Partner_Code__c"],
  "batch_id": "2f8c1a9e"
}</code></pre>



<p class="wp-block-paragraph">Two things to get right. Every name listed in <code>Dimensions</code> must also exist as a top-level property in the same JSON object, or extraction silently produces nothing. And note that <code>unmapped_field_names</code> and <code>batch_id</code> are properties, not dimensions: they ride along in the log event for querying but never become metric dimensions, which is exactly what you want.</p>



<p class="wp-block-paragraph">When the alarm fires, the investigation is a Logs Insights query against the same log group:</p>



<pre class="wp-block-code"><code>fields @timestamp, Object, UnmappedFields, unmapped_field_names
| filter Source = "salesforce" and UnmappedFields &gt; 0
| sort @timestamp desc
| limit 50</code></pre>



<p class="wp-block-paragraph">That hands you the new field names in seconds, which is the difference between a five-minute fix and an hour of guessing.</p>



<h2 class="wp-block-heading">Keeping CloudWatch data pipeline monitoring off the top of your bill</h2>



<p class="wp-block-paragraph">Custom metrics and log ingestion are consistently the two largest CloudWatch line items for data teams. Neither has a flat fee, so the cost is a direct function of choices you make in code. Three matter far more than the rest.</p>



<h3 class="wp-block-heading">Cardinality is the cost lever</h3>



<p class="wp-block-paragraph">A custom metric is billed per unique combination of namespace, metric name and dimension values. <code>Source</code> with five values and <code>Object</code> with twenty gives you a hundred metrics per metric name. Add <code>Tenant</code> with two hundred values and you have twenty thousand. Multiply by four metric names and the number stops being a rounding error.</p>



<p class="wp-block-paragraph">Never put a run ID, batch ID, record ID or raw error string in a dimension. Those belong in the log event. Dimensions are for things you would actually build an alarm on, and you are not going to build an alarm per batch.</p>



<h3 class="wp-block-heading">EMF over PutMetricData for anything high-frequency</h3>



<p class="wp-block-paragraph"><code>PutMetricData</code> is a synchronous API call on your critical path. In Lambda that means duration you pay for, plus a request that can throttle and needs handling. EMF avoids both: you write JSON to stdout, CloudWatch Logs extracts the metrics asynchronously, your function returns. For a per-run summary either is fine. For anything per batch or per record, EMF is the one I reach for.</p>



<h3 class="wp-block-heading">The Infrequent Access log class will break this silently</h3>



<p class="wp-block-paragraph">This one has caught competent people. CloudWatch Logs offers a Standard and an Infrequent Access log class, and IA has a lower ingestion price, which makes it an obvious target during a cost review. According to the AWS documentation, the IA class does not support embedded metric format, metric filters or subscription filters. Logs Insights queries still work.</p>



<p class="wp-block-paragraph">So if somebody moves your pipeline log group to IA to save money, every EMF-derived metric stops being extracted. The alarms built on those metrics go to <code>INSUFFICIENT_DATA</code>, or worse, sit in OK if they use the default missing-data behaviour. The logs still arrive. The dashboard still exists. The metrics are just gone.</p>



<p class="wp-block-paragraph">Log class cannot be changed after a log group is created, so recovery means a new Standard log group and redirecting output to it. Keep metric-backing log groups on Standard and point IA at the genuinely archival stuff: verbose debug output, raw payload dumps, anything you only read during a forensic investigation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Routing: one page, not four</h2>



<p class="wp-block-paragraph">Build all four signals and one broken pipeline fires the heartbeat, volume and freshness alarms within minutes of each other. Three pages, one problem. That is how alert fatigue starts.</p>



<p class="wp-block-paragraph">Composite alarms fix it. Build one composite alarm per pipeline whose rule combines the child alarms with OR, put the notification action there, and leave the child alarms with no actions at all. They become diagnostic detail you read after the page rather than three separate interruptions.</p>



<p class="wp-block-paragraph">SNS to email is fine for one person and poor past that, since it has no deduplication, escalation or acknowledgement. With more than one person on call, route through something with a schedule such as PagerDuty, Opsgenie or Better Stack. If your team already lives in Grafana Cloud or Datadog for application monitoring, pulling CloudWatch alarm state in through their AWS integration beats maintaining two alerting surfaces.</p>



<h2 class="wp-block-heading">Troubleshooting: the metric exists but the alarm never fires</h2>



<p class="wp-block-paragraph">Roughly in order of how often I have seen each one.</p>



<ul class="wp-block-list"><li><strong>Dimension mismatch.</strong> The alarm&#8217;s dimension set must match the published metric&#8217;s dimension set exactly, all of them, in the same values. A metric published with <code>Source</code> and <code>Object</code> is a different metric from one published with only <code>Source</code>. Run <code>aws cloudwatch list-metrics --namespace "Pipelines/Ingest"</code> and compare the output against your alarm definition character by character.</li><li><strong>Alarm stuck in INSUFFICIENT_DATA.</strong> The metric is not being published at all for that dimension set, or the period is shorter than the publishing interval so most periods are empty. Lengthen the period to at least the emission interval.</li><li><strong>Alarm stuck in OK during a real outage.</strong> Classic default <code>missing</code> behaviour. Any alarm whose purpose is to detect absence needs <code>--treat-missing-data breaching</code>.</li><li><strong>EMF metrics never appear.</strong> Check the log group&#8217;s class first, then confirm every dimension name in the <code>Dimensions</code> array exists as a top-level property in the same JSON object, then confirm the JSON is a single line with no wrapping prefix from your logger.</li><li><strong>Freshness spikes on a schedule.</strong> Almost always a timezone mismatch between how the SaaS API reports modification timestamps and how you convert them. Normalise everything to UTC epoch seconds at the boundary and stop converting anywhere else.</li><li><strong>Row counts double intermittently.</strong> Retries emitting the metric twice for one logical run. Emit volume metrics once, after the write is confirmed, not inside the retried block.</li></ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Alarming only on job failure, which misses every silent-success failure mode.</li><li>Leaving <code>TreatMissingData</code> at its default on alarms designed to detect absence.</li><li>Alarming on zero rows instead of on unexpectedly low rows, so partial extractions slip through.</li><li>Emitting a processing timestamp as the freshness metric, which is always current and therefore always useless.</li><li>Putting high-cardinality identifiers into metric dimensions and then being surprised by the bill.</li><li>Moving pipeline log groups to the Infrequent Access class without checking what depends on metric extraction.</li><li>Wiring notification actions to every child alarm instead of to one composite alarm per pipeline.</li><li>Building dashboards nobody looks at instead of alarms that reach a person.</li></ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list"><li>Emit liveness, volume, freshness and a shape metric for every source. Four metrics per source is a small amount of code and covers most of what actually breaks.</li><li>Define alarms in Terraform or CloudFormation alongside the pipeline, so a new source cannot ship without monitoring.</li><li>Use consistent dimension names across every pipeline. <code>Source</code> and <code>Object</code> everywhere makes search expressions and dashboards trivial; a mix of <code>source</code>, <code>system</code> and <code>connector</code> makes them impossible.</li><li>Set alarm descriptions to a one-line runbook. The person paged at 3am should not have to guess what the alarm means.</li><li>Test the absence case deliberately. Disable the schedule in a non-production account and confirm the heartbeat alarm fires. An untested absence alarm is an assumption.</li><li>Set explicit log retention on every pipeline log group. Never-expire is the default and it compounds quietly.</li><li>Keep an eye on the source side too, not just yours. Most SaaS APIs publish their own rate limit or API usage counters, and pulling those into a metric is often the earliest possible warning.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is CloudWatch enough on its own for data pipeline monitoring?</h3>



<p class="wp-block-paragraph">For alerting, yes, if your workloads are on AWS. It has metrics, alarms, composite alarms and log querying, and it is already collecting service metrics you would otherwise have to ship somewhere. Where it gets weak is exploratory analysis and cross-source correlation, which is where teams add Grafana, Datadog or a dedicated data-observability tool on top. Run CloudWatch for the paging path first and add the analysis layer when you have an actual gap.</p>



<h3 class="wp-block-heading">How do I monitor a pipeline that runs on a managed service like Glue or AppFlow?</h3>



<p class="wp-block-paragraph">The same four signals apply; only the emission point changes. Managed services publish their own metrics, but those describe the service&#8217;s execution, not your data. Where you cannot add code inside the job, hook the completion event through EventBridge to a small Lambda function that inspects what landed and emits the volume and freshness metrics. It is a handful of lines and it is the only way to get data-level signals out of a job you do not control.</p>



<h3 class="wp-block-heading">Should each tenant get its own metric in a multi-tenant pipeline?</h3>



<p class="wp-block-paragraph">Usually not as a dimension. Tenant counts grow, and cardinality is billed per unique dimension combination. Emit aggregate metrics with <code>Source</code> and <code>Object</code> dimensions, put the tenant identifier in the log event as a property, and use Logs Insights to slice by tenant when you need to. Break out a dimension per tenant only for the small number of accounts where a contractual SLA makes it worth the cost.</p>



<h3 class="wp-block-heading">What is a sensible freshness threshold?</h3>



<p class="wp-block-paragraph">Start at two schedule intervals plus your worst observed run duration, then adjust after watching real data for a couple of weeks. The threshold that matters is the one the business would notice, so if a stakeholder can tolerate half a day of staleness on a nightly export, do not page at four hours. Match the alarm to the actual consequence.</p>



<h3 class="wp-block-heading">Why does my alarm sit in INSUFFICIENT_DATA instead of ALARM?</h3>



<p class="wp-block-paragraph">CloudWatch has no data points to evaluate for that metric and dimension set. Either nothing is publishing, the dimensions do not match what is published, or the alarm period is shorter than the interval at which you emit. Confirm the metric exists with <code>list-metrics</code> first, and only then look at the alarm configuration.</p>



<h3 class="wp-block-heading">How much does this cost to run?</h3>



<p class="wp-block-paragraph">It depends on three things you control: the number of unique custom metrics, the volume of log data ingested, and how much data your Logs Insights queries scan. Alarms and dashboards are minor by comparison. Keep dimension cardinality low, set retention deliberately, and query narrow time windows, and the monitoring stays well below the cost of the pipeline it protects. Check current rates on the AWS pricing page rather than trusting any number in a blog post, including this one.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Effective CloudWatch data pipeline monitoring is built around absence, not around errors. Your SaaS integrations will fail far more often by succeeding at nothing than by throwing an exception, and every default in CloudWatch treats an absent metric as a non-event.</p>



<p class="wp-block-paragraph">If you only change one thing after reading this, add a heartbeat metric with <code>--treat-missing-data breaching</code> to your most important pipeline, then go and disable its schedule in a test account and confirm the alarm actually fires. If it does not, you have just found out on your own terms instead of from a stakeholder asking why the numbers look wrong.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your pipeline monitoring?</h2>



<p class="wp-block-paragraph">I work with teams running SaaS-to-warehouse pipelines on AWS, usually at the point where something broke quietly and nobody wants that to happen twice. Things I can help with here:</p>



<ul class="wp-block-list"><li>Auditing existing CloudWatch alarms for silent-failure gaps, particularly missing-data configuration and dimension mismatches that leave alarms permanently green.</li><li>Instrumenting liveness, volume, freshness and data-quality metrics into pipelines built on Lambda, Glue, Step Functions or self-hosted schedulers.</li><li>Designing EMF logging and metric namespaces that stay useful as source and object counts grow, without cardinality running away.</li><li>Building composite alarms and routing so one broken pipeline produces one actionable page rather than a cluster of them.</li><li>Cutting CloudWatch spend on log ingestion and custom metrics without losing the signals you actually rely on.</li><li>Writing Terraform modules that ship monitoring alongside every new source connector, so coverage is structural rather than remembered.</li></ul>



<p class="wp-block-paragraph">If you have a pipeline that has gone quiet, send me the alarm configuration and a Logs Insights output from the run in question and I will tell you what it is missing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/">CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Data Pipeline Operations Dashboard That Catches Silent Failures</title>
		<link>https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 15 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[OpenTelemetry]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[PromQL]]></category>
		<category><![CDATA[Pushgateway]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[StatsD]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=227</guid>

					<description><![CDATA[<p>Green DAGs and wrong numbers are the same morning. This is how to build a data pipeline operations dashboard around the four questions that actually matter, where each signal comes from, and how to turn the panels into alerts that fire before your users notice.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/">Building a Data Pipeline Operations Dashboard That Catches Silent Failures</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually lands around nine in the morning, from someone in finance or ops: &#8220;Is the revenue dashboard broken, or is it just late?&#8221;</p>



<p class="wp-block-paragraph">So you open the orchestrator. Every task is green. The run started on schedule, finished inside its usual window, exit code zero across the board. Nothing paged. And the numbers are still wrong.</p>



<p class="wp-block-paragraph">That gap is the whole reason to build a data pipeline operations dashboard. Not to watch DAGs go green, which the orchestrator UI already does perfectly well, but to answer the questions the orchestrator structurally cannot answer. This post covers the four signals worth putting on a screen, where each one actually comes from, how to wire them into Prometheus and Grafana without inventing a metrics platform, and how to turn the panels into alerts that fire before a human notices.</p>



<h2 class="wp-block-heading">Why green pipelines still ship bad data</h2>



<p class="wp-block-paragraph">An orchestrator tracks process outcomes. Did the Python exit cleanly, did the SQL statement return without an error, did the container terminate zero. That is a real signal and you should keep it. It is just answering a different question than the one your users care about.</p>



<p class="wp-block-paragraph">Consider what a perfectly successful run looks like when the upstream source has quietly changed:</p>



<ul class="wp-block-list">
<li>An API starts paginating differently and your extractor pulls the first page only. It ran fine. It loaded 500 rows instead of 500,000.</li>
<li>A source column gets renamed and your transform selects it with a <code>coalesce</code> default. It ran fine. Every row now says &#8220;unknown&#8221;.</li>
<li>A partition arrives late, so the incremental filter finds nothing to process. It ran fine. It did nothing.</li>
<li>An upstream team backfills, and your dedupe key no longer holds. It ran fine. Everything is doubled.</li>
</ul>



<p class="wp-block-paragraph">None of those raise an exception. All four are visible in under a second if you are tracking row counts and freshness at the table level. This is the failure family that eats the most incident time, because the clock does not start when the pipeline breaks. It starts when a human notices, which can be days later, and by then the bad data has been copied into reports, cached in a BI extract, and quoted in a meeting.</p>



<h2 class="wp-block-heading">The four questions a data pipeline operations dashboard has to answer</h2>



<p class="wp-block-paragraph">Resist the instinct to start from what your tools can emit. Start from the questions you ask during an incident, in the order you ask them. Everything that does not answer one of these is a second-page panel.</p>



<h3 class="wp-block-heading">Did it run at all?</h3>



<p class="wp-block-paragraph">Sounds trivial, and it is the one most homegrown dashboards get wrong. A panel showing &#8220;last run: success&#8221; does not distinguish between a pipeline that succeeded ten minutes ago and one that succeeded eleven days ago and has not been scheduled since. You want time since last successful completion, as a number that grows, not a status pill that sits there looking calm.</p>



<h3 class="wp-block-heading">Did it finish in time?</h3>



<p class="wp-block-paragraph">Duration matters less than duration against a deadline. A load that takes forty minutes is fine if the report is read at 9am and the run starts at 6am. The same forty minutes is an incident if someone moved the schedule to 8:40. Put the deadline on the panel as a threshold line, not in a runbook. Trending duration also catches slow decay: a stage that creeps from five minutes to fifty usually means data growth, a bad query plan, or resource contention, and it will breach your window eventually.</p>



<h3 class="wp-block-heading">Did the right amount of data arrive?</h3>



<p class="wp-block-paragraph">Rows in versus rows out, per stage. This is the highest value panel per unit of effort on the entire dashboard. A drop from a million rows to twenty thousand is not subtle once you plot it, but it is completely invisible in an exit code. Compare against the same weekday from the previous week rather than against yesterday, because most business data has a weekly shape and Monday-versus-Sunday comparisons generate noise you will learn to ignore.</p>



<h3 class="wp-block-heading">Is the data still the shape you expect?</h3>



<p class="wp-block-paragraph">Null rates, distinct counts on key columns, and schema changes. This is the panel almost nobody builds, and it is where the expensive failures live. A gradual rise in null rate on a joining key means an upstream system is degrading and your joins are silently dropping rows. You do not need statistical anomaly detection to catch that. You need the number plotted over time next to the others.</p>



<p class="wp-block-paragraph">There is a fifth question that belongs on a different surface: what breaks downstream when this table is wrong. That is lineage, and it answers blast radius rather than health. Keep it one click away, not on the operations screen.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where each signal actually comes from</h2>



<p class="wp-block-paragraph">Three sources, three different collection mechanisms. This is the part that takes an afternoon rather than a sprint, provided you do not try to build a unified agent.</p>



<h3 class="wp-block-heading">The orchestrator, for run and timing signals</h3>



<p class="wp-block-paragraph">Airflow has emitted StatsD metrics for a long time, and that path is still the one most deployments run. Airflow speaks StatsD, so you put a <code>statsd_exporter</code> in front of it to translate into Prometheus format. Enable it in the metrics section of your config, or with the equivalent <code>AIRFLOW__METRICS__</code> environment variables if you are on containers:</p>



<pre class="wp-block-code"><code>[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 9125
statsd_prefix = airflow
metrics_allow_list = scheduler,executor,dagrun,pool,triggerer</code></pre>



<p class="wp-block-paragraph">Two things worth knowing before you roll this out. First, <code>metrics_allow_list</code> is not optional at any real scale. Airflow emits per-DAG and per-task metric names, so without a prefix filter your cardinality grows with every DAG anyone adds, and you find out when the time series database starts struggling. Second, Airflow expects the StatsD destination to be resolvable at startup, so a scheduler that boots before the exporter can fail to come up. Order your dependencies accordingly.</p>



<p class="wp-block-paragraph">StatsD has no concept of labels, so the exporter has to reconstruct them from the dotted metric name. That mapping file is the fiddly part:</p>



<pre class="wp-block-code"><code>mappings:
  - match: "*.dag.*.*.duration"
    match_metric_type: observer
    name: "airflow_task_duration"
    labels:
      airflow_id: "$1"
      dag_id: "$2"
      task_id: "$3"</code></pre>



<p class="wp-block-paragraph">The wildcards map positionally onto <code>$1</code>, <code>$2</code>, <code>$3</code>, and <code>match_metric_type</code> has to match what Airflow actually sends, timer versus counter versus gauge. Get that wrong and the metric silently never appears at the exporter, which is a genuinely annoying half hour of debugging. Check the exporter&#8217;s own metrics endpoint before you go looking at Prometheus, because if it is not there it never left Airflow.</p>



<p class="wp-block-paragraph">Airflow also has a native OpenTelemetry path for both metrics and traces, which skips the StatsD translation entirely and gives you real labels rather than reconstructed ones:</p>



<pre class="wp-block-code"><code>[metrics]
otel_on = True
otel_host = otel-collector
otel_port = 4318
otel_prefix = airflow
otel_interval_milliseconds = 30000</code></pre>



<p class="wp-block-paragraph">The port there is the collector&#8217;s OTLP HTTP receiver, and you need an OpenTelemetry Collector or compatible endpoint in the path. If you are standing up a new stack, this is the one I would reach for, because the label problem disappears and traces give you per-task spans under a DAG run parent span, which is worth a lot during a slow-pipeline investigation. If you already have working StatsD mappings and dashboards built on them, migrating is real work for a modest gain, and the honest answer is that it can wait. Metric names do shift between Airflow major versions either way, so pin your dashboard queries to the metrics reference for the version you actually run.</p>



<p class="wp-block-paragraph">Managed Airflow changes the plumbing but not the shape. On Amazon MWAA the metrics land in CloudWatch instead, and Google Cloud Composer and Astronomer each have their own export path. The four questions do not change.</p>



<h3 class="wp-block-heading">The transformation layer, for data-level signals</h3>



<p class="wp-block-paragraph">If you use dbt, most of what you need is already being written to disk and thrown away. Every invocation produces <code>run_results.json</code> in the target directory, containing per-node status and timing, and <code>dbt source freshness</code> produces its own <code>sources.json</code>. The <code>manifest.json</code> holds the dependency graph, which is your lineage.</p>



<p class="wp-block-paragraph">The cheapest useful thing you can do is parse the failures out at the end of the run:</p>



<pre class="wp-block-code"><code>jq -r '.results[]
  | select(.status != "success")
  | [.unique_id, .status, (.execution_time | tostring)]
  | @tsv' target/run_results.json</code></pre>



<p class="wp-block-paragraph">One trap if you run dbt in ephemeral Kubernetes pods: the target directory dies with the pod. Upload the artifacts to object storage as the last step inside the same task that ran dbt, not as a downstream task, or you will find your history is empty exactly when you want it.</p>



<p class="wp-block-paragraph">Elementary is the low-friction option here. It installs as a dbt package with an on-run-end hook that loads the artifacts into your warehouse as ordinary tables, which means your dashboard can query test results and model timings in SQL alongside everything else. Great Expectations and Soda cover the same validation ground with more expressive checks and more setup. The commercial platforms, Monte Carlo and the vendors around it, add automated anomaly detection and column-level lineage across the whole stack, and the genuine case for them is that they find the checks you did not think to write. The trade-off is that you are paying per monitored table for statistical inference, and on a stack of twenty tables that inference is not telling you much that a row-count panel and a null-rate panel would not.</p>



<p class="wp-block-paragraph">Without dbt, you write the check yourself and it is still short. One query per critical table, run on a schedule:</p>



<pre class="wp-block-code"><code>SELECT
  COUNT(*)                                        AS row_count,
  MAX(updated_at)                                 AS last_event,
  SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) AS null_keys
FROM analytics.orders
WHERE loaded_at &gt;= CURRENT_DATE - 1;</code></pre>



<h3 class="wp-block-heading">Batch jobs, and the Pushgateway caveat</h3>



<p class="wp-block-paragraph">Prometheus pulls. Batch jobs finish and vanish long before a scrape interval comes round, so there is nothing to pull from. The Pushgateway exists for exactly this: short-lived service-level jobs push their results to it, and Prometheus scrapes the gateway instead.</p>



<pre class="wp-block-code"><code>cat &lt;&lt;EOF | curl --data-binary @- 
  http://pushgateway:9091/metrics/job/orders_sync
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $(date +%s)
# TYPE pipeline_rows_loaded gauge
pipeline_rows_loaded ${ROWS}
EOF</code></pre>



<p class="wp-block-paragraph">Now the caveat, which is the single most common way a pipeline dashboard lies to you. The Pushgateway is a cache, not an aggregator, and it never forgets. Push a success metric, then have the job stop running entirely, and that success sits there being scraped forever. Your dashboard stays green because the last known value is green, and the fact that nothing has updated it in six days is not something a status panel can see.</p>



<p class="wp-block-paragraph">Two defences, and you want both. First, always push a timestamp gauge rather than a boolean status, so staleness is arithmetic rather than inference. Second, alert on absence as well as on value, because the Pushgateway also exposes <code>push_time_seconds</code> per group, which tells you when anything last arrived regardless of what it said. Prometheus&#8217;s usual safety net, the <code>up</code> metric going to zero when a target disappears, does not apply here: Prometheus is scraping the gateway successfully, and the gateway is happily serving stale data.</p>



<p class="wp-block-paragraph">Also keep instance identifiers out of the grouping labels. Push per service, not per worker, or you accumulate a permanent time series for every container that ever ran.</p>



<h2 class="wp-block-heading">Laying out the dashboard so it works during an incident</h2>



<p class="wp-block-paragraph">The layout constraint is that someone is reading this at 7am on a phone with a stakeholder waiting. Three rows, top to bottom, in the order you need them:</p>



<ol class="wp-block-list">
<li><strong>Freshness across everything.</strong> One panel, one row per critical table, showing time since last successful load against its threshold. If this row is clean, the pipelines are fine and the problem is elsewhere. That is a genuinely useful thing to establish in five seconds.</li>
<li><strong>Volume and duration per pipeline.</strong> Row counts with a week-over-week comparison, and stage duration with the deadline drawn on. This is where you localise a problem to a specific pipeline and stage.</li>
<li><strong>Data quality detail.</strong> Null rates, failed test counts, schema change events. This is where you work out what actually went wrong once you know where.</li>
</ol>



<p class="wp-block-paragraph">The freshness query in PromQL, given the timestamp gauge above:</p>



<pre class="wp-block-code"><code>time() - max by (job) (pipeline_last_success_timestamp_seconds)</code></pre>



<p class="wp-block-paragraph">Grafana renders that nicely as a table with per-row thresholds, which is what makes the top row scannable. If you are self-hosting, Prometheus plus Grafana on a modest VPS from Hetzner, InterServer or DigitalOcean handles a surprising amount of pipeline metrics, because pipeline telemetry is low-volume compared to application metrics. Grafana Cloud&#8217;s free tier is a reasonable alternative if you do not want to own the storage, and Datadog or New Relic make sense if your organisation is already paying for one of them and you would rather have one pane than a better one.</p>



<p class="wp-block-paragraph">Keep it to one screen. Every panel you add past that dilutes the ones that matter, and a dashboard nobody can read at a glance is a dashboard nobody opens.</p>



<h2 class="wp-block-heading">Turning panels into alerts</h2>



<p class="wp-block-paragraph">A dashboard that requires someone to look at it is a dashboard that finds problems the morning after. The freshness panel converts into an alert almost directly:</p>



<pre class="wp-block-code"><code>groups:
  - name: pipelines
    rules:
      - alert: PipelineStale
        expr: time() - max by (job) (pipeline_last_success_timestamp_seconds) &gt; 5400
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} has not completed successfully in over 90 minutes"

      - alert: PipelineMetricsMissing
        expr: absent(pipeline_last_success_timestamp_seconds{job="orders_sync"})
        for: 30m
        labels:
          severity: page
        annotations:
          summary: "No metrics received from orders_sync"</code></pre>



<p class="wp-block-paragraph">The first catches a pipeline that is late or broken. The second catches the one that stopped reporting entirely, which the first cannot see, because an expression over a series that no longer exists returns nothing and an empty result does not fire. That pairing is the whole trick, and it applies to every push-based metric you collect.</p>



<p class="wp-block-paragraph">Set the threshold from the deadline the data has to meet, not from how long the job usually takes. Ninety minutes is not &#8220;the run takes an hour so give it some room&#8221;. It should be the point past which someone downstream is materially affected.</p>



<h2 class="wp-block-heading">Troubleshooting a dashboard that shows nothing useful</h2>



<p class="wp-block-paragraph"><strong>The metric never appears in Prometheus.</strong> Work backwards along the path, one hop at a time. Curl the statsd_exporter&#8217;s own metrics endpoint first. If the metric is not there, it never left the orchestrator, so the problem is the config or an allow-list that filtered it out. If it is there but not in Prometheus, the problem is your scrape config. Do not start in Grafana.</p>



<p class="wp-block-paragraph"><strong>The metric appears with no labels, or with the DAG name baked into the metric name.</strong> Your statsd_exporter mapping did not match. Check the wildcard count against the actual dotted name, and check <code>match_metric_type</code>, which is the usual culprit.</p>



<p class="wp-block-paragraph"><strong>Everything is green but the data is old.</strong> Classic Pushgateway staleness. Query <code>push_time_seconds</code> for the affected group and compare it to now. If it has not moved, the job is not running and you are looking at a cached value.</p>



<p class="wp-block-paragraph"><strong>Volume alerts fire every Monday.</strong> You are comparing against the previous day on data with a weekly cycle. Compare against the same day last week using an offset, or exclude weekends from the baseline.</p>



<p class="wp-block-paragraph"><strong>Gaps in the duration graph.</strong> If the metric is a timer sent only on completion, a run that is still going or that failed early sends nothing. That is correct behaviour, and it is why liveness needs its own panel rather than being inferred from the duration series.</p>



<p class="wp-block-paragraph"><strong>Prometheus memory climbing after rollout.</strong> Cardinality. Per-DAG and per-task metrics multiply, and any label carrying a run identifier, timestamp or UUID creates a permanent new series. Tighten the allow list and drop the offending labels.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating job success as data correctness. This is the entire problem, and every other mistake is a variation of it.</li>
<li>Pushing a boolean status instead of a timestamp, so staleness becomes invisible.</li>
<li>Alerting on threshold breaches but never on metric absence.</li>
<li>Building the dashboard before agreeing what &#8220;healthy&#8221; means for each table with the people who consume it.</li>
<li>Instrumenting every pipeline at once instead of the three that matter, then abandoning the project when the noise gets unmanageable.</li>
<li>Putting instance or pod identifiers into Pushgateway grouping labels and accumulating dead series forever.</li>
<li>Alerting on things nobody will act on at 3am. An alert that gets acknowledged and ignored twice has trained your team to ignore it permanently.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Baseline before you threshold. Collect for two or three weeks and look at the actual shape of the data before deciding what abnormal means.</li>
<li>Start with freshness on your three most critical tables. It catches more real problems per line of code than anything else you can instrument.</li>
<li>Define freshness thresholds from the consumer&#8217;s deadline, and write the consumer&#8217;s name next to the threshold so the number has an owner.</li>
<li>Version dashboards and alert rules in Git alongside the pipelines. A dashboard edited live in the UI is a dashboard that will be silently different next quarter.</li>
<li>Emit row counts at stage boundaries as a habit, not as a special case. In versus out at each hop is what turns &#8220;the numbers are wrong&#8221; into &#8220;stage three drops rows&#8221;.</li>
<li>Review alerts monthly and delete the ones that never fired usefully. Alert rules are code and they rot the same way.</li>
<li>Keep lineage accessible but off the main screen. It answers a different question and it competes for attention with the ones you need first.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the difference between pipeline monitoring and data observability?</h3>



<p class="wp-block-paragraph">Monitoring watches the process: did the job run, did it succeed, how long did it take. Observability watches the data the process produced: is it fresh, is there the right amount, has its shape changed. You need both, and the second is the one most teams are missing when they say their dashboards did not catch something.</p>



<h3 class="wp-block-heading">Do I need a commercial data observability platform?</h3>



<p class="wp-block-paragraph">Not to start. Freshness, volume and null-rate checks on your critical tables cover most real incidents and cost an afternoon. Commercial platforms earn their keep when you have hundreds of tables, several teams changing them independently, and nobody with time to hand-write checks. The honest signal that you have outgrown the homegrown version is when maintaining the checks becomes its own recurring task.</p>



<h3 class="wp-block-heading">Should I use StatsD or OpenTelemetry for Airflow metrics?</h3>



<p class="wp-block-paragraph">New stack, OpenTelemetry: you get real labels instead of reconstructing them from dotted names, and traces come along with it. Existing stack with working StatsD mappings, stay put until you have another reason to touch it. Metric coverage and naming differ between the two paths and between Airflow versions, so verify against the metrics reference for your version rather than copying a dashboard from a blog post.</p>



<h3 class="wp-block-heading">How do I monitor a pipeline that only runs weekly?</h3>



<p class="wp-block-paragraph">Same timestamp gauge, much larger threshold, and lean harder on the absence alert. Infrequent pipelines are where Pushgateway staleness does the most damage, because the last successful push can sit there for a fortnight looking entirely healthy. Check <code>push_time_seconds</code> for those groups specifically.</p>



<h3 class="wp-block-heading">What should a data pipeline operations dashboard show first?</h3>



<p class="wp-block-paragraph">Time since last successful load, per critical table, with its threshold visible. That one panel answers the question you will be asked most often, and when it is clean it rules out the entire pipeline layer in a glance.</p>



<h3 class="wp-block-heading">How many alerts is the right number?</h3>



<p class="wp-block-paragraph">Few enough that every page gets investigated. If someone can name an alert they routinely dismiss without looking, you have too many, and the cost is not the noise itself but the habit it builds. Start with freshness and absence per critical pipeline, add volume, and only go further when a real incident shows you a gap.</p>



<h3 class="wp-block-heading">Can I build this without an orchestrator like Airflow?</h3>



<p class="wp-block-paragraph">Yes. Cron jobs, Step Functions, GitHub Actions and shell scripts can all push a timestamp and a row count to a Pushgateway in three lines. The orchestrator gives you the run and timing signals for free, but it is the data-level signals that do the heavy lifting, and those come from querying the warehouse regardless of what triggered the load.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A data pipeline operations dashboard is not a nicer view of your orchestrator. It exists because the orchestrator can only tell you that a process finished, and the question you are actually being asked is whether the data is right.</p>



<p class="wp-block-paragraph">If you build one panel this week, build time since last successful load, sourced from a timestamp gauge, alerted on both threshold and absence. It costs an afternoon, it catches the failure mode that hides longest, and it turns &#8220;is the dashboard broken or just late?&#8221; into a question you can answer before anyone has to ask it.</p>



<h2 class="wp-block-heading">Need help building or fixing this?</h2>



<p class="wp-block-paragraph">I work with teams on pipeline observability and the plumbing underneath it. Things I regularly help with:</p>



<ul class="wp-block-list">
<li>Designing freshness, volume and quality checks for your critical tables, with thresholds derived from real consumer deadlines rather than guesses</li>
<li>Wiring Airflow into Prometheus and Grafana, including the statsd_exporter mappings and the cardinality controls that keep it affordable</li>
<li>Migrating an existing StatsD setup to the OpenTelemetry path without losing your dashboard history</li>
<li>Getting dbt artifacts out of ephemeral pods and into somewhere you can query them, with Elementary or with plain object storage</li>
<li>Auditing an existing dashboard and alert set to find what is stale, what will never fire, and what is quietly training your team to ignore pages</li>
<li>Standing up a self-hosted Prometheus and Grafana stack on a VPS, or sizing a managed alternative if owning the storage is not worth it to you</li>
</ul>



<p class="wp-block-paragraph">If you have a dashboard that went green through an incident, send me the panel queries and the alert rules and I will tell you where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-operations-dashboard/">Building a Data Pipeline Operations Dashboard That Catches Silent Failures</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</title>
		<link>https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 05 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Anomaly Detection]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[DQDL]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=148</guid>

					<description><![CDATA[<p>A SaaS admin changes a field and your pipeline stays green while the numbers drift. A practical guide to AWS Glue Data Quality for SaaS sources: where to run the checks, why nested payloads need flattening before DQDL can see them, which rule catches which failure, and the dynamic rules that pass silently because they have no history yet.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/">AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Nobody deployed anything. That is the part that makes it hard to explain.</p>



<p class="wp-block-paragraph">A Salesforce admin renames a custom field on Tuesday afternoon. Your connector keeps running, your Glue job keeps succeeding, the crawler happily adds a new column and quietly stops populating the old one. Three weeks later someone asks why the pipeline report has a cliff in it, and you spend an afternoon reconstructing a change nobody in engineering made and nobody logged.</p>



<p class="wp-block-paragraph">This post is about using AWS Glue Data Quality to catch that class of breakage. It is aimed at people pulling from SaaS sources into S3 or a lakehouse: Salesforce, HubSpot, Zendesk, Stripe, Shopify, Zoho, an internal partner API. I will cover where to run the checks, the flattening constraint that will bite you before anything else does, which DQDL rule actually catches which SaaS failure, when dynamic rules and anomaly detection quietly do nothing, and what should happen when a rule fails.</p>



<h2 class="wp-block-heading">Why SaaS data breaks in ways your own database doesn&#8217;t</h2>



<p class="wp-block-paragraph">When you own the source database, the schema is under change control: someone writes a migration, it gets reviewed, and if it breaks a consumer there is a commit to point at. A SaaS source has none of that. Three consequences:</p>



<ul class="wp-block-list">
<li><strong>The schema is edited by people who are not engineers.</strong> A sales ops admin adding a picklist value is a production schema change with no review, no notice and no version number.</li>

<li><strong>The payload is nested.</strong> SaaS APIs return structs and arrays: line items, related objects, custom field bags. That shape matters more than you would expect, for reasons in the next section but one.</li>

<li><strong>Absence is ambiguous.</strong> A field missing from a payload might mean &#8220;not set,&#8221; &#8220;not permitted for this user,&#8221; or &#8220;we removed it.&#8221; The connector usually cannot tell you which, and neither can your table.</li>
</ul>



<p class="wp-block-paragraph">Glue Data Quality is a rules engine built on Deequ that evaluates declarative checks written in DQDL, the Data Quality Definition Language, against a dataset. It is well suited to the first and third problems. The second one needs work before it is useful at all.</p>



<h2 class="wp-block-heading">Decide where the checks run before you write a rule</h2>



<p class="wp-block-paragraph">There are two places to evaluate a ruleset, and they answer different questions.</p>



<h3 class="wp-block-heading">Inside the Glue ETL job</h3>



<p class="wp-block-paragraph">The <code>EvaluateDataQuality</code> transform runs mid-pipeline against a DynamicFrame. It answers &#8220;is this batch safe to load?&#8221; and it can act on the answer: stop the job, or split failing rows into a quarantine prefix. This is also the only placement where row-level results exist, which matters because &#8220;which 400 rows are bad&#8221; is a far more useful answer than &#8220;the rule failed.&#8221;</p>



<h3 class="wp-block-heading">Against a Data Catalog table</h3>



<p class="wp-block-paragraph">You can attach a ruleset to a catalog table and run it on a schedule, independent of any job. This answers &#8220;is the table currently trustworthy?&#8221; and surfaces a quality score analysts can see before they query. The trade-off is that it is after the fact: by the time a catalog evaluation fails, the bad batch is already queryable. Analyzers are also not supported for catalog evaluations, so some dynamic-rule tooling behaves differently there.</p>



<p class="wp-block-paragraph">In practice I run both. The in-job ruleset is small and strict, and exists to stop garbage landing. The catalog ruleset is broader and looser, and exists so a table&#8217;s state is visible to people who will never open Glue Studio.</p>



<h2 class="wp-block-heading">The flattening problem that comes first</h2>



<p class="wp-block-paragraph">Here is the constraint that catches most people on SaaS data, stated plainly in the AWS documentation and easy to skim past: <strong>DQDL cannot evaluate nested or list-type columns.</strong> No structs, no arrays.</p>



<p class="wp-block-paragraph">Almost every interesting SaaS payload is nested. A Salesforce opportunity carries a related account struct, a Stripe invoice carries an array of line items, a Zendesk ticket carries a custom fields array of key-value pairs. Point a ruleset at the raw landed data and the fields you most want to check are the ones the engine cannot see.</p>



<p class="wp-block-paragraph">So you flatten first, with Glue&#8217;s relationalize or an explicit projection, then evaluate. That works, but it changes what your rules mean:</p>



<ul class="wp-block-list">
<li><strong>Your rules now describe the projection, not the payload.</strong> If your flattening step drops a field, no rule will notice, because from DQDL&#8217;s point of view that field was never there.</li>

<li><strong>Exploding an array changes the grain.</strong> One invoice with five line items becomes five rows. Every <code>RowCount</code> rule you write against that table is now counting line items, and a customer who starts itemising differently will move your row count without any data being wrong.</li>

<li><strong>Custom-field bags need pivoting, not flattening.</strong> A key-value array flattened naively gives you rows, not columns, and you cannot write a <code>ColumnValues</code> rule against a value whose meaning depends on a sibling key column. Pivot the ones you care about into real columns and check those.</li>
</ul>



<p class="wp-block-paragraph">The practical rule: put your quality checks immediately after the flattening step, and treat the flattening step itself as code that needs its own test. Glue Data Quality will not cover it for you.</p>



<h2 class="wp-block-heading">Matching SaaS breakage to the rule that catches it</h2>



<p class="wp-block-paragraph">DQDL has a long rule list and most guides just enumerate it. More useful is the other direction: what actually goes wrong with SaaS sources, and what catches each one.</p>



<h3 class="wp-block-heading">A field is renamed, removed, or added</h3>



<p class="wp-block-paragraph">This is the Tuesday-afternoon scenario. Two rules, doing two different jobs:</p>



<pre class="wp-block-code"><code>Rules = [
    ColumnExists "annual_revenue__c",
    ColumnCount = 47
]</code></pre>



<p class="wp-block-paragraph"><code>ColumnExists</code> protects the specific fields your downstream models depend on. <code>ColumnCount</code> protects against everything else: a field silently added, or one you forgot you needed. The first is precise and the second is a tripwire, and you want both because a rename shows up as one column gone and one column arrived, which leaves the count unchanged.</p>



<p class="wp-block-paragraph"><code>SchemaMatch</code> compares against a reference dataset&#8217;s schema, which is stricter and useful if you maintain a golden reference table. Bear in mind a hard column count on a SaaS object fails every time an admin adds a field, so decide whether that blocks or just warns.</p>



<h3 class="wp-block-heading">A field still exists but stopped being populated</h3>



<p class="wp-block-paragraph">Permissions change, a workflow gets disabled, an integration user loses field-level access. The column is still in the schema and is now all nulls.</p>



<pre class="wp-block-code"><code>Rules = [
    IsComplete "opportunity_id",
    Completeness "lead_source" &gt; 0.85,
    Completeness "email" &gt; 0.60 where "record_type = 'Contact'"
]</code></pre>



<p class="wp-block-paragraph"><code>IsComplete</code> is the absolute version: no nulls at all, right for keys. <code>Completeness</code> takes a threshold, which is what you want for optional business fields where a hard rule would fail constantly. The <code>where</code> clause is the one people miss: SaaS objects are usually polymorphic, and a completeness threshold that makes sense for contacts is nonsense across the whole table. Scope the rule to the record type.</p>



<h3 class="wp-block-heading">The sync ran short</h3>



<p class="wp-block-paragraph">A paginated extract stops early, a rate limit gets absorbed by retries, an incremental cursor drifts. The job succeeds and delivers less data than it should have. A static <code>RowCount &gt; 1000</code> stops catching this the moment you cross a thousand rows. Dynamic rules compare against your own history instead:</p>



<pre class="wp-block-code"><code>Rules = [
    RowCount &gt; min(last(3)),
    Sum "amount" &gt; avg(last(5)) * 0.7,
    DetectAnomalies "RowCount"
]

Analyzers = [
    RowCount,
    DistinctValuesCount "owner_id"
]</code></pre>



<p class="wp-block-paragraph"><code>last(k)</code> pulls the last k recorded values of that metric for the same dataset; with k greater than one you need an aggregation like <code>avg</code>, <code>min</code> or <code>max</code> to reduce them to one number. <code>DetectAnomalies</code> goes further, using forecasting over stored history to flag deviations with no threshold from you at all. That is the right tool for anything seasonal: a B2B pipeline that dips every weekend trips a naive trailing-average rule every Monday, while anomaly detection learns the shape.</p>



<p class="wp-block-paragraph">Analyzers deserve their own mention: they gather a metric without asserting anything, which builds the history dynamic rules and anomaly detection need. <code>DistinctValuesCount</code> on an owner or account column is a strong early signal for a partial sync, because row count can look fine while the number of distinct accounts represented quietly halves.</p>



<h3 class="wp-block-heading">Records arrive twice, or the key stops being a key</h3>



<p class="wp-block-paragraph">Replays, overlapping incremental windows and connector retries all produce duplicates, and SaaS &#8220;IDs&#8221; are less unique than vendor docs imply once you have merged sandboxes or multiple accounts.</p>



<pre class="wp-block-code"><code>Rules = [
    IsPrimaryKey "id",
    Uniqueness "external_reference" &gt; 0.99
]</code></pre>



<p class="wp-block-paragraph"><code>IsPrimaryKey</code> asserts unique and non-null together. <code>Uniqueness</code> with a threshold is the softer version for fields that are mostly-but-not-perfectly unique, which describes a lot of SaaS reference fields honestly.</p>



<h3 class="wp-block-heading">Objects stop agreeing with each other</h3>



<p class="wp-block-paragraph">Multi-object extracts rot quietly. Opportunities reference accounts that were never synced; invoice totals stop matching line item sums because one object&#8217;s extract failed and the other&#8217;s didn&#8217;t.</p>



<p class="wp-block-paragraph"><code>ReferentialIntegrity</code> checks that values in one dataset&#8217;s column exist in another&#8217;s. <code>RowCountMatch</code> and <code>AggregateMatch</code> compare counts and summary metrics across two datasets. These catch a broken join before an analyst does, and they need the multi-frame form of the transform with the second dataset supplied as an additional source.</p>



<h3 class="wp-block-heading">The sync stopped and nobody noticed</h3>



<p class="wp-block-paragraph">The quietest failure of all: nothing is wrong with the data, there just isn&#8217;t any new data. <code>DataFreshness</code> checks a timestamp column against a window. It is deliberately simple and its comparison granularity is limited, so for anything more nuanced the custom SQL rule is the escape hatch, running a query against the dataset and evaluating the result against an expression. DQDL is case sensitive and that rule type&#8217;s exact capitalisation has appeared more than one way across the docs and the visual builder, so copy it from the current DQDL reference rather than from memory.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Static, dynamic and ML: when each one lies to you</h2>



<p class="wp-block-paragraph">All three check styles have a mode where they pass while the data is wrong. Knowing which is which is most of the skill.</p>



<ul class="wp-block-list">
<li><strong>Static rules lie by going stale.</strong> A threshold that was correct when you wrote it stops being correct as the business grows, and a rule that can no longer fail is worse than no rule because it looks like coverage.</li>

<li><strong>Dynamic rules lie when they have no history.</strong> This one is genuinely nasty. <code>last(k)</code> reads previously recorded values for that metric, and if there aren&#8217;t any, it falls back to a default of zero. Add <code>RowCount &gt; avg(last(3))</code> to an existing job today and it will pass trivially for the first few runs, because almost any row count beats zero. It looks like a working check from the moment you deploy it. It isn&#8217;t one until it has accumulated runs.</li>

<li><strong>Anomaly detection lies while it is warming up, and after you retrain it badly.</strong> It needs a run of history before its forecasts mean anything, and if you leave a period of genuinely bad data in the training window, it learns that the bad shape is normal. You can exclude specific statistics from the calculation in the Data Quality tab, and you should, after any incident.</li>
</ul>



<p class="wp-block-paragraph">So when you add a dynamic or ML-based rule, write down the date and treat it as unarmed for the first couple of weeks. Keep a crude static rule alongside it in the meantime.</p>



<h2 class="wp-block-heading">What should happen when a rule fails</h2>



<p class="wp-block-paragraph">The transform returns a collection. The two members you care about are the row-level outcomes, which give you the original data with an evaluation result column appended per row, and the rule outcomes, which give you one row per rule with its verdict.</p>



<pre class="wp-block-code"><code>from awsgluedq.transforms import EvaluateDataQuality

dq_results = EvaluateDataQuality().process_rows(
    frame=flattened_opportunities,
    ruleset=ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "salesforce_opportunity",
        "enableDataQualityCloudWatchMetrics": "true",
        "enableDataQualityResultsPublishing": "true",
    },
)

row_level = SelectFromCollection.apply(
    dfc=dq_results, key="rowLevelOutcomes"
)</code></pre>



<p class="wp-block-paragraph"><code>dataQualityEvaluationContext</code> is the namespace your CloudWatch metrics land under. Name it after the source object rather than the job, because you will eventually want to alert per object.</p>



<p class="wp-block-paragraph">From there, three response patterns:</p>



<ol class="wp-block-list">
<li><strong>Fail fast.</strong> Stop the job on failure so nothing lands. Right for structural rules: if the primary key isn&#8217;t unique, downstream is going to be wrong no matter what you do next.</li>

<li><strong>Quarantine.</strong> Filter the row-level outcomes on the evaluation result column, write passing rows to the target and failing rows to a separate prefix. Right for value-level rules where 2% bad records shouldn&#8217;t block 98% good ones. The important discipline is that somebody has to actually look at the quarantine prefix; an unmonitored quarantine is a delete with extra steps.</li>

<li><strong>Warn and continue.</strong> Let everything land, publish the metrics, alert on the score. Right for new rules you don&#8217;t trust yet, and for the catalog-side ruleset.</li>
</ol>



<p class="wp-block-paragraph">Whichever you pick, get the failure out of Glue and into wherever your team actually looks. Results and anomaly predictions can be written back to catalog tables so they are queryable with plain SQL, and CloudWatch metrics plus EventBridge events give you the hook for paging or pausing a downstream pipeline. A Grafana or Datadog dashboard fed from those metrics beats a score in a console tab nobody opens.</p>



<h2 class="wp-block-heading">Troubleshooting AWS Glue Data Quality on SaaS pipelines</h2>



<h3 class="wp-block-heading">&#8220;Input data does not include column X&#8221;</h3>



<p class="wp-block-paragraph">A rule references a column the engine cannot find. On SaaS data that means one of three things: the field disappeared from the source, your flattening step renamed it, or the Data Catalog is out of sync with the underlying table. Check the catalog schema against the landed files before touching the ruleset; rerunning the crawler resolves a surprising share of these.</p>



<h3 class="wp-block-heading">A rule type isn&#8217;t recognised</h3>



<p class="wp-block-paragraph">DQDL is case sensitive and the rule set has grown over releases, so a rule copied from an older blog post can parse fine in your head and not at all in the engine. Check the rule name against the current DQDL rule type reference, and check that the feature is available on your Glue version. Anomaly detection in particular has a version floor.</p>



<h3 class="wp-block-heading">A dynamic rule never fails</h3>



<p class="wp-block-paragraph">Almost always the empty-history problem described above. Look at the statistics for that metric in the Data Quality tab. If there are fewer recorded points than your <code>last(k)</code> asks for, the rule is not doing what you think. It will start working on its own; the mistake is trusting it in the meantime.</p>



<h3 class="wp-block-heading">Evaluation is slow or the job runs out of memory</h3>



<p class="wp-block-paragraph">Each rule triggers analyzer computation over the dataset, and a large ruleset on a wide flattened SaaS table adds up fast. The transform exposes caching and observations-scope options to tune this. Before reaching for a bigger worker type, cut the ruleset down: most contain a dozen recommended rules nobody reads the results of.</p>



<h3 class="wp-block-heading">A &#8220;column reference is ambiguous&#8221; error</h3>



<p class="wp-block-paragraph">Classic flattening artefact. Two nested paths collapsed to the same leaf name, so <code>datechecked</code> now matches two things. Fix it in the projection with explicit aliases rather than trying to disambiguate inside DQDL.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Accepting the recommended ruleset wholesale. Recommendations are generated from a sample of current data, so they encode today&#8217;s accidents as tomorrow&#8217;s requirements.</li>

<li>Writing rules against nested data and assuming they cover the whole payload.</li>

<li>Deploying a dynamic rule and treating it as armed on day one.</li>

<li>Applying whole-table completeness thresholds to a polymorphic object instead of scoping with a <code>where</code> clause.</li>

<li>Quarantining failed rows to a prefix nobody monitors.</li>

<li>Writing <code>RowCount</code> rules against an exploded array without noticing the grain changed.</li>

<li>Failing the job on every rule, including cosmetic ones, until the team starts rerunning with checks disabled.</li>

<li>Keeping the ruleset in the console instead of in version control next to the job.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Split rules into blocking and reporting tiers, and be honest about which is which. A rule that always fails and never stops anything trains people to ignore the tab.</li>

<li>Pair <code>ColumnExists</code> with <code>ColumnCount</code>. One protects the fields you named, the other catches the drift you did not anticipate.</li>

<li>Add analyzers early, even for metrics you have no rule for yet. History has to accumulate before dynamic rules and anomaly detection are worth anything, and you cannot backfill it.</li>

<li>Keep DQDL in Git alongside the job definition and deploy it with Terraform or CloudFormation. Rulesets are code and should move through the same review as the pipeline they guard.</li>

<li>Namespace the evaluation context per source object so CloudWatch metrics and alerts can be routed per vendor.</li>

<li>Exclude incident periods from the anomaly detection training window once the incident is resolved.</li>

<li>Surface results where stakeholders already look. Writing outcomes back to catalog tables makes them queryable from Athena, which beats a score in a console tab.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much does AWS Glue Data Quality cost?</h3>



<p class="wp-block-paragraph">It bills on Glue DPU-hours like any other Glue workload, so cost is a function of evaluation time and DPUs used. The dimension that surprises people is anomaly detection, which consumes additional capacity per statistic analysed, so enabling it broadly across many columns and tables is where the bill moves. Model it against the current Glue pricing page; rates change and vary by region.</p>



<h3 class="wp-block-heading">Can it check data before it lands in S3?</h3>



<p class="wp-block-paragraph">Yes, if the check runs inside the Glue job. The transform evaluates a DynamicFrame in memory, so you can validate after extraction and before the write. That is the placement that lets you stop a bad load rather than report on one.</p>



<h3 class="wp-block-heading">Does it work on nested JSON from a SaaS API?</h3>



<p class="wp-block-paragraph">Not directly. DQDL does not evaluate nested or list-type columns, so you have to flatten the structure first and run the ruleset against the flattened frame. Plan the flattening deliberately, because your rules only ever see what the projection exposes.</p>



<h3 class="wp-block-heading">How does it compare to Great Expectations, Soda or Monte Carlo?</h3>



<p class="wp-block-paragraph">Glue Data Quality&#8217;s real advantage is that it is already inside the runtime and the catalog: no extra infrastructure, no separate agent, one IAM story. The disadvantages are real too. DQDL is less expressive than Python-based assertions, the nested-data limitation is a hard wall, and it only covers what Glue can read. Great Expectations gives you more expressive checks at the cost of running it yourself; Monte Carlo and similar platforms add lineage and cross-warehouse coverage Glue does not attempt. Glue-centric stack, start here. Several engines, a standalone tool fits better.</p>



<h3 class="wp-block-heading">Should I use the recommended rules?</h3>



<p class="wp-block-paragraph">As a starting draft, yes. As a ruleset, no. Recommendations are inferred from a sample of the data as it is today, which means any existing quality problem gets encoded as the expected baseline. Read every generated rule and delete the ones you would not have written yourself.</p>



<h3 class="wp-block-heading">Can I reconcile row counts against the SaaS source itself?</h3>



<p class="wp-block-paragraph">Not directly; it only sees datasets it can read. The usual pattern is to have the extraction step record the count the API reported, land that alongside the data as a control table, then use <code>RowCountMatch</code> or <code>AggregateMatch</code> to compare the two. That turns &#8220;did we get everything&#8221; into a rule the engine can evaluate.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">AWS Glue Data Quality is not really a data quality product. It is a way of writing down what you believe about a dataset so a machine can check it every run, and its value is entirely determined by whether those beliefs are written precisely enough to be wrong.</p>



<p class="wp-block-paragraph">For SaaS sources that means two habits. Flatten deliberately and check immediately after, because your rules only see the projection. And treat every new dynamic or ML-based rule as unarmed until it has history, because a rule that cannot fail yet looks exactly like a rule that is passing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with data quality on your Glue pipelines?</h2>



<p class="wp-block-paragraph">I work with teams whose SaaS pipelines run green while the numbers drift. Things I can help with:</p>



<ul class="wp-block-list">
<li>Writing a DQDL ruleset for a specific SaaS object that catches schema drift instead of just restating the current schema.</li>

<li>Designing the flattening and pivoting step for nested payloads and custom-field bags so the checks actually cover what matters.</li>

<li>Setting up quarantine and fail-fast paths in Glue ETL, with the routing and alerting that makes a quarantine prefix something people act on.</li>

<li>Cross-object reconciliation between SaaS extracts and control counts recorded at extraction time.</li>

<li>Tuning anomaly detection and dynamic rules, including cleaning incident periods out of the training window.</li>

<li>Getting data quality results out of the console and into Grafana, CloudWatch alarms or an Athena-queryable table.</li>
</ul>



<p class="wp-block-paragraph">If you have a ruleset that keeps passing while the data drifts, send me the DQDL and a schema dump. That is usually enough to see where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/">AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
