<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>ETL | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/etl/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/etl/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 13:07:08 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>ETL | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/etl/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates</title>
		<link>https://john-nessime.com/blog/devops/data-pipeline-freshness-monitoring/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 08 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[Deadman Alerting]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Kafka]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[PromQL]]></category>
		<category><![CDATA[Pushgateway]]></category>
		<category><![CDATA[SLO]]></category>
		<category><![CDATA[SRE]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=407</guid>

					<description><![CDATA[<p>Every task green, zero errors, and the dashboard still shows yesterday's numbers. Run status describes your code, not your data. Here is how to instrument freshness, latency and failure rate so your pipeline tells you before an analyst does.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-freshness-monitoring/">Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message lands at 8:40 in the morning. &#8220;Are the revenue numbers right? They look like yesterday&#8217;s.&#8221; You open the orchestrator and every task is green. The run finished in eleven minutes, comfortably inside its window. Zero errors. Zero retries. The pipeline did exactly what it was told to do, and the number on the dashboard is a day old.</p>



<p class="wp-block-paragraph">That gap is the entire problem. Run status tells you the code executed. It tells you nothing about whether data arrived. Data pipeline freshness monitoring is the signal that closes the gap, and most teams add it only after an analyst finds the stale table first.</p>



<p class="wp-block-paragraph">This post covers the three signals worth instrumenting for a batch or streaming pipeline: how current the data is, how long it took to get there, and how often the thing breaks. For each one I&#8217;ll cover what to measure, where to measure it, the queries and config that produce it, and the specific ways the measurement lies to you.</p>



<h2 class="wp-block-heading">The signal that lies to you first: run status</h2>



<p class="wp-block-paragraph">Job success is a statement about your code path, not about your data. A pipeline that reads an empty S3 prefix, transforms zero rows, writes zero rows and exits cleanly has succeeded. So has one whose upstream API silently started returning an empty page after a token rotation. So has one whose incremental watermark got stuck and now re-reads the same already-loaded slice on every run.</p>



<p class="wp-block-paragraph">All three are green. All three are producing stale data. This is why the ordering matters: freshness is the signal that fires first in a real incident, and run status is usually the last one to notice anything at all.</p>



<p class="wp-block-paragraph">So keep run status. It&#8217;s cheap and it catches crashes. Just stop treating it as your top-line health indicator.</p>



<h2 class="wp-block-heading">Freshness: measure the table, not the job</h2>



<p class="wp-block-paragraph">Freshness is the age of the newest record in a dataset. It&#8217;s measured against the dataset itself, not against the process that filled it, and that distinction is what makes it useful. If your job disappears entirely, freshness keeps climbing and keeps alerting. If your job succeeds while doing nothing, freshness keeps climbing and keeps alerting. It&#8217;s the one signal that survives both failure modes.</p>



<h3 class="wp-block-heading">Two clocks, and you need both</h3>



<p class="wp-block-paragraph">Every row usually carries two timestamps: when the event happened in the source system, and when your pipeline wrote it. Track the maximum of each.</p>



<ul class="wp-block-list">
<li><strong>Max event time</strong> tells you whether the upstream system is still producing. If this stops moving, the problem is upstream of you.</li>



<li><strong>Max load time</strong> tells you whether your pipeline is still writing. If this stops moving while event time is fine, the problem is yours.</li>
</ul>



<p class="wp-block-paragraph">Watching only one of them means every stale-data page starts with twenty minutes of figuring out which side of the boundary the fault sits on. Watching both answers that in the alert body.</p>



<p class="wp-block-paragraph">A freshness probe is a small scheduled query. This one is PostgreSQL syntax, and it returns both clocks plus a volume check in a single round trip:</p>



<pre class="wp-block-code"><code>select
  'orders' as dataset,
  extract(epoch from max(event_ts))  as max_event_ts,
  extract(epoch from max(loaded_at)) as max_loaded_at,
  count(*) filter (where loaded_at &gt;= now() - interval '1 hour') as rows_last_hour
from analytics.orders;</code></pre>



<p class="wp-block-paragraph">The row count matters. A pipeline can advance its load timestamp while writing almost nothing, which is what a partially broken source looks like. Freshness alone will not catch that; freshness plus volume will.</p>



<p class="wp-block-paragraph">Run that probe on a schedule that&#8217;s independent of the pipeline, push the results as gauges, and alert on age in PromQL:</p>



<pre class="wp-block-code"><code># Newest row is older than 90 minutes
time() - max by (dataset) (dataset_max_loaded_timestamp_seconds) &gt; 5400

# The probe itself has stopped reporting: a deadman check
absent(dataset_max_loaded_timestamp_seconds{dataset="orders"})</code></pre>



<p class="wp-block-paragraph">That second rule is the one people forget. A freshness metric that disappears looks identical to a healthy silence on a graph. <code>absent()</code> is what turns &#8220;no data&#8221; into a page.</p>



<h3 class="wp-block-heading">Where dbt fits</h3>



<p class="wp-block-paragraph">If you&#8217;re already running dbt, source freshness is built in and worth using before you write anything custom. You declare thresholds per source, and dbt queries the maximum of your timestamp column and compares it to now.</p>



<pre class="wp-block-code"><code>sources:
  - name: raw_shop
    schema: raw
    config:
      loaded_at_field: _loaded_at
      freshness:
        warn_after: {count: 2, period: hour}
        error_after: {count: 6, period: hour}
    tables:
      - name: orders
      - name: refunds
        config:
          freshness:
            warn_after: {count: 12, period: hour}
            error_after: {count: 24, period: hour}</code></pre>



<p class="wp-block-paragraph">Three details that trip people up. First, <code>dbt source freshness</code> is a separate command; <code>dbt build</code> does not run it, so a green build says nothing about source staleness. Second, it exits non-zero when a source hits its <code>error_after</code> threshold, which makes it a natural gate at the top of a job: fail fast rather than building models on stale input. Third, results land in <code>target/sources.json</code>, which is the artifact you parse if you want to distinguish warn from error, or ship the numbers into Prometheus rather than just failing the run.</p>



<pre class="wp-block-code"><code>dbt source freshness --select source:raw_shop --output target/freshness.json</code></pre>



<p class="wp-block-paragraph">One caveat on the YAML above: dbt moved these keys under a <code>config:</code> block in recent releases, and <code>loaded_at_field</code> followed later. Older projects nest them directly under the source. Check what your project&#8217;s version expects before copying, because a misplaced key fails quietly by simply not calculating freshness at all.</p>



<h2 class="wp-block-heading">Latency: name the clock before you name the number</h2>



<p class="wp-block-paragraph">&#8220;Our pipeline latency is twelve minutes&#8221; is meaningless until you say which two points you measured between. There are at least three plausible definitions, and teams routinely argue past each other because they&#8217;re each using a different one.</p>



<ul class="wp-block-list">
<li><strong>Run duration.</strong> Start to end of the job. Easy, and mostly useless for anyone downstream.</li>



<li><strong>Ingestion latency.</strong> Source extract to target load. This is what you control.</li>



<li><strong>End-to-end latency.</strong> Event time to the moment the row is queryable. This is what the business actually feels, and it includes queue wait, scheduler delay, and every upstream hop you don&#8217;t own.</li>
</ul>



<p class="wp-block-paragraph">Publish end-to-end as the headline and keep the others as breakdown. If you only track run duration, a scheduler backlog that delays every run by forty minutes is completely invisible to you: each individual run still takes eleven minutes.</p>



<p class="wp-block-paragraph">Instrument per stage with a histogram so you can ask percentile questions later without re-instrumenting:</p>



<pre class="wp-block-code"><code>histogram_quantile(
  0.95,
  sum by (le, stage) (rate(pipeline_stage_duration_seconds_bucket[6h]))
)</code></pre>



<p class="wp-block-paragraph">Use the median for capacity planning and p95 or p99 for the SLO. Averages hide the exact tail that generates the complaints, and on a pipeline that runs a few dozen times a day, a single pathological run is a real fraction of your day.</p>



<p class="wp-block-paragraph">For streaming, the equivalent is consumer lag expressed in time rather than offsets. Offset lag of fifty thousand messages means nothing without a rate; two hundred seconds of lag means something to everyone. Kafka Lag Exporter popularised this by interpolating a time estimate from observed offset and timestamp samples, exposing <code>kafka_consumergroup_group_max_lag_seconds</code> alongside the offset-based <code>kafka_consumergroup_group_lag</code>. Worth knowing: that project&#8217;s repository has been archived and is read-only, so if you&#8217;re starting fresh, check whether your broker vendor or a maintained fork covers it before you deploy something unmaintained into the critical path.</p>



<h2 class="wp-block-heading">Failure rate: decide what counts as a failure</h2>



<p class="wp-block-paragraph">Failure rate is trivially easy to compute and surprisingly easy to compute wrongly. The denominator and the definition both need a decision.</p>



<p class="wp-block-paragraph">Count runs, not tasks. A DAG with sixty tasks where one flaps on a transient network error looks catastrophic at task level and fine at run level. The run is the unit the consumer cares about.</p>



<p class="wp-block-paragraph">Count a run that succeeded on its third retry as a success for availability and a failure for a separate reliability metric. Both are true and they answer different questions. If your only metric folds retries into success, you will never see the slow degradation of an upstream API until it stops responding entirely.</p>



<pre class="wp-block-code"><code>sum by (pipeline) (rate(pipeline_runs_total{result="failure"}[6h]))
/
sum by (pipeline) (rate(pipeline_runs_total[6h]))</code></pre>



<p class="wp-block-paragraph">And add a category label for the failure reason at push time: source unavailable, schema mismatch, permission denied, timeout, validation failed. The rate tells you something is wrong. The category tells you who to wake up. Without it, every failure alert costs you a log dive before you can even route the incident.</p>



<h2 class="wp-block-heading">Getting metrics out of a job that exits</h2>



<p class="wp-block-paragraph">Prometheus scrapes. Batch jobs finish and vanish. Pushgateway bridges that: the job pushes before exiting, and Pushgateway holds the values for Prometheus to scrape on its own schedule.</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
set -euo pipefail

JOB="orders_load"
PGW="http://pushgateway.internal:9091"

start=$(date +%s)
if python /opt/pipelines/load_orders.py; then
  result=0
else
  result=1
fi
end=$(date +%s)

{
  cat &lt;&lt;EOF
# TYPE pipeline_run_duration_seconds gauge
pipeline_run_duration_seconds $((end - start))
# TYPE pipeline_last_run_timestamp_seconds gauge
pipeline_last_run_timestamp_seconds $end
# TYPE pipeline_last_run_success gauge
pipeline_last_run_success $((1 - result))
EOF
  if [ "$result" -eq 0 ]; then
    cat &lt;&lt;EOF
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $end
EOF
  fi
} | curl --fail --data-binary @- "$PGW/metrics/job/$JOB"

exit $result</code></pre>



<p class="wp-block-paragraph">The conditional block is doing real work. curl&#8217;s <code>--data-binary</code> issues a POST, and a POST to Pushgateway replaces only the metrics whose names appear in the payload, leaving the rest of the group intact. So a failing run updates the run timestamp and the success flag while leaving the previous <code>pipeline_last_success_timestamp_seconds</code> exactly where it was. That&#8217;s what lets <code>time() - pipeline_last_success_timestamp_seconds</code> keep climbing across consecutive failures. Send a PUT instead and you replace the whole group, wiping the value you needed.</p>



<p class="wp-block-paragraph">On the Prometheus side, one setting is not optional:</p>



<pre class="wp-block-code"><code>scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway.internal:9091']</code></pre>



<p class="wp-block-paragraph">Without <code>honor_labels: true</code>, Prometheus overwrites the <code>job</code> label your pipeline pushed with the scrape job&#8217;s own name, and every pipeline in your estate collapses into one indistinguishable series called <code>pushgateway</code>.</p>



<p class="wp-block-paragraph">The trap worth internalising: Pushgateway never forgets. Metrics persist until something explicitly deletes them or the process restarts. A pipeline you decommissioned last quarter is still cheerfully reporting a success timestamp and a duration, and it looks alive on every dashboard. Treat Pushgateway as a cache of past executions, not a picture of current state, and use the <code>push_time_seconds</code> gauge that Pushgateway attaches to each group to tell the difference between a fresh push and a fossil.</p>



<h3 class="wp-block-heading">If you&#8217;re on Airflow, the SLA feature is gone</h3>



<p class="wp-block-paragraph">This one will bite anyone upgrading. The <code>sla</code> and <code>sla_miss_callback</code> parameters were removed in Airflow 3.0, and the replacement, Deadline Alerts, arrived in 3.1. DAGs carrying the old configuration need manual migration; they don&#8217;t quietly keep working.</p>



<pre class="wp-block-code"><code>from datetime import timedelta

from airflow.sdk import AsyncCallback, DAG, DeadlineAlert, DeadlineReference
from airflow.providers.slack.notifications.slack_webhook import SlackWebhookNotifier

with DAG(
    dag_id="orders_load",
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_QUEUED_AT,
        interval=timedelta(minutes=45),
        callback=AsyncCallback(
            SlackWebhookNotifier,
            kwargs={"text": "orders_load has not finished 45 minutes after queuing."},
        ),
    ),
):
    ...</code></pre>



<p class="wp-block-paragraph">Note the reference point. Measuring from when the run was queued, rather than from its logical date, means scheduler backlog counts against the deadline. That&#8217;s usually what you want, because a run that sat in a queue for an hour is late to its consumers regardless of how fast it executed once it started.</p>



<p class="wp-block-paragraph">There&#8217;s a structural weakness here worth naming: any alert that lives inside the orchestrator dies with the orchestrator. If the scheduler is down, nothing evaluates your deadline and nothing notifies anyone. The freshness probe from earlier is the answer, and it needs to run somewhere else. A small VPS from a provider like Contabo or InterServer running Prometheus and Alertmanager, or a hosted option like Grafana Cloud, gives you a watcher outside the blast radius of the thing being watched. This is the single highest-value piece of monitoring most data teams are missing.</p>



<h2 class="wp-block-heading">Alerting without burning your on-call</h2>



<p class="wp-block-paragraph">The fastest way to make all of this worthless is to alert on every threshold crossing. Three rules keep it survivable.</p>



<ol class="wp-block-list">
<li><strong>Set thresholds from observed behaviour, not from wishes.</strong> Look at a month of actual freshness values and set <code>warn_after</code> a comfortable margin above the normal worst case. A source that habitually passes at eleven hours against a twelve-hour threshold is not healthy, it&#8217;s one upstream hiccup from paging you.</li>



<li><strong>Page on consumer impact, not internal events.</strong> A task retry is not an incident. A dataset breaching the freshness commitment its consumers rely on is. Route everything else to a channel someone reads in the morning.</li>



<li><strong>Use burn rate over multiple windows for SLOs.</strong> A short window catches fast breakage, a long window catches slow erosion, and requiring both to fire filters out the transient spikes that generate most false pages.</li>
</ol>



<p class="wp-block-paragraph">One more: alert on the absence of your own telemetry. A freshness gauge that stops updating is indistinguishable from one that&#8217;s fine, right up until someone asks about the numbers.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>Freshness alert fires but the data looks current.</strong> Almost always a timezone problem. Your <code>loaded_at</code> column is in local time, <code>now()</code> is in UTC, and the offset shows up as a constant bias in the age. Store load timestamps in UTC and cast explicitly at read time.</p>



<p class="wp-block-paragraph"><strong>Freshness flaps in and out of breach.</strong> Your check runs too close to the expected arrival. If loads land around six and your probe runs at five past, a fifteen-minute upstream delay produces an intermittent failure that trains everyone to ignore the alert. Move the probe later or widen the threshold.</p>



<p class="wp-block-paragraph"><strong>Every pipeline reports as one series.</strong> <code>honor_labels: true</code> is missing from the Pushgateway scrape config.</p>



<p class="wp-block-paragraph"><strong>A decommissioned pipeline still shows healthy.</strong> Stale group in Pushgateway. Delete the group and add a check on <code>push_time_seconds</code> so the next one surfaces on its own.</p>



<p class="wp-block-paragraph"><strong>Latency looks fine but consumers say data is late.</strong> You&#8217;re measuring run duration and they&#8217;re feeling end-to-end. Add queue wait and event-time-to-load and the gap will be obvious.</p>



<p class="wp-block-paragraph"><strong>Freshness passes, row count is near zero.</strong> The load timestamp advanced without meaningful data. This is a broken source or a stuck watermark, and it&#8217;s the reason the volume check belongs in the same probe.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating job success as the health signal and discovering staleness through a human.</li>



<li>Running the freshness check inside the same pipeline it&#8217;s meant to police.</li>



<li>Tracking freshness without volume, so a zero-row load reads as healthy.</li>



<li>Reporting latency without saying which two timestamps it spans.</li>



<li>Putting run IDs, batch IDs or timestamps into Prometheus labels, which multiplies your series count without bound. Those belong in traces or logs.</li>



<li>Alerting on task-level failures instead of run-level outcomes, then muting the whole channel a week later.</li>



<li>Assuming Pushgateway reflects current state rather than the last thing anyone pushed.</li>
</ul>



<h2 class="wp-block-heading">Best practices for data pipeline freshness monitoring</h2>



<ul class="wp-block-list">
<li>Start with your three most-used tables. Full coverage is a project; three tables is an afternoon and catches most of the pain.</li>



<li>Write down the freshness commitment per dataset in plain language, then encode it. &#8220;Yesterday&#8217;s orders are complete by 07:00&#8221; converts directly into a threshold.</li>



<li>Name metrics consistently across pipelines. A shared prefix and a stable label set is what makes one dashboard work for all of them.</li>



<li>Keep labels low-cardinality: pipeline, dataset, stage, environment, result. Nothing unbounded.</li>



<li>Emit a failure category alongside every failure so alerts route themselves.</li>



<li>Run the deadman check on infrastructure that doesn&#8217;t share a failure domain with the pipeline.</li>



<li>Link every alert to a runbook that names the owner and the first three things to check.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the difference between pipeline latency and data freshness?</h3>



<p class="wp-block-paragraph">Latency measures how long a specific batch of data took to travel from source to destination. Freshness measures how old the newest available record is right now, regardless of whether anything is currently running. A pipeline can have excellent latency and terrible freshness if it stopped being triggered.</p>



<h3 class="wp-block-heading">How often should freshness checks run?</h3>



<p class="wp-block-paragraph">Frequently enough that you find out before your consumers do. A useful rule is roughly a quarter of your tolerance window: if data may be up to four hours old, check hourly. Freshness probes are cheap single-aggregate queries, so the limiting factor is usually warehouse billing rather than load.</p>



<h3 class="wp-block-heading">Do I need a data observability platform for this?</h3>



<p class="wp-block-paragraph">Not to start. Freshness, latency and failure rate for a handful of critical datasets is a scheduled query, a push, and a few alert rules. Commercial platforms earn their cost when you need automatic column-level lineage, anomaly detection across hundreds of tables, or coverage of assets nobody has explicitly instrumented. That&#8217;s a real problem at scale, and a genuinely expensive one to build yourself. It&#8217;s just not the problem you have on day one.</p>



<h3 class="wp-block-heading">How do I monitor freshness for a table that only updates weekly?</h3>



<p class="wp-block-paragraph">Set the threshold from the schedule plus a delivery margin, and add a deadman rule so a missing metric alerts on its own. For genuinely static reference tables, disable freshness explicitly rather than leaving a check that always warns; in dbt that means setting freshness to null for the table.</p>



<h3 class="wp-block-heading">Should freshness checks fail the pipeline or just warn?</h3>



<p class="wp-block-paragraph">Both, in different places. Checking source freshness at the start of a job and failing hard prevents you building models on stale input, which is the cheapest bug to prevent and the most expensive to unwind. Checking output freshness after the fact should alert rather than fail, because the run is already over.</p>



<h3 class="wp-block-heading">What percentile should I use for a latency SLO?</h3>



<p class="wp-block-paragraph">p95 for most internal analytics pipelines, p99 where downstream systems make automated decisions on the data. Track the median separately for capacity planning. Never use the average as the SLO number, because it hides exactly the tail that produces complaints.</p>



<h3 class="wp-block-heading">Does this work for streaming pipelines too?</h3>



<p class="wp-block-paragraph">Yes, with different plumbing. Freshness becomes consumer lag measured in time, latency becomes event-time to availability-time, and failure rate becomes connector and task state plus dead-letter volume. The reasoning is identical; only the source of the numbers changes.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you remember one thing, make it this: measure the data, not the job. Run status, task counts and duration all describe your code. Only freshness describes what your consumers actually receive, and it&#8217;s the one signal that stays honest when the pipeline succeeds at doing nothing.</p>



<p class="wp-block-paragraph">Good data pipeline freshness monitoring is not a platform purchase. It&#8217;s a scheduled query that reports the age and volume of your most important tables, pushed somewhere durable, with a deadman rule so that silence is treated as a failure rather than as health. Add latency broken down by stage and failure rate categorised by reason, and you can answer &#8220;is the data good right now&#8221; without opening a single log file.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help instrumenting your pipelines?</h2>



<p class="wp-block-paragraph">Most of my consulting work in this area is retrofitting observability onto pipelines that already exist and can&#8217;t be paused. Specifically:</p>



<ul class="wp-block-list">
<li>Defining freshness and latency commitments per dataset, then translating them into thresholds and alert rules that hold up on-call.</li>



<li>Adding freshness and volume probes to existing warehouses without touching the pipelines themselves.</li>



<li>Wiring batch jobs into Prometheus through Pushgateway, including the grouping-key and stale-metric problems that bite six months later.</li>



<li>Migrating Airflow DAGs off the removed SLA feature onto Deadline Alerts, or onto external checks that survive a scheduler outage.</li>



<li>Building the Grafana dashboard that answers &#8220;is the data good right now&#8221; in one screen, with drill-down by stage and failure category.</li>



<li>Cutting alert noise on pipelines where the channel has already been muted, by moving from task-level events to consumer-impact SLOs.</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me a DAG file, a scrape config, or a screenshot of the dashboard you don&#8217;t trust, and I&#8217;ll tell you what I&#8217;d instrument first and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/data-pipeline-freshness-monitoring/">Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Zoho Analytics Import vs Live Connect: The Trade-Offs Nobody Mentions</title>
		<link>https://john-nessime.com/blog/technical-guides/zoho-analytics-import-vs-live-connect/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 30 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Caching]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Live Connect]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Query Timeout]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho Analytics]]></category>
		<category><![CDATA[Zoho Databridge]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=346</guid>

					<description><![CDATA[<p>Import mode fails quietly and Live Connect fails loudly. A practical breakdown of what each Zoho Analytics connection mode actually does, where each one wins, the four levers that should drive the choice, and a decision procedure you can run before you build the workspace.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zoho-analytics-import-vs-live-connect/">Zoho Analytics Import vs Live Connect: The Trade-Offs Nobody Mentions</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on a Monday. Someone in finance opens the revenue dashboard, compares it against the operational system, and asks why one says 41 and the other says 47. Nothing is broken. No sync failed. No alert fired. The dashboard is just showing you a snapshot from six hours ago, and absolutely nothing on the screen tells you that.</p>



<p class="wp-block-paragraph">That is the real cost of picking the wrong data connection mode, and it is why the choice between Zoho Analytics import vs Live Connect deserves more than five minutes during setup. Both modes work. Both are documented. But they fail in completely different ways, and one of those failure modes is silent.</p>



<p class="wp-block-paragraph">This post covers what each mode actually does under the hood, where each one genuinely wins, the four levers that should drive the decision, a short procedure for running that decision yourself, and the arguments people repeat that don&#8217;t hold up once you have a workspace in production.</p>



<h2 class="wp-block-heading">What the two modes actually do</h2>



<p class="wp-block-paragraph">Strip away the marketing language and the difference is where the data lives when someone opens a report.</p>



<p class="wp-block-paragraph"><strong>Data Import</strong> copies rows out of your source and stores them inside Zoho Analytics. Reports query that stored copy. You schedule refreshes to pull new data on an interval, and between refreshes your dashboard is a photograph, not a window.</p>



<p class="wp-block-paragraph"><strong>Live Connect</strong> does not copy anything. Zoho pulls metadata about your tables and columns, then generates SQL against your source database every single time a report or dashboard is opened. Your database does the work. Zoho does the rendering.</p>



<p class="wp-block-paragraph">That one architectural difference cascades into everything else: what you get billed for, which features you can use, who owns schema changes, and what happens when a query gets slow.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Import mode: where it wins and where it hurts</h2>



<h3 class="wp-block-heading">Where import mode wins</h3>



<p class="wp-block-paragraph">Import mode is the default for a reason, and the reasons are good ones.</p>



<ul class="wp-block-list">
<li><strong>Report speed is yours to control.</strong> Because the data sits in Zoho&#8217;s storage, load time doesn&#8217;t depend on how busy your production database is at 9am on a Monday.</li>



<li><strong>You get the full feature surface.</strong> Query Tables, custom import queries, data type changes on imported columns, and blending several unrelated sources into one workspace all work in import mode.</li>



<li><strong>Your production database is insulated.</strong> An analyst who builds an accidental many-to-many join hits a copy, not the system taking customer orders.</li>



<li><strong>Incremental fetch keeps sync cheap.</strong> Rather than re-pulling a whole table on every schedule, you nominate a column that identifies new rows and pull only what changed.</li>



<li><strong>It works on every plan.</strong> There is no tier gate on importing data, which matters if you are prototyping before anyone has approved a budget.</li>
</ul>



<h3 class="wp-block-heading">Where import mode hurts</h3>



<p class="wp-block-paragraph">The first problem is the one from the opening: <strong>staleness is invisible</strong>. A Zoho Analytics dashboard built on imported data looks exactly the same whether it refreshed two minutes ago or two days ago. There is no built-in &#8220;as of&#8221; banner on the chart. If your refresh schedule is daily and someone is making a call at 4pm, they are working from this morning&#8217;s world.</p>



<p class="wp-block-paragraph">The second is <strong>billing shape</strong>. Zoho Analytics prices on stored rows, counted as the sum of all rows across every table in every workspace in the account. Import mode is therefore the mode that grows your bill. Every table you sync &#8220;just in case&#8221; is a line item. Rates change, so check the current pricing page rather than trusting any figure you read in a blog post, but understand the mechanism: rows stored equals money.</p>



<p class="wp-block-paragraph">Third, <strong>the copy drifts and nobody owns it</strong>. Six months in, a table has a scheduled import that has been silently failing, or someone changed a transformation in the source and the imported copy still reflects the old logic. Import mode creates a second version of the truth, and second versions need maintenance.</p>



<p class="wp-block-paragraph">Fourth, <strong>refresh frequency is plan-gated</strong>. How often you can schedule a sync depends on your subscription tier. If your reporting requirement is &#8220;within the hour&#8221; and your plan permits fewer refreshes per day than that implies, import mode cannot meet the requirement no matter how you configure it.</p>



<h2 class="wp-block-heading">Live Connect: where it wins and where it hurts</h2>



<h3 class="wp-block-heading">Where Live Connect wins</h3>



<ul class="wp-block-list">
<li><strong>There is exactly one copy of the data.</strong> No reconciliation conversation, because there is nothing to reconcile. What the report shows is what the database holds at that moment.</li>



<li><strong>Sensitive data stays where it already is.</strong> For teams with a data residency position, or a security team that objects to bulk-copying customer records into a BI vendor&#8217;s storage, this is often the deciding argument on its own.</li>



<li><strong>Foreign keys carry over.</strong> Tables linked by foreign keys in the source get lookup relationships created automatically in Zoho, which is a genuine time saver compared to wiring lookup columns by hand.</li>



<li><strong>Row storage isn&#8217;t the cost driver.</strong> You&#8217;re not paying to warehouse a duplicate of a table you already pay to host on Amazon RDS, Snowflake, or a Postgres instance on your own VPS.</li>
</ul>



<h3 class="wp-block-heading">Where Live Connect hurts</h3>



<p class="wp-block-paragraph">Here is the one that catches people, and it is worth reading twice: <strong>Live Connect does not automatically track schema changes</strong>. Add a column, rename a column, drop a table, change a data type in the source, and Zoho will not follow along. Reports built on the affected objects land in a Mismatch state and you have to reconcile them manually through the Sync Design option in the data source settings.</p>



<p class="wp-block-paragraph">This is the inverse of what most people assume. &#8220;Live&#8221; sounds like &#8220;always current in every respect.&#8221; It means live <em>data</em>, not live <em>structure</em>. A routine migration that renames <code>customer_id</code> to <code>account_id</code> will break dashboards that nobody looked at that week, and you&#8217;ll find out from a user rather than from a monitor.</p>



<p class="wp-block-paragraph">The rest of the trade-offs:</p>



<ul class="wp-block-list">
<li><strong>Query timeout.</strong> Zoho documents a default query timeout of 2.5 minutes on Live Connect. Anything slower fails. On a source that hasn&#8217;t been tuned for analytical queries, that ceiling arrives faster than you expect.</li>



<li><strong>No Custom Query at import time.</strong> You can&#8217;t hand Zoho a filtered SELECT to narrow what it sees. The documented workaround is to create a view in the source database and connect to that instead.</li>



<li><strong>No Query Tables.</strong> The SQL-over-your-workspace feature depends on data being stored locally. Zoho notes Query Table support for selected databases is enabled on request, so it is worth asking, but do not plan around it.</li>



<li><strong>No data type changes in Zoho.</strong> If a column is text in the source and should be a date, you fix it in the source or in a view. There is no in-Zoho override.</li>



<li><strong>Lookups only within the connection.</strong> You can define lookup rules between tables in the same live connection, not across to other data sources. Reports from different sources can still be combined on a dashboard, but that is a presentation-layer join, not a data-layer one.</li>



<li><strong>It is gated to higher plans.</strong> Zoho&#8217;s documentation places Live Connect on the Premium and Enterprise tiers. If you&#8217;re on a lower plan, the decision has already been made for you.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The four levers behind Zoho Analytics import vs Live Connect</h2>



<p class="wp-block-paragraph">Feature checklists make this look like a close call. It usually isn&#8217;t, once you work out which of these four you actually care about.</p>



<p class="wp-block-paragraph"><strong>1. Your freshness contract.</strong> Not &#8220;how fresh would be nice.&#8221; What decision gets made from this dashboard, and how wrong can the data be before that decision goes bad? A weekly pipeline review tolerates a nightly sync comfortably. A dispatcher deciding which order to fulfil next does not. Write the tolerance down as a number of minutes before you look at any feature table.</p>



<p class="wp-block-paragraph"><strong>2. Where the cost lands.</strong> Import mode moves cost onto your Zoho row count. Live Connect moves it onto your database, in query load and in whatever you pay for the compute that serves it. Neither is free. A warehouse that bills per query scanned is a different economic proposition under Live Connect than a fixed-price MySQL instance on Contabo or InterServer, where extra query load costs nothing until you outgrow the box.</p>



<p class="wp-block-paragraph"><strong>3. Where transformation happens.</strong> If your model needs joins, unions, derived columns, and cleanup, import mode gives you Query Tables to do that inside Zoho. Live Connect pushes all of it down into the source, which means you need database write access and someone comfortable maintaining views. That&#8217;s a team-capability question as much as a technical one.</p>



<p class="wp-block-paragraph"><strong>4. Who owns schema change.</strong> Under Live Connect, every source migration is potentially a reporting incident. That is fine if the same team owns both, and painful if the application team ships migrations without telling anyone. Import mode is more forgiving here, since Zoho can automatically pick up column additions and deletions on synced tables.</p>



<h2 class="wp-block-heading">A decision procedure you can run in ten minutes</h2>



<ol class="wp-block-list">
<li><strong>Check the plan gate first.</strong> If you&#8217;re not on a tier that includes Live Connect, stop. Import mode, tuned well, is the answer.</li>



<li><strong>Write down the freshness tolerance in minutes.</strong> If the number is larger than your plan&#8217;s shortest available sync interval, import mode satisfies it. If it&#8217;s smaller, Live Connect is the only mode that can meet it.</li>



<li><strong>Time the heaviest query.</strong> Take the ugliest aggregation your dashboard will need and run it against the source with realistic filters. If it takes anywhere near 2.5 minutes, Live Connect will fail under real usage. Tune the source or choose import.</li>



<li><strong>List the transformations you need.</strong> If any of them require a Query Table, joining across unrelated sources, or changing a column&#8217;s data type in Zoho, import mode is doing that work.</li>



<li><strong>Ask who ships schema migrations.</strong> If it isn&#8217;t the people who own the dashboards, budget for Sync Design work every release, or use import mode.</li>



<li><strong>Check the security position last, because it can override everything.</strong> If the answer to &#8220;may customer records be stored in a third-party BI platform&#8221; is no, Live Connect is the only option and you build around its constraints.</li>
</ol>



<p class="wp-block-paragraph">In practice, mixed estates are common and reasonable. Live Connect for the operational workspace where freshness is the point, import mode for the historical and blended reporting where transformation and speed matter more.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<p class="wp-block-paragraph"><em>&#8220;Live Connect is real-time, so it&#8217;s more accurate.&#8221;</em> It&#8217;s more current. Accuracy is a modelling problem, not a transport problem. A live connection to a table with duplicate lookup keys produces wrong numbers instantly instead of wrong numbers on a schedule.</p>



<p class="wp-block-paragraph"><em>&#8220;Live Connect is cheaper because you don&#8217;t pay for rows.&#8221;</em> Sometimes. You&#8217;ve moved the cost, not removed it. Every dashboard open is a set of queries against your source. On per-query or per-compute-second billing, an enthusiastic team with auto-refreshing dashboards can generate a bill that makes row storage look like a bargain.</p>



<p class="wp-block-paragraph"><em>&#8220;Just enable the cache and Live Connect gets fast.&#8221;</em> Zoho does offer report caching on Live Connect workspaces, with a refresh interval you set in minutes. It genuinely helps. But note what you&#8217;ve just built: a copy of the data with a staleness window, which is the thing you chose Live Connect to avoid. Caching is a legitimate tuning tool, not a free upgrade.</p>



<p class="wp-block-paragraph"><em>&#8220;We&#8217;ll switch modes later if it doesn&#8217;t work out.&#8221;</em> Switching is not a toggle. Zoho does not permit importing data into an existing Live Connect workspace, and the two modes support different feature sets, so a migration means rebuilding the workspace and its reports. Choose deliberately the first time.</p>



<h2 class="wp-block-heading">If you go Live Connect, tune the source first</h2>



<p class="wp-block-paragraph">Live Connect performance is your database&#8217;s performance. Zoho&#8217;s own guidance is blunt about this, and it matches what you&#8217;d do for any reporting workload.</p>



<p class="wp-block-paragraph">Index the columns that lookups and joins run through. Without them, every dashboard open is a table scan:</p>



<pre class="wp-block-code"><code>-- Index the join key that Zoho will generate queries against
CREATE INDEX idx_orders_customer_id ON orders (customer_id);

-- Confirm the planner actually uses it (PostgreSQL)
EXPLAIN ANALYZE
SELECT customer_id, SUM(total)
FROM orders
WHERE order_date &gt;= CURRENT_DATE - INTERVAL '90 days'
GROUP BY customer_id;</code></pre>



<p class="wp-block-paragraph">Because Custom Query isn&#8217;t available in Live Connect, a view is how you narrow and shape what Zoho sees. Connect to the view rather than the raw table:</p>



<pre class="wp-block-code"><code>CREATE VIEW reporting_orders AS
SELECT
    o.id,
    o.customer_id,
    o.order_date,
    o.total,
    CASE WHEN o.status = 'complete' THEN 1 ELSE 0 END AS is_complete
FROM orders o
WHERE o.order_date &gt;= CURRENT_DATE - INTERVAL '2 years';</code></pre>



<p class="wp-block-paragraph">For calculations that get repeated across many reports, a materialized view precomputes the work so the 2.5 minute ceiling stops being a threat. You trade a refresh schedule for query speed, which is a much smaller staleness window than a full import.</p>



<p class="wp-block-paragraph">The rest of the list is short and worth following:</p>



<ul class="wp-block-list">
<li>Keep formula columns simple, and prefer CASE WHEN over deeply nested IF statements.</li>



<li>Make sure lookup columns hold unique values. Duplicate keys degrade query performance badly and quietly inflate aggregates.</li>



<li>Avoid many-to-many joins and circular join paths in your model.</li>



<li>Give Zoho a database user with SELECT plus metadata read on exactly the tables it needs, and nothing more. Missing metadata permission is a common cause of a table that connects but never loads.</li>



<li>Decide your connectivity path deliberately: allowlisting Zoho&#8217;s published IP ranges for a cloud database, or Zoho Databridge as an agent when the database sits in a private network. SSL and SSH tunneling are both supported on the connection.</li>



<li>Put a change-notification step in your migration checklist. If a release renames or drops a column, someone should run Sync Design before users find the Mismatch.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I switch a workspace from import mode to Live Connect?</h3>



<p class="wp-block-paragraph">Not as an in-place setting change. Zoho does not permit importing data into an existing Live Connect workspace, and the feature sets differ enough that reports built one way won&#8217;t simply carry over. Treat a mode change as a rebuild and plan the reporting downtime accordingly.</p>



<h3 class="wp-block-heading">Does Live Connect count against my row limit?</h3>



<p class="wp-block-paragraph">Zoho&#8217;s row-based pricing counts rows stored across your workspace tables, and Live Connect doesn&#8217;t store your source data in Zoho. The practical implication is that Live Connect shifts your constraint from storage to source query capacity. Confirm the exact treatment for your account with Zoho before you build a cost model on it, since add-ons and plan definitions change.</p>



<h3 class="wp-block-heading">Why do my Live Connect reports suddenly show a view-cannot-be-accessed error?</h3>



<p class="wp-block-paragraph">Almost always because a table or column the report depends on was renamed or dropped in the source. Live Connect doesn&#8217;t track structural changes automatically. Open the data source settings, check the Mismatch tab, and use Sync Design to remap. Insufficient SELECT or metadata permissions on the source produce a similar symptom.</p>



<h3 class="wp-block-heading">What is incremental fetch, and does it apply to both modes?</h3>



<p class="wp-block-paragraph">Incremental fetch is an import-mode feature. Instead of re-pulling every row on each schedule, you nominate a column that identifies new records, such as an auto-incrementing ID or a timestamp, and choose whether new rows are appended or matched and updated. It cuts sync time dramatically on large transactional tables. It has no meaning under Live Connect, where nothing is fetched on a schedule in the first place.</p>



<h3 class="wp-block-heading">Can I combine Live Connect tables with data from other sources?</h3>



<p class="wp-block-paragraph">Not at the data layer. Lookup rules only work between tables within the same live connection. You can place reports from a Live Connect workspace and reports from other sources side by side on a dashboard, but you cannot join them into a single dataset the way you can with imported tables in one workspace.</p>



<h3 class="wp-block-heading">Is Live Connect available for on-premise and private databases?</h3>



<p class="wp-block-paragraph">Yes, through Zoho Databridge, the agent that bridges a database in a private network to Zoho&#8217;s servers. You choose the connection type during setup, so the same agent supports either mode. If your database is publicly reachable instead, you allowlist Zoho&#8217;s published IP ranges and connect directly.</p>



<h3 class="wp-block-heading">Which mode is better for a Zoho CRM or Zoho Desk workspace?</h3>



<p class="wp-block-paragraph">Neither, in the sense that the question doesn&#8217;t apply. Zoho&#8217;s own application connectors sync data into Analytics on a schedule, which behaves like import mode. Live Connect is for external databases and data warehouses. If you need faster refreshes on Zoho app data, the lever is the connector&#8217;s sync schedule and your plan tier, not the connection mode.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The Zoho Analytics import vs Live Connect decision is not a performance question. It&#8217;s a question about which failure you&#8217;d rather have.</p>



<p class="wp-block-paragraph">Import mode fails quietly. Numbers stay confidently wrong until someone reconciles them by hand, and the bill grows with every table you sync out of habit. Live Connect fails loudly. Someone renames a column and a dashboard breaks in front of a user, or a heavy query hits the timeout during the Monday review.</p>



<p class="wp-block-paragraph">Loud failures are usually easier to live with, because you find out about them. But they only work if you have the source database, the access to tune it, and a change process that includes reporting. If you don&#8217;t have those three, import mode with an honest refresh schedule and a documented staleness window will serve you far better than a live connection you can&#8217;t maintain.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your Zoho Analytics setup?</h2>



<p class="wp-block-paragraph">Most of the Zoho Analytics work I get called into is somebody discovering the mode choice was made in five minutes eighteen months ago. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing an existing workspace and telling you honestly whether the connection mode still fits what you&#8217;re using it for</li>



<li>Profiling and tuning the source database behind a Live Connect workspace so dashboards stop hitting the query timeout</li>



<li>Designing reporting views and materialized views that push transformation into the database where Live Connect can use it</li>



<li>Setting up Zoho Databridge for databases in private networks, including SSL and SSH tunneling and the firewall rules around it</li>



<li>Configuring incremental fetch on import-mode tables to cut sync times and stop row counts drifting upward</li>



<li>Building a schema-change process so source migrations don&#8217;t turn into reporting incidents</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me the slow query, the Mismatch screenshot, or the EXPLAIN output from the aggregation that&#8217;s timing out. That&#8217;s usually enough to tell where the actual problem is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zoho-analytics-import-vs-live-connect/">Zoho Analytics Import vs Live Connect: The Trade-Offs Nobody Mentions</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</title>
		<link>https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Customer 360]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Identity Resolution]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Reverse ETL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Stripe]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Zendesk]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=350</guid>

					<description><![CDATA[<p>Three CRM records, one person, and nothing in the logs to explain it. A practical walkthrough of what actually breaks when you unify Shopify, Stripe, CRM and support data: guest-checkout identity, duplicate and out-of-order webhooks, revenue that never ties to payouts, deletion requests that miss half your copies, and reverse ETL write loops. Organized by failure family, with schema and handler patterns you can apply directly.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The project usually starts with a screenshot. Someone in support pastes three CRM records into a channel, all carrying the same person&#8217;s name, and asks which one is real. One has the order history. One has the subscription. The third has four tickets attached and no orders at all.</p>



<p class="wp-block-paragraph">Nobody broke anything. Shopify, Stripe, the CRM and the helpdesk each created a customer record using whatever identifier they had at the moment they needed one, and none of those identifiers agreed. The systems are all correct in isolation. The person is the thing that got lost.</p>



<p class="wp-block-paragraph">This post covers what actually breaks when you unify Shopify, Stripe, CRM and support data, and how to build the join so it survives guest checkouts, duplicate webhooks, refunds, currency conversion and deletion requests. It&#8217;s organized by failure family rather than by tool, because the tools get acquired and renamed every couple of years and the failures never change.</p>



<h2 class="wp-block-heading">Failure family one: the join key is not the email address</h2>



<p class="wp-block-paragraph">Every one of these systems will happily give you an email address, which is exactly why so many pipelines join on it and quietly produce garbage.</p>



<p class="wp-block-paragraph">Start with where identity actually lives in each system:</p>



<ul class="wp-block-list">
<li><strong>Shopify</strong> has a customer ID, but an order does not have to be attached to one. Guest checkout produces an order with an email on it and no durable customer record behind it.</li>

<li><strong>Stripe</strong> has a customer ID, but only if a Customer object was created. Checkout Sessions that don&#8217;t create one are associated with guest customers instead, which Stripe groups in the Dashboard based on the same card, email or phone. That grouping is read-only and it is not an ID you can join on.</li>

<li><strong>The CRM</strong> keys on email in almost every default configuration, and cheerfully creates a second contact when the email differs by one character.</li>

<li><strong>The helpdesk</strong> keys on the requester&#8217;s email, which is whatever the person typed into a contact form or sent the ticket from. It is very often not the address on the order.</li>
</ul>



<p class="wp-block-paragraph">Here&#8217;s the part that stays invisible until someone finally checks: nothing errors. Every dashboard still returns a number. Lifetime value gets silently split across three profiles, so your LTV looks lower and your repeat-purchase rate looks worse than it is. Support metrics look <em>better</em> than they are, because a customer coming back with the same unresolved problem shows up as a brand new contact. You can run like that for a year without a single failed job.</p>



<h3 class="wp-block-heading">Build an identifier table, not a fuzzy match</h3>



<p class="wp-block-paragraph">The fix is boring and it works: a surrogate person key, and a separate table of every identifier ever observed for that person, with the source that reported it.</p>



<pre class="wp-block-code"><code>-- One row per real human, generated by you, owned by you.
CREATE TABLE person (
  person_key      uuid PRIMARY KEY,
  created_at      timestamptz NOT NULL DEFAULT now(),
  merged_into     uuid NULL REFERENCES person(person_key)
);

-- One row per identifier per source. This is the whole trick.
CREATE TABLE person_identifier (
  person_key      uuid NOT NULL REFERENCES person(person_key),
  source          text NOT NULL,   -- shopify | stripe | crm | helpdesk
  id_type         text NOT NULL,   -- email | phone | shopify_customer_id | stripe_customer_id
  id_value        text NOT NULL,   -- already normalized
  first_seen_at   timestamptz NOT NULL,
  last_seen_at    timestamptz NOT NULL,
  PRIMARY KEY (source, id_type, id_value)
);</code></pre>



<p class="wp-block-paragraph">Normalization happens before the row is written, not in the join. Lowercase and trim emails, store phone numbers in E.164, strip whitespace from external IDs. Resist the urge to get clever with Gmail&#8217;s dot and plus handling. Stripping <code>+tags</code> is a business policy decision with real consequences, not a technical detail, and the same rule applied to a domain that treats plus addresses as distinct mailboxes will merge two genuinely different people.</p>



<p class="wp-block-paragraph">Deterministic matching first, always. Exact matches on shared identifiers, transitively closed. Probabilistic matching on name plus address plus device is worth reaching for only when you have exhausted the deterministic keys, and it has an asymmetric failure mode: a household sharing one card gets collapsed into one person, and unmerging afterwards is expensive because every downstream system has already copied the merged key into its own records. Merges are cheap to make and painful to undo. Bias toward leaving two records separate.</p>



<p class="wp-block-paragraph">One thing worth doing early on the Stripe side: when you create a Customer, write your own identifier into <code>metadata</code>. Stripe allows fifty key-value pairs per object, with keys up to forty characters and values up to five hundred, and it only returns metadata for requests made with a secret key. That&#8217;s enough room for a person key and an order reference, which turns a future fuzzy join into an exact one. Never put anything sensitive there.</p>



<h2 class="wp-block-heading">Failure family two: webhooks arrive twice, out of order, or not at all</h2>



<p class="wp-block-paragraph">Both platforms are explicit about this, and both get treated as if they weren&#8217;t.</p>



<p class="wp-block-paragraph">Stripe delivers at least once, does not guarantee ordering, and retries failed deliveries with exponential backoff for up to three days. The event <code>id</code> (the one starting <code>evt_</code>) stays identical across every retry, which makes it your deduplication key. Shopify sends <code>X-Shopify-Webhook-Id</code> for the same purpose, signs the raw body into <code>X-Shopify-Hmac-Sha256</code>, and states plainly that ordering isn&#8217;t guaranteed within a topic or across topics for the same resource. Shopify&#8217;s own documentation goes further and tells you not to rely on webhooks as your only source of data.</p>



<p class="wp-block-paragraph">For a customer-data pipeline specifically, out-of-order delivery is nastier than duplication. Consider a <code>customers/update</code> that changes someone&#8217;s email arriving before the <code>orders/create</code> that was placed under the old address. Process those in arrival order and you attach the order to a person who doesn&#8217;t exist yet, or worse, to the wrong one. Order your writes by the event&#8217;s own timestamp, using <code>X-Shopify-Triggered-At</code> or the <code>updated_at</code> in the payload, and make profile updates last-writer-wins on that timestamp rather than on insertion time.</p>



<h3 class="wp-block-heading">Do almost nothing in the handler</h3>



<p class="wp-block-paragraph">Verify the signature, claim the event ID, write the raw body to durable storage, return 2xx. That&#8217;s it. Identity resolution, CRM lookups and enrichment calls all belong in a worker reading from that store. A handler that tries to resolve identity inline will eventually be slow enough to time out, which triggers a retry, which arrives while the first attempt is still running.</p>



<p class="wp-block-paragraph">Make the claim atomic rather than check-then-act, because two retries genuinely can land concurrently:</p>



<pre class="wp-block-code"><code>-- Atomic claim. If this inserts zero rows, someone already has it.
INSERT INTO webhook_event (source, event_id, topic, received_at, payload)
VALUES ($1, $2, $3, now(), $4)
ON CONFLICT (source, event_id) DO NOTHING
RETURNING event_id;</code></pre>



<p class="wp-block-paragraph">Return a 2xx for a duplicate. A duplicate is a success from the sender&#8217;s point of view, and returning an error just schedules more of them.</p>



<p class="wp-block-paragraph">Signature verification has one recurring trap on the Shopify side: it runs against the <em>raw</em> body. Any JSON body parser that runs first will change the bytes and every signature will fail. Capture the raw body before parsing, and compare in constant time.</p>



<pre class="wp-block-code"><code>import base64, hashlib, hmac

def verify_shopify(raw_body: bytes, header_value: str, client_secret: str) -&gt; bool:
    digest = hmac.new(
        client_secret.encode("utf-8"),
        raw_body,
        hashlib.sha256,
    ).digest()
    expected = base64.b64encode(digest).decode("utf-8")
    return hmac.compare_digest(expected, header_value)</code></pre>



<p class="wp-block-paragraph">Worth knowing before you spend an afternoon on it: after rotating an app&#8217;s client secret, Shopify notes it can take up to an hour before signatures are generated with the new one. If verification starts failing right after a rotation, that&#8217;s usually why.</p>



<p class="wp-block-paragraph">And because delivery isn&#8217;t guaranteed, webhooks alone are never enough. Run a reconciliation job that pulls orders and customers modified since the last watermark and fills the gaps. This is the job that saves you during an outage on your side, and it&#8217;s the one people skip. If you&#8217;d rather not build the buffering and replay layer, managed webhook infrastructure like Hookdeck exists for exactly this, and self-hosting a small queue and worker on a modest VPS from somewhere like Contabo or InterServer is perfectly reasonable too. The point is that something has to catch what the webhook missed.</p>



<h2 class="wp-block-heading">Failure family three: &#8220;revenue&#8221; means four different things</h2>



<p class="wp-block-paragraph">This is the one that turns into a meeting. Finance says the number is wrong. Analytics says the number is right. Both are correct, because they&#8217;re measuring different things and nobody wrote down which.</p>



<ul class="wp-block-list">
<li>Shopify reports gross sales at the moment the order is placed, in the store&#8217;s currency and the store&#8217;s timezone.</li>

<li>Stripe pays out in batches, net of processing fees, refunds and disputes, on its own schedule, in the settlement currency of the account.</li>

<li>A refund issued in Shopify and a refund issued in Stripe are two different events that may or may not be the same underlying money movement.</li>

<li>A dispute lands days or weeks after the order and reverses revenue in a period that has already been reported.</li>
</ul>



<p class="wp-block-paragraph">There&#8217;s a structural trap here too. If the store runs Shopify Payments, there is no Stripe account for you to query. Shopify Payments is built on Stripe&#8217;s processing infrastructure, but it&#8217;s Shopify&#8217;s gateway, and payouts, fees and disputes live in the Shopify admin rather than behind a Stripe API key. Plenty of businesses end up with Shopify Payments handling storefront orders and a separate Stripe account handling subscriptions or invoicing, which means two payment systems, two payout schedules and one customer who has no idea any of this exists.</p>



<p class="wp-block-paragraph">Do not build a single <code>revenue</code> column. Build separate fact tables at their natural grain (orders, order line items, payments, refunds, disputes, payouts) plus a bridging table that links a payout back to the transactions inside it. Then let each team define its own metric on top. Finance gets net settled cash. Marketing gets gross booked revenue. Neither has to be wrong for the other to be right.</p>



<p class="wp-block-paragraph">Store every timestamp in UTC and keep the original timezone as a separate column. Store every amount in minor units as an integer alongside its currency code, and store the FX rate that was actually applied rather than recalculating it later from a rates table. Recalculated FX is the single most common source of &#8220;the number moved and nobody changed anything.&#8221;</p>



<p class="wp-block-paragraph">On ingestion, Stripe Data Pipeline can push Stripe data directly into Snowflake, Redshift or Databricks as a data share, which removes a connector from the picture if you&#8217;re already on one of those. Otherwise a managed ELT tool like Fivetran or a self-hosted Airbyte will handle Shopify, Stripe, the CRM and the helpdesk with off-the-shelf connectors. Both routes are fine. The connector is genuinely the easy part.</p>



<h2 class="wp-block-heading">Failure family four: deletion is the requirement nobody models</h2>



<p class="wp-block-paragraph">If you&#8217;re building this as a Shopify app, three compliance webhooks are mandatory: <code>customers/data_request</code>, <code>customers/redact</code> and <code>shop/redact</code>. You acknowledge with a 200 and complete the action within thirty days. <code>shop/redact</code> fires forty-eight hours after uninstall and covers the shop&#8217;s data. These are enforced at app review, so you&#8217;ll find out about them either way.</p>



<p class="wp-block-paragraph">The interesting problem isn&#8217;t receiving the webhook. It&#8217;s that by the time it arrives, that person&#8217;s email exists in your raw landing zone, your staging tables, your identity table, your customer view, a BI extract, the CRM, the helpdesk, and a Slack message from a debugging session six months ago. A delete that only touches the golden record is not a delete.</p>



<p class="wp-block-paragraph">The design decision that makes this tractable is to hold personal data in exactly one place. Every downstream table carries <code>person_key</code> and nothing else. Facts, aggregates, models, extracts and syncs all reference the key. When a redaction request lands, you redact one row set in the identifier table and the rest of the warehouse is already anonymous. Getting this right at schema-design time costs an afternoon; retrofitting it costs a quarter.</p>



<p class="wp-block-paragraph">Practical pieces that make it real:</p>



<ul class="wp-block-list">
<li>Tag every table and column that can contain an identifier, in your catalog or in dbt metadata, and treat the tag list as the deletion scope.</li>

<li>Partition the raw landing zone so you can rewrite a partition rather than scan an entire bucket looking for one email.</li>

<li>Keep a deletion queue with a status per downstream system, including the CRM and the helpdesk, which have their own APIs and their own idea of what deletion means.</li>

<li>Write down what you retain and why. Financial records generally have to survive a deletion request for tax and dispute reasons. Say so rather than promising a hard delete you can&#8217;t perform.</li>
</ul>



<h2 class="wp-block-heading">Failure family five: pushing the unified view back out</h2>



<p class="wp-block-paragraph">A customer view nobody can act on is a report. The value shows up when a support agent opens a ticket and can see subscription status and lifetime order count without switching tabs.</p>



<p class="wp-block-paragraph">That&#8217;s reverse ETL, and the market is well served: Hightouch, Census (now part of Fivetran), Polytomic, RudderStack, and iPaaS-style tools like Workato if you need branching logic rather than field syncs. The warehouse-native argument is straightforward and mostly correct: your models already live there, and you avoid maintaining a second copy of the customer profile inside a vendor&#8217;s rigid schema. The counter-argument is also real. Reverse ETL is batch by nature, so if you need sub-minute reaction to an event, a packaged CDP or a direct event pipeline will serve you better, and you&#8217;ll be running several vendors plus warehouse compute plus the people to maintain the models.</p>



<p class="wp-block-paragraph">Three things go wrong here more or less every time.</p>



<p class="wp-block-paragraph"><strong>The write loop.</strong> You compute a churn-risk field and sync it into the CRM. Your ELT connector reads the CRM back into the warehouse. Now the computed field is an input to the model that computes it. This is genuinely hard to spot because nothing errors, the value just drifts. Namespace every synced field with a consistent prefix and explicitly exclude that namespace from ingestion.</p>



<p class="wp-block-paragraph"><strong>The resync stampede.</strong> Incremental syncs push diffs and stay small. Change a model definition and the next run wants to update every record, which will meet a helpdesk API rate limit at speed. Cap the batch size, and treat a full resync as a planned operation rather than something a merge to main can trigger.</p>



<p class="wp-block-paragraph"><strong>Syncing logic instead of keys.</strong> Push <code>person_key</code> into a custom field on the CRM contact and the helpdesk user. It makes the key searchable by agents, it gives the next pipeline a deterministic join, and it stops three teams from each rebuilding their own version of your matching rules.</p>



<h2 class="wp-block-heading">A pipeline shape that holds up</h2>



<p class="wp-block-paragraph">Put together, the order of operations matters more than the tool choice:</p>



<ol class="wp-block-list">
<li>Land raw. Every webhook body and every API page goes to immutable storage first, keyed by source and event ID, before anything interprets it.</li>

<li>Deduplicate on the provider&#8217;s event ID, atomically, with a retention window longer than the provider&#8217;s retry window.</li>

<li>Normalize identifiers into the identifier table as rows arrive, with the source recorded.</li>

<li>Resolve identity deterministically into <code>person_key</code>, as a scheduled job, never inside a request handler.</li>

<li>Build fact tables per system at their own grain, keyed on <code>person_key</code>, with no raw PII.</li>

<li>Model the customer view on top, in dbt or whatever your team already uses. The Fivetran <code>dbt_customer360</code> package is a reasonable starting point if you want to see how someone else structured the identity step.</li>

<li>Activate a small, named, namespaced set of fields back into the CRM and helpdesk.</li>

<li>Run reconciliation and freshness checks on a schedule, and alert on them like you would any other production job.</li>
</ol>



<h2 class="wp-block-heading">Troubleshooting: symptom to root cause</h2>



<p class="wp-block-paragraph">These are the ones that come up repeatedly, and the check that usually settles them.</p>



<ul class="wp-block-list">
<li><strong>Warehouse customer count is much higher than Shopify&#8217;s.</strong> Guest orders are minting a new person on every purchase. Check how many <code>person</code> rows have exactly one identifier and one order.</li>

<li><strong>Warehouse count is much lower.</strong> An over-eager merge rule. Look for person keys with an implausible number of distinct emails or shipping addresses attached.</li>

<li><strong>BI revenue doesn&#8217;t match Stripe payouts.</strong> Grain mismatch, not a data loss. Reconcile a single payout against its constituent transactions before touching anything else.</li>

<li><strong>A CRM field keeps flipping between two values.</strong> Write loop. Check whether the field&#8217;s namespace is excluded from your ELT source configuration.</li>

<li><strong>Recent orders missing, older ones fine.</strong> Webhook delivery gap. Confirm the reconciliation job ran, and check its watermark rather than trusting its exit code.</li>

<li><strong>Every Shopify signature suddenly fails.</strong> Either a body parser is running before verification, or the client secret was rotated within the last hour.</li>

<li><strong>Tickets attached to the wrong customer after a merge.</strong> The merge happened in the warehouse but the synced key in the helpdesk still points at the old person. Merges need to propagate outward, not just inward.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Joining on email in the model layer instead of resolving identity once, upstream.</li>

<li>Treating webhooks as the system of record instead of as a low-latency hint, with a reconciliation pull behind them.</li>

<li>Doing identity resolution, enrichment or CRM writes inside the webhook handler.</li>

<li>Ordering profile updates by arrival time rather than by the event&#8217;s own timestamp.</li>

<li>One <code>revenue</code> column that finance and marketing are both told to use.</li>

<li>Storing amounts as floats, or recalculating historical FX from a rates table.</li>

<li>Copying PII into every downstream table because it was convenient at the time.</li>

<li>Reaching for probabilistic matching before the deterministic keys have been exhausted.</li>

<li>Assuming a Shopify store on Shopify Payments has a Stripe account you can query.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Own the surrogate key. Never let a vendor&#8217;s internal ID become the primary identity of a person in your systems.</li>

<li>Write your <code>person_key</code> into Stripe <code>metadata</code> and Shopify customer metafields at creation time, so future joins are exact rather than inferred.</li>

<li>Keep raw payloads immutable and replayable. Every model bug you&#8217;ll ever have is cheaper to fix if you can reprocess from raw.</li>

<li>Version your matching rules and record which version produced each merge, so you can audit and reverse a bad rule.</li>

<li>Alert on freshness and on row-count deltas per source, not just on job success. A connector returning zero rows exits cleanly.</li>

<li>Reconcile one payout end to end by hand before you trust the automated version. It&#8217;s the fastest way to find a grain error.</li>

<li>Document the metric definitions next to the models. Most &#8220;the data is wrong&#8221; tickets are definition disputes.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a CDP to unify Shopify, Stripe, CRM and support data?</h3>



<p class="wp-block-paragraph">Not necessarily. If you already run a warehouse and have someone comfortable with dbt, ELT plus modelling plus reverse ETL covers the same ground with more control over the data model. A packaged CDP earns its cost when you need real-time behavioural segmentation, when nobody on the team can maintain models, or when marketing needs to ship changes without waiting on data engineering.</p>



<h3 class="wp-block-heading">How do I handle guest checkout customers?</h3>



<p class="wp-block-paragraph">Create a person from the order&#8217;s email anyway, and mark the record as identity-weak. When that email later appears attached to a real Shopify customer or Stripe Customer object, the deterministic match links them and the order history follows. The mistake is discarding guest orders because they have no customer ID, which is exactly the revenue you most want attributed.</p>



<h3 class="wp-block-heading">Should I sync everything into the CRM or keep it in the warehouse?</h3>



<p class="wp-block-paragraph">Keep the full picture in the warehouse and sync a deliberately small set of fields outward. The test is whether an agent or a rep will act on the field during a conversation. Lifetime value, subscription status, open ticket count and churn risk pass. A hundred-column profile does not, and it turns every model change into a rate-limit incident.</p>



<h3 class="wp-block-heading">Why don&#8217;t my Shopify sales match my Stripe payouts?</h3>



<p class="wp-block-paragraph">Because they&#8217;re measuring different things at different times. Shopify shows gross sales at order time; Stripe pays out batched amounts net of fees, refunds and disputes, and the timing differs by days. Add currency conversion and a timezone mismatch and the numbers will never tie without an explicit bridging table linking payouts to the transactions inside them.</p>



<h3 class="wp-block-heading">Can I use email as the primary key if my store requires account creation?</h3>



<p class="wp-block-paragraph">You can get away with it longer, but it&#8217;s still a bad primary key. People change email addresses, use one address for orders and another for support, and share addresses within a household. A surrogate key with an identifier table costs very little up front and means an email change is a new row rather than a broken join and a duplicate customer.</p>



<h3 class="wp-block-heading">How do I test the identity resolution before trusting it?</h3>



<p class="wp-block-paragraph">Run it in shadow mode. Compute person keys without activating them, then have support pick twenty customers they know are duplicates and twenty they know are distinct, and check what the resolver did. That exercise finds normalization bugs and over-eager merge rules faster than any aggregate metric, because the people who answer the tickets already know where the duplicates are.</p>



<h3 class="wp-block-heading">What&#8217;s the minimum viable version of this?</h3>



<p class="wp-block-paragraph">Raw landing with deduplication, an identifier table with deterministic matching, one fact table per source, and a single synced field into the helpdesk. That&#8217;s a couple of weeks of work and it removes most of the pain. Everything above it is refinement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion: unify Shopify, Stripe, CRM and support data around a key you own</h2>



<p class="wp-block-paragraph">If you take one thing from this: the hard part of unifying Shopify, Stripe, CRM and support data is not moving the data. Connectors are commodity. The hard part is deciding what a customer <em>is</em>, writing that decision down as a key you own, and then refusing to let any downstream system quietly invent its own version of it.</p>



<p class="wp-block-paragraph">Do the identity work first, land everything raw so you can reprocess when you get it wrong, and keep personal data in one place so deletion stays a single operation. The dashboards will follow. Skip that order and you get exactly the screenshot this post opened with, eighteen months later and with a lot more rows behind it.</p>



<h2 class="wp-block-heading">Need help wiring this together?</h2>



<p class="wp-block-paragraph">I work on exactly this kind of plumbing: the unglamorous layer between an ecommerce platform, a payment processor and the tools your team actually opens all day. Things I can help with here:</p>



<ul class="wp-block-list">
<li>Designing the identity model (surrogate keys, identifier tables, deterministic matching rules) and auditing an existing one for over-merging.</li>

<li>Building idempotent, signature-verified webhook intake for Shopify and Stripe, with a dead-letter path and a reconciliation job behind it.</li>

<li>Setting up ELT into Snowflake, BigQuery, Redshift or Postgres, whether with Fivetran, self-hosted Airbyte, or Stripe Data Pipeline where it fits.</li>

<li>Modelling orders, payments, refunds, disputes and payouts so finance and marketing can each get a number they trust.</li>

<li>Reverse ETL back into the CRM and helpdesk without write loops, rate-limit incidents or a hundred junk fields.</li>

<li>Making deletion and data-subject requests actually work across the warehouse, the raw landing zone and every downstream copy.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re stuck on a specific piece, send me the concrete thing: a webhook payload, a schema dump, the query that disagrees with your Stripe payout report. It&#8217;s much easier to say something useful about a real artifact than about a diagram.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Backfilling Historical API Data into S3 Without Silent Gaps</title>
		<link>https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 24 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Backfill]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Conditional Requests]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[S3 Inventory]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=268</guid>

					<description><![CDATA[<p>A backfill that exits zero can still be missing a week of data, and nothing will tell you. This is a practical guide to the failure families behind silent gaps: pagination drift under a mutating source, retries that duplicate pages, prefix layouts designed for writes instead of reads, the seam where backfill meets live ingest, and the storage class rules that make mistakes expensive. Includes deterministic key derivation, S3 conditional writes, Athena partition projection, and a per-window manifest pattern that turns completeness into something you can query.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/">Backfilling Historical API Data into S3 Without Silent Gaps</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The backfill exits zero. Every window processed, nothing red in the logs, the object count in the bucket looks about right for the range. You close the ticket. Weeks later an analyst asks why the chart dips for one week in the middle of the historical range, and nobody can answer, because as far as every system involved is concerned, nothing failed.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of the problem. When you backfill historical API data into S3, the failure to worry about isn&#8217;t the one that throws. It&#8217;s the one that writes fewer records than it should and reports success anyway. Bytes transferred look plausible. Nothing goes red. The data is quietly wrong, and you find out when someone downstream trusts it.</p>



<p class="wp-block-paragraph">This post covers the failure families that produce that outcome and the design decisions that make them impossible rather than merely unlikely: pagination that drifts under a mutating source, retries that write the same page twice under different keys, a prefix layout chosen while thinking about writes instead of reads, the seam where the backfill meets the live pipeline, and the storage class trap that makes a cheap-looking archive expensive to fix.</p>



<h2 class="wp-block-heading">Why a backfill that finishes cleanly can still be wrong</h2>



<p class="wp-block-paragraph">A live pipeline has a natural error signal. Data stops arriving, a lag metric climbs, someone notices. A backfill has none of that. It runs once, against a range nobody is watching in real time, and then it&#8217;s over. The only evidence it leaves behind is a set of objects in a bucket.</p>



<p class="wp-block-paragraph">So the question you need to answer months later isn&#8217;t &#8220;did the job succeed?&#8221; It&#8217;s &#8220;is this window complete, and how do I know?&#8221; If your design can&#8217;t answer that from artifacts on disk, you don&#8217;t have a backfill. You have a large amount of data of unknown provenance.</p>



<h2 class="wp-block-heading">Failure family 1: the source moved while you paginated</h2>



<p class="wp-block-paragraph">This one bites hardest and shows up least. Most REST APIs paginate historical queries with an offset or page number. You ask for page 1, page 2, page 3. Meanwhile the underlying table is still accepting writes.</p>



<p class="wp-block-paragraph">If a record is inserted into the middle of the result set between your page 4 and page 5 requests, everything after it shifts down by one. The row that would have started page 5 is now the last row of page 4, which you already fetched. You request page 5 and that record is gone. Not errored. Gone. Insertions cause skips, deletions cause duplicates, and the job still exits zero.</p>



<h3 class="wp-block-heading">What to do instead</h3>



<ul class="wp-block-list"><li><strong>Prefer keyset or cursor pagination.</strong> If the API offers an opaque cursor or a &#8220;records after ID X&#8221; parameter, use it. The read position is anchored to a value rather than a count, so inserts elsewhere in the set can&#8217;t shift it.</li><li><strong>Sort by an immutable column.</strong> If you&#8217;re stuck with offsets, sort by something that never changes after insert, usually the primary key or the creation timestamp. Sorting by a last-modified column while records are being modified is the worst case: rows physically move within the ordering as you walk it.</li><li><strong>Bound every request on both ends.</strong> Don&#8217;t ask for &#8220;everything before now.&#8221; Ask for a closed interval with an upper bound already in the past. A closed window can&#8217;t grow while you read it.</li><li><strong>Know which timestamp the filter applies to.</strong> Filtering on created-at gives you records that came into existence in that window. Filtering on updated-at gives you records touched in it, which for a mutable source is a different set entirely.</li></ul>



<p class="wp-block-paragraph">That last point causes arguments. Backfill on created-at, then run incremental syncs on updated-at, and your historical range never picks up edits made to old records. That may be correct for your use case. It is not correct by default, and it&#8217;s worth writing down which one you chose.</p>



<h2 class="wp-block-heading">Failure family 2: the retry that wrote the same page twice</h2>



<p class="wp-block-paragraph">Backfills retry. They have to, because you&#8217;re hammering an API for hours and something will time out. The problem is what the retry writes.</p>



<p class="wp-block-paragraph">If your object key contains a UUID, a run identifier or a wall-clock timestamp, a retried page produces a second object with a different key and the same records inside. You now have duplicates that listing will never reveal, because both objects are legitimately present and neither is obviously a copy. Derive the key from the request instead of the run, so the same window plus the same cursor always produces the same key and a retry overwrites rather than accumulates.</p>



<pre class="wp-block-code"><code>import hashlib
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    config=Config(
        retries={"total_max_attempts": 8, "mode": "standard"},
        max_pool_connections=32,
    ),
)

def page_key(dataset, window, cursor):
    # Deterministic: the same page always maps to the same key.
    token = hashlib.sha256(cursor.encode()).hexdigest()[:16]
    return f"raw/{dataset}/dt={window}/page-{token}.json.gz"

def put_once(bucket, key, body):
    try:
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfNoneMatch="*")
        return "written"
    except ClientError as err:
        if err.response["Error"]["Code"] == "PreconditionFailed":
            return "already_present"
        raise</code></pre>



<p class="wp-block-paragraph">Two things in there are worth explaining rather than copying.</p>



<p class="wp-block-paragraph"><code>IfNoneMatch="*"</code> is a conditional write. S3 accepts the PUT only if no object exists at that key and returns a 412 <code>PreconditionFailed</code> otherwise. It&#8217;s supported on <code>PutObject</code> and <code>CompleteMultipartUpload</code>. That turns &#8220;did I already write this page?&#8221; into a single request with no read-before-write race. If two workers hit the same key concurrently, S3 can instead return a 409 <code>ConditionalRequestConflict</code>, which AWS documents as retryable, so handle the two codes differently: retry on 409, skip on 412.</p>



<p class="wp-block-paragraph">The same thing from the CLI, useful for testing the behaviour by hand:</p>



<pre class="wp-block-code"><code>aws s3api put-object 
  --bucket your-bucket 
  --key raw/orders/dt=WINDOW/page-abc123.json.gz 
  --body page.json.gz 
  --if-none-match "*"</code></pre>



<p class="wp-block-paragraph">The <code>retries</code> block matters for a reason that trips people up. In botocore, <code>max_attempts</code> means different things depending on where you set it: in the AWS config file or the environment variable it counts total requests including the first, but in a <code>Config</code> object it counts retries on top of the first. <code>total_max_attempts</code> exists only in a <code>Config</code> object and always means total, so using it removes the ambiguity. There&#8217;s also an <code>adaptive</code> mode that layers client-side rate limiting on top of <code>standard</code>; AWS still labels it experimental, so I reach for <code>standard</code> plus my own concurrency cap first.</p>



<h2 class="wp-block-heading">Failure family 3: a layout designed for writing, not reading</h2>



<p class="wp-block-paragraph">The prefix structure you pick during the backfill is the one you&#8217;re stuck with. Reorganising later means rewriting every object, and copy requests aren&#8217;t free.</p>



<h3 class="wp-block-heading">Partition on event time, not ingestion time</h3>



<p class="wp-block-paragraph">The tempting layout is the date you ran the job. It&#8217;s easy, it&#8217;s monotonic, and it&#8217;s useless. Every query anyone writes against historical data filters on when the thing happened, not when you happened to fetch it. If your partitions are ingestion dates, every one of those queries becomes a full scan. Use Hive-style keys so query engines prune without extra configuration:</p>



<pre class="wp-block-code"><code>s3://your-bucket/raw/orders/dt=YYYY-MM-DD/page-&lt;token&gt;.json.gz
s3://your-bucket/curated/orders/dt=YYYY-MM-DD/part-&lt;n&gt;.parquet</code></pre>



<p class="wp-block-paragraph">Keep the raw landing zone and the curated zone separate. Raw is whatever the API gave you, unmodified, so you can re-derive everything if your parsing logic turns out to be wrong. Curated is columnar and typed. Merging them saves nothing and costs you the ability to reprocess.</p>



<h3 class="wp-block-heading">Don&#8217;t let the write path throttle itself</h3>



<p class="wp-block-paragraph">S3 documents at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes. The word doing the work is <em>partitioned</em>. That rate applies once S3 has split the keyspace out for that prefix, and AWS is explicit that the scaling is gradual rather than instant. During the ramp you can see 503 <code>SlowDown</code> responses even when your sustained rate looks well inside the documented ceiling.</p>



<p class="wp-block-paragraph">For a backfill this is mostly good news, since date-partitioned keys spread writes across many prefixes as you work through the range. It becomes a problem when you parallelise <em>within</em> a single day and drive thousands of writes per second into one <code>dt=</code> prefix. Two mitigations: back off with jitter instead of retrying 503s immediately, and shard workers across windows rather than within them.</p>



<h3 class="wp-block-heading">Register partitions without a scan</h3>



<p class="wp-block-paragraph">A backfill can create thousands of partitions at once. <code>MSCK REPAIR TABLE</code> handles that badly. AWS documents that it only adds partitions and never removes them, that it can take a long time on large tables, and that it can time out leaving the catalog partially updated. Partial success with no error is exactly the failure mode this whole post is about.</p>



<p class="wp-block-paragraph">Partition projection sidesteps the catalog. Athena computes the partition list from table properties instead of looking it up, so a backfilled range is queryable the moment the objects exist.</p>



<pre class="wp-block-code"><code>TBLPROPERTIES (
  'projection.enabled'          = 'true',
  'projection.dt.type'          = 'date',
  'projection.dt.format'        = 'yyyy-MM-dd',
  'projection.dt.range'         = 'START_DATE,NOW',
  'projection.dt.interval'      = '1',
  'projection.dt.interval.unit' = 'DAYS',
  'storage.location.template'   = 's3://your-bucket/curated/orders/dt=${dt}/'
)</code></pre>



<p class="wp-block-paragraph">Replace <code>START_DATE</code> with the earliest date in your backfill. The trade-off is worth stating plainly: projection generates every partition in the range whether or not data exists there, so an over-wide range on a sparse dataset wastes planning effort. And if the S3 layout ever stops matching <code>storage.location.template</code>, queries return nothing rather than erroring. Set the range to your actual data, not to a comfortable margin.</p>



<h2 class="wp-block-heading">Failure family 4: the seam between backfill and live pipeline</h2>



<p class="wp-block-paragraph">Two processes write to the same table: a backfill walking backwards through history, and a live ingest walking forwards. They meet somewhere, and that meeting point is where records go missing.</p>



<p class="wp-block-paragraph">The instinct is to make the handoff exact. Backfill up to timestamp T, start streaming from T. Don&#8217;t. Clock skew between your runner and the API, records that arrive late at the source, and rounding in the API&#8217;s own filtering will all conspire to open a hole at exactly T.</p>



<p class="wp-block-paragraph">Overlap deliberately instead. Run the backfill past the point where live ingest started, by a margin generously larger than any plausible source lag, then deduplicate on a natural key during curation. Duplicates you can remove with a query. Gaps you can only fix by going back to the API, and by then the retention window may have closed.</p>



<h2 class="wp-block-heading">Failure family 5: the storage class you can&#8217;t undo</h2>



<p class="wp-block-paragraph">Historical data feels like archive data, so the instinct is to write it straight into a cold class or transition it immediately. Read the mechanics first, because the billing rules for cold classes are structural and pro-rated against you.</p>



<ul class="wp-block-list"><li>S3 Standard-IA and One Zone-IA carry a 30-day minimum storage duration and a 128 KB minimum billable object size.</li><li>Glacier Instant Retrieval and Glacier Flexible Retrieval carry a 90-day minimum duration; Deep Archive carries 180 days. Delete, overwrite or transition earlier and you&#8217;re charged the remainder.</li><li>Objects in Glacier Flexible Retrieval and Deep Archive are billed for 40 KB of extra per-object metadata on top of the object itself.</li></ul>



<p class="wp-block-paragraph">Read those together and the conclusion is uncomfortable. A backfill that writes millions of small JSON pages into an archive class pays a size floor on every one of them, plus metadata overhead, plus a minimum duration that makes fixing the mistake expensive. Those pages are also the objects you&#8217;re most likely to want to reprocess, which is exactly the overwrite that triggers the early-deletion charge.</p>



<p class="wp-block-paragraph">What I&#8217;d actually do: land raw pages in Standard, compact them into larger Parquet files in the curated zone, verify completeness, and only then apply a lifecycle rule to the raw prefix. Compaction is what makes cold storage sane, because it turns a size floor per tiny object into a rounding error on a large one. It also fixes the small-file problem in Athena, which is a separate cost line.</p>



<p class="wp-block-paragraph">One more cost note: if your backfill runs from EC2 in a private subnet, every byte pulled from the API and pushed to S3 may route through a NAT gateway with per-GB processing charges. A gateway VPC endpoint for S3 removes the S3 half of that. It&#8217;s a five-minute change people usually discover after the bill.</p>



<h2 class="wp-block-heading">Proving the backfill is complete</h2>



<p class="wp-block-paragraph">Here&#8217;s the part most backfills skip, and the part that makes everything above verifiable rather than hopeful. Write a manifest object per window, alongside the data, describing what that window contains.</p>



<pre class="wp-block-code"><code>{
  "dataset": "orders",
  "window": "&lt;dt value&gt;",
  "filter_field": "created_at",
  "range_start_inclusive": "&lt;iso8601&gt;",
  "range_end_exclusive": "&lt;iso8601&gt;",
  "pages_written": "&lt;int&gt;",
  "records_written": "&lt;int&gt;",
  "source_reported_total": "&lt;int or null&gt;",
  "last_cursor": "&lt;opaque token&gt;",
  "schema_fingerprint": "&lt;sha256 of sorted field names&gt;"
}</code></pre>



<p class="wp-block-paragraph">The manifest turns completeness into something you can query. A window with no manifest was never finished. A window whose <code>records_written</code> disagrees with <code>source_reported_total</code> is suspect. A window whose <code>schema_fingerprint</code> differs from its neighbours tells you the API changed shape partway through your range, which is otherwise close to undetectable.</p>



<p class="wp-block-paragraph">Write the manifest last, after every page for that window is confirmed in S3. A manifest written optimistically at the start is worse than no manifest at all, because it asserts a completeness nobody checked.</p>



<h2 class="wp-block-heading">Troubleshooting a backfill that already ran</h2>



<ol class="wp-block-list"><li><strong>Count objects per window and look at the shape.</strong> Use S3 Inventory rather than a live <code>ListObjectsV2</code> walk on a large bucket. Plot objects per <code>dt=</code> partition. Real data has a rhythm, weekday peaks and weekend troughs. A window that breaks the rhythm without a business reason is your gap.</li><li><strong>Reconcile record counts against the source.</strong> Many APIs return a total count on the first page of a filtered query. Pull that for a sample of windows and compare with what&#8217;s in S3. If you kept manifests, this is one query.</li><li><strong>Check for duplicate natural keys within a partition.</strong> A <code>GROUP BY</code> on the record ID with <code>HAVING COUNT(*) &gt; 1</code> finds retry duplicates immediately. Duplicates clustered at window boundaries mean pagination drift; duplicates spread evenly mean your keys weren&#8217;t deterministic.</li><li><strong>Look at boundary hours, not whole days.</strong> Cursor drift and clock skew concentrate at the edges of each request window. Aggregate to the hour and the missing slice usually becomes obvious even when daily totals look fine.</li><li><strong>Verify the layout still matches the table definition.</strong> If you&#8217;re using partition projection and a query returns zero rows, the first suspect is <code>storage.location.template</code> not matching the actual prefix, trailing slashes included.</li></ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Treating an exit code of zero as evidence of completeness.</li><li>Putting a run ID or timestamp in the object key, which makes retries additive instead of idempotent.</li><li>Partitioning by ingestion date because it&#8217;s what the job naturally knows.</li><li>Making the backfill and live-ingest boundary exact instead of overlapping.</li><li>Writing millions of small objects straight into an archive class and meeting the minimum object size and duration rules afterwards.</li><li>Running <code>MSCK REPAIR TABLE</code> over thousands of new partitions and trusting that it completed.</li><li>Backfilling on a last-modified field while assuming you captured creation events.</li><li>Sharding workers within a single day, concentrating writes on one prefix and inviting 503s.</li></ul>



<h2 class="wp-block-heading">Best practices when you backfill historical API data into S3</h2>



<ul class="wp-block-list"><li>Make every window independently re-runnable. If re-running one day is routine rather than scary, you&#8217;ll actually fix problems when you find them.</li><li>Derive object keys deterministically from window plus cursor, and use conditional writes so a retry is a no-op.</li><li>Bound every API request on both ends, and never let the upper bound be &#8220;now&#8221;.</li><li>Write raw responses untouched before parsing anything. Parsing bugs are common; re-fetching a closed API window often isn&#8217;t possible.</li><li>Emit a manifest per window, written last, and reconcile it against source counts.</li><li>Compact to Parquet before applying any lifecycle transition.</li><li>Run the job somewhere it can survive for hours. A long backfill on a laptop over hotel wifi is a bad plan; a small dedicated VPS from a provider like Contabo or InterServer, or a spot-priced EC2 instance, costs little and won&#8217;t sleep halfway through.</li><li>Watch progress somewhere other than the terminal. Pushing records-per-window into CloudWatch or Grafana gives you a curve you can eyeball, and an unusually flat stretch is a gap you can catch while the source data is still retrievable.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Should I write raw JSON or convert to Parquet during the backfill?</h3>



<p class="wp-block-paragraph">Both, in two zones. Write the raw response body first so the fetch step has no dependency on your schema assumptions, then convert as a separate job. Converting inline couples an expensive, rate-limited API walk to a parsing step that will eventually throw on an unexpected field, and you don&#8217;t want to re-fetch years of data because of a type error.</p>



<h3 class="wp-block-heading">How large should each object be?</h3>



<p class="wp-block-paragraph">In the raw zone, one object per API page is fine and keeps idempotency simple. In the curated zone, aim for files in the tens to low hundreds of megabytes. Query engines pay a fixed overhead per file, so thousands of tiny Parquet files scan slower and cost more than a few large ones holding identical data.</p>



<h3 class="wp-block-heading">Can I run the backfill in parallel across date windows?</h3>



<p class="wp-block-paragraph">Yes, and that&#8217;s the right axis, because each window is independent and lands on its own prefix. The binding constraint is almost always the source API&#8217;s rate limit rather than S3. Add a global concurrency cap and honour any documented limit or <code>Retry-After</code> header. Getting your credentials throttled or suspended mid-backfill is a worse outcome than finishing slowly.</p>



<h3 class="wp-block-heading">Do I need AWS Glue or Airflow for this?</h3>



<p class="wp-block-paragraph">Not to start. A single script that takes a window as an argument, writes pages and a manifest, and is safe to re-run covers most backfills. Orchestration earns its place when you need retries across hundreds of windows, dependency ordering, or a visible run history. Glue and Step Functions fit AWS-native stacks; Airflow gives you more control and more to operate. Pick after the script works, not before.</p>



<h3 class="wp-block-heading">What if the API caps how far back I can query?</h3>



<p class="wp-block-paragraph">Then the retention boundary is your real deadline and it should drive priority. Fetch the oldest reachable data first, not the most recent, because the newest data is still available tomorrow and the oldest may not be. If the vendor offers a bulk export or reporting endpoint separate from the standard API, it&#8217;s usually faster and less rate-limited than paginating.</p>



<h3 class="wp-block-heading">Does any of this change with S3-compatible storage instead of AWS?</h3>



<p class="wp-block-paragraph">The layout, idempotency and manifest patterns carry over unchanged to Cloudflare R2, Backblaze B2, MinIO and similar. What varies is API-level feature support. Conditional writes with <code>If-None-Match</code>, request-rate behaviour and storage class semantics differ by provider, so verify the specific behaviour you&#8217;re depending on against that provider&#8217;s documentation rather than assuming parity with S3.</p>



<h3 class="wp-block-heading">How do I know the backfill is actually finished?</h3>



<p class="wp-block-paragraph">When every expected window has a manifest, every manifest&#8217;s record count reconciles against the source, and a duplicate check on natural keys returns nothing. Not when the script exits.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A backfill isn&#8217;t finished when the job exits. It&#8217;s finished when you can prove the range is complete without re-reading the source. Deterministic keys, conditional writes, closed windows, manifests and deliberate overlap all exist to make that proof cheap.</p>



<p class="wp-block-paragraph">If you&#8217;re about to backfill historical API data into S3, spend the extra hour on the key derivation and the manifest before you start the run. The alternative is learning about a gap from a chart, months later, with the source retention window already closed behind you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with a backfill?</h2>



<p class="wp-block-paragraph">This is a large part of what I do. If you&#8217;re staring at a historical range and an API with a rate limit, I can help with:</p>



<ul class="wp-block-list"><li>Designing an idempotent, re-runnable backfill job with deterministic keys and per-window manifests, so a failed day is a one-command fix.</li><li>Choosing a partition layout that serves your actual query patterns, and setting up Athena or Glue tables with partition projection so thousands of backfilled partitions become queryable without catalog scans.</li><li>Auditing a backfill that already ran, reconciling record counts against the source and finding the gaps and duplicates before your dashboards do.</li><li>Handling awkward source APIs: offset-only pagination, aggressive rate limits, undocumented result caps, schema drift partway through a historical range.</li><li>Compaction and lifecycle design, so cold storage actually saves money instead of hitting minimum object size and minimum duration charges.</li><li>Wiring the seam between the backfill and your live pipeline so there&#8217;s no gap at the handover, plus the monitoring that would have caught it.</li></ul>



<p class="wp-block-paragraph">Send me something concrete: a sample API response, your current key layout, or the object counts per partition. That&#8217;s usually enough to spot where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/">Backfilling Historical API Data into S3 Without Silent Gaps</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</title>
		<link>https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 21 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Amazon Managed Grafana]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Metric Math]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[MWAA]]></category>
		<category><![CDATA[Notification Policies]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=260</guid>

					<description><![CDATA[<p>A Glue job that stops running emits no metrics, so the dashboard stays green and the alert quietly resolves itself. Here is why Grafana monitoring for AWS data pipelines misses that failure, and the heartbeat, metric math and no-data configuration that closes the gap, along with the CloudWatch query costs and IAM boundaries nobody warns you about.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/">Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on a Monday. An analyst pings you: &#8220;the dashboard is showing last week&#8217;s numbers.&#8221; You open Grafana, look at the pipeline board, and every panel is calm. No red. No spikes. Error counts flat at zero.</p>



<p class="wp-block-paragraph">Then you check the Glue job history and the last successful run was six days ago. The job has not failed. It has not run at all. Because it never ran, it never emitted a metric, so CloudWatch recorded nothing and Grafana drew nothing. The panel is not green because the pipeline is healthy. It is green because the pipeline is silent.</p>



<p class="wp-block-paragraph">That gap is the most common problem I see with <strong>Grafana monitoring for AWS data pipelines</strong>, and it is not a Grafana bug. It follows from how AWS batch services emit metrics and how alert engines read an empty result set. This post covers that failure and the three that follow close behind: wrong signal, cost, and access. Then the dashboard layout, the alerting rules, and a troubleshooting list for empty panels.</p>



<h2 class="wp-block-heading">Failure family one: the pipeline stopped and nothing turned red</h2>



<p class="wp-block-paragraph">Application metrics are continuous. A web server emits request counts every minute forever, so a gap in the line genuinely means something is wrong.</p>



<p class="wp-block-paragraph">Pipeline metrics are not like that. Glue reports job metrics only while a job executes, and only if job metrics are enabled and the script initialises a GlueContext. Step Functions publishes execution metrics when executions happen. MWAA publishes DAG metrics when DAGs run. Between runs the metric does not exist. A daily job produces a few minutes of data and twenty-three hours of nothing.</p>



<p class="wp-block-paragraph">So the question that decides whether your monitoring works is: what does your alert engine do with an empty result?</p>



<h3 class="wp-block-heading">No Data and MissingSeries are not the same thing</h3>



<ul class="wp-block-list">
<li><strong>No Data</strong> means the query returned no data points at all. Grafana handles this with a per-rule setting and by default raises a separate alert named <code>DatasourceNoData</code>. That alert does not inherit your rule&#8217;s labels and it fires immediately, skipping the pending period. If your notification policy routes on a team label, it will not match.</li>



<li><strong>MissingSeries</strong> means the query still returns data but one series has disappeared. That is the dead-job case exactly: a query for failed tasks across all Glue jobs still returns rows for the jobs that ran, and the dead one just stops existing. Grafana holds the last state for a couple of evaluation cycles, then evicts the instance with a transition to Normal, annotated with <code>grafana_state_reason</code> set to <code>MissingSeries</code>.</li>
</ul>



<p class="wp-block-paragraph">Read that again, because it is the whole problem. A job that stops running produces an alert that resolves itself and sends a cheerful &#8220;back to normal&#8221; notification. Nobody investigates a resolution.</p>



<h3 class="wp-block-heading">The fix: stop inferring health from absence</h3>



<p class="wp-block-paragraph">Do not build the &#8220;did it run&#8221; signal out of metrics that only exist when it runs. Publish a heartbeat: a custom metric your pipeline writes once, deliberately, at the point you consider the run complete.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-data 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions PipelineName=orders_daily 
  --value 1 
  --unit Count</code></pre>



<p class="wp-block-paragraph">Put that at the end of the Glue script, in the final Step Functions state, or in the last task of the DAG. Now &#8220;did it run&#8221; has a positive signal instead of an inferred one.</p>



<p class="wp-block-paragraph">The heartbeat is still sparse, so you also need to tell the evaluator that absence means failure. In a CloudWatch alarm, that is the missing-data setting: the options are <code>missing</code>, <code>ignore</code>, <code>breaching</code> and <code>notBreaching</code>, and for a heartbeat you want <code>breaching</code>.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name "orders_daily-heartbeat-missing" 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions Name=PipelineName,Value=orders_daily 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 26 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts</code></pre>



<p class="wp-block-paragraph">One subtlety here bites people. CloudWatch looks back over an evaluation range wider than your evaluation periods, and if it finds enough real data points in that wider range, the missing-data setting is ignored entirely. For a once-a-day metric that behaviour can suppress the alarm you just built. The reliable way around it is metric math: fill the gaps before the alarm evaluates anything.</p>



<pre class="wp-block-code"><code>FILL(m1, 0)</code></pre>



<p class="wp-block-paragraph">Now every period has a value, zero means &#8220;no run completed&#8221;, and a plain threshold behaves the way you expected. <code>FILL(m1, REPEAT)</code> and <code>FILL(m1, LINEAR)</code> carry the last known value or interpolate instead, which suits gauges better than heartbeats.</p>



<p class="wp-block-paragraph">On the Grafana side, set the no-data state explicitly on each rule and build a notification policy that catches <code>alertname=DatasourceNoData</code>, otherwise those alerts go nowhere. The heartbeat still matters even with that in place, because it turns a disappearing series into a present series with a bad value, and every rule engine handles that correctly.</p>



<h2 class="wp-block-heading">Failure family two: you are monitoring the compute, not the data</h2>



<p class="wp-block-paragraph">The second failure is subtler. The job runs, succeeds, takes its usual time, and writes nothing, because an upstream API returned an empty page or a partition filter matched no files. Executor CPU and heap will not tell you that. Two signals will.</p>



<p class="wp-block-paragraph"><strong>Freshness</strong> is time since the last successful completion, per pipeline. It maps directly onto &#8220;can I trust the report on my screen&#8221;, which is the only question your stakeholders are asking. Derive it from the heartbeat.</p>



<p class="wp-block-paragraph"><strong>Volume</strong> is rows or bytes written per run, compared against the same run last week. A drop to zero with a successful exit code is the classic empty-load bug. For Spark jobs on Glue the built-in counters give you a usable start:</p>



<ul class="wp-block-list">
<li><code>glue.driver.aggregate.recordsRead</code> and <code>glue.driver.aggregate.bytesRead</code> for input volume</li>



<li><code>glue.driver.aggregate.numFailedTasks</code> for partial failures that do not fail the job</li>



<li><code>glue.driver.s3.filesystem.write_bytes</code> and the <code>glue.ALL</code> equivalents for output across executors</li>
</ul>



<p class="wp-block-paragraph">Glue reports these roughly every thirty seconds as deltas from the previous report, so Sum over the run window is usually what you want rather than Average. Beyond those, the signals I put on a board first are <code>ExecutionsStarted</code> minus <code>ExecutionsSucceeded</code> from the <code>AWS/States</code> namespace, since aborted and timed-out executions land in separate counters and failure count alone misses them; DAG metrics from the <code>AmazonMWAA</code> namespace, remembering that <code>metrics.metrics_allow_list</code> and <code>metrics.metrics_block_list</code> control what reaches CloudWatch at all; consumer lag rather than throughput on streaming ingestion, because a stream will happily accept records nobody is reading; and message age on dead letter queues, which is a near-perfect proxy for &#8220;something is failing repeatedly and nobody noticed&#8221;.</p>



<h2 class="wp-block-heading">Failure family three: the dashboard costs more than the pipeline</h2>



<p class="wp-block-paragraph">Grafana&#8217;s CloudWatch data source uses <code>GetMetricData</code> to fetch samples and <code>ListMetrics</code> when you pick dimensions in the editor. <code>GetMetricData</code> does not qualify for the CloudWatch API free tier, and it is billed per metric requested rather than per API call. One request pulling five hundred metrics is five hundred billable metric requests.</p>



<p class="wp-block-paragraph">Now multiply: panels, times series per panel, times statistics per series, times refresh rate, times every user with the board open, times every alert rule on its own schedule. A wildcard query matching every Glue job in the account looks tidy in the editor and quietly becomes the largest line item in your monitoring bill. Asking for more than five statistics on one metric counts as an extra metric request too.</p>



<p class="wp-block-paragraph">The levers, roughly in order of return:</p>



<ol class="wp-block-list">
<li>Set a sane refresh interval. Metrics that arrive every thirty seconds do not need a five-second refresh, and a daily batch board does not need auto-refresh at all.</li>



<li>Replace wildcard dimension matching with explicit resource lists on always-open boards. Keep wildcards for ad hoc investigation.</li>



<li>Query only the regions you run in. It is easy to leave a data source pointed at a region you migrated out of.</li>



<li>Use template variables instead of one duplicated board per team.</li>



<li>If you pull a large metric catalogue continuously, evaluate CloudWatch metric streams. Streams push through Amazon Data Firehose to a destination you control, and AWS recommends that path for third-party tools with heavy <code>GetMetricData</code> usage. The trade-off is genuine: you lose metric math and Metrics Insights against CloudWatch directly, and you pick up storage and Firehose ingestion costs instead.</li>
</ol>



<p class="wp-block-paragraph">To find which client is driving the spend, enable CloudTrail data events on <code>GetMetricData</code>. It surfaces the calling IAM principal and source address, which is usually enough to identify the forgotten dashboard.</p>



<h2 class="wp-block-heading">Failure family four: access, accounts and regions</h2>



<p class="wp-block-paragraph">Grafana ships CloudWatch support natively, so there is no plugin to install. What there is, is an IAM policy people habitually make too broad or too narrow. This is the read-only set the data source needs:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarms",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetMetricStatistics"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingRegions",
      "Effect": "Allow",
      "Action": "ec2:DescribeRegions",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Prefer an assumed role over long-lived access keys. If Grafana runs on EC2 or ECS in the same account, the instance or task role is cleanest and there is no secret to rotate.</p>



<p class="wp-block-paragraph">Two boundaries to plan for. CloudWatch quotas are defined per account and per region, so a multi-account estate can hit limits in one account while the rest are fine, and each needs its own quota increase. And metrics do not cross regions: a pipeline that reads in one region and writes in another needs both on the board, and the missing half is a common reason a panel looks empty for no obvious cause.</p>



<h2 class="wp-block-heading">Where to run Grafana for this</h2>



<p class="wp-block-paragraph">Three options, and the choice is mostly about viewer count and who carries the operational load.</p>



<ul class="wp-block-list">
<li><strong>Amazon Managed Grafana</strong> bills per active user license per workspace, split into editor and viewer tiers, with at least one editor license required per workspace even if nobody logs in. An active user is anyone who logged in or made an API request during the billing cycle. Excellent for a small engineering team, expensive if you hand read-only boards to a large business audience.</li>



<li><strong>Grafana Cloud</strong> from Grafana Labs bills on usage rather than seats, which flips that calculation. It also brings the AWS observability integrations and Grafana Alloy if you want CloudWatch metrics in a Prometheus-compatible store.</li>



<li><strong>Self-hosted</strong> is cheapest per seat and most work. A small VPS from a provider like Contabo or InterServer runs Grafana for a handful of pipelines comfortably, and you own upgrades, TLS renewal and backups. One licensing note: Grafana&#8217;s open source core moved from Apache 2.0 to AGPLv3, while plugins and agents stayed Apache-licensed. If your legal team has a policy on AGPL, have that conversation before deployment.</li>
</ul>



<h2 class="wp-block-heading">Laying out a board people actually read</h2>



<p class="wp-block-paragraph">Organise by question, not by service. Whoever opens this board on a Monday morning has one question, and the answer should be visible without scrolling.</p>



<ol class="wp-block-list">
<li><strong>Freshness table.</strong> One row per pipeline, hours since last successful completion, threshold colour tied to that pipeline&#8217;s actual SLA. Most viewers read only this row.</li>



<li><strong>Volume against baseline.</strong> Records or bytes written per run, overlaid with the same period a week ago.</li>



<li><strong>Failure counts and durations</strong>, grouped by stage so a slow extract is distinguishable from a slow load.</li>



<li><strong>Resource detail at the bottom.</strong> Executor memory, DPU usage, throttling. Useful during an incident, noise the rest of the time.</li>
</ol>



<p class="wp-block-paragraph">Use a template variable for pipeline name rather than one board per pipeline. It keeps the query count down and new pipelines appear automatically. For cross-pipeline aggregation without fanning out into individual series, CloudWatch Metrics Insights runs a SQL-style query inside CloudWatch and the Grafana data source supports it:</p>



<pre class="wp-block-code"><code>SELECT SUM(ExecutionsFailed)
FROM SCHEMA("AWS/States", StateMachineArn)
GROUP BY StateMachineArn</code></pre>



<p class="wp-block-paragraph">Each <code>GetMetricData</code> operation can carry only one Metrics Insights query, so treat it as a way to consolidate one fan-out, not to run several at once.</p>



<h2 class="wp-block-heading">Troubleshooting: why the panel is empty</h2>



<ol class="wp-block-list">
<li><strong>Check the region on the query, not the data source.</strong> Grafana allows a per-query region override and it is easy to leave one on default.</li>



<li><strong>Widen the time range to seven days.</strong> If data appears, the metric is sparse and you have a heartbeat problem, not a query problem.</li>



<li><strong>Check the statistic.</strong> Sum and Average behave very differently on delta counters, and empty-looking panels are often Average over a window that is mostly gaps.</li>



<li><strong>Check dimensions match exactly.</strong> With Match Exact on, every dimension must be specified, so a metric published with two dimensions returns nothing when queried with one. Turning Match Exact off makes Grafana generate a search expression instead, which is the usual fix.</li>



<li><strong>Check job metrics are enabled.</strong> For Glue you need both the job metrics option and a GlueContext in the script. Without the GlueContext, nothing is emitted regardless of the console setting.</li>



<li><strong>Check IAM.</strong> A role with <code>ListMetrics</code> but not <code>GetMetricData</code> populates every dropdown perfectly and returns no data, which is confusing until you have seen it once.</li>



<li><strong>Check for throttling.</strong> Panels that are intermittently empty under load rather than consistently empty usually mean a CloudWatch API quota for that account and region.</li>
</ol>



<p class="wp-block-paragraph">For alert rules behaving oddly rather than panels, treat the Error state separately from No Data. Grafana raises <code>DatasourceError</code> on evaluation timeouts or repeated query failures, governed by the <code>evaluation_timeout</code> and <code>max_attempts</code> settings. Slow CloudWatch queries under quota pressure trip this, and the resulting alert looks nothing like the one you wrote.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Alerting on failure counts only. A job that never starts cannot fail, so failure-count alerting is blind to the most common outage.</li>



<li>Leaving no-data handling on the default and never routing <code>DatasourceNoData</code>. The alert fires into the void.</li>



<li>Treating a MissingSeries resolution as good news. If alerts resolve without anyone acting, check the reason annotation.</li>



<li>Importing a large community dashboard pack wholesale. Most of those panels use wildcards, and you have just subscribed to a permanent <code>GetMetricData</code> bill for metrics nobody reads.</li>



<li>Monitoring the orchestrator but not the destination. Step Functions reports a clean success while the load that ran inside it wrote zero rows.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Emit a heartbeat from every scheduled pipeline using one consistent namespace and dimension scheme. Consistency is what lets a single dashboard cover everything.</li>



<li>Write down a freshness SLA per pipeline, then set the threshold from it. &#8220;Alert if stale&#8221; only means something once someone says how stale is too stale.</li>



<li>Provision dashboards and alert rules as code in the same repository as the pipeline. A pipeline that ships without its monitoring is not finished.</li>



<li>Route by severity, not by service. Freshness breaches page; executor memory warnings go to a channel.</li>



<li>Break something on purpose to test the alert. Disable the schedule on a non-production pipeline and confirm you get paged. An untested alert is a guess.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a plugin to connect Grafana to CloudWatch?</h3>



<p class="wp-block-paragraph">No. Grafana includes native support for the Amazon CloudWatch data source. Configure credentials or an assumed role, set a default region, and query. The same data source also reaches CloudWatch Logs and X-Ray.</p>



<h3 class="wp-block-heading">Why does my Grafana alert resolve itself when the pipeline is still broken?</h3>



<p class="wp-block-paragraph">Almost certainly MissingSeries eviction. When one series in a multi-series query disappears, Grafana holds the last state briefly and then transitions the instance to Normal with <code>grafana_state_reason</code> set to <code>MissingSeries</code>. It is not treated as No Data. A heartbeat metric with gaps filled to zero stops the series disappearing in the first place.</p>



<h3 class="wp-block-heading">CloudWatch dashboards or Grafana for AWS data pipelines?</h3>



<p class="wp-block-paragraph">CloudWatch dashboards are the shorter path if everything sits in one account and the visualisation options suit you. Grafana earns its place when you need pipeline metrics correlated with a warehouse query, a database, a trace or a business KPI on one screen, and when alerting has to route to several destinations. Most data platforms end up wanting the correlation.</p>



<h3 class="wp-block-heading">How do I monitor a Glue job that succeeds but writes no data?</h3>



<p class="wp-block-paragraph">Alert on output volume rather than exit status. Use the Glue write byte counters, or better, emit your own row-count metric after the write so it reflects what your code believes it wrote. Compare against the equivalent run a week earlier, because fixed thresholds age badly as data grows.</p>



<h3 class="wp-block-heading">Does Grafana querying CloudWatch cost money?</h3>



<p class="wp-block-paragraph">Yes. Grafana uses <code>GetMetricData</code>, which is billed per metric requested and excluded from the CloudWatch API free tier. Cost scales with panels multiplied by series multiplied by refresh frequency multiplied by viewers, so wildcards on an auto-refreshing board are the expensive pattern. Check current rates on the CloudWatch pricing page and do the arithmetic for your own board.</p>



<h3 class="wp-block-heading">Amazon Managed Grafana or self-hosted?</h3>



<p class="wp-block-paragraph">It comes down to viewer count. Managed Grafana bills per active user per workspace, which suits a small engineering team and gets costly when read-only dashboards go out widely. Self-hosting removes the per-seat cost and adds upgrade, TLS and backup work. Grafana Cloud sits between them on usage-based billing. Count likely viewers first, then choose.</p>



<h3 class="wp-block-heading">What is the minimum useful monitoring for a new pipeline?</h3>



<p class="wp-block-paragraph">A heartbeat proving the run completed, a row or byte count proving it moved something, and an alert that fires when the heartbeat is missing for longer than the agreed freshness window. Everything else is refinement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Effective <strong>Grafana monitoring for AWS data pipelines</strong> is not about drawing more metrics. It is about refusing to infer health from absence. Batch pipelines are silent most of the time, and every monitoring system in existence reads silence as calm unless you tell it otherwise.</p>



<p class="wp-block-paragraph">Publish a heartbeat. Fill the gaps with zero. Set your no-data behaviour deliberately. Then alert on freshness and volume, because those are the two things people downstream will notice before you do. Do that, and a green panel means something.</p>



<h2 class="wp-block-heading">Need help building this on your AWS estate?</h2>



<p class="wp-block-paragraph">I design and build monitoring for AWS data platforms. This is the kind of work I take on:</p>



<ul class="wp-block-list">
<li>Heartbeat and freshness instrumentation across Glue, Step Functions, MWAA and Lambda pipelines, with one consistent namespace scheme so a single dashboard covers everything</li>



<li>Grafana dashboards built around freshness and volume rather than executor charts nobody reads</li>



<li>Alert rules and notification policies that handle No Data and MissingSeries correctly, routed to Slack, Teams, Telegram or email by severity</li>



<li>CloudWatch query cost reviews, tracing <code>GetMetricData</code> spend back to the dashboard or rule causing it</li>



<li>Cross-account and multi-region data source setup with least-privilege IAM roles instead of long-lived keys</li>



<li>Grafana deployment and hardening on Amazon Managed Grafana, Grafana Cloud or a self-hosted instance behind TLS</li>
</ul>



<p class="wp-block-paragraph">If you have a pipeline that failed quietly, send me the job definition, the alert rule, or a screenshot of the panel that stayed green. I would rather look at the actual thing than talk in generalities.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/">Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</title>
		<link>https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 19 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AppFlow]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event Relay]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Named Credentials]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Platform Events]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Pub/Sub API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=209</guid>

					<description><![CDATA[<p>Most Salesforce AWS integrations don't fail loudly, they drift. This guide covers the four integration paths between Salesforce and AWS, when to use each, the silent failure mode hiding in every one of them, and the reconciliation job you have to build yourself.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the ops channel asks why the opportunity count on the QuickSight dashboard doesn&#8217;t match the count in the Salesforce report. It&#8217;s off by a few hundred. Nobody can answer that, and worse, nobody can say when the two numbers stopped agreeing. Every pipeline is green. The scheduled flow reports successful runs. The EventBridge rule shows invocations. Everything is healthy except the data.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of most Salesforce AWS integration problems. They&#8217;re rarely outages. They&#8217;re divergence, accumulating quietly while every status page stays green, and you find out about it when someone in finance runs a number past a customer.</p>



<p class="wp-block-paragraph">This guide covers the four integration paths between Salesforce and AWS, which one to reach for and when, the specific failure each one hides, and the reconciliation layer you&#8217;ll have to build yourself because neither vendor ships it.</p>



<h2 class="wp-block-heading">Why a Salesforce AWS integration fails silently</h2>



<p class="wp-block-paragraph">Start here, because it changes how you design everything downstream. There are three mechanisms that lose data without raising an error anywhere you&#8217;re looking.</p>



<p class="wp-block-paragraph"><strong>Events sent to an unassociated partner event bus are dropped on the floor.</strong> When you create an Event Relay, Salesforce creates a partner event source in your AWS account. Until you associate that source with an event bus, EventBridge discards anything the partner publishes to it. AWS is explicit that those events are not persisted at rest. There&#8217;s no queue holding them for you, no backlog to drain once you finish the setup. If the relay is running and the bus isn&#8217;t associated, you are streaming into a void.</p>



<p class="wp-block-paragraph"><strong>The Salesforce event bus retains high-volume events for 72 hours.</strong> Platform events and change data capture events sit on the event bus for three days, and each one carries a replay ID so a durable subscriber can resume from where it left off. Past that window, they&#8217;re purged. Salesforce doesn&#8217;t guarantee availability beyond it. So a relay that goes into an error state on a Friday afternoon is a data loss event by Monday morning, and the only symptom during those three days is an absence of messages, which looks identical to a quiet weekend.</p>



<p class="wp-block-paragraph"><strong>Deletes don&#8217;t show up in query-based syncs.</strong> A scheduled pull selects records that exist. Records that stopped existing between runs are simply not in the result set, so your S3 prefix or Redshift table keeps a row that Salesforce no longer has. Nothing errors. The row count on the AWS side just creeps upward relative to the CRM, forever, until someone notices.</p>



<p class="wp-block-paragraph">Every architecture decision below is really a decision about which of these three you&#8217;re accepting and how you&#8217;re going to detect it.</p>



<h2 class="wp-block-heading">A Salesforce AWS integration starts with a direction, not a service</h2>



<p class="wp-block-paragraph">The most common mistake I see is picking a service first, usually whichever one appeared in a conference demo, and then bending the requirement to fit it. Two questions decide the path, and they take about a minute to answer.</p>



<ol class="wp-block-list">
<li><strong>Which way does the data move?</strong> Salesforce to AWS, or AWS to Salesforce. If the answer is &#8220;both&#8221;, you have two integrations, not one, and they will have different failure modes and different owners.</li>

<li><strong>What shape is the workload?</strong> Batch, where you want a consistent set of records on a schedule, or event-driven, where you want a reaction within seconds of a change.</li>
</ol>



<p class="wp-block-paragraph">That gives you four cells, and each cell has a default answer.</p>



<h3 class="wp-block-heading">Salesforce to AWS, batch: Amazon AppFlow</h3>



<p class="wp-block-paragraph">Amazon AppFlow is the managed answer for pulling Salesforce objects into S3, Redshift, or the Glue Data Catalog on a schedule or on demand. It handles the OAuth connection, field mapping, and incremental pulls without you writing or hosting anything. For analytics loads it&#8217;s usually the right default, and it&#8217;s the path of least resistance if the destination is a data lake you&#8217;re already cataloguing with Glue and querying with Athena.</p>



<p class="wp-block-paragraph">The trade-off is that it&#8217;s a polling client, and it spends your Salesforce API allocation to do it. AWS documents this plainly with an example: an hourly flow pulling five pages of data makes 120 API calls a day, and all of them count against the org&#8217;s 24-hour API request limit. Multiply that by every object you&#8217;re syncing and every environment you&#8217;re syncing from, and AppFlow can become the largest single consumer of an allocation that Apex callouts, integration users, and third-party tools are also drawing on.</p>



<p class="wp-block-paragraph">The failure mode that catches people: if a field referenced in the flow&#8217;s mapping is deleted in Salesforce, the flow run fails. Someone cleans up an unused custom field on a Tuesday and your nightly load breaks on Wednesday, in a service that nobody on the Salesforce side knows exists. AppFlow will automatically pick up newly created Salesforce fields when the destination is S3, but the reverse case, a removed field, needs a human to edit the mapping. Put the flow names somewhere the Salesforce admins will see them before they run a field cleanup.</p>



<h3 class="wp-block-heading">Salesforce to AWS, event-driven: Event Relay and Amazon EventBridge</h3>



<p class="wp-block-paragraph">Event Relay subscribes to a channel on the Salesforce event bus and streams platform events and change data capture events into Amazon EventBridge as a partner event source. No middleware, no subscriber you have to keep alive, no polling and therefore no API allocation burn on the read path. From EventBridge you fan out to Lambda, Step Functions, SQS, Firehose, whatever the downstream is.</p>



<p class="wp-block-paragraph">Setup runs across both consoles, and the order matters. On the Salesforce side you create a named credential that points at your AWS account and region. It uses no authentication protocol, because trust is established by the partner event source relationship rather than by a credential in the request. The region goes in uppercase, which is the single most common reason a first attempt doesn&#8217;t work.</p>



<pre class="wp-block-code"><code>POST /services/data/vXX.X/tooling/sobjects/NamedCredential/

{
  "FullName": "AwsRelayCredential",
  "Metadata": {
    "label": "AwsRelayCredential",
    "endpoint": "arn:aws:US-EAST-1:111122223333",
    "principalType": "NamedUser",
    "protocol": "NoAuthentication",
    "generateAuthorizationHeader": true
  }
}</code></pre>



<p class="wp-block-paragraph">The endpoint is not a URL. It&#8217;s an ARN-shaped string carrying the AWS region and the twelve-digit account ID, and that&#8217;s the whole addressing scheme. Get the account wrong and the partner event source appears in a stranger&#8217;s console, not yours.</p>



<p class="wp-block-paragraph">You then create an event channel, add the platform events or change events you want as channel members, and create the relay configuration against that channel. A relay is created in a stopped state deliberately, so you get a chance to verify the AWS account and region before anything flows. That pause is the useful part of the design, not an inconvenience to click through.</p>



<p class="wp-block-paragraph">Now switch to AWS and associate the source. This is the step that, skipped, produces the silent drop described earlier.</p>



<pre class="wp-block-code"><code># Find the partner event source Salesforce created in your account
aws events list-event-sources --region us-east-1

# Create the matching bus. Both values must be the source name, character for character.
aws events create-event-bus 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --event-source-name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1

# Confirm the source moved from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph"><code>describe-event-source</code> returning <code>ACTIVE</code> is your proof that a matching bus exists and is enabled. <code>PENDING</code> means either you haven&#8217;t created the bus or it&#8217;s deactivated, and every event published in that state is gone. This is the single check worth wiring into a synthetic monitor, because it&#8217;s cheap to run and it catches the most expensive mistake.</p>



<p class="wp-block-paragraph">Only once the source is active do you set the relay state to run. From the Salesforce side, the feedback object is where relay errors surface rather than in any log you&#8217;d normally tail:</p>



<pre class="wp-block-code"><code>SELECT Id, RemoteResource, Status, ErrorMessage, ErrorTime, ErrorIdentifier
FROM EventRelayFeedback
WHERE EventRelayConfigId = '7k2XXXXXXXXXXXXXXX'</code></pre>



<p class="wp-block-paragraph">If you&#8217;d rather not use Event Relay at all, the Pub/Sub API is the direct route: a gRPC interface to the same event bus, delivering Avro-encoded payloads, which you subscribe to from your own consumer. That&#8217;s more code and a process to keep running, on ECS or a VPS from a provider like InterServer or Hetzner if you&#8217;re keeping it outside AWS, but you get explicit control over replay ID checkpointing. Teams already running Kafka often go this way and land the stream through a managed connector from Confluent instead. Event Relay is the lower-operational-cost option; the Pub/Sub API is the one to reach for when you need the checkpoint under your own control.</p>



<h3 class="wp-block-heading">AWS to Salesforce, event-driven: EventBridge API Destinations</h3>



<p class="wp-block-paragraph">Going the other way, an API destination lets EventBridge call the Salesforce REST API directly, with no Lambda in the middle. You create a connection holding OAuth client credentials from a Salesforce connected app, point the destination at your org&#8217;s My Domain endpoint, and route matched events to it. Publishing back into an inbound platform event is the common pattern, so the write lands on the Salesforce event bus and Apex or Flow picks it up from there.</p>



<p class="wp-block-paragraph">Two constraints shape the design, and both are documented:</p>



<ul class="wp-block-list">
<li><strong>Five second client execution timeout.</strong> If Salesforce takes longer than that to respond, EventBridge times out the request and treats it as a retryable failure. Anything you make synchronous behind that endpoint, a trigger doing a callout, a validation chain, a busy org at peak, will eat into it.</li>

<li><strong>The default retry policy is generous, which is the problem.</strong> For event buses, EventBridge will retry a failed delivery for up to 24 hours and up to 185 times by default. If the endpoint is genuinely broken, that&#8217;s a day of retries against your org, consuming API calls, before the event is discarded with nowhere to go.</li>
</ul>



<p class="wp-block-paragraph">Tighten the policy and always attach a dead letter queue. Failed events land somewhere you can inspect and replay rather than evaporating at the end of the retry window:</p>



<pre class="wp-block-code"><code>{
  "Id": "salesforce-writeback",
  "Arn": "arn:aws:events:us-east-1:111122223333:api-destination/sf-platform-event/&lt;id&gt;",
  "RoleArn": "arn:aws:iam::111122223333:role/EventBridgeApiDestinationRole",
  "RetryPolicy": {
    "MaximumRetryAttempts": 20,
    "MaximumEventAgeInSeconds": 3600
  },
  "DeadLetterConfig": {
    "Arn": "arn:aws:sqs:us-east-1:111122223333:salesforce-writeback-dlq"
  }
}</code></pre>



<p class="wp-block-paragraph">Set the invocation rate limit on the destination to something your org can absorb. It exists specifically so you can align delivery with a third-party API&#8217;s limits, and Salesforce is very much a third-party API with limits.</p>



<p class="wp-block-paragraph">One thing worth knowing before you design around it: API destinations don&#8217;t support mutual TLS. If your security review requires mTLS on outbound calls, you&#8217;re putting a Lambda or a proxy in the path regardless.</p>



<h3 class="wp-block-heading">AWS to Salesforce, batch: Lambda and the Bulk API</h3>



<p class="wp-block-paragraph">For loading volume back into the org, nothing managed does this well and you write it yourself. A Lambda or a container job authenticates with the JWT bearer flow against a connected app, submits a Bulk API job, and polls for results. The REST API is fine for small writes; past a few thousand records the per-call overhead and the API allocation make the Bulk API the only sensible choice.</p>



<p class="wp-block-paragraph">This is the cell where you own everything: retries, idempotency, partial-failure handling, and backoff when the org throttles you. Use external ID fields and upsert semantics rather than insert, so a retried batch converges instead of duplicating. Store the job ID before you submit, not after, so a Lambda timeout mid-flight leaves you something to reconcile against. And keep the Salesforce credential in Secrets Manager with rotation wired up, because a hardcoded connected app secret in an environment variable is the finding that shows up in every audit.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The network layer: public internet, PrivateLink, or Private Connect</h2>



<p class="wp-block-paragraph">All four paths above work over the public internet with TLS, and for most workloads that&#8217;s genuinely fine. The question is whether your compliance posture or your risk appetite says otherwise.</p>



<p class="wp-block-paragraph">Salesforce Private Connect is the managed private path. Salesforce operates a transit VPC in each supported AWS region, and you create a PrivateLink connection between that transit VPC and yours. Traffic between the two never traverses the public internet. The terminology is worth getting right because it trips people up in design reviews:</p>



<ul class="wp-block-list">
<li><strong>Inbound</strong> means traffic flowing <em>into</em> Salesforce. AWS to Salesforce, or on-premises to Salesforce.</li>

<li><strong>Outbound</strong> means traffic flowing <em>out of</em> Salesforce. Salesforce calling a service in your VPC.</li>
</ul>



<p class="wp-block-paragraph">The direction is named from Salesforce&#8217;s point of view, not yours, and getting it backwards means provisioning the wrong connection.</p>



<p class="wp-block-paragraph">AppFlow can ride on this too. Choosing the PrivateLink option when you create the Salesforce connection has AppFlow route through Private Connect and manage the endpoint lifecycle for you, which is a meaningful simplification over building it by hand.</p>



<p class="wp-block-paragraph">Where I&#8217;d push back: Private Connect is a licensed add-on, and each VPC needs its own connection, so a multi-account AWS estate with separate VPCs per environment multiplies the cost. If the driver is a checkbox on a questionnaire rather than a real threat model, TLS with strict IP allowlisting on the Salesforce side and a well-scoped connected app gets you most of the way for nothing. If the driver is a regulator who wants to see that customer data never touches a public network path, Private Connect is the clean answer and the audit conversation ends quickly. Decide which one you&#8217;re in before you start pricing it.</p>



<h2 class="wp-block-heading">The reconciliation job nobody builds</h2>



<p class="wp-block-paragraph">This is the part that separates an integration that survives from one that quietly rots, and it&#8217;s about thirty lines of code plus a schedule.</p>



<p class="wp-block-paragraph">Run a count comparison on a window that both sides can express. Nightly is plenty:</p>



<pre class="wp-block-code"><code>-- Salesforce side, via the REST query endpoint
SELECT COUNT(Id) FROM Opportunity
WHERE LastModifiedDate = LAST_N_DAYS:1

-- AWS side, same window, in Athena
SELECT COUNT(DISTINCT id) FROM crm.opportunity
WHERE last_modified_date &gt;= date_add('day', -1, current_date)</code></pre>



<p class="wp-block-paragraph">If the two numbers differ by more than a small tolerance for clock skew at the window edges, alert. That single check catches a stopped relay, a failed flow, a mapping that silently dropped a field, and a partner event source sitting in <code>PENDING</code>. It catches all of them without you having to instrument each path separately, which is exactly why it&#8217;s worth building first.</p>



<p class="wp-block-paragraph">For the delete problem, Salesforce exposes a dedicated endpoint. Ask it for records removed in a window and apply those deletions on the AWS side:</p>



<pre class="wp-block-code"><code>GET /services/data/vXX.X/sobjects/Opportunity/deleted/
    ?start=&lt;ISO-8601 timestamp, URL encoded&gt;
    &amp;end=&lt;ISO-8601 timestamp, URL encoded&gt;</code></pre>



<p class="wp-block-paragraph">The window you can ask about is bounded by how long deleted records stay recoverable, so run it more often than that bound, not less. Change data capture delivers delete events too, which is the better answer when it&#8217;s available, but the endpoint is what you use to backfill after a gap, and after a 72-hour gap it&#8217;s the <em>only</em> thing that will tell you what you missed.</p>



<p class="wp-block-paragraph">Put both checks on one dashboard alongside your AppFlow run status and EventBridge failed-invocation metrics. CloudWatch will do it, and if you&#8217;re already running Grafana Cloud or Datadog for the rest of the estate, pull the Salesforce-side numbers in as a custom metric so one panel answers &#8220;are the two systems agreeing right now&#8221;.</p>



<h2 class="wp-block-heading">Troubleshooting a Salesforce AWS integration</h2>



<ul class="wp-block-list">
<li><strong>Relay is running, nothing arrives in EventBridge.</strong> Run <code>describe-event-source</code>. If the state is <code>PENDING</code>, the bus was never created or has been deactivated, and everything published so far is gone. Create the bus, then treat the gap as a backfill job.</li>

<li><strong>Relay won&#8217;t start, or flips to an error state.</strong> Query <code>EventRelayFeedback</code> for the config. The usual causes are a wrong AWS account ID in the named credential, a region in lowercase, or the channel having no members.</li>

<li><strong>Events arrive but no rule matches.</strong> Check the rule is attached to the partner event bus, not the default bus. This one costs people an afternoon. Match on the source prefix and the detail type, and test with an event pattern before wiring a real target.</li>

<li><strong>An AppFlow flow that ran for months suddenly fails.</strong> Look for a field in the mapping that no longer exists in the source object. Also check for a bad gateway error, which points at a connection configured with the wrong instance URL and is fixed by deleting the connection and recreating it rather than editing it.</li>

<li><strong>API destination returns 401 intermittently.</strong> The connection refreshes its OAuth token when it sees a 401 or 407. Persistent 401s usually mean the connected app has no usable scopes for the client credentials flow, or the run-as user was changed. Check the connected app&#8217;s scopes before you touch anything in AWS.</li>

<li><strong>Everything times out at almost exactly five seconds.</strong> That&#8217;s the API destination timeout, not Salesforce being slow. Move the work behind an inbound platform event so the HTTP call returns immediately and the processing happens asynchronously in the org.</li>

<li><strong>Unexplained API limit exhaustion.</strong> Sum the scheduled AppFlow runs against page counts before blaming a rogue integration user. A frequent flow across several objects adds up faster than most people expect.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Starting the relay before associating the partner event bus, and assuming the events queued somewhere.</li>

<li>Treating the 72-hour retention as a comfortable buffer. It&#8217;s three days including a weekend, and an unmonitored relay burns through it without a single alert.</li>

<li>Using change data capture as a replacement for an initial load. It streams changes from the moment you enable it; it doesn&#8217;t backfill history.</li>

<li>Building the whole integration bidirectionally through one mechanism, then discovering you&#8217;ve created an echo loop where a write from AWS fires a change event that triggers another write.</li>

<li>Sizing the AppFlow schedule for freshness without checking what it does to the org&#8217;s API allocation.</li>

<li>Leaving the default 24-hour EventBridge retry policy on an API destination with no dead letter queue.</li>

<li>Provisioning Private Connect for every VPC because it&#8217;s the secure-sounding option, without modelling the per-connection cost across environments.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Build the reconciliation check before you build the second integration path. It&#8217;s the only thing that tells you the whole system is working.</li>

<li>Monitor the partner event source state, not just rule invocations. Invocations go to zero on a quiet day and on a broken day, and the state field distinguishes them.</li>

<li>Give every integration its own connected app and its own integration user. Shared credentials make it impossible to attribute API consumption or revoke one path without breaking three.</li>

<li>Make writes idempotent with external IDs and upsert. Every path here retries, and some retry aggressively.</li>

<li>Version your event payloads. Platform events are immutable once published, and adding a required field later means a new event definition, not an edit.</li>

<li>Keep the AWS-side resource names discoverable from the Salesforce side. Put the flow name or event bus name in the named credential label so a Salesforce admin can see what&#8217;s attached before they change something.</li>

<li>Define the whole thing in Terraform or CloudFormation on the AWS side and in metadata on the Salesforce side, so a sandbox refresh doesn&#8217;t leave you clicking through two consoles from memory.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need MuleSoft to integrate Salesforce with AWS?</h3>



<p class="wp-block-paragraph">No. For the four patterns above, the native services cover it without an integration platform. MuleSoft, Boomi, and similar tools earn their place when you&#8217;re orchestrating across many systems, need transformation logic that lives outside both platforms, or have a governance requirement for a single integration layer. For a two-system integration, they&#8217;re usually more machinery than the problem needs.</p>



<h3 class="wp-block-heading">Does Event Relay use my Salesforce API allocation?</h3>



<p class="wp-block-paragraph">Event delivery counts against your platform event delivery allocation, which is a separate budget from the 24-hour API request limit that REST calls and AppFlow polling draw on. That separation is one of the strongest arguments for the event-driven path when API consumption is already tight in the org.</p>



<h3 class="wp-block-heading">Can I replay Salesforce events that AWS missed?</h3>



<p class="wp-block-paragraph">Within the 72-hour retention window, yes, using replay IDs from a durable subscriber. Beyond it, no. The events are purged and there&#8217;s no recovery path through the event bus. Backfilling from that point means querying the objects directly and using the deleted-records endpoint to catch removals, which is exactly why the reconciliation job matters.</p>



<h3 class="wp-block-heading">Should I use AppFlow or write my own Lambda extractor?</h3>



<p class="wp-block-paragraph">Start with AppFlow. It removes the auth handling, the pagination, and the scheduling, and for standard object pulls into S3 or Redshift it&#8217;s hard to justify writing that yourself. Move to a custom extractor when you need transformation logic AppFlow can&#8217;t express, when you&#8217;re hitting connector limitations on a specific object, or when you want tighter control over API consumption than a fixed schedule allows.</p>



<h3 class="wp-block-heading">How do I send data from AWS back into Salesforce without a Lambda?</h3>



<p class="wp-block-paragraph">EventBridge API destinations call the Salesforce REST API directly using OAuth client credentials from a connected app. Point them at an inbound platform event endpoint so the write returns fast and processing happens asynchronously in the org. You still need a Lambda if you require mTLS or payload transformation beyond what the input transformer handles.</p>



<h3 class="wp-block-heading">Is Private Connect required for a secure Salesforce AWS integration?</h3>



<p class="wp-block-paragraph">Not required. Everything runs over TLS by default. Private Connect removes the public internet from the path entirely, which matters when a regulator or a security review asks you to demonstrate it. It&#8217;s a licensed add-on with a per-connection cost, so model it against your actual VPC count before committing.</p>



<h3 class="wp-block-heading">Why does my EventBridge rule never fire even though events are arriving?</h3>



<p class="wp-block-paragraph">Almost always because the rule is on the default event bus rather than the partner event bus. Partner events land on their own bus and rules are bus-scoped, so a correct pattern on the wrong bus matches nothing and reports nothing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AWS integration doesn&#8217;t usually break loudly. It drifts. A relay stops, a mapping goes stale, a delete never propagates, and the two systems keep answering questions confidently with different numbers. The services themselves are well built and the setup is genuinely not hard once you know the region goes in uppercase and the bus has to be associated first.</p>



<p class="wp-block-paragraph">So pick the path by direction and shape, wire the failure modes you&#8217;ve accepted into alerts, and build the nightly count comparison before you build anything clever. That one job is worth more than any amount of pipeline polish, because it&#8217;s the only thing in the architecture that can tell you the whole thing is still true.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Salesforce and AWS setup?</h2>



<p class="wp-block-paragraph">I work with teams on the infrastructure side of CRM integrations, usually where the data is disagreeing and nobody can prove why. Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing an existing Salesforce to AWS pipeline and finding where records are being lost between the two.</li>

<li>Standing up Event Relay into EventBridge end to end, including the partner bus association and the monitoring that proves it&#8217;s live.</li>

<li>Designing the AppFlow schedule and object set so it doesn&#8217;t eat the org&#8217;s API allocation.</li>

<li>Building the reconciliation and delete-propagation jobs, with alerting in CloudWatch, Grafana, or whatever you already run.</li>

<li>Writing the write-back path properly: Bulk API jobs, idempotent upserts, dead letter queues, and credential rotation through Secrets Manager.</li>

<li>Working out whether Private Connect is worth the licence in your specific environment, or whether TLS and tighter scoping gets you there.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a flow that&#8217;s failing, a relay stuck in an error state, or two record counts that won&#8217;t agree, send me the config or the error and I&#8217;ll tell you what I&#8217;d check first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</title>
		<link>https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Delta Token]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[ODP]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[SAP]]></category>
		<category><![CDATA[SAP OData]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPC]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=213</guid>

					<description><![CDATA[<p>The AWS Glue SAP OData connector demos beautifully and then drifts. A practical look at delta token invalidation, deletes that never delete, partitioning limits on ODP entities, and the licensing line between ODP-RFC and ODP-OData.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/">SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The Glue job had been green for weeks. Every run finished, every run wrote Parquet, nobody had a reason to look at it. Then someone from finance mentioned that a report was short by a few hundred rows, and the reconciliation that followed turned up the real problem: the job had not actually been reading changes for most of that time. It had been reading nothing, successfully.</p>



<p class="wp-block-paragraph">That is the shape of most trouble with SAP data integration with AWS Glue. The connector does not usually crash. It returns an empty or partial result set, the job exits zero, and your data lake quietly diverges from the source system. By the time anyone notices, you have no idea which run was the last correct one.</p>



<p class="wp-block-paragraph">This post covers the parts that bite after the proof of concept: how delta tokens actually behave, why deleted records keep showing up in your reports, where partitioning stops working, and the licensing distinction between two things that both get called &#8220;ODP&#8221; and are not the same thing at all. There is a troubleshooting section at the end for when a job is already broken.</p>



<h2 class="wp-block-heading">First, get the ODP terminology right, because licensing depends on it</h2>



<p class="wp-block-paragraph">Operational Data Provisioning is SAP&#8217;s framework for exposing extraction interfaces in a delta-aware way. A provider (a BW extractor, an ABAP CDS view, SLT, a HANA information view) writes changes into an Operational Delta Queue, and subscribers pull from that queue. This part is uncontroversial.</p>



<p class="wp-block-paragraph">What matters is that there are two ways to talk to ODP, and SAP treats them very differently. SAP Note 3255746 restricts third-party and customer applications from using the RFC modules of the ODP Data Replication API against ABAP sources. That note has been sharpened over several revisions, from &#8220;unsupported&#8221; to &#8220;not permitted&#8221;, and SAP has stated it reserves the right to introduce technical measures and audit unpermitted use. A number of established extraction tools were built directly on ODP-RFC and have had to move.</p>



<p class="wp-block-paragraph">The AWS Glue SAP OData connector takes the other path. It talks OData over HTTPS to an activated SAP Gateway service, and uses the ODP framework through that OData layer rather than through RFC. That is the route SAP points customers at. If you are being asked in an architecture review whether your pipeline is on the wrong side of Note 3255746, this distinction is the whole answer, and it is worth having the note number written down before the meeting.</p>



<p class="wp-block-paragraph">Two honest caveats. First, notes get revised, so read the current version rather than a vendor blog summarising it, including this one. Second, &#8220;permitted&#8221; is not the same as &#8220;free&#8221;: you still need the OData service activated, a service user with the right authorisations, and a Basis team willing to own the queue growth on their side.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the delta token is shared state you do not own</h2>



<p class="wp-block-paragraph">This is the one that produces silent drift, and it is the reason the finance report was short.</p>



<p class="wp-block-paragraph">When you read an ODP-enabled entity with change data capture switched on, the connector asks SAP to initialise a delta queue subscription. SAP hands back a delta token. That token is your bookmark. On the next run you pass it back, and you get only what changed since.</p>



<pre class="wp-block-code"><code>sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "ENABLE_CDC": "true"
    },
    transformation_ctx="initial_load"
)</code></pre>



<p class="wp-block-paragraph"><code>ENABLE_CDC</code> defaults to false. Leave it off and you get a full load every single run, which is expensive but at least obvious. Turn it on and the token appears in the <code>DELTA_TOKEN</code> column, populated only on the last row of the frame. That last row is not a business record. It is a carrier for the token, and if you write your frame straight to S3 without stripping it you have just added a junk row to your table.</p>



<p class="wp-block-paragraph">The subsequent run passes the token back:</p>



<pre class="wp-block-code"><code>sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "DELTA_TOKEN": last_known_token
    },
    transformation_ctx="incremental_load"
)</code></pre>



<h3 class="wp-block-heading">Why this goes wrong</h3>



<p class="wp-block-paragraph">A delta token is bound to a service collection <em>and a user</em>. If a new initialisation runs with <code>ENABLE_CDC</code> set true for that same collection and user, SAP invalidates every token previously issued from the earlier initialisation. Not &#8220;expires eventually&#8221;. Invalidates, immediately.</p>



<p class="wp-block-paragraph">So picture two Glue jobs sharing one service account, both reading the same extractor. Someone redeploys job B with the CDC flag still set from testing. Job B re-initialises. Job A&#8217;s stored token is now dead, and the next time job A runs you get:</p>



<pre class="wp-block-code"><code>Could not open data access via extraction API RODPS_REPL_ODP_OPEN</code></pre>



<p class="wp-block-paragraph">An error is the good outcome. The bad outcome is code that catches the exception, falls back to a full load, and writes it over the top of an incremental table with no dedupe. Or code that re-initialises on failure, which fixes job A and breaks job B, and now the two jobs take turns invalidating each other forever while both keep exiting zero.</p>



<p class="wp-block-paragraph">Practical rules that come out of this:</p>



<ul class="wp-block-list">
<li>One SAP service user per consuming pipeline. Sharing a user is what couples independent jobs together.</li>

<li>Persist the token somewhere durable and versioned. DynamoDB, Parameter Store, S3 with versioning enabled, all fine. Job bookmarks are not a substitute, because the token is source state, not read position.</li>

<li>Never auto re-initialise on error. Fail loudly, page someone, and make the reinitialisation a deliberate human action with a documented backfill.</li>

<li>Alert on zero-change runs. A pipeline that returns no changes for several consecutive runs against a busy table is telling you something, and it is rarely &#8220;business is quiet&#8221;.</li>
</ul>



<h3 class="wp-block-heading">Retention on the SAP side is a hard deadline</h3>



<p class="wp-block-paragraph">The delta queue is not infinite. In transaction <code>ODQMON</code> the Basis team configures retention, and a reorganisation job cleans up data past that window. If your pipeline is down longer than the retention period, the changes you missed are gone from the queue and no token will bring them back. You are into a full reload and a reconciliation.</p>



<p class="wp-block-paragraph">Two things to find out before you go live: what the retention setting is, and whether the cleanup job is actually scheduled. Then make sure your pipeline&#8217;s alerting escalates well inside that window. A weekend outage on a queue with a short retention is a very expensive Monday.</p>



<p class="wp-block-paragraph">Related: if anyone has been testing extraction with <code>RODPS_REPL_TEST</code>, that program registers as a real additional subscriber rather than simulating one. Because the queue is shared and data is only cleaned once every subscriber has collected it, forgotten test subscriptions hold delta data indefinitely and the underlying queue tables grow. Ask whoever ran the tests to reset those subscriptions in ODQMON.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: deletes arrive as a column, not as a deletion</h2>



<p class="wp-block-paragraph">CDC extraction from ODP entities returns a couple of extra fields alongside your business columns. <code>DML_STATUS</code> carries <code>UPDATED</code> for inserts and updates, and <code>DELETED</code> for records removed at source. <code>GLUE_FETCH_SQ</code> is a per-record sequence derived from an epoch timestamp, in receipt order, and it is your tie-breaker when the same key changes twice inside one delta.</p>



<p class="wp-block-paragraph">Nothing acts on those fields for you. If your job appends the frame to S3 and calls it done, a deleted sales order is now a row in your lake with <code>DML_STATUS = DELETED</code> and every other column populated exactly as before. Any query that does not filter on that column counts it. Cancelled orders keep appearing in revenue. Closed accounts stay open. The pipeline is green, the numbers are wrong, and the discrepancy grows slowly enough that nobody catches it early.</p>



<p class="wp-block-paragraph">You have to apply the changes rather than accumulate them. Broadly two options:</p>



<ol class="wp-block-list">
<li><strong>Merge into a table format that supports it.</strong> Apache Iceberg, which Glue supports natively, gives you a real merge with delete handling. Land the raw delta, then merge on the business key with <code>GLUE_FETCH_SQ</code> deciding the winner within a batch, deleting where the status says deleted.</li>

<li><strong>Keep an append-only history and build a current-state view.</strong> Cheaper to write, and it preserves an audit trail. It also means every downstream consumer has to use the view, and someone eventually will not.</li>
</ol>



<p class="wp-block-paragraph">I reach for Iceberg first on anything that feeds reporting, because the correctness lives in the table rather than in the discipline of whoever writes the next query. Append-only is the right call when you genuinely need history, or when the consumer is a downstream warehouse like Amazon Redshift that will do its own merge anyway.</p>



<p class="wp-block-paragraph">Worth knowing: not every ODP provider emits deletes reliably. Some older extractors express them as reversal records rather than a delete flag. Confirm the behaviour for your specific entity with the functional owner before you build merge logic on an assumption.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: pagination and partitioning do not compose</h2>



<p class="wp-block-paragraph">The connector&#8217;s default page size is 50,000 records, and you can raise it up to 500,000. Anything above that is ignored and the maximum is used instead. Raising the page size cuts the number of round trips, which is usually the dominant cost on a wide table across a slow link.</p>



<p class="wp-block-paragraph">The catch is what your SAP Gateway will tolerate. A large page means a large response the ABAP stack has to assemble and hold, and Gateway or an intervening load balancer may cut it off before it completes. Symptoms are timeouts, connection resets, or a job that appears to paginate without ever finishing. Tune the page size with your Basis team, not in isolation, and treat it as a negotiation between round trips and response size rather than a value you set once.</p>



<p class="wp-block-paragraph">On partitioning, there is a specific documented limit that catches people out: ODP entities are not compatible with record-based partitioning, because pagination is already handled through skip and delta tokens. For record-based partitioning, the default concurrency is forced to null regardless of what you asked for. So the obvious lever for making a slow ODP extraction faster is not available.</p>



<p class="wp-block-paragraph">What you can still do on non-ODP OData entities is field-based partitioning, splitting a read across a bounded range:</p>



<ul class="wp-block-list">
<li><code>PARTITION_FIELD</code> is the field to split on, typically a date.</li>

<li><code>LOWER_BOUND</code> is inclusive, <code>UPPER_BOUND</code> is exclusive. Getting that backwards is how you duplicate or drop a boundary day.</li>

<li><code>NUM_PARTITIONS</code> defaults to 1, so leaving it out means you did not partition at all.</li>

<li>Where a limit and a partition are both applied, the limit wins. That is the usual cause of &#8220;I partitioned it and got fewer rows&#8221;.</li>
</ul>



<p class="wp-block-paragraph">For non-ODP entities you can also do timestamp-based incremental reads with <code>filteringExpression</code>, pointing at whatever last-changed field the entity exposes. Note the casing: that option is camelCase while most of the others are upper snake case, which trips people up when a filter appears to be ignored.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family four: connection, auth and network</h2>



<p class="wp-block-paragraph">These fail early and loudly, which makes them the easy ones. The trap is that they fail during setup, when you have three people on a call and someone suggests loosening something to move on.</p>



<p class="wp-block-paragraph">The connector supports two authentication types: CUSTOM, which is SAP basic authentication, and OAUTH2. The custom secret in AWS Secrets Manager takes this shape:</p>



<pre class="wp-block-code"><code>{
  "basicAuthUsername": "&lt;your SAP username&gt;",
  "basicAuthPassword": "&lt;your SAP password&gt;",
  "basicAuthDisableSSO": "&lt;True/False&gt;",
  "customAuthenticationType": "CustomBasicAuth"
}</code></pre>



<p class="wp-block-paragraph">For OAuth 2.0 the client secret goes into Secrets Manager under <code>USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET</code>, and the client ID, authorisation code URL, token URL and scopes are supplied on the connection itself.</p>



<p class="wp-block-paragraph">Basic auth is genuinely easier for a scheduled pipeline: no interactive redirect, no refresh token to keep alive, and rotation is a Secrets Manager operation. OAuth gets you short-lived tokens and revocation you can actually reason about, which is what a security review will want. Both are defensible. What is not defensible is one shared service user across every pipeline, for the delta token reasons above as much as the audit ones.</p>



<p class="wp-block-paragraph">A few things that reliably waste a morning:</p>



<ul class="wp-block-list">
<li>The service path on the connection is the <em>catalog</em> service path, not the path to your object. The entity path goes in <code>ENTITY_NAME</code> on the job.</li>

<li>The OData service has to be activated in the SAP system. Until it is, nothing you do on the AWS side matters.</li>

<li>For a non-VPC connection, the application host URL must be reachable from the public internet. If your SAP system sits in a private network, configure VPC connectivity properly rather than opening it up.</li>

<li>The Glue role needs Secrets Manager read plus the EC2 network interface permissions when a VPC is involved. A missing <code>ec2:CreateNetworkInterface</code> shows up as a connection failure that looks like a network problem.</li>

<li>Private certificate authorities need the chain in a form Glue accepts. If you are wiring this up yourself, expect it to take a pass or two.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Choosing your path: Glue jobs, zero-ETL, or something else</h2>



<p class="wp-block-paragraph">The same connector is reachable through more than one product surface, and the choice is really about how much state you want to own.</p>



<ul class="wp-block-list">
<li><strong>Glue Studio or script jobs.</strong> You control page size, filters, output format, transformation and scheduling, and you own delta token persistence. Pick this when you need real transformation in flight, or when the merge logic has to live next to the extraction.</li>

<li><strong>Glue zero-ETL integrations.</strong> AWS manages token state internally, runs the initial full load and then the incremental cycle, and preserves the token on failure so retries do not lose data. Less code, less control. Worth noting that once an incremental update field is chosen for a connection and entity, a later integration on the same pair cannot pick a different one.</li>

<li><strong>Amazon AppFlow.</strong> Simpler, flow-shaped, well suited to modest volumes and quick prototypes. Less appropriate once you need Spark-scale processing.</li>

<li><strong>Commercial replication tools.</strong> Fivetran, Airbyte and SAP-specialist vendors all play here. They cost money and add a vendor, and in exchange they absorb exactly the state management this post is about. If your team has no SAP depth, that trade is often correct.</li>
</ul>



<p class="wp-block-paragraph">On cost, Glue bills for the resources a job consumes for as long as it runs, so the levers are the same as any Spark workload: fewer round trips, less data pulled, fewer idle executors, and not re-reading history you already have. Rates change and vary by region, so model it against current pricing rather than a figure from a blog post. If your incremental run is pulling the same volume as your initial load, the delta mechanism is not working and the bill is the symptom.</p>



<p class="wp-block-paragraph">Also worth writing back. The connector supports write operations to SAP OData, using <code>INSERT</code> and <code>UPDATE</code>, with <code>ID_FIELD_NAMES</code> required on update to identify the external ID field. Useful for enrichment loops, and something to treat with a good deal more caution than a read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">&#8220;Could not open data access via extraction API RODPS_REPL_ODP_OPEN&#8221;</h3>



<p class="wp-block-paragraph">Your token is invalid or expired. Find out why before you re-initialise, because re-initialising destroys the evidence. Check ODQMON for a recent initialisation on that collection and user, check whether another job or a test program shares the credentials, and check whether the request you wanted has aged out of retention. Then re-initialise deliberately and reconcile the gap.</p>



<h3 class="wp-block-heading">The job succeeds but returns no rows</h3>



<p class="wp-block-paragraph">Most often a stale or wrong token, a filter predicate that matches nothing, or CDC never having been initialised so the connector has nothing to hand back. Run once without <code>ENABLE_CDC</code> against a small filtered slice and confirm you get data at all. That separates &#8220;no changes&#8221; from &#8220;not reading&#8221;.</p>



<h3 class="wp-block-heading">Pagination never terminates</h3>



<p class="wp-block-paragraph">Usually the SAP side enforcing its own page limit under your requested size, so the connector keeps requesting more. Reduce the page size, verify the filter is actually being applied server side rather than after retrieval, and have Basis confirm what Gateway is capping.</p>



<h3 class="wp-block-heading">Row counts drift over time</h3>



<p class="wp-block-paragraph">Almost always unhandled <code>DML_STATUS = DELETED</code> records, or a token invalidation that silently turned an incremental into a partial. Count distinct business keys in your target against a direct count in SAP for a bounded date range. Make that comparison a scheduled check, not something you run when someone complains.</p>



<h3 class="wp-block-heading">Duplicates after a rerun</h3>



<p class="wp-block-paragraph">A delta was consumed, the write failed, and the token advanced anyway. Store the token only after the write commits, never before. If you are appending rather than merging, add a dedupe on business key ordered by <code>GLUE_FETCH_SQ</code>.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Sharing one SAP service user across multiple pipelines, which couples their delta tokens together.</li>

<li>Automatic re-initialisation in an exception handler. It converts a loud failure into permanent silent drift.</li>

<li>Writing the token-carrier row into the target table as if it were data.</li>

<li>Appending CDC output and ignoring <code>DML_STATUS</code>.</li>

<li>Persisting the token before the write succeeds.</li>

<li>Assuming an entity is ODP-enabled because it looks analytical. Confirm it, because the incremental mechanism differs.</li>

<li>Leaving test subscriptions from <code>RODPS_REPL_TEST</code> registered against the queue.</li>

<li>Building the pipeline without ever asking Basis what the ODQ retention window is.</li>
</ul>



<h2 class="wp-block-heading">Best practices for SAP data integration with AWS Glue</h2>



<ul class="wp-block-list">
<li>Treat the delta token as production state. Versioned storage, a documented recovery procedure, and a written-down answer to &#8220;how do we backfill&#8221;.</li>

<li>One service user per pipeline, one connection per environment. Isolation is the cheapest fix for the worst failure.</li>

<li>Land raw, merge second. Keep the untouched connector output so you can replay a merge without going back to SAP.</li>

<li>Alert on absence, not just errors. Zero rows for several consecutive runs against a busy entity should page someone. Push Glue metrics into CloudWatch and forward them wherever you already look, whether that is Grafana Cloud, Datadog or your own Prometheus.</li>

<li>Reconcile on a schedule against SAP for a bounded window, and record the result. Drift you can see is a bug; drift you cannot see is an incident.</li>

<li>Start with a filtered slice and a small page size while you are building. Get it correct, then tune throughput.</li>

<li>Write down what each field means while the functional owner is still on the call. SAP field names do not explain themselves later.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does the AWS Glue SAP OData connector fall under SAP Note 3255746?</h3>



<p class="wp-block-paragraph">The note restricts the RFC modules of the ODP Data Replication API. The Glue connector uses OData over HTTPS against an activated Gateway service, which is the interface SAP directs customers toward. Read the current revision of the note yourself and get it confirmed with your SAP account team before you rely on any summary, including this one.</p>



<h3 class="wp-block-heading">Where should I store the delta token?</h3>



<p class="wp-block-paragraph">Anywhere durable and versioned that your job can write to transactionally: DynamoDB, Parameter Store, or S3 with versioning. What matters more than the store is the ordering. Write it only after the target write commits, and keep the previous value so you can roll back after a bad run.</p>



<h3 class="wp-block-heading">Can I run two Glue jobs against the same SAP entity?</h3>



<p class="wp-block-paragraph">Yes, but give them separate SAP service users. A delta token is scoped to the service collection and the user, so two jobs on one user will invalidate each other&#8217;s tokens whenever either re-initialises.</p>



<h3 class="wp-block-heading">Why do deleted SAP records still appear in my data lake?</h3>



<p class="wp-block-paragraph">Because the connector reports them rather than applying them. Deletes come back as rows with <code>DML_STATUS</code> set to <code>DELETED</code>. If your pipeline appends without a merge, those rows sit in the target looking like live data. Merge into Iceberg, or filter through a current-state view that every consumer actually uses.</p>



<h3 class="wp-block-heading">What page size should I use?</h3>



<p class="wp-block-paragraph">The default is 50,000 and the ceiling is 500,000. Larger pages mean fewer round trips but bigger responses for SAP to assemble, and Gateway or a load balancer may cut them short. Start at the default, raise it while watching for timeouts, and agree the ceiling with your Basis team.</p>



<h3 class="wp-block-heading">Can I partition an ODP extraction for speed?</h3>



<p class="wp-block-paragraph">Not with record-based partitioning. ODP entities handle pagination through skip and delta tokens, so concurrency is forced to null there regardless of your input. Field-based partitioning on a bounded range is available for non-ODP OData entities. For ODP, tune page size and narrow the entity instead.</p>



<h3 class="wp-block-heading">Can Glue write data back into SAP?</h3>



<p class="wp-block-paragraph">Yes. The SAP OData connector supports insert and update operations, with the external ID field named through <code>ID_FIELD_NAMES</code> on updates. Test that against a sandbox client with a functional owner watching, because a bad write into an ERP system is a very different class of incident to a bad read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">SAP data integration with AWS Glue is not hard because the connector is difficult to configure. Configuration takes an afternoon. It is hard because the correctness of your pipeline depends on a token held in a queue on a system your team does not administer, subject to a retention policy you may not have been told about, and shared with every other subscriber using the same credentials.</p>



<p class="wp-block-paragraph">Design for that. Isolate the users, persist the token deliberately, apply deletes rather than collecting them, and alert on silence as loudly as you alert on errors. Do that and the connector is a good piece of engineering that will run for a long time without your attention. Skip it and you get a job that stays green while the numbers quietly stop being true.</p>



<h2 class="wp-block-heading">Need a hand with an SAP to AWS pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. If you are standing one up, or you have one that has started drifting, I can help with:</p>



<ul class="wp-block-list">
<li>Designing delta token state management that survives failed runs, redeploys and concurrent pipelines.</li>

<li>Building merge logic that applies <code>DML_STATUS</code> deletes correctly into Iceberg or Redshift instead of accumulating ghost rows.</li>

<li>Diagnosing <code>RODPS_REPL_ODP_OPEN</code> failures, silent zero-row runs and row count drift, and working out how far back the damage goes.</li>

<li>Setting up SAP OData connections properly: Secrets Manager, least-privilege IAM, VPC connectivity and private certificate authorities.</li>

<li>Tuning page size, filters and job sizing so incremental runs cost what an incremental run should cost.</li>

<li>Adding reconciliation and absence alerting so drift shows up in a dashboard rather than in a finance report.</li>
</ul>



<p class="wp-block-paragraph">Send me a job log, a connection config, or the row counts that stopped matching, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/">SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=215</guid>

					<description><![CDATA[<p>A GraphQL source can hand you a 200 OK, a populated data block, and a quietly broken column in the same response. Here is how to build a GraphQL data ingestion pipeline on AWS that catches partial errors, respects cost-based rate limits, resumes cleanly from a cursor, and notices when the schema moves under you.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The row counts looked fine. That was the problem.</p>



<p class="wp-block-paragraph">A nightly sync from a partner&#8217;s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.</p>



<p class="wp-block-paragraph">The API had started returning a field-level error on that one field. Status code 200. A populated <code>data</code> block. An <code>errors</code> array sitting right next to it. The fetcher parsed <code>data</code>, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.</p>



<p class="wp-block-paragraph">This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.</p>



<h2 class="wp-block-heading">Why GraphQL ingestion breaks the habits REST taught you</h2>



<p class="wp-block-paragraph">Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server&#8217;s business, and rate limits are counted in requests. GraphQL invalidates all three.</p>



<ul class="wp-block-list">
<li><strong>The status code is not the contract.</strong> The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.</li>

<li><strong>You own the response shape.</strong> The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.</li>

<li><strong>Cost is query-shaped, not endpoint-shaped.</strong> One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.</li>
</ul>



<p class="wp-block-paragraph">Each of those becomes a distinct failure family. Let&#8217;s take them in the order they bite.</p>



<h2 class="wp-block-heading">Failure one: the 200 OK that quietly ate a column</h2>



<p class="wp-block-paragraph">This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.</p>



<ul class="wp-block-list">
<li><strong>Request errors</strong> happen before execution starts: syntax errors, validation failures, bad variable coercion. No <code>data</code> key is returned at all. Nothing executed.</li>

<li><strong>Field errors</strong> happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get <code>data</code> and <code>errors</code> together.</li>
</ul>



<p class="wp-block-paragraph">Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL&#8217;s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.</p>



<p class="wp-block-paragraph">The fix is to make the response body the contract. Classify before you commit anything:</p>



<pre class="wp-block-code"><code>import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]</code></pre>



<p class="wp-block-paragraph">Two things worth explaining. The <code>Accept</code> header asks for <code>application/graphql-response+json</code>, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don&#8217;t support it fall back to <code>application/json</code>, so including both is free.</p>



<p class="wp-block-paragraph">The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.</p>



<h3 class="wp-block-heading">Deciding what &#8220;partial&#8221; means for your data</h3>



<p class="wp-block-paragraph">Failing the whole page on any error is the safe default and the one I reach for first. But it&#8217;s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.</p>



<p class="wp-block-paragraph">The workable middle ground is to classify by path. Every error object carries a <code>path</code> array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.</p>



<h2 class="wp-block-heading">Failure two: rate limits that never send a 429</h2>



<p class="wp-block-paragraph">Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.</p>



<p class="wp-block-paragraph">GitHub&#8217;s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and <code>x-ratelimit-remaining</code> set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.</p>



<p class="wp-block-paragraph">The other half of the problem is that request counting is the wrong unit. Shopify&#8217;s GraphQL Admin API and several others price queries by <em>calculated cost</em>, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case <em>requested</em> cost, executes, then refunds the difference between requested and <em>actual</em> cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.</p>



<p class="wp-block-paragraph">The good news is that these APIs tell you the state of the bucket on every response, under the <code>extensions</code> key:</p>



<pre class="wp-block-code"><code>{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant&#8217;s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against <code>currentlyAvailable</code>, and if you&#8217;re short, sleep for the shortfall divided by <code>restoreRate</code>.</p>



<p class="wp-block-paragraph">On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:</p>



<pre class="wp-block-code"><code>query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}</code></pre>



<p class="wp-block-paragraph">Two more things about concurrency. First, if you fan out across Lambda invocations, each worker&#8217;s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it&#8217;s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can&#8217;t go wrong.</p>



<p class="wp-block-paragraph">Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.</p>



<h2 class="wp-block-heading">Failure three: cursors that don&#8217;t survive a restart</h2>



<p class="wp-block-paragraph">Most production GraphQL schemas follow the Relay cursor connections convention: <code>first</code> and <code>after</code> arguments, a <code>pageInfo</code> object with <code>hasNextPage</code> and <code>endCursor</code>, and edges carrying opaque per-item cursors.</p>



<pre class="wp-block-code"><code>query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.</p>



<p class="wp-block-paragraph">The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.</p>



<p class="wp-block-paragraph">Three habits make restarts safe:</p>



<ol class="wp-block-list">
<li><strong>Checkpoint after the write, not after the fetch.</strong> A cursor recorded before the page lands in S3 is a promise you can&#8217;t keep.</li>

<li><strong>Store a high watermark alongside the cursor.</strong> The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.</li>

<li><strong>Overlap the watermark deliberately.</strong> Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.</li>
</ol>



<p class="wp-block-paragraph">A DynamoDB item per source stream is enough state for all of this:</p>



<pre class="wp-block-code"><code>{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "&lt;updatedAt of newest committed record&gt;",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}</code></pre>



<p class="wp-block-paragraph">Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest <code>updatedAt</code>. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you&#8217;ll have traded silent loss for silent inflation.</p>



<h2 class="wp-block-heading">Failure four: the schema moved and nobody told you</h2>



<p class="wp-block-paragraph">GraphQL&#8217;s deprecation story is genuinely better than REST&#8217;s. Fields carry a <code>@deprecated</code> directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.</p>



<p class="wp-block-paragraph">The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.</p>



<p class="wp-block-paragraph">Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.</p>



<p class="wp-block-paragraph">The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring the GraphQL data ingestion pipeline on AWS</h2>



<p class="wp-block-paragraph">With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.</p>



<h3 class="wp-block-heading">Where the fetch loop runs</h3>



<p class="wp-block-paragraph">Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:</p>



<pre class="wp-block-code"><code>"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}</code></pre>



<p class="wp-block-paragraph">The execution role needs four permissions: <code>states:InvokeHTTPEndpoint</code>, <code>events:RetrieveConnectionCredentials</code>, and both <code>secretsmanager:GetSecretValue</code> and <code>secretsmanager:DescribeSecret</code> scoped to the EventBridge-managed secret.</p>



<p class="wp-block-paragraph">It&#8217;s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.</p>



<p class="wp-block-paragraph">The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow&#8217;s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.</p>



<p class="wp-block-paragraph">Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda&#8217;s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.</p>



<p class="wp-block-paragraph">One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.</p>



<h3 class="wp-block-heading">The landing zone</h3>



<p class="wp-block-paragraph">Land raw responses first, unmodified, including the <code>errors</code> and <code>extensions</code> keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.</p>



<ul class="wp-block-list">
<li><strong>Partition by ingest time, not event time.</strong> Event-time partitioning requires reading the record before you know where to put it, which means you can&#8217;t write until you&#8217;ve parsed. Ingest-time keys let the raw write be dumb and fast.</li>

<li><strong>Put the run identifier in the key.</strong> When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.</li>

<li><strong>Define Glue tables explicitly rather than crawling.</strong> You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.</li>

<li><strong>Convert to columnar in the curated layer.</strong> Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.</li>
</ul>



<p class="wp-block-paragraph">If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don&#8217;t need.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Runs succeed but a column is all null.</strong> Field errors. Pull a raw response from the landing zone and look at the <code>errors</code> array. The <code>path</code> tells you exactly which field.</li>

<li><strong>Runs succeed but ingest zero rows.</strong> Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.</li>

<li><strong>Intermittent failures partway through pagination.</strong> Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.</li>

<li><strong>Query fails immediately with no <code>data</code> key.</strong> Request error. Validate against current introspection; something in the schema moved.</li>

<li><strong>Throttled despite staying under the documented limit.</strong> Either concurrent workers are sharing a bucket they each think they own, or you&#8217;re being charged requested cost on near-empty pages.</li>

<li><strong>HTTP Task fails on large pages.</strong> The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.</li>

<li><strong>Row counts drift upward over time.</strong> Overlap without deduplication. Check that the curated layer actually resolves on the primary key.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a 200 as success and never reading the body.</li>

<li>Retrying request errors. They are deterministic; retrying just spends budget to fail identically.</li>

<li>Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.</li>

<li>Checkpointing the cursor before the data is durable.</li>

<li>Requesting the maximum page size on every call while being charged the worst-case cost for it.</li>

<li>Selecting entire objects because it was easier during development.</li>

<li>Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.</li>

<li>Overlapping restart windows with no downstream deduplication.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify every response into clean, partial, or request-failed before anything downstream sees it.</li>

<li>Persist raw responses with <code>errors</code> and <code>extensions</code> intact, then parse from the raw layer.</li>

<li>Alarm on row-count deltas and partial-error rate, not just on execution status.</li>

<li>Read the throttle state the API hands you and back off before you hit the wall, not after.</li>

<li>Keep both a cursor and a watermark, and know which one is authoritative on restart.</li>

<li>Diff the schema on a schedule so deprecations reach you before they reach your data.</li>

<li>Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.</li>

<li>Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you&#8217;re correlating child executions across a fan-out.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Step Functions call a GraphQL API without a Lambda function?</h3>



<p class="wp-block-paragraph">Yes. An HTTP Task using the <code>arn:aws:states:::http:invoke</code> resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.</p>



<h3 class="wp-block-heading">How do I detect GraphQL errors when the status code is always 200?</h3>



<p class="wp-block-paragraph">Inspect the body. If there&#8217;s no <code>data</code> key, nothing executed and you have a request error. If <code>data</code> is present alongside a non-empty <code>errors</code> array, you have a partial response and the data is incomplete. Only a response with <code>data</code> and no errors is unambiguously clean.</p>



<h3 class="wp-block-heading">Does AWS AppSync fit into this?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it&#8217;s what you&#8217;d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else&#8217;s GraphQL API, AppSync isn&#8217;t part of the picture.</p>



<h3 class="wp-block-heading">Should I store raw GraphQL responses or parsed records in S3?</h3>



<p class="wp-block-paragraph">Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they&#8217;re the only way to answer &#8220;was this field null in the source, or did my parser drop it?&#8221; weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.</p>



<h3 class="wp-block-heading">How do I run an incremental sync from a GraphQL API?</h3>



<p class="wp-block-paragraph">Find the schema&#8217;s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you&#8217;re limited to full pulls or webhook-driven change capture.</p>



<h3 class="wp-block-heading">Why am I throttled when I&#8217;m making very few requests?</h3>



<p class="wp-block-paragraph">Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.</p>



<h3 class="wp-block-heading">Is GraphQL a worse ingestion source than REST?</h3>



<p class="wp-block-paragraph">Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.</p>



<p class="wp-block-paragraph">So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with a GraphQL ingestion pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. Specifically:</p>



<ul class="wp-block-list">
<li>Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected</li>

<li>Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts</li>

<li>Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination</li>

<li>Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap</li>

<li>Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard</li>

<li>Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green</li>
</ul>



<p class="wp-block-paragraph">Send me a run log, a state machine definition, or a raw response with an <code>errors</code> array in it, and I&#8217;ll tell you what it&#8217;s costing you.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</title>
		<link>https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 16 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Zendesk]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=223</guid>

					<description><![CDATA[<p>AWS Glue zero-ETL replicates seven Zendesk entities, but only three of them ever remove a row. Here is how that gap quietly skews CSAT and knowledge base counts, plus the three IAM layers to wire, the two settings you cannot change after creation, and the CloudWatch metrics that make drift visible before someone spots it in a meeting.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/">Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The support lead pings you on a Tuesday. The CSAT figure on the exec dashboard is higher than the one in Zendesk Explore, and somebody noticed during the meeting. Nothing failed. The Glue integration is green. CloudWatch shows ingestion succeeding on every interval. And yet the warehouse and the source disagree, and have quietly disagreed for weeks.</p>



<p class="wp-block-paragraph">That divergence is not a bug. It is documented behaviour, sitting in one table in the AWS docs that almost everyone scrolls past. This post covers what Zendesk data integration with AWS Glue zero-ETL actually replicates, the entities where deletes silently never propagate, the configuration choices you cannot undo after creation, how to wire the three IAM roles involved, and how to monitor the thing so drift shows up as an alarm instead of a meeting.</p>



<h2 class="wp-block-heading">What Zendesk AWS Glue zero-ETL actually gives you</h2>



<p class="wp-block-paragraph">Zero-ETL is AWS&#8217;s name for managed replication. You create a Glue connection to Zendesk, pick the entities you want, pick a target, and Glue handles the initial snapshot, the schema mapping, and ongoing change data capture. No Spark job. No bookmark logic. No pagination code hammering the Zendesk API and no retry handling for it.</p>



<p class="wp-block-paragraph">Supported targets are a general purpose Amazon S3 bucket through the lakehouse architecture of Amazon SageMaker, S3 Tables through the same lakehouse, Redshift Managed Storage, or an Amazon Redshift data warehouse directly. Data lands as Apache Iceberg, which is what makes row-level updates and deletes viable on object storage at all.</p>



<p class="wp-block-paragraph">What you give up is control. No transform step, no server-side filter, no column pruning at ingest. You get the entity as the connector sees it and you shape it downstream. For most support analytics that trade is fine. It stops being fine the moment your compliance team asks why deleted records are still queryable.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The delete gap: four of seven entities never remove a row</h2>



<p class="wp-block-paragraph">AWS Glue zero-ETL supports seven Zendesk entities. Only three of them replicate deletes.</p>



<ul class="wp-block-list">
<li><strong>tickets</strong> &#8211; create, update and delete all propagate</li>



<li><strong>users</strong> &#8211; create, update and delete all propagate</li>



<li><strong>organizations</strong> &#8211; create, update and delete all propagate</li>



<li><strong>satisfaction-rating</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>articles</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>calls</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>legs</strong> (call legs) &#8211; create and update only, <strong>no delete</strong></li>
</ul>



<p class="wp-block-paragraph">When a satisfaction rating disappears in Zendesk, the row stays in your lakehouse. Permanently. Nothing errors, <code>IngestionSucceeded</code> fires as usual, and the row count in that table only ever goes up.</p>



<p class="wp-block-paragraph">Now think about what your CSAT query does. It averages a score column across a set of rows. If removed ratings never leave the target, your average drifts further from the source every cycle, and the skew is rarely random because retracted or disputed ratings are exactly the ones most likely to be removed. The same mechanism inflates knowledge base article counts, call volume, and any call leg analysis you build. It stays invisible until two dashboards get compared side by side.</p>



<h3 class="wp-block-heading">How to design around it</h3>



<p class="wp-block-paragraph">You cannot make the connector track deletes it does not track. So stop trying, and split the problem instead.</p>



<ol class="wp-block-list">
<li><strong>Run two integrations, not one.</strong> Put tickets, users and organizations in a continuously synced integration. Put the four append-only entities in a second integration you can tear down and rebuild on a cadence that matches how much drift you can tolerate.</li>



<li><strong>Treat the append-only tables as an event log, not as state.</strong> Query them through a view that reconciles against a periodic authoritative count, rather than trusting row presence to mean the record still exists.</li>



<li><strong>Prefer status fields over row existence.</strong> Where an entity exposes a field describing its state, filter on that field downstream. Check which fields your entities actually return before you build on this, because coverage varies by entity.</li>



<li><strong>Reconcile on a schedule.</strong> Pull counts from the Zendesk API for the four entities and compare against the target. Alert on divergence, not on an absolute number.</li>
</ol>



<p class="wp-block-paragraph">Splitting the integration costs you a second set of IAM wiring and a second thing to monitor. It buys you the ability to rebuild the drifting half without touching the half that is correct. That is a good trade.</p>



<h2 class="wp-block-heading">Entity coverage is narrower than the Zendesk API</h2>



<p class="wp-block-paragraph">Seven entities is not the Zendesk API. Ticket comments, ticket metrics, ticket audits, groups, views, macros and SLA policy definitions are not part of the zero-ETL entity set. If your reporting needs first response time, the audit trail behind a status change, or custom field definitions, zero-ETL alone will not get you there.</p>



<p class="wp-block-paragraph">The useful part is that the Glue connection you create is reusable. The same connection backs a normal Glue ETL job, so you can run zero-ETL for the bulk entities and a scheduled Spark job for the ones it does not cover:</p>



<pre class="wp-block-code"><code># Read a Zendesk entity through the same connection
# used by the zero-ETL integration. ENTITY_NAME takes
# the entity name, not the label, e.g. "tickets".
zendesk_read = glueContext.create_dynamic_frame.from_options(
    connection_type="Zendesk",
    connection_options={
        "connectionName": "my-zendesk-connection",
        "ENTITY_NAME": "tickets",
        "API_VERSION": "v2"
    }
)</code></pre>



<p class="wp-block-paragraph">One schema detail worth knowing before you write a single downstream query: the connector converts struct and list types to strings. Zendesk custom fields arrive as arrays of objects, so they land as serialized text, not as a nested column you can address with dot notation. Partitioning is also not supported on the Zendesk source. Plan for a parse step in your silver layer rather than discovering it in a broken dashboard.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Two decisions you cannot change after creation</h2>



<p class="wp-block-paragraph">Most Glue settings are editable. These two are not, and getting them wrong means deleting the integration and starting over.</p>



<h3 class="wp-block-heading">Continuous sync versus on-demand snapshot</h3>



<p class="wp-block-paragraph">By default the integration syncs continuously. Enable the on-demand snapshot setting and it does a single one-time replication with no ongoing CDC. That setting is locked once the integration exists. If you enabled it for a proof of concept and then decided you wanted CDC, you are recreating the integration and re-running the full load.</p>



<h3 class="wp-block-heading">Refresh interval, and why your target choice locks it</h3>



<p class="wp-block-paragraph">The refresh interval controls how often CDC pulls run. It accepts anything from 15 minutes up to 8640 minutes, which is six days. Here is the part that catches people:</p>



<ul class="wp-block-list">
<li>If the target is <strong>Amazon Redshift</strong>, the refresh interval cannot be modified after creation.</li>



<li>For <strong>other targets</strong>, including the SageMaker lakehouse on S3, you can change it later.</li>
</ul>



<p class="wp-block-paragraph">So the target you pick is not only an architecture decision, it is a flexibility decision. If you are not certain what freshness the business actually needs, land in the lakehouse first and query Redshift over it. You keep the ability to tune the interval once you have real usage data instead of a guess made in week one.</p>



<p class="wp-block-paragraph">On cost: AWS does not charge separately for the integration itself. You pay for everything around it, target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. A 15 minute interval on seven entities produces far more small-file churn and log volume than a 6 hour one. Set the interval from the freshness a human will actually act on, not from the smallest number the field accepts.</p>



<h2 class="wp-block-heading">Wiring the three roles</h2>



<p class="wp-block-paragraph">There are three separate permission layers here and they fail in different ways, which is why &#8220;check IAM&#8221; is useless advice on its own.</p>



<ul class="wp-block-list">
<li><strong>The Zendesk OAuth credential.</strong> The connector uses the authorization code grant, so you get redirected to Zendesk to log in and approve. You can rely on the Glue-managed client application and supply only your instance URL, or register your own OAuth app in the Zendesk admin center and provide your own client ID and secret. The resulting access token does not expire, which is convenient and also means a revoked app in Zendesk is something you will only discover through failed ingestions.</li>



<li><strong>The source role.</strong> This is what lets the integration read through the connection. It is a prerequisite for SaaS sources and it is the step most people miss, because creating the connection successfully does not mean the integration can use it.</li>



<li><strong>The target role and catalog policy.</strong> This is what lets the integration write. A misconfigured catalog resource policy puts the integration into <code>NEEDS_ATTENTION</code>, not into a clear error at creation time.</li>
</ul>



<p class="wp-block-paragraph">The source role policy looks like this. Note <code>glue:RefreshOAuth2Tokens</code>, which is the one people leave out and then spend an afternoon debugging:</p>



<pre class="wp-block-code"><code>{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GlueConnections",
            "Effect": "Allow",
            "Action": [
                "glue:GetConnections",
                "glue:GetConnection"
            ],
            "Resource": [
                "arn:aws:glue:*:111122223333:catalog",
                "arn:aws:glue:us-east-1:111122223333:connection/*"
            ]
        },
        {
            "Sid": "GlueActionBasedPermissions",
            "Effect": "Allow",
            "Action": [
                "glue:ListEntities",
                "glue:RefreshOAuth2Tokens"
            ],
            "Resource": ["*"]
        },
        {
            "Sid": "CloudWatchLogging",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": ["*"]
        }
    ]
}</code></pre>



<p class="wp-block-paragraph">The trust policy allows <code>glue.amazonaws.com</code> to assume it. Then you attach the role to the connection, which is a CLI-only step and does not appear in the console flow:</p>



<pre class="wp-block-code"><code>aws glue create-integration-resource-property 
  --resource-arn arn:aws:glue:us-east-1:123456789012:connection/my-zendesk-connection 
  --source-processing-properties "{"RoleArn" : "arn:aws:iam::123456789012:role/zendesk-zetl-source"}" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">If you skip this, the connection tests fine and the integration still cannot read. That mismatch is the single most common reason a first attempt sits in <code>NEEDS_ATTENTION</code> with no obvious cause.</p>



<h2 class="wp-block-heading">Setting up the integration</h2>



<ol class="wp-block-list">
<li>If you are using your own OAuth app, register it in the Zendesk admin center under the API settings and note the client ID and secret. Store the secret in AWS Secrets Manager, one secret per Glue connection.</li>



<li>In Glue Studio, create a connection under Data Connections. Choose Zendesk as the connection type, supply your instance URL and environment, and complete the OAuth redirect.</li>



<li>Create the source role with the policy above and attach it to the connection using <code>create-integration-resource-property</code>.</li>



<li>Prepare the target: the Glue database or S3 Table bucket, the target role, the catalog resource policy, and any Lake Formation grants your account&#8217;s permission model requires.</li>



<li>Create the integration. Select entities, set the refresh interval, and decide continuous sync versus on-demand snapshot. Both of those are locked from here.</li>



<li>Watch the first full load complete before you point anything at the tables. Full load and CDC are separate load types in the metrics, and the numbers look very different.</li>
</ol>



<p class="wp-block-paragraph">Once it works, move it into code. Glue zero-ETL integrations are supported by CloudFormation and the AWS CDK, which is the difference between a thing one person built in the console and a thing your team can redeploy into another account.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Monitoring: the metrics that expose drift</h2>



<p class="wp-block-paragraph">Zero-ETL publishes to the <code>AWS/Glue/ZeroETL</code> CloudWatch namespace, with dimensions for <code>integrationArn</code>, <code>loadType</code> and <code>tableName</code>. The metrics are <code>InsertCount</code>, <code>UpdateCount</code>, <code>DeleteCount</code>, <code>IngestionSucceeded</code>, <code>IngestionFailed</code> and <code>LastSyncTimestamp</code>.</p>



<p class="wp-block-paragraph">The obvious alarm is on failure:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name zendesk-zetl-ingestion-failed 
  --namespace AWS/Glue/ZeroETL 
  --metric-name IngestionFailed 
  --dimensions Name=integrationArn,Value=&lt;your-integration-arn&gt; 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 1 
  --threshold 1 
  --comparison-operator GreaterThanOrEqualToThreshold 
  --treat-missing-data notBreaching</code></pre>



<p class="wp-block-paragraph">That one catches loud failures. It does not catch the quiet one, which is an integration that stops running entirely. For that, alarm on <code>IngestionSucceeded</code> with <code>--treat-missing-data breaching</code> over a window longer than your refresh interval. Absence of success is the signal, not presence of failure.</p>



<p class="wp-block-paragraph">And here is the one specific to this post. Because <code>DeleteCount</code> carries a <code>tableName</code> dimension, you can see the delete gap directly in a graph. Plot <code>DeleteCount</code> per table. Tickets, users and organizations will show a nonzero line. Satisfaction ratings, articles, calls and legs will sit flat at zero forever. That flat line is not a broken metric, it is the behaviour, and having it on a dashboard is the cheapest way to keep the whole team aware of it.</p>



<p class="wp-block-paragraph">Glue also writes a system table into the target database recording the outcome of each full load and CDC run, with per-run record, insert and delete counts. Query that when CloudWatch is not granular enough. If you already run Grafana Cloud or Datadog, pulling the <code>AWS/Glue/ZeroETL</code> namespace in alongside your existing infrastructure dashboards puts pipeline health next to everything else on call already watches.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph">Integrations move through Creating, Active, Modifying, Syncing, Needs attention, Failed and Deleting. What each one means in practice:</p>



<ul class="wp-block-list">
<li><strong>Syncing</strong> means it hit a recoverable error and is re-seeding data. Not an emergency. Let it finish before you touch anything.</li>



<li><strong>Needs attention</strong> means you have to fix something, usually connection credentials, the source role, the target role, or the catalog resource policy. Once you fix it, there is no manual recovery action. Glue retries automatically on an exponential backoff schedule, so the gap between attempts grows over time. If you fixed a policy and nothing happened after two minutes, that is expected. Wait it out.</li>



<li><strong>Failed</strong> is terminal. Delete the integration and recreate it. There is no repair path.</li>
</ul>



<p class="wp-block-paragraph">For anything else, the CloudWatch logs emitted after each full load or CDC run carry the actual root cause. The console status is a summary, and a fairly lossy one. Read the logs first.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Building CSAT, article count or call volume metrics on raw row counts from the four entities that never delete.</li>



<li>Creating the connection and assuming the integration can use it. The source role attachment is a separate CLI step.</li>



<li>Choosing Redshift as the direct target during a proof of concept and locking a refresh interval you picked arbitrarily.</li>



<li>Setting a 15 minute interval because it was available, then being surprised by small-file churn and CloudWatch log volume.</li>



<li>Writing downstream queries that address Zendesk custom fields as nested columns. They arrive as strings.</li>



<li>Alarming only on <code>IngestionFailed</code>, so a stalled integration produces silence and silence looks like health.</li>



<li>Reaching for zero-ETL when the requirement is ticket comments or ticket metrics, which are not in the entity set at all.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Split delete-tracking and append-only entities into separate integrations so you can rebuild one without disturbing the other.</li>



<li>Land in the SageMaker lakehouse rather than straight into Redshift unless you are certain about freshness. It keeps the refresh interval editable.</li>



<li>Put <code>DeleteCount</code> per table on a dashboard. It makes an invisible behaviour visible to everyone, not just whoever read the docs.</li>



<li>Define the integration in CloudFormation or CDK once it works, so the next environment is a deploy rather than a repeat of the console clicking.</li>



<li>Keep a silver layer between the replicated tables and your BI tool. That is where you parse the stringified structs and apply the reconciliation logic, and it means schema surprises break one view instead of every dashboard.</li>



<li>Document which entities do not track deletes somewhere your analysts will actually read, next to the tables, not in a wiki page nobody opens.</li>
</ul>



<h3 class="wp-block-heading">When to use something else</h3>



<p class="wp-block-paragraph">Zero-ETL is a good fit when your target is already AWS, your entities are in the supported set, and you want AWS to own the pipeline. It is the wrong fit if you need entities outside that list, need transformation at ingest, or need the same Zendesk data landing somewhere that is not AWS. Managed connector platforms like Fivetran or Airbyte cover a wider slice of the Zendesk API and will replicate to non-AWS destinations, at the cost of another vendor relationship, another bill, and data leaving your account boundary on the way through. If the seven entities cover your reporting, zero-ETL is cheaper and simpler. If they do not, forcing it is worse than paying for a tool that fits.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Which Zendesk entities does AWS Glue zero-ETL support?</h3>



<p class="wp-block-paragraph">Tickets, users, organizations, satisfaction ratings, articles, calls and call legs. Tickets, users and organizations replicate creates, updates and deletes. The other four replicate creates and updates only.</p>



<h3 class="wp-block-heading">Can I get ticket comments or ticket metrics through zero-ETL?</h3>



<p class="wp-block-paragraph">No. They are not in the entity set. Use a Glue ETL job against the same Zendesk connection, or a connector platform with broader API coverage, and join the result to your replicated tables downstream.</p>



<h3 class="wp-block-heading">How fresh can the data be?</h3>



<p class="wp-block-paragraph">The refresh interval goes down to 15 minutes and up to six days. This is near real time replication, not streaming. If you need sub-minute latency on Zendesk events, zero-ETL is the wrong mechanism and you want webhooks into an event pipeline instead.</p>



<h3 class="wp-block-heading">Can I change the refresh interval later?</h3>



<p class="wp-block-paragraph">Only if the target is not Redshift. With Redshift as the target the interval is fixed at creation. With the SageMaker lakehouse and other targets you can modify it afterwards.</p>



<h3 class="wp-block-heading">Why is my integration stuck in NEEDS_ATTENTION after I fixed the permissions?</h3>



<p class="wp-block-paragraph">Because recovery is automatic but not immediate. Glue retries on exponential backoff, so if the integration has been unhealthy for a while the next retry may be some way off. There is no manual recovery command. Confirm the fix is correct, then wait.</p>



<h3 class="wp-block-heading">Does zero-ETL cost extra on top of Glue?</h3>



<p class="wp-block-paragraph">AWS does not bill a separate charge for the integration itself. You pay for the surrounding services: target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. Refresh interval is the main lever, since it drives how many small commits and log entries you generate.</p>



<h3 class="wp-block-heading">Why do my Zendesk custom fields look like text?</h3>



<p class="wp-block-paragraph">The connector converts struct and list types to strings. Custom fields come through as serialized values rather than nested columns, so parse them in a downstream layer instead of querying them directly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">Zendesk AWS Glue zero-ETL removes a genuinely tedious pipeline from your plate. The setup is fiddly in the IAM layer and then it mostly runs itself, which is exactly the point of a managed integration.</p>



<p class="wp-block-paragraph">The one thing worth carrying out of this post: four of the seven supported entities never replicate a delete. Satisfaction ratings, articles, calls and call legs only grow. Nothing warns you, every metric stays green, and your numbers drift a little further from Zendesk with every cycle. Design for that on day one, put <code>DeleteCount</code> per table on a dashboard, and you will not be the person explaining the discrepancy in a meeting six months from now.</p>



<h2 class="wp-block-heading">Need help with your Zendesk to AWS data pipeline?</h2>



<p class="wp-block-paragraph">I work with teams building and fixing replication pipelines on AWS. On this kind of setup, that usually means:</p>



<ul class="wp-block-list">
<li>Auditing an existing Zendesk zero-ETL integration and quantifying how far the target has drifted from the source</li>



<li>Designing the split between continuously synced and append-only entities, including the rebuild cadence and reconciliation queries</li>



<li>Untangling the source role, target role and catalog policy layers when an integration sits in <code>NEEDS_ATTENTION</code> with no clear cause</li>



<li>Building the silver layer that parses stringified Zendesk custom fields into something your BI tool can actually use</li>



<li>Setting up CloudWatch alarms and dashboards that catch a stalled integration, not just a failed one</li>



<li>Moving a console-built integration into CloudFormation or CDK so it can be deployed across accounts</li>
</ul>



<p class="wp-block-paragraph">Send me the integration status, a CloudWatch log excerpt, or the query that is giving you the wrong number, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/">Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</title>
		<link>https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 15 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Anomaly Detection]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=225</guid>

					<description><![CDATA[<p>Your SaaS pipeline will fail far more often by succeeding at nothing than by throwing an exception, and every CloudWatch default treats an absent metric as a non-event. Here are the four signals worth alarming on: liveness, volume, freshness and shape, plus the missing-data traps that leave alarms permanently green.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/">CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said the Salesforce numbers looked wrong on the executive dashboard. I opened the pipeline. Every run for the past several days was green: the state machine succeeded, no failed states, no Lambda errors, no alarms in the account. The pipeline had been running flawlessly and writing an empty file every single time.</p>



<p class="wp-block-paragraph">That gap is the whole problem with CloudWatch data pipeline monitoring, and it is why most teams find out about broken SaaS integrations from a business user rather than from a page. A web server that breaks returns 500s, and your error-rate alarm fires. A pipeline that breaks frequently returns exit code 0, writes a zero-byte object, and reports success. The connected app got revoked, a scope was dropped during an admin cleanup, or a filter clause quietly started matching nothing. None of the AWS-native metrics have an opinion about any of that.</p>



<p class="wp-block-paragraph">This post is about what to measure instead: the four signals worth alarming on, the CloudWatch defaults that leave those alarms silently in the wrong state, how to emit custom metrics without turning your bill into a line item someone asks about, and what to check when a metric exists but the alarm still refuses to fire.</p>



<h2 class="wp-block-heading">&#8220;Did the job fail?&#8221; is the weakest question you can ask</h2>



<p class="wp-block-paragraph">Job status is a proxy for pipeline health, and a bad one. It tells you the orchestrator finished its instructions. It says nothing about whether those instructions produced data.</p>



<p class="wp-block-paragraph">SaaS sources make this worse than usual. A REST API returning HTTP 200 with an empty result array is, to your extraction code, a completely successful call. Pagination that stops on the first page because a cursor was reset is successful. An OAuth refresh that returns a token with fewer scopes than yesterday is successful right up until one object starts coming back empty. Every one of those is a green run.</p>



<p class="wp-block-paragraph">Build the monitoring around four questions the job outcome cannot answer:</p>



<ol class="wp-block-list"><li><strong>Liveness</strong> &#8211; did the run happen at all?</li><li><strong>Volume</strong> &#8211; did it move a plausible amount of data?</li><li><strong>Freshness</strong> &#8211; how old is the newest record that actually landed?</li><li><strong>Shape</strong> &#8211; are the fields and values still what downstream expects?</li></ol>



<p class="wp-block-paragraph">Each needs a different kind of metric, and each fails silently in its own way.</p>



<h2 class="wp-block-heading">Signal 1: liveness, and the missing-data trap that hides it</h2>



<p class="wp-block-paragraph">Step Functions publishes execution metrics to the <code>AWS/States</code> namespace, including <code>ExecutionsStarted</code>, <code>ExecutionsSucceeded</code>, <code>ExecutionsFailed</code>, <code>ExecutionsTimedOut</code>, <code>ExecutionsAborted</code> and <code>ExecutionTime</code>. The obvious move is an alarm on <code>ExecutionsFailed</code>. Most teams stop there, and that alarm is close to useless on its own.</p>



<p class="wp-block-paragraph">Here is why. Those metrics are published as a result of executions. If somebody disables the EventBridge schedule rule during an incident and forgets to re-enable it, no execution starts, so no data point is published for that dimension at all. Your alarm has nothing to evaluate.</p>



<p class="wp-block-paragraph">What happens next depends entirely on a parameter most people never set. CloudWatch alarms accept a <code>TreatMissingData</code> setting with four values: <code>breaching</code>, <code>notBreaching</code>, <code>ignore</code> and <code>missing</code>. The default is <code>missing</code>, which means the alarm keeps its current state when there is nothing to evaluate. An alarm that was sitting in OK stays in OK. Forever. It is green because it is blind, and there is no visual difference between the two.</p>



<p class="wp-block-paragraph">The fix is not to change <code>TreatMissingData</code> on the failure alarm. It is to add a separate heartbeat metric that your pipeline emits on completion, and alarm on the <em>absence</em> of that heartbeat:</p>



<pre class="wp-block-code"><code># Emitted at the very end of a successful run, per source system.
aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --value 1 
  --unit Count 
  --dimensions Source=salesforce,Stage=extract</code></pre>



<p class="wp-block-paragraph">Then the alarm that actually catches a dead scheduler:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name "ingest-salesforce-no-heartbeat" 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --dimensions Name=Source,Value=salesforce Name=Stage,Value=extract 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --datapoints-to-alarm 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts</code></pre>



<p class="wp-block-paragraph">The two flags doing the real work are <code>--comparison-operator LessThanThreshold</code> and <code>--treat-missing-data breaching</code>. The first catches a run that completes but emits nothing. The second catches a run that never happens at all. Without the second flag, silence reads as health.</p>



<p class="wp-block-paragraph">Set <code>--period</code> comfortably longer than your schedule interval so one slow run does not page anyone, and use <code>--datapoints-to-alarm</code> to require consecutive breaches. One caveat: <code>breaching</code> is right for a heartbeat you control, but a poor blanket default across the account, because plenty of AWS metrics are only emitted when non-zero. Throttling counters are legitimately sparse, and forcing them to breach on silence just makes noise.</p>



<h2 class="wp-block-heading">Signal 2: volume, and why a static threshold beats anomaly detection at first</h2>



<p class="wp-block-paragraph">A heartbeat proves the run happened. It does not prove the run did anything. Emit a row count, per source, per object:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RecordsWritten" 
  --value 14238 
  --unit Count 
  --dimensions Source=salesforce,Object=opportunity</code></pre>



<p class="wp-block-paragraph">Alarm on that being unexpectedly low rather than on it being zero. Zero-row alarms feel precise, but the more common failure is partial: a cursor that only advances through the first page, a rate limit that truncates the pull, an incremental sync that silently narrows its window. Those deliver a fraction of the expected rows, not none of them.</p>



<p class="wp-block-paragraph">There is a subtlety here that trips people. If a run fails before emitting <code>RecordsWritten</code> at all, a low-volume alarm on that metric sees missing data, not a zero. Metric math fixes it cleanly:</p>



<pre class="wp-block-code"><code>FILL(m1, 0)</code></pre>



<p class="wp-block-paragraph">Wrapping the metric in <code>FILL</code> with a value of 0 substitutes zero for every empty period, so an absent run and an empty run both breach the same alarm. You get one alert instead of two competing ones, and it means what you think it means.</p>



<p class="wp-block-paragraph">CloudWatch anomaly detection is the obvious upgrade for seasonal volume. It models expected behaviour and alarms on deviation from the band, which handles a Monday-morning spike that a static threshold cannot. I still would not reach for it first: it needs a stretch of representative history before the band is trustworthy, and during that stretch it will be confidently wrong. Start with a static floor well below your observed minimum, and move to anomaly detection once you know what normal looks like and can see the fixed number fighting you.</p>



<h2 class="wp-block-heading">Signal 3: freshness, the metric almost nobody emits</h2>



<p class="wp-block-paragraph">This is the one that separates a pipeline you can trust from one you merely watch. Liveness and volume can both look perfect while the data is stale, because a job that re-reads the same window every run produces rows, produces a heartbeat, and produces nothing new.</p>



<p class="wp-block-paragraph">The metric to emit is lag: the difference in seconds between now and the maximum source-side modification timestamp in the batch you just landed. Not your own processing timestamp, which will always look current. The source system&#8217;s timestamp, the one that tells you how far behind the truth you are.</p>



<pre class="wp-block-code"><code>import time
import boto3

cw = boto3.client("cloudwatch")

def emit_freshness(source, obj, max_source_ts_epoch):
    """max_source_ts_epoch: max LastModifiedDate in this batch, as epoch seconds."""
    lag_seconds = int(time.time()) - int(max_source_ts_epoch)
    cw.put_metric_data(
        Namespace="Pipelines/Ingest",
        MetricData=[{
            "MetricName": "SourceLagSeconds",
            "Value": lag_seconds,
            "Unit": "Seconds",
            "Dimensions": [
                {"Name": "Source", "Value": source},
                {"Name": "Object", "Value": obj},
            ],
        }],
    )</code></pre>



<p class="wp-block-paragraph">Alarm with <code>GreaterThanThreshold</code>, and set the threshold to roughly two schedule intervals plus your worst observed run duration. An hourly pipeline that takes twelve minutes on a bad day gets a threshold somewhere around two and a half hours. Tighter than that and you page on ordinary jitter.</p>



<p class="wp-block-paragraph">Freshness is also the one metric worth putting in front of non-engineers. &#8220;Your Salesforce data is currently 40 minutes behind&#8221; is a sentence a stakeholder can act on; &#8220;ExecutionsSucceeded is 1&#8221; is not. If you surface dashboards to a business audience through Grafana Cloud or QuickSight over the CloudWatch data source, lag is the panel that earns its space.</p>



<h2 class="wp-block-heading">Signal 4: shape, using logs instead of counters</h2>



<p class="wp-block-paragraph">Shape problems are schema drift and value drift. A SaaS admin renames a custom field, a picklist gains a value your mapping does not handle, a required field starts arriving null for a subset of records. The rows keep flowing. The row count looks fine. Downstream joins quietly drop records.</p>



<p class="wp-block-paragraph">Counters alone are the wrong shape here, because what you need at 2am is not &#8220;the null rate rose&#8221;, it is &#8220;which records, and what changed&#8221;. Embedded Metric Format is built for exactly that split: you write one structured JSON log line, CloudWatch extracts the numbers as real metrics you can alarm on, and the full event stays queryable in Logs Insights for the investigation.</p>



<pre class="wp-block-code"><code>{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "Pipelines/Quality",
        "Dimensions": [["Source", "Object"]],
        "Metrics": [
          { "Name": "NullRequiredFields", "Unit": "Count" },
          { "Name": "UnmappedFields", "Unit": "Count" }
        ]
      }
    ]
  },
  "Source": "salesforce",
  "Object": "opportunity",
  "NullRequiredFields": 412,
  "UnmappedFields": 2,
  "unmapped_field_names": ["Renewal_Tier__c", "Partner_Code__c"],
  "batch_id": "2f8c1a9e"
}</code></pre>



<p class="wp-block-paragraph">Two things to get right. Every name listed in <code>Dimensions</code> must also exist as a top-level property in the same JSON object, or extraction silently produces nothing. And note that <code>unmapped_field_names</code> and <code>batch_id</code> are properties, not dimensions: they ride along in the log event for querying but never become metric dimensions, which is exactly what you want.</p>



<p class="wp-block-paragraph">When the alarm fires, the investigation is a Logs Insights query against the same log group:</p>



<pre class="wp-block-code"><code>fields @timestamp, Object, UnmappedFields, unmapped_field_names
| filter Source = "salesforce" and UnmappedFields &gt; 0
| sort @timestamp desc
| limit 50</code></pre>



<p class="wp-block-paragraph">That hands you the new field names in seconds, which is the difference between a five-minute fix and an hour of guessing.</p>



<h2 class="wp-block-heading">Keeping CloudWatch data pipeline monitoring off the top of your bill</h2>



<p class="wp-block-paragraph">Custom metrics and log ingestion are consistently the two largest CloudWatch line items for data teams. Neither has a flat fee, so the cost is a direct function of choices you make in code. Three matter far more than the rest.</p>



<h3 class="wp-block-heading">Cardinality is the cost lever</h3>



<p class="wp-block-paragraph">A custom metric is billed per unique combination of namespace, metric name and dimension values. <code>Source</code> with five values and <code>Object</code> with twenty gives you a hundred metrics per metric name. Add <code>Tenant</code> with two hundred values and you have twenty thousand. Multiply by four metric names and the number stops being a rounding error.</p>



<p class="wp-block-paragraph">Never put a run ID, batch ID, record ID or raw error string in a dimension. Those belong in the log event. Dimensions are for things you would actually build an alarm on, and you are not going to build an alarm per batch.</p>



<h3 class="wp-block-heading">EMF over PutMetricData for anything high-frequency</h3>



<p class="wp-block-paragraph"><code>PutMetricData</code> is a synchronous API call on your critical path. In Lambda that means duration you pay for, plus a request that can throttle and needs handling. EMF avoids both: you write JSON to stdout, CloudWatch Logs extracts the metrics asynchronously, your function returns. For a per-run summary either is fine. For anything per batch or per record, EMF is the one I reach for.</p>



<h3 class="wp-block-heading">The Infrequent Access log class will break this silently</h3>



<p class="wp-block-paragraph">This one has caught competent people. CloudWatch Logs offers a Standard and an Infrequent Access log class, and IA has a lower ingestion price, which makes it an obvious target during a cost review. According to the AWS documentation, the IA class does not support embedded metric format, metric filters or subscription filters. Logs Insights queries still work.</p>



<p class="wp-block-paragraph">So if somebody moves your pipeline log group to IA to save money, every EMF-derived metric stops being extracted. The alarms built on those metrics go to <code>INSUFFICIENT_DATA</code>, or worse, sit in OK if they use the default missing-data behaviour. The logs still arrive. The dashboard still exists. The metrics are just gone.</p>



<p class="wp-block-paragraph">Log class cannot be changed after a log group is created, so recovery means a new Standard log group and redirecting output to it. Keep metric-backing log groups on Standard and point IA at the genuinely archival stuff: verbose debug output, raw payload dumps, anything you only read during a forensic investigation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Routing: one page, not four</h2>



<p class="wp-block-paragraph">Build all four signals and one broken pipeline fires the heartbeat, volume and freshness alarms within minutes of each other. Three pages, one problem. That is how alert fatigue starts.</p>



<p class="wp-block-paragraph">Composite alarms fix it. Build one composite alarm per pipeline whose rule combines the child alarms with OR, put the notification action there, and leave the child alarms with no actions at all. They become diagnostic detail you read after the page rather than three separate interruptions.</p>



<p class="wp-block-paragraph">SNS to email is fine for one person and poor past that, since it has no deduplication, escalation or acknowledgement. With more than one person on call, route through something with a schedule such as PagerDuty, Opsgenie or Better Stack. If your team already lives in Grafana Cloud or Datadog for application monitoring, pulling CloudWatch alarm state in through their AWS integration beats maintaining two alerting surfaces.</p>



<h2 class="wp-block-heading">Troubleshooting: the metric exists but the alarm never fires</h2>



<p class="wp-block-paragraph">Roughly in order of how often I have seen each one.</p>



<ul class="wp-block-list"><li><strong>Dimension mismatch.</strong> The alarm&#8217;s dimension set must match the published metric&#8217;s dimension set exactly, all of them, in the same values. A metric published with <code>Source</code> and <code>Object</code> is a different metric from one published with only <code>Source</code>. Run <code>aws cloudwatch list-metrics --namespace "Pipelines/Ingest"</code> and compare the output against your alarm definition character by character.</li><li><strong>Alarm stuck in INSUFFICIENT_DATA.</strong> The metric is not being published at all for that dimension set, or the period is shorter than the publishing interval so most periods are empty. Lengthen the period to at least the emission interval.</li><li><strong>Alarm stuck in OK during a real outage.</strong> Classic default <code>missing</code> behaviour. Any alarm whose purpose is to detect absence needs <code>--treat-missing-data breaching</code>.</li><li><strong>EMF metrics never appear.</strong> Check the log group&#8217;s class first, then confirm every dimension name in the <code>Dimensions</code> array exists as a top-level property in the same JSON object, then confirm the JSON is a single line with no wrapping prefix from your logger.</li><li><strong>Freshness spikes on a schedule.</strong> Almost always a timezone mismatch between how the SaaS API reports modification timestamps and how you convert them. Normalise everything to UTC epoch seconds at the boundary and stop converting anywhere else.</li><li><strong>Row counts double intermittently.</strong> Retries emitting the metric twice for one logical run. Emit volume metrics once, after the write is confirmed, not inside the retried block.</li></ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Alarming only on job failure, which misses every silent-success failure mode.</li><li>Leaving <code>TreatMissingData</code> at its default on alarms designed to detect absence.</li><li>Alarming on zero rows instead of on unexpectedly low rows, so partial extractions slip through.</li><li>Emitting a processing timestamp as the freshness metric, which is always current and therefore always useless.</li><li>Putting high-cardinality identifiers into metric dimensions and then being surprised by the bill.</li><li>Moving pipeline log groups to the Infrequent Access class without checking what depends on metric extraction.</li><li>Wiring notification actions to every child alarm instead of to one composite alarm per pipeline.</li><li>Building dashboards nobody looks at instead of alarms that reach a person.</li></ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list"><li>Emit liveness, volume, freshness and a shape metric for every source. Four metrics per source is a small amount of code and covers most of what actually breaks.</li><li>Define alarms in Terraform or CloudFormation alongside the pipeline, so a new source cannot ship without monitoring.</li><li>Use consistent dimension names across every pipeline. <code>Source</code> and <code>Object</code> everywhere makes search expressions and dashboards trivial; a mix of <code>source</code>, <code>system</code> and <code>connector</code> makes them impossible.</li><li>Set alarm descriptions to a one-line runbook. The person paged at 3am should not have to guess what the alarm means.</li><li>Test the absence case deliberately. Disable the schedule in a non-production account and confirm the heartbeat alarm fires. An untested absence alarm is an assumption.</li><li>Set explicit log retention on every pipeline log group. Never-expire is the default and it compounds quietly.</li><li>Keep an eye on the source side too, not just yours. Most SaaS APIs publish their own rate limit or API usage counters, and pulling those into a metric is often the earliest possible warning.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is CloudWatch enough on its own for data pipeline monitoring?</h3>



<p class="wp-block-paragraph">For alerting, yes, if your workloads are on AWS. It has metrics, alarms, composite alarms and log querying, and it is already collecting service metrics you would otherwise have to ship somewhere. Where it gets weak is exploratory analysis and cross-source correlation, which is where teams add Grafana, Datadog or a dedicated data-observability tool on top. Run CloudWatch for the paging path first and add the analysis layer when you have an actual gap.</p>



<h3 class="wp-block-heading">How do I monitor a pipeline that runs on a managed service like Glue or AppFlow?</h3>



<p class="wp-block-paragraph">The same four signals apply; only the emission point changes. Managed services publish their own metrics, but those describe the service&#8217;s execution, not your data. Where you cannot add code inside the job, hook the completion event through EventBridge to a small Lambda function that inspects what landed and emits the volume and freshness metrics. It is a handful of lines and it is the only way to get data-level signals out of a job you do not control.</p>



<h3 class="wp-block-heading">Should each tenant get its own metric in a multi-tenant pipeline?</h3>



<p class="wp-block-paragraph">Usually not as a dimension. Tenant counts grow, and cardinality is billed per unique dimension combination. Emit aggregate metrics with <code>Source</code> and <code>Object</code> dimensions, put the tenant identifier in the log event as a property, and use Logs Insights to slice by tenant when you need to. Break out a dimension per tenant only for the small number of accounts where a contractual SLA makes it worth the cost.</p>



<h3 class="wp-block-heading">What is a sensible freshness threshold?</h3>



<p class="wp-block-paragraph">Start at two schedule intervals plus your worst observed run duration, then adjust after watching real data for a couple of weeks. The threshold that matters is the one the business would notice, so if a stakeholder can tolerate half a day of staleness on a nightly export, do not page at four hours. Match the alarm to the actual consequence.</p>



<h3 class="wp-block-heading">Why does my alarm sit in INSUFFICIENT_DATA instead of ALARM?</h3>



<p class="wp-block-paragraph">CloudWatch has no data points to evaluate for that metric and dimension set. Either nothing is publishing, the dimensions do not match what is published, or the alarm period is shorter than the interval at which you emit. Confirm the metric exists with <code>list-metrics</code> first, and only then look at the alarm configuration.</p>



<h3 class="wp-block-heading">How much does this cost to run?</h3>



<p class="wp-block-paragraph">It depends on three things you control: the number of unique custom metrics, the volume of log data ingested, and how much data your Logs Insights queries scan. Alarms and dashboards are minor by comparison. Keep dimension cardinality low, set retention deliberately, and query narrow time windows, and the monitoring stays well below the cost of the pipeline it protects. Check current rates on the AWS pricing page rather than trusting any number in a blog post, including this one.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Effective CloudWatch data pipeline monitoring is built around absence, not around errors. Your SaaS integrations will fail far more often by succeeding at nothing than by throwing an exception, and every default in CloudWatch treats an absent metric as a non-event.</p>



<p class="wp-block-paragraph">If you only change one thing after reading this, add a heartbeat metric with <code>--treat-missing-data breaching</code> to your most important pipeline, then go and disable its schedule in a test account and confirm the alarm actually fires. If it does not, you have just found out on your own terms instead of from a stakeholder asking why the numbers look wrong.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your pipeline monitoring?</h2>



<p class="wp-block-paragraph">I work with teams running SaaS-to-warehouse pipelines on AWS, usually at the point where something broke quietly and nobody wants that to happen twice. Things I can help with here:</p>



<ul class="wp-block-list"><li>Auditing existing CloudWatch alarms for silent-failure gaps, particularly missing-data configuration and dimension mismatches that leave alarms permanently green.</li><li>Instrumenting liveness, volume, freshness and data-quality metrics into pipelines built on Lambda, Glue, Step Functions or self-hosted schedulers.</li><li>Designing EMF logging and metric namespaces that stay useful as source and object counts grow, without cardinality running away.</li><li>Building composite alarms and routing so one broken pipeline produces one actionable page rather than a cluster of them.</li><li>Cutting CloudWatch spend on log ingestion and custom metrics without losing the signals you actually rely on.</li><li>Writing Terraform modules that ship monitoring alongside every new source connector, so coverage is structural rather than remembered.</li></ul>



<p class="wp-block-paragraph">If you have a pipeline that has gone quiet, send me the alarm configuration and a Logs Insights output from the run in question and I will tell you what it is missing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/">CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
