{"id":225,"date":"2026-08-15T21:00:00","date_gmt":"2026-08-15T18:00:00","guid":{"rendered":"https:\/\/john-nessime.com\/blog\/?p=225"},"modified":"2026-08-06T22:32:36","modified_gmt":"2026-08-06T19:32:36","slug":"cloudwatch-data-pipeline-monitoring","status":"publish","type":"post","link":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/","title":{"rendered":"CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">The ticket said the Salesforce numbers looked wrong on the executive dashboard. I opened the pipeline. Every run for the past several days was green: the state machine succeeded, no failed states, no Lambda errors, no alarms in the account. The pipeline had been running flawlessly and writing an empty file every single time.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That gap is the whole problem with CloudWatch data pipeline monitoring, and it is why most teams find out about broken SaaS integrations from a business user rather than from a page. A web server that breaks returns 500s, and your error-rate alarm fires. A pipeline that breaks frequently returns exit code 0, writes a zero-byte object, and reports success. The connected app got revoked, a scope was dropped during an admin cleanup, or a filter clause quietly started matching nothing. None of the AWS-native metrics have an opinion about any of that.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This post is about what to measure instead: the four signals worth alarming on, the CloudWatch defaults that leave those alarms silently in the wrong state, how to emit custom metrics without turning your bill into a line item someone asks about, and what to check when a metric exists but the alarm still refuses to fire.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">&#8220;Did the job fail?&#8221; is the weakest question you can ask<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Job status is a proxy for pipeline health, and a bad one. It tells you the orchestrator finished its instructions. It says nothing about whether those instructions produced data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">SaaS sources make this worse than usual. A REST API returning HTTP 200 with an empty result array is, to your extraction code, a completely successful call. Pagination that stops on the first page because a cursor was reset is successful. An OAuth refresh that returns a token with fewer scopes than yesterday is successful right up until one object starts coming back empty. Every one of those is a green run.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Build the monitoring around four questions the job outcome cannot answer:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li><strong>Liveness<\/strong> &#8211; did the run happen at all?<\/li><li><strong>Volume<\/strong> &#8211; did it move a plausible amount of data?<\/li><li><strong>Freshness<\/strong> &#8211; how old is the newest record that actually landed?<\/li><li><strong>Shape<\/strong> &#8211; are the fields and values still what downstream expects?<\/li><\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Each needs a different kind of metric, and each fails silently in its own way.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Signal 1: liveness, and the missing-data trap that hides it<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Step Functions publishes execution metrics to the <code>AWS\/States<\/code> namespace, including <code>ExecutionsStarted<\/code>, <code>ExecutionsSucceeded<\/code>, <code>ExecutionsFailed<\/code>, <code>ExecutionsTimedOut<\/code>, <code>ExecutionsAborted<\/code> and <code>ExecutionTime<\/code>. The obvious move is an alarm on <code>ExecutionsFailed<\/code>. Most teams stop there, and that alarm is close to useless on its own.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here is why. Those metrics are published as a result of executions. If somebody disables the EventBridge schedule rule during an incident and forgets to re-enable it, no execution starts, so no data point is published for that dimension at all. Your alarm has nothing to evaluate.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">What happens next depends entirely on a parameter most people never set. CloudWatch alarms accept a <code>TreatMissingData<\/code> setting with four values: <code>breaching<\/code>, <code>notBreaching<\/code>, <code>ignore<\/code> and <code>missing<\/code>. The default is <code>missing<\/code>, which means the alarm keeps its current state when there is nothing to evaluate. An alarm that was sitting in OK stays in OK. Forever. It is green because it is blind, and there is no visual difference between the two.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The fix is not to change <code>TreatMissingData<\/code> on the failure alarm. It is to add a separate heartbeat metric that your pipeline emits on completion, and alarm on the <em>absence<\/em> of that heartbeat:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Emitted at the very end of a successful run, per source system.\naws cloudwatch put-metric-data \n  --namespace \"Pipelines\/Ingest\" \n  --metric-name \"RunCompleted\" \n  --value 1 \n  --unit Count \n  --dimensions Source=salesforce,Stage=extract<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then the alarm that actually catches a dead scheduler:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>aws cloudwatch put-metric-alarm \n  --alarm-name \"ingest-salesforce-no-heartbeat\" \n  --namespace \"Pipelines\/Ingest\" \n  --metric-name \"RunCompleted\" \n  --dimensions Name=Source,Value=salesforce Name=Stage,Value=extract \n  --statistic Sum \n  --period 3600 \n  --evaluation-periods 2 \n  --datapoints-to-alarm 2 \n  --threshold 1 \n  --comparison-operator LessThanThreshold \n  --treat-missing-data breaching \n  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The two flags doing the real work are <code>--comparison-operator LessThanThreshold<\/code> and <code>--treat-missing-data breaching<\/code>. The first catches a run that completes but emits nothing. The second catches a run that never happens at all. Without the second flag, silence reads as health.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Set <code>--period<\/code> comfortably longer than your schedule interval so one slow run does not page anyone, and use <code>--datapoints-to-alarm<\/code> to require consecutive breaches. One caveat: <code>breaching<\/code> is right for a heartbeat you control, but a poor blanket default across the account, because plenty of AWS metrics are only emitted when non-zero. Throttling counters are legitimately sparse, and forcing them to breach on silence just makes noise.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Signal 2: volume, and why a static threshold beats anomaly detection at first<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A heartbeat proves the run happened. It does not prove the run did anything. Emit a row count, per source, per object:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>aws cloudwatch put-metric-data \n  --namespace \"Pipelines\/Ingest\" \n  --metric-name \"RecordsWritten\" \n  --value 14238 \n  --unit Count \n  --dimensions Source=salesforce,Object=opportunity<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Alarm on that being unexpectedly low rather than on it being zero. Zero-row alarms feel precise, but the more common failure is partial: a cursor that only advances through the first page, a rate limit that truncates the pull, an incremental sync that silently narrows its window. Those deliver a fraction of the expected rows, not none of them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There is a subtlety here that trips people. If a run fails before emitting <code>RecordsWritten<\/code> at all, a low-volume alarm on that metric sees missing data, not a zero. Metric math fixes it cleanly:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>FILL(m1, 0)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Wrapping the metric in <code>FILL<\/code> with a value of 0 substitutes zero for every empty period, so an absent run and an empty run both breach the same alarm. You get one alert instead of two competing ones, and it means what you think it means.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">CloudWatch anomaly detection is the obvious upgrade for seasonal volume. It models expected behaviour and alarms on deviation from the band, which handles a Monday-morning spike that a static threshold cannot. I still would not reach for it first: it needs a stretch of representative history before the band is trustworthy, and during that stretch it will be confidently wrong. Start with a static floor well below your observed minimum, and move to anomaly detection once you know what normal looks like and can see the fixed number fighting you.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Signal 3: freshness, the metric almost nobody emits<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This is the one that separates a pipeline you can trust from one you merely watch. Liveness and volume can both look perfect while the data is stale, because a job that re-reads the same window every run produces rows, produces a heartbeat, and produces nothing new.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The metric to emit is lag: the difference in seconds between now and the maximum source-side modification timestamp in the batch you just landed. Not your own processing timestamp, which will always look current. The source system&#8217;s timestamp, the one that tells you how far behind the truth you are.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import time\nimport boto3\n\ncw = boto3.client(\"cloudwatch\")\n\ndef emit_freshness(source, obj, max_source_ts_epoch):\n    \"\"\"max_source_ts_epoch: max LastModifiedDate in this batch, as epoch seconds.\"\"\"\n    lag_seconds = int(time.time()) - int(max_source_ts_epoch)\n    cw.put_metric_data(\n        Namespace=\"Pipelines\/Ingest\",\n        MetricData=[{\n            \"MetricName\": \"SourceLagSeconds\",\n            \"Value\": lag_seconds,\n            \"Unit\": \"Seconds\",\n            \"Dimensions\": [\n                {\"Name\": \"Source\", \"Value\": source},\n                {\"Name\": \"Object\", \"Value\": obj},\n            ],\n        }],\n    )<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Alarm with <code>GreaterThanThreshold<\/code>, and set the threshold to roughly two schedule intervals plus your worst observed run duration. An hourly pipeline that takes twelve minutes on a bad day gets a threshold somewhere around two and a half hours. Tighter than that and you page on ordinary jitter.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Freshness is also the one metric worth putting in front of non-engineers. &#8220;Your Salesforce data is currently 40 minutes behind&#8221; is a sentence a stakeholder can act on; &#8220;ExecutionsSucceeded is 1&#8221; is not. If you surface dashboards to a business audience through Grafana Cloud or QuickSight over the CloudWatch data source, lag is the panel that earns its space.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Signal 4: shape, using logs instead of counters<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Shape problems are schema drift and value drift. A SaaS admin renames a custom field, a picklist gains a value your mapping does not handle, a required field starts arriving null for a subset of records. The rows keep flowing. The row count looks fine. Downstream joins quietly drop records.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Counters alone are the wrong shape here, because what you need at 2am is not &#8220;the null rate rose&#8221;, it is &#8220;which records, and what changed&#8221;. Embedded Metric Format is built for exactly that split: you write one structured JSON log line, CloudWatch extracts the numbers as real metrics you can alarm on, and the full event stays queryable in Logs Insights for the investigation.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"_aws\": {\n    \"Timestamp\": 1700000000000,\n    \"CloudWatchMetrics\": [\n      {\n        \"Namespace\": \"Pipelines\/Quality\",\n        \"Dimensions\": [[\"Source\", \"Object\"]],\n        \"Metrics\": [\n          { \"Name\": \"NullRequiredFields\", \"Unit\": \"Count\" },\n          { \"Name\": \"UnmappedFields\", \"Unit\": \"Count\" }\n        ]\n      }\n    ]\n  },\n  \"Source\": \"salesforce\",\n  \"Object\": \"opportunity\",\n  \"NullRequiredFields\": 412,\n  \"UnmappedFields\": 2,\n  \"unmapped_field_names\": [\"Renewal_Tier__c\", \"Partner_Code__c\"],\n  \"batch_id\": \"2f8c1a9e\"\n}<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Two things to get right. Every name listed in <code>Dimensions<\/code> must also exist as a top-level property in the same JSON object, or extraction silently produces nothing. And note that <code>unmapped_field_names<\/code> and <code>batch_id<\/code> are properties, not dimensions: they ride along in the log event for querying but never become metric dimensions, which is exactly what you want.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When the alarm fires, the investigation is a Logs Insights query against the same log group:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>fields @timestamp, Object, UnmappedFields, unmapped_field_names\n| filter Source = \"salesforce\" and UnmappedFields &gt; 0\n| sort @timestamp desc\n| limit 50<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">That hands you the new field names in seconds, which is the difference between a five-minute fix and an hour of guessing.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Keeping CloudWatch data pipeline monitoring off the top of your bill<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Custom metrics and log ingestion are consistently the two largest CloudWatch line items for data teams. Neither has a flat fee, so the cost is a direct function of choices you make in code. Three matter far more than the rest.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Cardinality is the cost lever<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">A custom metric is billed per unique combination of namespace, metric name and dimension values. <code>Source<\/code> with five values and <code>Object<\/code> with twenty gives you a hundred metrics per metric name. Add <code>Tenant<\/code> with two hundred values and you have twenty thousand. Multiply by four metric names and the number stops being a rounding error.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Never put a run ID, batch ID, record ID or raw error string in a dimension. Those belong in the log event. Dimensions are for things you would actually build an alarm on, and you are not going to build an alarm per batch.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">EMF over PutMetricData for anything high-frequency<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><code>PutMetricData<\/code> is a synchronous API call on your critical path. In Lambda that means duration you pay for, plus a request that can throttle and needs handling. EMF avoids both: you write JSON to stdout, CloudWatch Logs extracts the metrics asynchronously, your function returns. For a per-run summary either is fine. For anything per batch or per record, EMF is the one I reach for.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The Infrequent Access log class will break this silently<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">This one has caught competent people. CloudWatch Logs offers a Standard and an Infrequent Access log class, and IA has a lower ingestion price, which makes it an obvious target during a cost review. According to the AWS documentation, the IA class does not support embedded metric format, metric filters or subscription filters. Logs Insights queries still work.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So if somebody moves your pipeline log group to IA to save money, every EMF-derived metric stops being extracted. The alarms built on those metrics go to <code>INSUFFICIENT_DATA<\/code>, or worse, sit in OK if they use the default missing-data behaviour. The logs still arrive. The dashboard still exists. The metrics are just gone.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Log class cannot be changed after a log group is created, so recovery means a new Standard log group and redirecting output to it. Keep metric-backing log groups on Standard and point IA at the genuinely archival stuff: verbose debug output, raw payload dumps, anything you only read during a forensic investigation.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Routing: one page, not four<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Build all four signals and one broken pipeline fires the heartbeat, volume and freshness alarms within minutes of each other. Three pages, one problem. That is how alert fatigue starts.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Composite alarms fix it. Build one composite alarm per pipeline whose rule combines the child alarms with OR, put the notification action there, and leave the child alarms with no actions at all. They become diagnostic detail you read after the page rather than three separate interruptions.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">SNS to email is fine for one person and poor past that, since it has no deduplication, escalation or acknowledgement. With more than one person on call, route through something with a schedule such as PagerDuty, Opsgenie or Better Stack. If your team already lives in Grafana Cloud or Datadog for application monitoring, pulling CloudWatch alarm state in through their AWS integration beats maintaining two alerting surfaces.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Troubleshooting: the metric exists but the alarm never fires<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Roughly in order of how often I have seen each one.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Dimension mismatch.<\/strong> The alarm&#8217;s dimension set must match the published metric&#8217;s dimension set exactly, all of them, in the same values. A metric published with <code>Source<\/code> and <code>Object<\/code> is a different metric from one published with only <code>Source<\/code>. Run <code>aws cloudwatch list-metrics --namespace \"Pipelines\/Ingest\"<\/code> and compare the output against your alarm definition character by character.<\/li><li><strong>Alarm stuck in INSUFFICIENT_DATA.<\/strong> The metric is not being published at all for that dimension set, or the period is shorter than the publishing interval so most periods are empty. Lengthen the period to at least the emission interval.<\/li><li><strong>Alarm stuck in OK during a real outage.<\/strong> Classic default <code>missing<\/code> behaviour. Any alarm whose purpose is to detect absence needs <code>--treat-missing-data breaching<\/code>.<\/li><li><strong>EMF metrics never appear.<\/strong> Check the log group&#8217;s class first, then confirm every dimension name in the <code>Dimensions<\/code> array exists as a top-level property in the same JSON object, then confirm the JSON is a single line with no wrapping prefix from your logger.<\/li><li><strong>Freshness spikes on a schedule.<\/strong> Almost always a timezone mismatch between how the SaaS API reports modification timestamps and how you convert them. Normalise everything to UTC epoch seconds at the boundary and stop converting anywhere else.<\/li><li><strong>Row counts double intermittently.<\/strong> Retries emitting the metric twice for one logical run. Emit volume metrics once, after the write is confirmed, not inside the retried block.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Common mistakes<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li>Alarming only on job failure, which misses every silent-success failure mode.<\/li><li>Leaving <code>TreatMissingData<\/code> at its default on alarms designed to detect absence.<\/li><li>Alarming on zero rows instead of on unexpectedly low rows, so partial extractions slip through.<\/li><li>Emitting a processing timestamp as the freshness metric, which is always current and therefore always useless.<\/li><li>Putting high-cardinality identifiers into metric dimensions and then being surprised by the bill.<\/li><li>Moving pipeline log groups to the Infrequent Access class without checking what depends on metric extraction.<\/li><li>Wiring notification actions to every child alarm instead of to one composite alarm per pipeline.<\/li><li>Building dashboards nobody looks at instead of alarms that reach a person.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Best practices worth the effort<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li>Emit liveness, volume, freshness and a shape metric for every source. Four metrics per source is a small amount of code and covers most of what actually breaks.<\/li><li>Define alarms in Terraform or CloudFormation alongside the pipeline, so a new source cannot ship without monitoring.<\/li><li>Use consistent dimension names across every pipeline. <code>Source<\/code> and <code>Object<\/code> everywhere makes search expressions and dashboards trivial; a mix of <code>source<\/code>, <code>system<\/code> and <code>connector<\/code> makes them impossible.<\/li><li>Set alarm descriptions to a one-line runbook. The person paged at 3am should not have to guess what the alarm means.<\/li><li>Test the absence case deliberately. Disable the schedule in a non-production account and confirm the heartbeat alarm fires. An untested absence alarm is an assumption.<\/li><li>Set explicit log retention on every pipeline log group. Never-expire is the default and it compounds quietly.<\/li><li>Keep an eye on the source side too, not just yours. Most SaaS APIs publish their own rate limit or API usage counters, and pulling those into a metric is often the earliest possible warning.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">FAQ<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Is CloudWatch enough on its own for data pipeline monitoring?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">For alerting, yes, if your workloads are on AWS. It has metrics, alarms, composite alarms and log querying, and it is already collecting service metrics you would otherwise have to ship somewhere. Where it gets weak is exploratory analysis and cross-source correlation, which is where teams add Grafana, Datadog or a dedicated data-observability tool on top. Run CloudWatch for the paging path first and add the analysis layer when you have an actual gap.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How do I monitor a pipeline that runs on a managed service like Glue or AppFlow?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The same four signals apply; only the emission point changes. Managed services publish their own metrics, but those describe the service&#8217;s execution, not your data. Where you cannot add code inside the job, hook the completion event through EventBridge to a small Lambda function that inspects what landed and emits the volume and freshness metrics. It is a handful of lines and it is the only way to get data-level signals out of a job you do not control.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Should each tenant get its own metric in a multi-tenant pipeline?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Usually not as a dimension. Tenant counts grow, and cardinality is billed per unique dimension combination. Emit aggregate metrics with <code>Source<\/code> and <code>Object<\/code> dimensions, put the tenant identifier in the log event as a property, and use Logs Insights to slice by tenant when you need to. Break out a dimension per tenant only for the small number of accounts where a contractual SLA makes it worth the cost.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What is a sensible freshness threshold?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Start at two schedule intervals plus your worst observed run duration, then adjust after watching real data for a couple of weeks. The threshold that matters is the one the business would notice, so if a stakeholder can tolerate half a day of staleness on a nightly export, do not page at four hours. Match the alarm to the actual consequence.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Why does my alarm sit in INSUFFICIENT_DATA instead of ALARM?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">CloudWatch has no data points to evaluate for that metric and dimension set. Either nothing is publishing, the dimensions do not match what is published, or the alarm period is shorter than the interval at which you emit. Confirm the metric exists with <code>list-metrics<\/code> first, and only then look at the alarm configuration.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How much does this cost to run?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">It depends on three things you control: the number of unique custom metrics, the volume of log data ingested, and how much data your Logs Insights queries scan. Alarms and dashboards are minor by comparison. Keep dimension cardinality low, set retention deliberately, and query narrow time windows, and the monitoring stays well below the cost of the pipeline it protects. Check current rates on the AWS pricing page rather than trusting any number in a blog post, including this one.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The one thing worth remembering<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Effective CloudWatch data pipeline monitoring is built around absence, not around errors. Your SaaS integrations will fail far more often by succeeding at nothing than by throwing an exception, and every default in CloudWatch treats an absent metric as a non-event.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you only change one thing after reading this, add a heartbeat metric with <code>--treat-missing-data breaching<\/code> to your most important pipeline, then go and disable its schedule in a test account and confirm the alarm actually fires. If it does not, you have just found out on your own terms instead of from a stakeholder asking why the numbers look wrong.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Need a second pair of eyes on your pipeline monitoring?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I work with teams running SaaS-to-warehouse pipelines on AWS, usually at the point where something broke quietly and nobody wants that to happen twice. Things I can help with here:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Auditing existing CloudWatch alarms for silent-failure gaps, particularly missing-data configuration and dimension mismatches that leave alarms permanently green.<\/li><li>Instrumenting liveness, volume, freshness and data-quality metrics into pipelines built on Lambda, Glue, Step Functions or self-hosted schedulers.<\/li><li>Designing EMF logging and metric namespaces that stay useful as source and object counts grow, without cardinality running away.<\/li><li>Building composite alarms and routing so one broken pipeline produces one actionable page rather than a cluster of them.<\/li><li>Cutting CloudWatch spend on log ingestion and custom metrics without losing the signals you actually rely on.<\/li><li>Writing Terraform modules that ship monitoring alongside every new source connector, so coverage is structural rather than remembered.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">If you have a pipeline that has gone quiet, send me the alarm configuration and a Logs Insights output from the run in question and I will tell you what it is missing.<\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/www.upwork.com\/freelancers\/~01f15a912ad84a6620\" target=\"_blank\" rel=\"noreferrer noopener\">Work with me on Upwork<\/a><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Your SaaS pipeline will fail far more often by succeeding at nothing than by throwing an exception, and every CloudWatch default treats an absent metric as a non-event. Here are the four signals worth alarming on: liveness, volume, freshness and shape, plus the missing-data traps that leave alarms permanently green.<\/p>\n","protected":false},"author":1,"featured_media":226,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25,24,52],"tags":[98,235,93,186,185,187,169,349,155,234,348,157,13,96,124,233,118,332],"class_list":["post-225","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-computing","category-devops","category-technical-guides","tag-alerting","tag-anomaly-detection","tag-aws","tag-aws-lambda","tag-cloudwatch","tag-cost-optimization","tag-data-engineering","tag-data-freshness","tag-data-integration","tag-data-observability","tag-embedded-metric-format","tag-etl","tag-monitoring","tag-observability","tag-pipeline-design","tag-schema-drift","tag-sre","tag-step-functions","entry","has-media"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>CloudWatch Data Pipeline Monitoring That Catches Silence<\/title>\n<meta name=\"description\" content=\"CloudWatch data pipeline monitoring for SaaS integrations: catch silent failures, stale data and empty runs before a stakeholder reports them.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"CloudWatch Data Pipeline Monitoring That Catches Silence\" \/>\n<meta property=\"og:description\" content=\"CloudWatch data pipeline monitoring for SaaS integrations: catch silent failures, stale data and empty runs before a stakeholder reports them.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/\" \/>\n<meta property=\"og:site_name\" content=\"John Nessime\" \/>\n<meta property=\"article:published_time\" content=\"2026-08-15T18:00:00+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/john-nessime.com\/blog\/wp-content\/uploads\/2026\/08\/cloudwatch-data-pipeline-monitoring.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"627\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"John Nessime\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"John Nessime\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"15 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/\"},\"author\":{\"name\":\"John Nessime\",\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/#\\\/schema\\\/person\\\/ede0b56d0c808f123f57d5d796902105\"},\"headline\":\"CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing\",\"datePublished\":\"2026-08-15T18:00:00+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/\"},\"wordCount\":3192,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/#\\\/schema\\\/person\\\/ede0b56d0c808f123f57d5d796902105\"},\"image\":{\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/08\\\/cloudwatch-data-pipeline-monitoring.png\",\"keywords\":[\"Alerting\",\"Anomaly Detection\",\"AWS\",\"AWS Lambda\",\"CloudWatch\",\"Cost Optimization\",\"Data Engineering\",\"Data Freshness\",\"Data Integration\",\"Data Observability\",\"Embedded Metric Format\",\"ETL\",\"Monitoring\",\"Observability\",\"Pipeline Design\",\"Schema Drift\",\"SRE\",\"Step Functions\"],\"articleSection\":[\"Cloud Computing\",\"DevOps\",\"Technical Guides\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/\",\"url\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/\",\"name\":\"CloudWatch Data Pipeline Monitoring That Catches Silence\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/08\\\/cloudwatch-data-pipeline-monitoring.png\",\"datePublished\":\"2026-08-15T18:00:00+00:00\",\"description\":\"CloudWatch data pipeline monitoring for SaaS integrations: catch silent failures, stale data and empty runs before a stakeholder reports them.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/#primaryimage\",\"url\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/08\\\/cloudwatch-data-pipeline-monitoring.png\",\"contentUrl\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/08\\\/cloudwatch-data-pipeline-monitoring.png\",\"width\":1200,\"height\":627,\"caption\":\"Diagram contrasting a single green SUCCEEDED pipeline status with four separate CloudWatch signals: liveness passing, volume empty, freshness stale and field shape drifting.\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/devops\\\/cloudwatch-data-pipeline-monitoring\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/\",\"name\":\"John Nessime\",\"description\":\"Cloud, DevOps, Data &amp; AI \u2014 Built, Tested, Explained\",\"publisher\":{\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/#\\\/schema\\\/person\\\/ede0b56d0c808f123f57d5d796902105\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/#\\\/schema\\\/person\\\/ede0b56d0c808f123f57d5d796902105\",\"name\":\"John Nessime\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/cropped-jn.png\",\"url\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/cropped-jn.png\",\"contentUrl\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/cropped-jn.png\",\"width\":512,\"height\":512,\"caption\":\"John Nessime\"},\"logo\":{\"@id\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/cropped-jn.png\"},\"description\":\"AWS Certified Solutions Architect helping businesses build reliable cloud, data, reporting, and automation solutions. I help startups, agencies, and growing businesses replace manual processes and disconnected data with practical AWS architectures, clean data pipelines, useful dashboards, and maintainable automation.\",\"sameAs\":[\"https:\\\/\\\/john-nessime.com\\\/blog\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/john-m-nessime\"],\"url\":\"https:\\\/\\\/john-nessime.com\\\/blog\\\/author\\\/johnnessime\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"CloudWatch Data Pipeline Monitoring That Catches Silence","description":"CloudWatch data pipeline monitoring for SaaS integrations: catch silent failures, stale data and empty runs before a stakeholder reports them.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/","og_locale":"en_US","og_type":"article","og_title":"CloudWatch Data Pipeline Monitoring That Catches Silence","og_description":"CloudWatch data pipeline monitoring for SaaS integrations: catch silent failures, stale data and empty runs before a stakeholder reports them.","og_url":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/","og_site_name":"John Nessime","article_published_time":"2026-08-15T18:00:00+00:00","og_image":[{"width":1200,"height":627,"url":"https:\/\/john-nessime.com\/blog\/wp-content\/uploads\/2026\/08\/cloudwatch-data-pipeline-monitoring.png","type":"image\/png"}],"author":"John Nessime","twitter_card":"summary_large_image","twitter_misc":{"Written by":"John Nessime","Est. reading time":"15 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/#article","isPartOf":{"@id":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/"},"author":{"name":"John Nessime","@id":"https:\/\/john-nessime.com\/blog\/#\/schema\/person\/ede0b56d0c808f123f57d5d796902105"},"headline":"CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing","datePublished":"2026-08-15T18:00:00+00:00","mainEntityOfPage":{"@id":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/"},"wordCount":3192,"commentCount":0,"publisher":{"@id":"https:\/\/john-nessime.com\/blog\/#\/schema\/person\/ede0b56d0c808f123f57d5d796902105"},"image":{"@id":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/#primaryimage"},"thumbnailUrl":"https:\/\/john-nessime.com\/blog\/wp-content\/uploads\/2026\/08\/cloudwatch-data-pipeline-monitoring.png","keywords":["Alerting","Anomaly Detection","AWS","AWS Lambda","CloudWatch","Cost Optimization","Data Engineering","Data Freshness","Data Integration","Data Observability","Embedded Metric Format","ETL","Monitoring","Observability","Pipeline Design","Schema Drift","SRE","Step Functions"],"articleSection":["Cloud Computing","DevOps","Technical Guides"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/","url":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/","name":"CloudWatch Data Pipeline Monitoring That Catches Silence","isPartOf":{"@id":"https:\/\/john-nessime.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/#primaryimage"},"image":{"@id":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/#primaryimage"},"thumbnailUrl":"https:\/\/john-nessime.com\/blog\/wp-content\/uploads\/2026\/08\/cloudwatch-data-pipeline-monitoring.png","datePublished":"2026-08-15T18:00:00+00:00","description":"CloudWatch data pipeline monitoring for SaaS integrations: catch silent failures, stale data and empty runs before a stakeholder reports them.","breadcrumb":{"@id":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/#primaryimage","url":"https:\/\/john-nessime.com\/blog\/wp-content\/uploads\/2026\/08\/cloudwatch-data-pipeline-monitoring.png","contentUrl":"https:\/\/john-nessime.com\/blog\/wp-content\/uploads\/2026\/08\/cloudwatch-data-pipeline-monitoring.png","width":1200,"height":627,"caption":"Diagram contrasting a single green SUCCEEDED pipeline status with four separate CloudWatch signals: liveness passing, volume empty, freshness stale and field shape drifting."},{"@type":"BreadcrumbList","@id":"https:\/\/john-nessime.com\/blog\/devops\/cloudwatch-data-pipeline-monitoring\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/john-nessime.com\/blog\/"},{"@type":"ListItem","position":2,"name":"CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing"}]},{"@type":"WebSite","@id":"https:\/\/john-nessime.com\/blog\/#website","url":"https:\/\/john-nessime.com\/blog\/","name":"John Nessime","description":"Cloud, DevOps, Data &amp; AI \u2014 Built, Tested, Explained","publisher":{"@id":"https:\/\/john-nessime.com\/blog\/#\/schema\/person\/ede0b56d0c808f123f57d5d796902105"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/john-nessime.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/john-nessime.com\/blog\/#\/schema\/person\/ede0b56d0c808f123f57d5d796902105","name":"John Nessime","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/john-nessime.com\/blog\/wp-content\/uploads\/2026\/07\/cropped-jn.png","url":"https:\/\/john-nessime.com\/blog\/wp-content\/uploads\/2026\/07\/cropped-jn.png","contentUrl":"https:\/\/john-nessime.com\/blog\/wp-content\/uploads\/2026\/07\/cropped-jn.png","width":512,"height":512,"caption":"John Nessime"},"logo":{"@id":"https:\/\/john-nessime.com\/blog\/wp-content\/uploads\/2026\/07\/cropped-jn.png"},"description":"AWS Certified Solutions Architect helping businesses build reliable cloud, data, reporting, and automation solutions. I help startups, agencies, and growing businesses replace manual processes and disconnected data with practical AWS architectures, clean data pipelines, useful dashboards, and maintainable automation.","sameAs":["https:\/\/john-nessime.com\/blog","https:\/\/www.linkedin.com\/in\/john-m-nessime"],"url":"https:\/\/john-nessime.com\/blog\/author\/johnnessime\/"}]}},"_links":{"self":[{"href":"https:\/\/john-nessime.com\/blog\/wp-json\/wp\/v2\/posts\/225","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/john-nessime.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/john-nessime.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/john-nessime.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/john-nessime.com\/blog\/wp-json\/wp\/v2\/comments?post=225"}],"version-history":[{"count":1,"href":"https:\/\/john-nessime.com\/blog\/wp-json\/wp\/v2\/posts\/225\/revisions"}],"predecessor-version":[{"id":236,"href":"https:\/\/john-nessime.com\/blog\/wp-json\/wp\/v2\/posts\/225\/revisions\/236"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/john-nessime.com\/blog\/wp-json\/wp\/v2\/media\/226"}],"wp:attachment":[{"href":"https:\/\/john-nessime.com\/blog\/wp-json\/wp\/v2\/media?parent=225"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/john-nessime.com\/blog\/wp-json\/wp\/v2\/categories?post=225"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/john-nessime.com\/blog\/wp-json\/wp\/v2\/tags?post=225"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}