Every task green, zero errors, and the dashboard still shows yesterday's numbers. Run status describes your code, not your data. Here is how to instrument freshness, latency and failure rate so your pipeline tells you before an analyst does.
The rule went red at 02:04 and nobody found out until 09:15. Rule evaluation and notification delivery are two different systems, and only one of them is visible in the UI. A practical setup for Grafana alert notifications to email, Slack and Telegram, plus how to detect the day delivery quietly stops.
Most AWS accounts have dozens of CloudWatch alarms and about three anyone trusts. A practical guide to missing-data settings, M-out-of-N evaluation, metric math, composite alarms and mute rules, and how to test the notification path before you need it.
n8n's Prometheus endpoint tells you the process is alive and busy. It does not tell you a workflow stopped producing correct output. This post walks through the three failure families that slip past a default n8n dashboard, the PromQL that catches each one, and how to close the gap with a second source of truth from the executions API.
A Glue job that stops running emits no metrics, so the dashboard stays green and the alert quietly resolves itself. Here is why Grafana monitoring for AWS data pipelines misses that failure, and the heartbeat, metric math and no-data configuration that closes the gap, along with the CloudWatch query costs and IAM boundaries nobody warns you about.
Uptime checks and RUM measure different things, and each one is structurally blind to failures the other catches. A practical breakdown of what synthetic probes miss, why RUM goes quiet during a real outage, and how to combine them without paying for two overlapping platforms.
Your SaaS pipeline will fail far more often by succeeding at nothing than by throwing an exception, and every CloudWatch default treats an absent metric as a non-event. Here are the four signals worth alarming on: liveness, volume, freshness and shape, plus the missing-data traps that leave alarms permanently green.
Green DAGs and wrong numbers are the same morning. This is how to build a data pipeline operations dashboard around the four questions that actually matter, where each signal comes from, and how to turn the panels into alerts that fire before your users notice.
The error count was zero every day for three weeks. So was the invocation count. A stopped integration and a healthy one produce identical graphs, and every CloudWatch default is tuned to stay quiet when data stops arriving. Four signals worth emitting, and the alarm config that actually fires.
Metrics tell you that something is wrong. Traces tell you where. Logs tell you why. Use them in that order and each step narrows the search — start with logs and you are grepping a firehose. Here is what each one is actually for, and the field that connects all three.