Every task green, zero errors, and the dashboard still shows yesterday's numbers. Run status describes your code, not your data. Here is how to instrument freshness, latency and failure rate so your pipeline tells you before an analyst does.
The rule went red at 02:04 and nobody found out until 09:15. Rule evaluation and notification delivery are two different systems, and only one of them is visible in the UI. A practical setup for Grafana alert notifications to email, Slack and Telegram, plus how to detect the day delivery quietly stops.
n8n's Prometheus endpoint tells you the process is alive and busy. It does not tell you a workflow stopped producing correct output. This post walks through the three failure families that slip past a default n8n dashboard, the PromQL that catches each one, and how to close the gap with a second source of truth from the executions API.
High I/O wait is a symptom, not a diagnosis. It rises when nothing is wrong and stays flat when something is badly wrong, and it only sees one layer of the storage stack. This is how to read iowait and PSI correctly, then work through the five failure families that actually produce storage stalls: device saturation, memory pressure, synchronous writes, throttling, and latency that lives above the block layer entirely.
A Glue job that stops running emits no metrics, so the dashboard stays green and the alert quietly resolves itself. Here is why Grafana monitoring for AWS data pipelines misses that failure, and the heartbeat, metric math and no-data configuration that closes the gap, along with the CloudWatch query costs and IAM boundaries nobody warns you about.
Uptime checks and RUM measure different things, and each one is structurally blind to failures the other catches. A practical breakdown of what synthetic probes miss, why RUM goes quiet during a real outage, and how to combine them without paying for two overlapping platforms.
Green DAGs and wrong numbers are the same morning. This is how to build a data pipeline operations dashboard around the four questions that actually matter, where each signal comes from, and how to turn the panels into alerts that fire before your users notice.
The error count was zero every day for three weeks. So was the invocation count. A stopped integration and a healthy one produce identical graphs, and every CloudWatch default is tuned to stay quiet when data stops arriving. Four signals worth emitting, and the alarm config that actually fires.
Loki was fast until someone added one more label. Now queries time out during incidents and the index no longer fits in memory. Here's the model you have to internalise, what replaced Promtail, and how to keep labels, retention and storage costs under control.
Metrics tell you that something is wrong. Traces tell you where. Logs tell you why. Use them in that order and each step narrows the search — start with logs and you are grepping a firehose. Here is what each one is actually for, and the field that connects all three.