Every task green, zero errors, and the dashboard still shows yesterday's numbers. Run status describes your code, not your data. Here is how to instrument freshness, latency and failure rate so your pipeline tells you before an analyst does.
The rule went red at 02:04 and nobody found out until 09:15. Rule evaluation and notification delivery are two different systems, and only one of them is visible in the UI. A practical setup for Grafana alert notifications to email, Slack and Telegram, plus how to detect the day delivery quietly stops.
Most AWS accounts have dozens of CloudWatch alarms and about three anyone trusts. A practical guide to missing-data settings, M-out-of-N evaluation, metric math, composite alarms and mute rules, and how to test the notification path before you need it.
n8n's Prometheus endpoint tells you the process is alive and busy. It does not tell you a workflow stopped producing correct output. This post walks through the three failure families that slip past a default n8n dashboard, the PromQL that catches each one, and how to close the gap with a second source of truth from the executions API.
High I/O wait is a symptom, not a diagnosis. It rises when nothing is wrong and stays flat when something is badly wrong, and it only sees one layer of the storage stack. This is how to read iowait and PSI correctly, then work through the five failure families that actually produce storage stalls: device saturation, memory pressure, synchronous writes, throttling, and latency that lives above the block layer entirely.
A Glue job that stops running emits no metrics, so the dashboard stays green and the alert quietly resolves itself. Here is why Grafana monitoring for AWS data pipelines misses that failure, and the heartbeat, metric math and no-data configuration that closes the gap, along with the CloudWatch query costs and IAM boundaries nobody warns you about.
Uptime checks and RUM measure different things, and each one is structurally blind to failures the other catches. A practical breakdown of what synthetic probes miss, why RUM goes quiet during a real outage, and how to combine them without paying for two overlapping platforms.
Your SaaS pipeline will fail far more often by succeeding at nothing than by throwing an exception, and every CloudWatch default treats an absent metric as a non-event. Here are the four signals worth alarming on: liveness, volume, freshness and shape, plus the missing-data traps that leave alarms permanently green.
Green DAGs and wrong numbers are the same morning. This is how to build a data pipeline operations dashboard around the four questions that actually matter, where each signal comes from, and how to turn the panels into alerts that fire before your users notice.
Third-party API spend is the one production signal with no error rate attached to it, which is why it creeps up quietly. This is a working engineer's guide to reducing SaaS API costs by changing the shape of your calls: reading the billing unit before you optimise anything, killing pointless polling with conditional requests and webhooks, collapsing N+1 patterns, caching with stampede protection and per-tenant keys, stopping your own retry amplification, and attributing spend so you can prove the work paid off.