The same output showed roughly 11 GB in the available column. Nothing was wrong, and nothing had been wrong for months. The only thing broken was the alert that fired.

Low free memory on Linux is one of the most reliably misread signals in server operations. It looks like an emergency, it triggers scaling decisions, and it is almost always noise. This post covers what free is really telling you, which four signals actually indicate memory trouble, the one situation where a low free number genuinely matters, and what to put in your alert rules instead.

What the free command is actually reporting

free does not measure anything. It parses /proc/meminfo and prints a handful of those fields in columns. Every number it shows comes straight from the kernel and is readable directly if you want to skip the formatting.

A typical reading on a busy server looks roughly like this:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       3.4Gi       231Mi       128Mi        12Gi        11Gi
Swap:          2.0Gi          0B       2.0Gi

Read that left to right and the story is boring. Applications hold a few gigabytes. The kernel filled almost everything else with page cache and buffers, because every file that gets read or written passes through memory on the way, and directory metadata lands in reclaimable slab. What is left over, genuinely untouched, is the free column.

That last part is the bit people miss. Free memory is not a reserve. It is waste. A page holding a cached copy of a file your application reads every few seconds is doing useful work. A page sitting empty is doing nothing, and the kernel converts it into cache the moment it has a reason to. A server that has been up for a week and still shows plenty of free memory is a server that is barely touching its disk.

If you want the raw source rather than the formatted view:

$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SReclaimable|SwapTotal|SwapFree|Dirty):' /proc/meminfo

Two options are worth knowing. free -w switches to wide mode and splits buffers and cache into separate columns, which helps when attributing memory to a specific I/O pattern. free -h -s 5 reprints every five seconds, a crude but effective trend view while you reproduce a problem.

Signal one: available memory, not free memory

The available column maps to MemAvailable in /proc/meminfo, and it is the kernel’s own estimate of how much memory a new workload could allocate without pushing the system into swap. It has been present since kernel 3.14, so every distribution you are likely to be running exposes it.

What matters is that it is not free plus cache. That naive sum is what people calculated before MemAvailable existed, and it overstates the truth. The kernel starts from free pages, subtracts the reserves it refuses to dip below, then adds back the portion of file-backed page cache and reclaimable slab it believes it could actually release. Both additions are capped, because some cache is pinned and some reclaimable slab objects are in use.

The kernel documentation is explicit that this is an estimate whose accuracy varies from system to system, so treat it as a strong hint rather than an exact figure. It is still far better than anything you can compute yourself, and it is the number your dashboards should be built on.

If you change one thing after reading this: point your memory alerts at available, not at free. Everything else in this post is refinement on top of that.

Signal two: swap activity, not swap usage

The second most common false alarm is swap. Someone sees a few hundred megabytes of swap in use and concludes the machine is thrashing. It usually isn’t.

Swap usage is a stock: pages moved to disk at some point, possibly during a backup weeks ago, and never needed since. Leaving them there is correct, and pulling them back to satisfy a dashboard would evict something useful. Swap activity is a flow, and that is the part that hurts, because a page fault serviced from disk is orders of magnitude slower than one serviced from RAM.

vmstat gives you the flow. The si and so columns are swap-in and swap-out, in blocks per second over the sampling interval:

$ vmstat 5 6

Discard the first row. vmstat reports averages since boot on its first line, which is almost never what you want. Read the rows after that. If si and so are both consistently non-zero, the machine is simultaneously reading pages back to serve current work and writing pages out to make room for new allocations. That is thrashing, and it is a real signal with no ambiguity attached.

A single non-zero sample is not that. Brief swap activity during a log rotation or a backup window is normal. Sustained activity across several consecutive samples is what to escalate.

Signal three: memory pressure, which is the one that catches things early

Pressure Stall Information is the signal I reach for first now, and it is badly underused. It has been in the kernel since 4.20 and it answers a question none of the other tools answer directly: how much time are my processes losing because memory is contended?

$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

The some line is the share of wall-clock time in which at least one task was stalled waiting on memory. The full line is the share in which every non-idle task was stalled, meaning nothing useful happened at all. The averages cover ten seconds, one minute and five minutes, and total is a cumulative counter in microseconds that updates continuously, so it catches a two-second stall the running averages smooth away entirely.

Here is why this beats a percentage threshold. Utilization tells you how much of a resource is consumed; pressure tells you whether that consumption is costing you anything. A machine can sit at high utilization for months with zero pressure, which means the sizing is fine. It can also sit at moderate utilization while full climbs, which means reclaim is working hard and latency is already suffering with no threshold crossed.

With cgroup v2 mounted, the same data exists per cgroup in each directory’s memory.pressure file, so you can attribute pressure to a specific service rather than the whole machine. Prometheus node_exporter collects the system-level PSI counters, and if you already ship metrics to something like Grafana Cloud these graph well beside your latency panels.

Signal four: the OOM log, which is the only unambiguous one

When memory really does run out, the kernel writes a record. It does not do this quietly and it does not do it conditionally. If there is no OOM record, no process was killed for memory, full stop.

$ sudo journalctl -k --since "24 hours ago" | grep -i "oom"
$ sudo dmesg -T | grep -i "killed process"

The kill line names the victim and reports its memory breakdown at the moment of death: total virtual size, anonymous resident set, file-backed resident set, shared memory resident set, and the oom_score_adj that was in effect. The line above it, where a process “invoked oom-killer”, tells you which allocation triggered the whole event, and that process is frequently not the one that got killed.

That distinction is the most useful thing in the log. The kernel picks a victim by badness score, driven largely by how much memory a process holds relative to what is available to it, adjusted by oom_score_adj. A slow leak in a small helper can push the machine over the edge and get your database killed instead, because the database is simply the largest thing running. Read both lines before deciding who the culprit is.

If you have a service that should survive at almost any cost, systemd exposes OOMScoreAdjust= in the unit file, which persists across restarts in a way that writing to /proc/<pid>/oom_score_adj does not. Use it sparingly. Protecting a process that is itself leaking just moves the crash somewhere less convenient.

When low free memory on Linux really is the problem

Everything above assumes you are looking at a whole machine. Inside a container, the accounting changes, and this is the failure mode that is invisible until it isn’t.

The cgroup memory controller charges page cache to the cgroup that caused it. Clean, evictable, perfectly reclaimable cache still counts toward memory.current, and that is what gets compared against memory.max. A container reading a large file has just charged that file to its own budget.

$ cat /sys/fs/cgroup/memory.current
$ cat /sys/fs/cgroup/memory.max
$ grep -E '^(anon|file|slab_reclaimable) ' /sys/fs/cgroup/memory.stat

The split in memory.stat is what you actually want. anon is memory your process genuinely holds; file is page cache charged to you that the kernel can take back under pressure. A container at its limit on mostly file is in a very different position from one at its limit on mostly anon.

Two things follow from this that catch people out repeatedly:

• Reclaim is not instant. Under heavy pressure the reclaim path itself runs slowly, so a container can be OOM-killed while still holding clean page cache that was theoretically available.
• Running free inside a container lies to you. Unless something virtualizes /proc, you are reading the host’s /proc/meminfo, which knows nothing about your limit. The cgroup files are the only truth here.

cgroup v2 also has memory.high, a soft limit. Crossing it triggers reclaim and throttling rather than a kill, which turns a hard failure into a slow one. That is usually the better trade, and it is worth setting deliberately rather than leaving it at its default.

What to alert on

The alert on the ticket at the top of this post was watching the wrong number. Here is what a memory alert set looks like when it watches the right ones:

• Available memory below a percentage of total, sustained. Ten percent for ten minutes is a reasonable starting shape. A single thirty-second dip is noise and should never page anyone.
• Sustained swap-in and swap-out together. Rate of change on the swap counters, not the absolute amount of swap in use.
• Memory PSI rising. The some line moving off the floor is your early warning. The full line moving at all is your late one.
• Any OOM kill, ever. This one is binary and should always generate a ticket, even if the service recovered on its own.
• Per-container memory events. The cgroup memory.events file counts limit hits, and a rising count is a limit that needs revisiting.

Notice that free memory does not appear anywhere in that list. There is no threshold on the free column that produces a useful alert, because the number is a byproduct of the kernel’s caching behaviour rather than a measure of anything you care about.

Troubleshooting: which tool answers which question

When a memory question lands on you, the order you check things in matters more than the tools.

1. Did anything actually die? Check the kernel log for OOM records first. If there are none, this is a performance question and not a capacity failure, which changes everything about what you do next.
2. Is the machine stalling? Read /proc/pressure/memory. Zero pressure with low free memory means you are done. Go close the ticket.
3. Is it swapping right now? Run vmstat 5 for half a minute and watch si and so. Ignore the first row.
4. Who is holding the memory? ps aux --sort=-%mem | head -15 gets the shortlist. Resident set size double-counts shared pages, so summing the column overstates the total.
5. Is it the kernel rather than a process? If the process list does not account for the usage, check slabtop and SUnreclaim in /proc/meminfo. Unreclaimable slab growth is a different problem, usually a driver, filesystem or network workload creating enormous numbers of kernel objects.
6. Is it growing? A leak is a slope, not a level, and one reading tells you nothing. This is the whole argument for having the graph before you need it.

One more that saves time on crowded boxes: resident set size is the wrong measure of what a process really costs, because shared libraries are counted against every process that maps them. Proportional set size divides shared pages among the processes sharing them, and tools reporting PSS give a far more honest answer when you are deciding what to move off a busy VPS.

Common mistakes

• Dropping caches to make the number look better. Writing to /proc/sys/vm/drop_caches discards clean page cache and reclaimable slab. The kernel documentation says plainly that using it outside testing or debugging is not recommended, because everything dropped has to be re-read from disk. You trade a nicer-looking free output for an I/O spike and worse latency. It is a benchmarking tool, not an operations tool.
• Setting vm.swappiness to zero as a reflex. It does not disable swap and it does not make the machine faster. It makes the kernel strongly prefer evicting page cache over swapping anonymous pages, which on a database host means throwing away exactly the cache you wanted. Change it for a measured reason, not by default.
• Running with no swap at all. Swap is not just an overflow area. Without it the kernel loses the option of paging out genuinely cold anonymous memory, which means the only lever left under pressure is evicting cache or killing something. A modest swap file gives reclaim somewhere to go.
• Summing RSS across processes and comparing it to total RAM. Shared pages are counted once per process. The sum will exceed reality, sometimes dramatically.
• Resizing the instance before finding the slope. More RAM buys time against a leak. It does not fix one, and the bill arrives every month afterwards.

Best practices

• Graph available memory, swap rate and memory PSI on one panel. Together they read as a story; individually each one misleads.
• Keep the raw counters, not just derived percentages. A percentage of an instance size that has since changed cannot answer “was this happening last week”.
• Set container limits from observed anon usage plus headroom, not a round number someone picked, and put memory.high below memory.max so pressure shows up as slowness before it shows up as a kill.
• Alert on rate of change for anything that can leak. The steady climb is the signal; the level it reached is just when you happened to look.
• Right-size deliberately. On providers such as Contabo or InterServer, where RAM is a meaningful share of the monthly bill, the gap between a real requirement and a panicked upgrade is money you keep every month.
• Write the explanation down where your team will find it. Every organization rediscovers the free-versus-available confusion independently, usually at three in the morning.

Frequently asked questions

Why is free memory so low on my Linux server?

Because the kernel used it for page cache and buffers, which is the intended behaviour. Any file read or written passes through memory, and the kernel keeps those pages in case they are needed again. It reclaims them automatically the moment an application needs the space. Check the available column instead.

What is the difference between free and available in free -h?

Free is memory that is completely unused. Available is the kernel’s estimate of how much a new process could allocate without swapping: free memory plus the portion of cache and reclaimable slab it believes it could release. Available is almost always much larger, and it answers the question you are actually asking.

Should I clear the page cache to free up memory?

No. The kernel reclaims cache automatically under pressure, and more selectively than a blanket drop. Writing to drop_caches forces everything to be re-read from disk, and the kernel documentation advises against using it outside testing or debugging. Doing it on a schedule is a strong hint something else is misconfigured.

Is swap being used a bad sign?

Not by itself. Pages that were swapped out during a quiet period and never needed again are fine where they are. What matters is swap traffic: sustained non-zero si and so in vmstat at the same time means the machine is thrashing. Alert on the rate, not the amount.

How do I tell if a process was killed for running out of memory?

Search the kernel log. journalctl -k filtered for “oom”, or dmesg -T filtered for “killed process”, shows a record naming the process, its PID and its memory breakdown at the time. The kernel always logs this, so if there is no record the process died for some other reason.

Why does my container get OOM-killed when memory looks fine inside it?

Because free inside a container usually reads the host’s memory information, not the cgroup limit being enforced. Read memory.current, memory.max and memory.stat instead. Page cache charged to the container counts toward its limit, so a read-heavy workload can hit the ceiling without any process holding much anonymous memory.

What is a good memory alert threshold?

Available memory below roughly ten percent of total, sustained for ten minutes, is a defensible starting point to tune against your own workload. Pair it with memory PSI and an unconditional alert on any OOM kill. A threshold on the free column, at any value, will page you for nothing.

Conclusion

Low free memory on Linux is the expected steady state of a healthy, working server. The kernel fills memory with cache because empty memory earns nothing, and hands it back the instant an application asks. The free column is a byproduct of that behaviour, not a health metric, and alerting on it produces exactly the kind of urgent ticket that wastes an afternoon and ends in a change request nobody needed.

The one thing worth remembering: free tells you what is idle, available tells you what you can have, and pressure tells you whether any of it is costing you. Watch the second and third, log the OOM events, and read the cgroup files rather than free whenever a container is involved.

Need help reading your memory signals?

Most of the memory work I get asked for is not “add more RAM”. It is working out whether the number someone is worried about means anything. I can help with:

• Rebuilding memory alerts around available memory, swap rate and PSI so they stop paging on normal cache behaviour
• Diagnosing repeated OOM kills, including the cases where the process being killed is not the one causing the problem
• Setting container and pod memory limits from observed anonymous usage rather than guesses, with a soft limit so pressure surfaces before a kill does
• Separating a genuine leak from normal cache growth using retained metrics rather than a single reading
• Right-sizing servers and instances honestly, up or down, based on what the workload actually needs
• Building the dashboard that turns the next memory question into a five-minute conversation instead of an incident

If you have a free -h output, a vmstat capture or an OOM log line that is bothering you, send it over and I will tell you what it means.

The post Low Free Memory on Linux Isn’t a Problem. Here’s What Actually Is. appeared first on John Nessime.

The crontab entry is still there. The script still works when you run it by hand. Nothing in the monitoring went red, because nothing was watching whether the job ran at all. It was watching whether the server was up.

That is the real question behind cron vs systemd timers, and it has almost nothing to do with syntax. Both tools can run a command at 2 a.m. The difference is what happens on the night the command doesn’t run, and how long it takes anyone to find out.

This post compares the two by failure family rather than by feature list: missed runs, overlapping runs, environment surprises, output that goes nowhere, and jobs that eat the machine. Then a decision procedure, the same job written both ways, and how to debug each one when it goes quiet.

The failure that actually decides it

Scheduled jobs fail silently far more often than they fail loudly. A job that crashes with a stack trace gets fixed the same week. A job that stopped firing six weeks ago gets found during an incident.

Cron’s default answer to “what happened?” is to mail the job’s output to the owning user. On a server with no local mail transfer agent configured, which is most servers built in the last decade, that output is generated and then discarded. The daemon logs that it started the command. It does not log what the command printed, and in most implementations it does not log the exit status either.

So you get a log line saying cron ran your backup, every night, forever, including on the nights the backup wrote nothing. That line is the trap. It looks like evidence.

systemd timers hand this to journald instead. Standard output and standard error from the triggered service land in the journal, tagged with the unit name, alongside the exit status and the runtime. That is the single biggest practical difference between the two, and it is the reason most of my new scheduled work goes into timers.

Neither one alerts you. Both will happily not run for a month while you sleep well. Observability is not monitoring, and I’ll come back to that.

Where cron still wins

Cron is one line in one file. There is no unit to reload, no second file to keep in sync, no install section to get wrong. crontab -e, five fields, a command, done. For a job that trims a log directory on a box you own, that economy is not nothing.

It also runs where systemd does not. BSD hosts, Alpine images, containers, appliance firmware, embedded boxes, older enterprise systems still on a legacy init. If your configuration management has to cover a fleet that is not uniformly systemd, cron is the common denominator and it is not close.

Per-user scheduling is genuinely simpler too. A user runs crontab -e and owns their own jobs, with no root involvement and no unit files in /etc. On shared hosting and on control panels like DirectAdmin or cPanel, this is the only model available, and the panel writes the crontab for you.

Where cron loses

• No dependency ordering. Cron cannot wait for the network to be up or the database to be ready. It fires at the wall clock time and hopes.
• No overlap protection. If the 2 a.m. run is still going at 3 a.m., the 3 a.m. run starts anyway, on top of it.
• No resource containment. A runaway job competes with production for CPU, memory and disk with nothing standing between them.
• Minute resolution, and no catch-up if the machine was off.
• A syntax with sharp edges that fail quietly, which is the next section.

Where systemd timers win

A timer separates when from what. The .timer unit holds the schedule. The .service unit holds the job. That split feels like bureaucracy until the first time you need to run the job right now, out of band, without touching the schedule. Then it is just systemctl start myjob.service, and the run is logged exactly like a scheduled one.

Everything systemd knows how to do to a service, it will do to a scheduled job. Ordering after other units. Restart-on-failure. Runtime caps. Memory and CPU limits through cgroups. A dedicated user. A private /tmp. A failure handler unit that fires when the job exits non-zero.

You also get a schedule you can interrogate before it bites you. systemctl list-timers prints every timer on the box with its next and last elapse. systemd-analyze calendar takes an expression and tells you exactly when it would fire, which is the closest thing either scheduler has to a unit test.

Where systemd timers lose

• Two files, a daemon-reload and an enable for every job. That is real friction when you have forty small jobs.
• The calendar syntax is more expressive than cron’s and harder to eyeball. You will reach for systemd-analyze calendar more often than you expect.
• Portability is gone. These units run on systemd hosts and nowhere else.
• User timers need lingering enabled, or they stop when the user logs out and never start at boot. This surprises people once, memorably.
• The journal is not a log file. If your log shipping only tails /var/log, timer output is invisible to it until you point the shipper at journald.

Cron vs systemd timers, by failure family

The machine was off at 2 a.m.

Cron has no memory. If the box was down, rebooting, or suspended when the schedule came round, that run is gone. Nothing catches it up and nothing records that it was skipped.

The traditional fix is anacron, which tracks the last successful run per job in a timestamp file and runs the job shortly after boot if the interval has lapsed. It is why the scripts in /etc/cron.daily still run on a laptop that is closed every night. The catch is that anacron works in whole days and only covers the cron.daily, cron.weekly and cron.monthly directories by default. It does not help your custom job at 02:00.

systemd folds that behaviour into a single directive:

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

With Persistent=true, systemd writes a timestamp under /var/lib/systemd/timers/ each time the timer fires. On boot it compares that stamp against the schedule, and if a run was missed it triggers immediately. This matters more than it sounds on any VPS you reboot for kernel updates, and a great deal on hosts at providers like Contabo or InterServer where you take maintenance windows on their timetable rather than yours.

One caveat worth knowing: “immediately” means immediately. If you have twelve persistent timers and the box has been off for a week, all twelve fire within seconds of boot, on a machine that is still starting services. Pair Persistent=true with RandomizedDelaySec= so the catch-up is spread out.

Two copies of the same job

This is the one that corrupts data. A sync job normally takes four minutes and runs every five. One night the remote end is slow, the run takes eleven minutes, and cron starts two more copies while the first is still writing.

Cron will not stop this. You have to do it yourself, and the standard tool is flock from util-linux:

*/5 * * * * /usr/bin/flock -n /var/lock/sync.lock /usr/local/bin/sync.sh

-n means fail immediately rather than queue behind the running copy. Without it you build a backlog of processes all waiting for the same lock, which turns an overlap problem into a fork bomb with a slow fuse.

systemd gets this for free from its own model. A unit is either active or it isn’t. If sync.service is still running when the timer elapses, there is no second copy to start, because starting an already-active unit is a no-op. You do not have to remember anything.

One default is worth knowing. A calendar timer schedules its next elapse from the previous trigger time, so if a run overran, that next elapse is already in the past and the service fires again the moment it finishes. Newer systemd releases add a DeferReactivation= boolean that schedules from when the service went inactive instead, so an overrunning job waits for the next real slot. Check your distribution’s systemd version before relying on it.

The environment your job actually gets

“It works when I run it manually” is the single most common scheduled-job bug, and both schedulers cause it, for the same reason: neither one gives your job an interactive login shell.

Cron runs the command with a short, hardcoded environment. Your shell profile is not sourced. Your PATH is much shorter than the one you tested with, which is why a script that calls docker, wp, aws or a version-manager shim works from your terminal and fails from cron. The working directory is the user’s home, not wherever you happened to be standing.

Cron adds one hazard that is entirely its own. Inside a crontab, the percent sign is a metacharacter: it becomes a newline, and everything after the first one is fed to the command as standard input. So this looks fine and is broken:

# Truncated at the first %. The redirect never happens.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

# Escaped. This is what you meant.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

There is no warning. The line parses, cron runs the truncated fragment, and the log says the job started. Anything with a date + format string in a crontab deserves a second look.

systemd services get a minimal environment too, so the class of bug is identical. What differs is that the fix is declarative and lives with the unit rather than being hidden inside a wrapper script:

[Service]
Type=oneshot
User=deploy
WorkingDirectory=/srv/app
Environment=PATH=/usr/local/bin:/usr/bin:/bin
EnvironmentFile=/etc/myapp/backup.env
ExecStart=/usr/local/bin/backup.sh

The honest summary: absolute paths for every binary solve most of this on either scheduler. The rest is solved by never putting logic in the schedule. Put it in a script and schedule the script.

Everything firing at once

Write 0 0 * * * on fifty hosts and fifty hosts hit your backup target, your package mirror or your API at exactly midnight. The classic cron answer is to hash the hostname into a minute offset in configuration management, which works and which every team reinvents.

systemd has two directives here and they pull in opposite directions, which is the part people get wrong:

• RandomizedDelaySec= spreads timers apart. It adds a delay between zero and the value you set, so identical schedules on many hosts land at different moments.
• AccuracySec= pulls timers together. It lets systemd shift the firing time within a window to coalesce wakeups and save power. It defaults to one minute, which is why a timer set to 02:00:00 often fires a few seconds late.

If you want a timer to fire close to the stated second, lower AccuracySec=. If you want to stagger a fleet, raise RandomizedDelaySec=. Setting only AccuracySec= to a large value and expecting it to spread load is the mistake, because coalescing is the opposite of spreading.

A job that eats the box

Cron has no answer here beyond nice and ionice in front of the command, and a script that hangs forever will hang forever. A systemd service is a cgroup, so the limits are declarative and enforced:

[Service]
Type=oneshot
MemoryMax=512M
CPUQuota=40%
IOWeight=20
TimeoutStartSec=30min
OnFailure=notify-failure@%n.service
ExecStart=/usr/local/bin/reindex.sh

The two directives that earn their keep on a shared box are TimeoutStartSec=, which kills a oneshot job that has hung, and OnFailure=, which starts another unit when this one exits non-zero. That second one is how you get a scheduled job to page you without wiring an alerting call into every script.

The same job written both ways

A nightly database dump that must not overlap, must not run before the database is up, and must catch up if the host was rebooted. In cron:

0 2 * * * /usr/bin/flock -n /var/lock/db-backup.lock /usr/local/bin/db-backup.sh >> /var/log/db-backup.log 2>&1

That handles overlap and captures output. It does not handle “the database wasn’t ready” or “the host was down”, and the log file is now your problem to rotate.

The systemd version is two files. First the service, which is the job:

# /etc/systemd/system/db-backup.service
[Unit]
Description=Nightly application database dump
After=network-online.target mariadb.service
Wants=network-online.target

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/db-backup.sh
TimeoutStartSec=45min

Then the timer, which is only the schedule:

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run the nightly database dump

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=15min

[Install]
WantedBy=timers.target

Because the two units share a base name, you do not need a Unit= line. systemd pairs db-backup.timer with db-backup.service by convention. Add Unit= only when the names differ.

Then the part people skip, which is verifying it before walking away:

# Check the units parse before you enable anything
systemd-analyze verify /etc/systemd/system/db-backup.*

# Confirm the calendar expression means what you think
systemd-analyze calendar --iterations=5 "*-*-* 02:00:00"

systemctl daemon-reload
systemctl enable --now db-backup.timer

# Next and last elapse for this timer
systemctl list-timers db-backup.timer

# Run it now, out of band, without touching the schedule
systemctl start db-backup.service
journalctl -u db-backup.service -n 50 --no-pager

Note that you enable the timer, not the service. Enabling the service instead would try to run the job at every boot, which is a memorable way to learn the difference.

How I’d decide

Work down this list and stop at the first line that applies. It resolves most cases in under a minute.

1. The host might not be systemd, now or later. Use cron. Portability beats every other consideration on this list.
2. The job is inside a container. Use neither. Use your orchestrator’s scheduler, or a timer on the host that invokes the container.
3. A missed run matters, or the job must not overlap, or it needs to wait for another service. Use a timer. Every one of those is a directive rather than a wrapper script.
4. The job is heavy: a dump, a reindex, an import, anything that can pin a core or fill memory. Use a timer, for the cgroup limits and the runtime cap.
5. You need to see output after the fact without building a logging path first. Use a timer. The journal is already there.
6. You need sub-minute scheduling. Use a timer with a monotonic directive. Cron’s floor is one minute.
7. Otherwise, cron is fine, and one line beats two files.

The migration question answers itself from that. Do not convert forty working crontab lines in a weekend. Write new scheduled work as timers, convert the jobs that have actually burned you, and leave the rest. Mixing both on one host is normal and supported. Just never schedule the same job in both places, which is easier to do than it sounds when a configuration management run and a manual edit disagree.

Diagnosing a job that never fired

Cron

Establish what actually happened before you theorise. The daemon logs every job it starts, so the log tells you which of three situations you are in: cron never tried, cron tried and the command failed, or cron never saw the entry at all.

# Is the daemon running at all?
systemctl status crond      # RHEL, AlmaLinux, Rocky
systemctl status cron       # Debian, Ubuntu

# What did it start, and when?
journalctl -u crond --since "24 hours ago"
grep CRON /var/log/syslog | tail -20     # Debian family
tail -50 /var/log/cron                   # RHEL family

# Whose crontab are you actually looking at?
crontab -l -u deploy
ls -l /etc/cron.d/

Then work through the usual suspects, roughly in order of how often they turn out to be the cause:

• Wrong crontab. The job is in root’s crontab and you are reading the deploy user’s, or it’s in /etc/cron.d where there is an extra user field before the command.
• PATH, or an unescaped percent sign. Dump what the job really sees with a temporary entry like * * * * * env > /tmp/cron-env.txt 2>&1, then compare it against your shell.
• Permissions. The script is not executable, or the crontab owner cannot read it.
• The schedule is not what you think. When both day-of-month and day-of-week are restricted, cron runs the job when either matches, not both. 0 0 15 * 1 fires on the 15th and on every Monday.
• The command captured no output, so a real error was mailed into the void. Append >> /var/log/myjob.log 2>&1 and wait for the next run.

systemd

# Is the timer loaded and scheduled? --all shows inactive ones too
systemctl list-timers --all

# Timer state, last trigger, next elapse
systemctl status db-backup.timer

# What the job printed, and how it exited
journalctl -u db-backup.service --since "7 days ago"

# Prove the schedule independently of the unit
systemd-analyze calendar --iterations=3 "Mon..Fri *-*-* 06:30:00"

• The timer is not enabled. Creating the files does nothing. It needs daemon-reload and enable --now.
• You enabled the service instead of the timer. Check with systemctl is-enabled on both.
• A user timer with no lingering. Under systemctl --user, the manager stops when the session ends unless you run loginctl enable-linger for that user.
• The calendar expression is valid but wrong. Valid syntax that never matches will not error. systemd-analyze calendar is how you find out.
• The service failed before your script ran. A missing User=, an ExecStart path that does not exist, a WorkingDirectory that is not there. The journal names the reason.

Arguments that don’t survive contact

“Cron is deprecated.” It isn’t. It is packaged and supported on every mainstream distribution, and control panels, hosting providers and countless application installers write crontab entries as a matter of course. Nobody is removing it.

“Timers are too complicated for small jobs.” Two files is more typing, not more complexity. The complexity that matters is the wrapper script you write to get locking, logging and a runtime cap out of cron, and how well the next person understands it. Compare like for like.

“The journal means I have logging sorted.” The journal is a ring buffer with a size cap. If nobody set SystemMaxUse= deliberately and nobody ships journal entries anywhere else, your evidence has a retention window you never chose. Check it before you rely on it.

“Switching to timers will stop jobs failing silently.” This is the one I’d push back on hardest. Timers make failures visible. They do not make them noticed. Neither scheduler will ever tell you that a job did not run, because absence is not an event. That needs a dead man’s switch: the job pings a heartbeat endpoint on success, and something external alerts when the ping stops. Healthchecks.io, Cronitor and Better Stack all sell exactly this, and Healthchecks is open source if you would rather self-host it next to your own Grafana. If you already run Prometheus, writing a completion timestamp through the node exporter’s textfile collector and alerting on its age gets you the same property without another vendor.

“Just move everything to timers.” A mass migration of working jobs is a lot of change for no new capability, and every converted job is a chance to fat-finger a path. Convert on cause, not on principle.

Frequently asked questions

Are systemd timers better than cron?

For production jobs on a systemd host, generally yes: logging, dependency ordering, overlap protection, catch-up and resource limits are all built in rather than bolted on. For a one-line job on a box you own, cron’s simplicity is a real advantage and the timer buys you nothing you will use.

Is cron deprecated or being removed?

No. Cron implementations are still packaged and maintained across the major distributions. Some distributions do not install a cron daemon in minimal images by default, which is a packaging decision rather than a deprecation, so check with your package manager before assuming a fresh host has one.

Can cron and systemd timers coexist on the same server?

Yes, and most real servers run both. The cron daemon itself is usually managed as a systemd service. The only rule is that a given job should be scheduled in exactly one place. Double-scheduling is what produces the “why did this run twice” ticket.

How do I see when a systemd timer will next run?

systemctl list-timers shows next and last elapse for all active timers; add --all to include inactive ones. To test an expression without creating a unit, use systemd-analyze calendar "Mon..Fri *-*-* 06:30:00" with --iterations=N for more fire times.

How do I stop a cron job from overlapping itself?

Wrap it in flock -n with a dedicated lock file. The -n flag makes the new run exit straight away instead of queuing. Do not use a PID file you wrote yourself; it will leak a stale lock the first time the job is killed, and then the job stops running forever.

Do systemd timers work inside Docker containers?

Not usefully. A container normally runs a single process without systemd as PID 1, so there is no service manager to own a timer. Schedule from outside the container instead: a host timer that runs the container, or the scheduler your platform provides. Podman is the exception worth knowing, since it can generate systemd units for containers and the host’s systemd drives them.

How do I convert a crontab line to a systemd timer?

Move the command into a Type=oneshot service with an absolute ExecStart, then translate the five cron fields into an OnCalendar= expression of the form DayOfWeek Year-Month-Day Hour:Minute:Second. Verify the expression with systemd-analyze calendar, enable the timer, then remove the crontab line only after you have seen a successful run in the journal.

The one thing worth remembering

Cron vs systemd timers is not a contest between an old tool and a new one. Cron is a scheduler. A systemd timer is a scheduler attached to a supervisor, and the supervisor is what you are actually buying: dependency ordering, overlap protection, catch-up, resource limits, and output that lands somewhere you can read it.

Pick timers when a missed or duplicated run costs you something. Pick cron when it doesn’t. And whichever you pick, add a heartbeat, because the failure that hurts is not the job that crashes. It’s the job that stopped running while everything stayed green.

Need a second pair of eyes on your scheduled jobs?

Scheduling is one of those areas where the problem is rarely the syntax. It’s the job nobody has checked since the person who wrote it left. This is the kind of work I take on:

• Auditing every crontab and timer on a host and telling you which jobs have quietly stopped running
• Converting fragile cron entries to systemd timers with locking, runtime caps and failure handlers, without a big-bang migration
• Fixing the classic “works by hand, fails on schedule” environment and PATH problems, including the percent-sign trap
• Adding heartbeat monitoring and alerting so a missed run pages someone instead of sitting there
• Backup and dump jobs that verify their own output rather than exiting zero on an empty file
• Untangling duplicate schedules left behind by control panels, deploy scripts and configuration management

If something on your server isn’t running when it should, send me the crontab line or the unit file and the journal output around the time it should have fired. That’s usually enough to tell you what’s wrong before we talk about anything else.

The post Cron vs systemd Timers: The Failure Modes That Decide It appeared first on John Nessime.

The rule was fine. The query was fine. The evaluation ran on schedule and the state transition happened exactly when it should have. What failed was everything after that, and none of it made a sound.

That is the failure mode worth designing around. Rule evaluation and notification delivery are two different systems, and a red rule only proves the first one works. Grafana alert notifications run against SMTP servers, Slack’s API and Telegram’s bot endpoint, all of which fail in their own quiet ways. A rejected login, a revoked webhook, a message over a length limit: each produces a log line and nothing else. The alert list still shows the rule firing, so everything looks healthy from the one place you’d think to check.

This post covers the practical setup for the three channels most small and mid-sized teams actually use: email, Slack and Telegram. It covers what breaks in each of them, how to route alerts so the right ones interrupt people and the rest don’t, how to provision the whole thing as files instead of clicking through the UI, and how to detect the case where delivery has stopped entirely.

Where a firing alert actually goes

Before touching any integration, it helps to know how many hops sit between “the rule fired” and “my phone buzzed”.

• The rule evaluates on its own schedule and produces alert instances, each carrying labels.
• Those instances go to an Alertmanager. Grafana ships its own and can also use an external Prometheus Alertmanager.
• The Alertmanager applies mute timings, groups instances, and walks the notification policy tree for a match.
• The matched policy names a contact point, which holds one or more integrations.
• Each integration makes its own outbound call to a third-party service.

Steps one and two are visible in the Grafana UI. Steps three through five are where things go wrong, and they are visible only in logs. That asymmetry is the whole problem.

One thing worth knowing early: a contact point can hold several integrations at once, so a single contact point named critical can fire email, Slack and Telegram together. Building one contact point per channel instead makes routing much harder than it needs to be.

Email: the channel everyone assumes works

Email is the one people configure first and verify least. The catch is that the email contact point holds only the recipient addresses. The actual SMTP credentials live in grafana.ini, so a perfectly valid contact point can sit there doing nothing because the server underneath was never configured.

[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
# Wrap the password in triple quotes if it contains # or ;
password = """your-smtp-password"""
from_address = alerts@example.com
from_name = Grafana Alerts
skip_verify = false
# Default is OpportunisticStartTLS
startTLS_policy = MandatoryStartTLS

Several details in that block matter more than they look.

• The port is part of host. There is no separate port setting. Port 587 with STARTTLS is the normal choice; 465 is implicit TLS and behaves differently, which is usually why one fails cleanly and the other hangs.
• startTLS_policy decides how hard Grafana insists on encryption. The default is opportunistic: it upgrades the connection if the server offers it and quietly continues in the clear if not. On a relay you control, mandatory turns a silent downgrade into a visible failure.
• from_address has to be something the relay accepts. Managed relays reject mail from unverified domains after the connection succeeds, which is why the Grafana side looks fine.
• Leave skip_verify false. It exists for internal relays with self-signed certificates. Turning it on to clear an error removes the check that was telling you the certificate is wrong.

Changes to this file need a Grafana restart. Nothing in the UI reloads SMTP settings.

Deliverability is a separate problem from delivery

Sending directly from a VPS is the fastest way to get alerts filed as spam. A fresh IP range from a provider like Contabo or InterServer has no sending reputation, and a mail server that emits one message a week during incidents never builds one.

Point Grafana at a transactional relay instead. Amazon SES, Postmark, Mailgun and Brevo all do the same job at this volume. Whichever you pick, publish SPF and DKIM records for the sending domain and set a DMARC policy, then send a real alert to a Gmail address and a Microsoft 365 address and check the spam folder. That is the only test that tells you anything.

Slack: webhook or bot token

Grafana supports two ways of posting to Slack, and the choice has consequences beyond setup time.

Incoming webhooks are the quick path. Slack generates a URL under hooks.slack.com/services/ that is bound to a single channel, and Grafana posts to it. There’s no OAuth flow and no scopes to reason about. The trade-off is that the URL is the credential and it maps to exactly one channel, so a team with six alert channels ends up managing six secrets with no central view of them.

A Slack app with a bot token takes longer. You create the app, add the chat:write.public scope so it can post to public channels without being invited to each one, then copy the bot user OAuth token, which starts with xoxb-. In Grafana you supply that token plus the channel ID as the recipient. One credential covers every channel, permissions are visible in one place, and revoking it kills all posting at once.

Webhooks win when you have one or two channels and want to be done in five minutes. The bot token wins as soon as channels start multiplying or someone asks what the integration is permitted to do. I reach for the webhook on small setups and regret it around the fourth channel.

Two constraints either way. A Slack integration targets one channel, so several channels means several integrations. And notification templates control the message title and body text, not its visual structure, so rich block layouts aren’t on the table.

Telegram: cheap paging, one nasty edge

Telegram is genuinely useful for out-of-hours alerts because it pushes to a phone without anyone paying for a paging product. The setup is two values: a bot API token and a chat ID.

1. Message BotFather and send /newbot. The username has to end in bot or _bot.
2. Copy the HTTP API token it returns. That’s the bot API token Grafana wants.
3. Create a group for alerts and add the bot to it. A bot cannot message a chat it has never seen.
4. Send a slash command in the group, such as /start@your_bot_name. An explicit command is the reliable way to make the group visible to the bot.
5. Fetch the chat ID from the Telegram API.

curl -s "https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates" | jq '.result[].message.chat'

The id field in that output is what goes in Grafana’s Chat ID field. Group chat IDs are negative. Copy the minus sign; leaving it off is the single most common reason a Telegram contact point tests as failing.

Now the edge case, and it’s the reason Telegram belongs in a post about invisible failures. Telegram caps messages at 4096 UTF-8 characters. If you’re using a parse mode other than none, truncating at that boundary can leave the markup unbalanced, and Telegram rejects the whole message as malformed. You don’t get a shortened alert. You get nothing.

This is exactly backwards, because the message most likely to blow the limit is a large grouped alert during a real outage. The channel works perfectly for months of single alerts, then goes quiet the first time forty instances fire together. Three ways to deal with it, in the order I’d try them:

• Write a deliberately short notification template for Telegram: alert name, severity, instance count and a dashboard link. Treat it as a pager, not a report.
• Narrow group_by on the policy that feeds Telegram so a single notification covers fewer instances.
• Set the parse mode to none, which makes truncation harmless at the cost of formatting.

Routing: split by severity, not by tool

The most common design mistake is a policy tree organised around channels: a Slack policy, an email policy, a Telegram policy. That gives you no way to express the thing you actually care about, which is that some alerts should wake someone up and most should not.

Put severity on the alert rule as a label and route on that instead. Contact points then become bundles of channels chosen to match urgency.

• severity=critical goes to a contact point holding Telegram and Slack. Something is broken and someone needs to look now.
• severity=warning goes to Slack only, in a channel people read during working hours.
• Everything else falls through to the default policy and lands in email, where it accumulates without interrupting anyone.

Four timing settings control how noisy this is, and each one does something specific:

• group_by decides what counts as the same notification. Grouping by alertname and a service label means one cascading failure produces one message instead of forty.
• group_wait (default 30 seconds) is how long to hold the first notification for a new group, so related instances that fire a few seconds apart arrive together.
• group_interval (default 5 minutes) is the minimum gap before a group that has already notified sends again because new instances joined it.
• repeat_interval (default 4 hours) is how often an unresolved alert re-notifies with nothing new to say.

The temptation is to drop repeat_interval to fifteen minutes so nothing gets forgotten. Resist it. Teams that do this stop reading the channel within a month, which converts a routing problem into a much worse cultural one. Longer intervals plus a real escalation path beats short intervals every time.

Grafana also offers a simpler path: pick a contact point directly on the alert rule form and skip label matching entirely. Reasonable when a rule has one obvious destination. The trade-off is that routing knowledge moves into individual rules, so changing where a class of alerts goes means editing every rule instead of one policy.

Provision it as files, not clicks

Contact points built through the UI are invisible to review, easy to delete by accident, and impossible to recreate after a bad restore. If Grafana alert notifications matter enough to build, they matter enough to keep in Git. Grafana reads alerting configuration from files in the provisioning/alerting directory under its provisioning path at startup.

Before writing any YAML by hand, build one contact point in the UI, test it, then use the export option on the contact points list to dump it as YAML. That gives you the exact setting keys for that integration type instead of the ones you half-remember from a blog post. Setting names are not consistent between integration types, and a guessed key produces a config that loads without error and sends nothing.

apiVersion: 1

contactPoints:
  - orgId: 1
    name: critical-oncall
    receivers:
      # Telegram receiver goes here. Export it from the UI first:
      # the setting key names differ from what you would guess.
      - uid: critical-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS
        disableResolveMessage: false

  - orgId: 1
    name: warnings-slack
    receivers:
      - uid: warnings-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS

  - orgId: 1
    name: email-digest
    receivers:
      - uid: email-digest
        type: email
        settings:
          addresses: ops@example.com
          singleEmail: false

And the policy tree that uses them:

apiVersion: 1

policies:
  - orgId: 1
    receiver: email-digest
    group_by: [grafana_folder, alertname]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: critical-oncall
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 1h
      - receiver: warnings-slack
        matchers:
          - severity = warning
        repeat_interval: 12h

Two things to know about provisioned resources. By default they become read-only in the UI, which is the point but surprises people the first time. And a contact point that a policy references cannot be deleted until the policy stops referencing it, so removals have to happen in the right order.

Keep tokens out of the files themselves. Grafana supports environment variable substitution in provisioning files, which means the YAML can live in Git and the secrets can come from your existing secret store.

Detecting the silence

Everything above still leaves the original problem unsolved. If the Slack app gets revoked or the SMTP password rotates, notifications stop and Grafana does not tell you.

The fix is a heartbeat alert, and it’s worth the twenty minutes. Create a rule whose condition is always true, give it a low severity label, route it to each contact point with a short repeat_interval, and point the resulting notification at an external dead-man’s-switch service. Healthchecks.io, Cronitor and Better Stack all offer this, and Healthchecks.io can be self-hosted if you’d rather not add a dependency. The service expects a signal on a schedule and alerts you when it stops arriving.

The logic is inverted on purpose. You aren’t monitoring infrastructure here, you’re monitoring the path that tells you about your infrastructure, and it has to be watched by something that isn’t itself.

As a second layer, Grafana exposes its own internal metrics on a Prometheus-compatible endpoint, including counters for notification attempts and failures. If you’re already scraping Grafana, alert on that failure counter increasing. It won’t catch a credential that works but delivers to a dead mailbox, which is why the heartbeat comes first.

Troubleshooting: the rule fires, nothing arrives

Work down this list in order. Each step rules out one hop, and stopping early is how people spend an afternoon on the wrong layer.

1. Test the contact point in isolation. The Test button sends a notification without touching rules or policies. If it fails, the problem is credentials or network and nothing else matters yet.
2. Watch the Grafana logs during the test. On a systemd install, run journalctl -u grafana-server -f while you click Test. SMTP, Slack and Telegram delivery errors all surface here, usually specific enough to fix directly.
3. Compare the instance’s labels against your matchers. A policy matching severity = critical never sees severity = Critical. Matchers are exact and case-sensitive. Read the firing instance’s real labels, not the ones you think the rule sets.
4. Look for a policy earlier in the tree that swallowed it. Evaluation stops at the first match unless the policy continues. A broad route near the top silently absorbs everything below it.
5. Check active silences and mute timings. A silence created during last month’s maintenance with a generous matcher is the classic. So is a mute timing in the wrong timezone.
6. Check whether you’re inside repeat_interval. A red alert now does not mean a notification is due now. This looks identical to a broken integration and isn’t.
7. For email, read the relay’s own logs. Grafana handing the message off is not delivery. Bounces and spam filing happen after Grafana’s involvement ends.

Common mistakes

• Configuring the email contact point and never touching grafana.ini. The contact point is only a recipient list. Without the SMTP block, it has nothing to send with.
• Routing by channel instead of severity. It feels tidy and it makes urgency impossible to express.
• Dropping the minus sign from a Telegram group chat ID. Group IDs are negative and the field takes the value literally.
• Sending verbose templates to Telegram. The 4096-character limit turns your biggest incident into your quietest one.
• Testing with the Test button and calling it done. Test bypasses the policy tree entirely. It proves the credential works, not that a real alert would ever reach it.
• Leaving resolved notifications on for every channel. Resolution messages in a pager channel double the volume and train people to ignore it.
• Storing webhook URLs and bot tokens in the UI only. They exist in one place, are invisible to review, and disappear with the database.

Best practices for Grafana alert notifications

• Set a consistent severity label on every rule from day one. Retrofitting labels across dozens of rules is miserable work.
• Use one contact point per urgency level, holding as many integrations as that level needs.
• Put a dashboard or runbook link in an annotation your templates render. An alert that says what broke but not where to look costs the responder five minutes every time.
• Write channel-specific templates: short for Telegram, richer for email, in between for Slack.
• Provision contact points and policies from files, with secrets injected from the environment, and run a heartbeat through every channel.
• Re-test every integration after credential rotation, Slack workspace changes or a Grafana upgrade. Those are the three events that quietly break delivery.

FAQ

Can one contact point send to email, Slack and Telegram at the same time?

Yes. A contact point holds multiple integrations and every one of them fires when that contact point is selected. This is the right way to build an escalation tier: one contact point per severity, with the channel mix chosen to match how urgently that severity needs attention.

Why does the Test button work but real alerts never arrive?

Test sends straight to the contact point and skips the policy tree. If it works and real alerts don’t, the break is in routing: matchers that don’t match, a broader policy earlier in the tree absorbing the alert, an active silence, or a mute timing. Open the firing instance, read its real labels, and compare them character by character against your matchers.

Do I need Grafana Cloud for any of this?

No. Contact points, notification policies, templates and all three integrations ship in the open source build. Grafana Cloud handles hosting and adds on-call tooling on top, which matters if you need rotations and escalation chains. For a team routing alerts to three channels, a self-hosted instance on a modest VPS does everything described here.

Should I use a Slack webhook or a bot token?

Webhook for one or two channels where speed of setup matters. Bot token once you have several channels, want one revocable credential, or need to see in one place what the integration is allowed to do. Both deliver identically; the difference is entirely in credential management.

Why did my Telegram alerts stop during a big outage?

Almost certainly the 4096-character limit. A large grouped notification exceeded it, truncation broke the message markup, and Telegram rejected the send outright. Shorten the template for that channel, group more narrowly, or drop the parse mode. It’s the failure mode most worth guarding against, because it only shows up when you need the channel most.

Can I keep using the Prometheus Alertmanager I already run?

Yes. Grafana can send to an external Alertmanager instead of its built-in one, and the contact point selector at the top of the alerting pages lets you choose which one you’re configuring. If you already have a mature Alertmanager routing config, keeping it and pointing Grafana at it is usually less work than migrating the tree.

The one thing to remember

A firing rule proves evaluation works. It proves nothing about whether anyone was told. Grafana alert notifications cross three or four systems after the rule turns red, and every one of them fails quietly by default.

So build the channels carefully, route by severity rather than by tool, then run a heartbeat through every path and let something outside Grafana watch it. The alternative is finding out during the incident, which is the one time you have no attention to spare.

Need alerting that reaches people?

Most of the Grafana setups I get asked to look at have good dashboards and alerting nobody trusts. Work I take on here:

• Fixing email delivery end to end: SMTP config, relay selection, SPF, DKIM and DMARC so alerts stop landing in spam.
• Setting up Slack and Telegram contact points properly, including bot apps, scoped tokens and channel-appropriate templates.
• Designing a policy tree around severity labels, with grouping and timing tuned so people keep reading the channel.
• Converting click-built alerting into provisioned YAML under version control, with secrets kept out of Git.
• Adding heartbeat monitoring so a broken notification path announces itself instead of waiting for an outage.
• Auditing alert rules for missing labels, missing runbook links, and rules that fire constantly and are therefore ignored.

Send me your notification policy export and the output of journalctl -u grafana-server from a period when an alert should have gone out, and I’ll tell you which hop is dropping it.

The post Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram appeared first on John Nessime.

That is the failure mode nobody catches during a review, and it is why most AWS accounts have dozens of alarms and about three anyone trusts. The rest either fire constantly and get muted, or never fire at all and get mistaken for coverage.

This post is about building useful CloudWatch alarms: which signals deserve a page, which settings decide whether an alarm fires correctly, how to collapse alarm storms without going blind, and how to prove the notification path works before you need it.

The test an alarm has to pass before it exists

Before the threshold, before the metric, one question: if this fires at 3am, what does the person on call do?

If the honest answer is “look at it and go back to bed”, it is not an alarm. It is a dashboard panel. That single filter removes most of the CPU, memory and disk alarms people inherit. High CPU is not an incident. High CPU while latency climbs and the queue backs up is an incident. Every alarm that fires without requiring action trains the on-call to ignore the channel, and that training is permanent.

• Page on symptoms a customer can feel: error rate, latency at a high percentile, queue age, failed jobs, expiring credentials.
• Ticket on causes that bite next week: disk trending toward full, approaching a service quota, a renewal that is not happening.
• Graph only on everything else. Resource utilisation is context during an incident, not the incident.

Encode that split in the alarm’s action, not in your head. A page goes to an SNS topic wired to PagerDuty or Opsgenie. A ticket goes to a different topic that opens a Jira issue or a Systems Manager OpsItem. Same engine, different consequences.

The alarm that goes quiet: missing data is the invisible failure

Every metric alarm has a TreatMissingData setting, and if you never touched it, it is missing. That default means the alarm ignores gaps entirely when deciding state. It keeps whatever state it had, or sits in INSUFFICIENT_DATA if there is nothing at all. For a metric that always reports, that is a silent hole: the instance dies, the metric stops, and the alarm does not go red. It just stops having an opinion.

• breaching for metrics that report continuously. If the data stops, something is wrong. Heartbeats, health checks, CPU on a box that should always be up.
• notBreaching for metrics that only exist when something bad happens. Error counts, throttled requests, dead letter queue depth. No data means no errors.
• ignore keeps the current state through gaps, so an alarm stays red until genuinely resolved rather than flapping to OK on a hole in the data.
• missing is the default, and defensible only when you decided it was right.

There is a second-order effect that surprises people. When an alarm evaluates, CloudWatch pulls a wider range of data points than the evaluation periods you configured. AWS calls that the evaluation range. If some points in that range are missing but enough real ones exist, CloudWatch evaluates against the real points and your missing-data setting never applies to the gaps.

The consequence: an alarm set to eight breaching points out of ten can fire on one breaching point followed by nine gaps, because the older breach is still inside the evaluation range and everything newer is absent. Teams hit this with load balancer 5XX alarms constantly, and it looks exactly like an unexplained false positive.

Setting notBreaching on sparse error metrics is usually the fix. Filling gaps with metric math is the other, and it is more explicit:

# Treat absent error data points as zero rather than as gaps.
# m1 is the raw error metric; e1 is what the alarm evaluates.
e1 = FILL(m1, 0)

One exception: alarms on metrics in the AWS/DynamoDB namespace always ignore missing data regardless of what you set. If you need a DynamoDB alarm that reacts to silence, build the fill into metric math.

Then find the ones already stuck. This is the fastest audit you can run on an inherited account:

aws cloudwatch describe-alarms 
  --state-value INSUFFICIENT_DATA 
  --query 'MetricAlarms[].[AlarmName,MetricName,StateReason]' 
  --output table

Anything on that list older than a deploy cycle is either watching something that no longer exists, or watching something that stopped reporting. Both deserve thirty seconds.

The alarm that cries wolf: periods, data points and windows

Three settings decide whether an alarm is jumpy: Period (how long each data point covers), EvaluationPeriods (how many recent points to look at), and DatapointsToAlarm (how many of those must breach).

M out of N beats consecutive breaches

Most people set evaluation periods to 3 and leave data points to alarm unset, which makes it 3 as well. That demands three consecutive breaching points. Real degradation is rarely that tidy; it flickers, one clean point resets your patience, and the alarm never fires.

Set them separately. Three out of five catches intermittent problems while still filtering single-point spikes. The breaching points do not need to be consecutive, only inside the last N.

aws cloudwatch put-metric-alarm 
  --alarm-name "api-5xx-elevated" 
  --namespace "AWS/ApplicationELB" 
  --metric-name HTTPCode_ELB_5XX_Count 
  --dimensions Name=LoadBalancer,Value=app/my-alb/0123456789abcdef 
  --statistic Sum 
  --period 60 
  --evaluation-periods 5 
  --datapoints-to-alarm 3 
  --threshold 25 
  --comparison-operator GreaterThanThreshold 
  --treat-missing-data notBreaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page

Read that as: over the last five minutes, if three or more individual minutes each saw more than 25 backend 5XX responses, page. Minutes with no data count as fine, because a load balancer with no errors publishes nothing.

Percentiles need a sample-count decision

Average latency hides everything interesting, so alarm on p95 or p99 using the extended statistic rather than --statistic Average.

The catch is low traffic. A p99 computed from four requests is noise, and by default CloudWatch still evaluates it. The EvaluateLowSampleCountPercentile setting controls that: set it to ignore and the alarm holds its current state when the sample count is too small to mean anything. On an overnight window or a quiet endpoint, that one setting removes a whole class of 4am pages.

Sliding versus wall clock windows

By default the evaluation window slides: it advances a minute at a time and is not aligned to the clock. That is what you want for latency and errors, because you want to know now.

CloudWatch also supports a wall clock window aligned to real boundaries in a time zone you pick: top of the hour, midnight, start of the calendar week. That is the right choice for anything measured per calendar period, like a daily batch job or a monthly threshold, because the alarm considers a completed period rather than a rolling one. Note that the wider evaluation range described earlier applies only to sliding windows, so wall clock alarms are easier to reason about but have no safety net for late-arriving data.

Alarming on the wrong number

A large share of useless alarms are correctly configured alarms watching a meaningless quantity.

Rates, not counts

An alarm on “more than 50 errors” is wrong twice. During a traffic peak, 50 errors out of half a million requests is nothing. At 3am, 50 errors out of 60 requests is a total outage that never crosses the threshold.

# m1 = Errors (Sum), m2 = Invocations (Sum)
# Guard the zero case so the alarm gets a data point at idle
# instead of a gap it has to interpret.
e1 = IF(m2 > 0, m1 / m2 * 100, 0)

Only the final expression should return data to the alarm; the underlying metrics are inputs. A rate alarm alone is still incomplete, though, because at very low volume one error is 100 percent. Pair it with a minimum-volume condition in a composite alarm, or fold the request count into the expression so tiny denominators cannot trip it.

Quotas and saturation, before they bite

Service quotas are the outage nobody sees coming, because everything is healthy right up until it is not. For services that publish usage metrics, the SERVICE_QUOTA metric math function pulls your current limit from Service Quotas, so you can alarm on percentage consumed instead of a number you hardcoded and forgot:

# m1 is a usage metric under the Usage namespace.
# Alarm on consumption as a share of the live quota.
e1 = m1 / SERVICE_QUOTA(m1) * 100

This is the definitive ticket-not-page alarm. Nobody needs waking for concurrency at 80 percent, but somebody needs to file a limit increase before the deploy that takes it to 100. Same category: queue age rather than queue depth. A deep queue draining fast is fine; a shallow queue whose oldest message is twenty minutes old means your consumers are stuck.

Cutting the storm: composite alarms, suppression and mute rules

One bad deploy in an Auto Scaling group with per-instance alarms produces forty notifications describing one event. A composite alarm evaluates a boolean rule over other alarms. The metric alarms stay exactly as they are and keep their own state, but stop being wired to your paging topic. The composite becomes the only thing that pages.

aws cloudwatch put-composite-alarm 
  --alarm-name "checkout-degraded" 
  --alarm-rule 'ALARM("checkout-latency-p99") AND ALARM("checkout-error-rate")' 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page

Latency alone might be a slow dependency. Errors alone might be a bad client. Both together is a real problem, and that is what earns a page.

The second half is suppression. A composite alarm accepts a suppressor alarm: while that alarm is in ALARM, the composite still evaluates but takes no actions. Point it at a deployment-in-progress alarm and you stop paging for known-bad windows.

aws cloudwatch put-composite-alarm 
  --alarm-name "checkout-degraded" 
  --alarm-rule 'ALARM("checkout-latency-p99") AND ALARM("checkout-error-rate")' 
  --actions-suppressor "deployment-in-progress" 
  --actions-suppressor-wait-period 60 
  --actions-suppressor-extension-period 60 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page

Those two periods are the part people skip and then get burned by. The wait period gives the suppressor time to reach ALARM before the composite decides to fire; the extension period gives the composite time to settle back to OK after the suppressor clears. AWS suggests 60 seconds for each, on the reasoning that metric alarms evaluate once a minute. Leave them at zero and you get paged in the gap between the deploy starting and the deploy alarm noticing.

CloudWatch has since added alarm mute rules, which handle the scheduled case directly. A mute rule targets specific alarms by name and mutes their actions during a one-time or recurring window. The alarms keep evaluating and keep changing state; only the actions are muted. That removes the EventBridge and Lambda contraption teams used to build for maintenance windows. Two things to keep straight: mute rules take precedence over composite action suppression when both apply, and a mute rule targets alarms by name, so a rename in your Terraform or OpenTofu module silently orphans the rule.

Suppression is for known-bad windows you chose. If you are muting an alarm because it is annoying rather than because you scheduled the disruption, the alarm is wrong and muting it just hides that.

Test the path, not the threshold

An alarm you have never seen fire is a hypothesis. The threshold is the easy part. The fragile part is everything downstream: SNS subscriptions that were never confirmed, a Lambda whose role lost a permission, a chat webhook rotated three months ago, a topic policy that quietly rejects CloudWatch.

You can force a state transition without touching the metric:

aws cloudwatch set-alarm-state 
  --alarm-name "api-5xx-elevated" 
  --state-value ALARM 
  --state-reason "Routing test, ticket OPS-1234"

This drives the real actions, so the page really goes out. Tell the on-call first. The forced state is temporary; the next real evaluation overwrites it. Do this for every new alarm before you call it done, and again after any change to the notification stack. On a mixed estate where some workloads sit on a VPS at a provider like Contabo or InterServer alongside your AWS footprint, test each path separately, because a shared SNS topic creates the illusion that one confirmed subscription covers everything.

Then confirm the transition and the action both happened:

aws cloudwatch describe-alarm-history 
  --alarm-name "api-5xx-elevated" 
  --history-item-type Action 
  --max-records 10

What useful CloudWatch alarms cost you

Alarms bill per alarm metric per month, so the mental model that matters is this: a standard metric alarm counts as one, and a metric math alarm counts as one per metric referenced in the expression. An error-rate alarm built from errors and invocations bills as two, not one. Anomaly detection alarms carry their own multiplier because the band is evaluated alongside the metric. High-resolution alarms, with periods of 10, 20 or 30 seconds, are charged at a higher rate than standard ones, which makes them worth it for a handful of latency-critical paths and wasteful everywhere else.

Check current rates on the CloudWatch pricing page rather than trusting a number in a blog post, including this one. And note the bigger cost is usually not the alarm at all. It is the custom metrics feeding it, especially high-cardinality ones published per request or per container.

Troubleshooting an alarm that misbehaves

1. Stuck in INSUFFICIENT_DATA. Either the metric is not being published, or the alarm’s dimensions do not match the metric’s dimensions exactly. Dimension mismatch is the common one and it is silent: an alarm on a dimension set that never existed looks identical to an alarm on a dead resource.
2. Fired but nobody heard. Check describe-alarm-history filtered to Action items. Transition present but action missing points at the SNS topic policy or the subscription. Both present points downstream of SNS.
3. Fires with no visible breach on the graph. Almost always missing data plus the wider evaluation range. The state reason names how many points were breaching and how many were unknown.
4. Flapping between OK and ALARM. The period is too short for the metric’s natural variance, or data points to alarm equals evaluation periods. Widen the window before raising the threshold; raising the threshold loses sensitivity you may want later.
5. Composite alarm never fires. Confirm every child alarm named in the rule exists and is spelled correctly, then confirm no suppressor or mute rule is active. The console shows a mute indicator on affected alarms.

Common mistakes

• Leaving TreatMissingData at the default because you did not know it existed.
• Picking 80 percent CPU because it sounds like a threshold, on a service that normally runs at 75.
• Alarming per instance in an Auto Scaling group instead of on the aggregate a customer experiences.
• Wiring OK actions to the same paging channel, so every recovery is also an interruption.
• Alarm descriptions that restate the alarm name. The description is the only context the responder gets at 3am, so put the runbook link there.
• Anomaly detection on a metric with no stable pattern. The band is only as good as the history it learned from.

Best practices for useful CloudWatch alarms

• Set TreatMissingData explicitly on every alarm, and write down why in the description.
• Separate DatapointsToAlarm from EvaluationPeriods. M out of N is almost always better than N consecutive.
• Observe a metric across a full weekly cycle before fixing its threshold. Weekend traffic is a different distribution.
• Define alarms in code, in whatever you already use: Terraform, OpenTofu, CDK, CloudFormation. Console-created alarms drift and disappear.
• Use anomaly detection for the signal and composite alarms for the logic. An anomalous shape plus real user impact is a page; an anomalous shape alone is a graph.
• Review what fired last quarter. Any alarm that fired and was closed without action is a candidate for deletion. Send it to a Grafana panel instead.

Frequently asked questions

How many CloudWatch alarms should a service have?

Fewer than you think. A handful of symptom alarms that page, plus a set of cause alarms that open tickets, covers most services. The number that matters is not how many exist but how many fired last quarter and led to somebody doing something.

Should I use anomaly detection or static thresholds?

Static thresholds where a real limit exists: a disk that must not fill, a latency budget you promised. Anomaly detection where the metric has a genuine daily or weekly shape and no fixed correct value, like request volume. Anomaly detection needs history before it is useful and struggles when a deploy changes the pattern, so it is a poor fit for anything new.

Why did my alarm fire when the graph looks fine?

Usually missing data. CloudWatch pulls a wider evaluation range than the periods you configured, so an older breaching point can still count while newer points are simply absent. Read the state reason, which spells out how many points were breaching and how many were unknown, then reconsider the missing-data setting.

Can an alarm watch a metric in another AWS account?

Cross-account observability lets a monitoring account view and alarm on metrics from linked source accounts, which is the usual pattern for an AWS Organizations estate. The alternative is running alarms in each account and aggregating at the SNS or paging layer. That is more alarms to manage but keeps the blast radius per account smaller.

How do I stop alarms firing during a deployment?

Two options. An alarm mute rule on a schedule, if your deploys are predictable. Or a composite alarm with a deployment-in-progress alarm as its actions suppressor, if they are not. The suppressor approach suits continuous delivery better because it reacts to the actual deploy rather than a calendar guess.

Do CloudWatch alarms work on logs?

Indirectly. A metric filter on a log group turns matching log lines into a numeric metric, and you alarm on that metric like any other. Metric filters only emit data points when they match, so notBreaching is almost always the right missing-data setting for them.

Is CloudWatch enough, or do I need Datadog or Grafana?

For a single-account AWS workload, CloudWatch alarms plus SNS handle alerting fine, and staying inside AWS avoids another integration to maintain. Third-party platforms earn their keep when you need correlation across non-AWS systems, richer on-call routing, or long metric retention with fast queries. The honest test is whether you are already spending engineering time rebuilding those features on top of CloudWatch.

The one thing to keep

Useful CloudWatch alarms are not the ones with the cleverest thresholds. They are the ones where somebody decided, on purpose, what silence means.

Every alarm in your account is making a claim about the absence of data, whether you configured that claim or inherited it. Go through them, set TreatMissingData deliberately, split data points from evaluation periods, delete the ones nobody acts on, and force a state transition on the ones you keep so you know the page lands. Six alarms that fire correctly beat sixty that nobody reads.

Need a second pair of eyes on your alerting?

I work with teams on AWS monitoring that people actually trust. Typically that looks like:

• Auditing existing CloudWatch alarms and separating the ones that page, the ones that should open tickets, and the ones to delete.
• Fixing missing-data and evaluation settings so alarms stop firing on gaps and stop going silent on dead metrics.
• Building composite alarms, suppression and mute rules so one incident produces one page instead of forty.
• Writing metric math for error rates, queue age and quota headroom instead of raw counts.
• Codifying the whole alarm set in Terraform or OpenTofu so it survives the next person.
• Wiring CloudWatch into Grafana dashboards and testing every notification path end to end.

If you want a concrete starting point, send me the output of describe-alarms for one account and I will tell you which alarms are load-bearing and which are decoration.

The post CloudWatch Alarms That Are Actually Useful (and the Ones to Delete) appeared first on John Nessime.

That gap is the entire problem. Monitoring n8n workflow failures in Grafana is not hard because the tooling is missing. It’s hard because the built-in metrics endpoint answers a question you didn’t ask. It tells you the process is alive and busy. It does not tell you that a workflow stopped producing correct output.

This post covers what n8n’s Prometheus endpoint actually exposes, the three families of failure it handles badly, and the specific PromQL and alert rules that close each gap. There’s also a troubleshooting section for the case where /metrics returns nothing at all, which is the first wall most people hit.

What the n8n metrics endpoint actually gives you

n8n uses the prom-client library and serves metrics on the same port as the editor, at /metrics. The endpoint is off by default, and almost everything interesting inside it is off by default too, behind its own environment variable.

N8N_METRICS=true
N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true
N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL=true
N8N_METRICS_INCLUDE_NODE_TYPE_LABEL=true
N8N_METRICS_INCLUDE_QUEUE_METRICS=true

The first line opens the endpoint. The second turns on the workflow and node event counters, which is the part people assume is included and is not. The third attaches a workflow_id label to workflow metrics, without which you get a single instance-wide number and no way to tell which automation broke.

Those labels are opt-in for a reason worth understanding rather than working around. Every distinct label value creates a separate time series. Turn on workflow_id and you get one series per workflow per counter; add node_type and node counters fan out across every integration you use. On a few dozen workflows that’s fine. On several hundred with per-node labels, you’re signing up for a cardinality problem that surfaces later as slow queries and a Prometheus process that keeps getting OOM-killed.

With the event bus metrics enabled, the counters you get include n8n_workflow_started_total, n8n_workflow_success_total, n8n_node_started_total and n8n_node_finished_total. Queue mode adds n8n_scaling_mode_queue_jobs_active, _completed, _failed and _waiting, gathered from Bull and exposed on the main instance only.

Prometheus side, the scrape job is unremarkable:

scrape_configs:
  - job_name: n8n
    static_configs:
      - targets:
          - n8n-main:5678
          - n8n-worker-1:5678
    metrics_path: /metrics

Note both main and workers. Each process exposes its own endpoint and its own counters. Scrape only the main and you lose everything the workers did, which in queue mode is nearly all the real execution work.

One thing the n8n docs are emphatic about, and they’re right: don’t put /metrics on the public internet. It leaks workflow IDs, execution volumes and instance topology to anyone who asks. Keep the scrape path on a private network segment, a WireGuard tunnel, or your provider’s internal networking. If you’re running the whole stack on a single VPS from somewhere like Contabo or InterServer, binding n8n to the Docker network and scraping it by container name is enough. Publishing port 5678 to the host and firewalling it afterwards is not, because you will forget.

Failure family one: the execution that never happened

This is the one that costs the most and shows up the least. A schedule trigger gets deactivated during a deploy. A credential expires and the workflow is switched off rather than fixed. An upstream service stops calling a webhook. Nothing errors. Nothing increments. The workflow simply stops existing as far as your metrics are concerned.

Every error-rate alert you write is blind to this, because an error rate needs a denominator. Zero executions divided by zero executions is not an alert, it’s a NaN.

The instinct is to write this:

sum by (workflow_id) (increase(n8n_workflow_started_total[6h])) == 0

It looks correct and it does not work. When a workflow stops running, its series stops being exported. Prometheus marks it stale after a few missed scrapes and instant queries stop returning it. There is no series left for == 0 to match against. The alert never fires, because the thing you wanted to alert on is exactly the thing that removed the data.

What you want is a function that returns something when a series is absent:

absent_over_time(
  n8n_workflow_started_total{workflow_id="AbC123XyZ"}[6h]
)

absent_over_time returns 1 if the matched series produced no samples in the window and nothing at all if it did. That inversion is the point. The alert fires on silence.

The cost is that absent_over_time needs an exact matcher, so you can’t write it once for everything. That’s more acceptable than it sounds. You don’t need a heartbeat on all two hundred workflows, you need one on the six a human would notice within an hour. Write those six by hand and treat the list as part of the runbook.

One Grafana-specific trap. When a query returns nothing, Grafana’s alert rule enters a No Data state, which is configurable separately from the alert condition and defaults to firing in most versions. Set it deliberately. For an absent_over_time rule, No Data means the whole scrape target is gone, which is a different incident from one workflow going quiet, and you probably want it routed differently.

Failure family two: the failure you can see but can’t attribute

You have the counters. Something is failing. Which workflow?

Without N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL, the answer is: no idea. The counters are instance-wide totals. You know the number moved and that’s it. That’s a fine signal for capacity planning and useless during an incident.

Turn the label on and you get IDs, which are opaque strings like AbC123XyZ. A dashboard full of those is technically attributable and practically unreadable at 3am. Newer n8n versions solve this with a gauge that maps IDs to names, enabled with N8N_METRICS_INCLUDE_WORKFLOW_INFO:

n8n_workflow_info{workflow_id="VaQPuPmx9tPpo6BX",workflow_name="Invoice sync"} 1

It’s a constant-1 gauge that exists only to be joined against. The pattern is a standard Prometheus vector match:

sum by (workflow_name) (
  rate(n8n_workflow_started_total[5m])
  * on(workflow_id) group_left(workflow_name)
  n8n_workflow_info
)

The multiplication is arithmetically a no-op because the gauge is always 1. Its only job is to carry the workflow_name label across with group_left. Do this once in a Grafana variable or a recording rule rather than in every panel, or you’ll be copy-pasting the join for the rest of your life.

On an older build without the info gauge, the fallback is a hand-maintained relabel config or a Grafana value mapping. Both drift, and the relabel version drifts silently.

Failure family three: the failure that counts as a success

This is the invisible one, and it’s why a green n8n dashboard is not evidence of anything.

n8n gives you several deliberate ways to make a node’s failure not be an execution failure. They’re all reasonable features. Together they mean the success counter and “the automation worked” are different statements.

• Continue on error. A node set to continue passes its error downstream instead of stopping the run. The execution finishes. It finishes as a success.
• Error output branches. A node with a separate error output routes failures to a different path. That path might log and stop. The execution still completes normally.
• Never Error on HTTP responses. The HTTP Request node can be configured to treat any status code as a valid response. A run of 500s from an upstream API becomes a workflow that processed zero useful records, successfully.
• Empty result sets. A query that returns nothing is not an error. A sync that syncs nothing, every night, looks exactly like a sync that has nothing to do.

There are also user reports on the n8n community forum of started and success counters matching even when a node failed outright, in queue mode. I have not reproduced that and I’d treat it as unconfirmed rather than a known behavior. But it’s the right instinct: don’t assume the success counter is a strict complement of failure. Verify it on your own instance before you build alerting on the assumption.

For webhook-triggered workflows, newer n8n versions give you a real answer here. The n8n_webhook_request_duration_seconds histogram carries a status_code label, so you can alert on what the caller actually received rather than on what n8n thought about the run:

sum by (workflow_id) (
  rate(n8n_webhook_request_duration_seconds_count{status_code!~"2.."}[5m])
)

Worth flagging: the n8n documentation shows this example with status_code!="2..", using the equality operator. In PromQL, != compares against a literal string, so that matcher excludes only a status code whose literal value is the three characters 2.., which never occurs. It matches everything, including your 200s. Use !~ for the regex form. Test any label matcher you copy from documentation by running it in Prometheus and checking the series count against what you expect.

Getting a second source of truth

For everything that isn’t a webhook, the metrics endpoint alone won’t close this gap. You need a second signal that reads execution outcomes from n8n’s own store.

The public REST API exposes exactly that. Executions can be filtered by status, and the failed ones are one request away:

curl -s -H "X-N8N-API-KEY: $N8N_API_KEY" 
  "https://n8n.example.com/api/v1/executions?status=error&limit=50"

Two ways to get that into Grafana. Write a small exporter that polls the API on a timer and serves its own /metrics, or run a scheduled job that pushes counts to a Prometheus Pushgateway. The exporter is more work and behaves correctly. Pushgateway is faster to build and carries the usual caveat: pushed metrics persist until deleted, so a job that stops running leaves a stale value sitting there looking healthy. That reintroduces failure family one through the back door.

Two caveats on the API. There’s an open report that the status=crashed list filter returns executions whose stored status is something else entirely when re-fetched by ID, which would make crash alerting built on the list endpoint unreliable. Check the behavior on your version before depending on it. And n8n’s log streaming feature, which forwards execution events to syslog or a webhook in real time, is an Enterprise capability. If you’re on the community edition, the API poll is your path.

The third option, and often the best one for a small setup, is an Error Trigger workflow. n8n lets you designate a workflow that runs whenever another workflow fails. Point it at Pushgateway, at Loki, or at a Grafana webhook contact point. It’s native, it needs no API key, and it gives you the error message rather than just a count. The trade-off is that it only fires on real execution failures, so it inherits every blind spot in this section.

The Grafana panels that earn their space

Four panels cover most of what you need. Everything past that is usually decoration.

1. Executions per workflow, by name. A time series of rate(n8n_workflow_started_total[5m]) joined to n8n_workflow_info. Its real job is showing you the lines that flatten to zero.
2. Success ratio. rate(n8n_workflow_success_total[15m]) / rate(n8n_workflow_started_total[15m]). Anything below 1 is worth a look. Anything above 1 means your window is catching a restart and the counters reset.
3. Queue depth and failed jobs. n8n_scaling_mode_queue_jobs_waiting next to increase(n8n_scaling_mode_queue_jobs_failed[1h]). A waiting count that climbs and never drains means workers are dead or starved, which looks identical to “no failures” on every other panel.
4. Webhook non-2xx rate. The query from the previous section. This is the only panel that reports what the outside world experienced.

Skip the node-level breakdown unless you have a specific reason. It is the most expensive thing you can enable in cardinality terms and it answers a question you can answer faster by opening the execution in the n8n UI.

Alert rules worth paging on

1. Named workflow silent. absent_over_time on each business-critical workflow, window set to roughly twice its expected interval. This is the rule that catches the invoice sync.
2. Success ratio below threshold. Sustained for 15 minutes, not instantaneous. A single failed run is not a page.
3. Queue waiting count rising. Compare the current value to fifteen minutes ago rather than to a fixed number, so the threshold survives traffic growth.
4. Webhook error rate. Per workflow, so a single noisy endpoint doesn’t mask the rest.
5. Scrape target down. The up metric. Obvious, and routinely forgotten, and it invalidates every other rule on this list.

Troubleshooting: the endpoint returns nothing useful

• “Cannot GET /metrics”. N8N_METRICS isn’t set on the process you’re hitting, or it’s set as a string that n8n doesn’t read as true. In Docker Compose, quote it as "true" and confirm with docker compose exec n8n env | grep N8N_METRICS.
• Endpoint responds but has no n8n_ counters. You have the default Node.js process metrics and nothing else. Set N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true and restart.
• Counters exist but no workflow_id label. N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL is off. Restart is required; it’s read at startup.
• Queue metrics are zero or missing. They’re gathered from Bull and exposed on the main instance, not the workers. Querying the worker endpoint gives you nothing. Confirm you’re looking at the right target.
• Counters look wrong after a deploy. They reset to zero when the process restarts. Always wrap them in rate() or increase(), which handle counter resets. Raw counter values on a dashboard will lie to you every time you deploy.
• Prometheus can’t reach the target. Usually a Docker network mismatch. Scrape by container name on the shared network rather than by published host port, and check the target page in Prometheus for the actual error rather than guessing.

Common mistakes

• Treating the success counter as proof the automation worked. It proves the run finished.
• Building only error-rate alerts, so the workflows that stop entirely go unmonitored.
• Enabling every N8N_METRICS_INCLUDE_* variable on day one and discovering the cardinality bill a month later.
• Scraping only the main instance in queue mode and losing all worker-side execution data.
• Copying label matchers out of documentation without running them and checking the returned series count.
• Exposing /metrics publicly because it’s “just numbers”.

Best practices

• Decide which workflows are business-critical and write a silence alert for each one by name. Six good heartbeats beat two hundred generic panels.
• Run a second source of truth. Metrics tell you about volume, the executions API tells you about outcomes. You need both.
• Push the ID-to-name join into a recording rule so panels stay readable and cheap.
• Break a staging workflow on purpose and confirm the alert fires. An untested alert is a decoration.
• If you’d rather not run Prometheus and Grafana yourself, Grafana Cloud will scrape a remote endpoint and gives you alerting without a control plane to maintain. Weigh that against your metrics leaving your network.

FAQ

Does n8n expose a failed-executions counter?

Not a reliable per-workflow one. You get started and success counters from the event bus, and a failed-jobs counter for queue mode. Deriving failures by subtracting success from started works only if you’ve verified on your own instance that a failing run genuinely doesn’t increment success, which is exactly the assumption that breaks when nodes are set to continue on error.

Can I monitor n8n Cloud this way?

No. The /metrics endpoint isn’t available on n8n Cloud. On Cloud you’re limited to the public API and the built-in insights views, so the executions-API approach in this post is the only one that transfers.

Why does my workflow error rate look like zero when I know things are failing?

Almost always one of three things: the event bus metrics aren’t enabled so you’re only seeing Node.js process metrics, the failures are being absorbed by continue-on-error settings and recorded as successes, or you’re scraping the main instance while the work happens on workers.

How do I alert when a scheduled workflow stops running entirely?

Use absent_over_time against that workflow’s started counter with an exact workflow_id matcher, and set the window to about twice the expected interval. A threshold comparison won’t work, because a workflow that stops running stops producing the series you’d compare against.

Is enabling the workflow_id label safe on a large instance?

It multiplies your n8n series count by the number of workflows. On a few dozen it’s negligible. Past a few hundred, combined with node-type labels, watch your Prometheus memory and head series count before and after. Enable the workflow label, leave the node label off, and only turn the node one on when you have a specific question it answers.

Should I use OpenTelemetry instead of Prometheus for n8n?

They solve different problems. Metrics answer “how many, how often, how fast” cheaply and forever. Traces answer “where did this run spend its time and what did it call”, which is better for debugging a slow multi-service workflow. Tracing n8n usually means a custom image with the OpenTelemetry SDK loaded before startup, so it’s a project rather than an environment variable. Start with metrics; add tracing when you have a latency question metrics can’t answer.

Where do the n8n counters go when the container restarts?

They reset to zero. They’re in-process counters with no persistence. Prometheus handles this correctly as long as your queries use rate() or increase(). Any panel showing a raw counter value will appear to lose all its history on every deploy.

The one thing to take away

Monitoring n8n workflow failures in Grafana works, but only once you stop asking the metrics endpoint a question it can’t answer. It reports activity. Activity and correctness are different things, and the space between them is where the expensive outages live.

So build for absence first. A silence alert on the handful of workflows a human would miss will catch more real incidents than any error-rate panel, because errors are loud and stopped schedules are not. Then add attribution with the workflow label, then close the false-success gap with the executions API or an Error Trigger workflow. In that order, because each one is cheap and the value drops off fast after the third.

Need help wiring this up?

I build and maintain observability stacks for self-hosted automation platforms, and n8n is a common one. If any of the above sounds like a week you’d rather not spend, here’s what I can take off your plate:

• Enabling and securing the n8n metrics endpoint on a private network path, including queue-mode setups with multiple workers
• Building Prometheus scrape configs, recording rules and the ID-to-name join so your Grafana panels are readable rather than a wall of hashes
• Writing silence and error-rate alert rules that survive counter resets, Grafana No Data states and traffic growth
• Adding a second source of truth from the n8n executions API, either as a small exporter or an Error Trigger workflow feeding Loki or Pushgateway
• Auditing an existing setup for cardinality problems, stale Pushgateway series and alerts that quietly stopped being able to fire
• Running a failure drill against staging so you know which alerts actually work before you need them

Send me your Compose file, your prometheus.yml, or a screenshot of the panel that looks fine while things are on fire, and I’ll tell you what’s missing.

The post Monitoring n8n Workflow Failures in Grafana: Why Your Green Dashboard Is Lying appeared first on John Nessime.

Sometimes the disk really is dying. More often it isn’t, and the number you’re staring at is not measuring what you think it’s measuring.

This post covers troubleshooting high I/O wait the way I’d actually do it on a production box: what the metric means, why it lies in both directions, and how to work through the five failure families that produce it.

What high I/O wait actually measures

Here’s the definition that matters: iowait is a subcategory of idle time. For a given CPU, it’s the time that CPU spent doing nothing while at least one task scheduled on it had an outstanding block I/O request.

Two consequences fall out of that, and both will burn you. High iowait can mean nothing is wrong. If the box has spare CPU and one process is pulling a large file off disk, the CPU genuinely has nothing else to do. Copying a backup will push iowait toward 100 percent on an idle host, and that’s a working system, not a fault.

Low iowait can hide a severe problem. This is the one that costs people days. If the CPU has other runnable work, the kernel context switches to it and that time gets counted as user or system, not iowait. A busy application server can be stalling badly on storage while wa reads two percent, because the CPU is never actually idle.

Add the averaging problem on top. The wa figure in top is averaged across every core. One thread pinned to one core and blocked hard on disk shows up as roughly three percent on a 32-core machine. Press 1 in top to break out per-CPU lines before you conclude anything from the aggregate.

So iowait is a hint, not a verdict. Treat it as the thing that starts the investigation, never the thing that ends it.

Start with pressure, not percentages

Pressure Stall Information is the metric I reach for first, and it exists precisely because iowait is a poor proxy for pain. It answers the question you actually care about: how much time did tasks lose because they couldn’t get I/O?

cat /proc/pressure/io

You get two lines. some is the share of time at least one task was stalled waiting on I/O. full is the share of time every non-idle task was stalled at once, which is CPU capacity being burned for nothing. The avg10, avg60 and avg300 fields are percentages over the last 10, 60 and 300 seconds. total is a cumulative microsecond counter since boot, which is what you want for graphing because averages smooth away short spikes.

The practical rule I use: a rising full line on io is a real user-visible problem almost every time. A high some line with a flat full line usually means one noisy job is waiting while everything else gets on with its life.

If /proc/pressure doesn’t exist, the kernel was built without CONFIG_PSI, or it was built with PSI compiled in but disabled by default, in which case you need psi=1 on the kernel command line. On cgroup v2 systems you also get per-cgroup pressure, which is how you narrow from “the host is stalling” to “this container is stalling” in one command:

# Pressure for one systemd slice, useful on any modern distro
cat /sys/fs/cgroup/system.slice/io.pressure

# Pressure for every leaf cgroup that has it, sorted by nothing,
# just eyeball the full lines
find /sys/fs/cgroup -name io.pressure -exec sh -c 
  'printf "%s: " "$1"; sed -n 2p "$1"' _ {} ;

If you run Prometheus, node_exporter has exposed these since it gained a pressure collector. The counters are node_pressure_io_waiting_seconds_total for some and node_pressure_io_stalled_seconds_total for full. Rate those and you have a far better storage alert than any threshold on %iowait will ever be. Grafana Cloud and Netdata both chart PSI out of the box if you’d rather not build the panels yourself.

Failure family one: the device really is saturated

This is the case everyone assumes, so get it confirmed or eliminated quickly.

iostat -xy 1

The -x gives extended per-device stats. The -y matters more than people realise: without it, the first report is a since-boot average, and reading that first block is the single most common iostat mistake there is. It will tell you the disk has been fine all year, which is true and useless.

Now the columns worth your attention:

• r_await and w_await: average milliseconds a read or write took, queue time included. This is the closest thing to “how slow does storage feel”. Split reads from writes, because they fail very differently.
• aqu-sz: average queue depth. Requests either queued or in service. A depth that climbs while throughput stays flat is the signature of a genuinely saturated device.
• r/s, w/s, rkB/s, wkB/s: the actual workload. Compare against what the device or volume class is rated for.
• %util: treat with suspicion, see below.

The %util trap. That column counts the fraction of time at least one request was in flight. On a spinning disk that serviced one request at a time, it was a fair saturation signal. On NVMe and on cloud volumes that handle many requests in parallel, a device can sit at 100 percent while running at a fraction of its real capacity. Provisioning a bigger volume off the back of a 99 percent %util reading is a common and expensive way to change nothing. Use aqu-sz and the await columns instead. Similarly, svctm was deprecated and dropped from newer sysstat output because it was unreliable on parallel devices; ignore it if an old build still prints it.

Once you know the device is busy, find the culprit. pidstat -d 1 gives per-process read and write rates from the same sysstat package and needs nothing special enabled. iotop -o is friendlier, but on kernels from 5.14 onward per-task delay accounting is off by default, so the SWAPIN and IO> columns come back empty until you turn it on:

sudo sysctl kernel.task_delayacct=1
sudo iotop -oPa
# turn it back off when you're done, it costs a little scheduler overhead
sudo sysctl kernel.task_delayacct=0

And check the boring things before the clever ones. dmesg -T | grep -iE 'i/o error|reset|medium' catches a failing drive or a flapping controller. smartctl -a /dev/sda catches reallocated sectors on hardware you own. If either of those turns something up, stop reading and replace the disk.

Failure family two: memory pressure in an I/O costume

This is the failure family I’d check second, and it’s invisible if you only look at storage.

When RAM gets tight, the kernel starts evicting page cache and swapping. Both generate disk traffic. The disk gets busy, iowait climbs, and every graph points at storage while the actual constraint is memory. Give the box more RAM and the “disk problem” disappears.

# si and so are swap-in and swap-out in KB/s.
# Sustained non-zero values are your answer.
vmstat 1

# Are we thrashing writeback? Dirty pages waiting to be written.
grep -E 'Dirty|Writeback' /proc/meminfo

# The direct question
cat /proc/pressure/memory

If memory pressure and I/O pressure rise together, fix memory first. Everything else is downstream.

There’s a related trap on the write side. The kernel buffers writes in page cache and flushes them in the background, governed by vm.dirty_background_ratio and vm.dirty_ratio. Cross the background threshold and flusher threads start writing. Cross the hard vm.dirty_ratio threshold and writing processes are forced to do writeback themselves, synchronously. That transition is brutal, and on a box with plenty of RAM and a modest disk the defaults let you accumulate gigabytes of dirty pages and then pay for all of them at once. Lowering the ratios makes writeback more frequent and less spiky, trading away a little write batching. Measure before you commit to it.

Failure family three: small synchronous writes

Throughput can be trivial and the system still crawls. A workload doing thousands of tiny fsync calls a second is bound by round-trip latency, not bandwidth, and no amount of extra megabytes per second will help.

Databases are the usual source, because durability has a cost and that cost is a flush. MySQL and MariaDB expose it through innodb_flush_log_at_trx_commit; PostgreSQL through synchronous_commit. Loosening either buys real throughput and trades away durability on power loss or crash. That can be a perfectly reasonable trade for a cache or an analytics replica, and a terrible one for anything holding orders or payments. Decide it explicitly, write down what you decided, and don’t let it be a thing somebody changed once at 2am.

The other common source is logging. Application logs, access logs, container JSON logs and a database sharing one volume will starve each other. Splitting hot write paths onto separate devices is unglamorous and works.

Failure family four: throttled, not saturated

A throttled device and a saturated device look almost identical from inside the guest: high await, growing queue, miserable latency. The difference is that no amount of tuning inside the box will fix a throttle. Three flavours are worth knowing.

• Cloud volume limits. Managed block storage gives you a baseline of IOPS and throughput, sometimes with a burst allowance on top. Burn the burst and you drop to baseline, and everything that was fine in testing falls over in production. The tell is a latency cliff that arrives at a consistent point into a workload rather than tracking load smoothly. On AWS, watch the volume-level CloudWatch metrics such as BurstBalance on burstable volume types and VolumeQueueLength; other providers publish equivalents. The fix is a volume class change, not a sysctl.
• cgroup limits you set yourself. On cgroup v2, io.max caps a cgroup at a given bytes-per-second or IOPS figure, and io.weight shares bandwidth proportionally under contention. A limit that made sense when it was written can become the bottleneck a year later, and nothing in the container’s own metrics announces it.
• Neighbours on shared hardware. On oversubscribed VPS plans your storage is shared, and someone else’s backup job is your latency spike. This is where cheap and expensive hosting genuinely differ. Budget providers like Contabo and InterServer are fine value for a lot of workloads, but if storage latency is load-bearing for you, look for plans with dedicated NVMe rather than shared pools, and test before you migrate anything that matters.

# Per-device I/O accounting for a cgroup, including any throttling
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.stat

# Is a cap set? An empty file means no limit on that cgroup.
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.max

One more signal while you’re here. In top, the st column is steal time, meaning the hypervisor gave your vCPU’s slice to someone else. Steal and iowait rising together on a shared host is a strong hint that the problem is the neighbourhood, not your configuration.

Failure family five: the latency isn’t in the block layer

Here’s the failure mode that wastes the most time, because every storage metric you check will look healthy.

iostat measures the block layer. It cannot see anything above it. So if your latency is in NFS, in a network filesystem, in an overlay or FUSE mount, in filesystem lock contention, or in a journal that’s serialising writes, the device will report sub-millisecond service times while your application waits tens of milliseconds per operation.

The diagnosis is the gap. Measure latency at the application’s layer and at the block layer, and if the two numbers disagree wildly, the problem lives in between.

# Block layer latency as a histogram. Look for a bimodal shape
# or a long tail rather than the average.
sudo biolatency-bpfcc -m 10 1      # Debian/Ubuntu naming
sudo /usr/share/bcc/tools/biolatency -m 10 1   # RHEL family

# Filesystem operations slower than 10ms, with the process and file
sudo ext4slower-bpfcc 10
sudo xfsslower-bpfcc 10

# Everything at the VFS layer, higher overhead, use briefly
sudo fileslower-bpfcc 10

If ext4slower is full of hundred-millisecond operations and biolatency shows everything completing in microseconds, you have your answer and it is not the disk. For NFS specifically, nfsiostat splits out RTT from execution time, which separates network cost from server cost. Install the tooling with bpfcc-tools on Debian and Ubuntu, or bcc-tools on RHEL, AlmaLinux and Rocky.

A working order for troubleshooting high I/O wait

On an unfamiliar box, this is the sequence I work through. Each step either finds the problem or eliminates a family.

1. Read /proc/pressure/io. If full is near zero, the stall is not hurting the whole system and you should widen the search before you tune storage.
2. Check /proc/pressure/memory at the same time. If it’s rising too, fix memory first.
3. Run iostat -xy 1 for thirty seconds. Note r_await, w_await and aqu-sz per device. Ignore %util.
4. List processes stuck in uninterruptible sleep: ps -eo state,pid,comm | awk '$1 ~ /^D/'. That’s your blocked set, and it names the workload.
5. Attribute the traffic with pidstat -d 1. Confirm the process you suspect is the process actually doing the I/O.
6. If await is high, decide saturated versus throttled. Check cgroup io.max, check provider volume metrics, check steal time.
7. If await is low but the application is slow, jump to the layer gap. Run biolatency and ext4slower together and compare.
8. Only now consider tuning: dirty ratios, I/O scheduler, flush settings, splitting volumes.

Common mistakes

• Reading iostat’s first report. It’s a since-boot average. Use -y, or discard the first block.
• Trusting %util on NVMe or cloud volumes. Parallel devices break the assumption it was built on.
• Alerting on a fixed %iowait threshold. It fires on healthy backups and stays silent during real stalls on busy hosts.
• Changing the I/O scheduler first. Going from mq-deadline to none on NVMe is sometimes right, but it’s a tuning step, not a diagnosis, and it fixes nothing if you’re throttled.
• Adding swap to fix swapping. More swap on a memory-starved box means more disk traffic, not less.
• Stopping at the block layer. If every storage metric looks clean and the app is still slow, you’re measuring at the wrong layer, not measuring a healthy system.
• Testing with the wrong benchmark. A sequential dd tells you nothing about a workload doing random 4K synchronous writes. Use fio and match the real access pattern.

Best practices worth the effort

• Record PSI continuously. Storage incidents are short and bursty. Without history you’re guessing about something that already ended.
• Capture a baseline while things are healthy. Save an iostat -xy sample and a PSI reading from a normal day. “Is 8ms bad?” is unanswerable without one.
• Keep sar running. The sysstat package can archive metrics on a schedule, which means sar -d can show you what happened at 3am. Enable it before you need it.
• Separate hot write paths. Database data, database logs and application logs on separate volumes stops one from starving the others.
• Cap container logs. An uncapped JSON log driver quietly filling the root volume is a recurring cause of host-wide stalls.
• Alert on latency and pressure, not utilisation. Users feel w_await and PSI full. Nobody has ever felt a percentage of device busy time.

Frequently asked questions

What is a normal or acceptable I/O wait percentage?

There isn’t one, and any article that gives you a single number is guessing. Iowait scales with how idle your CPUs are, so the same workload produces very different figures on a busy server and a quiet one. Judge against your own baseline, and prefer PSI, which measures lost time directly rather than as a share of idle.

Can high I/O wait be caused by something other than the disk?

Regularly. Memory pressure driving swap and page cache eviction is the most common non-disk cause. Network filesystems, hypervisor contention on shared hosts, and cgroup or cloud volume throttling all produce the same symptom. The block device being genuinely slow is one possibility among several.

Why does iostat show 100% utilisation when the disk isn’t busy?

Because %util measures the fraction of time at least one request was in flight, which made sense for devices that handled one request at a time. NVMe drives and cloud volumes process many requests in parallel, so they can be continuously occupied while nowhere near capacity. Read aqu-sz and the await columns instead.

How do I find which process is causing I/O wait?

Start with pidstat -d 1, which needs no special kernel settings. List processes in uninterruptible sleep with ps -eo state,pid,comm and filter for state D to see who is actually blocked. iotop -o gives a live view but needs kernel.task_delayacct=1 on recent kernels for its percentage columns to populate.

Does adding more RAM reduce I/O wait?

Often, yes, and it’s underrated. More RAM means a larger page cache, fewer reads reaching the device, and less swapping. If vmstat shows sustained swap activity or memory pressure tracks I/O pressure, RAM is likely the cheapest fix available.

Should I change the I/O scheduler to fix high I/O wait?

Only after you know the device is the bottleneck. Check the current setting with cat /sys/block/sda/queue/scheduler. On NVMe, none is a common and sensible default because the hardware already does the queuing. On spinning disks, mq-deadline or bfq give better latency behaviour. It’s a tuning knob, not a diagnostic, and it will not help if you’re being throttled.

Why is my application slow when iostat says storage is fine?

Because iostat only sees the block layer. Latency in NFS, FUSE, overlay filesystems, journal serialisation or filesystem locking is entirely invisible to it. Run ext4slower or xfsslower alongside biolatency: if the filesystem is slow and the device is fast, the gap between them is your problem.

The one thing worth remembering

High I/O wait is a symptom that points in a general direction, not a diagnosis. It’s a slice of idle time, averaged across cores, measured at one specific layer of the stack. It goes up when nothing is wrong and stays flat when something is very wrong.

Read PSI to find out whether anyone is actually suffering. Read r_await, w_await and aqu-sz to find out whether the device is the constraint. Compare filesystem latency against block latency to find out whether you’re even looking at the right layer. Do those three things and the answer usually falls out in a few minutes, instead of after a week of arguing with your hosting provider about a volume that was fine all along.

Need someone to look at your storage stalls?

I work with teams on exactly this kind of problem, usually the ones that have already survived a round of “the host says the disk is fine”. Things I can help with:

• Diagnosing a live or recurring I/O stall and identifying which failure family it belongs to
• Setting up PSI, sar and per-cgroup I/O metrics in Prometheus and Grafana, with alerts that fire on latency rather than utilisation
• Separating cloud volume throttling from genuine device saturation, and sizing storage correctly instead of guessing upward
• Tuning writeback, dirty ratios and I/O schedulers for a specific workload, with before-and-after measurements
• Tracing container and Kubernetes I/O contention through cgroup v2 limits, overlay filesystems and log drivers
• Benchmarking with fio against your real access pattern so capacity decisions rest on something other than a dd run

If you’ve got an iostat capture, a PSI reading or just a graph that doesn’t make sense, send it over and I’ll tell you what I see in it.

The post Troubleshooting High I/O Wait Without Blaming the Disk appeared first on John Nessime.

Then you check the Glue job history and the last successful run was six days ago. The job has not failed. It has not run at all. Because it never ran, it never emitted a metric, so CloudWatch recorded nothing and Grafana drew nothing. The panel is not green because the pipeline is healthy. It is green because the pipeline is silent.

That gap is the most common problem I see with Grafana monitoring for AWS data pipelines, and it is not a Grafana bug. It follows from how AWS batch services emit metrics and how alert engines read an empty result set. This post covers that failure and the three that follow close behind: wrong signal, cost, and access. Then the dashboard layout, the alerting rules, and a troubleshooting list for empty panels.

Failure family one: the pipeline stopped and nothing turned red

Application metrics are continuous. A web server emits request counts every minute forever, so a gap in the line genuinely means something is wrong.

Pipeline metrics are not like that. Glue reports job metrics only while a job executes, and only if job metrics are enabled and the script initialises a GlueContext. Step Functions publishes execution metrics when executions happen. MWAA publishes DAG metrics when DAGs run. Between runs the metric does not exist. A daily job produces a few minutes of data and twenty-three hours of nothing.

So the question that decides whether your monitoring works is: what does your alert engine do with an empty result?

No Data and MissingSeries are not the same thing

• No Data means the query returned no data points at all. Grafana handles this with a per-rule setting and by default raises a separate alert named DatasourceNoData. That alert does not inherit your rule’s labels and it fires immediately, skipping the pending period. If your notification policy routes on a team label, it will not match.
• MissingSeries means the query still returns data but one series has disappeared. That is the dead-job case exactly: a query for failed tasks across all Glue jobs still returns rows for the jobs that ran, and the dead one just stops existing. Grafana holds the last state for a couple of evaluation cycles, then evicts the instance with a transition to Normal, annotated with grafana_state_reason set to MissingSeries.

Read that again, because it is the whole problem. A job that stops running produces an alert that resolves itself and sends a cheerful “back to normal” notification. Nobody investigates a resolution.

The fix: stop inferring health from absence

Do not build the “did it run” signal out of metrics that only exist when it runs. Publish a heartbeat: a custom metric your pipeline writes once, deliberately, at the point you consider the run complete.

aws cloudwatch put-metric-data 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions PipelineName=orders_daily 
  --value 1 
  --unit Count

Put that at the end of the Glue script, in the final Step Functions state, or in the last task of the DAG. Now “did it run” has a positive signal instead of an inferred one.

The heartbeat is still sparse, so you also need to tell the evaluator that absence means failure. In a CloudWatch alarm, that is the missing-data setting: the options are missing, ignore, breaching and notBreaching, and for a heartbeat you want breaching.

aws cloudwatch put-metric-alarm 
  --alarm-name "orders_daily-heartbeat-missing" 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions Name=PipelineName,Value=orders_daily 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 26 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts

One subtlety here bites people. CloudWatch looks back over an evaluation range wider than your evaluation periods, and if it finds enough real data points in that wider range, the missing-data setting is ignored entirely. For a once-a-day metric that behaviour can suppress the alarm you just built. The reliable way around it is metric math: fill the gaps before the alarm evaluates anything.

FILL(m1, 0)

Now every period has a value, zero means “no run completed”, and a plain threshold behaves the way you expected. FILL(m1, REPEAT) and FILL(m1, LINEAR) carry the last known value or interpolate instead, which suits gauges better than heartbeats.

On the Grafana side, set the no-data state explicitly on each rule and build a notification policy that catches alertname=DatasourceNoData, otherwise those alerts go nowhere. The heartbeat still matters even with that in place, because it turns a disappearing series into a present series with a bad value, and every rule engine handles that correctly.

Failure family two: you are monitoring the compute, not the data

The second failure is subtler. The job runs, succeeds, takes its usual time, and writes nothing, because an upstream API returned an empty page or a partition filter matched no files. Executor CPU and heap will not tell you that. Two signals will.

Freshness is time since the last successful completion, per pipeline. It maps directly onto “can I trust the report on my screen”, which is the only question your stakeholders are asking. Derive it from the heartbeat.

Volume is rows or bytes written per run, compared against the same run last week. A drop to zero with a successful exit code is the classic empty-load bug. For Spark jobs on Glue the built-in counters give you a usable start:

• glue.driver.aggregate.recordsRead and glue.driver.aggregate.bytesRead for input volume
• glue.driver.aggregate.numFailedTasks for partial failures that do not fail the job
• glue.driver.s3.filesystem.write_bytes and the glue.ALL equivalents for output across executors

Glue reports these roughly every thirty seconds as deltas from the previous report, so Sum over the run window is usually what you want rather than Average. Beyond those, the signals I put on a board first are ExecutionsStarted minus ExecutionsSucceeded from the AWS/States namespace, since aborted and timed-out executions land in separate counters and failure count alone misses them; DAG metrics from the AmazonMWAA namespace, remembering that metrics.metrics_allow_list and metrics.metrics_block_list control what reaches CloudWatch at all; consumer lag rather than throughput on streaming ingestion, because a stream will happily accept records nobody is reading; and message age on dead letter queues, which is a near-perfect proxy for “something is failing repeatedly and nobody noticed”.

Failure family three: the dashboard costs more than the pipeline

Grafana’s CloudWatch data source uses GetMetricData to fetch samples and ListMetrics when you pick dimensions in the editor. GetMetricData does not qualify for the CloudWatch API free tier, and it is billed per metric requested rather than per API call. One request pulling five hundred metrics is five hundred billable metric requests.

Now multiply: panels, times series per panel, times statistics per series, times refresh rate, times every user with the board open, times every alert rule on its own schedule. A wildcard query matching every Glue job in the account looks tidy in the editor and quietly becomes the largest line item in your monitoring bill. Asking for more than five statistics on one metric counts as an extra metric request too.

The levers, roughly in order of return:

1. Set a sane refresh interval. Metrics that arrive every thirty seconds do not need a five-second refresh, and a daily batch board does not need auto-refresh at all.
2. Replace wildcard dimension matching with explicit resource lists on always-open boards. Keep wildcards for ad hoc investigation.
3. Query only the regions you run in. It is easy to leave a data source pointed at a region you migrated out of.
4. Use template variables instead of one duplicated board per team.
5. If you pull a large metric catalogue continuously, evaluate CloudWatch metric streams. Streams push through Amazon Data Firehose to a destination you control, and AWS recommends that path for third-party tools with heavy GetMetricData usage. The trade-off is genuine: you lose metric math and Metrics Insights against CloudWatch directly, and you pick up storage and Firehose ingestion costs instead.

To find which client is driving the spend, enable CloudTrail data events on GetMetricData. It surfaces the calling IAM principal and source address, which is usually enough to identify the forgotten dashboard.

Failure family four: access, accounts and regions

Grafana ships CloudWatch support natively, so there is no plugin to install. What there is, is an IAM policy people habitually make too broad or too narrow. This is the read-only set the data source needs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarms",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetMetricStatistics"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingRegions",
      "Effect": "Allow",
      "Action": "ec2:DescribeRegions",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Prefer an assumed role over long-lived access keys. If Grafana runs on EC2 or ECS in the same account, the instance or task role is cleanest and there is no secret to rotate.

Two boundaries to plan for. CloudWatch quotas are defined per account and per region, so a multi-account estate can hit limits in one account while the rest are fine, and each needs its own quota increase. And metrics do not cross regions: a pipeline that reads in one region and writes in another needs both on the board, and the missing half is a common reason a panel looks empty for no obvious cause.

Where to run Grafana for this

Three options, and the choice is mostly about viewer count and who carries the operational load.

• Amazon Managed Grafana bills per active user license per workspace, split into editor and viewer tiers, with at least one editor license required per workspace even if nobody logs in. An active user is anyone who logged in or made an API request during the billing cycle. Excellent for a small engineering team, expensive if you hand read-only boards to a large business audience.
• Grafana Cloud from Grafana Labs bills on usage rather than seats, which flips that calculation. It also brings the AWS observability integrations and Grafana Alloy if you want CloudWatch metrics in a Prometheus-compatible store.
• Self-hosted is cheapest per seat and most work. A small VPS from a provider like Contabo or InterServer runs Grafana for a handful of pipelines comfortably, and you own upgrades, TLS renewal and backups. One licensing note: Grafana’s open source core moved from Apache 2.0 to AGPLv3, while plugins and agents stayed Apache-licensed. If your legal team has a policy on AGPL, have that conversation before deployment.

Laying out a board people actually read

Organise by question, not by service. Whoever opens this board on a Monday morning has one question, and the answer should be visible without scrolling.

1. Freshness table. One row per pipeline, hours since last successful completion, threshold colour tied to that pipeline’s actual SLA. Most viewers read only this row.
2. Volume against baseline. Records or bytes written per run, overlaid with the same period a week ago.
3. Failure counts and durations, grouped by stage so a slow extract is distinguishable from a slow load.
4. Resource detail at the bottom. Executor memory, DPU usage, throttling. Useful during an incident, noise the rest of the time.

Use a template variable for pipeline name rather than one board per pipeline. It keeps the query count down and new pipelines appear automatically. For cross-pipeline aggregation without fanning out into individual series, CloudWatch Metrics Insights runs a SQL-style query inside CloudWatch and the Grafana data source supports it:

SELECT SUM(ExecutionsFailed)
FROM SCHEMA("AWS/States", StateMachineArn)
GROUP BY StateMachineArn

Each GetMetricData operation can carry only one Metrics Insights query, so treat it as a way to consolidate one fan-out, not to run several at once.

Troubleshooting: why the panel is empty

1. Check the region on the query, not the data source. Grafana allows a per-query region override and it is easy to leave one on default.
2. Widen the time range to seven days. If data appears, the metric is sparse and you have a heartbeat problem, not a query problem.
3. Check the statistic. Sum and Average behave very differently on delta counters, and empty-looking panels are often Average over a window that is mostly gaps.
4. Check dimensions match exactly. With Match Exact on, every dimension must be specified, so a metric published with two dimensions returns nothing when queried with one. Turning Match Exact off makes Grafana generate a search expression instead, which is the usual fix.
5. Check job metrics are enabled. For Glue you need both the job metrics option and a GlueContext in the script. Without the GlueContext, nothing is emitted regardless of the console setting.
6. Check IAM. A role with ListMetrics but not GetMetricData populates every dropdown perfectly and returns no data, which is confusing until you have seen it once.
7. Check for throttling. Panels that are intermittently empty under load rather than consistently empty usually mean a CloudWatch API quota for that account and region.

For alert rules behaving oddly rather than panels, treat the Error state separately from No Data. Grafana raises DatasourceError on evaluation timeouts or repeated query failures, governed by the evaluation_timeout and max_attempts settings. Slow CloudWatch queries under quota pressure trip this, and the resulting alert looks nothing like the one you wrote.

Common mistakes

• Alerting on failure counts only. A job that never starts cannot fail, so failure-count alerting is blind to the most common outage.
• Leaving no-data handling on the default and never routing DatasourceNoData. The alert fires into the void.
• Treating a MissingSeries resolution as good news. If alerts resolve without anyone acting, check the reason annotation.
• Importing a large community dashboard pack wholesale. Most of those panels use wildcards, and you have just subscribed to a permanent GetMetricData bill for metrics nobody reads.
• Monitoring the orchestrator but not the destination. Step Functions reports a clean success while the load that ran inside it wrote zero rows.

Best practices worth the effort

• Emit a heartbeat from every scheduled pipeline using one consistent namespace and dimension scheme. Consistency is what lets a single dashboard cover everything.
• Write down a freshness SLA per pipeline, then set the threshold from it. “Alert if stale” only means something once someone says how stale is too stale.
• Provision dashboards and alert rules as code in the same repository as the pipeline. A pipeline that ships without its monitoring is not finished.
• Route by severity, not by service. Freshness breaches page; executor memory warnings go to a channel.
• Break something on purpose to test the alert. Disable the schedule on a non-production pipeline and confirm you get paged. An untested alert is a guess.

Frequently asked questions

Do I need a plugin to connect Grafana to CloudWatch?

No. Grafana includes native support for the Amazon CloudWatch data source. Configure credentials or an assumed role, set a default region, and query. The same data source also reaches CloudWatch Logs and X-Ray.

Why does my Grafana alert resolve itself when the pipeline is still broken?

Almost certainly MissingSeries eviction. When one series in a multi-series query disappears, Grafana holds the last state briefly and then transitions the instance to Normal with grafana_state_reason set to MissingSeries. It is not treated as No Data. A heartbeat metric with gaps filled to zero stops the series disappearing in the first place.

CloudWatch dashboards or Grafana for AWS data pipelines?

CloudWatch dashboards are the shorter path if everything sits in one account and the visualisation options suit you. Grafana earns its place when you need pipeline metrics correlated with a warehouse query, a database, a trace or a business KPI on one screen, and when alerting has to route to several destinations. Most data platforms end up wanting the correlation.

How do I monitor a Glue job that succeeds but writes no data?

Alert on output volume rather than exit status. Use the Glue write byte counters, or better, emit your own row-count metric after the write so it reflects what your code believes it wrote. Compare against the equivalent run a week earlier, because fixed thresholds age badly as data grows.

Does Grafana querying CloudWatch cost money?

Yes. Grafana uses GetMetricData, which is billed per metric requested and excluded from the CloudWatch API free tier. Cost scales with panels multiplied by series multiplied by refresh frequency multiplied by viewers, so wildcards on an auto-refreshing board are the expensive pattern. Check current rates on the CloudWatch pricing page and do the arithmetic for your own board.

Amazon Managed Grafana or self-hosted?

It comes down to viewer count. Managed Grafana bills per active user per workspace, which suits a small engineering team and gets costly when read-only dashboards go out widely. Self-hosting removes the per-seat cost and adds upgrade, TLS and backup work. Grafana Cloud sits between them on usage-based billing. Count likely viewers first, then choose.

What is the minimum useful monitoring for a new pipeline?

A heartbeat proving the run completed, a row or byte count proving it moved something, and an alert that fires when the heartbeat is missing for longer than the agreed freshness window. Everything else is refinement.

The one thing to take away

Effective Grafana monitoring for AWS data pipelines is not about drawing more metrics. It is about refusing to infer health from absence. Batch pipelines are silent most of the time, and every monitoring system in existence reads silence as calm unless you tell it otherwise.

Publish a heartbeat. Fill the gaps with zero. Set your no-data behaviour deliberately. Then alert on freshness and volume, because those are the two things people downstream will notice before you do. Do that, and a green panel means something.

Need help building this on your AWS estate?

I design and build monitoring for AWS data platforms. This is the kind of work I take on:

• Heartbeat and freshness instrumentation across Glue, Step Functions, MWAA and Lambda pipelines, with one consistent namespace scheme so a single dashboard covers everything
• Grafana dashboards built around freshness and volume rather than executor charts nobody reads
• Alert rules and notification policies that handle No Data and MissingSeries correctly, routed to Slack, Teams, Telegram or email by severity
• CloudWatch query cost reviews, tracing GetMetricData spend back to the dashboard or rule causing it
• Cross-account and multi-region data source setup with least-privilege IAM roles instead of long-lived keys
• Grafana deployment and hardening on Amazon Managed Grafana, Grafana Cloud or a self-hosted instance behind TLS

If you have a pipeline that failed quietly, send me the job definition, the alert rule, or a screenshot of the panel that stayed green. I would rather look at the actual thing than talk in generalities.

The post Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem appeared first on John Nessime.

Both are accurate. The probe hits the homepage every minute from a datacenter, gets a 200 back, and moves on. It never logs in, never adds anything to a cart, and never runs the third-party payment script that is throwing a JavaScript error on one browser version. The check was answering a question nobody was asking.

That gap is what the synthetic vs real user monitoring argument is actually about. It isn’t a tooling preference or a budget line. It’s about which failures each approach is structurally incapable of seeing, and what happens when you only run one of them.

This post covers what each method really measures, the blind spot in each that bites hardest, how to choose when you can only afford one, and how to wire both together so they explain each other instead of arguing.

The blind spot that bites hardest: silence is not health

Start with the one that catches people out, because it is invisible until the worst possible moment.

Real user monitoring is passive. A script in the page collects timings and errors from actual visitors and beacons them back. No visitors, no data. So when your site is genuinely, completely down, the browser never loads, the RUM script never executes, and your error rate goes to zero.

Zero errors. A flat, quiet, beautiful dashboard. During an outage.

I’ve seen teams build alert rules on RUM error counts and then wonder why nothing fired. A threshold alert on “errors above X” cannot fire on a metric that has stopped reporting. If you alert on RUM at all, you need a companion rule on the absence of data, and that rule needs a sensible window, because low-traffic sites go quiet at 3am for entirely normal reasons.

The mirror-image problem on the synthetic side is quieter but just as expensive. A scheduled probe only ever tests what you told it to test. Every path you didn’t script is unmonitored, and it looks exactly the same on the dashboard as a path you did script and that’s working. Coverage gaps and healthy systems are visually identical.

So: synthetic monitoring is blind to what you forgot to check. RUM is blind when nobody is looking. Neither of those is a small caveat.

Synthetic monitoring: what a probe actually measures

A synthetic check is an active request on a schedule, from infrastructure you don’t own, against an endpoint you do. It ranges from a plain HTTP GET up to a full scripted browser session that logs in and walks a purchase flow.

The simplest useful version is something you can run by hand right now. This asks curl to throw away the body and print only the timing breakdown, which tells you where the time went rather than just how much of it there was:

curl -o /dev/null -s -w 
  'dns=%{time_namelookup} tcp=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total} code=%{http_code}n' 
  https://example.com/

Each value is cumulative from the start of the request, so you subtract to get the phase. A slow time_namelookup is DNS. A gap between time_connect and time_appconnect is the TLS handshake. A gap between time_appconnect and time_starttransfer is your application thinking. That last one is the number that actually correlates with user-visible slowness on a server-rendered page.

If you’re already running Prometheus, the self-hosted version of this is the blackbox exporter. It probes endpoints from outside and exposes the result as metrics. The config below is the whole trick: Prometheus doesn’t scrape the target, it scrapes the exporter and passes the target as a URL parameter.

scrape_configs:
  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com/
          - https://example.com/checkout
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

Those three relabel rules have to run in that order: copy the target into the query parameter, keep the original URL as the instance label so your graphs are readable, then overwrite the scrape address with the exporter. Get the order wrong and Prometheus scrapes the exporter’s own metrics endpoint, you get no probe_* series at all, and everything looks fine because there’s nothing red to see.

The metrics you’ll actually alert on:

groups:
  - name: blackbox
    rules:
      - alert: EndpointDown
        expr: probe_success == 0
        for: 3m
        labels:
          severity: critical
      - alert: CertExpiringSoon
        expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 21
        for: 1h
        labels:
          severity: warning

The certificate one is worth setting up even if you do nothing else. It’s the single highest-value synthetic check most small teams are missing, and it catches a failure mode that takes an entire site offline with no warning and no gradual degradation to notice first.

Where synthetic monitoring wins

• It works with zero traffic. A B2B app with forty users, a staging environment, a newly launched site: synthetic is the only thing that will tell you anything at all.
• It’s a controlled baseline. Same location, same schedule, same request. When the number moves, something on your side moved. RUM numbers shift because a marketing campaign brought in traffic on older phones.
• It tests paths users haven’t taken yet. A new region, a failover route, a payment provider you just switched to. RUM only ever sees sessions that already happened.
• It runs before deploy. The same scripted journey can gate a release in CI and then run on a schedule against production.
• It’s what an SLA is measured against. Availability commitments need a defined, repeatable observer. Real users are not that.

Where synthetic monitoring falls down

• The probe is on a better connection than your users. Datacenter fibre, a modern browser engine, no browser extensions, no battery throttling. It is a best case, permanently.
• Coverage equals imagination. Whatever you didn’t think to script is a silent gap, and it will be the thing that breaks.
• Scripted journeys rot. Every selector change, every A/B test, every consent banner redesign can break the check. Flaky synthetics that cry wolf get muted, and muted checks are worse than no checks.
• Cost scales multiplicatively. Frequency times locations times journeys. A one-minute browser check from eight regions is a very different bill from one HTTP check every five minutes.

Real user monitoring: what the browser actually reports

RUM is a small JavaScript agent that reads the browser’s own performance APIs and ships the results somewhere. Google’s web-vitals library is the reference implementation for the metrics that matter, and it’s deliberately tiny:

import { onCLS, onINP, onLCP } from 'web-vitals';

function send(metric) {
  navigator.sendBeacon('/rum', JSON.stringify({
    name:   metric.name,
    value:  metric.value,
    rating: metric.rating,
    id:     metric.id,
    path:   location.pathname
  }));
}

onCLS(send);
onINP(send);
onLCP(send);

Three things are worth understanding about that snippet. The callbacks fire when the metric is final, not on a timer, which is why Interaction to Next Paint can only be reported near the end of a session. sendBeacon is used instead of fetch because it survives the page being closed. And metric.rating gives you the good / needs-improvement / poor bucket without you hardcoding thresholds that Google may revise.

If you’d rather not build the collection side, the managed options split roughly into full-stack platforms (Datadog RUM, New Relic, Dynatrace), error-tracking tools that grew into RUM (Sentry), and open-source agents you can point at your own backend. Grafana Faro is the notable one in that last group: a browser SDK that ships web vitals, errors and traces to a collector you run yourself or to Grafana Cloud, which means your frontend signals land next to your Prometheus metrics and Loki logs instead of in a separate tab.

Where RUM wins

• It’s the only source of truth for Core Web Vitals. Google assesses LCP, INP and CLS from field data at the 75th percentile, via the Chrome User Experience Report. A perfect Lighthouse score is a lab result and carries no weight in that assessment.
• It surfaces the long tail. Mid-range Android on a congested mobile network is a cohort no synthetic probe represents, and it’s frequently the cohort that fails.
• It segments the way the business thinks. By country, by ISP, by device class, by page template, by traffic source. That’s how you find out the paid social landing page is the slow one.
• It catches client-side-only failures. A third-party tag that starts blocking render, a JS exception on one browser version, a consent script that stalls. The HTML came back 200. The page is still broken.

Where RUM falls down

• No traffic, no signal — the silence problem above, plus useless percentiles on any low-volume page.
• It’s reactive by construction. Every data point is a user who already had the bad experience. RUM cannot warn you about anything.
• You are sampled, ad-blocked and consent-gated. A meaningful slice of visitors never report, and they’re not a random slice. Privacy-conscious, extension-heavy users skew toward desktop and toward certain regions.
• The agent costs what it measures. You’re adding JavaScript to a page in order to find out how much JavaScript is slowing it down. Keep the agent small and load it late.
• Field data lags. CrUX aggregates over a trailing window, so a fix you shipped this week won’t show up in the assessment for weeks. Ship it, then wait, and resist re-litigating.

Synthetic vs real user monitoring: the honest split

Strip away the vendor framing and the division is clean. Synthetic answers “is the thing I defined still working?” RUM answers “what are people actually experiencing right now?”

Which means they belong to different jobs. Synthetic owns availability, contractual uptime, certificate and DNS health, and regression gates in CI. RUM owns experience, Core Web Vitals, conversion-adjacent performance work, and prioritisation. If you find yourself arguing about which is “better”, you’re comparing a smoke detector to a customer satisfaction survey.

Synthetic tells you something broke. RUM tells you whether it mattered, and to whom.

How I’d decide what to run first

Most teams can’t stand up both properly in the same quarter. Here’s the order I’d work in.

1. Get an external uptime check up today. Any of them. UptimeRobot, Better Stack, Pingdom, or blackbox exporter if you already run Prometheus. It must live outside your own infrastructure, or it dies with the thing it’s watching. Do check the current terms on free tiers before you build a client’s alerting on one, since several now restrict commercial use.
2. Add TLS expiry and DNS resolution checks. Cheap, and they catch total outages that have no warning phase.
3. Decide whether you have traffic. If a typical page gets meaningful daily sessions, RUM will produce usable percentiles and should come next. If it doesn’t, skip RUM entirely for now and spend the effort on scripted journeys instead.
4. Script your two or three money paths. Login, search, checkout. Not twelve. Each one is code you now have to maintain. Checkly is built around exactly this if you want the checks living in your repo as Playwright specs; Grafana Cloud Synthetic Monitoring covers the same ground with k6 scripts if your dashboards already live in Grafana.
5. Add probe locations only where you have users. Frequency times locations is your bill. Two well-chosen regions beat eight decorative ones.
6. Layer RUM last, and start read-only. Look at it for a month before you attach a single alert to it. You need to know what normal looks like per page template.

One thing worth checking before you buy anything: what your hosting already gives you. Managed WordPress plans and VPS providers like InterServer often include basic availability alerting and a status page, and Cloudflare exposes edge-side request and error data for anything behind it. That may be enough of the availability layer that you can spend the budget on the experience layer instead.

Mistakes that look like monitoring

• Running the check from inside the network it monitors. If your probe and your app share a VPC, a datacenter, or a DNS resolver, a shared failure takes out both and you learn nothing.
• Probing a health endpoint that always returns 200. A route that returns a hardcoded OK without touching the database is testing your web server’s ability to serve a string. Make the check exercise a real dependency, or add a content assertion on a page that only renders correctly when the database answers.
• Alerting on averages. A mean page load time hides every user who had a terrible one. Use p75 for experience work, because that’s what Google assesses, and p95 or p99 when you’re hunting outliers.
• Alerting on a single failed probe. One-off network blips are constant. Require consecutive failures, or failures from more than one location, before you wake anyone.
• Treating a Lighthouse score as a monitoring signal. It’s a diagnostic on a simulated device. Useful for finding the cause, worthless as an assessment of what users get.
• Letting synthetic traffic pollute analytics. Scripted browser checks look like sessions. Filter them out by user agent or a header, or your conversion rate quietly develops a dent.
• No alert on missing data. Applies to both sides. A probe that stopped running and a RUM stream that stopped reporting both look like calm.

Making the two views agree

The payoff for running both isn’t two dashboards. It’s that each one narrows the search when the other fires.

• Use consistent labels across both. Same page-template names, same region names, same environment tags. Otherwise correlating them is manual archaeology every time.
• Define availability from synthetic, experience from RUM, and write it down. avg_over_time(probe_success[30d]) is your uptime figure. p75 INP is your experience figure. Don’t let the two SLOs drift into using each other’s data.
• Let RUM tell you what to script next. When RUM shows a page template failing for one device class, that’s a synthetic check waiting to be written, throttled to match.
• Let synthetic explain RUM regressions. RUM says LCP got worse on Tuesday. Synthetic’s stored waterfall from Tuesday tells you which resource started arriving late.
• Alert on synthetic, investigate with RUM. Pages should come from the deterministic signal. The noisy, high-cardinality one is where you go to figure out the blast radius.
• Keep the scripted journeys in version control alongside the app they test, so a selector change and its check update land in the same pull request.

Frequently asked questions

Is uptime monitoring the same thing as synthetic monitoring?

Uptime monitoring is the simplest form of synthetic monitoring: a scheduled request checking for a successful response. Full synthetic monitoring extends that to multi-step browser journeys, API sequences, DNS, TCP and traceroute checks. Same category, different depth.

Can real user monitoring replace uptime checks?

No, and this is the most expensive misunderstanding in the whole synthetic vs real user monitoring discussion. RUM depends on pages loading. When the site is down, the RUM stream goes quiet rather than red, so an outage looks identical to a slow night unless you have explicitly alerted on the absence of data.

How often should synthetic checks run?

Work backwards from how fast you need to detect a problem, then account for the consecutive failures you require before alerting. A one-minute interval with a three-failure threshold means roughly three minutes to detection. Cheap HTTP checks can run frequently; browser journeys usually run every five to fifteen minutes because they cost far more per execution.

Does RUM slow down the site it’s measuring?

Slightly, and it’s a real trade-off rather than a rounding error. A minimal web vitals collector adds very little; a full session-replay agent is a much heavier thing to load. Load the agent asynchronously, sample sessions rather than capturing every one, and measure the agent’s own cost before you decide it’s free.

Why does my synthetic check pass while PageSpeed Insights fails?

They’re measuring different things from different places. A synthetic HTTP probe checks that a response came back; PageSpeed Insights reports both a throttled lab simulation and field data from real Chrome users. Neither is wrong. If the field section fails, real users on slower devices are having a worse time than your probe is.

Do I need both if I run a small brochure site?

Usually not. A low-traffic site gets almost nothing out of RUM because there aren’t enough sessions to make a percentile meaningful. Run solid external uptime and certificate checks, look at Search Console’s Core Web Vitals report for whatever field data exists, and add RUM when traffic justifies it.

Can I self-host all of this?

Yes. Blackbox exporter plus Prometheus and Alertmanager covers synthetic; Grafana Faro with a collector you run covers RUM. The catch is that self-hosted probes lose the one property that makes external monitoring valuable, which is independence. Run at least one check from a network you don’t control, even if everything else is yours.

The one thing worth remembering

Synthetic vs real user monitoring isn’t a choice between two products that do the same job. It’s a choice about which blind spot you’re willing to live with, and the answer to that changes as your traffic changes.

If you take one thing away: a quiet dashboard is not evidence of health. Synthetic monitoring goes quiet when you never wrote the check. RUM goes quiet when the site is down. Both silences look exactly like everything being fine, and both need an explicit alert on missing data before you can trust the green.

Need help getting your monitoring to match reality?

This is a good chunk of what I do. If your uptime page says one thing and your users say another, I can help with:

• Setting up external synthetic checks with sensible thresholds, so alerts fire on real failures and stop firing on network noise
• Blackbox exporter, Prometheus and Alertmanager configuration, including TLS expiry and DNS resolution alerting
• Scripting the two or three user journeys that actually earn money, and keeping them in version control next to the app
• Adding real user monitoring with Grafana Faro or a hosted RUM platform, sized so the agent doesn’t become the performance problem
• Building Grafana dashboards that put availability and Core Web Vitals side by side with consistent labels, so you can correlate them during an incident
• Auditing existing monitoring for coverage gaps, missing-data alerts, and checks that pass for the wrong reasons

Send me a screenshot of your current alert rules, a scrape config, or just the endpoint that keeps going green while people complain, and I’ll tell you what it’s missing.

The post Synthetic vs Real User Monitoring: What Your Green Dashboard Isn’t Telling You appeared first on John Nessime.

That gap is the whole problem with CloudWatch data pipeline monitoring, and it is why most teams find out about broken SaaS integrations from a business user rather than from a page. A web server that breaks returns 500s, and your error-rate alarm fires. A pipeline that breaks frequently returns exit code 0, writes a zero-byte object, and reports success. The connected app got revoked, a scope was dropped during an admin cleanup, or a filter clause quietly started matching nothing. None of the AWS-native metrics have an opinion about any of that.

This post is about what to measure instead: the four signals worth alarming on, the CloudWatch defaults that leave those alarms silently in the wrong state, how to emit custom metrics without turning your bill into a line item someone asks about, and what to check when a metric exists but the alarm still refuses to fire.

“Did the job fail?” is the weakest question you can ask

Job status is a proxy for pipeline health, and a bad one. It tells you the orchestrator finished its instructions. It says nothing about whether those instructions produced data.

SaaS sources make this worse than usual. A REST API returning HTTP 200 with an empty result array is, to your extraction code, a completely successful call. Pagination that stops on the first page because a cursor was reset is successful. An OAuth refresh that returns a token with fewer scopes than yesterday is successful right up until one object starts coming back empty. Every one of those is a green run.

Build the monitoring around four questions the job outcome cannot answer:

1. Liveness – did the run happen at all?
2. Volume – did it move a plausible amount of data?
3. Freshness – how old is the newest record that actually landed?
4. Shape – are the fields and values still what downstream expects?

Each needs a different kind of metric, and each fails silently in its own way.

Signal 1: liveness, and the missing-data trap that hides it

Step Functions publishes execution metrics to the AWS/States namespace, including ExecutionsStarted, ExecutionsSucceeded, ExecutionsFailed, ExecutionsTimedOut, ExecutionsAborted and ExecutionTime. The obvious move is an alarm on ExecutionsFailed. Most teams stop there, and that alarm is close to useless on its own.

Here is why. Those metrics are published as a result of executions. If somebody disables the EventBridge schedule rule during an incident and forgets to re-enable it, no execution starts, so no data point is published for that dimension at all. Your alarm has nothing to evaluate.

What happens next depends entirely on a parameter most people never set. CloudWatch alarms accept a TreatMissingData setting with four values: breaching, notBreaching, ignore and missing. The default is missing, which means the alarm keeps its current state when there is nothing to evaluate. An alarm that was sitting in OK stays in OK. Forever. It is green because it is blind, and there is no visual difference between the two.

The fix is not to change TreatMissingData on the failure alarm. It is to add a separate heartbeat metric that your pipeline emits on completion, and alarm on the absence of that heartbeat:

# Emitted at the very end of a successful run, per source system.
aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --value 1 
  --unit Count 
  --dimensions Source=salesforce,Stage=extract

Then the alarm that actually catches a dead scheduler:

aws cloudwatch put-metric-alarm 
  --alarm-name "ingest-salesforce-no-heartbeat" 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --dimensions Name=Source,Value=salesforce Name=Stage,Value=extract 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --datapoints-to-alarm 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts

The two flags doing the real work are --comparison-operator LessThanThreshold and --treat-missing-data breaching. The first catches a run that completes but emits nothing. The second catches a run that never happens at all. Without the second flag, silence reads as health.

Set --period comfortably longer than your schedule interval so one slow run does not page anyone, and use --datapoints-to-alarm to require consecutive breaches. One caveat: breaching is right for a heartbeat you control, but a poor blanket default across the account, because plenty of AWS metrics are only emitted when non-zero. Throttling counters are legitimately sparse, and forcing them to breach on silence just makes noise.

Signal 2: volume, and why a static threshold beats anomaly detection at first

A heartbeat proves the run happened. It does not prove the run did anything. Emit a row count, per source, per object:

aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RecordsWritten" 
  --value 14238 
  --unit Count 
  --dimensions Source=salesforce,Object=opportunity

Alarm on that being unexpectedly low rather than on it being zero. Zero-row alarms feel precise, but the more common failure is partial: a cursor that only advances through the first page, a rate limit that truncates the pull, an incremental sync that silently narrows its window. Those deliver a fraction of the expected rows, not none of them.

There is a subtlety here that trips people. If a run fails before emitting RecordsWritten at all, a low-volume alarm on that metric sees missing data, not a zero. Metric math fixes it cleanly:

FILL(m1, 0)

Wrapping the metric in FILL with a value of 0 substitutes zero for every empty period, so an absent run and an empty run both breach the same alarm. You get one alert instead of two competing ones, and it means what you think it means.

CloudWatch anomaly detection is the obvious upgrade for seasonal volume. It models expected behaviour and alarms on deviation from the band, which handles a Monday-morning spike that a static threshold cannot. I still would not reach for it first: it needs a stretch of representative history before the band is trustworthy, and during that stretch it will be confidently wrong. Start with a static floor well below your observed minimum, and move to anomaly detection once you know what normal looks like and can see the fixed number fighting you.

Signal 3: freshness, the metric almost nobody emits

This is the one that separates a pipeline you can trust from one you merely watch. Liveness and volume can both look perfect while the data is stale, because a job that re-reads the same window every run produces rows, produces a heartbeat, and produces nothing new.

The metric to emit is lag: the difference in seconds between now and the maximum source-side modification timestamp in the batch you just landed. Not your own processing timestamp, which will always look current. The source system’s timestamp, the one that tells you how far behind the truth you are.

import time
import boto3

cw = boto3.client("cloudwatch")

def emit_freshness(source, obj, max_source_ts_epoch):
    """max_source_ts_epoch: max LastModifiedDate in this batch, as epoch seconds."""
    lag_seconds = int(time.time()) - int(max_source_ts_epoch)
    cw.put_metric_data(
        Namespace="Pipelines/Ingest",
        MetricData=[{
            "MetricName": "SourceLagSeconds",
            "Value": lag_seconds,
            "Unit": "Seconds",
            "Dimensions": [
                {"Name": "Source", "Value": source},
                {"Name": "Object", "Value": obj},
            ],
        }],
    )

Alarm with GreaterThanThreshold, and set the threshold to roughly two schedule intervals plus your worst observed run duration. An hourly pipeline that takes twelve minutes on a bad day gets a threshold somewhere around two and a half hours. Tighter than that and you page on ordinary jitter.

Freshness is also the one metric worth putting in front of non-engineers. “Your Salesforce data is currently 40 minutes behind” is a sentence a stakeholder can act on; “ExecutionsSucceeded is 1” is not. If you surface dashboards to a business audience through Grafana Cloud or QuickSight over the CloudWatch data source, lag is the panel that earns its space.

Signal 4: shape, using logs instead of counters

Shape problems are schema drift and value drift. A SaaS admin renames a custom field, a picklist gains a value your mapping does not handle, a required field starts arriving null for a subset of records. The rows keep flowing. The row count looks fine. Downstream joins quietly drop records.

Counters alone are the wrong shape here, because what you need at 2am is not “the null rate rose”, it is “which records, and what changed”. Embedded Metric Format is built for exactly that split: you write one structured JSON log line, CloudWatch extracts the numbers as real metrics you can alarm on, and the full event stays queryable in Logs Insights for the investigation.

{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "Pipelines/Quality",
        "Dimensions": [["Source", "Object"]],
        "Metrics": [
          { "Name": "NullRequiredFields", "Unit": "Count" },
          { "Name": "UnmappedFields", "Unit": "Count" }
        ]
      }
    ]
  },
  "Source": "salesforce",
  "Object": "opportunity",
  "NullRequiredFields": 412,
  "UnmappedFields": 2,
  "unmapped_field_names": ["Renewal_Tier__c", "Partner_Code__c"],
  "batch_id": "2f8c1a9e"
}

Two things to get right. Every name listed in Dimensions must also exist as a top-level property in the same JSON object, or extraction silently produces nothing. And note that unmapped_field_names and batch_id are properties, not dimensions: they ride along in the log event for querying but never become metric dimensions, which is exactly what you want.

When the alarm fires, the investigation is a Logs Insights query against the same log group:

fields @timestamp, Object, UnmappedFields, unmapped_field_names
| filter Source = "salesforce" and UnmappedFields > 0
| sort @timestamp desc
| limit 50

That hands you the new field names in seconds, which is the difference between a five-minute fix and an hour of guessing.

Keeping CloudWatch data pipeline monitoring off the top of your bill

Custom metrics and log ingestion are consistently the two largest CloudWatch line items for data teams. Neither has a flat fee, so the cost is a direct function of choices you make in code. Three matter far more than the rest.

Cardinality is the cost lever

A custom metric is billed per unique combination of namespace, metric name and dimension values. Source with five values and Object with twenty gives you a hundred metrics per metric name. Add Tenant with two hundred values and you have twenty thousand. Multiply by four metric names and the number stops being a rounding error.

Never put a run ID, batch ID, record ID or raw error string in a dimension. Those belong in the log event. Dimensions are for things you would actually build an alarm on, and you are not going to build an alarm per batch.

EMF over PutMetricData for anything high-frequency

PutMetricData is a synchronous API call on your critical path. In Lambda that means duration you pay for, plus a request that can throttle and needs handling. EMF avoids both: you write JSON to stdout, CloudWatch Logs extracts the metrics asynchronously, your function returns. For a per-run summary either is fine. For anything per batch or per record, EMF is the one I reach for.

The Infrequent Access log class will break this silently

This one has caught competent people. CloudWatch Logs offers a Standard and an Infrequent Access log class, and IA has a lower ingestion price, which makes it an obvious target during a cost review. According to the AWS documentation, the IA class does not support embedded metric format, metric filters or subscription filters. Logs Insights queries still work.

So if somebody moves your pipeline log group to IA to save money, every EMF-derived metric stops being extracted. The alarms built on those metrics go to INSUFFICIENT_DATA, or worse, sit in OK if they use the default missing-data behaviour. The logs still arrive. The dashboard still exists. The metrics are just gone.

Log class cannot be changed after a log group is created, so recovery means a new Standard log group and redirecting output to it. Keep metric-backing log groups on Standard and point IA at the genuinely archival stuff: verbose debug output, raw payload dumps, anything you only read during a forensic investigation.

Routing: one page, not four

Build all four signals and one broken pipeline fires the heartbeat, volume and freshness alarms within minutes of each other. Three pages, one problem. That is how alert fatigue starts.

Composite alarms fix it. Build one composite alarm per pipeline whose rule combines the child alarms with OR, put the notification action there, and leave the child alarms with no actions at all. They become diagnostic detail you read after the page rather than three separate interruptions.

SNS to email is fine for one person and poor past that, since it has no deduplication, escalation or acknowledgement. With more than one person on call, route through something with a schedule such as PagerDuty, Opsgenie or Better Stack. If your team already lives in Grafana Cloud or Datadog for application monitoring, pulling CloudWatch alarm state in through their AWS integration beats maintaining two alerting surfaces.

Troubleshooting: the metric exists but the alarm never fires

Roughly in order of how often I have seen each one.

• Dimension mismatch. The alarm’s dimension set must match the published metric’s dimension set exactly, all of them, in the same values. A metric published with Source and Object is a different metric from one published with only Source. Run aws cloudwatch list-metrics --namespace "Pipelines/Ingest" and compare the output against your alarm definition character by character.
• Alarm stuck in INSUFFICIENT_DATA. The metric is not being published at all for that dimension set, or the period is shorter than the publishing interval so most periods are empty. Lengthen the period to at least the emission interval.
• Alarm stuck in OK during a real outage. Classic default missing behaviour. Any alarm whose purpose is to detect absence needs --treat-missing-data breaching.
• EMF metrics never appear. Check the log group’s class first, then confirm every dimension name in the Dimensions array exists as a top-level property in the same JSON object, then confirm the JSON is a single line with no wrapping prefix from your logger.
• Freshness spikes on a schedule. Almost always a timezone mismatch between how the SaaS API reports modification timestamps and how you convert them. Normalise everything to UTC epoch seconds at the boundary and stop converting anywhere else.
• Row counts double intermittently. Retries emitting the metric twice for one logical run. Emit volume metrics once, after the write is confirmed, not inside the retried block.

Common mistakes

• Alarming only on job failure, which misses every silent-success failure mode.
• Leaving TreatMissingData at its default on alarms designed to detect absence.
• Alarming on zero rows instead of on unexpectedly low rows, so partial extractions slip through.
• Emitting a processing timestamp as the freshness metric, which is always current and therefore always useless.
• Putting high-cardinality identifiers into metric dimensions and then being surprised by the bill.
• Moving pipeline log groups to the Infrequent Access class without checking what depends on metric extraction.
• Wiring notification actions to every child alarm instead of to one composite alarm per pipeline.
• Building dashboards nobody looks at instead of alarms that reach a person.

Best practices worth the effort

• Emit liveness, volume, freshness and a shape metric for every source. Four metrics per source is a small amount of code and covers most of what actually breaks.
• Define alarms in Terraform or CloudFormation alongside the pipeline, so a new source cannot ship without monitoring.
• Use consistent dimension names across every pipeline. Source and Object everywhere makes search expressions and dashboards trivial; a mix of source, system and connector makes them impossible.
• Set alarm descriptions to a one-line runbook. The person paged at 3am should not have to guess what the alarm means.
• Test the absence case deliberately. Disable the schedule in a non-production account and confirm the heartbeat alarm fires. An untested absence alarm is an assumption.
• Set explicit log retention on every pipeline log group. Never-expire is the default and it compounds quietly.
• Keep an eye on the source side too, not just yours. Most SaaS APIs publish their own rate limit or API usage counters, and pulling those into a metric is often the earliest possible warning.

FAQ

Is CloudWatch enough on its own for data pipeline monitoring?

For alerting, yes, if your workloads are on AWS. It has metrics, alarms, composite alarms and log querying, and it is already collecting service metrics you would otherwise have to ship somewhere. Where it gets weak is exploratory analysis and cross-source correlation, which is where teams add Grafana, Datadog or a dedicated data-observability tool on top. Run CloudWatch for the paging path first and add the analysis layer when you have an actual gap.

How do I monitor a pipeline that runs on a managed service like Glue or AppFlow?

The same four signals apply; only the emission point changes. Managed services publish their own metrics, but those describe the service’s execution, not your data. Where you cannot add code inside the job, hook the completion event through EventBridge to a small Lambda function that inspects what landed and emits the volume and freshness metrics. It is a handful of lines and it is the only way to get data-level signals out of a job you do not control.

Should each tenant get its own metric in a multi-tenant pipeline?

Usually not as a dimension. Tenant counts grow, and cardinality is billed per unique dimension combination. Emit aggregate metrics with Source and Object dimensions, put the tenant identifier in the log event as a property, and use Logs Insights to slice by tenant when you need to. Break out a dimension per tenant only for the small number of accounts where a contractual SLA makes it worth the cost.

What is a sensible freshness threshold?

Start at two schedule intervals plus your worst observed run duration, then adjust after watching real data for a couple of weeks. The threshold that matters is the one the business would notice, so if a stakeholder can tolerate half a day of staleness on a nightly export, do not page at four hours. Match the alarm to the actual consequence.

Why does my alarm sit in INSUFFICIENT_DATA instead of ALARM?

CloudWatch has no data points to evaluate for that metric and dimension set. Either nothing is publishing, the dimensions do not match what is published, or the alarm period is shorter than the interval at which you emit. Confirm the metric exists with list-metrics first, and only then look at the alarm configuration.

How much does this cost to run?

It depends on three things you control: the number of unique custom metrics, the volume of log data ingested, and how much data your Logs Insights queries scan. Alarms and dashboards are minor by comparison. Keep dimension cardinality low, set retention deliberately, and query narrow time windows, and the monitoring stays well below the cost of the pipeline it protects. Check current rates on the AWS pricing page rather than trusting any number in a blog post, including this one.

The one thing worth remembering

Effective CloudWatch data pipeline monitoring is built around absence, not around errors. Your SaaS integrations will fail far more often by succeeding at nothing than by throwing an exception, and every default in CloudWatch treats an absent metric as a non-event.

If you only change one thing after reading this, add a heartbeat metric with --treat-missing-data breaching to your most important pipeline, then go and disable its schedule in a test account and confirm the alarm actually fires. If it does not, you have just found out on your own terms instead of from a stakeholder asking why the numbers look wrong.

Need a second pair of eyes on your pipeline monitoring?

I work with teams running SaaS-to-warehouse pipelines on AWS, usually at the point where something broke quietly and nobody wants that to happen twice. Things I can help with here:

• Auditing existing CloudWatch alarms for silent-failure gaps, particularly missing-data configuration and dimension mismatches that leave alarms permanently green.
• Instrumenting liveness, volume, freshness and data-quality metrics into pipelines built on Lambda, Glue, Step Functions or self-hosted schedulers.
• Designing EMF logging and metric namespaces that stay useful as source and object counts grow, without cardinality running away.
• Building composite alarms and routing so one broken pipeline produces one actionable page rather than a cluster of them.
• Cutting CloudWatch spend on log ingestion and custom metrics without losing the signals you actually rely on.
• Writing Terraform modules that ship monitoring alongside every new source connector, so coverage is structural rather than remembered.

If you have a pipeline that has gone quiet, send me the alarm configuration and a Logs Insights output from the run in question and I will tell you what it is missing.

The post CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing appeared first on John Nessime.

So you open the orchestrator. Every task is green. The run started on schedule, finished inside its usual window, exit code zero across the board. Nothing paged. And the numbers are still wrong.

That gap is the whole reason to build a data pipeline operations dashboard. Not to watch DAGs go green, which the orchestrator UI already does perfectly well, but to answer the questions the orchestrator structurally cannot answer. This post covers the four signals worth putting on a screen, where each one actually comes from, how to wire them into Prometheus and Grafana without inventing a metrics platform, and how to turn the panels into alerts that fire before a human notices.

Why green pipelines still ship bad data

An orchestrator tracks process outcomes. Did the Python exit cleanly, did the SQL statement return without an error, did the container terminate zero. That is a real signal and you should keep it. It is just answering a different question than the one your users care about.

Consider what a perfectly successful run looks like when the upstream source has quietly changed:

• An API starts paginating differently and your extractor pulls the first page only. It ran fine. It loaded 500 rows instead of 500,000.
• A source column gets renamed and your transform selects it with a coalesce default. It ran fine. Every row now says “unknown”.
• A partition arrives late, so the incremental filter finds nothing to process. It ran fine. It did nothing.
• An upstream team backfills, and your dedupe key no longer holds. It ran fine. Everything is doubled.

None of those raise an exception. All four are visible in under a second if you are tracking row counts and freshness at the table level. This is the failure family that eats the most incident time, because the clock does not start when the pipeline breaks. It starts when a human notices, which can be days later, and by then the bad data has been copied into reports, cached in a BI extract, and quoted in a meeting.

The four questions a data pipeline operations dashboard has to answer

Resist the instinct to start from what your tools can emit. Start from the questions you ask during an incident, in the order you ask them. Everything that does not answer one of these is a second-page panel.

Did it run at all?

Sounds trivial, and it is the one most homegrown dashboards get wrong. A panel showing “last run: success” does not distinguish between a pipeline that succeeded ten minutes ago and one that succeeded eleven days ago and has not been scheduled since. You want time since last successful completion, as a number that grows, not a status pill that sits there looking calm.

Did it finish in time?

Duration matters less than duration against a deadline. A load that takes forty minutes is fine if the report is read at 9am and the run starts at 6am. The same forty minutes is an incident if someone moved the schedule to 8:40. Put the deadline on the panel as a threshold line, not in a runbook. Trending duration also catches slow decay: a stage that creeps from five minutes to fifty usually means data growth, a bad query plan, or resource contention, and it will breach your window eventually.

Did the right amount of data arrive?

Rows in versus rows out, per stage. This is the highest value panel per unit of effort on the entire dashboard. A drop from a million rows to twenty thousand is not subtle once you plot it, but it is completely invisible in an exit code. Compare against the same weekday from the previous week rather than against yesterday, because most business data has a weekly shape and Monday-versus-Sunday comparisons generate noise you will learn to ignore.

Is the data still the shape you expect?

Null rates, distinct counts on key columns, and schema changes. This is the panel almost nobody builds, and it is where the expensive failures live. A gradual rise in null rate on a joining key means an upstream system is degrading and your joins are silently dropping rows. You do not need statistical anomaly detection to catch that. You need the number plotted over time next to the others.

There is a fifth question that belongs on a different surface: what breaks downstream when this table is wrong. That is lineage, and it answers blast radius rather than health. Keep it one click away, not on the operations screen.

Where each signal actually comes from

Three sources, three different collection mechanisms. This is the part that takes an afternoon rather than a sprint, provided you do not try to build a unified agent.

The orchestrator, for run and timing signals

Airflow has emitted StatsD metrics for a long time, and that path is still the one most deployments run. Airflow speaks StatsD, so you put a statsd_exporter in front of it to translate into Prometheus format. Enable it in the metrics section of your config, or with the equivalent AIRFLOW__METRICS__ environment variables if you are on containers:

[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 9125
statsd_prefix = airflow
metrics_allow_list = scheduler,executor,dagrun,pool,triggerer

Two things worth knowing before you roll this out. First, metrics_allow_list is not optional at any real scale. Airflow emits per-DAG and per-task metric names, so without a prefix filter your cardinality grows with every DAG anyone adds, and you find out when the time series database starts struggling. Second, Airflow expects the StatsD destination to be resolvable at startup, so a scheduler that boots before the exporter can fail to come up. Order your dependencies accordingly.

StatsD has no concept of labels, so the exporter has to reconstruct them from the dotted metric name. That mapping file is the fiddly part:

mappings:
  - match: "*.dag.*.*.duration"
    match_metric_type: observer
    name: "airflow_task_duration"
    labels:
      airflow_id: "$1"
      dag_id: "$2"
      task_id: "$3"

The wildcards map positionally onto $1, $2, $3, and match_metric_type has to match what Airflow actually sends, timer versus counter versus gauge. Get that wrong and the metric silently never appears at the exporter, which is a genuinely annoying half hour of debugging. Check the exporter’s own metrics endpoint before you go looking at Prometheus, because if it is not there it never left Airflow.

Airflow also has a native OpenTelemetry path for both metrics and traces, which skips the StatsD translation entirely and gives you real labels rather than reconstructed ones:

[metrics]
otel_on = True
otel_host = otel-collector
otel_port = 4318
otel_prefix = airflow
otel_interval_milliseconds = 30000

The port there is the collector’s OTLP HTTP receiver, and you need an OpenTelemetry Collector or compatible endpoint in the path. If you are standing up a new stack, this is the one I would reach for, because the label problem disappears and traces give you per-task spans under a DAG run parent span, which is worth a lot during a slow-pipeline investigation. If you already have working StatsD mappings and dashboards built on them, migrating is real work for a modest gain, and the honest answer is that it can wait. Metric names do shift between Airflow major versions either way, so pin your dashboard queries to the metrics reference for the version you actually run.

Managed Airflow changes the plumbing but not the shape. On Amazon MWAA the metrics land in CloudWatch instead, and Google Cloud Composer and Astronomer each have their own export path. The four questions do not change.

The transformation layer, for data-level signals

If you use dbt, most of what you need is already being written to disk and thrown away. Every invocation produces run_results.json in the target directory, containing per-node status and timing, and dbt source freshness produces its own sources.json. The manifest.json holds the dependency graph, which is your lineage.

The cheapest useful thing you can do is parse the failures out at the end of the run:

jq -r '.results[]
  | select(.status != "success")
  | [.unique_id, .status, (.execution_time | tostring)]
  | @tsv' target/run_results.json

One trap if you run dbt in ephemeral Kubernetes pods: the target directory dies with the pod. Upload the artifacts to object storage as the last step inside the same task that ran dbt, not as a downstream task, or you will find your history is empty exactly when you want it.

Elementary is the low-friction option here. It installs as a dbt package with an on-run-end hook that loads the artifacts into your warehouse as ordinary tables, which means your dashboard can query test results and model timings in SQL alongside everything else. Great Expectations and Soda cover the same validation ground with more expressive checks and more setup. The commercial platforms, Monte Carlo and the vendors around it, add automated anomaly detection and column-level lineage across the whole stack, and the genuine case for them is that they find the checks you did not think to write. The trade-off is that you are paying per monitored table for statistical inference, and on a stack of twenty tables that inference is not telling you much that a row-count panel and a null-rate panel would not.

Without dbt, you write the check yourself and it is still short. One query per critical table, run on a schedule:

SELECT
  COUNT(*)                                        AS row_count,
  MAX(updated_at)                                 AS last_event,
  SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) AS null_keys
FROM analytics.orders
WHERE loaded_at >= CURRENT_DATE - 1;

Batch jobs, and the Pushgateway caveat

Prometheus pulls. Batch jobs finish and vanish long before a scrape interval comes round, so there is nothing to pull from. The Pushgateway exists for exactly this: short-lived service-level jobs push their results to it, and Prometheus scrapes the gateway instead.

cat <<EOF | curl --data-binary @- 
  http://pushgateway:9091/metrics/job/orders_sync
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $(date +%s)
# TYPE pipeline_rows_loaded gauge
pipeline_rows_loaded ${ROWS}
EOF

Now the caveat, which is the single most common way a pipeline dashboard lies to you. The Pushgateway is a cache, not an aggregator, and it never forgets. Push a success metric, then have the job stop running entirely, and that success sits there being scraped forever. Your dashboard stays green because the last known value is green, and the fact that nothing has updated it in six days is not something a status panel can see.

Two defences, and you want both. First, always push a timestamp gauge rather than a boolean status, so staleness is arithmetic rather than inference. Second, alert on absence as well as on value, because the Pushgateway also exposes push_time_seconds per group, which tells you when anything last arrived regardless of what it said. Prometheus’s usual safety net, the up metric going to zero when a target disappears, does not apply here: Prometheus is scraping the gateway successfully, and the gateway is happily serving stale data.

Also keep instance identifiers out of the grouping labels. Push per service, not per worker, or you accumulate a permanent time series for every container that ever ran.

Laying out the dashboard so it works during an incident

The layout constraint is that someone is reading this at 7am on a phone with a stakeholder waiting. Three rows, top to bottom, in the order you need them:

1. Freshness across everything. One panel, one row per critical table, showing time since last successful load against its threshold. If this row is clean, the pipelines are fine and the problem is elsewhere. That is a genuinely useful thing to establish in five seconds.
2. Volume and duration per pipeline. Row counts with a week-over-week comparison, and stage duration with the deadline drawn on. This is where you localise a problem to a specific pipeline and stage.
3. Data quality detail. Null rates, failed test counts, schema change events. This is where you work out what actually went wrong once you know where.

The freshness query in PromQL, given the timestamp gauge above:

time() - max by (job) (pipeline_last_success_timestamp_seconds)

Grafana renders that nicely as a table with per-row thresholds, which is what makes the top row scannable. If you are self-hosting, Prometheus plus Grafana on a modest VPS from Hetzner, InterServer or DigitalOcean handles a surprising amount of pipeline metrics, because pipeline telemetry is low-volume compared to application metrics. Grafana Cloud’s free tier is a reasonable alternative if you do not want to own the storage, and Datadog or New Relic make sense if your organisation is already paying for one of them and you would rather have one pane than a better one.

Keep it to one screen. Every panel you add past that dilutes the ones that matter, and a dashboard nobody can read at a glance is a dashboard nobody opens.

Turning panels into alerts

A dashboard that requires someone to look at it is a dashboard that finds problems the morning after. The freshness panel converts into an alert almost directly:

groups:
  - name: pipelines
    rules:
      - alert: PipelineStale
        expr: time() - max by (job) (pipeline_last_success_timestamp_seconds) > 5400
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} has not completed successfully in over 90 minutes"

      - alert: PipelineMetricsMissing
        expr: absent(pipeline_last_success_timestamp_seconds{job="orders_sync"})
        for: 30m
        labels:
          severity: page
        annotations:
          summary: "No metrics received from orders_sync"

The first catches a pipeline that is late or broken. The second catches the one that stopped reporting entirely, which the first cannot see, because an expression over a series that no longer exists returns nothing and an empty result does not fire. That pairing is the whole trick, and it applies to every push-based metric you collect.

Set the threshold from the deadline the data has to meet, not from how long the job usually takes. Ninety minutes is not “the run takes an hour so give it some room”. It should be the point past which someone downstream is materially affected.

Troubleshooting a dashboard that shows nothing useful

The metric never appears in Prometheus. Work backwards along the path, one hop at a time. Curl the statsd_exporter’s own metrics endpoint first. If the metric is not there, it never left the orchestrator, so the problem is the config or an allow-list that filtered it out. If it is there but not in Prometheus, the problem is your scrape config. Do not start in Grafana.

The metric appears with no labels, or with the DAG name baked into the metric name. Your statsd_exporter mapping did not match. Check the wildcard count against the actual dotted name, and check match_metric_type, which is the usual culprit.

Everything is green but the data is old. Classic Pushgateway staleness. Query push_time_seconds for the affected group and compare it to now. If it has not moved, the job is not running and you are looking at a cached value.

Volume alerts fire every Monday. You are comparing against the previous day on data with a weekly cycle. Compare against the same day last week using an offset, or exclude weekends from the baseline.

Gaps in the duration graph. If the metric is a timer sent only on completion, a run that is still going or that failed early sends nothing. That is correct behaviour, and it is why liveness needs its own panel rather than being inferred from the duration series.

Prometheus memory climbing after rollout. Cardinality. Per-DAG and per-task metrics multiply, and any label carrying a run identifier, timestamp or UUID creates a permanent new series. Tighten the allow list and drop the offending labels.

Common mistakes

• Treating job success as data correctness. This is the entire problem, and every other mistake is a variation of it.
• Pushing a boolean status instead of a timestamp, so staleness becomes invisible.
• Alerting on threshold breaches but never on metric absence.
• Building the dashboard before agreeing what “healthy” means for each table with the people who consume it.
• Instrumenting every pipeline at once instead of the three that matter, then abandoning the project when the noise gets unmanageable.
• Putting instance or pod identifiers into Pushgateway grouping labels and accumulating dead series forever.
• Alerting on things nobody will act on at 3am. An alert that gets acknowledged and ignored twice has trained your team to ignore it permanently.

Best practices

• Baseline before you threshold. Collect for two or three weeks and look at the actual shape of the data before deciding what abnormal means.
• Start with freshness on your three most critical tables. It catches more real problems per line of code than anything else you can instrument.
• Define freshness thresholds from the consumer’s deadline, and write the consumer’s name next to the threshold so the number has an owner.
• Version dashboards and alert rules in Git alongside the pipelines. A dashboard edited live in the UI is a dashboard that will be silently different next quarter.
• Emit row counts at stage boundaries as a habit, not as a special case. In versus out at each hop is what turns “the numbers are wrong” into “stage three drops rows”.
• Review alerts monthly and delete the ones that never fired usefully. Alert rules are code and they rot the same way.
• Keep lineage accessible but off the main screen. It answers a different question and it competes for attention with the ones you need first.

Frequently asked questions

What is the difference between pipeline monitoring and data observability?

Monitoring watches the process: did the job run, did it succeed, how long did it take. Observability watches the data the process produced: is it fresh, is there the right amount, has its shape changed. You need both, and the second is the one most teams are missing when they say their dashboards did not catch something.

Do I need a commercial data observability platform?

Not to start. Freshness, volume and null-rate checks on your critical tables cover most real incidents and cost an afternoon. Commercial platforms earn their keep when you have hundreds of tables, several teams changing them independently, and nobody with time to hand-write checks. The honest signal that you have outgrown the homegrown version is when maintaining the checks becomes its own recurring task.

Should I use StatsD or OpenTelemetry for Airflow metrics?

New stack, OpenTelemetry: you get real labels instead of reconstructing them from dotted names, and traces come along with it. Existing stack with working StatsD mappings, stay put until you have another reason to touch it. Metric coverage and naming differ between the two paths and between Airflow versions, so verify against the metrics reference for your version rather than copying a dashboard from a blog post.

How do I monitor a pipeline that only runs weekly?

Same timestamp gauge, much larger threshold, and lean harder on the absence alert. Infrequent pipelines are where Pushgateway staleness does the most damage, because the last successful push can sit there for a fortnight looking entirely healthy. Check push_time_seconds for those groups specifically.

What should a data pipeline operations dashboard show first?

Time since last successful load, per critical table, with its threshold visible. That one panel answers the question you will be asked most often, and when it is clean it rules out the entire pipeline layer in a glance.

How many alerts is the right number?

Few enough that every page gets investigated. If someone can name an alert they routinely dismiss without looking, you have too many, and the cost is not the noise itself but the habit it builds. Start with freshness and absence per critical pipeline, add volume, and only go further when a real incident shows you a gap.

Can I build this without an orchestrator like Airflow?

Yes. Cron jobs, Step Functions, GitHub Actions and shell scripts can all push a timestamp and a row count to a Pushgateway in three lines. The orchestrator gives you the run and timing signals for free, but it is the data-level signals that do the heavy lifting, and those come from querying the warehouse regardless of what triggered the load.

The one thing to take away

A data pipeline operations dashboard is not a nicer view of your orchestrator. It exists because the orchestrator can only tell you that a process finished, and the question you are actually being asked is whether the data is right.

If you build one panel this week, build time since last successful load, sourced from a timestamp gauge, alerted on both threshold and absence. It costs an afternoon, it catches the failure mode that hides longest, and it turns “is the dashboard broken or just late?” into a question you can answer before anyone has to ask it.

Need help building or fixing this?

I work with teams on pipeline observability and the plumbing underneath it. Things I regularly help with:

• Designing freshness, volume and quality checks for your critical tables, with thresholds derived from real consumer deadlines rather than guesses
• Wiring Airflow into Prometheus and Grafana, including the statsd_exporter mappings and the cardinality controls that keep it affordable
• Migrating an existing StatsD setup to the OpenTelemetry path without losing your dashboard history
• Getting dbt artifacts out of ephemeral pods and into somewhere you can query them, with Elementary or with plain object storage
• Auditing an existing dashboard and alert set to find what is stale, what will never fire, and what is quietly training your team to ignore pages
• Standing up a self-hosted Prometheus and Grafana stack on a VPS, or sizing a managed alternative if owning the storage is not worth it to you

If you have a dashboard that went green through an incident, send me the panel queries and the alert rules and I will tell you where the gap is.

The post Building a Data Pipeline Operations Dashboard That Catches Silent Failures appeared first on John Nessime.