The same output showed roughly 11 GB in the available column. Nothing was wrong, and nothing had been wrong for months. The only thing broken was the alert that fired.

Low free memory on Linux is one of the most reliably misread signals in server operations. It looks like an emergency, it triggers scaling decisions, and it is almost always noise. This post covers what free is really telling you, which four signals actually indicate memory trouble, the one situation where a low free number genuinely matters, and what to put in your alert rules instead.

What the free command is actually reporting

free does not measure anything. It parses /proc/meminfo and prints a handful of those fields in columns. Every number it shows comes straight from the kernel and is readable directly if you want to skip the formatting.

A typical reading on a busy server looks roughly like this:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       3.4Gi       231Mi       128Mi        12Gi        11Gi
Swap:          2.0Gi          0B       2.0Gi

Read that left to right and the story is boring. Applications hold a few gigabytes. The kernel filled almost everything else with page cache and buffers, because every file that gets read or written passes through memory on the way, and directory metadata lands in reclaimable slab. What is left over, genuinely untouched, is the free column.

That last part is the bit people miss. Free memory is not a reserve. It is waste. A page holding a cached copy of a file your application reads every few seconds is doing useful work. A page sitting empty is doing nothing, and the kernel converts it into cache the moment it has a reason to. A server that has been up for a week and still shows plenty of free memory is a server that is barely touching its disk.

If you want the raw source rather than the formatted view:

$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SReclaimable|SwapTotal|SwapFree|Dirty):' /proc/meminfo

Two options are worth knowing. free -w switches to wide mode and splits buffers and cache into separate columns, which helps when attributing memory to a specific I/O pattern. free -h -s 5 reprints every five seconds, a crude but effective trend view while you reproduce a problem.

Signal one: available memory, not free memory

The available column maps to MemAvailable in /proc/meminfo, and it is the kernel’s own estimate of how much memory a new workload could allocate without pushing the system into swap. It has been present since kernel 3.14, so every distribution you are likely to be running exposes it.

What matters is that it is not free plus cache. That naive sum is what people calculated before MemAvailable existed, and it overstates the truth. The kernel starts from free pages, subtracts the reserves it refuses to dip below, then adds back the portion of file-backed page cache and reclaimable slab it believes it could actually release. Both additions are capped, because some cache is pinned and some reclaimable slab objects are in use.

The kernel documentation is explicit that this is an estimate whose accuracy varies from system to system, so treat it as a strong hint rather than an exact figure. It is still far better than anything you can compute yourself, and it is the number your dashboards should be built on.

If you change one thing after reading this: point your memory alerts at available, not at free. Everything else in this post is refinement on top of that.

Signal two: swap activity, not swap usage

The second most common false alarm is swap. Someone sees a few hundred megabytes of swap in use and concludes the machine is thrashing. It usually isn’t.

Swap usage is a stock: pages moved to disk at some point, possibly during a backup weeks ago, and never needed since. Leaving them there is correct, and pulling them back to satisfy a dashboard would evict something useful. Swap activity is a flow, and that is the part that hurts, because a page fault serviced from disk is orders of magnitude slower than one serviced from RAM.

vmstat gives you the flow. The si and so columns are swap-in and swap-out, in blocks per second over the sampling interval:

$ vmstat 5 6

Discard the first row. vmstat reports averages since boot on its first line, which is almost never what you want. Read the rows after that. If si and so are both consistently non-zero, the machine is simultaneously reading pages back to serve current work and writing pages out to make room for new allocations. That is thrashing, and it is a real signal with no ambiguity attached.

A single non-zero sample is not that. Brief swap activity during a log rotation or a backup window is normal. Sustained activity across several consecutive samples is what to escalate.

Signal three: memory pressure, which is the one that catches things early

Pressure Stall Information is the signal I reach for first now, and it is badly underused. It has been in the kernel since 4.20 and it answers a question none of the other tools answer directly: how much time are my processes losing because memory is contended?

$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

The some line is the share of wall-clock time in which at least one task was stalled waiting on memory. The full line is the share in which every non-idle task was stalled, meaning nothing useful happened at all. The averages cover ten seconds, one minute and five minutes, and total is a cumulative counter in microseconds that updates continuously, so it catches a two-second stall the running averages smooth away entirely.

Here is why this beats a percentage threshold. Utilization tells you how much of a resource is consumed; pressure tells you whether that consumption is costing you anything. A machine can sit at high utilization for months with zero pressure, which means the sizing is fine. It can also sit at moderate utilization while full climbs, which means reclaim is working hard and latency is already suffering with no threshold crossed.

With cgroup v2 mounted, the same data exists per cgroup in each directory’s memory.pressure file, so you can attribute pressure to a specific service rather than the whole machine. Prometheus node_exporter collects the system-level PSI counters, and if you already ship metrics to something like Grafana Cloud these graph well beside your latency panels.

Signal four: the OOM log, which is the only unambiguous one

When memory really does run out, the kernel writes a record. It does not do this quietly and it does not do it conditionally. If there is no OOM record, no process was killed for memory, full stop.

$ sudo journalctl -k --since "24 hours ago" | grep -i "oom"
$ sudo dmesg -T | grep -i "killed process"

The kill line names the victim and reports its memory breakdown at the moment of death: total virtual size, anonymous resident set, file-backed resident set, shared memory resident set, and the oom_score_adj that was in effect. The line above it, where a process “invoked oom-killer”, tells you which allocation triggered the whole event, and that process is frequently not the one that got killed.

That distinction is the most useful thing in the log. The kernel picks a victim by badness score, driven largely by how much memory a process holds relative to what is available to it, adjusted by oom_score_adj. A slow leak in a small helper can push the machine over the edge and get your database killed instead, because the database is simply the largest thing running. Read both lines before deciding who the culprit is.

If you have a service that should survive at almost any cost, systemd exposes OOMScoreAdjust= in the unit file, which persists across restarts in a way that writing to /proc/<pid>/oom_score_adj does not. Use it sparingly. Protecting a process that is itself leaking just moves the crash somewhere less convenient.

When low free memory on Linux really is the problem

Everything above assumes you are looking at a whole machine. Inside a container, the accounting changes, and this is the failure mode that is invisible until it isn’t.

The cgroup memory controller charges page cache to the cgroup that caused it. Clean, evictable, perfectly reclaimable cache still counts toward memory.current, and that is what gets compared against memory.max. A container reading a large file has just charged that file to its own budget.

$ cat /sys/fs/cgroup/memory.current
$ cat /sys/fs/cgroup/memory.max
$ grep -E '^(anon|file|slab_reclaimable) ' /sys/fs/cgroup/memory.stat

The split in memory.stat is what you actually want. anon is memory your process genuinely holds; file is page cache charged to you that the kernel can take back under pressure. A container at its limit on mostly file is in a very different position from one at its limit on mostly anon.

Two things follow from this that catch people out repeatedly:

• Reclaim is not instant. Under heavy pressure the reclaim path itself runs slowly, so a container can be OOM-killed while still holding clean page cache that was theoretically available.
• Running free inside a container lies to you. Unless something virtualizes /proc, you are reading the host’s /proc/meminfo, which knows nothing about your limit. The cgroup files are the only truth here.

cgroup v2 also has memory.high, a soft limit. Crossing it triggers reclaim and throttling rather than a kill, which turns a hard failure into a slow one. That is usually the better trade, and it is worth setting deliberately rather than leaving it at its default.

What to alert on

The alert on the ticket at the top of this post was watching the wrong number. Here is what a memory alert set looks like when it watches the right ones:

• Available memory below a percentage of total, sustained. Ten percent for ten minutes is a reasonable starting shape. A single thirty-second dip is noise and should never page anyone.
• Sustained swap-in and swap-out together. Rate of change on the swap counters, not the absolute amount of swap in use.
• Memory PSI rising. The some line moving off the floor is your early warning. The full line moving at all is your late one.
• Any OOM kill, ever. This one is binary and should always generate a ticket, even if the service recovered on its own.
• Per-container memory events. The cgroup memory.events file counts limit hits, and a rising count is a limit that needs revisiting.

Notice that free memory does not appear anywhere in that list. There is no threshold on the free column that produces a useful alert, because the number is a byproduct of the kernel’s caching behaviour rather than a measure of anything you care about.

Troubleshooting: which tool answers which question

When a memory question lands on you, the order you check things in matters more than the tools.

1. Did anything actually die? Check the kernel log for OOM records first. If there are none, this is a performance question and not a capacity failure, which changes everything about what you do next.
2. Is the machine stalling? Read /proc/pressure/memory. Zero pressure with low free memory means you are done. Go close the ticket.
3. Is it swapping right now? Run vmstat 5 for half a minute and watch si and so. Ignore the first row.
4. Who is holding the memory? ps aux --sort=-%mem | head -15 gets the shortlist. Resident set size double-counts shared pages, so summing the column overstates the total.
5. Is it the kernel rather than a process? If the process list does not account for the usage, check slabtop and SUnreclaim in /proc/meminfo. Unreclaimable slab growth is a different problem, usually a driver, filesystem or network workload creating enormous numbers of kernel objects.
6. Is it growing? A leak is a slope, not a level, and one reading tells you nothing. This is the whole argument for having the graph before you need it.

One more that saves time on crowded boxes: resident set size is the wrong measure of what a process really costs, because shared libraries are counted against every process that maps them. Proportional set size divides shared pages among the processes sharing them, and tools reporting PSS give a far more honest answer when you are deciding what to move off a busy VPS.

Common mistakes

• Dropping caches to make the number look better. Writing to /proc/sys/vm/drop_caches discards clean page cache and reclaimable slab. The kernel documentation says plainly that using it outside testing or debugging is not recommended, because everything dropped has to be re-read from disk. You trade a nicer-looking free output for an I/O spike and worse latency. It is a benchmarking tool, not an operations tool.
• Setting vm.swappiness to zero as a reflex. It does not disable swap and it does not make the machine faster. It makes the kernel strongly prefer evicting page cache over swapping anonymous pages, which on a database host means throwing away exactly the cache you wanted. Change it for a measured reason, not by default.
• Running with no swap at all. Swap is not just an overflow area. Without it the kernel loses the option of paging out genuinely cold anonymous memory, which means the only lever left under pressure is evicting cache or killing something. A modest swap file gives reclaim somewhere to go.
• Summing RSS across processes and comparing it to total RAM. Shared pages are counted once per process. The sum will exceed reality, sometimes dramatically.
• Resizing the instance before finding the slope. More RAM buys time against a leak. It does not fix one, and the bill arrives every month afterwards.

Best practices

• Graph available memory, swap rate and memory PSI on one panel. Together they read as a story; individually each one misleads.
• Keep the raw counters, not just derived percentages. A percentage of an instance size that has since changed cannot answer “was this happening last week”.
• Set container limits from observed anon usage plus headroom, not a round number someone picked, and put memory.high below memory.max so pressure shows up as slowness before it shows up as a kill.
• Alert on rate of change for anything that can leak. The steady climb is the signal; the level it reached is just when you happened to look.
• Right-size deliberately. On providers such as Contabo or InterServer, where RAM is a meaningful share of the monthly bill, the gap between a real requirement and a panicked upgrade is money you keep every month.
• Write the explanation down where your team will find it. Every organization rediscovers the free-versus-available confusion independently, usually at three in the morning.

Frequently asked questions

Why is free memory so low on my Linux server?

Because the kernel used it for page cache and buffers, which is the intended behaviour. Any file read or written passes through memory, and the kernel keeps those pages in case they are needed again. It reclaims them automatically the moment an application needs the space. Check the available column instead.

What is the difference between free and available in free -h?

Free is memory that is completely unused. Available is the kernel’s estimate of how much a new process could allocate without swapping: free memory plus the portion of cache and reclaimable slab it believes it could release. Available is almost always much larger, and it answers the question you are actually asking.

Should I clear the page cache to free up memory?

No. The kernel reclaims cache automatically under pressure, and more selectively than a blanket drop. Writing to drop_caches forces everything to be re-read from disk, and the kernel documentation advises against using it outside testing or debugging. Doing it on a schedule is a strong hint something else is misconfigured.

Is swap being used a bad sign?

Not by itself. Pages that were swapped out during a quiet period and never needed again are fine where they are. What matters is swap traffic: sustained non-zero si and so in vmstat at the same time means the machine is thrashing. Alert on the rate, not the amount.

How do I tell if a process was killed for running out of memory?

Search the kernel log. journalctl -k filtered for “oom”, or dmesg -T filtered for “killed process”, shows a record naming the process, its PID and its memory breakdown at the time. The kernel always logs this, so if there is no record the process died for some other reason.

Why does my container get OOM-killed when memory looks fine inside it?

Because free inside a container usually reads the host’s memory information, not the cgroup limit being enforced. Read memory.current, memory.max and memory.stat instead. Page cache charged to the container counts toward its limit, so a read-heavy workload can hit the ceiling without any process holding much anonymous memory.

What is a good memory alert threshold?

Available memory below roughly ten percent of total, sustained for ten minutes, is a defensible starting point to tune against your own workload. Pair it with memory PSI and an unconditional alert on any OOM kill. A threshold on the free column, at any value, will page you for nothing.

Conclusion

Low free memory on Linux is the expected steady state of a healthy, working server. The kernel fills memory with cache because empty memory earns nothing, and hands it back the instant an application asks. The free column is a byproduct of that behaviour, not a health metric, and alerting on it produces exactly the kind of urgent ticket that wastes an afternoon and ends in a change request nobody needed.

The one thing worth remembering: free tells you what is idle, available tells you what you can have, and pressure tells you whether any of it is costing you. Watch the second and third, log the OOM events, and read the cgroup files rather than free whenever a container is involved.

Need help reading your memory signals?

Most of the memory work I get asked for is not “add more RAM”. It is working out whether the number someone is worried about means anything. I can help with:

• Rebuilding memory alerts around available memory, swap rate and PSI so they stop paging on normal cache behaviour
• Diagnosing repeated OOM kills, including the cases where the process being killed is not the one causing the problem
• Setting container and pod memory limits from observed anonymous usage rather than guesses, with a soft limit so pressure surfaces before a kill does
• Separating a genuine leak from normal cache growth using retained metrics rather than a single reading
• Right-sizing servers and instances honestly, up or down, based on what the workload actually needs
• Building the dashboard that turns the next memory question into a five-minute conversation instead of an incident

If you have a free -h output, a vmstat capture or an OOM log line that is bothering you, send it over and I will tell you what it means.

The post Low Free Memory on Linux Isn’t a Problem. Here’s What Actually Is. appeared first on John Nessime.

The crontab entry is still there. The script still works when you run it by hand. Nothing in the monitoring went red, because nothing was watching whether the job ran at all. It was watching whether the server was up.

That is the real question behind cron vs systemd timers, and it has almost nothing to do with syntax. Both tools can run a command at 2 a.m. The difference is what happens on the night the command doesn’t run, and how long it takes anyone to find out.

This post compares the two by failure family rather than by feature list: missed runs, overlapping runs, environment surprises, output that goes nowhere, and jobs that eat the machine. Then a decision procedure, the same job written both ways, and how to debug each one when it goes quiet.

The failure that actually decides it

Scheduled jobs fail silently far more often than they fail loudly. A job that crashes with a stack trace gets fixed the same week. A job that stopped firing six weeks ago gets found during an incident.

Cron’s default answer to “what happened?” is to mail the job’s output to the owning user. On a server with no local mail transfer agent configured, which is most servers built in the last decade, that output is generated and then discarded. The daemon logs that it started the command. It does not log what the command printed, and in most implementations it does not log the exit status either.

So you get a log line saying cron ran your backup, every night, forever, including on the nights the backup wrote nothing. That line is the trap. It looks like evidence.

systemd timers hand this to journald instead. Standard output and standard error from the triggered service land in the journal, tagged with the unit name, alongside the exit status and the runtime. That is the single biggest practical difference between the two, and it is the reason most of my new scheduled work goes into timers.

Neither one alerts you. Both will happily not run for a month while you sleep well. Observability is not monitoring, and I’ll come back to that.

Where cron still wins

Cron is one line in one file. There is no unit to reload, no second file to keep in sync, no install section to get wrong. crontab -e, five fields, a command, done. For a job that trims a log directory on a box you own, that economy is not nothing.

It also runs where systemd does not. BSD hosts, Alpine images, containers, appliance firmware, embedded boxes, older enterprise systems still on a legacy init. If your configuration management has to cover a fleet that is not uniformly systemd, cron is the common denominator and it is not close.

Per-user scheduling is genuinely simpler too. A user runs crontab -e and owns their own jobs, with no root involvement and no unit files in /etc. On shared hosting and on control panels like DirectAdmin or cPanel, this is the only model available, and the panel writes the crontab for you.

Where cron loses

• No dependency ordering. Cron cannot wait for the network to be up or the database to be ready. It fires at the wall clock time and hopes.
• No overlap protection. If the 2 a.m. run is still going at 3 a.m., the 3 a.m. run starts anyway, on top of it.
• No resource containment. A runaway job competes with production for CPU, memory and disk with nothing standing between them.
• Minute resolution, and no catch-up if the machine was off.
• A syntax with sharp edges that fail quietly, which is the next section.

Where systemd timers win

A timer separates when from what. The .timer unit holds the schedule. The .service unit holds the job. That split feels like bureaucracy until the first time you need to run the job right now, out of band, without touching the schedule. Then it is just systemctl start myjob.service, and the run is logged exactly like a scheduled one.

Everything systemd knows how to do to a service, it will do to a scheduled job. Ordering after other units. Restart-on-failure. Runtime caps. Memory and CPU limits through cgroups. A dedicated user. A private /tmp. A failure handler unit that fires when the job exits non-zero.

You also get a schedule you can interrogate before it bites you. systemctl list-timers prints every timer on the box with its next and last elapse. systemd-analyze calendar takes an expression and tells you exactly when it would fire, which is the closest thing either scheduler has to a unit test.

Where systemd timers lose

• Two files, a daemon-reload and an enable for every job. That is real friction when you have forty small jobs.
• The calendar syntax is more expressive than cron’s and harder to eyeball. You will reach for systemd-analyze calendar more often than you expect.
• Portability is gone. These units run on systemd hosts and nowhere else.
• User timers need lingering enabled, or they stop when the user logs out and never start at boot. This surprises people once, memorably.
• The journal is not a log file. If your log shipping only tails /var/log, timer output is invisible to it until you point the shipper at journald.

Cron vs systemd timers, by failure family

The machine was off at 2 a.m.

Cron has no memory. If the box was down, rebooting, or suspended when the schedule came round, that run is gone. Nothing catches it up and nothing records that it was skipped.

The traditional fix is anacron, which tracks the last successful run per job in a timestamp file and runs the job shortly after boot if the interval has lapsed. It is why the scripts in /etc/cron.daily still run on a laptop that is closed every night. The catch is that anacron works in whole days and only covers the cron.daily, cron.weekly and cron.monthly directories by default. It does not help your custom job at 02:00.

systemd folds that behaviour into a single directive:

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

With Persistent=true, systemd writes a timestamp under /var/lib/systemd/timers/ each time the timer fires. On boot it compares that stamp against the schedule, and if a run was missed it triggers immediately. This matters more than it sounds on any VPS you reboot for kernel updates, and a great deal on hosts at providers like Contabo or InterServer where you take maintenance windows on their timetable rather than yours.

One caveat worth knowing: “immediately” means immediately. If you have twelve persistent timers and the box has been off for a week, all twelve fire within seconds of boot, on a machine that is still starting services. Pair Persistent=true with RandomizedDelaySec= so the catch-up is spread out.

Two copies of the same job

This is the one that corrupts data. A sync job normally takes four minutes and runs every five. One night the remote end is slow, the run takes eleven minutes, and cron starts two more copies while the first is still writing.

Cron will not stop this. You have to do it yourself, and the standard tool is flock from util-linux:

*/5 * * * * /usr/bin/flock -n /var/lock/sync.lock /usr/local/bin/sync.sh

-n means fail immediately rather than queue behind the running copy. Without it you build a backlog of processes all waiting for the same lock, which turns an overlap problem into a fork bomb with a slow fuse.

systemd gets this for free from its own model. A unit is either active or it isn’t. If sync.service is still running when the timer elapses, there is no second copy to start, because starting an already-active unit is a no-op. You do not have to remember anything.

One default is worth knowing. A calendar timer schedules its next elapse from the previous trigger time, so if a run overran, that next elapse is already in the past and the service fires again the moment it finishes. Newer systemd releases add a DeferReactivation= boolean that schedules from when the service went inactive instead, so an overrunning job waits for the next real slot. Check your distribution’s systemd version before relying on it.

The environment your job actually gets

“It works when I run it manually” is the single most common scheduled-job bug, and both schedulers cause it, for the same reason: neither one gives your job an interactive login shell.

Cron runs the command with a short, hardcoded environment. Your shell profile is not sourced. Your PATH is much shorter than the one you tested with, which is why a script that calls docker, wp, aws or a version-manager shim works from your terminal and fails from cron. The working directory is the user’s home, not wherever you happened to be standing.

Cron adds one hazard that is entirely its own. Inside a crontab, the percent sign is a metacharacter: it becomes a newline, and everything after the first one is fed to the command as standard input. So this looks fine and is broken:

# Truncated at the first %. The redirect never happens.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

# Escaped. This is what you meant.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

There is no warning. The line parses, cron runs the truncated fragment, and the log says the job started. Anything with a date + format string in a crontab deserves a second look.

systemd services get a minimal environment too, so the class of bug is identical. What differs is that the fix is declarative and lives with the unit rather than being hidden inside a wrapper script:

[Service]
Type=oneshot
User=deploy
WorkingDirectory=/srv/app
Environment=PATH=/usr/local/bin:/usr/bin:/bin
EnvironmentFile=/etc/myapp/backup.env
ExecStart=/usr/local/bin/backup.sh

The honest summary: absolute paths for every binary solve most of this on either scheduler. The rest is solved by never putting logic in the schedule. Put it in a script and schedule the script.

Everything firing at once

Write 0 0 * * * on fifty hosts and fifty hosts hit your backup target, your package mirror or your API at exactly midnight. The classic cron answer is to hash the hostname into a minute offset in configuration management, which works and which every team reinvents.

systemd has two directives here and they pull in opposite directions, which is the part people get wrong:

• RandomizedDelaySec= spreads timers apart. It adds a delay between zero and the value you set, so identical schedules on many hosts land at different moments.
• AccuracySec= pulls timers together. It lets systemd shift the firing time within a window to coalesce wakeups and save power. It defaults to one minute, which is why a timer set to 02:00:00 often fires a few seconds late.

If you want a timer to fire close to the stated second, lower AccuracySec=. If you want to stagger a fleet, raise RandomizedDelaySec=. Setting only AccuracySec= to a large value and expecting it to spread load is the mistake, because coalescing is the opposite of spreading.

A job that eats the box

Cron has no answer here beyond nice and ionice in front of the command, and a script that hangs forever will hang forever. A systemd service is a cgroup, so the limits are declarative and enforced:

[Service]
Type=oneshot
MemoryMax=512M
CPUQuota=40%
IOWeight=20
TimeoutStartSec=30min
OnFailure=notify-failure@%n.service
ExecStart=/usr/local/bin/reindex.sh

The two directives that earn their keep on a shared box are TimeoutStartSec=, which kills a oneshot job that has hung, and OnFailure=, which starts another unit when this one exits non-zero. That second one is how you get a scheduled job to page you without wiring an alerting call into every script.

The same job written both ways

A nightly database dump that must not overlap, must not run before the database is up, and must catch up if the host was rebooted. In cron:

0 2 * * * /usr/bin/flock -n /var/lock/db-backup.lock /usr/local/bin/db-backup.sh >> /var/log/db-backup.log 2>&1

That handles overlap and captures output. It does not handle “the database wasn’t ready” or “the host was down”, and the log file is now your problem to rotate.

The systemd version is two files. First the service, which is the job:

# /etc/systemd/system/db-backup.service
[Unit]
Description=Nightly application database dump
After=network-online.target mariadb.service
Wants=network-online.target

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/db-backup.sh
TimeoutStartSec=45min

Then the timer, which is only the schedule:

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run the nightly database dump

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=15min

[Install]
WantedBy=timers.target

Because the two units share a base name, you do not need a Unit= line. systemd pairs db-backup.timer with db-backup.service by convention. Add Unit= only when the names differ.

Then the part people skip, which is verifying it before walking away:

# Check the units parse before you enable anything
systemd-analyze verify /etc/systemd/system/db-backup.*

# Confirm the calendar expression means what you think
systemd-analyze calendar --iterations=5 "*-*-* 02:00:00"

systemctl daemon-reload
systemctl enable --now db-backup.timer

# Next and last elapse for this timer
systemctl list-timers db-backup.timer

# Run it now, out of band, without touching the schedule
systemctl start db-backup.service
journalctl -u db-backup.service -n 50 --no-pager

Note that you enable the timer, not the service. Enabling the service instead would try to run the job at every boot, which is a memorable way to learn the difference.

How I’d decide

Work down this list and stop at the first line that applies. It resolves most cases in under a minute.

1. The host might not be systemd, now or later. Use cron. Portability beats every other consideration on this list.
2. The job is inside a container. Use neither. Use your orchestrator’s scheduler, or a timer on the host that invokes the container.
3. A missed run matters, or the job must not overlap, or it needs to wait for another service. Use a timer. Every one of those is a directive rather than a wrapper script.
4. The job is heavy: a dump, a reindex, an import, anything that can pin a core or fill memory. Use a timer, for the cgroup limits and the runtime cap.
5. You need to see output after the fact without building a logging path first. Use a timer. The journal is already there.
6. You need sub-minute scheduling. Use a timer with a monotonic directive. Cron’s floor is one minute.
7. Otherwise, cron is fine, and one line beats two files.

The migration question answers itself from that. Do not convert forty working crontab lines in a weekend. Write new scheduled work as timers, convert the jobs that have actually burned you, and leave the rest. Mixing both on one host is normal and supported. Just never schedule the same job in both places, which is easier to do than it sounds when a configuration management run and a manual edit disagree.

Diagnosing a job that never fired

Cron

Establish what actually happened before you theorise. The daemon logs every job it starts, so the log tells you which of three situations you are in: cron never tried, cron tried and the command failed, or cron never saw the entry at all.

# Is the daemon running at all?
systemctl status crond      # RHEL, AlmaLinux, Rocky
systemctl status cron       # Debian, Ubuntu

# What did it start, and when?
journalctl -u crond --since "24 hours ago"
grep CRON /var/log/syslog | tail -20     # Debian family
tail -50 /var/log/cron                   # RHEL family

# Whose crontab are you actually looking at?
crontab -l -u deploy
ls -l /etc/cron.d/

Then work through the usual suspects, roughly in order of how often they turn out to be the cause:

• Wrong crontab. The job is in root’s crontab and you are reading the deploy user’s, or it’s in /etc/cron.d where there is an extra user field before the command.
• PATH, or an unescaped percent sign. Dump what the job really sees with a temporary entry like * * * * * env > /tmp/cron-env.txt 2>&1, then compare it against your shell.
• Permissions. The script is not executable, or the crontab owner cannot read it.
• The schedule is not what you think. When both day-of-month and day-of-week are restricted, cron runs the job when either matches, not both. 0 0 15 * 1 fires on the 15th and on every Monday.
• The command captured no output, so a real error was mailed into the void. Append >> /var/log/myjob.log 2>&1 and wait for the next run.

systemd

# Is the timer loaded and scheduled? --all shows inactive ones too
systemctl list-timers --all

# Timer state, last trigger, next elapse
systemctl status db-backup.timer

# What the job printed, and how it exited
journalctl -u db-backup.service --since "7 days ago"

# Prove the schedule independently of the unit
systemd-analyze calendar --iterations=3 "Mon..Fri *-*-* 06:30:00"

• The timer is not enabled. Creating the files does nothing. It needs daemon-reload and enable --now.
• You enabled the service instead of the timer. Check with systemctl is-enabled on both.
• A user timer with no lingering. Under systemctl --user, the manager stops when the session ends unless you run loginctl enable-linger for that user.
• The calendar expression is valid but wrong. Valid syntax that never matches will not error. systemd-analyze calendar is how you find out.
• The service failed before your script ran. A missing User=, an ExecStart path that does not exist, a WorkingDirectory that is not there. The journal names the reason.

Arguments that don’t survive contact

“Cron is deprecated.” It isn’t. It is packaged and supported on every mainstream distribution, and control panels, hosting providers and countless application installers write crontab entries as a matter of course. Nobody is removing it.

“Timers are too complicated for small jobs.” Two files is more typing, not more complexity. The complexity that matters is the wrapper script you write to get locking, logging and a runtime cap out of cron, and how well the next person understands it. Compare like for like.

“The journal means I have logging sorted.” The journal is a ring buffer with a size cap. If nobody set SystemMaxUse= deliberately and nobody ships journal entries anywhere else, your evidence has a retention window you never chose. Check it before you rely on it.

“Switching to timers will stop jobs failing silently.” This is the one I’d push back on hardest. Timers make failures visible. They do not make them noticed. Neither scheduler will ever tell you that a job did not run, because absence is not an event. That needs a dead man’s switch: the job pings a heartbeat endpoint on success, and something external alerts when the ping stops. Healthchecks.io, Cronitor and Better Stack all sell exactly this, and Healthchecks is open source if you would rather self-host it next to your own Grafana. If you already run Prometheus, writing a completion timestamp through the node exporter’s textfile collector and alerting on its age gets you the same property without another vendor.

“Just move everything to timers.” A mass migration of working jobs is a lot of change for no new capability, and every converted job is a chance to fat-finger a path. Convert on cause, not on principle.

Frequently asked questions

Are systemd timers better than cron?

For production jobs on a systemd host, generally yes: logging, dependency ordering, overlap protection, catch-up and resource limits are all built in rather than bolted on. For a one-line job on a box you own, cron’s simplicity is a real advantage and the timer buys you nothing you will use.

Is cron deprecated or being removed?

No. Cron implementations are still packaged and maintained across the major distributions. Some distributions do not install a cron daemon in minimal images by default, which is a packaging decision rather than a deprecation, so check with your package manager before assuming a fresh host has one.

Can cron and systemd timers coexist on the same server?

Yes, and most real servers run both. The cron daemon itself is usually managed as a systemd service. The only rule is that a given job should be scheduled in exactly one place. Double-scheduling is what produces the “why did this run twice” ticket.

How do I see when a systemd timer will next run?

systemctl list-timers shows next and last elapse for all active timers; add --all to include inactive ones. To test an expression without creating a unit, use systemd-analyze calendar "Mon..Fri *-*-* 06:30:00" with --iterations=N for more fire times.

How do I stop a cron job from overlapping itself?

Wrap it in flock -n with a dedicated lock file. The -n flag makes the new run exit straight away instead of queuing. Do not use a PID file you wrote yourself; it will leak a stale lock the first time the job is killed, and then the job stops running forever.

Do systemd timers work inside Docker containers?

Not usefully. A container normally runs a single process without systemd as PID 1, so there is no service manager to own a timer. Schedule from outside the container instead: a host timer that runs the container, or the scheduler your platform provides. Podman is the exception worth knowing, since it can generate systemd units for containers and the host’s systemd drives them.

How do I convert a crontab line to a systemd timer?

Move the command into a Type=oneshot service with an absolute ExecStart, then translate the five cron fields into an OnCalendar= expression of the form DayOfWeek Year-Month-Day Hour:Minute:Second. Verify the expression with systemd-analyze calendar, enable the timer, then remove the crontab line only after you have seen a successful run in the journal.

The one thing worth remembering

Cron vs systemd timers is not a contest between an old tool and a new one. Cron is a scheduler. A systemd timer is a scheduler attached to a supervisor, and the supervisor is what you are actually buying: dependency ordering, overlap protection, catch-up, resource limits, and output that lands somewhere you can read it.

Pick timers when a missed or duplicated run costs you something. Pick cron when it doesn’t. And whichever you pick, add a heartbeat, because the failure that hurts is not the job that crashes. It’s the job that stopped running while everything stayed green.

Need a second pair of eyes on your scheduled jobs?

Scheduling is one of those areas where the problem is rarely the syntax. It’s the job nobody has checked since the person who wrote it left. This is the kind of work I take on:

• Auditing every crontab and timer on a host and telling you which jobs have quietly stopped running
• Converting fragile cron entries to systemd timers with locking, runtime caps and failure handlers, without a big-bang migration
• Fixing the classic “works by hand, fails on schedule” environment and PATH problems, including the percent-sign trap
• Adding heartbeat monitoring and alerting so a missed run pages someone instead of sitting there
• Backup and dump jobs that verify their own output rather than exiting zero on an empty file
• Untangling duplicate schedules left behind by control panels, deploy scripts and configuration management

If something on your server isn’t running when it should, send me the crontab line or the unit file and the journal output around the time it should have fired. That’s usually enough to tell you what’s wrong before we talk about anything else.

The post Cron vs systemd Timers: The Failure Modes That Decide It appeared first on John Nessime.

The rule was fine. The query was fine. The evaluation ran on schedule and the state transition happened exactly when it should have. What failed was everything after that, and none of it made a sound.

That is the failure mode worth designing around. Rule evaluation and notification delivery are two different systems, and a red rule only proves the first one works. Grafana alert notifications run against SMTP servers, Slack’s API and Telegram’s bot endpoint, all of which fail in their own quiet ways. A rejected login, a revoked webhook, a message over a length limit: each produces a log line and nothing else. The alert list still shows the rule firing, so everything looks healthy from the one place you’d think to check.

This post covers the practical setup for the three channels most small and mid-sized teams actually use: email, Slack and Telegram. It covers what breaks in each of them, how to route alerts so the right ones interrupt people and the rest don’t, how to provision the whole thing as files instead of clicking through the UI, and how to detect the case where delivery has stopped entirely.

Where a firing alert actually goes

Before touching any integration, it helps to know how many hops sit between “the rule fired” and “my phone buzzed”.

• The rule evaluates on its own schedule and produces alert instances, each carrying labels.
• Those instances go to an Alertmanager. Grafana ships its own and can also use an external Prometheus Alertmanager.
• The Alertmanager applies mute timings, groups instances, and walks the notification policy tree for a match.
• The matched policy names a contact point, which holds one or more integrations.
• Each integration makes its own outbound call to a third-party service.

Steps one and two are visible in the Grafana UI. Steps three through five are where things go wrong, and they are visible only in logs. That asymmetry is the whole problem.

One thing worth knowing early: a contact point can hold several integrations at once, so a single contact point named critical can fire email, Slack and Telegram together. Building one contact point per channel instead makes routing much harder than it needs to be.

Email: the channel everyone assumes works

Email is the one people configure first and verify least. The catch is that the email contact point holds only the recipient addresses. The actual SMTP credentials live in grafana.ini, so a perfectly valid contact point can sit there doing nothing because the server underneath was never configured.

[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
# Wrap the password in triple quotes if it contains # or ;
password = """your-smtp-password"""
from_address = alerts@example.com
from_name = Grafana Alerts
skip_verify = false
# Default is OpportunisticStartTLS
startTLS_policy = MandatoryStartTLS

Several details in that block matter more than they look.

• The port is part of host. There is no separate port setting. Port 587 with STARTTLS is the normal choice; 465 is implicit TLS and behaves differently, which is usually why one fails cleanly and the other hangs.
• startTLS_policy decides how hard Grafana insists on encryption. The default is opportunistic: it upgrades the connection if the server offers it and quietly continues in the clear if not. On a relay you control, mandatory turns a silent downgrade into a visible failure.
• from_address has to be something the relay accepts. Managed relays reject mail from unverified domains after the connection succeeds, which is why the Grafana side looks fine.
• Leave skip_verify false. It exists for internal relays with self-signed certificates. Turning it on to clear an error removes the check that was telling you the certificate is wrong.

Changes to this file need a Grafana restart. Nothing in the UI reloads SMTP settings.

Deliverability is a separate problem from delivery

Sending directly from a VPS is the fastest way to get alerts filed as spam. A fresh IP range from a provider like Contabo or InterServer has no sending reputation, and a mail server that emits one message a week during incidents never builds one.

Point Grafana at a transactional relay instead. Amazon SES, Postmark, Mailgun and Brevo all do the same job at this volume. Whichever you pick, publish SPF and DKIM records for the sending domain and set a DMARC policy, then send a real alert to a Gmail address and a Microsoft 365 address and check the spam folder. That is the only test that tells you anything.

Slack: webhook or bot token

Grafana supports two ways of posting to Slack, and the choice has consequences beyond setup time.

Incoming webhooks are the quick path. Slack generates a URL under hooks.slack.com/services/ that is bound to a single channel, and Grafana posts to it. There’s no OAuth flow and no scopes to reason about. The trade-off is that the URL is the credential and it maps to exactly one channel, so a team with six alert channels ends up managing six secrets with no central view of them.

A Slack app with a bot token takes longer. You create the app, add the chat:write.public scope so it can post to public channels without being invited to each one, then copy the bot user OAuth token, which starts with xoxb-. In Grafana you supply that token plus the channel ID as the recipient. One credential covers every channel, permissions are visible in one place, and revoking it kills all posting at once.

Webhooks win when you have one or two channels and want to be done in five minutes. The bot token wins as soon as channels start multiplying or someone asks what the integration is permitted to do. I reach for the webhook on small setups and regret it around the fourth channel.

Two constraints either way. A Slack integration targets one channel, so several channels means several integrations. And notification templates control the message title and body text, not its visual structure, so rich block layouts aren’t on the table.

Telegram: cheap paging, one nasty edge

Telegram is genuinely useful for out-of-hours alerts because it pushes to a phone without anyone paying for a paging product. The setup is two values: a bot API token and a chat ID.

1. Message BotFather and send /newbot. The username has to end in bot or _bot.
2. Copy the HTTP API token it returns. That’s the bot API token Grafana wants.
3. Create a group for alerts and add the bot to it. A bot cannot message a chat it has never seen.
4. Send a slash command in the group, such as /start@your_bot_name. An explicit command is the reliable way to make the group visible to the bot.
5. Fetch the chat ID from the Telegram API.

curl -s "https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates" | jq '.result[].message.chat'

The id field in that output is what goes in Grafana’s Chat ID field. Group chat IDs are negative. Copy the minus sign; leaving it off is the single most common reason a Telegram contact point tests as failing.

Now the edge case, and it’s the reason Telegram belongs in a post about invisible failures. Telegram caps messages at 4096 UTF-8 characters. If you’re using a parse mode other than none, truncating at that boundary can leave the markup unbalanced, and Telegram rejects the whole message as malformed. You don’t get a shortened alert. You get nothing.

This is exactly backwards, because the message most likely to blow the limit is a large grouped alert during a real outage. The channel works perfectly for months of single alerts, then goes quiet the first time forty instances fire together. Three ways to deal with it, in the order I’d try them:

• Write a deliberately short notification template for Telegram: alert name, severity, instance count and a dashboard link. Treat it as a pager, not a report.
• Narrow group_by on the policy that feeds Telegram so a single notification covers fewer instances.
• Set the parse mode to none, which makes truncation harmless at the cost of formatting.

Routing: split by severity, not by tool

The most common design mistake is a policy tree organised around channels: a Slack policy, an email policy, a Telegram policy. That gives you no way to express the thing you actually care about, which is that some alerts should wake someone up and most should not.

Put severity on the alert rule as a label and route on that instead. Contact points then become bundles of channels chosen to match urgency.

• severity=critical goes to a contact point holding Telegram and Slack. Something is broken and someone needs to look now.
• severity=warning goes to Slack only, in a channel people read during working hours.
• Everything else falls through to the default policy and lands in email, where it accumulates without interrupting anyone.

Four timing settings control how noisy this is, and each one does something specific:

• group_by decides what counts as the same notification. Grouping by alertname and a service label means one cascading failure produces one message instead of forty.
• group_wait (default 30 seconds) is how long to hold the first notification for a new group, so related instances that fire a few seconds apart arrive together.
• group_interval (default 5 minutes) is the minimum gap before a group that has already notified sends again because new instances joined it.
• repeat_interval (default 4 hours) is how often an unresolved alert re-notifies with nothing new to say.

The temptation is to drop repeat_interval to fifteen minutes so nothing gets forgotten. Resist it. Teams that do this stop reading the channel within a month, which converts a routing problem into a much worse cultural one. Longer intervals plus a real escalation path beats short intervals every time.

Grafana also offers a simpler path: pick a contact point directly on the alert rule form and skip label matching entirely. Reasonable when a rule has one obvious destination. The trade-off is that routing knowledge moves into individual rules, so changing where a class of alerts goes means editing every rule instead of one policy.

Provision it as files, not clicks

Contact points built through the UI are invisible to review, easy to delete by accident, and impossible to recreate after a bad restore. If Grafana alert notifications matter enough to build, they matter enough to keep in Git. Grafana reads alerting configuration from files in the provisioning/alerting directory under its provisioning path at startup.

Before writing any YAML by hand, build one contact point in the UI, test it, then use the export option on the contact points list to dump it as YAML. That gives you the exact setting keys for that integration type instead of the ones you half-remember from a blog post. Setting names are not consistent between integration types, and a guessed key produces a config that loads without error and sends nothing.

apiVersion: 1

contactPoints:
  - orgId: 1
    name: critical-oncall
    receivers:
      # Telegram receiver goes here. Export it from the UI first:
      # the setting key names differ from what you would guess.
      - uid: critical-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS
        disableResolveMessage: false

  - orgId: 1
    name: warnings-slack
    receivers:
      - uid: warnings-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS

  - orgId: 1
    name: email-digest
    receivers:
      - uid: email-digest
        type: email
        settings:
          addresses: ops@example.com
          singleEmail: false

And the policy tree that uses them:

apiVersion: 1

policies:
  - orgId: 1
    receiver: email-digest
    group_by: [grafana_folder, alertname]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: critical-oncall
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 1h
      - receiver: warnings-slack
        matchers:
          - severity = warning
        repeat_interval: 12h

Two things to know about provisioned resources. By default they become read-only in the UI, which is the point but surprises people the first time. And a contact point that a policy references cannot be deleted until the policy stops referencing it, so removals have to happen in the right order.

Keep tokens out of the files themselves. Grafana supports environment variable substitution in provisioning files, which means the YAML can live in Git and the secrets can come from your existing secret store.

Detecting the silence

Everything above still leaves the original problem unsolved. If the Slack app gets revoked or the SMTP password rotates, notifications stop and Grafana does not tell you.

The fix is a heartbeat alert, and it’s worth the twenty minutes. Create a rule whose condition is always true, give it a low severity label, route it to each contact point with a short repeat_interval, and point the resulting notification at an external dead-man’s-switch service. Healthchecks.io, Cronitor and Better Stack all offer this, and Healthchecks.io can be self-hosted if you’d rather not add a dependency. The service expects a signal on a schedule and alerts you when it stops arriving.

The logic is inverted on purpose. You aren’t monitoring infrastructure here, you’re monitoring the path that tells you about your infrastructure, and it has to be watched by something that isn’t itself.

As a second layer, Grafana exposes its own internal metrics on a Prometheus-compatible endpoint, including counters for notification attempts and failures. If you’re already scraping Grafana, alert on that failure counter increasing. It won’t catch a credential that works but delivers to a dead mailbox, which is why the heartbeat comes first.

Troubleshooting: the rule fires, nothing arrives

Work down this list in order. Each step rules out one hop, and stopping early is how people spend an afternoon on the wrong layer.

1. Test the contact point in isolation. The Test button sends a notification without touching rules or policies. If it fails, the problem is credentials or network and nothing else matters yet.
2. Watch the Grafana logs during the test. On a systemd install, run journalctl -u grafana-server -f while you click Test. SMTP, Slack and Telegram delivery errors all surface here, usually specific enough to fix directly.
3. Compare the instance’s labels against your matchers. A policy matching severity = critical never sees severity = Critical. Matchers are exact and case-sensitive. Read the firing instance’s real labels, not the ones you think the rule sets.
4. Look for a policy earlier in the tree that swallowed it. Evaluation stops at the first match unless the policy continues. A broad route near the top silently absorbs everything below it.
5. Check active silences and mute timings. A silence created during last month’s maintenance with a generous matcher is the classic. So is a mute timing in the wrong timezone.
6. Check whether you’re inside repeat_interval. A red alert now does not mean a notification is due now. This looks identical to a broken integration and isn’t.
7. For email, read the relay’s own logs. Grafana handing the message off is not delivery. Bounces and spam filing happen after Grafana’s involvement ends.

Common mistakes

• Configuring the email contact point and never touching grafana.ini. The contact point is only a recipient list. Without the SMTP block, it has nothing to send with.
• Routing by channel instead of severity. It feels tidy and it makes urgency impossible to express.
• Dropping the minus sign from a Telegram group chat ID. Group IDs are negative and the field takes the value literally.
• Sending verbose templates to Telegram. The 4096-character limit turns your biggest incident into your quietest one.
• Testing with the Test button and calling it done. Test bypasses the policy tree entirely. It proves the credential works, not that a real alert would ever reach it.
• Leaving resolved notifications on for every channel. Resolution messages in a pager channel double the volume and train people to ignore it.
• Storing webhook URLs and bot tokens in the UI only. They exist in one place, are invisible to review, and disappear with the database.

Best practices for Grafana alert notifications

• Set a consistent severity label on every rule from day one. Retrofitting labels across dozens of rules is miserable work.
• Use one contact point per urgency level, holding as many integrations as that level needs.
• Put a dashboard or runbook link in an annotation your templates render. An alert that says what broke but not where to look costs the responder five minutes every time.
• Write channel-specific templates: short for Telegram, richer for email, in between for Slack.
• Provision contact points and policies from files, with secrets injected from the environment, and run a heartbeat through every channel.
• Re-test every integration after credential rotation, Slack workspace changes or a Grafana upgrade. Those are the three events that quietly break delivery.

FAQ

Can one contact point send to email, Slack and Telegram at the same time?

Yes. A contact point holds multiple integrations and every one of them fires when that contact point is selected. This is the right way to build an escalation tier: one contact point per severity, with the channel mix chosen to match how urgently that severity needs attention.

Why does the Test button work but real alerts never arrive?

Test sends straight to the contact point and skips the policy tree. If it works and real alerts don’t, the break is in routing: matchers that don’t match, a broader policy earlier in the tree absorbing the alert, an active silence, or a mute timing. Open the firing instance, read its real labels, and compare them character by character against your matchers.

Do I need Grafana Cloud for any of this?

No. Contact points, notification policies, templates and all three integrations ship in the open source build. Grafana Cloud handles hosting and adds on-call tooling on top, which matters if you need rotations and escalation chains. For a team routing alerts to three channels, a self-hosted instance on a modest VPS does everything described here.

Should I use a Slack webhook or a bot token?

Webhook for one or two channels where speed of setup matters. Bot token once you have several channels, want one revocable credential, or need to see in one place what the integration is allowed to do. Both deliver identically; the difference is entirely in credential management.

Why did my Telegram alerts stop during a big outage?

Almost certainly the 4096-character limit. A large grouped notification exceeded it, truncation broke the message markup, and Telegram rejected the send outright. Shorten the template for that channel, group more narrowly, or drop the parse mode. It’s the failure mode most worth guarding against, because it only shows up when you need the channel most.

Can I keep using the Prometheus Alertmanager I already run?

Yes. Grafana can send to an external Alertmanager instead of its built-in one, and the contact point selector at the top of the alerting pages lets you choose which one you’re configuring. If you already have a mature Alertmanager routing config, keeping it and pointing Grafana at it is usually less work than migrating the tree.

The one thing to remember

A firing rule proves evaluation works. It proves nothing about whether anyone was told. Grafana alert notifications cross three or four systems after the rule turns red, and every one of them fails quietly by default.

So build the channels carefully, route by severity rather than by tool, then run a heartbeat through every path and let something outside Grafana watch it. The alternative is finding out during the incident, which is the one time you have no attention to spare.

Need alerting that reaches people?

Most of the Grafana setups I get asked to look at have good dashboards and alerting nobody trusts. Work I take on here:

• Fixing email delivery end to end: SMTP config, relay selection, SPF, DKIM and DMARC so alerts stop landing in spam.
• Setting up Slack and Telegram contact points properly, including bot apps, scoped tokens and channel-appropriate templates.
• Designing a policy tree around severity labels, with grouping and timing tuned so people keep reading the channel.
• Converting click-built alerting into provisioned YAML under version control, with secrets kept out of Git.
• Adding heartbeat monitoring so a broken notification path announces itself instead of waiting for an outage.
• Auditing alert rules for missing labels, missing runbook links, and rules that fire constantly and are therefore ignored.

Send me your notification policy export and the output of journalctl -u grafana-server from a period when an alert should have gone out, and I’ll tell you which hop is dropping it.

The post Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram appeared first on John Nessime.

That is the thing about DirectAdmin PHP-FPM pools that catches people out. You think you are tuning a site. You are not. You are tuning a user, and every domain that user owns is sharing the same worker budget, the same queue, and the same memory ceiling.

This post covers where DirectAdmin actually keeps the pool config, how to size pm.max_children from numbers your server gives you rather than from a blog post’s magic value, which per-site PHP limits are worth setting and which are theatre, why OPcache does not behave the way you expect on a multi-tenant box, and how to make all of it survive the next CustomBuild run.

The design fact that explains most of your 502s

DirectAdmin generates one FPM pool per user account, per PHP version. The pool is named after the Linux user. It listens on a Unix socket named after the user. It runs as that user. Apache or Nginx hands PHP requests to that socket for every domain and subdomain in the account.

So if a reseller client has twelve domains under one username, those twelve sites share a single value of pm.max_children. There is no per-domain pool. This is a long-standing feature request rather than an oversight, and it has a real consequence: the noisiest site in an account sets the availability of every other site in that account.

Once you have internalised that, the fix for a lot of “mystery 502” tickets stops being a config tweak and starts being an account layout decision. If a site matters commercially, give it its own DirectAdmin user. That is the cleanest isolation the panel offers without going off-piste, and it costs you nothing but a bit of admin tidiness.

The alternative, if you have User jails enabled, is DirectAdmin’s isolated FPM mode. That runs a separate php-fpm master process per jailed user instead of a shared master with many pools:

da config-set isolated_fpm 1
systemctl restart directadmin
da taskq --run 'action=rewrite&value=httpd'

It only applies to users who already have Jailed enabled. The trade-off is memory: one master process per user, plus a separate OPcache region for each, adds up quickly. DirectAdmin mitigates that by letting idle per-user FPM instances exit after a period of inactivity, but on a box with sixty accounts you will feel it. I would reach for this on a server hosting a handful of paying clients, not on a general shared box.

Where the pool config actually lives

This trips up almost everyone the first time, because there are two files with nearly the same name and only one of them matters for user sites.

• /usr/local/php83/etc/php-fpm.conf is the master config for that PHP version. Editing it by hand is how you learn that CustomBuild overwrites it.
• /usr/local/directadmin/data/users/USERNAME/php/php-fpm83.conf is the generated per-user pool. This is the file that governs a customer’s sites.
• /usr/local/directadmin/data/templates/php-fpm.conf is the template those per-user files are rendered from.

Substitute your PHP version for 83 throughout. To see what every account is currently running:

ls /usr/local/directadmin/data/users/*/php/php-fpm*.conf

grep -H 'pm.max_children' /usr/local/directadmin/data/users/*/php/php-fpm*.conf

The second command is the one I run first on any server I have just inherited. It gives you the entire worker budget of the box on one screen, and it is usually the moment you discover that forty accounts are all sitting on the stock default.

The DirectAdmin template ships with pm = ondemand and a low default child count, driven by a MAX_CHILDREN token. Ondemand means no workers exist until a request arrives, and idle workers are reaped after a timeout. On a shared box with many mostly-idle accounts, that is the right default and I would leave it alone. On a server dedicated to one busy WordPress site, the spawn latency on the first request after an idle period is real and switching to dynamic is defensible. Most people who switch to dynamic on a shared server are trading memory they do not have for a latency win they will not notice.

Sizing pm.max_children without guessing

Every tuning guide gives you a formula. The formula is fine. The input everyone gets wrong is worker memory, because they use a number from someone else’s server instead of measuring their own. A lean WordPress install and the same install with a page builder, a security plugin and WooCommerce are not remotely the same workload.

Measure it. This averages the resident memory of the workers currently running in one pool:

ps -eo rss,args | grep '[p]hp-fpm: pool wpuser' 
  | awk '{sum+=$1; n++} END {if (n) printf "%d workers, avg %.0f MBn", n, sum/n/1024}'

The bracket trick in the grep pattern stops grep matching its own process. rss is resident set size in kilobytes, which is why the awk divides twice. Run it during a busy period, not at three in the morning, and run it a few times.

Two honest caveats about RSS. It overstates real consumption, because workers forked from the same master share a lot of pages, including the OPcache region. And it understates your worst case, because a worker that has not yet hit a memory-hungry request looks cheap. Treat the average as a planning figure, not a guarantee, and leave headroom.

Then the arithmetic:

1. Take total RAM and subtract what MySQL or MariaDB, the web server, mail, and the panel itself need. On a WordPress box the database is usually the second-largest consumer and it is not optional.
2. Reserve a further slice for page cache and burst. I aim to leave a comfortable margin free rather than allocating to the last megabyte.
3. Divide what is left by your measured per-worker figure. That is your ceiling for all pools combined.
4. Split that ceiling across accounts by actual traffic, not evenly.

Step three is where people go wrong on cheap VPS plans. If you are running a dozen WordPress sites on one of the low-RAM tiers from a provider like Contabo or InterServer, the honest answer is often that you cannot give every account a generous pool and you need to either cut the site count or move up a tier. Setting pm.max_children to a number your RAM cannot back does not buy capacity. It buys the OOM killer, and the OOM killer usually takes the database with it, which turns a slow site into a broken one.

To raise the server-wide default:

da config-set php_fpm_max_children_default 30
systemctl restart directadmin
da build rewrite_confs
systemctl restart php-fpm83

For a single account, go to Admin Level → Custom HTTPD Configurations → domain.com and open the php-fpm entry for that PHP version. In the |CUSTOM1| textarea, set the token:

|?MAX_CHILDREN=40|

That syntax is a template token override, not a config line. It tells the template renderer to use 40 instead of the default when it writes pm.max_children for that user. Writing pm.max_children = 40 in that box instead is the single most common mistake here, and it does something worse than nothing: the template still emits its own pm.max_children, you now have the directive twice in one pool, and FPM refuses to start. On a server where one broken pool file stops the whole master, that takes down every site on the box.

Rewrite and restart after any change, then confirm the value actually landed in the generated file rather than trusting the panel.

Per-site PHP limits, and the difference that matters

The |CUSTOM2| textarea on the same page is where PHP settings go. The syntax is FPM pool syntax, not php.ini syntax:

php_admin_value[memory_limit] = 512M
php_admin_value[max_execution_time] = 120
php_value[upload_max_filesize] = 64M
php_value[post_max_size] = 64M
php_value[max_input_vars] = 5000

The admin in php_admin_value is not decoration. Settings written with php_value can be raised at runtime by the application through ini_set(). Settings written with php_admin_value cannot. That distinction is the whole game on a multi-tenant server.

WordPress and a good number of plugins will happily try to raise memory_limit at runtime, and some optimisation plugins push it to values that would let one bad request consume a large slice of the machine. If you want a genuine ceiling, use php_admin_value. If you want a sensible starting point that a developer can raise for a legitimate import job, use php_value. I use admin for memory and execution time, plain for upload sizes, because upload sizes are a support-ticket generator and not a stability risk.

max_input_vars deserves its own note. It is the one that silently truncates a large WooCommerce product save or a menu with hundreds of items, with no error anywhere. The user reports “my changes did not save” and every log is clean. If you support content-heavy sites, raise it before someone asks.

Letting site owners set their own

Under FPM there is no .htaccess route for PHP settings, because FPM is a separate process and never sees the file. DirectAdmin enables the per-directory alternative by default: a .user.ini file in public_html, controlled by the user_ini.filename setting. It handles the PHP_INI_PERDIR and PHP_INI_USER class of settings only, so it will never override anything you set with php_admin_value. That is the point of it.

The WP-CLI gap

None of this applies to WP-CLI. Cron jobs and shell commands run through the CLI SAPI and read the CLI php.ini, not your pool. So a plugin update that works fine in the browser can die on memory in a cron-driven wp cron event run, and you will hunt through the pool config finding nothing wrong. Check the CLI limits separately when a scheduled task fails but the same action succeeds in wp-admin.

OPcache is shared, and that is the invisible one

OPcache allocates its shared memory region when the FPM master starts, before any pool exists. Every pool under that master inherits the same region. You cannot meaningfully give one account its own OPcache budget through a pool directive, because by the time the pool is read the memory has already been carved out.

Two consequences, and the second is the one nobody plans for.

First, capacity. The default OPcache size is comfortable for one application and thin for thirty WordPress installs, each with its own core, theme and plugin tree. When it fills, OPcache starts evicting and eventually restarts, and every site on the version gets slower at once with nothing in any error log to explain it. Watch the wasted-memory and restart counters rather than guessing.

Second, the sizing you actually need scales with the total number of PHP files across all accounts on that PHP version, not the biggest site. Consolidating everyone onto one PHP version looks tidy and quietly makes this worse. Spreading accounts across two or three PHP versions gives each master its own region, which is an underrated side effect of not forcing everyone to upgrade on the same day.

OPcache is configured through CustomBuild rather than the pool. Edit the customised copy under CustomBuild’s custom directory so your changes survive updates, then rebuild and restart the FPM master.

Making changes survive the next rebuild

Anything you type directly into a generated file is temporary. The next rewrite_confs, PHP update or panel update will flatten it, usually weeks later, and the regression will look like it came from nowhere.

There are three durable places to put changes, in increasing order of blast radius:

• Per user, all PHP versions. /usr/local/directadmin/data/users/USERNAME/php/php-fpm.conf.custom1 and .custom2. These feed the same CUSTOM tokens as the panel textareas, so the settings follow the account when the user switches PHP version in Domain Setup. This is the one I reach for first.
• Server-wide, all users, all versions. /usr/local/directadmin/data/templates/custom/php-fpm.conf.custom1 and .custom2. If either exists, its contents are injected into every pool file DirectAdmin writes. Good for a baseline you want everywhere.
• A full custom template. Copy /usr/local/directadmin/data/templates/php-fpm.conf to the custom subdirectory and edit it. This is the only route for pool directives that have no token, but you now own that file forever and stop receiving upstream template improvements. Use it when nothing else works, and leave a comment at the top saying why.

Global first, then per-user on top. Both land in the same token, so a per-user value does not replace a global one, it appends. For a directive that can only appear once in a pool, that is a duplicate and a failed restart.

Troubleshooting

“server reached max_children setting”

The pool name in that warning tells you which account, not which site. Find the log first rather than guessing at a path:

grep -i 'error_log' /usr/local/php83/etc/php-fpm.conf

Before raising the limit, check whether the requests are legitimate. Cross-reference the timestamps against each domain’s access log in that account. Nine times out of ten it is one of three things: an uncached admin-ajax.php loop from a plugin, a bot crawling faceted search or filtered archive URLs that bypass page cache, or wp-cron.php firing on every request on a site with a heavy scheduled task. Raising pm.max_children to absorb bot traffic is paying for garbage with RAM.

502 or 504 with no PHP error

The request is dying at the FPM boundary rather than inside PHP. Usual causes are a worker killed by request_terminate_timeout, a full listen queue, or the master failing to start after a config change. Check the FPM service status before you check anything in WordPress. If FPM did not start at all, the panel will still show your settings saved and every site on that PHP version will be down.

Changes in the panel that do nothing

Read the generated per-user file after every change. If the value is not in there, the rewrite did not run or the token was wrong. If it is in there and PHP still reports something else, the master was not restarted, or something later in the pool is overriding it. A temporary phpinfo() file served by the site in question settles the argument in seconds. Delete it immediately afterwards.

Session or temp files that are not where you left them

If PHP writes to /tmp and you cannot see the files over SSH, that is systemd’s PrivateTmp giving the FPM service its own private namespace. Nothing is broken. The files exist, in a mount only that process can see.

disable_functions that will not budge

disable_functions is special. From a pool you can add to the list but not remove from it, because the php.ini value has already been applied. To allow a function for one account you have to stop setting it in php.ini at all and drive the whole thing from templates instead. DirectAdmin documents that pattern, and it is worth reading before you start rather than after you have half-done it, because the intermediate state removes protection server-wide.

Common mistakes

• Editing /usr/local/phpXX/etc/php-fpm.conf directly and being surprised when it reverts.
• Writing pm.max_children = 40 into a CUSTOM box instead of the |?MAX_CHILDREN=40| token, producing a duplicate directive and a dead master.
• Sizing workers from a memory figure found online rather than measured on the box.
• Using php_value for memory_limit on shared hosting, then wondering how a plugin raised it to 2G.
• Assuming pool settings apply to WP-CLI and cron.
• Raising limits to absorb bot traffic instead of blocking or caching it upstream.
• Putting a client’s revenue site in the same account as their abandoned projects.

Best practices for tuning DirectAdmin PHP-FPM pools

• One DirectAdmin user per site that matters. Isolation you get for free beats isolation you have to engineer.
• Keep ondemand on shared servers. Only move to dynamic when a single busy site owns the box.
• Set a global baseline in the custom template, then override per user. Do not hand-tune forty files.
• Use php_admin_value for anything that protects the server and php_value for anything that is only a convenience.
• Cut PHP requests before you add workers. Full-page caching, a CDN in front, and Cloudflare or similar handling bots removes far more load than any pool setting.
• Disable WordPress’s request-triggered cron and drive it from a real system cron, so scheduled work stops competing with visitors for workers.
• Alert on saturation, not just uptime. Worker utilisation per pool and PHP-FPM error log entries are the signals worth having in Netdata, Prometheus, or whatever you already run. A site that is queueing looks perfectly healthy to an HTTP check.
• Re-measure after major plugin changes. A page builder or a new security plugin moves per-worker memory enough to invalidate your arithmetic.

Frequently asked questions

Can I create a separate PHP-FPM pool for each domain in DirectAdmin?

Not through the panel. Pools are generated per user, per PHP version, and every domain in the account shares one. The supported ways to get separation are to move the domain to its own DirectAdmin user, or to enable isolated FPM mode for jailed users, which gives each user their own master process. Anything else means custom systemd units you maintain yourself, outside the panel’s knowledge.

What is a good pm.max_children value for WordPress?

There is no portable number, because the input is your measured per-worker memory and your available RAM after the database. Measure the average resident size of your workers under real traffic, subtract everything else the server needs from total RAM, and divide. A lean brochure site and a WooCommerce store with thirty plugins can differ by a factor of three or more.

Where does DirectAdmin store PHP-FPM pool configuration?

Generated per-user pools live at /usr/local/directadmin/data/users/USERNAME/php/php-fpmXX.conf, rendered from the template at /usr/local/directadmin/data/templates/php-fpm.conf. Do not edit either directly. Put durable changes in the custom files under the user’s php directory or in the templates/custom directory.

Why do my PHP settings disappear after a CustomBuild update?

Because they were written into a generated file. CustomBuild and DirectAdmin regenerate those files on update and on any config rewrite. Only the custom template files and the CUSTOM token areas persist.

Does .htaccess work for PHP settings under PHP-FPM?

No. FPM is a separate process from the web server and never reads .htaccess. Use a .user.ini file in the web root for the settings PHP allows at that level, or set the rest in the pool.

Should I switch from ondemand to dynamic or static?

On a shared server, no. Ondemand exists precisely so that dozens of idle pools cost you nothing. On a server dedicated to one high-traffic site, dynamic removes cold-start latency and static gives the most predictable memory profile. The question to ask is whether your idle pools outnumber your busy ones.

Can I give one WordPress site its own OPcache?

Not from the pool config. OPcache’s shared memory is allocated by the FPM master at startup and shared by every pool under it. The practical routes are isolated FPM mode, which gives jailed users their own master, or moving the site to a different PHP version so it sits under a different master.

The one thing to take away

DirectAdmin PHP-FPM pools are a per-user boundary, not a per-site one. Every tuning decision that follows, how many workers to allow, whether a memory limit should be a suggestion or a wall, whether a client’s twelve domains belong in one account, comes out of that single fact.

Measure your workers before you size them, use the CUSTOM token files so your work survives the next rebuild, and watch pool saturation rather than waiting for the 502 ticket. The site that goes down is rarely the site that caused it.

Need a hand with a DirectAdmin server?

Most of the work I do on DirectAdmin boxes is this kind of thing: sites that are fine in isolation and fall over together. If that sounds familiar, I can help with:

• Auditing every pool on a server and producing a worker budget that matches the RAM you actually have
• Tracking down which domain in a shared account is saturating the pool, and why
• Building a global custom template so per-user tuning stops being forty hand-edited files
• Splitting a multi-site DirectAdmin account into isolated users without breaking mail, DNS or SSL
• OPcache and per-version sizing on servers hosting many WordPress installs
• Wiring up pool-level monitoring and alerts so saturation shows up before the ticket does

Send me a pool file, the output of that RSS command, or the last hour of your PHP-FPM error log, and I will tell you what I see.

The post DirectAdmin PHP-FPM Pools: Why One WordPress Site Takes Down the Others appeared first on John Nessime.

Sometimes the disk really is dying. More often it isn’t, and the number you’re staring at is not measuring what you think it’s measuring.

This post covers troubleshooting high I/O wait the way I’d actually do it on a production box: what the metric means, why it lies in both directions, and how to work through the five failure families that produce it.

What high I/O wait actually measures

Here’s the definition that matters: iowait is a subcategory of idle time. For a given CPU, it’s the time that CPU spent doing nothing while at least one task scheduled on it had an outstanding block I/O request.

Two consequences fall out of that, and both will burn you. High iowait can mean nothing is wrong. If the box has spare CPU and one process is pulling a large file off disk, the CPU genuinely has nothing else to do. Copying a backup will push iowait toward 100 percent on an idle host, and that’s a working system, not a fault.

Low iowait can hide a severe problem. This is the one that costs people days. If the CPU has other runnable work, the kernel context switches to it and that time gets counted as user or system, not iowait. A busy application server can be stalling badly on storage while wa reads two percent, because the CPU is never actually idle.

Add the averaging problem on top. The wa figure in top is averaged across every core. One thread pinned to one core and blocked hard on disk shows up as roughly three percent on a 32-core machine. Press 1 in top to break out per-CPU lines before you conclude anything from the aggregate.

So iowait is a hint, not a verdict. Treat it as the thing that starts the investigation, never the thing that ends it.

Start with pressure, not percentages

Pressure Stall Information is the metric I reach for first, and it exists precisely because iowait is a poor proxy for pain. It answers the question you actually care about: how much time did tasks lose because they couldn’t get I/O?

cat /proc/pressure/io

You get two lines. some is the share of time at least one task was stalled waiting on I/O. full is the share of time every non-idle task was stalled at once, which is CPU capacity being burned for nothing. The avg10, avg60 and avg300 fields are percentages over the last 10, 60 and 300 seconds. total is a cumulative microsecond counter since boot, which is what you want for graphing because averages smooth away short spikes.

The practical rule I use: a rising full line on io is a real user-visible problem almost every time. A high some line with a flat full line usually means one noisy job is waiting while everything else gets on with its life.

If /proc/pressure doesn’t exist, the kernel was built without CONFIG_PSI, or it was built with PSI compiled in but disabled by default, in which case you need psi=1 on the kernel command line. On cgroup v2 systems you also get per-cgroup pressure, which is how you narrow from “the host is stalling” to “this container is stalling” in one command:

# Pressure for one systemd slice, useful on any modern distro
cat /sys/fs/cgroup/system.slice/io.pressure

# Pressure for every leaf cgroup that has it, sorted by nothing,
# just eyeball the full lines
find /sys/fs/cgroup -name io.pressure -exec sh -c 
  'printf "%s: " "$1"; sed -n 2p "$1"' _ {} ;

If you run Prometheus, node_exporter has exposed these since it gained a pressure collector. The counters are node_pressure_io_waiting_seconds_total for some and node_pressure_io_stalled_seconds_total for full. Rate those and you have a far better storage alert than any threshold on %iowait will ever be. Grafana Cloud and Netdata both chart PSI out of the box if you’d rather not build the panels yourself.

Failure family one: the device really is saturated

This is the case everyone assumes, so get it confirmed or eliminated quickly.

iostat -xy 1

The -x gives extended per-device stats. The -y matters more than people realise: without it, the first report is a since-boot average, and reading that first block is the single most common iostat mistake there is. It will tell you the disk has been fine all year, which is true and useless.

Now the columns worth your attention:

• r_await and w_await: average milliseconds a read or write took, queue time included. This is the closest thing to “how slow does storage feel”. Split reads from writes, because they fail very differently.
• aqu-sz: average queue depth. Requests either queued or in service. A depth that climbs while throughput stays flat is the signature of a genuinely saturated device.
• r/s, w/s, rkB/s, wkB/s: the actual workload. Compare against what the device or volume class is rated for.
• %util: treat with suspicion, see below.

The %util trap. That column counts the fraction of time at least one request was in flight. On a spinning disk that serviced one request at a time, it was a fair saturation signal. On NVMe and on cloud volumes that handle many requests in parallel, a device can sit at 100 percent while running at a fraction of its real capacity. Provisioning a bigger volume off the back of a 99 percent %util reading is a common and expensive way to change nothing. Use aqu-sz and the await columns instead. Similarly, svctm was deprecated and dropped from newer sysstat output because it was unreliable on parallel devices; ignore it if an old build still prints it.

Once you know the device is busy, find the culprit. pidstat -d 1 gives per-process read and write rates from the same sysstat package and needs nothing special enabled. iotop -o is friendlier, but on kernels from 5.14 onward per-task delay accounting is off by default, so the SWAPIN and IO> columns come back empty until you turn it on:

sudo sysctl kernel.task_delayacct=1
sudo iotop -oPa
# turn it back off when you're done, it costs a little scheduler overhead
sudo sysctl kernel.task_delayacct=0

And check the boring things before the clever ones. dmesg -T | grep -iE 'i/o error|reset|medium' catches a failing drive or a flapping controller. smartctl -a /dev/sda catches reallocated sectors on hardware you own. If either of those turns something up, stop reading and replace the disk.

Failure family two: memory pressure in an I/O costume

This is the failure family I’d check second, and it’s invisible if you only look at storage.

When RAM gets tight, the kernel starts evicting page cache and swapping. Both generate disk traffic. The disk gets busy, iowait climbs, and every graph points at storage while the actual constraint is memory. Give the box more RAM and the “disk problem” disappears.

# si and so are swap-in and swap-out in KB/s.
# Sustained non-zero values are your answer.
vmstat 1

# Are we thrashing writeback? Dirty pages waiting to be written.
grep -E 'Dirty|Writeback' /proc/meminfo

# The direct question
cat /proc/pressure/memory

If memory pressure and I/O pressure rise together, fix memory first. Everything else is downstream.

There’s a related trap on the write side. The kernel buffers writes in page cache and flushes them in the background, governed by vm.dirty_background_ratio and vm.dirty_ratio. Cross the background threshold and flusher threads start writing. Cross the hard vm.dirty_ratio threshold and writing processes are forced to do writeback themselves, synchronously. That transition is brutal, and on a box with plenty of RAM and a modest disk the defaults let you accumulate gigabytes of dirty pages and then pay for all of them at once. Lowering the ratios makes writeback more frequent and less spiky, trading away a little write batching. Measure before you commit to it.

Failure family three: small synchronous writes

Throughput can be trivial and the system still crawls. A workload doing thousands of tiny fsync calls a second is bound by round-trip latency, not bandwidth, and no amount of extra megabytes per second will help.

Databases are the usual source, because durability has a cost and that cost is a flush. MySQL and MariaDB expose it through innodb_flush_log_at_trx_commit; PostgreSQL through synchronous_commit. Loosening either buys real throughput and trades away durability on power loss or crash. That can be a perfectly reasonable trade for a cache or an analytics replica, and a terrible one for anything holding orders or payments. Decide it explicitly, write down what you decided, and don’t let it be a thing somebody changed once at 2am.

The other common source is logging. Application logs, access logs, container JSON logs and a database sharing one volume will starve each other. Splitting hot write paths onto separate devices is unglamorous and works.

Failure family four: throttled, not saturated

A throttled device and a saturated device look almost identical from inside the guest: high await, growing queue, miserable latency. The difference is that no amount of tuning inside the box will fix a throttle. Three flavours are worth knowing.

• Cloud volume limits. Managed block storage gives you a baseline of IOPS and throughput, sometimes with a burst allowance on top. Burn the burst and you drop to baseline, and everything that was fine in testing falls over in production. The tell is a latency cliff that arrives at a consistent point into a workload rather than tracking load smoothly. On AWS, watch the volume-level CloudWatch metrics such as BurstBalance on burstable volume types and VolumeQueueLength; other providers publish equivalents. The fix is a volume class change, not a sysctl.
• cgroup limits you set yourself. On cgroup v2, io.max caps a cgroup at a given bytes-per-second or IOPS figure, and io.weight shares bandwidth proportionally under contention. A limit that made sense when it was written can become the bottleneck a year later, and nothing in the container’s own metrics announces it.
• Neighbours on shared hardware. On oversubscribed VPS plans your storage is shared, and someone else’s backup job is your latency spike. This is where cheap and expensive hosting genuinely differ. Budget providers like Contabo and InterServer are fine value for a lot of workloads, but if storage latency is load-bearing for you, look for plans with dedicated NVMe rather than shared pools, and test before you migrate anything that matters.

# Per-device I/O accounting for a cgroup, including any throttling
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.stat

# Is a cap set? An empty file means no limit on that cgroup.
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.max

One more signal while you’re here. In top, the st column is steal time, meaning the hypervisor gave your vCPU’s slice to someone else. Steal and iowait rising together on a shared host is a strong hint that the problem is the neighbourhood, not your configuration.

Failure family five: the latency isn’t in the block layer

Here’s the failure mode that wastes the most time, because every storage metric you check will look healthy.

iostat measures the block layer. It cannot see anything above it. So if your latency is in NFS, in a network filesystem, in an overlay or FUSE mount, in filesystem lock contention, or in a journal that’s serialising writes, the device will report sub-millisecond service times while your application waits tens of milliseconds per operation.

The diagnosis is the gap. Measure latency at the application’s layer and at the block layer, and if the two numbers disagree wildly, the problem lives in between.

# Block layer latency as a histogram. Look for a bimodal shape
# or a long tail rather than the average.
sudo biolatency-bpfcc -m 10 1      # Debian/Ubuntu naming
sudo /usr/share/bcc/tools/biolatency -m 10 1   # RHEL family

# Filesystem operations slower than 10ms, with the process and file
sudo ext4slower-bpfcc 10
sudo xfsslower-bpfcc 10

# Everything at the VFS layer, higher overhead, use briefly
sudo fileslower-bpfcc 10

If ext4slower is full of hundred-millisecond operations and biolatency shows everything completing in microseconds, you have your answer and it is not the disk. For NFS specifically, nfsiostat splits out RTT from execution time, which separates network cost from server cost. Install the tooling with bpfcc-tools on Debian and Ubuntu, or bcc-tools on RHEL, AlmaLinux and Rocky.

A working order for troubleshooting high I/O wait

On an unfamiliar box, this is the sequence I work through. Each step either finds the problem or eliminates a family.

1. Read /proc/pressure/io. If full is near zero, the stall is not hurting the whole system and you should widen the search before you tune storage.
2. Check /proc/pressure/memory at the same time. If it’s rising too, fix memory first.
3. Run iostat -xy 1 for thirty seconds. Note r_await, w_await and aqu-sz per device. Ignore %util.
4. List processes stuck in uninterruptible sleep: ps -eo state,pid,comm | awk '$1 ~ /^D/'. That’s your blocked set, and it names the workload.
5. Attribute the traffic with pidstat -d 1. Confirm the process you suspect is the process actually doing the I/O.
6. If await is high, decide saturated versus throttled. Check cgroup io.max, check provider volume metrics, check steal time.
7. If await is low but the application is slow, jump to the layer gap. Run biolatency and ext4slower together and compare.
8. Only now consider tuning: dirty ratios, I/O scheduler, flush settings, splitting volumes.

Common mistakes

• Reading iostat’s first report. It’s a since-boot average. Use -y, or discard the first block.
• Trusting %util on NVMe or cloud volumes. Parallel devices break the assumption it was built on.
• Alerting on a fixed %iowait threshold. It fires on healthy backups and stays silent during real stalls on busy hosts.
• Changing the I/O scheduler first. Going from mq-deadline to none on NVMe is sometimes right, but it’s a tuning step, not a diagnosis, and it fixes nothing if you’re throttled.
• Adding swap to fix swapping. More swap on a memory-starved box means more disk traffic, not less.
• Stopping at the block layer. If every storage metric looks clean and the app is still slow, you’re measuring at the wrong layer, not measuring a healthy system.
• Testing with the wrong benchmark. A sequential dd tells you nothing about a workload doing random 4K synchronous writes. Use fio and match the real access pattern.

Best practices worth the effort

• Record PSI continuously. Storage incidents are short and bursty. Without history you’re guessing about something that already ended.
• Capture a baseline while things are healthy. Save an iostat -xy sample and a PSI reading from a normal day. “Is 8ms bad?” is unanswerable without one.
• Keep sar running. The sysstat package can archive metrics on a schedule, which means sar -d can show you what happened at 3am. Enable it before you need it.
• Separate hot write paths. Database data, database logs and application logs on separate volumes stops one from starving the others.
• Cap container logs. An uncapped JSON log driver quietly filling the root volume is a recurring cause of host-wide stalls.
• Alert on latency and pressure, not utilisation. Users feel w_await and PSI full. Nobody has ever felt a percentage of device busy time.

Frequently asked questions

What is a normal or acceptable I/O wait percentage?

There isn’t one, and any article that gives you a single number is guessing. Iowait scales with how idle your CPUs are, so the same workload produces very different figures on a busy server and a quiet one. Judge against your own baseline, and prefer PSI, which measures lost time directly rather than as a share of idle.

Can high I/O wait be caused by something other than the disk?

Regularly. Memory pressure driving swap and page cache eviction is the most common non-disk cause. Network filesystems, hypervisor contention on shared hosts, and cgroup or cloud volume throttling all produce the same symptom. The block device being genuinely slow is one possibility among several.

Why does iostat show 100% utilisation when the disk isn’t busy?

Because %util measures the fraction of time at least one request was in flight, which made sense for devices that handled one request at a time. NVMe drives and cloud volumes process many requests in parallel, so they can be continuously occupied while nowhere near capacity. Read aqu-sz and the await columns instead.

How do I find which process is causing I/O wait?

Start with pidstat -d 1, which needs no special kernel settings. List processes in uninterruptible sleep with ps -eo state,pid,comm and filter for state D to see who is actually blocked. iotop -o gives a live view but needs kernel.task_delayacct=1 on recent kernels for its percentage columns to populate.

Does adding more RAM reduce I/O wait?

Often, yes, and it’s underrated. More RAM means a larger page cache, fewer reads reaching the device, and less swapping. If vmstat shows sustained swap activity or memory pressure tracks I/O pressure, RAM is likely the cheapest fix available.

Should I change the I/O scheduler to fix high I/O wait?

Only after you know the device is the bottleneck. Check the current setting with cat /sys/block/sda/queue/scheduler. On NVMe, none is a common and sensible default because the hardware already does the queuing. On spinning disks, mq-deadline or bfq give better latency behaviour. It’s a tuning knob, not a diagnostic, and it will not help if you’re being throttled.

Why is my application slow when iostat says storage is fine?

Because iostat only sees the block layer. Latency in NFS, FUSE, overlay filesystems, journal serialisation or filesystem locking is entirely invisible to it. Run ext4slower or xfsslower alongside biolatency: if the filesystem is slow and the device is fast, the gap between them is your problem.

The one thing worth remembering

High I/O wait is a symptom that points in a general direction, not a diagnosis. It’s a slice of idle time, averaged across cores, measured at one specific layer of the stack. It goes up when nothing is wrong and stays flat when something is very wrong.

Read PSI to find out whether anyone is actually suffering. Read r_await, w_await and aqu-sz to find out whether the device is the constraint. Compare filesystem latency against block latency to find out whether you’re even looking at the right layer. Do those three things and the answer usually falls out in a few minutes, instead of after a week of arguing with your hosting provider about a volume that was fine all along.

Need someone to look at your storage stalls?

I work with teams on exactly this kind of problem, usually the ones that have already survived a round of “the host says the disk is fine”. Things I can help with:

• Diagnosing a live or recurring I/O stall and identifying which failure family it belongs to
• Setting up PSI, sar and per-cgroup I/O metrics in Prometheus and Grafana, with alerts that fire on latency rather than utilisation
• Separating cloud volume throttling from genuine device saturation, and sizing storage correctly instead of guessing upward
• Tuning writeback, dirty ratios and I/O schedulers for a specific workload, with before-and-after measurements
• Tracing container and Kubernetes I/O contention through cgroup v2 limits, overlay filesystems and log drivers
• Benchmarking with fio against your real access pattern so capacity decisions rest on something other than a dd run

If you’ve got an iostat capture, a PSI reading or just a graph that doesn’t make sense, send it over and I’ll tell you what I see in it.

The post Troubleshooting High I/O Wait Without Blaming the Disk appeared first on John Nessime.

The site never went down. The uptime monitor stayed green all weekend. Analytics looked normal. And a few days of enquiries went somewhere nobody thought to look.

That’s the shape of most migration damage. Not the 502. A 502 gets fixed inside ten minutes because everyone sees it at once, including the client. The expensive failures are the ones with no symptom on the front end: mail that authenticates on the old IP and not the new one, a cron job that never came across, a certificate that is valid today and will fail to renew long after everyone has stopped paying attention, orders written to a database that is about to be deleted.

This post-migration checklist covers the first 48 hours after cutover. It’s ordered by when each failure actually becomes visible, not by category, because that ordering is what decides which problems you catch yourself and which ones reach you as a client ticket a fortnight later.

Why a post-migration checklist should be ordered by detection latency

Most migration checklists you’ll find are grouped by subsystem: DNS, SSL, email, SEO, performance. That’s a fine way to write a document and a bad way to work a cutover, because it treats a 500 error on the homepage and a broken certificate renewal as equally urgent. They aren’t. One will be reported to you within minutes by someone who isn’t you. The other has a detection latency measured in weeks, and it lands at 3am.

So sort your checks by how long the failure can hide. Short latency, catch it reactively. Long latency, go and look for it on purpose, while you still have the old server, the old data and a rollback path. That’s what makes 48 hours the right frame: long enough for a full daily cron cycle and for DNS to settle at any sane TTL, short enough that the old server is still sitting there with the evidence on it.

Hour 0 to 1: prove you are actually looking at the new server

Almost everyone starts by loading the site in a browser and saying it looks fine. That check is worth close to nothing an hour after cutover. Your laptop may still hold the old answer, your ISP resolver definitely might, and if you added a hosts entry during testing you’re now testing the thing you already tested. Ask several resolvers directly, then bypass DNS entirely and talk to the new IP.

# What is each public resolver answering, and how long will it keep answering it?
dig example.com A @1.1.1.1
dig example.com A @8.8.8.8
dig example.com A @9.9.9.9

# Talk to the new server directly, ignoring DNS completely
curl -sI --resolve example.com:443:203.0.113.10 https://example.com/

# Which certificate is that IP presenting for that hostname?
openssl s_client -connect 203.0.113.10:443 -servername example.com </dev/null 2>/dev/null 
  | openssl x509 -noout -subject -issuer -dates

The full dig answer section, not +short, matters here because it shows the remaining TTL. That number tells you how much longer stale answers can survive, which is the single most useful figure you have in the first hour.

The --resolve flag pins a hostname to an address for that one request, so you get the new server’s response with the correct SNI and Host header. It’s the closest thing to a definitive answer you’ll get while records are still in flight. If you want to see what the rest of the world sees, an exit in another country helps, and a VPN account with NordVPN or Surfshark is a cheaper way to get one than spinning up a box in a second region.

Things to confirm in this window:

• The response comes from the new server, verified by something unique to it rather than by the page looking right. A temporary header or a file at a path that only exists on the new box removes all doubt. Pass test: you can name the specific artefact that proves it.
• The correct virtual host is selected, not a catch-all default. Sites with several domains on one server fail here quietly, serving the wrong document root over a valid certificate. Pass test: every hostname you migrated returns its own content, checked individually.
• The certificate chain is complete on the new box, not just the leaf. Browsers often paper over a missing intermediate using cached data; API clients and payment callbacks do not. Pass test: an external SSL checker reports a complete chain from a machine that has never visited the site.
• Redirect behaviour matches the old server exactly. HTTP to HTTPS, www to apex or the reverse, trailing slashes. Pass test: a handful of known deep URLs return the same final status and location as before.
• Any hosts file entries you added for testing are gone. Leave one in place and you’ll misread every check you run for the next two days.

Hour 1 to 6: the split-brain window, where you lose data instead of uptime

This is the part that costs real money, and it’s the part almost nobody checks, because during it both servers are healthy. Some resolvers still hand out the old address. Those visitors reach the old application, which happily accepts their form submission, their order, their uploaded file, their account signup. Then you delete that server.

“Keep the old server running for 48 hours” is standard advice and it’s necessary, but on its own it makes the problem worse rather than better. A running old server that still accepts writes is a data-loss machine. What you want is an old server that stays reachable and stops accepting writes.

Three options, in order of how much I’d trust them:

1. Proxy the old server to the new one. Every request that lands on the old address gets forwarded to the new backend, so there is only ever one database. This is the only option that genuinely eliminates the split, and it’s worth the extra config on anything transactional.
2. Redirect the old server to a hostname that only resolves to the new IP. Cheap, works for GET traffic, does nothing useful for a POST that’s already in flight.
3. Put the old application into a read-only or maintenance state. Safe, but visitors on stale DNS see a broken site for the length of your TTL, so this only works if that number is genuinely small.

Whichever you pick, go and look at what the old box actually did after the cutover.

# On the OLD server: has anything been written since cutover?
grep '"POST ' /var/log/nginx/access.log | tail -n 50

# Apache layouts vary; on DirectAdmin the per-domain log lives here
grep '"POST ' /var/log/httpd/domains/example.com.log | tail -n 50

# Is the old box still receiving mail for the domain?
tail -n 200 /var/log/maillog

Every POST in that log after the cutover timestamp is a potential lost record. Take the list to the application: compare row counts or highest IDs on the write-heavy tables between old and new, and reconcile before you decommission anything.

Mail deserves its own pass. If MX records changed, messages go to whichever server the sending system had cached, and land in two different mailboxes. Keep both able to accept mail through the overlap, and check the old mailbox before you close the account, not after.

Two more things here that won’t show up in any log you’re watching:

• Anything pinned to an IP address. Payment gateway allowlists, webhook endpoints, database firewall rules, SMTP relay authorisations, third-party API allowlists. None of these follow a redirect. A webhook posted to the old IP returns a perfectly good 200 and the new application never hears about the payment.
• Outbound connections from the new IP. Your application is now calling other people’s APIs from an address they’ve never seen. If any of those partners allowlist by IP, the failure appears as a timeout in a background job rather than an error on a page.

Hour 6 to 24: everything that runs on a schedule

Files and databases get migrated. Scheduled work frequently does not, because it lives outside the document root and outside the database dump, and because nothing on the site breaks when it’s missing.

# Every user's crontab, not just root's
for u in $(cut -d: -f1 /etc/passwd); do echo "== $u"; crontab -l -u "$u" 2>/dev/null; done

# systemd timers, where a lot of "cron" actually lives now
systemctl list-timers --all

# WordPress: what is scheduled, and is anything overdue?
wp cron event list --fields=hook,next_run_relative,recurrence
wp config get DISABLE_WP_CRON

Run that on both servers and diff the output. That comparison is the whole check, and it takes two minutes.

The specific traps worth knowing:

• The old server’s cron is still running. Nobody stops it, because “keep it up for 48 hours” sounded like keep everything up. Now your nightly export runs twice, your reminder emails send twice, and your billing job runs against a database nobody is reading. Disable cron on the old box at cutover, before you disable anything else.
• WP-Cron was disabled in config and the replacement never came across. If DISABLE_WP_CRON is true and no system cron hits wp-cron.php, every scheduled task stops silently. Scheduled posts sit at “Missed schedule”, backups stop, WooCommerce action queues back up. The site itself works perfectly.
• The cron URL is wrong in a way that still returns success. A job calling an http:// URL that now redirects to HTTPS gets a 3xx, exits zero, and looks healthy in the log while the task never executes. Same story for a cron still pointing at the old hostname or an old absolute path.
• Backups. The job may have moved and still be writing to storage credentials, a mount, or a destination path that only existed on the old host. Don’t check that the backup job ran. Check that a file arrived at the destination, and that it’s the size you expect.
• Log rotation. A fresh server often has no rotation configured for application logs. Nothing goes wrong for three days, then the disk fills at 2am and every write fails at once.

Hour 24 to 48: the delayed detonations

These are the checks with the longest detection latency, which makes them the ones most worth doing deliberately.

Certificate renewal, not the certificate

A copied certificate is valid for weeks. What breaks in a migration is the machinery that replaces it: the webroot path in the renewal config no longer exists, the ACME account keys weren’t copied, port 80 is closed on the new firewall, or a WAF is now intercepting the challenge path. None of that surfaces until the renewal window opens, long after everyone has moved on.

# What does certbot think it manages, and how does it validate?
certbot certificates

# The only check that matters: will the NEXT renewal succeed?
certbot renew --dry-run

# Is anything actually going to run it?
systemctl list-timers | grep -i certbot

The dry run validates against Let’s Encrypt’s staging environment, so it exercises the real challenge path without consuming your issuance quota. Resist the urge to fix a failure with a forced renewal: Let’s Encrypt applies rate limits per registered domain and a separate limit on identical certificates, and burning through them during a migration is how a fixable config problem turns into a week of waiting.

Mail identity on an address with no history

If the new server sends mail, it’s sending from an IP that receiving systems have never seen. Authentication correctness and reputation are two separate problems and they fail differently. Authentication failures bounce or land in spam immediately. Reputation problems look like intermittent delivery to one provider and normal delivery to another.

# Reverse DNS for the new sending IP
dig +short -x 203.0.113.10

# Does SPF still authorise everything that sends for you?
dig +short TXT example.com

# DKIM selector (replace 'selector') and DMARC policy
dig +short TXT selector._domainkey.example.com
dig +short TXT _dmarc.example.com

Send a test message to an address at a major provider and read the full headers. The Authentication-Results header tells you whether SPF, DKIM and DMARC each passed, and crucially whether the passing domain aligns with the visible From address. SPF can pass for the envelope sender and still fail DMARC when the From domain differs, which is a failure mode you cannot see from the sending side.

Leave the old server’s SPF authorisation and DKIM public key published through the overlap. Mail that was queued or delayed on the old path still needs to verify. Sources differ on how long to keep them, and the honest answer is: until you have confirmed nothing is still sending or retrying from the old host.

If transactional mail matters to the business, the migration is a reasonable moment to stop sending it from a web server at all. Handing it to a dedicated provider such as Postmark, Mailgun or Amazon SES moves reputation off an address you’ve just changed and gives you delivery logs you can actually query.

Search crawl, redirects and staging leftovers

Crawlers hit harder than usual after a move, because they’re re-checking old URLs and discovering new ones at the same time. That surge is also your best early warning: it surfaces broken redirects faster than your own testing will.

• Fetch robots.txt from the live site and read it. A staging Disallow: / that survives cutover is the single most damaging thing on this list.
• View source on several page types and check for a leftover noindex tag and for canonical tags pointing at a staging hostname.
• Watch 404s in the access log rather than in a crawler report. It’s your missed-redirects list, sorted by how much anyone cares.
• Leave the old URLs crawlable through the redirect. Blocking them means the redirects are never seen.

Resource behaviour under real traffic

Staging never told you what the process pool does at peak. Twenty-four hours of production traffic does.

# Status code distribution since cutover
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

# Disk, and the one people forget
df -h
df -i

Compare the ratio of 5xx to total requests against the old server’s logs for the same weekday. Absolute numbers mislead here; the ratio doesn’t. Watch memory across a full daily cycle rather than at a single moment, because the interesting shape is a slow climb into a limit, not a spike.

Before you decommission the old server

The old server is the only copy of some things, and none of them are in your backup.

• Access and error logs covering the cutover window, which is your evidence for anything that surfaces next month.
• Mail spools, and any messages that arrived after the MX change.
• The full crontab set, exported to a file rather than a screenshot.
• Web server, PHP and database configuration, so you can diff a behavioural difference you notice later.
• A final database dump taken after write traffic stopped.

Keep the TTL low until you’ve decided you’re staying. Raising it back to normal is the last step of the migration, not the first thing you do once the site loads. Another month on an InterServer or Contabo VPS is cheap next to reconstructing a week of orders, and the overlap is the only period in which rolling back is still a five-minute operation.

Troubleshooting: three symptoms and what they usually mean

“It works for me but not for them.” Almost always a resolution split. Get the affected user to report what their resolver returns, then check whether the answer is stale DNS, a corporate resolver ignoring your TTL, or a CDN edge still holding the old origin. If you use Cloudflare or a similar proxy in front, remember the origin change and the DNS change are two separate things and only one of them propagates.

“The site is fine but our mail goes to spam now.” Work in this order: reverse DNS for the sending IP, then SPF including the new address, then DKIM signing on the new host, then DMARC alignment with the visible From domain. Only after all four pass should you start thinking about reputation and volume, and reputation on a new IP is a matter of ramping gradually rather than something you can configure.

“Random 500s that never happened in staging.” Look at limits before you look at code. Process pool size, memory limits, database connection caps, open file limits, and file ownership on directories the application writes to. Permission problems after an rsync or archive restore are common and produce errors that look nothing like permission problems from the front end.

Common mistakes in the first 48 hours

• Treating a green uptime check as verification. It confirms one URL returns 200. It says nothing about mail, cron, writes or checkout.
• Leaving the old server fully operational, cron and all, instead of leaving it reachable but inert.
• Testing while a hosts file entry is still in place.
• Raising the TTL back to normal on day one, which throws away your fast rollback.
• Checking that the certificate is valid instead of checking that renewal works.
• Silencing monitoring for the maintenance window and forgetting to unsilence it.
• Declaring the migration done before a full daily cron cycle has completed on the new server.

If you only do five things

1. Stop writes and cron on the old server, but keep it reachable.
2. Run certbot renew --dry-run and confirm a timer or cron will actually invoke it.
3. Send a real test email to a major provider and read the Authentication-Results header.
4. Diff the scheduled jobs on both servers, then confirm one real backup file landed at its destination.
5. Grep the old server’s access log for POST requests after the cutover, and reconcile anything you find.

FAQ

How long should I keep the old server running after a migration?

Long enough that no resolver anywhere is still handing out the old address, plus enough margin to reconcile data and pull logs. For most sites that’s a few days. The cost of the extra billing period is trivial compared with losing your rollback path, but keep it inert: reachable, not writable, with cron disabled.

Why is my site fine but email broken after a server migration?

Because web and mail are validated by different systems using different records. The web path only needs an A record and a certificate. Mail needs the MX record, reverse DNS on the sending IP, SPF listing that IP, DKIM signing configured on the new host, and DMARC alignment with your From domain. A migration can satisfy the first set completely and break every item in the second.

What should I check first if something looks wrong right after cutover?

Establish which server you are talking to before you diagnose anything else. Use curl --resolve to pin the hostname to the new IP and compare that response with what you get through normal DNS. Half of post-cutover panic is people debugging the old server.

Can I lose form submissions or orders during DNS propagation?

Yes, and it’s the most common serious loss in a migration. Visitors on cached DNS reach the old application, which writes to the old database. Nothing errors. Proxying the old server to the new backend removes the problem entirely; short of that, freeze writes during the overlap and check the old access log for POST requests before decommissioning.

Why did my SSL certificate stop renewing weeks after the migration?

The certificate itself migrated fine; the renewal configuration didn’t. Common causes are a webroot path that no longer exists, an ACME account directory that wasn’t copied, port 80 blocked by the new firewall, a proxy or WAF intercepting the challenge path, or no timer scheduled to run the renewal at all. A dry run within 48 hours catches all of these.

Do I need to tell search engines about a server migration?

If the URLs are unchanged and only the hosting moved, no announcement is required. Your job is to make sure crawlers can reach the new server, that nothing carries a staging noindex or a blocking robots rule, and that canonical tags point at live URLs. If the domain or URL structure changed, that’s a different migration with its own redirect and site-move process.

What monitoring should I add specifically for the 48-hour window?

Add checks for the things a homepage ping cannot see: a synthetic transaction through a real form or checkout, a heartbeat from each scheduled job, certificate expiry, and a content check that asserts the right text is present rather than just a 200 status. Most uptime platforms handle the first and last; heartbeat monitoring for cron is the one people skip and the one that catches the silent failures.

The one thing worth remembering

A migration doesn’t fail when the site goes down. It fails when the site stays up and something behind it stops, quietly, in a way nobody is looking for. Every item on this post-migration checklist exists because the failure it catches produces no visible symptom on the front end.

So work the 48 hours in order of detection latency. Prove which server you’re on, close the write path on the old one, diff the scheduled jobs, dry-run the certificate renewal, and read the headers on a real test email. Then, and only then, raise the TTL and let the old box go.

Need a second pair of eyes on a cutover?

Migrations are one of the few pieces of infrastructure work where the review matters more than the execution. Things I help with:

• Planning the cutover so the write path is closed on the old server rather than left open, including proxy-forward setups for transactional sites.
• Running the 48-hour verification pass on a migration someone else performed, and telling you plainly what’s still broken.
• Repairing certificate renewal, ACME challenge paths and reload hooks that stopped working after the move.
• Fixing mail on a new IP: reverse DNS, SPF, DKIM signing and DMARC alignment, or moving transactional mail off the web server entirely.
• Rebuilding scheduled jobs, backup destinations and log rotation on the new host, with heartbeat monitoring so silence becomes an alert.
• Reconciling data written to the old server during the split window, before anything gets deleted.

If you’re mid-migration or just came out of one, send me something concrete: the old server’s access log around the cutover, your crontab, or the headers from a test email. That’s usually enough to tell you what’s wrong.

The post Post-Migration Checklist: What Breaks Quietly in the First 48 Hours appeared first on John Nessime.

That is the shape of most trouble with SAP data integration with AWS Glue. The connector does not usually crash. It returns an empty or partial result set, the job exits zero, and your data lake quietly diverges from the source system. By the time anyone notices, you have no idea which run was the last correct one.

This post covers the parts that bite after the proof of concept: how delta tokens actually behave, why deleted records keep showing up in your reports, where partitioning stops working, and the licensing distinction between two things that both get called “ODP” and are not the same thing at all. There is a troubleshooting section at the end for when a job is already broken.

First, get the ODP terminology right, because licensing depends on it

Operational Data Provisioning is SAP’s framework for exposing extraction interfaces in a delta-aware way. A provider (a BW extractor, an ABAP CDS view, SLT, a HANA information view) writes changes into an Operational Delta Queue, and subscribers pull from that queue. This part is uncontroversial.

What matters is that there are two ways to talk to ODP, and SAP treats them very differently. SAP Note 3255746 restricts third-party and customer applications from using the RFC modules of the ODP Data Replication API against ABAP sources. That note has been sharpened over several revisions, from “unsupported” to “not permitted”, and SAP has stated it reserves the right to introduce technical measures and audit unpermitted use. A number of established extraction tools were built directly on ODP-RFC and have had to move.

The AWS Glue SAP OData connector takes the other path. It talks OData over HTTPS to an activated SAP Gateway service, and uses the ODP framework through that OData layer rather than through RFC. That is the route SAP points customers at. If you are being asked in an architecture review whether your pipeline is on the wrong side of Note 3255746, this distinction is the whole answer, and it is worth having the note number written down before the meeting.

Two honest caveats. First, notes get revised, so read the current version rather than a vendor blog summarising it, including this one. Second, “permitted” is not the same as “free”: you still need the OData service activated, a service user with the right authorisations, and a Basis team willing to own the queue growth on their side.

Failure family one: the delta token is shared state you do not own

This is the one that produces silent drift, and it is the reason the finance report was short.

When you read an ODP-enabled entity with change data capture switched on, the connector asks SAP to initialise a delta queue subscription. SAP hands back a delta token. That token is your bookmark. On the next run you pass it back, and you get only what changed since.

sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "ENABLE_CDC": "true"
    },
    transformation_ctx="initial_load"
)

ENABLE_CDC defaults to false. Leave it off and you get a full load every single run, which is expensive but at least obvious. Turn it on and the token appears in the DELTA_TOKEN column, populated only on the last row of the frame. That last row is not a business record. It is a carrier for the token, and if you write your frame straight to S3 without stripping it you have just added a junk row to your table.

The subsequent run passes the token back:

sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "DELTA_TOKEN": last_known_token
    },
    transformation_ctx="incremental_load"
)

Why this goes wrong

A delta token is bound to a service collection and a user. If a new initialisation runs with ENABLE_CDC set true for that same collection and user, SAP invalidates every token previously issued from the earlier initialisation. Not “expires eventually”. Invalidates, immediately.

So picture two Glue jobs sharing one service account, both reading the same extractor. Someone redeploys job B with the CDC flag still set from testing. Job B re-initialises. Job A’s stored token is now dead, and the next time job A runs you get:

Could not open data access via extraction API RODPS_REPL_ODP_OPEN

An error is the good outcome. The bad outcome is code that catches the exception, falls back to a full load, and writes it over the top of an incremental table with no dedupe. Or code that re-initialises on failure, which fixes job A and breaks job B, and now the two jobs take turns invalidating each other forever while both keep exiting zero.

Practical rules that come out of this:

• One SAP service user per consuming pipeline. Sharing a user is what couples independent jobs together.
• Persist the token somewhere durable and versioned. DynamoDB, Parameter Store, S3 with versioning enabled, all fine. Job bookmarks are not a substitute, because the token is source state, not read position.
• Never auto re-initialise on error. Fail loudly, page someone, and make the reinitialisation a deliberate human action with a documented backfill.
• Alert on zero-change runs. A pipeline that returns no changes for several consecutive runs against a busy table is telling you something, and it is rarely “business is quiet”.

Retention on the SAP side is a hard deadline

The delta queue is not infinite. In transaction ODQMON the Basis team configures retention, and a reorganisation job cleans up data past that window. If your pipeline is down longer than the retention period, the changes you missed are gone from the queue and no token will bring them back. You are into a full reload and a reconciliation.

Two things to find out before you go live: what the retention setting is, and whether the cleanup job is actually scheduled. Then make sure your pipeline’s alerting escalates well inside that window. A weekend outage on a queue with a short retention is a very expensive Monday.

Related: if anyone has been testing extraction with RODPS_REPL_TEST, that program registers as a real additional subscriber rather than simulating one. Because the queue is shared and data is only cleaned once every subscriber has collected it, forgotten test subscriptions hold delta data indefinitely and the underlying queue tables grow. Ask whoever ran the tests to reset those subscriptions in ODQMON.

Failure family two: deletes arrive as a column, not as a deletion

CDC extraction from ODP entities returns a couple of extra fields alongside your business columns. DML_STATUS carries UPDATED for inserts and updates, and DELETED for records removed at source. GLUE_FETCH_SQ is a per-record sequence derived from an epoch timestamp, in receipt order, and it is your tie-breaker when the same key changes twice inside one delta.

Nothing acts on those fields for you. If your job appends the frame to S3 and calls it done, a deleted sales order is now a row in your lake with DML_STATUS = DELETED and every other column populated exactly as before. Any query that does not filter on that column counts it. Cancelled orders keep appearing in revenue. Closed accounts stay open. The pipeline is green, the numbers are wrong, and the discrepancy grows slowly enough that nobody catches it early.

You have to apply the changes rather than accumulate them. Broadly two options:

1. Merge into a table format that supports it. Apache Iceberg, which Glue supports natively, gives you a real merge with delete handling. Land the raw delta, then merge on the business key with GLUE_FETCH_SQ deciding the winner within a batch, deleting where the status says deleted.
2. Keep an append-only history and build a current-state view. Cheaper to write, and it preserves an audit trail. It also means every downstream consumer has to use the view, and someone eventually will not.

I reach for Iceberg first on anything that feeds reporting, because the correctness lives in the table rather than in the discipline of whoever writes the next query. Append-only is the right call when you genuinely need history, or when the consumer is a downstream warehouse like Amazon Redshift that will do its own merge anyway.

Worth knowing: not every ODP provider emits deletes reliably. Some older extractors express them as reversal records rather than a delete flag. Confirm the behaviour for your specific entity with the functional owner before you build merge logic on an assumption.

Failure family three: pagination and partitioning do not compose

The connector’s default page size is 50,000 records, and you can raise it up to 500,000. Anything above that is ignored and the maximum is used instead. Raising the page size cuts the number of round trips, which is usually the dominant cost on a wide table across a slow link.

The catch is what your SAP Gateway will tolerate. A large page means a large response the ABAP stack has to assemble and hold, and Gateway or an intervening load balancer may cut it off before it completes. Symptoms are timeouts, connection resets, or a job that appears to paginate without ever finishing. Tune the page size with your Basis team, not in isolation, and treat it as a negotiation between round trips and response size rather than a value you set once.

On partitioning, there is a specific documented limit that catches people out: ODP entities are not compatible with record-based partitioning, because pagination is already handled through skip and delta tokens. For record-based partitioning, the default concurrency is forced to null regardless of what you asked for. So the obvious lever for making a slow ODP extraction faster is not available.

What you can still do on non-ODP OData entities is field-based partitioning, splitting a read across a bounded range:

• PARTITION_FIELD is the field to split on, typically a date.
• LOWER_BOUND is inclusive, UPPER_BOUND is exclusive. Getting that backwards is how you duplicate or drop a boundary day.
• NUM_PARTITIONS defaults to 1, so leaving it out means you did not partition at all.
• Where a limit and a partition are both applied, the limit wins. That is the usual cause of “I partitioned it and got fewer rows”.

For non-ODP entities you can also do timestamp-based incremental reads with filteringExpression, pointing at whatever last-changed field the entity exposes. Note the casing: that option is camelCase while most of the others are upper snake case, which trips people up when a filter appears to be ignored.

Failure family four: connection, auth and network

These fail early and loudly, which makes them the easy ones. The trap is that they fail during setup, when you have three people on a call and someone suggests loosening something to move on.

The connector supports two authentication types: CUSTOM, which is SAP basic authentication, and OAUTH2. The custom secret in AWS Secrets Manager takes this shape:

{
  "basicAuthUsername": "<your SAP username>",
  "basicAuthPassword": "<your SAP password>",
  "basicAuthDisableSSO": "<True/False>",
  "customAuthenticationType": "CustomBasicAuth"
}

For OAuth 2.0 the client secret goes into Secrets Manager under USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET, and the client ID, authorisation code URL, token URL and scopes are supplied on the connection itself.

Basic auth is genuinely easier for a scheduled pipeline: no interactive redirect, no refresh token to keep alive, and rotation is a Secrets Manager operation. OAuth gets you short-lived tokens and revocation you can actually reason about, which is what a security review will want. Both are defensible. What is not defensible is one shared service user across every pipeline, for the delta token reasons above as much as the audit ones.

A few things that reliably waste a morning:

• The service path on the connection is the catalog service path, not the path to your object. The entity path goes in ENTITY_NAME on the job.
• The OData service has to be activated in the SAP system. Until it is, nothing you do on the AWS side matters.
• For a non-VPC connection, the application host URL must be reachable from the public internet. If your SAP system sits in a private network, configure VPC connectivity properly rather than opening it up.
• The Glue role needs Secrets Manager read plus the EC2 network interface permissions when a VPC is involved. A missing ec2:CreateNetworkInterface shows up as a connection failure that looks like a network problem.
• Private certificate authorities need the chain in a form Glue accepts. If you are wiring this up yourself, expect it to take a pass or two.

Choosing your path: Glue jobs, zero-ETL, or something else

The same connector is reachable through more than one product surface, and the choice is really about how much state you want to own.

• Glue Studio or script jobs. You control page size, filters, output format, transformation and scheduling, and you own delta token persistence. Pick this when you need real transformation in flight, or when the merge logic has to live next to the extraction.
• Glue zero-ETL integrations. AWS manages token state internally, runs the initial full load and then the incremental cycle, and preserves the token on failure so retries do not lose data. Less code, less control. Worth noting that once an incremental update field is chosen for a connection and entity, a later integration on the same pair cannot pick a different one.
• Amazon AppFlow. Simpler, flow-shaped, well suited to modest volumes and quick prototypes. Less appropriate once you need Spark-scale processing.
• Commercial replication tools. Fivetran, Airbyte and SAP-specialist vendors all play here. They cost money and add a vendor, and in exchange they absorb exactly the state management this post is about. If your team has no SAP depth, that trade is often correct.

On cost, Glue bills for the resources a job consumes for as long as it runs, so the levers are the same as any Spark workload: fewer round trips, less data pulled, fewer idle executors, and not re-reading history you already have. Rates change and vary by region, so model it against current pricing rather than a figure from a blog post. If your incremental run is pulling the same volume as your initial load, the delta mechanism is not working and the bill is the symptom.

Also worth writing back. The connector supports write operations to SAP OData, using INSERT and UPDATE, with ID_FIELD_NAMES required on update to identify the external ID field. Useful for enrichment loops, and something to treat with a good deal more caution than a read.

Troubleshooting

“Could not open data access via extraction API RODPS_REPL_ODP_OPEN”

Your token is invalid or expired. Find out why before you re-initialise, because re-initialising destroys the evidence. Check ODQMON for a recent initialisation on that collection and user, check whether another job or a test program shares the credentials, and check whether the request you wanted has aged out of retention. Then re-initialise deliberately and reconcile the gap.

The job succeeds but returns no rows

Most often a stale or wrong token, a filter predicate that matches nothing, or CDC never having been initialised so the connector has nothing to hand back. Run once without ENABLE_CDC against a small filtered slice and confirm you get data at all. That separates “no changes” from “not reading”.

Pagination never terminates

Usually the SAP side enforcing its own page limit under your requested size, so the connector keeps requesting more. Reduce the page size, verify the filter is actually being applied server side rather than after retrieval, and have Basis confirm what Gateway is capping.

Row counts drift over time

Almost always unhandled DML_STATUS = DELETED records, or a token invalidation that silently turned an incremental into a partial. Count distinct business keys in your target against a direct count in SAP for a bounded date range. Make that comparison a scheduled check, not something you run when someone complains.

Duplicates after a rerun

A delta was consumed, the write failed, and the token advanced anyway. Store the token only after the write commits, never before. If you are appending rather than merging, add a dedupe on business key ordered by GLUE_FETCH_SQ.

Common mistakes

• Sharing one SAP service user across multiple pipelines, which couples their delta tokens together.
• Automatic re-initialisation in an exception handler. It converts a loud failure into permanent silent drift.
• Writing the token-carrier row into the target table as if it were data.
• Appending CDC output and ignoring DML_STATUS.
• Persisting the token before the write succeeds.
• Assuming an entity is ODP-enabled because it looks analytical. Confirm it, because the incremental mechanism differs.
• Leaving test subscriptions from RODPS_REPL_TEST registered against the queue.
• Building the pipeline without ever asking Basis what the ODQ retention window is.

Best practices for SAP data integration with AWS Glue

• Treat the delta token as production state. Versioned storage, a documented recovery procedure, and a written-down answer to “how do we backfill”.
• One service user per pipeline, one connection per environment. Isolation is the cheapest fix for the worst failure.
• Land raw, merge second. Keep the untouched connector output so you can replay a merge without going back to SAP.
• Alert on absence, not just errors. Zero rows for several consecutive runs against a busy entity should page someone. Push Glue metrics into CloudWatch and forward them wherever you already look, whether that is Grafana Cloud, Datadog or your own Prometheus.
• Reconcile on a schedule against SAP for a bounded window, and record the result. Drift you can see is a bug; drift you cannot see is an incident.
• Start with a filtered slice and a small page size while you are building. Get it correct, then tune throughput.
• Write down what each field means while the functional owner is still on the call. SAP field names do not explain themselves later.

Frequently asked questions

Does the AWS Glue SAP OData connector fall under SAP Note 3255746?

The note restricts the RFC modules of the ODP Data Replication API. The Glue connector uses OData over HTTPS against an activated Gateway service, which is the interface SAP directs customers toward. Read the current revision of the note yourself and get it confirmed with your SAP account team before you rely on any summary, including this one.

Where should I store the delta token?

Anywhere durable and versioned that your job can write to transactionally: DynamoDB, Parameter Store, or S3 with versioning. What matters more than the store is the ordering. Write it only after the target write commits, and keep the previous value so you can roll back after a bad run.

Can I run two Glue jobs against the same SAP entity?

Yes, but give them separate SAP service users. A delta token is scoped to the service collection and the user, so two jobs on one user will invalidate each other’s tokens whenever either re-initialises.

Why do deleted SAP records still appear in my data lake?

Because the connector reports them rather than applying them. Deletes come back as rows with DML_STATUS set to DELETED. If your pipeline appends without a merge, those rows sit in the target looking like live data. Merge into Iceberg, or filter through a current-state view that every consumer actually uses.

What page size should I use?

The default is 50,000 and the ceiling is 500,000. Larger pages mean fewer round trips but bigger responses for SAP to assemble, and Gateway or a load balancer may cut them short. Start at the default, raise it while watching for timeouts, and agree the ceiling with your Basis team.

Can I partition an ODP extraction for speed?

Not with record-based partitioning. ODP entities handle pagination through skip and delta tokens, so concurrency is forced to null there regardless of your input. Field-based partitioning on a bounded range is available for non-ODP OData entities. For ODP, tune page size and narrow the entity instead.

Can Glue write data back into SAP?

Yes. The SAP OData connector supports insert and update operations, with the external ID field named through ID_FIELD_NAMES on updates. Test that against a sandbox client with a functional owner watching, because a bad write into an ERP system is a very different class of incident to a bad read.

The one thing worth remembering

SAP data integration with AWS Glue is not hard because the connector is difficult to configure. Configuration takes an afternoon. It is hard because the correctness of your pipeline depends on a token held in a queue on a system your team does not administer, subject to a retention policy you may not have been told about, and shared with every other subscriber using the same credentials.

Design for that. Isolate the users, persist the token deliberately, apply deletes rather than collecting them, and alert on silence as loudly as you alert on errors. Do that and the connector is a good piece of engineering that will run for a long time without your attention. Skip it and you get a job that stays green while the numbers quietly stop being true.

Need a hand with an SAP to AWS pipeline?

This is the kind of work I take on regularly. If you are standing one up, or you have one that has started drifting, I can help with:

• Designing delta token state management that survives failed runs, redeploys and concurrent pipelines.
• Building merge logic that applies DML_STATUS deletes correctly into Iceberg or Redshift instead of accumulating ghost rows.
• Diagnosing RODPS_REPL_ODP_OPEN failures, silent zero-row runs and row count drift, and working out how far back the damage goes.
• Setting up SAP OData connections properly: Secrets Manager, least-privilege IAM, VPC connectivity and private certificate authorities.
• Tuning page size, filters and job sizing so incremental runs cost what an incremental run should cost.
• Adding reconciliation and absence alerting so drift shows up in a dashboard rather than in a finance report.

Send me a job log, a connection config, or the row counts that stopped matching, and I will tell you what I think is going on.

The post SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos appeared first on John Nessime.

That gap is the one that costs you, because it is invisible from the control plane. Kubernetes did exactly what you asked. The problem is that a readiness probe only answers one of the four questions that Kubernetes zero-downtime rollouts depend on, and it is not the question that usually bites.

This post walks through the four failure families in the order they usually show up: traffic arriving too early, traffic arriving too late, capacity disappearing faster than it is replaced, and the load balancer that was never watching your endpoints at all. Each one has a different fix, and stacking all four is what actually gets you a clean deploy.

What a zero-downtime rollout actually requires

Three independent things have to be true at the same time, and each is owned by a different part of the system.

• Capacity never dips below what traffic needs. Owned by the Deployment’s rolling update strategy.
• No pod receives traffic before it can serve it. Owned by the readiness probe.
• No pod receives traffic after it stops serving it. Owned by graceful termination, and this is the one nobody configures.

Miss any one and you get errors. Miss the third and you get errors that look like an application bug, because the pod that failed the request no longer exists by the time you go looking for it.

Failure family one: traffic arrives before the pod can serve it

This is the well-documented one. With no readiness probe, a pod is considered Ready as soon as its containers are running, which for most runtimes means “the process was forked”, not “the process can answer HTTP”. The pod lands in the Service’s EndpointSlice, kube-proxy programs it, and requests start arriving while the app is still loading config, warming a connection pool, or compiling templates.

The readiness probe that lies

Adding a probe is easy. Adding one that means something is the part people skip. Two patterns cause trouble:

A readiness endpoint that returns 200 from a static handler. If /healthz is wired up before the rest of the app is, the probe passes while the service is still useless. The readiness endpoint should be the last thing your app enables, after migrations have run, caches are primed and the pool has connected.

A readiness endpoint that checks downstream dependencies. This one feels correct and is actively dangerous. If your readiness check pings the database, then a five-second database blip marks every replica unready simultaneously. The Service loses all its endpoints, traffic has nowhere to go, and a brief degradation becomes a full outage. Readiness answers “can this pod serve traffic”, not “is the whole system healthy”. Check the dependency in your app’s request path and return a sensible error, or expose it on a separate diagnostic endpoint that nothing routes on.

Let the startup probe own the boot budget

For anything slow to boot, do not stretch initialDelaySeconds on the readiness probe until it covers the worst case. That delay applies on every restart forever, and it makes recovery slower than it needs to be. Use a startup probe instead: while it is running, liveness and readiness probes are suppressed entirely, and once it succeeds the fast probes take over.

# The startup probe owns the boot window. failureThreshold x periodSeconds
# is the total budget: 30 x 5 = 150 seconds before the kubelet gives up.
startupProbe:
  httpGet:
    path: /healthz/started
    port: http
  periodSeconds: 5
  failureThreshold: 30

# Readiness is fast and cheap. Two consecutive failures pull the pod out
# of rotation within about 6 seconds.
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: http
  periodSeconds: 3
  timeoutSeconds: 2
  failureThreshold: 2

# Liveness is deliberately slower and points at a DIFFERENT endpoint.
# This one restarts the container, so it should only fire on real deadlock.
livenessProbe:
  httpGet:
    path: /healthz/live
    port: http
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Pointing liveness and readiness at the same URL is a common and expensive mistake. Under load, a slow response fails both: readiness pulls the pod out of rotation, which is right, and liveness kills the container, which is wrong. You lose the pod entirely instead of letting it recover, and the remaining replicas absorb its traffic, get slower, and fail their own liveness checks. That is how a latency spike turns into a rolling restart of the whole Deployment.

Failure family two: traffic arrives after the pod has stopped

This is the invisible one, and in my experience it accounts for most of the leftover errors after someone has “already added readiness probes”.

When a pod is deleted, two things happen in parallel, not in sequence:

1. The kubelet begins the shutdown sequence: run the preStop hook if one exists, then send SIGTERM to the container’s main process.
2. The endpoints controller marks the pod as terminating in its EndpointSlice, and every consumer of that data has to notice and reprogram: kube-proxy on every node, your ingress controller, your service mesh sidecars, and any cloud load balancer with its own target registry.

Nothing coordinates those two tracks. Kubernetes offers no guarantee that routing has converged before your process gets SIGTERM. On a small cluster the gap might be under a second. On a busy cluster with a few hundred nodes and an ingress controller reconciling on its own schedule, it can be several seconds. Every request that lands in that window hits a socket that is closing or already closed, and your users see a 502 or a connection reset.

The preStop sleep, and what it does not do

The fix is to delay SIGTERM so the routing fabric gets a head start. A preStop hook must finish before the TERM signal is sent, so sleeping in it does exactly that.

spec:
  # Total budget for preStop + graceful shutdown. Default is 30.
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          # Native handler, run by the kubelet. No shell needed in the image,
          # which matters for distroless and scratch-based builds.
          sleep:
            seconds: 15

If you are on an older cluster without the native sleep handler, the equivalent is an exec hook, which does require a shell and a sleep binary in the image:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]

Be clear about what this buys you. The sleep does not drain anything. Your application keeps serving normally during it, unaware anything is happening. All it does is push SIGTERM later, so that by the time your process starts shutting down, nothing is sending it new work. Draining in-flight requests is still your application’s job, on receipt of SIGTERM: stop accepting new connections, finish what is open, then exit.

There is no universally correct sleep value, and anyone who gives you one is guessing. Measure it in your own cluster, on a normal weekday, and add margin.

The grace period is a shared budget

This trips people up. The terminationGracePeriodSeconds countdown starts when the pod is marked Terminating, which is before the preStop hook runs, not after. The hook and your application’s shutdown both spend from the same clock. If the grace period is 30, the hook sleeps 25, and your app needs 10 seconds to drain, the app gets SIGKILLed mid-drain and you have made things worse.

Size it as: preStop sleep + your longest realistic request + a few seconds of slack. And if your app holds long-lived connections such as WebSockets, gRPC streams or server-sent events, remember that removing an endpoint does nothing to connections that are already established. Either the grace period has to be long enough to see them out, or the client needs to reconnect on its own.

One more detail worth knowing before you go looking for it: probes accept their own terminationGracePeriodSeconds override, but only liveness and startup probes. You cannot set it on a readiness probe, because a failing readiness probe never kills anything.

Failure family three: the rollout deletes capacity faster than it adds it

The default rolling update strategy allows 25% of your pods to be unavailable during the update. On a Deployment with four replicas, that is one pod gone before its replacement is ready. If you are running near capacity, losing 25% of your fleet mid-deploy means queueing, timeouts and retries, none of which show up as a failed rollout.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 4
  # Rollout has 10 minutes to finish before it is marked failed.
  progressDeadlineSeconds: 600
  # A new pod must stay Ready this long before it counts. Catches pods
  # that pass the probe and then immediately crash.
  minReadySeconds: 15
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Never drop below the declared replica count.
      maxUnavailable: 0
      # Allow one extra pod on top while rolling.
      maxSurge: 1

Setting maxUnavailable: 0 is the right default for anything user-facing. The trade-off is real and worth stating: you now need headroom for one extra pod, and the rollout is slower because it is strictly sequential, waiting for each new pod to become Ready before removing an old one. On a Deployment with thirty replicas and a slow start-up, that can turn a two-minute deploy into fifteen. Raise maxSurge to a percentage if you have the node capacity and want the speed back.

minReadySeconds is the underrated one. Without it, a pod that passes its readiness probe and then falls over two seconds later still counts as a successful step, and the rollout marches on, replacing healthy pods with broken ones until the whole Deployment is bad. A short hold turns that into a stalled rollout you can catch, which is exactly what you want.

Rollouts are not the only thing that replaces pods, either. Node drains during a cluster upgrade evict pods too, and they respect PodDisruptionBudgets rather than your rolling update strategy. If you tuned one and not the other, an upgrade on a self-managed cluster, whether on cloud instances or your own VPS nodes from a provider like InterServer or Hetzner, will happily take out more replicas at once than any deploy ever did.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  # Keep at least three replicas up during voluntary disruptions.
  minAvailable: 3
  selector:
    matchLabels:
      app: api

Failure family four: the load balancer was never watching EndpointSlices

Everything above assumes traffic reaches your pods through a Service, where endpoint removal is the whole story. Plenty of production setups do not work that way.

Cloud load balancers that target pod IPs directly keep their own target registry. When a pod terminates, a controller has to call the cloud API to deregister it, that call has to be accepted, and the load balancer has to propagate the change to its own data plane. Then a configurable deregistration delay runs before the target is fully removed. That chain is slower and more variable than kube-proxy, often by an order of magnitude, and it explains why teams on managed clusters sometimes report that a preStop sleep of five seconds fixed nothing.

Two things help here:

• Pod readiness gates. These let an external controller add a condition to the pod that must be true before the pod counts as Ready. With a load balancer controller that supports them, a new pod is not considered Ready until the load balancer has actually registered it and passed its own health check. That closes the mirror image of the termination race, on the way in.
• Match your preStop sleep to the deregistration delay. If your target group is configured to drain for 30 seconds, a 5-second sleep is not enough. The sleep needs to cover the time until the load balancer genuinely stops sending new connections.

The same reasoning applies to an ingress controller running in the cluster. Ingress-NGINX, Traefik and friends watch EndpointSlices and reload their own config, which is fast but not instant, and each has its own graceful shutdown settings that need to be consistent with the pod’s grace period. If your edge sits behind Cloudflare or another CDN, be aware that it may retry an idempotent request against another origin, which quietly hides some of these errors from your users while leaving them in your logs. That is a good safety net, not a substitute for fixing the race.

Verifying a zero-downtime rollout instead of hoping

A rollout that has never been tested under load is a rollout you have not tested. The whole point of these failure modes is that they only appear when requests are in flight.

1. Generate steady traffic against the real ingress path. Not against a pod IP, and not through a port-forward. Those bypass exactly the layers you are trying to test.
2. Trigger a rollout with no image change, so you are testing the mechanism rather than your new code.
3. Watch EndpointSlices in a second terminal and note how long a terminating pod stays in the list.
4. Count non-200 responses. Anything above zero is a bug, not noise.
5. Repeat under realistic concurrency. A single-threaded curl loop will miss a 300ms window that a real traffic level would hit hundreds of times.

# Terminal 1: steady probe traffic, printing status and latency
while true; do
  curl -s -o /dev/null -w '%{http_code} %{time_total}n' https://api.example.com/
  sleep 0.2
done

# Terminal 2: watch endpoints appear and disappear in real time
kubectl get endpointslices -l kubernetes.io/service-name=api -w

# Terminal 3: roll the deployment without changing the image,
# then follow progress and stop on a stall
kubectl rollout restart deployment/api
kubectl rollout status deployment/api --timeout=10m

Do this once per service and record the numbers. If you already run Prometheus and Grafana, or a hosted stack like Grafana Cloud or Datadog, put a deploy annotation on your error-rate panel so the correlation is obvious next time instead of something someone has to notice by eye.

Troubleshooting

Rollout hangs, new pods never become Ready. The readiness probe is failing. kubectl describe pod shows the probe failure and the response it got. Check that the port name in the probe matches a declared containerPort name, that the path exists, and that the app is listening on all interfaces rather than only 127.0.0.1, which is a classic one when moving from a local docker-compose setup.

Probes fail only under load. Look at CPU limits before you look at anything else. A container being CPU-throttled cannot answer a probe within timeoutSeconds, and the default timeout is one second. This produces restart storms that look like an application bug and are actually a resource limit.

Errors persist after adding a preStop sleep. Either the sleep is shorter than your routing convergence time, or the load balancer is not driven by EndpointSlices at all. Time it: mark the moment of deletion, then watch how long the endpoint stays listed. Increase the sleep past that number and re-test.

Pods stuck Terminating for the full grace period. Your app is ignoring SIGTERM. This is extremely common when the container’s entrypoint is a shell script, because the shell runs as PID 1 and does not forward signals to the child. Use the exec form of ENTRYPOINT, or an init like tini, so your process actually receives the signal.

Rollout succeeded but the new version is broken. Roll back first, investigate second.

# Freeze a rollout mid-flight without reverting what has already landed
kubectl rollout pause deployment/api
kubectl rollout resume deployment/api

# Inspect revisions, then go back
kubectl rollout history deployment/api
kubectl rollout undo deployment/api --to-revision=3

Common mistakes

• Pointing liveness and readiness at the same endpoint, so a slow response restarts the container instead of just removing it from rotation.
• Checking databases or downstream APIs in the readiness probe, turning a dependency blip into a total outage.
• Leaving maxUnavailable at the default on a service running near capacity.
• Adding a preStop sleep longer than the grace period, so the app gets SIGKILLed before it can drain.
• Running a single replica and expecting a rolling update to be seamless. With one pod there is nothing to roll onto.
• Using RollingUpdate for workloads that cannot tolerate two versions running at once, such as a single-writer process or a schema change that is not backward compatible. Recreate exists for these, and it does mean downtime.
• Testing the deploy with no traffic flowing, which makes every one of these failure modes invisible.

Best practices

• Three separate endpoints: /healthz/started, /healthz/ready, /healthz/live. They answer different questions and should be allowed to disagree.
• maxUnavailable: 0 plus a maxSurge you have node capacity for, as the default for anything user-facing.
• A preStop sleep on every pod behind a Service, sized from a measurement rather than a blog post.
• A grace period that covers preStop plus your slowest realistic request, with slack.
• Handle SIGTERM properly in the application, and make sure it reaches PID 1.
• A PodDisruptionBudget on anything that matters, so node drains are as safe as deploys.
• minReadySeconds long enough to catch a pod that passes its probe and then dies.
• Annotate deploys on your dashboards, and treat any non-200 during a rollout as a defect rather than background noise.

FAQ

Do readiness probes alone give me zero-downtime deployments?

No. A readiness probe stops traffic reaching a pod that is not ready yet. It does nothing about the window between a pod being marked for deletion and the routing layer noticing, and nothing about capacity dipping during the rollout. You need the rolling update strategy and graceful termination as well.

How long should the preStop sleep be?

Long enough for everything that routes traffic to your pods to stop doing so. Measure it: delete a pod, watch the EndpointSlice, and time how long it stays listed. Behind a cloud load balancer, add its deregistration delay on top. Then confirm by running the load test with the value you picked.

Should the readiness probe check the database?

Almost never. It couples the availability of every replica to a single shared dependency, so one slow query can empty your Service of endpoints all at once. Handle dependency failures in the request path and return a meaningful error instead.

What is the difference between maxSurge and maxUnavailable?

maxSurge is how many pods you may run above the replica count during a rollout. maxUnavailable is how many you may drop below it. Surge costs resources; unavailability costs capacity. For zero downtime you want unavailability at zero and surge at whatever your nodes can absorb.

Why do I still get 502s on EKS, GKE or another managed cluster?

Cloud load balancers that target pod IPs maintain their own target registry, and deregistration goes through a cloud API rather than kube-proxy. That path is slower and more variable. Use pod readiness gates if your load balancer controller supports them, and size the preStop sleep against the target group’s deregistration delay rather than against kube-proxy.

Can I get zero-downtime rollouts with a StatefulSet?

The same probe and termination mechanics apply, but StatefulSets update pods one at a time in reverse ordinal order and cannot surge, so there is no extra pod covering the gap. Whether that is zero downtime depends entirely on whether your application tolerates losing one member at a time, which for most quorum-based systems it does and for a single-writer database it does not.

Does kubectl rollout status prove the deploy was clean?

It proves the desired number of pods reached Ready within the progress deadline. It says nothing about requests that failed on the way there. The only thing that proves a clean deploy is traffic flowing through the real ingress path with a zero error count.

The one thing worth remembering

Kubernetes zero-downtime rollouts are not a feature you turn on. They are the result of four separate things being configured correctly at once, and the readiness probe is only the most obvious of them. The one that actually drops requests in most clusters is the race at the other end: your process receives SIGTERM while the routing layer still believes the pod is a valid destination.

Delay the signal, size the grace period to cover the delay plus a real request, and then prove it with traffic flowing. Everything else is bookkeeping.

Need a second pair of eyes on your rollouts?

Most of this work is unglamorous: reading manifests, timing endpoint propagation, and finding the one Deployment that never got a preStop hook. Things I can help with:

• Auditing your Deployments and Helm charts for probe, surge and termination settings, with a prioritised list of what to change
• Measuring real endpoint propagation time in your cluster and sizing preStop and grace periods from that number
• Building a load-test harness that runs against a rollout in CI, so a regression fails the pipeline instead of the pager
• Tracking down 502s and connection resets that only appear during deploys, including cloud load balancer and ingress controller paths
• Splitting a single overloaded health endpoint into proper startup, readiness and liveness checks
• Adding PodDisruptionBudgets and drain-safety so cluster upgrades stop being an event

If you have a Deployment manifest, a kubectl describe pod output, or a graph showing your error rate during a deploy, send it over and I will tell you what I see.

The post Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn’t Saving You appeared first on John Nessime.

So you close the ticket. Then it happens again next week, to a different customer, on a different mailbox provider.

This is the shape of almost every WordPress email deliverability problem I’ve had to unpick. The site is not broken and the mail is genuinely leaving. It is being evaluated somewhere downstream and quietly filed into spam by a receiver that has decided it cannot prove who sent it. Nothing in WordPress will ever tell you this happened.

This post covers the four places that failure actually lives: how the message leaves your server, whether SPF survives evaluation, whether DKIM signs and aligns, and what DMARC does with the result. It is organised by failure family rather than by protocol tour, because the protocols are individually well documented and the failures are not.

The failure mode that costs you money: silent success

Start with the thing that makes this hard to debug. wp_mail() returns a boolean. That boolean tells you whether PHPMailer handed the message off without throwing. It tells you nothing about whether the message was delivered, and nothing at all about whether it reached an inbox rather than a spam folder.

So “WordPress says it sent” and “the customer got it” are separated by four or five systems you have no visibility into. That gap is where the money leaks: WooCommerce order confirmations, membership welcome mails, password resets, form notifications. All of them fail the same silent way.

So stop guessing about the boolean. WordPress fires an action when the handoff itself fails, and almost nobody hooks it:

// Log the real reason PHPMailer refused, instead of a bare false.
add_action( 'wp_mail_failed', function ( $error ) {
    error_log( 'wp_mail failed: ' . $error->get_error_message() );
} );

Then force a send from the command line so you are testing the mail path and not a contact form plugin:

wp eval 'var_dump( wp_mail( "you@example.com", "deliverability test", "body" ) );'

If that prints bool(true) and the mail still never arrives, you have ruled out WordPress entirely. Everything from here is transport and DNS.

Failure family one: how the message leaves the box

By default wp_mail() falls through to PHP’s mail(), which shells out to the local mail transfer agent. On a shared host or a default VPS build, that means your marketing site is now an email server with none of the reputation, warm-up, or bounce handling that implies.

Receivers are not being unreasonable about this. A message arriving from a web server IP, with no authentication, from a domain the IP has no relationship with, looks exactly like a compromised WordPress install pumping out spam. Because most of the time, it is.

The three transport options, honestly

Local mail(). Genuine case for it: zero dependencies, no credentials to leak, and it works fine when the box already runs a properly configured MTA with correct reverse DNS and a warmed IP. Where it fails: almost everywhere else. Most hosts either block outbound port 25 or run an MTA nobody has configured for authentication. Skip it unless you own the mail infrastructure and know its reputation.

Authenticated SMTP submission. WordPress connects to a real mail provider over port 587 with STARTTLS, or 465 with implicit TLS, and authenticates. This is the portable option, it works with any provider, and it is what I reach for first on a site I did not build. The cost is that SMTP is a chatty, connection-oriented protocol running inside a page request, so a slow or throttled provider makes checkout feel slow.

Provider HTTP API. The plugin posts the message to an API endpoint over 443 instead of speaking SMTP. Faster, immune to hosts that block SMTP ports outbound, and usually gives you richer delivery events back. The cost is real lock-in: your sending config is now provider-specific and moving means reconfiguring, not just changing a hostname.

On the plugin side, WP Mail SMTP and Post SMTP both cover the common providers and both have usable free tiers. For providers, transactional-focused services like Postmark, Amazon SES, Mailgun, Brevo and SendGrid all do this job well; the differences that matter are bounce webhook quality, log retention, and how quickly a human answers when your account gets throttled. If your host bundles a transactional relay, check whether it authenticates as your domain or the host’s shared domain before you rely on it. Providers like InterServer, Cloudways and Kinsta differ meaningfully here.

Prove the transport before you touch DNS

Before blaming records, confirm the server can actually reach the relay. This opens a TLS session against the submission port and shows you the SMTP banner and advertised capabilities:

openssl s_client -starttls smtp -crlf -connect smtp.example.net:587

-starttls smtp issues the STARTTLS upgrade for you, and -crlf sends proper line endings so the session does not hang. If this times out, your host is filtering outbound SMTP and you want the HTTP API transport instead of a longer argument with support.

To send a full test message end to end, swaks is the tool worth having on every box:

swaks --to you@example.com --from wp@example.com 
      --server smtp.example.net:587 --auth --tls

It prints the entire conversation, so an authentication failure or a policy rejection shows up as an actual SMTP response code rather than a shrug from a plugin.

Failure family two: SPF passes until it silently doesn’t

SPF is a DNS TXT record listing who may send using your domain in the envelope sender. It looks like this:

v=spf1 include:_spf.google.com include:sendgrid.net ~all

Two things about that record cause more WordPress email deliverability incidents than anything else in SPF.

The ten lookup budget

RFC 7208 caps SPF evaluation at ten mechanisms that require DNS lookups. include, a, mx, ptr, exists and redirect all count, and crucially so does everything nested inside someone else’s include. ip4 and ip6 cost nothing.

Cross the limit and the receiver returns permerror. That is not a soft fail. The receiver could not evaluate your record at all, treats SPF as failed, and every message from the domain is affected at once.

Here is why it bites so hard on WordPress sites specifically. Nobody adds ten includes in one sitting. Someone adds Google Workspace. Six months later the marketing team adds a newsletter platform. Then a CRM. Then a help desk. Then you add a transactional provider for the WooCommerce mail, and the record tips over. Nothing in DNS warns you, no plugin surfaces it, and the only visible signal is in DMARC aggregate reports you probably aren’t reading yet.

Check the record and count what it actually resolves to:

dig +short TXT example.com

Counting nested includes by hand is tedious and error-prone, so use a checker that expands them. When you are over budget, the durable fixes are removing senders you no longer use and moving distinct sending systems onto their own subdomains, each with their own SPF record. Flattening includes into raw IPs works, but you have just taken ownership of someone else’s IP ranges and they will change without telling you.

One record, and mind the envelope

Publish exactly one SPF record per domain. Two records is not additive, it is invalid, and receivers may fail the check outright.

Also remember SPF checks the envelope sender, not the address your visitor sees. On WordPress those can easily differ, especially when a plugin sets a friendly From address while the MTA uses the system user as the envelope. If you need to control it explicitly:

add_action( 'phpmailer_init', function ( $phpmailer ) {
    $phpmailer->Sender = 'bounces@example.com';
} );

Setting Sender sets the envelope from, which is what SPF evaluates and where bounces are returned. It is not the same field as From, and conflating the two is a classic source of “but SPF passes when I test it” confusion.

Failure family three: DKIM signs, but alignment is what counts

This is the invisible one, and if you take one thing from this post, take this.

DKIM adds a cryptographic signature to the message. The receiver fetches your public key from DNS at the selector your provider gave you and verifies the signature held. Check yours like this:

dig +short TXT selector1._domainkey.example.com

Straightforward enough. Here is the part that catches people: SPF and DKIM can both pass and DMARC can still fail.

DMARC does not care that a domain authenticated. It cares that the domain the recipient sees in the From header matches the domain that authenticated. That match is called alignment, and it is the whole point of DMARC. SPF aligns when the envelope sender domain matches the From domain. DKIM aligns when the signing domain in the d= tag matches the From domain. You need at least one of them aligned. Not both, but at least one.

Now picture the default setup on a WordPress site with a transactional provider. The From header says orders@example.com. The envelope sender is a bounce address on the provider’s own domain, so SPF authenticates the provider and does not align with you. And if you skipped the DNS records the provider asked you to add, the message is signed with the provider’s shared key, so d= is the provider’s domain and DKIM does not align either.

Both checks pass. Alignment fails. DMARC fails. Every deliverability tester you paste the record into says your DNS is fine.

The fix is nearly always the same: complete your provider’s domain authentication step so mail is signed as your own domain rather than their shared one. That usually means publishing a CNAME or TXT record at a selector under _domainkey, and often a CNAME for a custom return path so SPF aligns too.

Read the headers, don’t trust the dashboard

Send yourself a real message from the site and open the raw source. The Authentication-Results header the receiver added is the ground truth. You are looking for the SPF result, the DKIM result, the d= domain on the DKIM signature, and the DMARC verdict. If d= is not your domain and the SPF domain is not your domain, you have found your problem regardless of what any dashboard claims.

One practical note on the key itself: a 2048-bit DKIM public key does not fit in a single 255 character DNS string and has to be published as multiple concatenated strings. Most DNS providers handle this transparently. Some do not, and the symptom is a key that looks present but fails verification.

Failure family four: DMARC policy, and what just changed

DMARC ties the previous two together. It tells receivers what to do when neither SPF nor DKIM aligns, and it asks them to send you reports about it.

_dmarc.example.com. IN TXT "v=DMARC1; p=none; rua=mailto:dmarc@example.com"

Start at p=none. It changes nothing about how mail is handled and turns on the aggregate reports, which are the only view you will ever get of who is sending as your domain. Read them for a few weeks, find the legitimate senders you forgot about, fix their alignment, then move to p=quarantine and eventually p=reject.

Do not skip straight to p=reject on a domain you have not observed. You will reject your own invoicing system.

The specification itself was updated. RFC 7489, the original DMARC document from 2015, has been obsoleted by RFC 9989 together with RFC 9990 for aggregate reporting and RFC 9991 for failure reporting, and DMARC has moved onto the IETF standards track. Records still begin with v=DMARC1 and nothing you have published stops working, so this is not an emergency. Two changes are worth knowing:

• The pct tag is gone, along with rf and ri. Percentage-based staged rollout was implemented inconsistently by receivers and has been replaced by a simple t testing flag. If your record still carries a partial pct value from a rollout that never finished, that is worth cleaning up.
• Organisational domain discovery no longer depends on the Public Suffix List; receivers now walk the DNS tree. Single-domain owners will not notice. If you run a lot of subdomains, publishing explicit records rather than relying on inheritance removes the ambiguity.

There are also new optional tags. The one worth considering is np, which sets a policy for non-existent subdomains so attackers cannot spoof from a subdomain you never created, without changing your main policy.

Aggregate reports arrive as compressed XML and are unpleasant to read by hand. Feeding them to a processor is worth it; dmarcian, Postmark’s free DMARC digest and MXToolbox all do this, and Cloudflare can host the DNS side.

Are the bulk sender rules your problem?

Probably not, and it is worth saying so plainly because a lot of writing on this topic implies otherwise. Google’s stricter bulk sender requirements apply to domains sending roughly 5,000 or more messages a day to personal Gmail accounts. A typical WordPress business site sends a few dozen.

What changed for everyone is the direction of travel. Authentication that used to be a nice-to-have is now the baseline for being evaluated on your merits rather than treated as suspicious by default. If you do cross into bulk territory, the additional obligations are one-click unsubscribe on marketing mail via the List-Unsubscribe and List-Unsubscribe-Post headers, and keeping spam complaints low. Google’s published guidance is to stay under 0.1% and never reach 0.3%.

Domain strategy: keep resets away from newsletters

One structural decision saves more grief than any single record. Separate your transactional mail from your marketing mail at the domain level.

Send password resets and order confirmations from something like mail.example.com, and send campaigns from news.example.com. Each subdomain gets its own SPF record, which keeps you comfortably inside the lookup budget. More importantly, reputation is largely tracked per sending domain. When a campaign goes badly and complaints spike, the damage stays contained. Your password resets keep landing.

The trade-off is real: more DNS records to maintain, more DMARC reports to read, and a From address that is slightly less clean. On a site where a lost order confirmation costs actual revenue, that is a trade worth making.

Troubleshooting: a working order

1. Does the message leave? Hook wp_mail_failed and send via WP-CLI. A logged error here means credentials, ports or a plugin, not DNS.
2. Can the box reach the relay? openssl s_client -starttls smtp against the submission port. A timeout means outbound filtering.
3. What does the receiver say? Send to a mailbox you control, open the raw source, read Authentication-Results. This single header settles most arguments.
4. Is the DKIM d= your domain? If not, you have not completed provider domain authentication. Fix that before anything else.
5. Does SPF actually evaluate? Expand nested includes and count. A permerror looks like a pass in most naive testers.
6. What do the aggregate reports show? If mail is failing for a subset of recipients only, the reports will name the sending source you forgot about.

If Gmail is deferring rather than rejecting, the enhanced status code in the bounce tells you which check failed. Codes in the 4.7.x range point at authentication or alignment specifically rather than content, which narrows the search considerably.

Common mistakes

• Publishing two SPF records and assuming they merge. They do not.
• Treating a green tick from an online record checker as proof of delivery. Those tools validate syntax, not alignment on your actual mail.
• Jumping to p=reject before reading a single aggregate report.
• Setting a From address on a domain you do not control, such as a customer’s own address on a contact form. Use your domain in From and put theirs in Reply-To.
• Adding a new SaaS include without re-counting the lookup budget.
• Leaving a stale include for a service you stopped using two years ago, which is both a wasted lookup and a live authorisation.
• Configuring an SMTP plugin and never testing what happens when the provider is down. Mail queued nowhere is mail lost.

Best practices

• Route WordPress mail through a real provider, and prefer the HTTP API transport when the host filters SMTP ports.
• Complete provider domain authentication properly so DKIM signs as your domain. This is the single highest-value action on the list.
• Publish DMARC at p=none with rua on day one, even before you intend to enforce. The reports are the visibility you are missing.
• Keep SPF under the ten lookup budget with headroom for the next service someone adds.
• Split transactional and marketing sending onto separate subdomains.
• Enable email logging in your SMTP plugin so you have a record of what was attempted and when.
• Monitor DKIM key expiry and provider selector changes the same way you monitor TLS certificates. Both fail silently and both fail completely.

Frequently asked questions

Do I need an SMTP plugin if my host already sends mail?

Check what the host’s relay authenticates as. If it signs with the host’s shared domain rather than yours, DKIM will not align with your From address and DMARC will fail even though mail is technically going out. If it authenticates as your domain, you are fine without a plugin.

Is SPF enough on its own?

No. SPF breaks on forwarding, because the forwarding server is not in your record, and it authenticates the envelope sender rather than the visible From address. DKIM survives forwarding and gives you a second, independent path to alignment. Publish both.

Why do my emails reach Gmail but not Outlook, or the reverse?

Receivers weight signals differently and each maintains its own reputation view of your sending IP and domain. Divergent behaviour usually means your authentication is borderline rather than solid, so one receiver’s threshold catches it and another’s does not. Fix alignment rather than chasing individual receivers.

How long until DNS changes take effect?

As long as the TTL on the record, plus whatever the resolver in front of you decides to cache. Lower the TTL before you plan changes, not after. Verify with dig against an authoritative nameserver rather than trusting a browser-based checker.

Does DMARC stop people spoofing my domain?

At p=reject it stops exact-domain spoofing at receivers that honour the policy, which is most large ones. It does nothing about lookalike domains, display-name spoofing, or a compromised mailbox sending genuinely authenticated mail. It is one control, not a complete answer to phishing.

Should I update my DMARC record for the new RFCs?

Nothing breaks if you do not. At your next DNS edit, remove pct, rf and ri if present, and consider adding np if you want to close the non-existent subdomain gap. Leave the new tags unset otherwise, because the defaults are sensible.

Why did deliverability get worse after I moved hosts?

Usually because the old host had a relay configured that the new one does not, so PHP mail() is now sending direct from an unwarmed IP with no authentication. It is also common for a migration to move the site but not the DNS records the old provider had added. Re-verify SPF, DKIM selectors and the return path after any move.

Wrapping up

If you remember one thing about WordPress email deliverability, remember that authentication and alignment are different questions. SPF passing and DKIM verifying does not mean DMARC passes. The receiver is asking whether the domain your recipient can see is the domain that proved itself, and on a default WordPress plus transactional provider setup, the honest answer is often no.

Get the transport off PHP mail(), complete your provider’s domain authentication so d= is your own domain, keep SPF inside its lookup budget, and publish DMARC at p=none so you can finally see what is happening. Then read the reports for a month before you enforce anything.

The alternative is closing the same ticket every week and never learning why.

Need someone to fix this properly?

Mail problems are unusually well suited to outside help, because the failure is almost always in a place the site owner cannot see. This is work I take on regularly:

• Auditing an existing setup end to end and telling you exactly which of the four failure families you are in
• Rebuilding SPF records that have drifted past the ten lookup limit, without breaking a live sender
• Completing DKIM and return-path domain authentication with your provider so alignment actually holds
• Taking a domain from p=none to enforcement safely, using the aggregate reports rather than guesswork
• Splitting transactional and marketing sending onto separate subdomains with their own records and reputation
• Migrating WordPress mail off PHP mail() onto a provider, with logging and failure alerting wired up

If you want a second opinion, send me a raw message source with the full headers, or your current SPF and DMARC records, and I will tell you what I see.

The post Why Your WordPress Emails Vanish: SMTP, SPF, DKIM and DMARC appeared first on John Nessime.

Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker’s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.

This post covers Docker log rotation properly: what the defaults actually do, why the daemon.json change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.

The default is unbounded, and that is documented behaviour

Docker’s default logging driver is json-file. It captures stdout and stderr from the container’s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.

The important part is the option defaults. For json-file, max-size defaults to unlimited, max-file defaults to 1, and compress defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.

Docker’s own documentation is upfront about why. Keeping json-file unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.

Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:

# What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"

The gap between the first command and the second is where most of the pain lives, and we will come back to it.

To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container’s log file plus a small amount of metadata, so the sizes are close enough to be useful:

sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10

Container IDs are not memorable, so map the winner back to a name with docker ps --no-trunc or by grepping the ID against docker ps -aq. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.

Setting up Docker log rotation on the daemon

The daemon-wide fix goes in /etc/docker/daemon.json. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

What each key buys you:

• max-size caps a single file. Accepts an integer plus k, m or g. This is the setting that stops the runaway.
• max-file caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is only effective if max-size is also set, which is the single most common misconfiguration I see.
• compress gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.

Two things trip people up here. First, every value in log-opts must be a JSON string, including the numeric ones. Writing "max-file": 3 without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed daemon.json on a production host is a much worse afternoon than a full disk:

# Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  && echo "daemon.json parses OK"

sudo systemctl restart docker

Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is max-size multiplied by max-file. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.

Why your daemon.json change did nothing

This is the failure mode that costs the most time, because it looks exactly like the fix not working.

Logging configuration is baked into a container’s host config when the container is created. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because docker restart stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.

So the container that filled your disk keeps filling your disk, its log file still has no cap, and docker info cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.

To actually apply it, recreate the container and then verify rather than assuming:

# Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web

You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.

Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.

Pinning limits per service in Compose

Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides daemon.json entirely. A YAML anchor keeps it from turning into copy-paste sprawl:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

Be aware that the logging block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with its defaults, which for json-file means straight back to unbounded. Half-specifying is worse than not specifying at all.

You deleted the log file and df did not move

Classic 3am mistake, and it is worth understanding rather than memorising.

When you rm the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from ls, the daemon keeps writing into a file you can no longer see, and df reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.

truncate -s 0 is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:

sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"

There is a cost. Truncating out from under the daemon can leave an in-flight docker logs -f stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.

The same reasoning applies to logrotate rules pointed at /var/lib/docker/containers/*/*.log. Rotating with copytruncate works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker’s own docs warn against external tools touching those files. If the driver can rotate for you, let it.

Choosing a driver, and what each one costs

json-file

The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. docker logs works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.

local

Docker’s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it rotates and compresses by default, retaining five files of 20 MB each per container out of the box. docker logs works exactly as normal. The trade-off is that the format is Docker’s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is docker logs plus a shipper that talks to the daemon, this is the safest default you can pick.

journald

Hands container output to systemd’s journal, which means you get journalctl filtering, structured fields, and the journal’s own size management via SystemMaxUse in journald.conf. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.

The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at RateLimitIntervalSec and RateLimitBurst before you need them, not after.

syslog, fluentd and the remote drivers

These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.

What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.

none

Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because docker logs returns nothing and you will waste twenty minutes concluding the container is broken.

The blocking behaviour worth knowing about

By default the logging path is blocking. If the driver cannot keep up, the container’s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.

docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1

Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.

Dual logging, and why docker logs still works

If you switch to a remote driver and find docker logs still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the local driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.

Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.

Reclaiming space when the disk is already full

Order matters here. Free space first, fix the cause second, verify third.

1. Confirm logs are actually the problem. Compare du -sh /var/lib/docker/containers against docker system df, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.
2. Identify the largest log files with the du and sort command from earlier.
3. Copy anything you need out first, if the logs are part of an active investigation.
4. Truncate the offenders with truncate -s 0. Never rm.
5. Set the limits in daemon.json, validate the JSON, restart the daemon.
6. Recreate the containers so the limits take effect, then confirm with docker inspect on each one.
7. Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.

Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.

Troubleshooting

Rotation is configured but files still grow past the cap. The container predates the change. Check docker inspect --format '{{.HostConfig.LogConfig}}' and recreate it.

The daemon will not start after editing daemon.json. Almost always invalid JSON, or unquoted values in log-opts. Read the actual error with journalctl -u docker --no-pager -n 50 rather than guessing.

max-file is set but only one file ever appears. max-file does nothing without max-size. Set both.

docker logs -f hangs after showing existing output. Something truncated the file underneath the daemon, usually an external logrotate rule. Reattach to recover, then remove the rule.

docker logs returns nothing at all. Check the driver. none discards everything, and some remote drivers only serve reads through the dual-logging cache.

Disk usage did not drop after deleting log files. The daemon still holds the descriptors. Restart the daemon to release the inodes, and use truncate next time.

Logs vanish sooner than the retention you configured. On journald, check journal rate limiting and SystemMaxUse. On a remote driver in non-blocking mode, check whether the buffer is overflowing.

Common mistakes

• Assuming a daemon restart applies the new limits to running containers. It does not.
• Setting max-file without max-size and believing rotation is on.
• Writing numeric values unquoted in log-opts.
• Overwriting an existing daemon.json instead of merging, and silently dropping storage or network settings.
• Using rm on a live container’s log file and concluding the disk report is broken.
• Specifying a logging.driver in Compose without its options, which resets that service back to the driver’s own unbounded defaults.
• Pointing host logrotate at the Docker containers directory when the driver could have handled it.
• Sizing rotation per container without multiplying by container count against the actual disk.

Best practices

• Set logging limits in daemon.json as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.
• Prefer the local driver on standalone hosts unless something specifically needs to read the JSON files.
• Turn on compress. Log text compresses well enough that you get more retention for the same disk.
• Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.
• Do the multiplication. Worst-case host usage is max-size × max-file × container count, and it should be a fraction of your disk, not most of it.
• Alert on Docker data root usage at a threshold that leaves you time to act.
• Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.
• Verify with docker inspect after any change. docker info tells you the daemon’s intent, not what your containers carry.

Frequently asked questions

Does Docker rotate container logs automatically?

Not with the default driver. json-file ships with max-size unlimited and max-file set to 1, so a single file grows without bound. The local driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.

Where are Docker container logs stored on the host?

Under the Docker data root, one directory per container, which on a standard Linux install means /var/lib/docker/containers/. Rather than assembling the path by hand, ask the daemon: docker inspect --format '{{.LogPath}}' <container>. That works regardless of a custom data-root.

How do I clear Docker logs without stopping the container?

Truncate the file to zero bytes with truncate -s 0 on the path from {{.LogPath}}. Do not use rm: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.

Why did my daemon.json rotation settings not apply?

Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and docker restart reuses the same container object. Recreate the container, then confirm with docker inspect --format '{{.HostConfig.LogConfig}}'.

Should I use the local driver or json-file?

Use local if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use json-file if an agent or shipper on the host tails those files, because it cannot parse the local format. Either way docker logs behaves the same.

Can I use logrotate for Docker container logs instead?

You can, and plenty of hosts do, but the driver options are the better tool. A copytruncate rule pointed at the containers directory reaches into files the daemon treats as private, and can leave docker logs -f streams stalled after each rotation. Use logrotate for logs your applications write to mounted volumes, and let the driver handle stdout.

How much disk should I budget for container logs?

Worst case is max-size × max-file per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.

The one thing worth remembering

Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.

So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with docker inspect rather than trusting docker info. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.

Need a hand with container logging on your hosts?

Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:

• Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits
• Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number
• Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back
• Choosing between json-file, local, journald and shipping off-box, including the blocking behaviour trade-offs under load
• Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends
• Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot

If you want a second opinion, send me your daemon.json, the output of docker ps and a du -sh of your containers directory, and I will tell you what I would change.

The post Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation appeared first on John Nessime.