The same output showed roughly 11 GB in the available column. Nothing was wrong, and nothing had been wrong for months. The only thing broken was the alert that fired.

Low free memory on Linux is one of the most reliably misread signals in server operations. It looks like an emergency, it triggers scaling decisions, and it is almost always noise. This post covers what free is really telling you, which four signals actually indicate memory trouble, the one situation where a low free number genuinely matters, and what to put in your alert rules instead.

What the free command is actually reporting

free does not measure anything. It parses /proc/meminfo and prints a handful of those fields in columns. Every number it shows comes straight from the kernel and is readable directly if you want to skip the formatting.

A typical reading on a busy server looks roughly like this:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       3.4Gi       231Mi       128Mi        12Gi        11Gi
Swap:          2.0Gi          0B       2.0Gi

Read that left to right and the story is boring. Applications hold a few gigabytes. The kernel filled almost everything else with page cache and buffers, because every file that gets read or written passes through memory on the way, and directory metadata lands in reclaimable slab. What is left over, genuinely untouched, is the free column.

That last part is the bit people miss. Free memory is not a reserve. It is waste. A page holding a cached copy of a file your application reads every few seconds is doing useful work. A page sitting empty is doing nothing, and the kernel converts it into cache the moment it has a reason to. A server that has been up for a week and still shows plenty of free memory is a server that is barely touching its disk.

If you want the raw source rather than the formatted view:

$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SReclaimable|SwapTotal|SwapFree|Dirty):' /proc/meminfo

Two options are worth knowing. free -w switches to wide mode and splits buffers and cache into separate columns, which helps when attributing memory to a specific I/O pattern. free -h -s 5 reprints every five seconds, a crude but effective trend view while you reproduce a problem.

Signal one: available memory, not free memory

The available column maps to MemAvailable in /proc/meminfo, and it is the kernel’s own estimate of how much memory a new workload could allocate without pushing the system into swap. It has been present since kernel 3.14, so every distribution you are likely to be running exposes it.

What matters is that it is not free plus cache. That naive sum is what people calculated before MemAvailable existed, and it overstates the truth. The kernel starts from free pages, subtracts the reserves it refuses to dip below, then adds back the portion of file-backed page cache and reclaimable slab it believes it could actually release. Both additions are capped, because some cache is pinned and some reclaimable slab objects are in use.

The kernel documentation is explicit that this is an estimate whose accuracy varies from system to system, so treat it as a strong hint rather than an exact figure. It is still far better than anything you can compute yourself, and it is the number your dashboards should be built on.

If you change one thing after reading this: point your memory alerts at available, not at free. Everything else in this post is refinement on top of that.

Signal two: swap activity, not swap usage

The second most common false alarm is swap. Someone sees a few hundred megabytes of swap in use and concludes the machine is thrashing. It usually isn’t.

Swap usage is a stock: pages moved to disk at some point, possibly during a backup weeks ago, and never needed since. Leaving them there is correct, and pulling them back to satisfy a dashboard would evict something useful. Swap activity is a flow, and that is the part that hurts, because a page fault serviced from disk is orders of magnitude slower than one serviced from RAM.

vmstat gives you the flow. The si and so columns are swap-in and swap-out, in blocks per second over the sampling interval:

$ vmstat 5 6

Discard the first row. vmstat reports averages since boot on its first line, which is almost never what you want. Read the rows after that. If si and so are both consistently non-zero, the machine is simultaneously reading pages back to serve current work and writing pages out to make room for new allocations. That is thrashing, and it is a real signal with no ambiguity attached.

A single non-zero sample is not that. Brief swap activity during a log rotation or a backup window is normal. Sustained activity across several consecutive samples is what to escalate.

Signal three: memory pressure, which is the one that catches things early

Pressure Stall Information is the signal I reach for first now, and it is badly underused. It has been in the kernel since 4.20 and it answers a question none of the other tools answer directly: how much time are my processes losing because memory is contended?

$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

The some line is the share of wall-clock time in which at least one task was stalled waiting on memory. The full line is the share in which every non-idle task was stalled, meaning nothing useful happened at all. The averages cover ten seconds, one minute and five minutes, and total is a cumulative counter in microseconds that updates continuously, so it catches a two-second stall the running averages smooth away entirely.

Here is why this beats a percentage threshold. Utilization tells you how much of a resource is consumed; pressure tells you whether that consumption is costing you anything. A machine can sit at high utilization for months with zero pressure, which means the sizing is fine. It can also sit at moderate utilization while full climbs, which means reclaim is working hard and latency is already suffering with no threshold crossed.

With cgroup v2 mounted, the same data exists per cgroup in each directory’s memory.pressure file, so you can attribute pressure to a specific service rather than the whole machine. Prometheus node_exporter collects the system-level PSI counters, and if you already ship metrics to something like Grafana Cloud these graph well beside your latency panels.

Signal four: the OOM log, which is the only unambiguous one

When memory really does run out, the kernel writes a record. It does not do this quietly and it does not do it conditionally. If there is no OOM record, no process was killed for memory, full stop.

$ sudo journalctl -k --since "24 hours ago" | grep -i "oom"
$ sudo dmesg -T | grep -i "killed process"

The kill line names the victim and reports its memory breakdown at the moment of death: total virtual size, anonymous resident set, file-backed resident set, shared memory resident set, and the oom_score_adj that was in effect. The line above it, where a process “invoked oom-killer”, tells you which allocation triggered the whole event, and that process is frequently not the one that got killed.

That distinction is the most useful thing in the log. The kernel picks a victim by badness score, driven largely by how much memory a process holds relative to what is available to it, adjusted by oom_score_adj. A slow leak in a small helper can push the machine over the edge and get your database killed instead, because the database is simply the largest thing running. Read both lines before deciding who the culprit is.

If you have a service that should survive at almost any cost, systemd exposes OOMScoreAdjust= in the unit file, which persists across restarts in a way that writing to /proc/<pid>/oom_score_adj does not. Use it sparingly. Protecting a process that is itself leaking just moves the crash somewhere less convenient.

When low free memory on Linux really is the problem

Everything above assumes you are looking at a whole machine. Inside a container, the accounting changes, and this is the failure mode that is invisible until it isn’t.

The cgroup memory controller charges page cache to the cgroup that caused it. Clean, evictable, perfectly reclaimable cache still counts toward memory.current, and that is what gets compared against memory.max. A container reading a large file has just charged that file to its own budget.

$ cat /sys/fs/cgroup/memory.current
$ cat /sys/fs/cgroup/memory.max
$ grep -E '^(anon|file|slab_reclaimable) ' /sys/fs/cgroup/memory.stat

The split in memory.stat is what you actually want. anon is memory your process genuinely holds; file is page cache charged to you that the kernel can take back under pressure. A container at its limit on mostly file is in a very different position from one at its limit on mostly anon.

Two things follow from this that catch people out repeatedly:

• Reclaim is not instant. Under heavy pressure the reclaim path itself runs slowly, so a container can be OOM-killed while still holding clean page cache that was theoretically available.
• Running free inside a container lies to you. Unless something virtualizes /proc, you are reading the host’s /proc/meminfo, which knows nothing about your limit. The cgroup files are the only truth here.

cgroup v2 also has memory.high, a soft limit. Crossing it triggers reclaim and throttling rather than a kill, which turns a hard failure into a slow one. That is usually the better trade, and it is worth setting deliberately rather than leaving it at its default.

What to alert on

The alert on the ticket at the top of this post was watching the wrong number. Here is what a memory alert set looks like when it watches the right ones:

• Available memory below a percentage of total, sustained. Ten percent for ten minutes is a reasonable starting shape. A single thirty-second dip is noise and should never page anyone.
• Sustained swap-in and swap-out together. Rate of change on the swap counters, not the absolute amount of swap in use.
• Memory PSI rising. The some line moving off the floor is your early warning. The full line moving at all is your late one.
• Any OOM kill, ever. This one is binary and should always generate a ticket, even if the service recovered on its own.
• Per-container memory events. The cgroup memory.events file counts limit hits, and a rising count is a limit that needs revisiting.

Notice that free memory does not appear anywhere in that list. There is no threshold on the free column that produces a useful alert, because the number is a byproduct of the kernel’s caching behaviour rather than a measure of anything you care about.

Troubleshooting: which tool answers which question

When a memory question lands on you, the order you check things in matters more than the tools.

1. Did anything actually die? Check the kernel log for OOM records first. If there are none, this is a performance question and not a capacity failure, which changes everything about what you do next.
2. Is the machine stalling? Read /proc/pressure/memory. Zero pressure with low free memory means you are done. Go close the ticket.
3. Is it swapping right now? Run vmstat 5 for half a minute and watch si and so. Ignore the first row.
4. Who is holding the memory? ps aux --sort=-%mem | head -15 gets the shortlist. Resident set size double-counts shared pages, so summing the column overstates the total.
5. Is it the kernel rather than a process? If the process list does not account for the usage, check slabtop and SUnreclaim in /proc/meminfo. Unreclaimable slab growth is a different problem, usually a driver, filesystem or network workload creating enormous numbers of kernel objects.
6. Is it growing? A leak is a slope, not a level, and one reading tells you nothing. This is the whole argument for having the graph before you need it.

One more that saves time on crowded boxes: resident set size is the wrong measure of what a process really costs, because shared libraries are counted against every process that maps them. Proportional set size divides shared pages among the processes sharing them, and tools reporting PSS give a far more honest answer when you are deciding what to move off a busy VPS.

Common mistakes

• Dropping caches to make the number look better. Writing to /proc/sys/vm/drop_caches discards clean page cache and reclaimable slab. The kernel documentation says plainly that using it outside testing or debugging is not recommended, because everything dropped has to be re-read from disk. You trade a nicer-looking free output for an I/O spike and worse latency. It is a benchmarking tool, not an operations tool.
• Setting vm.swappiness to zero as a reflex. It does not disable swap and it does not make the machine faster. It makes the kernel strongly prefer evicting page cache over swapping anonymous pages, which on a database host means throwing away exactly the cache you wanted. Change it for a measured reason, not by default.
• Running with no swap at all. Swap is not just an overflow area. Without it the kernel loses the option of paging out genuinely cold anonymous memory, which means the only lever left under pressure is evicting cache or killing something. A modest swap file gives reclaim somewhere to go.
• Summing RSS across processes and comparing it to total RAM. Shared pages are counted once per process. The sum will exceed reality, sometimes dramatically.
• Resizing the instance before finding the slope. More RAM buys time against a leak. It does not fix one, and the bill arrives every month afterwards.

Best practices

• Graph available memory, swap rate and memory PSI on one panel. Together they read as a story; individually each one misleads.
• Keep the raw counters, not just derived percentages. A percentage of an instance size that has since changed cannot answer “was this happening last week”.
• Set container limits from observed anon usage plus headroom, not a round number someone picked, and put memory.high below memory.max so pressure shows up as slowness before it shows up as a kill.
• Alert on rate of change for anything that can leak. The steady climb is the signal; the level it reached is just when you happened to look.
• Right-size deliberately. On providers such as Contabo or InterServer, where RAM is a meaningful share of the monthly bill, the gap between a real requirement and a panicked upgrade is money you keep every month.
• Write the explanation down where your team will find it. Every organization rediscovers the free-versus-available confusion independently, usually at three in the morning.

Frequently asked questions

Why is free memory so low on my Linux server?

Because the kernel used it for page cache and buffers, which is the intended behaviour. Any file read or written passes through memory, and the kernel keeps those pages in case they are needed again. It reclaims them automatically the moment an application needs the space. Check the available column instead.

What is the difference between free and available in free -h?

Free is memory that is completely unused. Available is the kernel’s estimate of how much a new process could allocate without swapping: free memory plus the portion of cache and reclaimable slab it believes it could release. Available is almost always much larger, and it answers the question you are actually asking.

Should I clear the page cache to free up memory?

No. The kernel reclaims cache automatically under pressure, and more selectively than a blanket drop. Writing to drop_caches forces everything to be re-read from disk, and the kernel documentation advises against using it outside testing or debugging. Doing it on a schedule is a strong hint something else is misconfigured.

Is swap being used a bad sign?

Not by itself. Pages that were swapped out during a quiet period and never needed again are fine where they are. What matters is swap traffic: sustained non-zero si and so in vmstat at the same time means the machine is thrashing. Alert on the rate, not the amount.

How do I tell if a process was killed for running out of memory?

Search the kernel log. journalctl -k filtered for “oom”, or dmesg -T filtered for “killed process”, shows a record naming the process, its PID and its memory breakdown at the time. The kernel always logs this, so if there is no record the process died for some other reason.

Why does my container get OOM-killed when memory looks fine inside it?

Because free inside a container usually reads the host’s memory information, not the cgroup limit being enforced. Read memory.current, memory.max and memory.stat instead. Page cache charged to the container counts toward its limit, so a read-heavy workload can hit the ceiling without any process holding much anonymous memory.

What is a good memory alert threshold?

Available memory below roughly ten percent of total, sustained for ten minutes, is a defensible starting point to tune against your own workload. Pair it with memory PSI and an unconditional alert on any OOM kill. A threshold on the free column, at any value, will page you for nothing.

Conclusion

Low free memory on Linux is the expected steady state of a healthy, working server. The kernel fills memory with cache because empty memory earns nothing, and hands it back the instant an application asks. The free column is a byproduct of that behaviour, not a health metric, and alerting on it produces exactly the kind of urgent ticket that wastes an afternoon and ends in a change request nobody needed.

The one thing worth remembering: free tells you what is idle, available tells you what you can have, and pressure tells you whether any of it is costing you. Watch the second and third, log the OOM events, and read the cgroup files rather than free whenever a container is involved.

Need help reading your memory signals?

Most of the memory work I get asked for is not “add more RAM”. It is working out whether the number someone is worried about means anything. I can help with:

• Rebuilding memory alerts around available memory, swap rate and PSI so they stop paging on normal cache behaviour
• Diagnosing repeated OOM kills, including the cases where the process being killed is not the one causing the problem
• Setting container and pod memory limits from observed anonymous usage rather than guesses, with a soft limit so pressure surfaces before a kill does
• Separating a genuine leak from normal cache growth using retained metrics rather than a single reading
• Right-sizing servers and instances honestly, up or down, based on what the workload actually needs
• Building the dashboard that turns the next memory question into a five-minute conversation instead of an incident

If you have a free -h output, a vmstat capture or an OOM log line that is bothering you, send it over and I will tell you what it means.

The post Low Free Memory on Linux Isn’t a Problem. Here’s What Actually Is. appeared first on John Nessime.

The crontab entry is still there. The script still works when you run it by hand. Nothing in the monitoring went red, because nothing was watching whether the job ran at all. It was watching whether the server was up.

That is the real question behind cron vs systemd timers, and it has almost nothing to do with syntax. Both tools can run a command at 2 a.m. The difference is what happens on the night the command doesn’t run, and how long it takes anyone to find out.

This post compares the two by failure family rather than by feature list: missed runs, overlapping runs, environment surprises, output that goes nowhere, and jobs that eat the machine. Then a decision procedure, the same job written both ways, and how to debug each one when it goes quiet.

The failure that actually decides it

Scheduled jobs fail silently far more often than they fail loudly. A job that crashes with a stack trace gets fixed the same week. A job that stopped firing six weeks ago gets found during an incident.

Cron’s default answer to “what happened?” is to mail the job’s output to the owning user. On a server with no local mail transfer agent configured, which is most servers built in the last decade, that output is generated and then discarded. The daemon logs that it started the command. It does not log what the command printed, and in most implementations it does not log the exit status either.

So you get a log line saying cron ran your backup, every night, forever, including on the nights the backup wrote nothing. That line is the trap. It looks like evidence.

systemd timers hand this to journald instead. Standard output and standard error from the triggered service land in the journal, tagged with the unit name, alongside the exit status and the runtime. That is the single biggest practical difference between the two, and it is the reason most of my new scheduled work goes into timers.

Neither one alerts you. Both will happily not run for a month while you sleep well. Observability is not monitoring, and I’ll come back to that.

Where cron still wins

Cron is one line in one file. There is no unit to reload, no second file to keep in sync, no install section to get wrong. crontab -e, five fields, a command, done. For a job that trims a log directory on a box you own, that economy is not nothing.

It also runs where systemd does not. BSD hosts, Alpine images, containers, appliance firmware, embedded boxes, older enterprise systems still on a legacy init. If your configuration management has to cover a fleet that is not uniformly systemd, cron is the common denominator and it is not close.

Per-user scheduling is genuinely simpler too. A user runs crontab -e and owns their own jobs, with no root involvement and no unit files in /etc. On shared hosting and on control panels like DirectAdmin or cPanel, this is the only model available, and the panel writes the crontab for you.

Where cron loses

• No dependency ordering. Cron cannot wait for the network to be up or the database to be ready. It fires at the wall clock time and hopes.
• No overlap protection. If the 2 a.m. run is still going at 3 a.m., the 3 a.m. run starts anyway, on top of it.
• No resource containment. A runaway job competes with production for CPU, memory and disk with nothing standing between them.
• Minute resolution, and no catch-up if the machine was off.
• A syntax with sharp edges that fail quietly, which is the next section.

Where systemd timers win

A timer separates when from what. The .timer unit holds the schedule. The .service unit holds the job. That split feels like bureaucracy until the first time you need to run the job right now, out of band, without touching the schedule. Then it is just systemctl start myjob.service, and the run is logged exactly like a scheduled one.

Everything systemd knows how to do to a service, it will do to a scheduled job. Ordering after other units. Restart-on-failure. Runtime caps. Memory and CPU limits through cgroups. A dedicated user. A private /tmp. A failure handler unit that fires when the job exits non-zero.

You also get a schedule you can interrogate before it bites you. systemctl list-timers prints every timer on the box with its next and last elapse. systemd-analyze calendar takes an expression and tells you exactly when it would fire, which is the closest thing either scheduler has to a unit test.

Where systemd timers lose

• Two files, a daemon-reload and an enable for every job. That is real friction when you have forty small jobs.
• The calendar syntax is more expressive than cron’s and harder to eyeball. You will reach for systemd-analyze calendar more often than you expect.
• Portability is gone. These units run on systemd hosts and nowhere else.
• User timers need lingering enabled, or they stop when the user logs out and never start at boot. This surprises people once, memorably.
• The journal is not a log file. If your log shipping only tails /var/log, timer output is invisible to it until you point the shipper at journald.

Cron vs systemd timers, by failure family

The machine was off at 2 a.m.

Cron has no memory. If the box was down, rebooting, or suspended when the schedule came round, that run is gone. Nothing catches it up and nothing records that it was skipped.

The traditional fix is anacron, which tracks the last successful run per job in a timestamp file and runs the job shortly after boot if the interval has lapsed. It is why the scripts in /etc/cron.daily still run on a laptop that is closed every night. The catch is that anacron works in whole days and only covers the cron.daily, cron.weekly and cron.monthly directories by default. It does not help your custom job at 02:00.

systemd folds that behaviour into a single directive:

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

With Persistent=true, systemd writes a timestamp under /var/lib/systemd/timers/ each time the timer fires. On boot it compares that stamp against the schedule, and if a run was missed it triggers immediately. This matters more than it sounds on any VPS you reboot for kernel updates, and a great deal on hosts at providers like Contabo or InterServer where you take maintenance windows on their timetable rather than yours.

One caveat worth knowing: “immediately” means immediately. If you have twelve persistent timers and the box has been off for a week, all twelve fire within seconds of boot, on a machine that is still starting services. Pair Persistent=true with RandomizedDelaySec= so the catch-up is spread out.

Two copies of the same job

This is the one that corrupts data. A sync job normally takes four minutes and runs every five. One night the remote end is slow, the run takes eleven minutes, and cron starts two more copies while the first is still writing.

Cron will not stop this. You have to do it yourself, and the standard tool is flock from util-linux:

*/5 * * * * /usr/bin/flock -n /var/lock/sync.lock /usr/local/bin/sync.sh

-n means fail immediately rather than queue behind the running copy. Without it you build a backlog of processes all waiting for the same lock, which turns an overlap problem into a fork bomb with a slow fuse.

systemd gets this for free from its own model. A unit is either active or it isn’t. If sync.service is still running when the timer elapses, there is no second copy to start, because starting an already-active unit is a no-op. You do not have to remember anything.

One default is worth knowing. A calendar timer schedules its next elapse from the previous trigger time, so if a run overran, that next elapse is already in the past and the service fires again the moment it finishes. Newer systemd releases add a DeferReactivation= boolean that schedules from when the service went inactive instead, so an overrunning job waits for the next real slot. Check your distribution’s systemd version before relying on it.

The environment your job actually gets

“It works when I run it manually” is the single most common scheduled-job bug, and both schedulers cause it, for the same reason: neither one gives your job an interactive login shell.

Cron runs the command with a short, hardcoded environment. Your shell profile is not sourced. Your PATH is much shorter than the one you tested with, which is why a script that calls docker, wp, aws or a version-manager shim works from your terminal and fails from cron. The working directory is the user’s home, not wherever you happened to be standing.

Cron adds one hazard that is entirely its own. Inside a crontab, the percent sign is a metacharacter: it becomes a newline, and everything after the first one is fed to the command as standard input. So this looks fine and is broken:

# Truncated at the first %. The redirect never happens.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

# Escaped. This is what you meant.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

There is no warning. The line parses, cron runs the truncated fragment, and the log says the job started. Anything with a date + format string in a crontab deserves a second look.

systemd services get a minimal environment too, so the class of bug is identical. What differs is that the fix is declarative and lives with the unit rather than being hidden inside a wrapper script:

[Service]
Type=oneshot
User=deploy
WorkingDirectory=/srv/app
Environment=PATH=/usr/local/bin:/usr/bin:/bin
EnvironmentFile=/etc/myapp/backup.env
ExecStart=/usr/local/bin/backup.sh

The honest summary: absolute paths for every binary solve most of this on either scheduler. The rest is solved by never putting logic in the schedule. Put it in a script and schedule the script.

Everything firing at once

Write 0 0 * * * on fifty hosts and fifty hosts hit your backup target, your package mirror or your API at exactly midnight. The classic cron answer is to hash the hostname into a minute offset in configuration management, which works and which every team reinvents.

systemd has two directives here and they pull in opposite directions, which is the part people get wrong:

• RandomizedDelaySec= spreads timers apart. It adds a delay between zero and the value you set, so identical schedules on many hosts land at different moments.
• AccuracySec= pulls timers together. It lets systemd shift the firing time within a window to coalesce wakeups and save power. It defaults to one minute, which is why a timer set to 02:00:00 often fires a few seconds late.

If you want a timer to fire close to the stated second, lower AccuracySec=. If you want to stagger a fleet, raise RandomizedDelaySec=. Setting only AccuracySec= to a large value and expecting it to spread load is the mistake, because coalescing is the opposite of spreading.

A job that eats the box

Cron has no answer here beyond nice and ionice in front of the command, and a script that hangs forever will hang forever. A systemd service is a cgroup, so the limits are declarative and enforced:

[Service]
Type=oneshot
MemoryMax=512M
CPUQuota=40%
IOWeight=20
TimeoutStartSec=30min
OnFailure=notify-failure@%n.service
ExecStart=/usr/local/bin/reindex.sh

The two directives that earn their keep on a shared box are TimeoutStartSec=, which kills a oneshot job that has hung, and OnFailure=, which starts another unit when this one exits non-zero. That second one is how you get a scheduled job to page you without wiring an alerting call into every script.

The same job written both ways

A nightly database dump that must not overlap, must not run before the database is up, and must catch up if the host was rebooted. In cron:

0 2 * * * /usr/bin/flock -n /var/lock/db-backup.lock /usr/local/bin/db-backup.sh >> /var/log/db-backup.log 2>&1

That handles overlap and captures output. It does not handle “the database wasn’t ready” or “the host was down”, and the log file is now your problem to rotate.

The systemd version is two files. First the service, which is the job:

# /etc/systemd/system/db-backup.service
[Unit]
Description=Nightly application database dump
After=network-online.target mariadb.service
Wants=network-online.target

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/db-backup.sh
TimeoutStartSec=45min

Then the timer, which is only the schedule:

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run the nightly database dump

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=15min

[Install]
WantedBy=timers.target

Because the two units share a base name, you do not need a Unit= line. systemd pairs db-backup.timer with db-backup.service by convention. Add Unit= only when the names differ.

Then the part people skip, which is verifying it before walking away:

# Check the units parse before you enable anything
systemd-analyze verify /etc/systemd/system/db-backup.*

# Confirm the calendar expression means what you think
systemd-analyze calendar --iterations=5 "*-*-* 02:00:00"

systemctl daemon-reload
systemctl enable --now db-backup.timer

# Next and last elapse for this timer
systemctl list-timers db-backup.timer

# Run it now, out of band, without touching the schedule
systemctl start db-backup.service
journalctl -u db-backup.service -n 50 --no-pager

Note that you enable the timer, not the service. Enabling the service instead would try to run the job at every boot, which is a memorable way to learn the difference.

How I’d decide

Work down this list and stop at the first line that applies. It resolves most cases in under a minute.

1. The host might not be systemd, now or later. Use cron. Portability beats every other consideration on this list.
2. The job is inside a container. Use neither. Use your orchestrator’s scheduler, or a timer on the host that invokes the container.
3. A missed run matters, or the job must not overlap, or it needs to wait for another service. Use a timer. Every one of those is a directive rather than a wrapper script.
4. The job is heavy: a dump, a reindex, an import, anything that can pin a core or fill memory. Use a timer, for the cgroup limits and the runtime cap.
5. You need to see output after the fact without building a logging path first. Use a timer. The journal is already there.
6. You need sub-minute scheduling. Use a timer with a monotonic directive. Cron’s floor is one minute.
7. Otherwise, cron is fine, and one line beats two files.

The migration question answers itself from that. Do not convert forty working crontab lines in a weekend. Write new scheduled work as timers, convert the jobs that have actually burned you, and leave the rest. Mixing both on one host is normal and supported. Just never schedule the same job in both places, which is easier to do than it sounds when a configuration management run and a manual edit disagree.

Diagnosing a job that never fired

Cron

Establish what actually happened before you theorise. The daemon logs every job it starts, so the log tells you which of three situations you are in: cron never tried, cron tried and the command failed, or cron never saw the entry at all.

# Is the daemon running at all?
systemctl status crond      # RHEL, AlmaLinux, Rocky
systemctl status cron       # Debian, Ubuntu

# What did it start, and when?
journalctl -u crond --since "24 hours ago"
grep CRON /var/log/syslog | tail -20     # Debian family
tail -50 /var/log/cron                   # RHEL family

# Whose crontab are you actually looking at?
crontab -l -u deploy
ls -l /etc/cron.d/

Then work through the usual suspects, roughly in order of how often they turn out to be the cause:

• Wrong crontab. The job is in root’s crontab and you are reading the deploy user’s, or it’s in /etc/cron.d where there is an extra user field before the command.
• PATH, or an unescaped percent sign. Dump what the job really sees with a temporary entry like * * * * * env > /tmp/cron-env.txt 2>&1, then compare it against your shell.
• Permissions. The script is not executable, or the crontab owner cannot read it.
• The schedule is not what you think. When both day-of-month and day-of-week are restricted, cron runs the job when either matches, not both. 0 0 15 * 1 fires on the 15th and on every Monday.
• The command captured no output, so a real error was mailed into the void. Append >> /var/log/myjob.log 2>&1 and wait for the next run.

systemd

# Is the timer loaded and scheduled? --all shows inactive ones too
systemctl list-timers --all

# Timer state, last trigger, next elapse
systemctl status db-backup.timer

# What the job printed, and how it exited
journalctl -u db-backup.service --since "7 days ago"

# Prove the schedule independently of the unit
systemd-analyze calendar --iterations=3 "Mon..Fri *-*-* 06:30:00"

• The timer is not enabled. Creating the files does nothing. It needs daemon-reload and enable --now.
• You enabled the service instead of the timer. Check with systemctl is-enabled on both.
• A user timer with no lingering. Under systemctl --user, the manager stops when the session ends unless you run loginctl enable-linger for that user.
• The calendar expression is valid but wrong. Valid syntax that never matches will not error. systemd-analyze calendar is how you find out.
• The service failed before your script ran. A missing User=, an ExecStart path that does not exist, a WorkingDirectory that is not there. The journal names the reason.

Arguments that don’t survive contact

“Cron is deprecated.” It isn’t. It is packaged and supported on every mainstream distribution, and control panels, hosting providers and countless application installers write crontab entries as a matter of course. Nobody is removing it.

“Timers are too complicated for small jobs.” Two files is more typing, not more complexity. The complexity that matters is the wrapper script you write to get locking, logging and a runtime cap out of cron, and how well the next person understands it. Compare like for like.

“The journal means I have logging sorted.” The journal is a ring buffer with a size cap. If nobody set SystemMaxUse= deliberately and nobody ships journal entries anywhere else, your evidence has a retention window you never chose. Check it before you rely on it.

“Switching to timers will stop jobs failing silently.” This is the one I’d push back on hardest. Timers make failures visible. They do not make them noticed. Neither scheduler will ever tell you that a job did not run, because absence is not an event. That needs a dead man’s switch: the job pings a heartbeat endpoint on success, and something external alerts when the ping stops. Healthchecks.io, Cronitor and Better Stack all sell exactly this, and Healthchecks is open source if you would rather self-host it next to your own Grafana. If you already run Prometheus, writing a completion timestamp through the node exporter’s textfile collector and alerting on its age gets you the same property without another vendor.

“Just move everything to timers.” A mass migration of working jobs is a lot of change for no new capability, and every converted job is a chance to fat-finger a path. Convert on cause, not on principle.

Frequently asked questions

Are systemd timers better than cron?

For production jobs on a systemd host, generally yes: logging, dependency ordering, overlap protection, catch-up and resource limits are all built in rather than bolted on. For a one-line job on a box you own, cron’s simplicity is a real advantage and the timer buys you nothing you will use.

Is cron deprecated or being removed?

No. Cron implementations are still packaged and maintained across the major distributions. Some distributions do not install a cron daemon in minimal images by default, which is a packaging decision rather than a deprecation, so check with your package manager before assuming a fresh host has one.

Can cron and systemd timers coexist on the same server?

Yes, and most real servers run both. The cron daemon itself is usually managed as a systemd service. The only rule is that a given job should be scheduled in exactly one place. Double-scheduling is what produces the “why did this run twice” ticket.

How do I see when a systemd timer will next run?

systemctl list-timers shows next and last elapse for all active timers; add --all to include inactive ones. To test an expression without creating a unit, use systemd-analyze calendar "Mon..Fri *-*-* 06:30:00" with --iterations=N for more fire times.

How do I stop a cron job from overlapping itself?

Wrap it in flock -n with a dedicated lock file. The -n flag makes the new run exit straight away instead of queuing. Do not use a PID file you wrote yourself; it will leak a stale lock the first time the job is killed, and then the job stops running forever.

Do systemd timers work inside Docker containers?

Not usefully. A container normally runs a single process without systemd as PID 1, so there is no service manager to own a timer. Schedule from outside the container instead: a host timer that runs the container, or the scheduler your platform provides. Podman is the exception worth knowing, since it can generate systemd units for containers and the host’s systemd drives them.

How do I convert a crontab line to a systemd timer?

Move the command into a Type=oneshot service with an absolute ExecStart, then translate the five cron fields into an OnCalendar= expression of the form DayOfWeek Year-Month-Day Hour:Minute:Second. Verify the expression with systemd-analyze calendar, enable the timer, then remove the crontab line only after you have seen a successful run in the journal.

The one thing worth remembering

Cron vs systemd timers is not a contest between an old tool and a new one. Cron is a scheduler. A systemd timer is a scheduler attached to a supervisor, and the supervisor is what you are actually buying: dependency ordering, overlap protection, catch-up, resource limits, and output that lands somewhere you can read it.

Pick timers when a missed or duplicated run costs you something. Pick cron when it doesn’t. And whichever you pick, add a heartbeat, because the failure that hurts is not the job that crashes. It’s the job that stopped running while everything stayed green.

Need a second pair of eyes on your scheduled jobs?

Scheduling is one of those areas where the problem is rarely the syntax. It’s the job nobody has checked since the person who wrote it left. This is the kind of work I take on:

• Auditing every crontab and timer on a host and telling you which jobs have quietly stopped running
• Converting fragile cron entries to systemd timers with locking, runtime caps and failure handlers, without a big-bang migration
• Fixing the classic “works by hand, fails on schedule” environment and PATH problems, including the percent-sign trap
• Adding heartbeat monitoring and alerting so a missed run pages someone instead of sitting there
• Backup and dump jobs that verify their own output rather than exiting zero on an empty file
• Untangling duplicate schedules left behind by control panels, deploy scripts and configuration management

If something on your server isn’t running when it should, send me the crontab line or the unit file and the journal output around the time it should have fired. That’s usually enough to tell you what’s wrong before we talk about anything else.

The post Cron vs systemd Timers: The Failure Modes That Decide It appeared first on John Nessime.

The fix looked obvious. Drop in a limit_req_zone, apply it, reload, done. Except the reload changed nothing at all, and the reason took an embarrassingly long time to find.

This post covers Nginx rate limiting and bot mitigation the way it actually behaves in front of a real site: how the limiting modules key requests, what burst and nodelay really do to your traffic shape, why a limit that looks correct in the config can be doing nothing (or throttling everyone at once), and how to layer bot handling so you are not playing whack-a-mole with user agent strings forever.

The failure that hides: your limit is keyed on the wrong address

This is the one that bites hardest, because the config parses, nginx -t passes, and the error log stays quiet. Nothing tells you it is wrong.

If anything sits in front of Nginx (Cloudflare, another CDN, a load balancer, or a local Varnish tier), then $remote_addr is the address of that proxy, not the visitor. Key a rate limit on it and you get one of two outcomes, both bad:

• Every visitor arriving through a given edge node shares a single bucket. One aggressive client burns the budget and legitimate users start collecting 503s.
• Or your rate is generous enough that the shared bucket never fills, in which case the limit is decorative and the abusive client sails straight through.

The real IP module fixes this, but only if you trust the right sources. It rewrites $remote_addr and $binary_remote_addr from a header, and only for connections coming from addresses you have explicitly declared trustworthy.

# http context. One entry per CDN range you actually receive traffic from.
set_real_ip_from 203.0.113.0/24;
set_real_ip_from 2001:db8::/32;

real_ip_header CF-Connecting-IP;   # or X-Forwarded-For for a generic proxy
real_ip_recursive on;

Two things matter here more than the syntax.

First, never widen set_real_ip_from to 0.0.0.0/0. That tells Nginx to believe a client-supplied header from anyone, which means an attacker can pick a new “client IP” on every request and your per-IP limit becomes unlimited. It also poisons your logs, your deny rules and anything downstream that trusts the forwarded address.

Second, after the real IP module has run, $binary_remote_addr holds the visitor address and $realip_remote_addr holds the original connecting address (the proxy). Key your limits on $binary_remote_addr. Keying on $realip_remote_addr puts you right back in the shared-bucket failure, silently.

The proxy ranges change over time, so treat the trust list as a file that gets regenerated on a schedule and reloaded, not something you paste in once. On a self-managed VPS from a provider like Contabo or InterServer, a small cron job that fetches the current ranges, rewrites an include file, runs nginx -t, and only reloads on success is about twenty lines of shell and saves you from a very confusing afternoon later.

What limit_req actually does, and what burst really means

Nginx rate limiting for requests uses a leaky bucket. The zone stores, per key, the current number of excessive requests. The rate defines how fast that excess drains. Two directives do the work: limit_req_zone declares the bucket in the http context, limit_req applies it in http, server or location.

http {
    limit_req_zone $binary_remote_addr zone=general:10m rate=10r/s;
    limit_req_zone $binary_remote_addr zone=login:10m   rate=20r/m;

    limit_req_status 429;
    limit_req_log_level warn;
}

The rate is expressed in requests per second (r/s) or requests per minute (r/m). If you want less than one request per second, use r/m. There is no native per-hour or per-day unit in the stock module, which surprises people writing password-reset limits.

burst without nodelay is a queue, not a rejection

This is the second-most common misunderstanding after the keying problem. By default, excessive requests are delayed so that they come out at the configured rate. They are only rejected once the number waiting exceeds burst.

location /search/ {
    limit_req zone=general burst=20;
}

At 10r/s, a client that fires 21 requests at once gets one served immediately, nineteen held and released over roughly two seconds, and the last one rejected. Those nineteen held requests are still open connections on your server. You have not reduced concurrency, you have converted a spike into a slow trickle and paid for it in worker slots and file descriptors.

That is genuinely useful for smoothing traffic to a fragile upstream. It is a bad idea for abuse control, where holding the connection is exactly what the attacker wants.

nodelay, delay, and picking between them

nodelay changes the behaviour to: serve everything in the burst window immediately, reject everything past it. The bucket still drains at the configured rate, so a client that sustains the burst rate will hit the wall and stay there.

location /wp-login.php {
    limit_req zone=login burst=5 nodelay;
    limit_req_status 429;
    # ...fastcgi_pass or proxy_pass here
}

The delay parameter is the middle ground: it sets the point within the burst at which requests start being delayed rather than served immediately. burst=20 delay=10 means the first ten excess requests go straight through, the next ten get queued, and anything beyond is rejected.

How I decide: if a human clicking around your site could plausibly generate the spike, use nodelay or a delay that covers it. A single page load pulling twelve uncached assets through Nginx is one “request” to the user and twelve to the limiter. If the endpoint is machine-facing and expensive (search, export, login, an API write path), tighten the rate, keep the burst small, and use nodelay so rejection is fast and cheap.

Also set limit_req_status 429. The default is 503, which tells clients and monitoring that your server is broken rather than that they are being throttled. 429 is the correct semantic, and well-behaved clients back off on it.

Sizing the zone, and the number most guides get wrong

You will see “10m holds about 160,000 IP addresses” repeated everywhere. That is the 32-bit figure. Nginx documents the stored state for limit_req as 64 bytes on 32-bit platforms and 128 bytes on 64-bit platforms, so one megabyte holds roughly 16 thousand states on 32-bit and roughly 8 thousand on 64-bit. On the 64-bit server you are almost certainly running, a 10m zone is closer to 80,000 states.

This matters because of what happens when the zone fills. Nginx evicts the least recently used state. If it still cannot allocate, the request is terminated with your configured error status. A zone that is constantly thrashing under a distributed crawl will start returning errors for reasons that have nothing to do with the rate you configured. If you are limiting a high-cardinality key, give the zone room.

limit_conn solves a different problem, and you probably need both

limit_req caps how fast requests arrive. It does nothing about a client that opens sixty connections and reads each response one byte at a time. That is a concurrency problem, and it belongs to limit_conn.

http {
    limit_conn_zone $binary_remote_addr zone=perip:10m;
    limit_conn_status 429;
}

server {
    limit_conn perip 20;

    location /downloads/ {
        limit_conn perip 4;
        limit_rate_after 2m;
        limit_rate 512k;
    }
}

Three behaviours worth knowing before you pick a number.

• Not every connection counts. Nginx only counts a connection once the full request header has been read and the request is being processed. A client stalling mid-header is not counted here, which is why client_header_timeout and client_body_timeout still matter.
• Under HTTP/2 and HTTP/3, each concurrent request counts as a separate connection. A limit of 4 that felt generous on HTTP/1.1 can break a modern browser loading a page with many assets. This is a very common self-inflicted outage.
• The limit_conn state is smaller than the limit_req state, so the same zone size holds roughly twice as many entries.

The pairing is the point: limit_req protects against volume, limit_conn protects against occupancy. Rate limiting alone leaves you open to slow-read exhaustion, and connection limiting alone does nothing about a client making one fast request after another on a keepalive connection.

Choosing keys: IP is a starting point, not the answer

The key can be any variable or combination of variables. That flexibility is where most of the value is, because per-IP limits have a real accuracy problem in both directions.

On one side, carrier-grade NAT, corporate egress, university networks and shared VPN exit nodes (NordVPN, Surfshark and every other provider works this way) put many genuine users behind one address. A tight per-IP limit will hit them. On the other side, a distributed scraper renting a few hundred cheap addresses never touches a per-IP limit at all, because no single address is doing much.

So layer keys instead of tuning one harder:

limit_req_zone $binary_remote_addr zone=perip:20m    rate=10r/s;
limit_req_zone $server_name        zone=persite:1m   rate=200r/s;

Applied together, the per-IP zone catches the loud individual and the per-site zone caps total damage regardless of how many addresses are involved. When several limit_req directives match, all of them are evaluated and the most restrictive result wins. If any one of them rejects, the request is rejected.

One inheritance rule catches people out: limit_req directives are inherited from the enclosing level only if the current level defines none. Add a single limit_req inside a location and you have dropped every limit set at server level for that location. If you want both, restate both.

Allowlisting with an empty key

Requests whose key evaluates to an empty string are not accounted at all. That is the clean way to exempt monitoring probes, your office, or a payment webhook source without maintaining a parallel set of locations.

geo $limit {
    default        1;
    10.0.0.0/8     0;
    192.0.2.10/32  0;
}

map $limit $limit_key {
    0 "";                    # empty key, never limited
    1 $binary_remote_addr;
}

limit_req_zone $limit_key zone=perip:20m rate=10r/s;

Nginx evaluates map variables only when they are used, so declaring a large map costs nothing on requests that never reference it.

Roll it out in dry run, not in production

Both modules have a dry run mode. Excessive requests are counted in the shared memory zone and logged exactly as they would be, but nothing is actually rejected or delayed. Combined with the status variables, this gives you a real answer to “would this limit have hurt anyone” before you find out from a customer.

log_format ratelimit '$remote_addr $status "$request" '
                     'req=$limit_req_status conn=$limit_conn_status '
                     'ua="$http_user_agent"';

server {
    limit_req_dry_run  on;
    limit_conn_dry_run on;

    access_log /var/log/nginx/ratelimit.log ratelimit;
}

$limit_req_status reports PASSED, DELAYED, REJECTED, DELAYED_DRY_RUN or REJECTED_DRY_RUN. $limit_conn_status reports PASSED, REJECTED or REJECTED_DRY_RUN. A numbered rollout that has never embarrassed me:

1. Add the zones and the limit_req directives with dry run on, plus the log format above.
2. Run for a full traffic cycle. A week if you have weekly patterns, longer if you have a monthly billing spike.
3. Count the would-be rejections by user agent and by address. If real browsers appear, your rate or burst is wrong, not your users.
4. Turn dry run off on the least critical location first. Watch 429 rates alongside your normal error budget.
5. Only then apply it to login, checkout, or anything that costs you money when it breaks.

Shipping the rate limit log into Loki and graphing it in Grafana makes step three take minutes instead of an evening with awk. Any log platform with decent label cardinality handling works; the point is having rejection counts broken down by zone and user agent, over time, next to your latency graphs.

Bot mitigation in layers, cheapest first

Rate limiting treats all clients the same. Bot mitigation is about deciding that some clients deserve different treatment before the limiter ever sees them. The useful mental model is a set of layers ordered by cost and by how easy each is to defeat.

Layer one: robots.txt, and being honest about what it does

robots.txt is a request, not a control. Crawlers that respect it will honour a Disallow on your search endpoint, your faceted filter URLs and your infinite calendar pages, and that alone removes a surprising amount of load. Crawlers that do not respect it will ignore the file completely, and some will read it specifically to find out where the interesting URLs are.

Use it to shape well-behaved traffic. Never use it as a security boundary. And be careful about blanket disallows on search engines you actually want: deindexing yourself is a much more expensive mistake than a bit of crawler load.

Layer two: user agent classification with map

A map on $http_user_agent is cheap, readable, and easy to revise. It is also trivially spoofable, so treat it as traffic shaping rather than enforcement.

map $http_user_agent $ua_class {
    default          human;
    ""               empty;
    "~*bot|crawler|spider|scrape"  bot;
}

map $ua_class $bot_limit_key {
    bot   $binary_remote_addr;
    empty $binary_remote_addr;
    human "";
}

limit_req_zone $bot_limit_key zone=bots:10m rate=30r/m;

That gives declared bots a much tighter budget than browsers while leaving human traffic on the general limit. An empty user agent is worth its own class: almost nothing legitimate sends one, and it is a reliable signal for cheap scraping tools.

Two cautions. Broad substring matching on bot will catch strings you did not intend, so review a sample of your own logs before deploying the pattern. And blocking rather than throttling is a bigger decision than it looks: if you return 403 to a crawler you actually want, you find out weeks later in your search traffic, not in your error log.

Layer three: verifying bots that claim a name

The interesting case is a request claiming to be a major search crawler. Impersonating one is the standard way to bypass a user agent based allowlist, so a claim needs verification.

The method the search vendors themselves document is a forward-confirmed reverse DNS check: reverse-resolve the client address, confirm the resulting hostname belongs to the vendor’s domain, then forward-resolve that hostname and confirm it returns the original address. An attacker cannot fake this without controlling the vendor’s DNS. Several vendors also publish their crawler IP ranges as machine-readable lists, which avoids the DNS round trip entirely.

Here is the honest limitation: stock Nginx has no directive that does this. There is no built-in reverse DNS verification in the core modules. Your realistic options are a third-party module, doing the verification out of band and feeding the result into a geo block or an included allowlist file, or pushing the decision up to a CDN or WAF layer that already does it. Anyone showing you a pure stock-Nginx config that “verifies Googlebot” is verifying a string, not an identity.

This is also the point where an edge provider earns its keep. Cloudflare, Fastly and similar edge platforms maintain verified bot lists and fingerprinting that no single origin server can reproduce, and they drop the traffic before it costs you a worker. If your bot problem is large and distributed, the edge is the right layer and Nginx is your backstop, not your front line.

Layer four: escalation to the firewall

A client that collects hundreds of 429s and keeps going is not going to stop. Serving it a rejection still costs you a TCP handshake, a TLS negotiation and a log line. At that point the cheapest response is to stop answering at all.

Fail2ban is the usual tool: a filter that matches your rate limit rejections in the Nginx error or access log, an action that inserts a firewall drop, and a ban time long enough to be inconvenient. Two things to get right. Match on a pattern that only fires for real limit rejections, or you will ban yourself during a deploy. And if you are behind a CDN, your firewall sees the CDN address, not the client, so the ban has to be applied at the edge instead. Banning a Cloudflare range at your origin firewall takes your whole site offline, and it is a mistake people make exactly once.

Return code 444 is worth knowing here. It is Nginx-specific and closes the connection without any response, which is cheaper than generating an error page for a client that will never read it.

Troubleshooting Nginx rate limiting

The limit does nothing. Check the key first. Log $binary_remote_addr alongside $realip_remote_addr and confirm they differ in the way you expect. Then check inheritance: a limit_req in the matched location discards the server level ones. Confirm which location actually matched by adding $request_uri and a marker header while testing.

Real users are getting 429 or 503. Turn dry run back on and read the log rather than guessing. Usually it is one of three things: a burst too small for a normal page load’s asset count, a limit_conn value set for HTTP/1.1 assumptions on an HTTP/2 server, or a shared-egress network where many people genuinely share one address.

Nothing appears in the error log. The default log level for rate limit refusals is error, and delays are logged one level below whatever you set. If you set limit_req_log_level warn, delays go to info, which your log level may be filtering out entirely. Grep for the limiting messages directly:

grep 'limiting requests' /var/log/nginx/error.log | tail -50
grep 'limiting connections' /var/log/nginx/error.log | tail -50

Intermittent errors under heavy load with no matching rate. Suspect zone exhaustion. Enlarge the zone and see whether the pattern changes. High-cardinality keys and distributed crawls are the usual cause.

Health checks and uptime probes are being throttled. Add them to the allowlist map. Monitoring hitting a rate limit produces alerts about an outage that is not happening, which is worse than no monitoring.

Common mistakes

• Trusting forwarded headers from every source, which turns a per-IP limit into no limit at all.
• Leaving limit_req_status at the default 503, so throttling looks like an outage in your own monitoring.
• Using burst without nodelay on an abuse path, and quietly filling worker slots with held connections.
• Setting a limit_conn value based on HTTP/1.1 intuition on a server serving HTTP/2 or HTTP/3.
• Adding one limit_req in a location and silently dropping every server-level limit for that path.
• Blocking user agent substrings without checking a sample of real logs first.
• Applying identical limits to reads and writes. A cacheable GET and a POST that writes to the database do not deserve the same budget.
• Banning CDN address ranges at the origin firewall because the logs show them as the client.

Best practices worth the effort

• Verify the key before tuning the rate. Every other decision depends on it being correct.
• Always start in dry run and measure a full traffic cycle before enforcing.
• Use separate zones per traffic class. Search, login, API, static and general traffic have different shapes and should not share a bucket.
• Pair limit_req with limit_conn. The first caps arrival rate, the second caps occupancy, and neither substitutes for the other.
• Layer a per-server zone above your per-IP zone so distributed traffic still hits a ceiling.
• Allowlist with an empty key rather than duplicating location blocks.
• Export $limit_req_status to your log platform and graph rejections next to latency and upstream errors.
• Keep CDN trust ranges in a generated include file that is refreshed on a schedule and validated with nginx -t before reload.
• Throttle before you block. Blocking failures are invisible until they show up in your search traffic.

Frequently asked questions

Does Nginx rate limiting stop a DDoS attack?

It helps with application-layer floods that reach your server, because rejecting a request is far cheaper than processing it. It does nothing about volumetric attacks that saturate your network link, since that traffic never gets to Nginx to be limited. For that you need capacity in front of the origin: a CDN or scrubbing service. Treat rate limiting as protection against abuse and accidental floods, not as DDoS mitigation.

What rate should I actually set?

There is no universal number, and any guide that gives you one has not seen your traffic. Measure instead: run dry run mode, look at your real per-IP request distribution, and set the rate above the busiest legitimate client with headroom. Then set a separate, much tighter zone for the specific endpoints that are expensive or attacked.

Should I use 429 or 503 for rejected requests?

429 in nearly every case. It communicates that the client should slow down, well-behaved clients back off on it, and it keeps your 5xx metrics meaningful. The default is 503 purely for historical reasons.

Can Nginx rate limit per hour or per day?

Not with the stock module. It accepts requests per second and requests per minute only, so the slowest native rate is one request per minute expressed as 1r/m. Longer windows need a third-party module, a shared store such as Redis via a scripting module, or enforcement in the application itself. For something like a password reset limit, the application is usually the better place anyway, because it can key on the account rather than the address.

Do rate limit zones work across multiple Nginx servers?

Not in the open source build. Each instance keeps its own shared memory zone, so a client spread across four load-balanced servers effectively gets four times the budget. The sync parameter on limit_req_zone synchronises state across a cluster, but it is part of the commercial subscription. Without it, either divide your intended rate by your instance count, ensure your balancer pins clients consistently, or move the limit to a shared store.

Will blocking AI crawlers hurt my SEO?

Blocking training crawlers is separate from blocking search indexing crawlers, and the distinction is worth checking per vendor before you write a rule, because some vendors run several crawlers with different purposes under related names. The risk is collateral damage: a broad user agent pattern that also matches a search crawler will remove you from results, and you will not notice quickly. Verify your pattern against real log samples, and prefer throttling over outright blocking while you are unsure.

Where should rate limiting live: the CDN, Nginx, or the application?

All three, doing different jobs. The CDN handles volume and known-bad reputation before it costs you anything. Nginx handles per-client shaping and protects your workers from anything that gets through. The application handles limits that need identity or business context, like per-account quotas, which Nginx cannot see. Skipping the Nginx layer because you have a CDN leaves you exposed the moment someone finds your origin address.

The one thing worth remembering

Nginx rate limiting is not hard to configure. It is easy to configure into a state where it silently does nothing, or silently punishes the wrong people, and both failures look identical from the outside: a config that parses, a clean nginx -t, and no complaints until there are.

So get the key right, prove it with a log line rather than assuming it, and run every new limit in dry run until the data tells you it is safe. Layer bot handling by cost, throttle before you block, and be honest about which layer can actually verify an identity. Everything else is tuning, and tuning is easy once the foundation is not lying to you.

Need a second pair of eyes on your Nginx edge?

Most of the work here is diagnostic rather than clever. Things I regularly help with:

• Auditing an existing limit_req and limit_conn setup to find out whether it is actually limiting anything, including the keying and inheritance traps above.
• Fixing real client IP handling behind Cloudflare or another CDN, with a generated trust list that refreshes safely instead of drifting.
• Designing per-endpoint zones for WordPress, WooCommerce or API origins so login, search and write paths get budgets that match their real cost.
• Running a measured dry run rollout and reporting back what would have been rejected, broken down by user agent and network, before anything is enforced.
• Building bot classification and escalation: user agent maps, allowlists, Fail2ban integration, and knowing which decisions belong at the edge instead.
• Wiring rate limit telemetry into Grafana and Loki so throttling is something you can see on a dashboard rather than discover from a ticket.

If you would like a look at yours, send me the relevant server block, a sample of the access log, and whatever is currently going wrong. That is usually enough to say something useful straight away.

The post Nginx Rate Limiting and Bot Mitigation Without Blocking Real Users appeared first on John Nessime.

That is the thing about DirectAdmin PHP-FPM pools that catches people out. You think you are tuning a site. You are not. You are tuning a user, and every domain that user owns is sharing the same worker budget, the same queue, and the same memory ceiling.

This post covers where DirectAdmin actually keeps the pool config, how to size pm.max_children from numbers your server gives you rather than from a blog post’s magic value, which per-site PHP limits are worth setting and which are theatre, why OPcache does not behave the way you expect on a multi-tenant box, and how to make all of it survive the next CustomBuild run.

The design fact that explains most of your 502s

DirectAdmin generates one FPM pool per user account, per PHP version. The pool is named after the Linux user. It listens on a Unix socket named after the user. It runs as that user. Apache or Nginx hands PHP requests to that socket for every domain and subdomain in the account.

So if a reseller client has twelve domains under one username, those twelve sites share a single value of pm.max_children. There is no per-domain pool. This is a long-standing feature request rather than an oversight, and it has a real consequence: the noisiest site in an account sets the availability of every other site in that account.

Once you have internalised that, the fix for a lot of “mystery 502” tickets stops being a config tweak and starts being an account layout decision. If a site matters commercially, give it its own DirectAdmin user. That is the cleanest isolation the panel offers without going off-piste, and it costs you nothing but a bit of admin tidiness.

The alternative, if you have User jails enabled, is DirectAdmin’s isolated FPM mode. That runs a separate php-fpm master process per jailed user instead of a shared master with many pools:

da config-set isolated_fpm 1
systemctl restart directadmin
da taskq --run 'action=rewrite&value=httpd'

It only applies to users who already have Jailed enabled. The trade-off is memory: one master process per user, plus a separate OPcache region for each, adds up quickly. DirectAdmin mitigates that by letting idle per-user FPM instances exit after a period of inactivity, but on a box with sixty accounts you will feel it. I would reach for this on a server hosting a handful of paying clients, not on a general shared box.

Where the pool config actually lives

This trips up almost everyone the first time, because there are two files with nearly the same name and only one of them matters for user sites.

• /usr/local/php83/etc/php-fpm.conf is the master config for that PHP version. Editing it by hand is how you learn that CustomBuild overwrites it.
• /usr/local/directadmin/data/users/USERNAME/php/php-fpm83.conf is the generated per-user pool. This is the file that governs a customer’s sites.
• /usr/local/directadmin/data/templates/php-fpm.conf is the template those per-user files are rendered from.

Substitute your PHP version for 83 throughout. To see what every account is currently running:

ls /usr/local/directadmin/data/users/*/php/php-fpm*.conf

grep -H 'pm.max_children' /usr/local/directadmin/data/users/*/php/php-fpm*.conf

The second command is the one I run first on any server I have just inherited. It gives you the entire worker budget of the box on one screen, and it is usually the moment you discover that forty accounts are all sitting on the stock default.

The DirectAdmin template ships with pm = ondemand and a low default child count, driven by a MAX_CHILDREN token. Ondemand means no workers exist until a request arrives, and idle workers are reaped after a timeout. On a shared box with many mostly-idle accounts, that is the right default and I would leave it alone. On a server dedicated to one busy WordPress site, the spawn latency on the first request after an idle period is real and switching to dynamic is defensible. Most people who switch to dynamic on a shared server are trading memory they do not have for a latency win they will not notice.

Sizing pm.max_children without guessing

Every tuning guide gives you a formula. The formula is fine. The input everyone gets wrong is worker memory, because they use a number from someone else’s server instead of measuring their own. A lean WordPress install and the same install with a page builder, a security plugin and WooCommerce are not remotely the same workload.

Measure it. This averages the resident memory of the workers currently running in one pool:

ps -eo rss,args | grep '[p]hp-fpm: pool wpuser' 
  | awk '{sum+=$1; n++} END {if (n) printf "%d workers, avg %.0f MBn", n, sum/n/1024}'

The bracket trick in the grep pattern stops grep matching its own process. rss is resident set size in kilobytes, which is why the awk divides twice. Run it during a busy period, not at three in the morning, and run it a few times.

Two honest caveats about RSS. It overstates real consumption, because workers forked from the same master share a lot of pages, including the OPcache region. And it understates your worst case, because a worker that has not yet hit a memory-hungry request looks cheap. Treat the average as a planning figure, not a guarantee, and leave headroom.

Then the arithmetic:

1. Take total RAM and subtract what MySQL or MariaDB, the web server, mail, and the panel itself need. On a WordPress box the database is usually the second-largest consumer and it is not optional.
2. Reserve a further slice for page cache and burst. I aim to leave a comfortable margin free rather than allocating to the last megabyte.
3. Divide what is left by your measured per-worker figure. That is your ceiling for all pools combined.
4. Split that ceiling across accounts by actual traffic, not evenly.

Step three is where people go wrong on cheap VPS plans. If you are running a dozen WordPress sites on one of the low-RAM tiers from a provider like Contabo or InterServer, the honest answer is often that you cannot give every account a generous pool and you need to either cut the site count or move up a tier. Setting pm.max_children to a number your RAM cannot back does not buy capacity. It buys the OOM killer, and the OOM killer usually takes the database with it, which turns a slow site into a broken one.

To raise the server-wide default:

da config-set php_fpm_max_children_default 30
systemctl restart directadmin
da build rewrite_confs
systemctl restart php-fpm83

For a single account, go to Admin Level → Custom HTTPD Configurations → domain.com and open the php-fpm entry for that PHP version. In the |CUSTOM1| textarea, set the token:

|?MAX_CHILDREN=40|

That syntax is a template token override, not a config line. It tells the template renderer to use 40 instead of the default when it writes pm.max_children for that user. Writing pm.max_children = 40 in that box instead is the single most common mistake here, and it does something worse than nothing: the template still emits its own pm.max_children, you now have the directive twice in one pool, and FPM refuses to start. On a server where one broken pool file stops the whole master, that takes down every site on the box.

Rewrite and restart after any change, then confirm the value actually landed in the generated file rather than trusting the panel.

Per-site PHP limits, and the difference that matters

The |CUSTOM2| textarea on the same page is where PHP settings go. The syntax is FPM pool syntax, not php.ini syntax:

php_admin_value[memory_limit] = 512M
php_admin_value[max_execution_time] = 120
php_value[upload_max_filesize] = 64M
php_value[post_max_size] = 64M
php_value[max_input_vars] = 5000

The admin in php_admin_value is not decoration. Settings written with php_value can be raised at runtime by the application through ini_set(). Settings written with php_admin_value cannot. That distinction is the whole game on a multi-tenant server.

WordPress and a good number of plugins will happily try to raise memory_limit at runtime, and some optimisation plugins push it to values that would let one bad request consume a large slice of the machine. If you want a genuine ceiling, use php_admin_value. If you want a sensible starting point that a developer can raise for a legitimate import job, use php_value. I use admin for memory and execution time, plain for upload sizes, because upload sizes are a support-ticket generator and not a stability risk.

max_input_vars deserves its own note. It is the one that silently truncates a large WooCommerce product save or a menu with hundreds of items, with no error anywhere. The user reports “my changes did not save” and every log is clean. If you support content-heavy sites, raise it before someone asks.

Letting site owners set their own

Under FPM there is no .htaccess route for PHP settings, because FPM is a separate process and never sees the file. DirectAdmin enables the per-directory alternative by default: a .user.ini file in public_html, controlled by the user_ini.filename setting. It handles the PHP_INI_PERDIR and PHP_INI_USER class of settings only, so it will never override anything you set with php_admin_value. That is the point of it.

The WP-CLI gap

None of this applies to WP-CLI. Cron jobs and shell commands run through the CLI SAPI and read the CLI php.ini, not your pool. So a plugin update that works fine in the browser can die on memory in a cron-driven wp cron event run, and you will hunt through the pool config finding nothing wrong. Check the CLI limits separately when a scheduled task fails but the same action succeeds in wp-admin.

OPcache is shared, and that is the invisible one

OPcache allocates its shared memory region when the FPM master starts, before any pool exists. Every pool under that master inherits the same region. You cannot meaningfully give one account its own OPcache budget through a pool directive, because by the time the pool is read the memory has already been carved out.

Two consequences, and the second is the one nobody plans for.

First, capacity. The default OPcache size is comfortable for one application and thin for thirty WordPress installs, each with its own core, theme and plugin tree. When it fills, OPcache starts evicting and eventually restarts, and every site on the version gets slower at once with nothing in any error log to explain it. Watch the wasted-memory and restart counters rather than guessing.

Second, the sizing you actually need scales with the total number of PHP files across all accounts on that PHP version, not the biggest site. Consolidating everyone onto one PHP version looks tidy and quietly makes this worse. Spreading accounts across two or three PHP versions gives each master its own region, which is an underrated side effect of not forcing everyone to upgrade on the same day.

OPcache is configured through CustomBuild rather than the pool. Edit the customised copy under CustomBuild’s custom directory so your changes survive updates, then rebuild and restart the FPM master.

Making changes survive the next rebuild

Anything you type directly into a generated file is temporary. The next rewrite_confs, PHP update or panel update will flatten it, usually weeks later, and the regression will look like it came from nowhere.

There are three durable places to put changes, in increasing order of blast radius:

• Per user, all PHP versions. /usr/local/directadmin/data/users/USERNAME/php/php-fpm.conf.custom1 and .custom2. These feed the same CUSTOM tokens as the panel textareas, so the settings follow the account when the user switches PHP version in Domain Setup. This is the one I reach for first.
• Server-wide, all users, all versions. /usr/local/directadmin/data/templates/custom/php-fpm.conf.custom1 and .custom2. If either exists, its contents are injected into every pool file DirectAdmin writes. Good for a baseline you want everywhere.
• A full custom template. Copy /usr/local/directadmin/data/templates/php-fpm.conf to the custom subdirectory and edit it. This is the only route for pool directives that have no token, but you now own that file forever and stop receiving upstream template improvements. Use it when nothing else works, and leave a comment at the top saying why.

Global first, then per-user on top. Both land in the same token, so a per-user value does not replace a global one, it appends. For a directive that can only appear once in a pool, that is a duplicate and a failed restart.

Troubleshooting

“server reached max_children setting”

The pool name in that warning tells you which account, not which site. Find the log first rather than guessing at a path:

grep -i 'error_log' /usr/local/php83/etc/php-fpm.conf

Before raising the limit, check whether the requests are legitimate. Cross-reference the timestamps against each domain’s access log in that account. Nine times out of ten it is one of three things: an uncached admin-ajax.php loop from a plugin, a bot crawling faceted search or filtered archive URLs that bypass page cache, or wp-cron.php firing on every request on a site with a heavy scheduled task. Raising pm.max_children to absorb bot traffic is paying for garbage with RAM.

502 or 504 with no PHP error

The request is dying at the FPM boundary rather than inside PHP. Usual causes are a worker killed by request_terminate_timeout, a full listen queue, or the master failing to start after a config change. Check the FPM service status before you check anything in WordPress. If FPM did not start at all, the panel will still show your settings saved and every site on that PHP version will be down.

Changes in the panel that do nothing

Read the generated per-user file after every change. If the value is not in there, the rewrite did not run or the token was wrong. If it is in there and PHP still reports something else, the master was not restarted, or something later in the pool is overriding it. A temporary phpinfo() file served by the site in question settles the argument in seconds. Delete it immediately afterwards.

Session or temp files that are not where you left them

If PHP writes to /tmp and you cannot see the files over SSH, that is systemd’s PrivateTmp giving the FPM service its own private namespace. Nothing is broken. The files exist, in a mount only that process can see.

disable_functions that will not budge

disable_functions is special. From a pool you can add to the list but not remove from it, because the php.ini value has already been applied. To allow a function for one account you have to stop setting it in php.ini at all and drive the whole thing from templates instead. DirectAdmin documents that pattern, and it is worth reading before you start rather than after you have half-done it, because the intermediate state removes protection server-wide.

Common mistakes

• Editing /usr/local/phpXX/etc/php-fpm.conf directly and being surprised when it reverts.
• Writing pm.max_children = 40 into a CUSTOM box instead of the |?MAX_CHILDREN=40| token, producing a duplicate directive and a dead master.
• Sizing workers from a memory figure found online rather than measured on the box.
• Using php_value for memory_limit on shared hosting, then wondering how a plugin raised it to 2G.
• Assuming pool settings apply to WP-CLI and cron.
• Raising limits to absorb bot traffic instead of blocking or caching it upstream.
• Putting a client’s revenue site in the same account as their abandoned projects.

Best practices for tuning DirectAdmin PHP-FPM pools

• One DirectAdmin user per site that matters. Isolation you get for free beats isolation you have to engineer.
• Keep ondemand on shared servers. Only move to dynamic when a single busy site owns the box.
• Set a global baseline in the custom template, then override per user. Do not hand-tune forty files.
• Use php_admin_value for anything that protects the server and php_value for anything that is only a convenience.
• Cut PHP requests before you add workers. Full-page caching, a CDN in front, and Cloudflare or similar handling bots removes far more load than any pool setting.
• Disable WordPress’s request-triggered cron and drive it from a real system cron, so scheduled work stops competing with visitors for workers.
• Alert on saturation, not just uptime. Worker utilisation per pool and PHP-FPM error log entries are the signals worth having in Netdata, Prometheus, or whatever you already run. A site that is queueing looks perfectly healthy to an HTTP check.
• Re-measure after major plugin changes. A page builder or a new security plugin moves per-worker memory enough to invalidate your arithmetic.

Frequently asked questions

Can I create a separate PHP-FPM pool for each domain in DirectAdmin?

Not through the panel. Pools are generated per user, per PHP version, and every domain in the account shares one. The supported ways to get separation are to move the domain to its own DirectAdmin user, or to enable isolated FPM mode for jailed users, which gives each user their own master process. Anything else means custom systemd units you maintain yourself, outside the panel’s knowledge.

What is a good pm.max_children value for WordPress?

There is no portable number, because the input is your measured per-worker memory and your available RAM after the database. Measure the average resident size of your workers under real traffic, subtract everything else the server needs from total RAM, and divide. A lean brochure site and a WooCommerce store with thirty plugins can differ by a factor of three or more.

Where does DirectAdmin store PHP-FPM pool configuration?

Generated per-user pools live at /usr/local/directadmin/data/users/USERNAME/php/php-fpmXX.conf, rendered from the template at /usr/local/directadmin/data/templates/php-fpm.conf. Do not edit either directly. Put durable changes in the custom files under the user’s php directory or in the templates/custom directory.

Why do my PHP settings disappear after a CustomBuild update?

Because they were written into a generated file. CustomBuild and DirectAdmin regenerate those files on update and on any config rewrite. Only the custom template files and the CUSTOM token areas persist.

Does .htaccess work for PHP settings under PHP-FPM?

No. FPM is a separate process from the web server and never reads .htaccess. Use a .user.ini file in the web root for the settings PHP allows at that level, or set the rest in the pool.

Should I switch from ondemand to dynamic or static?

On a shared server, no. Ondemand exists precisely so that dozens of idle pools cost you nothing. On a server dedicated to one high-traffic site, dynamic removes cold-start latency and static gives the most predictable memory profile. The question to ask is whether your idle pools outnumber your busy ones.

Can I give one WordPress site its own OPcache?

Not from the pool config. OPcache’s shared memory is allocated by the FPM master at startup and shared by every pool under it. The practical routes are isolated FPM mode, which gives jailed users their own master, or moving the site to a different PHP version so it sits under a different master.

The one thing to take away

DirectAdmin PHP-FPM pools are a per-user boundary, not a per-site one. Every tuning decision that follows, how many workers to allow, whether a memory limit should be a suggestion or a wall, whether a client’s twelve domains belong in one account, comes out of that single fact.

Measure your workers before you size them, use the CUSTOM token files so your work survives the next rebuild, and watch pool saturation rather than waiting for the 502 ticket. The site that goes down is rarely the site that caused it.

Need a hand with a DirectAdmin server?

Most of the work I do on DirectAdmin boxes is this kind of thing: sites that are fine in isolation and fall over together. If that sounds familiar, I can help with:

• Auditing every pool on a server and producing a worker budget that matches the RAM you actually have
• Tracking down which domain in a shared account is saturating the pool, and why
• Building a global custom template so per-user tuning stops being forty hand-edited files
• Splitting a multi-site DirectAdmin account into isolated users without breaking mail, DNS or SSL
• OPcache and per-version sizing on servers hosting many WordPress installs
• Wiring up pool-level monitoring and alerts so saturation shows up before the ticket does

Send me a pool file, the output of that RSS command, or the last hour of your PHP-FPM error log, and I will tell you what I see.

The post DirectAdmin PHP-FPM Pools: Why One WordPress Site Takes Down the Others appeared first on John Nessime.

Sometimes the disk really is dying. More often it isn’t, and the number you’re staring at is not measuring what you think it’s measuring.

This post covers troubleshooting high I/O wait the way I’d actually do it on a production box: what the metric means, why it lies in both directions, and how to work through the five failure families that produce it.

What high I/O wait actually measures

Here’s the definition that matters: iowait is a subcategory of idle time. For a given CPU, it’s the time that CPU spent doing nothing while at least one task scheduled on it had an outstanding block I/O request.

Two consequences fall out of that, and both will burn you. High iowait can mean nothing is wrong. If the box has spare CPU and one process is pulling a large file off disk, the CPU genuinely has nothing else to do. Copying a backup will push iowait toward 100 percent on an idle host, and that’s a working system, not a fault.

Low iowait can hide a severe problem. This is the one that costs people days. If the CPU has other runnable work, the kernel context switches to it and that time gets counted as user or system, not iowait. A busy application server can be stalling badly on storage while wa reads two percent, because the CPU is never actually idle.

Add the averaging problem on top. The wa figure in top is averaged across every core. One thread pinned to one core and blocked hard on disk shows up as roughly three percent on a 32-core machine. Press 1 in top to break out per-CPU lines before you conclude anything from the aggregate.

So iowait is a hint, not a verdict. Treat it as the thing that starts the investigation, never the thing that ends it.

Start with pressure, not percentages

Pressure Stall Information is the metric I reach for first, and it exists precisely because iowait is a poor proxy for pain. It answers the question you actually care about: how much time did tasks lose because they couldn’t get I/O?

cat /proc/pressure/io

You get two lines. some is the share of time at least one task was stalled waiting on I/O. full is the share of time every non-idle task was stalled at once, which is CPU capacity being burned for nothing. The avg10, avg60 and avg300 fields are percentages over the last 10, 60 and 300 seconds. total is a cumulative microsecond counter since boot, which is what you want for graphing because averages smooth away short spikes.

The practical rule I use: a rising full line on io is a real user-visible problem almost every time. A high some line with a flat full line usually means one noisy job is waiting while everything else gets on with its life.

If /proc/pressure doesn’t exist, the kernel was built without CONFIG_PSI, or it was built with PSI compiled in but disabled by default, in which case you need psi=1 on the kernel command line. On cgroup v2 systems you also get per-cgroup pressure, which is how you narrow from “the host is stalling” to “this container is stalling” in one command:

# Pressure for one systemd slice, useful on any modern distro
cat /sys/fs/cgroup/system.slice/io.pressure

# Pressure for every leaf cgroup that has it, sorted by nothing,
# just eyeball the full lines
find /sys/fs/cgroup -name io.pressure -exec sh -c 
  'printf "%s: " "$1"; sed -n 2p "$1"' _ {} ;

If you run Prometheus, node_exporter has exposed these since it gained a pressure collector. The counters are node_pressure_io_waiting_seconds_total for some and node_pressure_io_stalled_seconds_total for full. Rate those and you have a far better storage alert than any threshold on %iowait will ever be. Grafana Cloud and Netdata both chart PSI out of the box if you’d rather not build the panels yourself.

Failure family one: the device really is saturated

This is the case everyone assumes, so get it confirmed or eliminated quickly.

iostat -xy 1

The -x gives extended per-device stats. The -y matters more than people realise: without it, the first report is a since-boot average, and reading that first block is the single most common iostat mistake there is. It will tell you the disk has been fine all year, which is true and useless.

Now the columns worth your attention:

• r_await and w_await: average milliseconds a read or write took, queue time included. This is the closest thing to “how slow does storage feel”. Split reads from writes, because they fail very differently.
• aqu-sz: average queue depth. Requests either queued or in service. A depth that climbs while throughput stays flat is the signature of a genuinely saturated device.
• r/s, w/s, rkB/s, wkB/s: the actual workload. Compare against what the device or volume class is rated for.
• %util: treat with suspicion, see below.

The %util trap. That column counts the fraction of time at least one request was in flight. On a spinning disk that serviced one request at a time, it was a fair saturation signal. On NVMe and on cloud volumes that handle many requests in parallel, a device can sit at 100 percent while running at a fraction of its real capacity. Provisioning a bigger volume off the back of a 99 percent %util reading is a common and expensive way to change nothing. Use aqu-sz and the await columns instead. Similarly, svctm was deprecated and dropped from newer sysstat output because it was unreliable on parallel devices; ignore it if an old build still prints it.

Once you know the device is busy, find the culprit. pidstat -d 1 gives per-process read and write rates from the same sysstat package and needs nothing special enabled. iotop -o is friendlier, but on kernels from 5.14 onward per-task delay accounting is off by default, so the SWAPIN and IO> columns come back empty until you turn it on:

sudo sysctl kernel.task_delayacct=1
sudo iotop -oPa
# turn it back off when you're done, it costs a little scheduler overhead
sudo sysctl kernel.task_delayacct=0

And check the boring things before the clever ones. dmesg -T | grep -iE 'i/o error|reset|medium' catches a failing drive or a flapping controller. smartctl -a /dev/sda catches reallocated sectors on hardware you own. If either of those turns something up, stop reading and replace the disk.

Failure family two: memory pressure in an I/O costume

This is the failure family I’d check second, and it’s invisible if you only look at storage.

When RAM gets tight, the kernel starts evicting page cache and swapping. Both generate disk traffic. The disk gets busy, iowait climbs, and every graph points at storage while the actual constraint is memory. Give the box more RAM and the “disk problem” disappears.

# si and so are swap-in and swap-out in KB/s.
# Sustained non-zero values are your answer.
vmstat 1

# Are we thrashing writeback? Dirty pages waiting to be written.
grep -E 'Dirty|Writeback' /proc/meminfo

# The direct question
cat /proc/pressure/memory

If memory pressure and I/O pressure rise together, fix memory first. Everything else is downstream.

There’s a related trap on the write side. The kernel buffers writes in page cache and flushes them in the background, governed by vm.dirty_background_ratio and vm.dirty_ratio. Cross the background threshold and flusher threads start writing. Cross the hard vm.dirty_ratio threshold and writing processes are forced to do writeback themselves, synchronously. That transition is brutal, and on a box with plenty of RAM and a modest disk the defaults let you accumulate gigabytes of dirty pages and then pay for all of them at once. Lowering the ratios makes writeback more frequent and less spiky, trading away a little write batching. Measure before you commit to it.

Failure family three: small synchronous writes

Throughput can be trivial and the system still crawls. A workload doing thousands of tiny fsync calls a second is bound by round-trip latency, not bandwidth, and no amount of extra megabytes per second will help.

Databases are the usual source, because durability has a cost and that cost is a flush. MySQL and MariaDB expose it through innodb_flush_log_at_trx_commit; PostgreSQL through synchronous_commit. Loosening either buys real throughput and trades away durability on power loss or crash. That can be a perfectly reasonable trade for a cache or an analytics replica, and a terrible one for anything holding orders or payments. Decide it explicitly, write down what you decided, and don’t let it be a thing somebody changed once at 2am.

The other common source is logging. Application logs, access logs, container JSON logs and a database sharing one volume will starve each other. Splitting hot write paths onto separate devices is unglamorous and works.

Failure family four: throttled, not saturated

A throttled device and a saturated device look almost identical from inside the guest: high await, growing queue, miserable latency. The difference is that no amount of tuning inside the box will fix a throttle. Three flavours are worth knowing.

• Cloud volume limits. Managed block storage gives you a baseline of IOPS and throughput, sometimes with a burst allowance on top. Burn the burst and you drop to baseline, and everything that was fine in testing falls over in production. The tell is a latency cliff that arrives at a consistent point into a workload rather than tracking load smoothly. On AWS, watch the volume-level CloudWatch metrics such as BurstBalance on burstable volume types and VolumeQueueLength; other providers publish equivalents. The fix is a volume class change, not a sysctl.
• cgroup limits you set yourself. On cgroup v2, io.max caps a cgroup at a given bytes-per-second or IOPS figure, and io.weight shares bandwidth proportionally under contention. A limit that made sense when it was written can become the bottleneck a year later, and nothing in the container’s own metrics announces it.
• Neighbours on shared hardware. On oversubscribed VPS plans your storage is shared, and someone else’s backup job is your latency spike. This is where cheap and expensive hosting genuinely differ. Budget providers like Contabo and InterServer are fine value for a lot of workloads, but if storage latency is load-bearing for you, look for plans with dedicated NVMe rather than shared pools, and test before you migrate anything that matters.

# Per-device I/O accounting for a cgroup, including any throttling
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.stat

# Is a cap set? An empty file means no limit on that cgroup.
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.max

One more signal while you’re here. In top, the st column is steal time, meaning the hypervisor gave your vCPU’s slice to someone else. Steal and iowait rising together on a shared host is a strong hint that the problem is the neighbourhood, not your configuration.

Failure family five: the latency isn’t in the block layer

Here’s the failure mode that wastes the most time, because every storage metric you check will look healthy.

iostat measures the block layer. It cannot see anything above it. So if your latency is in NFS, in a network filesystem, in an overlay or FUSE mount, in filesystem lock contention, or in a journal that’s serialising writes, the device will report sub-millisecond service times while your application waits tens of milliseconds per operation.

The diagnosis is the gap. Measure latency at the application’s layer and at the block layer, and if the two numbers disagree wildly, the problem lives in between.

# Block layer latency as a histogram. Look for a bimodal shape
# or a long tail rather than the average.
sudo biolatency-bpfcc -m 10 1      # Debian/Ubuntu naming
sudo /usr/share/bcc/tools/biolatency -m 10 1   # RHEL family

# Filesystem operations slower than 10ms, with the process and file
sudo ext4slower-bpfcc 10
sudo xfsslower-bpfcc 10

# Everything at the VFS layer, higher overhead, use briefly
sudo fileslower-bpfcc 10

If ext4slower is full of hundred-millisecond operations and biolatency shows everything completing in microseconds, you have your answer and it is not the disk. For NFS specifically, nfsiostat splits out RTT from execution time, which separates network cost from server cost. Install the tooling with bpfcc-tools on Debian and Ubuntu, or bcc-tools on RHEL, AlmaLinux and Rocky.

A working order for troubleshooting high I/O wait

On an unfamiliar box, this is the sequence I work through. Each step either finds the problem or eliminates a family.

1. Read /proc/pressure/io. If full is near zero, the stall is not hurting the whole system and you should widen the search before you tune storage.
2. Check /proc/pressure/memory at the same time. If it’s rising too, fix memory first.
3. Run iostat -xy 1 for thirty seconds. Note r_await, w_await and aqu-sz per device. Ignore %util.
4. List processes stuck in uninterruptible sleep: ps -eo state,pid,comm | awk '$1 ~ /^D/'. That’s your blocked set, and it names the workload.
5. Attribute the traffic with pidstat -d 1. Confirm the process you suspect is the process actually doing the I/O.
6. If await is high, decide saturated versus throttled. Check cgroup io.max, check provider volume metrics, check steal time.
7. If await is low but the application is slow, jump to the layer gap. Run biolatency and ext4slower together and compare.
8. Only now consider tuning: dirty ratios, I/O scheduler, flush settings, splitting volumes.

Common mistakes

• Reading iostat’s first report. It’s a since-boot average. Use -y, or discard the first block.
• Trusting %util on NVMe or cloud volumes. Parallel devices break the assumption it was built on.
• Alerting on a fixed %iowait threshold. It fires on healthy backups and stays silent during real stalls on busy hosts.
• Changing the I/O scheduler first. Going from mq-deadline to none on NVMe is sometimes right, but it’s a tuning step, not a diagnosis, and it fixes nothing if you’re throttled.
• Adding swap to fix swapping. More swap on a memory-starved box means more disk traffic, not less.
• Stopping at the block layer. If every storage metric looks clean and the app is still slow, you’re measuring at the wrong layer, not measuring a healthy system.
• Testing with the wrong benchmark. A sequential dd tells you nothing about a workload doing random 4K synchronous writes. Use fio and match the real access pattern.

Best practices worth the effort

• Record PSI continuously. Storage incidents are short and bursty. Without history you’re guessing about something that already ended.
• Capture a baseline while things are healthy. Save an iostat -xy sample and a PSI reading from a normal day. “Is 8ms bad?” is unanswerable without one.
• Keep sar running. The sysstat package can archive metrics on a schedule, which means sar -d can show you what happened at 3am. Enable it before you need it.
• Separate hot write paths. Database data, database logs and application logs on separate volumes stops one from starving the others.
• Cap container logs. An uncapped JSON log driver quietly filling the root volume is a recurring cause of host-wide stalls.
• Alert on latency and pressure, not utilisation. Users feel w_await and PSI full. Nobody has ever felt a percentage of device busy time.

Frequently asked questions

What is a normal or acceptable I/O wait percentage?

There isn’t one, and any article that gives you a single number is guessing. Iowait scales with how idle your CPUs are, so the same workload produces very different figures on a busy server and a quiet one. Judge against your own baseline, and prefer PSI, which measures lost time directly rather than as a share of idle.

Can high I/O wait be caused by something other than the disk?

Regularly. Memory pressure driving swap and page cache eviction is the most common non-disk cause. Network filesystems, hypervisor contention on shared hosts, and cgroup or cloud volume throttling all produce the same symptom. The block device being genuinely slow is one possibility among several.

Why does iostat show 100% utilisation when the disk isn’t busy?

Because %util measures the fraction of time at least one request was in flight, which made sense for devices that handled one request at a time. NVMe drives and cloud volumes process many requests in parallel, so they can be continuously occupied while nowhere near capacity. Read aqu-sz and the await columns instead.

How do I find which process is causing I/O wait?

Start with pidstat -d 1, which needs no special kernel settings. List processes in uninterruptible sleep with ps -eo state,pid,comm and filter for state D to see who is actually blocked. iotop -o gives a live view but needs kernel.task_delayacct=1 on recent kernels for its percentage columns to populate.

Does adding more RAM reduce I/O wait?

Often, yes, and it’s underrated. More RAM means a larger page cache, fewer reads reaching the device, and less swapping. If vmstat shows sustained swap activity or memory pressure tracks I/O pressure, RAM is likely the cheapest fix available.

Should I change the I/O scheduler to fix high I/O wait?

Only after you know the device is the bottleneck. Check the current setting with cat /sys/block/sda/queue/scheduler. On NVMe, none is a common and sensible default because the hardware already does the queuing. On spinning disks, mq-deadline or bfq give better latency behaviour. It’s a tuning knob, not a diagnostic, and it will not help if you’re being throttled.

Why is my application slow when iostat says storage is fine?

Because iostat only sees the block layer. Latency in NFS, FUSE, overlay filesystems, journal serialisation or filesystem locking is entirely invisible to it. Run ext4slower or xfsslower alongside biolatency: if the filesystem is slow and the device is fast, the gap between them is your problem.

The one thing worth remembering

High I/O wait is a symptom that points in a general direction, not a diagnosis. It’s a slice of idle time, averaged across cores, measured at one specific layer of the stack. It goes up when nothing is wrong and stays flat when something is very wrong.

Read PSI to find out whether anyone is actually suffering. Read r_await, w_await and aqu-sz to find out whether the device is the constraint. Compare filesystem latency against block latency to find out whether you’re even looking at the right layer. Do those three things and the answer usually falls out in a few minutes, instead of after a week of arguing with your hosting provider about a volume that was fine all along.

Need someone to look at your storage stalls?

I work with teams on exactly this kind of problem, usually the ones that have already survived a round of “the host says the disk is fine”. Things I can help with:

• Diagnosing a live or recurring I/O stall and identifying which failure family it belongs to
• Setting up PSI, sar and per-cgroup I/O metrics in Prometheus and Grafana, with alerts that fire on latency rather than utilisation
• Separating cloud volume throttling from genuine device saturation, and sizing storage correctly instead of guessing upward
• Tuning writeback, dirty ratios and I/O schedulers for a specific workload, with before-and-after measurements
• Tracing container and Kubernetes I/O contention through cgroup v2 limits, overlay filesystems and log drivers
• Benchmarking with fio against your real access pattern so capacity decisions rest on something other than a dd run

If you’ve got an iostat capture, a PSI reading or just a graph that doesn’t make sense, send it over and I’ll tell you what I see in it.

The post Troubleshooting High I/O Wait Without Blaming the Disk appeared first on John Nessime.

That is the real problem with trying to bootstrap a VPS with Ansible in a single run. The failures that cost you are not the ones that turn the recap red. They are the tasks that report changed, do nothing useful, and leave you believing the server is hardened when it is not.

This post covers the ordering that keeps you connected while you dismantle your own access, the silent failure modes that only surface on reboot, and how to make the second run of the playbook as safe as the first. The examples are Debian and Ubuntu flavoured with notes for RHEL-family images, and they assume a plain provider VPS from somewhere like InterServer, Hetzner, DigitalOcean or Vultr, not a pre-baked golden image.

What a one-run bootstrap actually has to survive

A bootstrap playbook is unusual because it modifies the thing it is standing on. Every other playbook you write assumes a working connection. This one has to change the user it connects as, the port it connects to, and the firewall that lets it connect at all, without dropping the run.

Four things break it, and they break it in different ways:

• The chicken-and-egg problem. Ansible needs Python on the target to run almost any module, and a minimal image may not have it.
• Lockout. You remove your own access before proving the replacement access works.
• Non-idempotent tasks. The first run is fine. The second run is where the damage happens.
• Missing collections. The playbook that worked on your laptop fails on the CI runner because half the modules do not ship with ansible-core.

Everything below is organised around those four, because that is how they actually show up.

The chicken-and-egg problem: no Python, no modules

Most modern cloud images ship Python 3 and this never comes up. Minimal Debian netinstall images and some stripped provider templates do not, and then your very first task fails with a message about the interpreter before it has done anything at all.

The escape hatch is ansible.builtin.raw, which pipes a command straight down the SSH connection without needing Python on the other end. Pair it with gather_facts: false, because fact gathering is itself a Python module and will fail before your bootstrap task ever runs.

- name: Make the host manageable
  hosts: new_servers
  gather_facts: false
  become: true
  tasks:
    - name: Check whether a Python interpreter exists
      ansible.builtin.raw: test -e /usr/bin/python3
      register: python_present
      changed_when: false
      failed_when: false

    - name: Install Python if it is missing
      ansible.builtin.raw: apt-get update && apt-get install -y python3
      when: python_present.rc != 0
      changed_when: true

    - name: Gather facts now that we can
      ansible.builtin.setup:

Two details worth understanding rather than copying. failed_when: false on the check turns a non-zero exit into data instead of a failure, so you can branch on it. And ansible.builtin.setup is the fact-gathering module called explicitly, which means from that point on ansible_distribution and friends are available even though the play started with gathering disabled.

If you are provisioning through a provider that supports user-data, doing this in cloud-init instead is cleaner. Let cloud-init handle first-boot truths, then let Ansible own everything that will change later.

Failure family one: locking yourself out while the playbook says ok

This is the one that actually costs a rebuild, and it is entirely an ordering problem. The tasks are all correct in isolation. Run them in the wrong sequence and the playbook cheerfully removes your way back in before it has confirmed the replacement works.

The order that survives

The rule is simple: nothing that removes access happens until something has proven the new access works. In practice:

1. Connect as whatever the provider handed you, usually root or a sudo-capable default user.
2. Create the admin account, add it to the sudo or wheel group, install its authorized key.
3. Write the sudoers drop-in and validate it with visudo -cf.
4. Prove you can log in as that account and escalate. This is the step everyone skips.
5. Install the firewall and allow SSH before enabling it.
6. Write the sshd configuration, validate it, restart, reconnect on the new port and confirm.
7. Only now disable root login and password authentication.

Step four is a real task, not a comment. It runs from the control node against the target, using the credentials you are about to depend on:

- name: Prove the new admin account can log in and escalate
  ansible.builtin.command: id -u
  become: true
  changed_when: false
  vars:
    ansible_user: "{{ admin_user }}"
    ansible_ssh_private_key_file: "{{ admin_key_path }}"

Setting connection variables at the task level means this single task uses the new identity while the rest of the play carries on as root. If the key is wrong or sudo is misconfigured, the play stops here with root login still intact, which is exactly what you want.

The socket activation trap

This is the failure that passes green and bites on reboot. On Ubuntu images from 22.10 onward, sshd is started through systemd socket activation rather than as a persistent service. ssh.socket owns the listening port, and Port in sshd_config is not what decides where the daemon listens.

So your playbook writes Port 2222, restarts the service, reports changed, and the box carries on listening on 22. Your existing session still works. Your firewall rule for 2222 looks fine. Nothing indicates a problem until something forces a reconnect.

The fix is to write a socket override as well, and to check which mechanism the image actually uses rather than assuming:

# Is the socket unit in play on this host?
systemctl is-enabled ssh.socket

# What is actually listening, and under which unit?
ss -tlnp | grep -E 'ssh|:22'

If ssh.socket is enabled, the override needs an empty ListenStream= first to clear the inherited default, then the real value. Miss the empty line and the host listens on both ports, which is worse than either outcome alone because your hardening looks applied and is not.

Honestly, the simpler answer for most single-server setups is to leave SSH on 22 and lean on key-only authentication plus a firewall source restriction. Moving the port cuts log noise from opportunistic scanners. It does not stop anyone who is actually looking at you.

Firewall ordering

UFW defaults to denying inbound traffic when you enable it. Enable first, add the SSH rule second, and the run dies mid-task with an UNREACHABLE that will not recover. Rules go in first, always:

- name: Allow SSH before the firewall is armed
  community.general.ufw:
    rule: allow
    port: "{{ ssh_port }}"
    proto: tcp

- name: Set default inbound policy
  community.general.ufw:
    direction: incoming
    policy: deny

- name: Enable UFW
  community.general.ufw:
    state: enabled

Two extra things bite here. Provider-level firewalls, the ones configured in the control panel rather than on the host, are a second layer that Ansible has no visibility into. If you open a port on the host and traffic still does not arrive, check there before debugging iptables. And if you moved the SSH port, the host firewall rule and the provider firewall rule both need updating, in that order, before the daemon restarts.

Failure family two: the second run is the dangerous one

A bootstrap playbook you run once is a shell script with extra syntax. The value only appears when you can run it again on the same host, in six months, and have it either do nothing or fix drift. That is idempotency, and the usual killers are shell and command tasks with no guard.

An unguarded shell task reports changed every single time, which means any handler it notifies fires every single time. That is how a playbook you thought was a no-op ends up restarting sshd on a production box during a routine drift check.

Guard them properly:

• creates: or removes: makes the task skip when the target state already exists. Prefer this.
• changed_when: false for read-only commands so they stop lying about changing things.
• when: driven by a registered check, for anything conditional.

The other idempotency trap is lineinfile against sshd_config. It works, and it is fragile in a specific way: the regex has to match the shipped file, and different distributions ship different defaults, commented or not. Writing a drop-in file is more predictable:

- name: Write the sshd hardening drop-in
  ansible.builtin.template:
    src: 99-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/99-hardening.conf
    owner: root
    group: root
    mode: "0600"
  notify: Restart sshd

- name: Validate the full sshd configuration
  ansible.builtin.command: sshd -t
  changed_when: false

Note that the validation runs against the whole configuration rather than the drop-in in isolation. A drop-in is not a valid standalone config file, so validate: on the template task with sshd -t -f %s will fail on a temporary file that has no host keys and no context. Run sshd -t as its own task afterwards instead, and let the handler restart only if it passes.

Two prerequisites for the drop-in approach. The base sshd_config needs an Include /etc/ssh/sshd_config.d/*.conf line, which recent Debian, Ubuntu and RHEL 9 images have but older ones may not. And in sshd configuration the first value for a keyword wins, so the include has to appear near the top of the file for your drop-in to override anything below it. Check both before relying on the mechanism.

Failure family three: the collections are not where you think

ansible-core ships the ansible.builtin namespace and essentially nothing else. The full ansible package bundles a large set of community collections, which is why a playbook that runs on your laptop can fail on a lean CI runner with a module-not-found error.

Things people assume are built in and are not: ufw and timezone live in community.general. authorized_key, sysctl and mount live in ansible.posix. Declare them and pin them:

# requirements.yml
collections:
  - name: community.general
  - name: ansible.posix

# install before the run, and in CI
ansible-galaxy collection install -r requirements.yml

Use fully qualified names everywhere. Short names still resolve, but they resolve through a lookup that can pick a different module when two collections define the same name. ansible-lint flags this and it is worth listening to.

Failure family four: secrets in the repository

Bootstrap playbooks accumulate secrets faster than anything else you write: initial passwords, API tokens for monitoring agents, TLS material, backup credentials. They also tend to be the first repository someone shares with a client.

Ansible Vault is the low-friction option and it is genuinely fine for a small estate. Encrypt the variable file, not the playbook, so diffs stay readable:

ansible-vault encrypt group_vars/all/vault.yml
ansible-playbook site.yml --ask-vault-pass

The honest trade-off: Vault gives you one shared passphrase with no rotation story and no audit trail. Once more than two people run the playbook, or a client needs their own credentials kept separate from yours, move to something with real access control. That is more moving parts, so do not reach for it on day one.

Putting it together: how to bootstrap a VPS with Ansible end to end

Structure matters less than order, but a shape that works well is a single site.yml calling roles in dependency order, with the risky access changes fenced off at the end.

# inventory.yml
new_servers:
  hosts:
    web01:
      ansible_host: 203.0.113.10
      ansible_user: root
      ansible_port: 22
  vars:
    admin_user: deploy
    ssh_port: 22

Then the run itself, in this order:

1. Reachability. Python check, fact gathering, timezone, hostname, package cache refresh.
2. Identity. Admin user, group membership, authorized key, sudoers drop-in with visudo -cf validation.
3. Verification gate. The task shown earlier. Nothing destructive runs before this passes.
4. Firewall. Rules first, policy second, enable third.
5. Unattended updates. unattended-upgrades on Debian and Ubuntu, dnf-automatic with its systemd timer on RHEL-family hosts.
6. Brute-force protection. fail2ban with your own /etc/fail2ban/jail.local, never edits to jail.conf.
7. Workload. Docker, a web server, a monitoring agent, whatever the box is for.
8. Access teardown. Disable root SSH login, disable password authentication, restart sshd, reconnect, confirm.

When step eight changes the port or user, the play needs to actually reconnect rather than reusing its cached SSH connection. Ansible keeps a persistent connection per host, so a config change alone will not be picked up:

- name: Point subsequent tasks at the new identity
  ansible.builtin.set_fact:
    ansible_user: "{{ admin_user }}"
    ansible_port: "{{ ssh_port }}"

- name: Drop the cached connection
  ansible.builtin.meta: reset_connection

- name: Confirm the host answers on the new settings
  ansible.builtin.wait_for_connection:
    timeout: 60

meta: reset_connection does not accept a when conditional, so it runs on every pass. That is a small cost for a play that works on the first run and every run after.

Test it before you point it at anything real

The cheapest test is a disposable VPS. Spin one up, run the playbook, reboot it, run the playbook again, then destroy it. The reboot is the important part, because that is what exposes the socket activation problem and any service you enabled but never actually started.

# catch YAML and structural errors instantly
ansible-playbook site.yml --syntax-check

# style and correctness, including FQCN and idempotency smells
ansible-lint

# show what would change without changing it
ansible-playbook -i inventory.yml site.yml --check --diff

One caveat on --check: it is not fully reliable on a bootstrap playbook. Tasks that depend on earlier tasks having run will fail or report nonsense in check mode, because the earlier change did not really happen. Treat a clean check run as a weak signal, not proof.

Troubleshooting

• UNREACHABLE on the very first task. Usually host key verification against a rebuilt host reusing an old IP. Remove the stale entry with ssh-keygen -R and re-add the correct one with ssh-keyscan. Disabling host key checking globally makes this go away and removes a real protection, so do not.
• UNREACHABLE immediately after a firewall task. You armed the firewall before allowing SSH. Recover through the provider console, not SSH.
• “Missing sudo password”. The sudoers drop-in is not in effect or has bad syntax. Validate with visudo -cf on the file itself before trusting it.
• Module not found under an FQCN. The collection is not installed on the machine running the playbook. Confirm with ansible-galaxy collection list.
• New group membership ignored. Adding a user to docker mid-play does not affect the already-open connection. Insert meta: reset_connection after the group change.
• fail2ban installed but not banning. On images without rsyslog, /var/log/auth.log may not exist, so a jail configured to read it never starts. Check fail2ban-client status and switch the jail to the systemd backend if the log file is absent.
• Everything green, port unchanged. Socket activation. Check systemctl is-enabled ssh.socket and ss -tlnp.

Common mistakes

• Disabling root login in the same play that creates the replacement user, with no verification step between them.
• Enabling UFW before the SSH allow rule exists.
• Assuming Port in sshd_config is authoritative on every distribution.
• Unguarded shell tasks that notify handlers, causing service restarts on every run.
• Editing jail.conf instead of creating jail.local, so the next package update silently reverts it.
• Committing plaintext secrets because “it is only a bootstrap playbook”.
• Testing on a host you cannot afford to lose, with no console access configured.
• Relying on --check as a substitute for a real run on a disposable host.

Best practices

• Know your recovery path before the first run. Provider console, serial console or recovery mode, tested once so you are not learning it under pressure.
• Put a verification task before every irreversible one, and let the play fail there rather than after.
• Prefer drop-in files over line edits for sshd, sudoers, fail2ban and sysctl. They survive package upgrades and are trivially removable.
• Validate configuration before restarting the service that reads it. sshd -t, visudo -cf, nginx -t.
• Pin collections in requirements.yml and install them in CI, not just locally.
• Run the playbook twice on a fresh host and read the second recap. Anything still reporting changed is a bug.
• Reboot once during testing. Half of the silent failures only appear after a restart.
• Get a monitoring agent on the box in the same run. A server that is hardened but unobserved is a server you will hear about from a customer first.

Frequently asked questions

Can you really bootstrap a VPS with Ansible in a single playbook run?

Yes, including changing the SSH user and port mid-run. The mechanism is set_fact to update the connection variables, meta: reset_connection to drop the cached connection, and wait_for_connection to confirm the new settings work before continuing. What you cannot do safely is change access and skip the confirmation.

Should I use cloud-init or Ansible for initial server setup?

Both, split by lifetime. cloud-init handles things that are true exactly once at first boot: the admin user, the SSH key, a minimal package set. Ansible handles everything that will be tuned again later. Putting hardening rules in cloud-init means the only way to change them is to rebuild the host.

Do I need Terraform or OpenTofu as well?

Only if you are creating the infrastructure, not just configuring it. Terraform and OpenTofu create the VPS, the DNS records and the provider firewall. Ansible configures what runs inside. For a handful of long-lived servers ordered through a provider panel, Ansible alone is enough and adding a second tool costs more than it returns.

Why does my playbook work the first time and fail the second?

Almost always because the first run changed the connection parameters and the inventory still describes the original ones. Either update the inventory to the post-bootstrap values and make the play tolerate both, or split bootstrap and steady-state into separate playbooks with separate inventory entries.

Is changing the SSH port worth doing?

It reduces log noise from untargeted scanning, which makes real events easier to spot. It is not a security control on its own. Key-only authentication, a default-deny firewall and fail2ban do the actual work. If moving the port complicates your automation, the honest answer is to leave it on 22.

How do I recover if the playbook locks me out?

Through the provider’s console or recovery environment, which is why you should know where it is before you need it. Most VPS providers offer a web-based console, some offer a rescue boot with the disk mounted. From there, revert the offending drop-in file or firewall rule, restart the service, and reconnect. Then add the verification step the playbook was missing.

Can this run against several servers at once?

Yes, but use serial: 1 the first few times. If there is an ordering bug in the play, running it against one host at a time means you lock yourself out of one server instead of a fleet.

The one thing worth remembering

When you bootstrap a VPS with Ansible, the individual tasks are the easy part. Creating a user, writing a firewall rule and hardening sshd are all well-documented and hard to get wrong in isolation. What decides whether the run leaves you with a working server or a rebuild is the sequence, and specifically whether you verify new access before removing old access.

Add the verification gate. Run the playbook twice. Reboot the box once. Those three habits catch nearly every silent failure described here, and they cost about ten minutes on a disposable host.

Need a bootstrap playbook you can actually rerun?

Most of the provisioning work I take on starts with a playbook that worked once and has been too frightening to run since. Things I help with:

• Building a bootstrap role that takes a blank VPS to a hardened, monitored server in one run, with a verification gate before every irreversible step.
• Auditing an existing playbook for idempotency, so the second and hundredth runs are as safe as the first.
• Untangling SSH, firewall and socket activation interactions on mixed Debian, Ubuntu and RHEL-family estates.
• Splitting cloud-init and Ansible responsibilities cleanly so hardening stays changeable after first boot.
• Moving secrets out of a repository into Vault or a managed secret store without breaking existing runs.
• Wiring the playbook into CI with linting, collection pinning and a disposable-host test that runs on every change.

Send me the playbook, a recap from a second run, or the output of ss -tlnp from the host that is behaving oddly, and I will tell you what I would change first.

The post Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out appeared first on John Nessime.

That framing is where most of these arguments go sideways. On one box serving a normal website, all three of these servers will handle your traffic without breaking a sweat. The web server is almost never the slow part. The difference shows up months later, at 2am, in who gets paged and why.

So this post compares Nginx vs Apache vs Caddy on the three things that genuinely change your on-call life: how each one behaves when it runs out of capacity, who owns the certificate lifecycle, and where configuration lives when someone other than you needs to change it. Then a decision procedure, the mistakes I keep running into, and an FAQ.

Why the benchmark comparison picks the wrong winner

Almost every throughput comparison you’ll find measures static file serving on an idle machine with no application behind it. That is a real measurement of a thing that is not your bottleneck. If you’re proxying to PHP-FPM, Gunicorn, or a Node process, the request spends milliseconds in the web server and everything else in your code and your database.

The honest exception: if you are running a CDN origin, terminating TLS at a high connection rate, or serving genuinely large volumes of static assets, the connection handling model does show up in your bill. That is a real workload and Nginx has earned its reputation there. It just isn’t most projects.

For everyone else, the useful question is not “which is fastest” but “which chores am I choosing to keep.” That’s what the rest of this is about.

Failure family one: the process model decides how you fall over

This is the invisible one. Nobody notices it during a migration and everybody notices it during a traffic spike.

Apache: a hard worker ceiling, and a queue behind it

Apache’s behaviour depends entirely on which Multi-Processing Module is loaded. With prefork, you get one process per connection, capped by MaxRequestWorkers. When you hit that cap, new connections don’t get rejected. They sit in the kernel’s listen backlog waiting for a worker to free up.

The symptom is the worst kind: the site “hangs.” Load average is unremarkable. The application log is quiet because the requests never reached the application. Your uptime check times out but a manual curl from the box works fine. People spend hours looking at the database.

Check which MPM you’re actually running before you theorise about anything:

# RHEL, AlmaLinux, Rocky
httpd -V | grep -i "Server MPM"

# Debian and Ubuntu
apache2ctl -V | grep -i "Server MPM"

The reason prefork survives is mod_php. It isn’t safe with Apache’s threaded MPMs, so PHP stacks historically pinned themselves to prefork and inherited its concurrency profile. The Apache project itself now points people at PHP-FPM over mod_proxy_fcgi instead, which frees you to run mpm_event and gets you HTTP/2 as a side effect, since mod_http2 disables itself under prefork.

If you inherit an Apache box, that migration is usually the single highest-value change available to you, and it does not require leaving Apache.

Nginx: cheap connections, different limits

Nginx runs a small number of worker processes, each with an event loop. An idle keepalive connection costs a file descriptor and a little memory, not a process. Ten thousand of them is not a crisis.

Its ceiling is worker_processes multiplied by worker_connections, plus whatever your file descriptor limit allows. When you hit it, Nginx says so plainly in the error log rather than silently queueing. That’s a genuine operational advantage: the failure announces itself.

The trade is that when Nginx is proxying, saturation moves downstream. You stop seeing worker exhaustion and start seeing 502s and 504s from an overloaded PHP-FPM pool or app server. The web server is fine; you now have to go find the real limit.

Caddy: goroutines, and a softer edge

Caddy is a Go binary and handles connections with goroutines that the runtime multiplexes onto OS threads. Practically, that sits between the other two: connections are cheap like Nginx, but you don’t tune a worker count at all.

The failure mode is correspondingly softer and, honestly, harder to reason about. Instead of a clean “workers exhausted” line, you get memory growth and garbage collection pressure. There’s no knob you turned wrong, which is pleasant right up until you need to explain what happened.

Failure family two: who owns the certificate lifecycle

This is the one that actually causes outages, and it is the sharpest genuine difference between the three.

With Nginx or Apache, certificate management is a separate moving part. Certbot or another ACME client writes new files to disk on a timer, and the running server keeps serving whatever it loaded at startup. If nothing tells it to reload, the certificate on disk is valid and the certificate on the wire is expired. Users see a browser interstitial while your monitoring, if it checks the filesystem, reports everything fine.

The fix is to attach the reload to the renewal itself, not to a separate cron entry that might drift:

# Run the renewal end to end without touching real certificates.
# This exercises the challenge and the hook, which is the point.
certbot renew --dry-run

# Attach the reload to renewal so it only fires when a cert actually changed
certbot renew --deploy-hook "systemctl reload nginx"

The distinction that matters: --deploy-hook runs only when a certificate was actually renewed, so you’re not reloading the server twice a day for nothing. A blanket post-hook reloads regardless.

Caddy removes this category of failure. Certificate issuance and renewal are built into the server itself, it swaps the certificate in memory when it renews, and there is no external hook to forget. Point several instances at the same storage backend and they coordinate rather than each hammering the CA. If you have ever been paged for an expired certificate, this alone is a serious argument.

It is not free of foot-guns, though. On-demand TLS, where Caddy obtains a certificate during the handshake for a hostname it has never seen, is genuinely useful for multi-tenant setups where customers point their own domains at you. Enabled without restriction, it is an open door: anyone who points a DNS record at your IP can make you request a certificate. Caddy’s own documentation is blunt that it must be both enabled and restricted, and the mechanism is an ask endpoint you host:

{
    on_demand_tls {
        ask http://localhost:9123/ask
    }
}

https:// {
    tls {
        on_demand
    }
    reverse_proxy 127.0.0.1:3000
}

Caddy asks that endpoint whether a hostname is allowed before it will issue. It should be a fast indexed lookup against your own tenant table, returning in milliseconds, because a TLS handshake is being held open while it answers. The older interval and burst rate-limiting options are no longer recommended and should be removed if you still have them in a config.

One more Caddy default worth knowing: the admin API listens on localhost port 2019 and accepts a full config replacement without authentication. That is fine bound to loopback and catastrophic if you helpfully expose it while debugging a container.

Failure family three: where the configuration lives

This is the axis people underweight, and it’s usually the one that actually decides the answer.

Apache supports .htaccess: per-directory configuration, read at request time, changeable by anything that can write to the directory. No reload, no root access. That is why the entire shared hosting world is built on Apache, and why control panels like cPanel and DirectAdmin, and hosts like InterServer, assume it. A WordPress plugin can rewrite its own rules and it just works.

The costs are real and they’re the mirror image of the benefit. With overrides enabled, Apache walks the directory tree looking for .htaccess files on every single request. And your security rules can be silently overridden by whatever gets deployed into a directory, by anyone who can deploy.

Nginx has no equivalent and this is deliberate. Everything is central, and a change requires a reload. Rewrites are handled with try_files rather than per-directory rule files. If you own the box, that’s a feature: nothing changes routing behind your back and the running configuration is exactly what’s in version control. If you host other people’s applications, it’s a support burden, because every customer rewrite rule becomes a ticket for you.

Caddy is central too, either a Caddyfile or JSON. The difference is that config changes can go through the admin API as an atomic swap rather than a file edit plus a signal, which suits automated pipelines nicely.

Nginx: where it wins and where it doesn’t

Where it wins: high connection counts on modest hardware, static and cached content, TLS termination in front of an application fleet, and the sheer weight of the ecosystem. Every stack you’ll meet has a known-good Nginx config, and everyone you hire has already read one.

Where it doesn’t: the DNS behaviour catches people constantly. A hostname written directly into proxy_pass, or into an upstream block, is resolved once at startup and cached for the lifetime of the process. When your upstream is a cloud load balancer or a container that moves, Nginx keeps sending traffic to an IP that stopped being yours. Nothing in the log says “stale DNS”; you just get connection timeouts to an address that looks plausible.

Forcing re-resolution means putting the name in a variable and giving Nginx a resolver, because that’s what moves the lookup from config parse time to request time:

location / {
    # resolver moves lookups to runtime; valid= caps how long a result is reused
    resolver 127.0.0.53 valid=30s ipv6=off;

    # the variable is the trick: it defers evaluation to per-request
    set $upstream_host app.internal.example.com;
    proxy_pass http://$upstream_host:8080$request_uri;
}

Note the $request_uri on the end. Once a variable appears in proxy_pass, Nginx stops doing its usual URI rewriting for you, so you have to pass the path explicitly. Miss that and every request lands on the upstream’s root. Also worth knowing: an upstream block defined elsewhere with the same name will take precedence over the runtime lookup, which produces a very confusing afternoon.

The other honest downside is governance. Nginx is owned by F5, and a longtime core developer forked it to freenginx after a dispute over how security issues were being handled, with an earlier fork, Angie, already in existence. Mainline Nginx remains overwhelmingly dominant and I would not choose a fork for a client project today. But “who decides what ships” is a fair question to ask about any dependency, and this one now has a public answer.

Apache: where it wins and where it doesn’t

Where it wins: anywhere applications need to configure the web server without touching the web server. Shared hosting, multi-tenant WordPress, legacy applications that ship their own rewrite rules. The module ecosystem is enormous and modules load at runtime rather than requiring a rebuild. And the documentation is, quietly, the best of the three: nearly every directive has a page explaining its interactions.

Where it doesn’t: memory per connection, if you’re still on prefork. Slow clients holding processes open is the specific scenario that hurts, because a process that’s waiting on a phone on bad mobile data is a process nobody else can use. Putting a CDN or a proxy in front absorbs a lot of that, which is a large part of why Apache origins behind Cloudflare behave so much better than bare ones.

I would not migrate a working Apache installation to Nginx purely for performance. I would move it from prefork and mod_php to mpm_event and PHP-FPM first, then measure, and quite often stop there.

Caddy: where it wins and where it doesn’t

Where it wins: anything where certificate management is the operational risk. Small teams without a dedicated ops person, internal tooling with a long tail of hostnames nobody tracks, multi-tenant products serving customer-owned domains, container stacks where a single static binary with no runtime dependencies is a real simplification. A working HTTPS reverse proxy is genuinely a few lines.

app.example.com {
    encode zstd gzip
    reverse_proxy app:3000
}

That obtains a certificate, redirects HTTP to HTTPS, negotiates compression and proxies, with no ACME client, no cron, no reload hook.

Where it doesn’t: the ecosystem gap is real. When something unusual breaks, there are fewer Stack Overflow answers and fewer engineers who have seen it before. Adding a plugin means rebuilding the binary with xcaddy rather than dropping in a module, which is fine in CI and annoying on a box you SSH into. And there’s a subtler cost: automation you don’t understand is still automation you’re responsible for. Being paged about ACME behaviour you’ve never had to learn is a worse afternoon than being paged about Certbot.

On upstream DNS, Caddy dials static upstreams by name so resolution happens at dial time rather than being frozen at config load. When the set of addresses changes, the dynamic a upstream module with a refresh interval is the built-in answer, which is a cleaner story than Nginx’s variable trick.

A decision procedure for Nginx vs Apache vs Caddy

Work down this list and stop at the first one that matches. It’ll be right more often than a benchmark.

1. Do applications you don’t control need to change server behaviour? If yes, Apache. Nothing else has .htaccess, and reimplementing it in tickets is not a plan.
2. Is TLS renewal your top operational risk? Small team, no dedicated ops, lots of hostnames, or customer-owned domains: Caddy. It deletes a whole class of incident.
3. Are you terminating a very high connection rate, or serving heavy static content? Nginx. This is the workload it was designed around.
4. Does the team already run one of these well? Keep it. Operational familiarity beats a 20% throughput difference you were never going to measure.
5. Is there a working Apache install and no specific complaint? Fix the MPM and PHP handler, then reassess. Most “we need Nginx” tickets are actually “we need to stop using prefork.”
6. None of the above? Nginx for anything that will be handed to another engineer. Caddy for anything you’ll run yourself.

Common mistakes

• Renewing certificates without reloading. The single most common self-inflicted TLS outage on Nginx and Apache. Use a deploy hook and verify it with a dry run.
• Migrating for benchmark numbers. You inherit a new failure mode and a team that has never debugged it, in exchange for latency nobody could perceive.
• Leaving Apache on prefork after moving to PHP-FPM. The migration is only half done, and you’re still paying prefork’s memory cost while also giving up HTTP/2.
• Hostnames baked into proxy_pass with dynamic upstreams. Works perfectly until the upstream IP changes, then fails in a way that looks like a network problem.
• Enabling Caddy’s on-demand TLS without an ask endpoint. You’ve handed strangers the ability to make you request certificates.
• Exposing Caddy’s admin API beyond loopback. Unauthenticated full config replacement is exactly as bad as it sounds.
• Monitoring the certificate file instead of the handshake. The file is almost never the thing that’s wrong. Check what the server actually presents, from outside the box.

How I’d decide, in practice

• Pick based on who has to change the config six months from now, not on requests per second today.
• Whichever you choose, dump the effective configuration rather than reading files: nginx -T, apachectl -S, caddy adapt --config /etc/caddy/Caddyfile. Include files hide surprises.
• Monitor TLS expiry as an external handshake check from a service like Better Stack or a Grafana Cloud synthetic, not as a filesystem check on the box.
• Put something in front. A CDN like Cloudflare or Fastly absorbs slow clients and connection floods, which flattens most of the architectural differences here.
• Run one server type per environment. Mixing Nginx in staging and Apache in production means your rewrite rules are untested by definition.
• Validate before reloading, every time. nginx -t, apachectl configtest and caddy validate all exist so you don’t find out from users.

Frequently asked questions

Is Caddy production ready?

Yes. It’s been running real workloads for years and its certificate automation removes more operational risk than its smaller ecosystem adds, for most teams. The caveat is support surface, not stability: when you hit something unusual, there are fewer people who have hit it before you.

Can I run Nginx and Apache together?

Yes, and it’s a common pattern: Nginx on 80 and 443 handling TLS and static files, proxying dynamic requests to Apache on a local port so applications keep their .htaccess behaviour. You get Nginx’s connection handling and Apache’s per-directory config. The cost is two servers to patch, two logs to correlate, and real client IP to forward correctly.

Which is best for WordPress?

Apache if plugins need to write rewrite rules, which on a typical multi-plugin site they do. Nginx if you control the site and are willing to translate those rules into try_files and static location blocks once. Plenty of high-traffic WordPress runs on Nginx; it just means plugin-generated rules are your job now.

Does Nginx really re-resolve DNS for upstreams?

Not by default. A literal hostname in proxy_pass or an upstream block is resolved at startup and reused indefinitely, regardless of TTL. You get runtime resolution by using a variable together with a resolver directive, and remembering that this changes how the request URI is passed.

Should I switch from mod_php to PHP-FPM?

In almost all cases yes. It’s the change that unlocks Apache’s threaded MPMs and HTTP/2, and it separates PHP’s lifecycle from the web server’s so you can restart one without the other. Check first for anything relying on PHP settings in Apache config or .htaccess, since those move to a .user.ini file with different syntax.

Does the freenginx fork mean I should stop using Nginx?

No. Mainline Nginx under F5 remains the overwhelmingly dominant option with the broadest packaging and support. The fork is worth knowing about as context on project governance rather than as a migration target, and I wouldn’t put a client on a fork without a specific reason.

Which uses the least memory?

Nginx, generally, with Caddy close behind and Apache furthest away when running prefork. But the gap narrows sharply once Apache is on mpm_event, and on a VPS from a provider like Hetzner or DigitalOcean the memory your application uses will dwarf all three. Measure your own stack before treating this as a deciding factor.

The one thing worth remembering

The Nginx vs Apache vs Caddy question is not a performance question. All three are fast enough for what you’re doing. What differs is which operational chores each one hands back to you: Apache gives applications control at the cost of memory and predictability, Nginx gives you central control at the cost of owning every rewrite rule and the certificate reload, and Caddy takes the certificate lifecycle off your plate at the cost of a smaller ecosystem when things get strange.

Pick the set of chores your team can actually carry. Then go fix the thing that’s genuinely slow, which is almost certainly not the web server.

Need a second opinion on your web server setup?

Most of the work I do here is unglamorous and pays for itself quickly. Things I can help with:

• Auditing an existing Nginx, Apache or Caddy config and telling you what will break under load before it does
• Migrating Apache from prefork and mod_php to mpm_event with PHP-FPM, without breaking existing .htaccess behaviour
• Translating a working Apache rewrite set into an Nginx configuration, with a test plan so you know it matches
• Fixing certificate renewal that runs but never reloads, including deploy hooks and external expiry monitoring
• Setting up Caddy for multi-tenant or customer-domain hosting with on-demand TLS properly restricted
• Diagnosing intermittent 502s, 504s and stale-upstream timeouts that look like network faults and aren’t

Send me your config, an error log excerpt, or the output of nginx -T and tell me what you’re seeing. I’d rather look at the real thing than guess from a description.

The post Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects appeared first on John Nessime.

That is the failure that actually bites when you provision servers by hand for a handful of clients. It is invisible for months. Nothing alerts. Nothing breaks. Then a disk dies, or a client asks for a second region, or you want to hand the whole thing over, and the gap between “the server works” and “the server is reproducible” turns into a weekend.

This post covers how to automate server provisioning for new clients in a way that survives that moment: what to put in the image, what to put in first boot, what to leave to configuration management, how to keep one client’s credentials from ever touching another client’s box, and how to prove a build is finished instead of assuming it.

The four failure families worth designing against

Automation is not the goal. Automation is what you reach for once you have decided which of these you refuse to keep living with. Pick the ones that apply to you and let them shape the build, because trying to solve all four at once is how provisioning projects stall.

1. You cannot rebuild it

The test is blunt: destroy the machine and rebuild it from your repository, with no shell access to the old one. If you cannot do that, you do not have a server, you have an artefact. Every hand-run apt install, every file edited over SSH at 1am, every “I’ll document that later” is a piece of the machine that exists nowhere but the machine.

This is the one that stays invisible longest. A hand-built box can run happily for two years. The cost lands entirely in one bad afternoon.

2. Every client is a slightly different server

Client A got fail2ban. Client B got it two months later with different jail settings. Client C never got it because that build happened on a Friday. Now a CVE lands in a package you installed on four of six boxes and you have to go look at each one to find out.

Drift is not just untidy. It destroys your ability to reason about the fleet as a fleet. Any statement that starts “all our clients have…” becomes a guess.

3. One credential, too much blast radius

This one is specific to consulting and it is the one people get wrong most often. If the same SSH key opens every client’s server, and the same API token provisions every client’s infrastructure, then a compromise of your laptop is a compromise of everybody. You are a shared dependency with none of the controls a shared dependency is supposed to have.

Worse, it is contractually awkward. Plenty of client agreements have language about access control that a single reused key quietly violates.

4. The handover you cannot perform

Engagements end. When one does, the client should be able to take a repository, run it, and get their infrastructure. If the only way to reproduce their environment is to hire you again, that is not leverage, it is a liability. It also makes you slower to say yes to the next client, because your capacity is tied up in machines only you understand.

How to automate server provisioning for new clients

The stack that holds up in practice has four layers, and the important part is the boundary between them. Most broken provisioning setups I see are broken because someone pushed work into the wrong layer.

Layer 1: the machine itself

Terraform or OpenTofu talks to your provider’s API and creates the server, the firewall rules, the volumes and the DNS records. Whether you run a VPS at InterServer, Hetzner, Vultr or DigitalOcean, or full cloud on AWS, the shape is the same: a declarative file that says what should exist.

Worth knowing before you commit: HashiCorp moved Terraform from the MPL to the Business Source License, which is source-available rather than open source, and the community forked the last MPL release as OpenTofu under the Linux Foundation. For an independent engineer building client infrastructure, either is workable. Terraform has the larger provider and module ecosystem and the commercial backing. OpenTofu gives you a licence that will not be renegotiated under you, which matters more when the code you write ends up in a client’s repository and out of your control. The HCL, provider protocol and state model are shared, so the switching cost is low in both directions. I lean OpenTofu for client work for the licence certainty, and that is a judgement call, not a technical verdict.

Keep state remote and locked from day one, and keep one state file per client. Shared state is the fastest way to turn a mistake on one engagement into an outage on another.

Layer 2: first boot with cloud-init

cloud-init runs once, on first boot, before you have any way in. Its job is narrow: create the account you will connect as, install its key, lock down password and root login, and get out of the way. Everything else belongs in the next layer.

#cloud-config
hostname: web01
fqdn: web01.acme.internal
timezone: Etc/UTC

users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz... provisioning key for this client only

# Refuse passwords and direct root logins from the very first boot,
# so there is no window where the box is reachable with weaker settings.
ssh_pwauth: false
disable_root: true

package_update: true
packages:
  - python3
  - sudo

Two deliberate omissions. There is no package_upgrade: true, because a full upgrade at first boot can run for minutes, holds the package lock, and is the single most common cause of the race described in the troubleshooting section below. Do the upgrade in Ansible where you can see it fail. And there is no application setup, because anything in runcmd is a script you will never test again.

python3 is there so Ansible has an interpreter to work with. Some images ship it already; installing it is cheap insurance.

Validate the file before it ever reaches a provider, because a YAML error in user-data fails silently and you find out by not being able to log in:

cloud-init schema --config-file user-data.yaml

Layer 3: convergence with Ansible

This is where the actual build lives: packages, users, firewall, TLS, web server, database, backup agent, monitoring agent, log shipping. Ansible is agentless and runs over SSH, which fits consulting well because you are not asking a client to accept a permanent daemon on their box.

The layout that scales across clients separates the shared build from the per-client differences:

infra/
  roles/                  # shared, versioned, the same for everyone
    base/
    hardening/
    web/
    backup/
  clients/
    acme/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
    globex/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
  site.yml

Roles are the product. Client directories are configuration. If you find yourself copying a role into a client directory and editing it, that is a variable you have not extracted yet.

Hand the inventory over from Terraform rather than maintaining it by hand. The simplest reliable version reads the outputs as JSON:

cd clients/acme/terraform

tofu output -json client_hosts 
  | jq -r '.value[] | "(.name) ansible_host=(.ipv4)"' 
  > ../inventory/hosts.ini

There are richer options, including inventory plugins that read Terraform state directly, but a generated file has a property worth keeping: you can read it, diff it, and commit it. When a provisioning run targets the wrong host, that diff is how you find out.

Layer 4: the verification gate

This is the layer almost everyone skips, and it is the one that turns provisioning from a script into a process. A playbook that finishes without errors has told you that no task raised an exception. It has not told you the server works.

Write a separate play whose only job is to assert the finished state, and run it as the last step of every provision:

- name: Verify the host is genuinely ready
  hosts: all
  gather_facts: true
  tasks:
    # raw, not command: this may run before Python is guaranteed present
    - name: Wait for cloud-init to finish
      ansible.builtin.raw: cloud-init status --wait
      changed_when: false

    - name: Collect service state
      ansible.builtin.service_facts:

    - name: Core services are running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services[ssh_unit].state == 'running'
        fail_msg: "Core services are not up on {{ inventory_hostname }}"

    - name: Site answers over TLS
      ansible.builtin.uri:
        url: "https://{{ client_domain }}/"
        status_code: 200
      delegate_to: localhost

Note ssh_unit as a variable rather than a literal: the SSH unit is ssh.service on Debian and Ubuntu but sshd.service on RHEL-family systems. Hard-coding it is how a verification play passes on one distro and fails confusingly on another.

The point of the gate is not that it catches exotic bugs. It is that it catches the ordinary ones: the service that installed but did not enable, the certificate that issued but did not get loaded, the firewall rule that opened the wrong port.

Keeping clients apart: secrets and access

Generate a fresh SSH keypair per client engagement. It costs you thirty seconds and it means revoking access to one client’s infrastructure is a single, contained action.

For secrets in the repository you have two credible options. Ansible Vault is built in and needs nothing extra, but it encrypts whole files, so your diffs become meaningless blobs and rotating a key across several clients is tedious. SOPS encrypts values while leaving keys readable, which means a code review can still see that a variable changed without seeing what it changed to, and it supports age, PGP and the cloud KMS services. For a repo that a client might eventually inherit, that readability is worth the extra tool.

# .sops.yaml at the repo root: different key per client
creation_rules:
  - path_regex: clients/acme/secrets/.*.ya?ml$
    age: age1acme...
  - path_regex: clients/globex/secrets/.*.ya?ml$
    age: age1globex...

Separate keys per client is the whole point. One shared key and you are back to a single blast radius with extra ceremony.

For handing the initial credentials to a client, use something with an audit trail and an expiry. A shared vault item in 1Password or Bitwarden is fine. Email and chat are not, because the secret then lives in two message histories neither of you controls.

If you want ongoing access without exposing SSH to the internet at all, an overlay network like Tailscale installed by the base role is a clean way to do it. The provisioning path still uses a public address for the first run, then the firewall closes it.

Troubleshooting a provisioning run

Ansible connects before cloud-init has finished

The most common failure by a wide margin. SSH comes up before cloud-init is done, Ansible connects, and then the first package task fails because cloud-init still holds the package manager lock. The error points at apt or dnf, which sends you looking in entirely the wrong place.

Fix it by blocking explicitly at the top of the first play, as in the verification snippet above. Check the state manually with:

cloud-init status --wait --long
sudo grep -i Traceback /var/log/cloud-init.log

One caveat that catches people: never put cloud-init status --wait inside a runcmd or bootcmd block. It will deadlock waiting for the process that is running it.

cloud-init appears to have done nothing

Two usual causes. Either the user-data was malformed, in which case cloud-init skipped it rather than erroring loudly, or the modules already ran on a previous boot and the semaphore files in /var/lib/cloud/instance/sem/ are preventing a rerun.

# what actually ran, and how long each stage took
cloud-init analyze show

# the human-readable output of scripts and package installs
sudo less /var/log/cloud-init-output.log

# on a throwaway test box only: wipe state and start over
sudo cloud-init clean --logs --reboot

Never run cloud-init clean on a production host. It removes cached state and the following boot will behave as a first boot.

Host key verification failed on a rebuilt server

Rebuild a machine on the same IP and every client with the old key in known_hosts refuses to connect. The tempting fix is to disable host key checking in ansible.cfg. Do not. That turns a real integrity check into a permanent no-op across every client you manage.

Remove the stale entry and pin the new one as part of the provisioning flow:

ssh-keygen -R 203.0.113.10
ssh-keyscan -H 203.0.113.10 >> ~/.ssh/known_hosts

Better still, if your provider exposes the host key fingerprint through its API or console output, compare it before you accept. That closes the trust-on-first-use gap properly.

The playbook reports changes on every single run

Almost always a shell or command task with no creates, changed_when or module equivalent. Ansible cannot know whether a raw command changed anything, so it assumes yes. Real changed counts are what let you spot drift, and a playbook that always reports twelve changes tells you nothing.

Run against an already-provisioned host in check mode. Anything that reports a change is either genuine drift or a task you need to fix:

ansible-playbook -i clients/acme/inventory/hosts.ini site.yml --check --diff

Common mistakes

• Putting the application build in runcmd. It runs once, never gets tested, and is invisible to every later change.
• One shared SSH key and one shared API token across all clients. Convenient right up to the moment it is not.
• Copying a role into a client directory to make one small change, instead of extracting a variable.
• Treating “the playbook exited zero” as proof the server is ready. It is proof no task threw.
• Local Terraform state on your laptop. It is a single point of failure that also cannot be handed over.
• Provisioning the server but not the DNS, TLS, monitoring and backups. A box nobody watches and nobody can restore is not finished.
• Automating the build and then fixing production by hand. One SSH session and you are back to a snowflake.

Best practices

• Rebuild before you need to. Destroy and rebuild a client’s staging host on a schedule. It is the only honest test of reproducibility.
• Version your roles and pin them per client. A shared role that everyone tracks on main means one commit changes six clients at once.
• Keep first boot minimal. If it can wait for Ansible, it should.
• Make monitoring part of provisioning. Whether that is Prometheus you run yourself, Grafana Cloud, or a lighter uptime service, the agent goes on in the base role, not as a follow-up ticket.
• Same for backups. An offsite target such as Backblaze B2 configured at build time, plus a restore test, or you have a backup you have never proven.
• Separate credentials per client, always. Keys, tokens, encryption keys, state backends.
• Write the runbook as you build. The provisioning repo’s README is the handover document. If it is accurate, the engagement can end cleanly.

Frequently asked questions

Do I need Terraform if I only manage a few VPS instances?

Not necessarily. If you are creating three servers a year through a provider’s control panel, the honest answer is that Terraform adds a state file to look after for very little return. Start with cloud-init and Ansible, which give you most of the reproducibility. Add Terraform or OpenTofu when the infrastructure around the server, such as firewalls, volumes, load balancers and DNS records, starts to be the part you keep getting wrong.

Should I build custom images instead of provisioning at boot?

Golden images cut boot time and remove dependency on package mirrors being up during a build. They also add a pipeline you have to maintain and rebuild for every security update. For client work at small scale, provisioning from a stock distribution image is usually the better trade, because the build stays readable and there is no stale image to forget about. Images earn their cost when you are launching machines often enough that boot time matters.

Terraform or OpenTofu for client projects?

Both work, and the configuration language is shared, so this is not a lock-in decision in the way it looks. Terraform has the deeper ecosystem and the commercial support path. OpenTofu is under the Linux Foundation with a licence that is unlikely to change, which is the stronger argument when the code will live in a client repository long after you have moved on. Ask the client whether they have a standard before you pick.

How do I automate server provisioning without exposing SSH to the internet?

The first run generally needs a reachable address, so scope the firewall rule to your own source IP for the duration of provisioning and close it in the same run. After that, reach the host through a bastion or an overlay network installed during the base role. Provider firewalls and host firewalls should both be in the automation, because relying on only one of them means a provider console change can silently open the box.

How do I stop configuration drift once the server is live?

Run the playbook in check mode on a schedule and look at what it reports. Any change on a host you have not touched is drift, and it is far cheaper to find it that way than during an incident. This only works if your tasks are properly idempotent, which is why the always-reports-changed problem above matters more than it looks.

What belongs in cloud-init versus Ansible?

cloud-init handles what must exist before you can connect: hostname, the login account, its SSH key, disabled password authentication, and a Python interpreter. Ansible handles everything you will ever want to change again. The dividing line is repeatability. cloud-init runs once and you cannot re-run it safely on a live host, so anything you might revisit belongs on the Ansible side.

How long should provisioning a new client server take?

Wall-clock time is the wrong measure. The useful measure is how much of it needs you. A build that takes twenty minutes but runs unattended from one command is in far better shape than a five-minute build with four manual steps, because the manual steps are where the divergence between clients gets introduced.

The one thing worth remembering

When you automate server provisioning for new clients, the deliverable is not the script. It is the ability to answer “can you rebuild this?” with yes, on any day, without opening a shell on the existing machine.

Everything else follows from that. Reproducibility gives you consistency across the fleet, because there is one definition. It gives you a clean handover, because the repository is the environment. It gives you the ability to take the next client on without dreading the build. And it costs the same afternoon that a single unplanned rebuild would have cost you anyway, just spent on your own schedule instead of at 2am.

Start with one client. Get their staging box rebuilding from scratch. Then make the second client’s build reuse the roles from the first. That is the whole path.

Need a provisioning setup you can actually hand over?

Most of the provisioning work I take on is not greenfield. It is a set of hand-built servers that work fine and nobody can reproduce. Things I can help with:

• Turning existing hand-built client servers into an Ansible role set, without a rebuild and without downtime
• Building a client onboarding repository: Terraform or OpenTofu, cloud-init, roles, per-client inventory and secrets
• Splitting shared credentials into per-client keys, tokens and state backends, with a revocation path that works
• Writing the verification playbook that decides whether a build is finished, and wiring it into CI
• Fixing idempotency so check-mode runs become a usable drift report instead of noise
• Preparing an environment for handover: runbook, access review, and a documented rebuild that the client can run themselves

If you want a second opinion on something specific, send the cloud-config, the playbook output, or a check-mode diff you cannot explain, and I will tell you what I see.

The post Automating Server Provisioning for New Clients Without Building a Snowflake Farm appeared first on John Nessime.

Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker’s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.

This post covers Docker log rotation properly: what the defaults actually do, why the daemon.json change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.

The default is unbounded, and that is documented behaviour

Docker’s default logging driver is json-file. It captures stdout and stderr from the container’s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.

The important part is the option defaults. For json-file, max-size defaults to unlimited, max-file defaults to 1, and compress defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.

Docker’s own documentation is upfront about why. Keeping json-file unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.

Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:

# What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"

The gap between the first command and the second is where most of the pain lives, and we will come back to it.

To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container’s log file plus a small amount of metadata, so the sizes are close enough to be useful:

sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10

Container IDs are not memorable, so map the winner back to a name with docker ps --no-trunc or by grepping the ID against docker ps -aq. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.

Setting up Docker log rotation on the daemon

The daemon-wide fix goes in /etc/docker/daemon.json. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

What each key buys you:

• max-size caps a single file. Accepts an integer plus k, m or g. This is the setting that stops the runaway.
• max-file caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is only effective if max-size is also set, which is the single most common misconfiguration I see.
• compress gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.

Two things trip people up here. First, every value in log-opts must be a JSON string, including the numeric ones. Writing "max-file": 3 without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed daemon.json on a production host is a much worse afternoon than a full disk:

# Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  && echo "daemon.json parses OK"

sudo systemctl restart docker

Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is max-size multiplied by max-file. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.

Why your daemon.json change did nothing

This is the failure mode that costs the most time, because it looks exactly like the fix not working.

Logging configuration is baked into a container’s host config when the container is created. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because docker restart stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.

So the container that filled your disk keeps filling your disk, its log file still has no cap, and docker info cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.

To actually apply it, recreate the container and then verify rather than assuming:

# Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web

You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.

Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.

Pinning limits per service in Compose

Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides daemon.json entirely. A YAML anchor keeps it from turning into copy-paste sprawl:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

Be aware that the logging block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with its defaults, which for json-file means straight back to unbounded. Half-specifying is worse than not specifying at all.

You deleted the log file and df did not move

Classic 3am mistake, and it is worth understanding rather than memorising.

When you rm the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from ls, the daemon keeps writing into a file you can no longer see, and df reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.

truncate -s 0 is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:

sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"

There is a cost. Truncating out from under the daemon can leave an in-flight docker logs -f stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.

The same reasoning applies to logrotate rules pointed at /var/lib/docker/containers/*/*.log. Rotating with copytruncate works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker’s own docs warn against external tools touching those files. If the driver can rotate for you, let it.

Choosing a driver, and what each one costs

json-file

The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. docker logs works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.

local

Docker’s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it rotates and compresses by default, retaining five files of 20 MB each per container out of the box. docker logs works exactly as normal. The trade-off is that the format is Docker’s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is docker logs plus a shipper that talks to the daemon, this is the safest default you can pick.

journald

Hands container output to systemd’s journal, which means you get journalctl filtering, structured fields, and the journal’s own size management via SystemMaxUse in journald.conf. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.

The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at RateLimitIntervalSec and RateLimitBurst before you need them, not after.

syslog, fluentd and the remote drivers

These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.

What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.

none

Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because docker logs returns nothing and you will waste twenty minutes concluding the container is broken.

The blocking behaviour worth knowing about

By default the logging path is blocking. If the driver cannot keep up, the container’s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.

docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1

Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.

Dual logging, and why docker logs still works

If you switch to a remote driver and find docker logs still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the local driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.

Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.

Reclaiming space when the disk is already full

Order matters here. Free space first, fix the cause second, verify third.

1. Confirm logs are actually the problem. Compare du -sh /var/lib/docker/containers against docker system df, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.
2. Identify the largest log files with the du and sort command from earlier.
3. Copy anything you need out first, if the logs are part of an active investigation.
4. Truncate the offenders with truncate -s 0. Never rm.
5. Set the limits in daemon.json, validate the JSON, restart the daemon.
6. Recreate the containers so the limits take effect, then confirm with docker inspect on each one.
7. Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.

Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.

Troubleshooting

Rotation is configured but files still grow past the cap. The container predates the change. Check docker inspect --format '{{.HostConfig.LogConfig}}' and recreate it.

The daemon will not start after editing daemon.json. Almost always invalid JSON, or unquoted values in log-opts. Read the actual error with journalctl -u docker --no-pager -n 50 rather than guessing.

max-file is set but only one file ever appears. max-file does nothing without max-size. Set both.

docker logs -f hangs after showing existing output. Something truncated the file underneath the daemon, usually an external logrotate rule. Reattach to recover, then remove the rule.

docker logs returns nothing at all. Check the driver. none discards everything, and some remote drivers only serve reads through the dual-logging cache.

Disk usage did not drop after deleting log files. The daemon still holds the descriptors. Restart the daemon to release the inodes, and use truncate next time.

Logs vanish sooner than the retention you configured. On journald, check journal rate limiting and SystemMaxUse. On a remote driver in non-blocking mode, check whether the buffer is overflowing.

Common mistakes

• Assuming a daemon restart applies the new limits to running containers. It does not.
• Setting max-file without max-size and believing rotation is on.
• Writing numeric values unquoted in log-opts.
• Overwriting an existing daemon.json instead of merging, and silently dropping storage or network settings.
• Using rm on a live container’s log file and concluding the disk report is broken.
• Specifying a logging.driver in Compose without its options, which resets that service back to the driver’s own unbounded defaults.
• Pointing host logrotate at the Docker containers directory when the driver could have handled it.
• Sizing rotation per container without multiplying by container count against the actual disk.

Best practices

• Set logging limits in daemon.json as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.
• Prefer the local driver on standalone hosts unless something specifically needs to read the JSON files.
• Turn on compress. Log text compresses well enough that you get more retention for the same disk.
• Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.
• Do the multiplication. Worst-case host usage is max-size × max-file × container count, and it should be a fraction of your disk, not most of it.
• Alert on Docker data root usage at a threshold that leaves you time to act.
• Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.
• Verify with docker inspect after any change. docker info tells you the daemon’s intent, not what your containers carry.

Frequently asked questions

Does Docker rotate container logs automatically?

Not with the default driver. json-file ships with max-size unlimited and max-file set to 1, so a single file grows without bound. The local driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.

Where are Docker container logs stored on the host?

Under the Docker data root, one directory per container, which on a standard Linux install means /var/lib/docker/containers/. Rather than assembling the path by hand, ask the daemon: docker inspect --format '{{.LogPath}}' <container>. That works regardless of a custom data-root.

How do I clear Docker logs without stopping the container?

Truncate the file to zero bytes with truncate -s 0 on the path from {{.LogPath}}. Do not use rm: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.

Why did my daemon.json rotation settings not apply?

Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and docker restart reuses the same container object. Recreate the container, then confirm with docker inspect --format '{{.HostConfig.LogConfig}}'.

Should I use the local driver or json-file?

Use local if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use json-file if an agent or shipper on the host tails those files, because it cannot parse the local format. Either way docker logs behaves the same.

Can I use logrotate for Docker container logs instead?

You can, and plenty of hosts do, but the driver options are the better tool. A copytruncate rule pointed at the containers directory reaches into files the daemon treats as private, and can leave docker logs -f streams stalled after each rotation. Use logrotate for logs your applications write to mounted volumes, and let the driver handle stdout.

How much disk should I budget for container logs?

Worst case is max-size × max-file per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.

The one thing worth remembering

Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.

So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with docker inspect rather than trusting docker info. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.

Need a hand with container logging on your hosts?

Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:

• Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits
• Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number
• Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back
• Choosing between json-file, local, journald and shipping off-box, including the blocking behaviour trade-offs under load
• Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends
• Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot

If you want a second opinion, send me your daemon.json, the output of docker ps and a du -sh of your containers directory, and I will tell you what I would change.

The post Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation appeared first on John Nessime.

That is the moment most people properly meet the command line, usually at the worst possible time. The dashboard is fine for editing a post. It is a bad tool for anything that touches a thousand rows, twelve plugins, or two servers at once.

This is a working list of the WP-CLI commands that save hours, grouped by the job that eats the hours rather than by command name. Each section leads with the failure mode, because with a couple of these the command finishes, prints Success, and leaves you with a subtly broken database that nobody notices for a week.

The migration hour: search-replace, and the one that bites back

Changing a domain by running SQL against wp_posts is how sites break. WordPress stores a great deal of configuration as PHP serialized arrays, and serialized strings carry their own byte length. Change https://old.example.com to https://new.example.org with a plain UPDATE ... REPLACE() and the length prefix no longer matches the string. PHP refuses to unserialize the value, returns false, and the widget or page-builder section silently renders nothing.

The whole point of wp search-replace is that it unserializes, replaces, and reserializes, so the lengths stay correct. Run it in dry-run mode first, every single time:

wp search-replace 'https://old.example.com' 'https://new.example.org' 
  --all-tables-with-prefix 
  --skip-columns=guid 
  --precise 
  --report-changed-only 
  --dry-run

What each flag is actually doing:

• --dry-run runs the whole operation and prints the report without writing. The report has a “Type” column: PHP means the value was serialized and handled properly, SQL means a plain string replacement. If you expected hundreds of changes and see zero, your search string is wrong, not the tool.
• --all-tables-with-prefix widens the scope to every table sharing your prefix, not just the tables registered with $wpdb. Plugins that create their own tables live here. --all-tables goes wider still and will happily rewrite tables belonging to a completely different application sharing the database, so reach for the prefix version first.
• --skip-columns=guid is the one people leave off. The guid column is a permanent identifier for feed readers, not a URL to be followed. Rewriting it makes every historical post look brand new to anything that has subscribed to your feed.
• --precise forces PHP-side serialization handling on every column instead of letting MySQL do the simple cases. It is slower. On a large database it is noticeably slower. It is also the difference between “probably fine” and “definitely correct”.
• --report-changed-only trims the report to tables that actually changed, which makes the output readable instead of a wall of zeros.

Read the dry-run output. When the numbers look right, run the identical command without --dry-run.

The step everyone forgets

You changed the database. Every cache layer still holds the old values. On a site with Redis or Memcached behind a persistent object cache drop-in, the old URLs will keep being served from memory and you will spend twenty minutes convinced the replacement failed.

wp cache flush
wp transient delete --expired
wp rewrite flush

Be aware that on multisite with a shared persistent object cache, wp cache flush can clear entries for every site on the network, not just the one you are working on. Then purge the page cache and the CDN through their own controls. If you are on Cloudflare, the database change means nothing until the edge cache is purged too.

Finally, re-run the dry-run. It should report zero remaining replacements. If it does not, you have a second URL format hiding somewhere: protocol-relative //old.example.com, or an escaped form like https://old.example.com inside JSON stored in an option. Run a second pass for each shape you find.

The plugin conflict hour: bisect without touching the dashboard

Classic scenario: something fatals, the admin is white, and the usual advice is “deactivate all plugins and switch to a default theme.” On a production site that means real downtime while you click through a list.

WP-CLI has two global parameters that skip loading plugins and themes for the duration of a single command. Nothing is deactivated. Nothing is written. The site keeps serving visitors exactly as it was.

# Does WP-CLI work at all with nothing loaded?
wp --skip-plugins --skip-themes option get siteurl

# Is it the theme?
wp --skip-themes option get siteurl

# Skip one specific plugin
wp --skip-plugins=akismet option get siteurl

If the first command works and the unmodified one does not, the fault is in your plugins or theme, not in WP-CLI or WordPress core. From there you can walk the active list one at a time. This one-liner from the WP-CLI handbook does the walk for you:

wp plugin list --field=name --status=active --skip-plugins 
  | xargs -n1 -I % wp --skip-plugins=% plugin get % --field=name

The inner --skip-plugins on the listing command matters. Without it, a plugin broken badly enough to crash the bootstrap will also crash the command that is trying to build the list, and you get nothing.

The honest limitation: skipping a plugin at the CLI does not reproduce browser-side behaviour. If the bug only appears for logged-in users on a specific admin screen, this narrows the field but will not fully confirm the culprit. It still turns a thirty-minute click-through into about two minutes of scrolling output.

The silent-cron hour: scheduled jobs that quietly stopped

This is the invisible failure. Nobody opens a ticket saying “cron is broken.” They open a ticket saying backups have not run since last month, or scheduled posts are stuck in the future, or order emails stopped going out. WP-Cron is not a scheduler. It is a queue that gets drained when someone loads a page, which means a low-traffic site drains it rarely and a fully page-cached site may never drain it at all.

# Can the site reach its own wp-cron.php?
wp cron test

# What is scheduled, and how overdue is it?
wp cron event list

# Drain everything that is due, right now, in this process
wp cron event run --due-now

In wp cron event list, the column to read is next_run_relative. If events are showing as long overdue, the queue is not being drained and you have found your problem.

The fix is to stop relying on page loads. Disable the loopback trigger in wp-config.php, above the “stop editing” line:

define( 'DISABLE_WP_CRON', true );

Then add a real system cron entry for the user that owns the site files:

* * * * * /usr/local/bin/wp --path=/var/www/example.com cron event run --due-now --quiet

Why the CLI form and not a curl of wp-cron.php: wp cron event run executes the events in the current PHP process. No HTTP loopback is involved, so it cannot be defeated by a firewall rule, a reverse proxy, HTTP authentication on staging, or a host that rate-limits self-requests. Errors surface in the cron mail or the log you redirect to, instead of vanishing into a request nobody reads.

One warning. Set DISABLE_WP_CRON and then forget the crontab entry, and you have upgraded an unreliable scheduler into one that never runs at all. Verify with wp cron event list a few minutes later before you close the ticket.

The bulk-edit hour: piping IDs instead of clicking pages

The pattern that does the real work here is --format=ids feeding another command. Once that clicks, most “there is no plugin for this” tasks become one line.

# How many revisions are we carrying?
wp post list --post_type=revision --format=count

# Delete them, skipping the trash
wp post delete $(wp post list --post_type=revision --format=ids) --force

# Every administrator on the site, in one table
wp user list --role=administrator --fields=ID,user_login,user_email

# Regenerate only the thumbnail sizes that are missing
wp media regenerate --only-missing --yes

Two things to know before you run the delete. It is irreversible, so take a database export first. And if the inner command returns nothing because there are no revisions, the outer command gets no arguments and will error rather than doing nothing quietly. That is a harmless failure, but it looks alarming the first time.

Revisions come back unless you cap them. Adding define( 'WP_POST_REVISIONS', 5 ); to wp-config.php keeps enough history to recover from a bad edit without letting wp_posts grow without limit.

For anything you want to script rather than read, add --format=json and pipe it into jq. Every list command in WP-CLI supports it, which makes the whole tool usable as a data source for monitoring and reporting rather than just an admin replacement.

The “is this thing compromised?” hour: checksums

A site is behaving oddly, there is a suspicious redirect, and the first instinct is to install a security scanner. Before that, run the two commands that compare what is on disk against the official hashes on WordPress.org.

wp core verify-checksums
wp plugin verify-checksums --all

These download the published checksums for your exact version and locale and report every core or plugin file that has been modified, plus files that should not exist at all. The core command deliberately avoids loading WordPress, so it still works when the site itself is too broken to boot.

Know the blind spots before you trust a clean result:

• Only plugins hosted on the WordPress.org directory can be verified. Premium plugins, custom code and anything from a private repository have no published source of truth and will be skipped or warned about.
• Nothing in wp-content/uploads is covered. A dropped PHP file in an uploads subdirectory is a common backdoor and checksums will never see it.
• Themes are not covered by an equivalent bundled command.
• Getting a locale or version mismatch produces alarming warnings that are not real findings. Pass --version and --locale explicitly if the output looks wrong.

A clean checksum run is not proof the site is clean. A dirty one is proof it is not, which is worth a great deal in the first five minutes of an incident.

The database hour: find the bloat before you optimise anything

“The site is slow” is not actionable. “One table is carrying most of the database” is. Start by looking, not by running an optimiser.

# Size per table, sorted by whatever is worst
wp db size --tables --human-readable

# Take an export before touching anything
wp db export ~/backups/example-com.sql

# Run a query and strip the ASCII table borders
wp db query 'SELECT option_value FROM wp_options WHERE option_name="home"' --skip-column-names

In practice the offenders are nearly always wp_options stuffed with autoloaded rows and orphaned transients, wp_postmeta from a plugin that never cleans up after itself, or wp_posts full of revisions. Autoloaded options are the sneaky one, because every row marked for autoload is loaded on every request, including requests that will never look at it.

Two clean-up commands worth knowing, and the difference between them:

wp transient delete --expired
wp transient delete --all

The first removes only what has already expired and is safe to run on a schedule. The second removes everything, including live cache entries, which forces WordPress to rebuild them on the next requests. That is safe but produces a short performance dip, so it belongs in a maintenance window rather than in a cron job.

Note that on a site with a persistent object cache drop-in installed, transients bypass the database entirely and wrap the object cache instead. If wp transient delete --all reports nothing, that is usually the reason and not a failure.

The setup that makes everything above faster

If you manage more than one site, the biggest single time saving is not a command at all. It is aliases. Put a wp-cli.yml in a project directory, or ~/.wp-cli/config.yml for a global one:

@production:
  ssh: deploy@example.com/var/www/example.com
@staging:
  ssh: deploy@staging.example.com/var/www/staging

Now you run commands against a remote site from your own machine, without an SSH session:

wp @staging plugin list --status=active
wp @production cron event list

This needs WP-CLI installed on the remote host, and it needs your host to actually give you SSH. That varies more than people expect. A plain VPS from a provider like InterServer or DigitalOcean gives you root and you install what you like. Managed WordPress hosts differ: some ship WP-CLI preinstalled and hand you an SSH key immediately, others restrict which commands you can run, and a few give you no shell at all on entry-level plans. Worth checking before you build a workflow that depends on it.

Two optional packages are worth the install if you do maintenance work regularly:

wp package install wp-cli/doctor-command
wp package install wp-cli/profile-command

wp doctor check --all
wp profile stage

wp doctor runs a set of health checks, including one for autoloaded options size, and gives you a pass or warn per check. wp profile breaks a request into stages so you can see where the time is going before you start guessing.

The honest trade-off: these are separate packages, not bundled commands, and they get less attention than the core set. If you are working across client servers you do not control, installing packages everywhere is friction you may not want. On your own infrastructure they earn their place. For continuous visibility rather than point-in-time checks you still want proper monitoring, whether that is a self-hosted Prometheus and Grafana stack or a hosted uptime service.

Troubleshooting: when WP-CLI itself will not run

Before blaming the tool, run wp cli info. It prints the PHP binary in use, the PHP version, the loaded php.ini, and the WP-CLI version. Most problems are visible in that output.

• “This does not seem to be a WordPress installation.” You are in the wrong directory, or the document root is elsewhere. Pass --path=/full/path/to/wordpress rather than guessing.
• Wrong PHP version. On a control panel server the shell PHP is often not the PHP the site runs on. Check the binary in wp cli info, and invoke the right one explicitly if they differ, for example /usr/local/php83/bin/php $(which wp).
• Memory exhaustion on large operations. The CLI uses its own memory limit, not the web one. Raise it for a single command with php -d memory_limit=512M $(which wp) ... instead of editing configuration globally.
• A fatal error the moment you type anything. Confirm with wp --skip-plugins --skip-themes cli info. If that works, the fault is in site code.
• A warning about running as root. WP-CLI is telling you that files created by this command will be owned by root and the web server may not be able to write to them later. --allow-root silences the warning without fixing the ownership problem. Run as the site user instead, unless you are inside a container where root is the normal case.

Common mistakes

• Running search-replace without --dry-run because you have done it before and it worked.
• Reaching for --all-tables by reflex when --all-tables-with-prefix is what you meant, on a server where several sites share one database.
• Forgetting to flush the object cache and CDN after a database write, then re-running the replacement several times chasing a ghost.
• Setting DISABLE_WP_CRON without adding the system cron entry.
• Treating a clean verify-checksums result as proof that a site is not compromised.
• Running destructive commands over SSH without screen or tmux, then losing the connection mid-write on a large table.
• Using --allow-root as a habit and leaving root-owned files scattered through wp-content.

Best practices

1. Export the database before any command that writes. wp db export takes seconds and has saved more afternoons than any other command in this post.
2. Dry-run first, read the report, then repeat the command verbatim without the flag. Do not retype it.
3. Run as the site user, not root, so file ownership stays correct.
4. Use aliases in wp-cli.yml so you cannot run a staging command against production by muscle memory.
5. Verify after the write. Re-run the read-only version of whatever you just did and confirm it reports zero remaining work.
6. Test the restore path, not just the backup. An export you have never imported is a hope, not a backup.
7. Keep WP-CLI current with wp cli update, particularly before working on a site running a recent PHP release.

Frequently asked questions

Is wp search-replace safe to run on a live site?

Safer than raw SQL, because it handles serialized data correctly. Still a write against production. Export the database first, dry-run first, and use --skip-columns=guid. On a large database the operation can hold the site in a slightly inconsistent state for the duration, so a quiet window helps.

Why do my changes not show up after search-replace?

Almost always caching. Run wp cache flush, purge the page cache and the CDN, then reload. If the old value still appears in a dry-run report, you have a second URL format stored somewhere, such as a protocol-relative or escaped variant.

Does WP-CLI need SSH access?

It needs shell access on a machine that can reach the WordPress files and database. That usually means SSH. Some managed hosts provide a browser terminal instead, and some restrict the available commands. On a self-managed VPS you install it yourself and nothing is restricted.

What is the difference between –skip-plugins and deactivating a plugin?

--skip-plugins prevents the plugin from loading for that one command only. Nothing is written to the database and visitors are unaffected. Deactivating changes site state and takes the plugin offline for everyone, which is why the skip flags are the right diagnostic tool on production.

Can I schedule WP-CLI commands with cron?

Yes, and that is the recommended way to run WordPress scheduled events reliably. Use the full path to the wp binary, pass --path, run as the site user, and redirect output so failures are visible rather than silent.

Will verify-checksums detect all malware?

No. It compares core files and WordPress.org-hosted plugins against published hashes. It does not cover themes, premium or custom plugins, or anything in the uploads directory. Treat a failure as a definite finding and a pass as one data point among several.

Which WP-CLI commands should I learn first?

wp db export, wp search-replace --dry-run, wp --skip-plugins, and wp cron event list. Those four cover backup, migration, conflict isolation and the most common silent failure on a WordPress site.

The one thing worth remembering

Most of the WP-CLI commands that save hours do it by turning a job you cannot see into a job you can read. --dry-run shows you the write before it happens. wp cron event list shows you a queue that was failing silently. verify-checksums shows you a file that changed without anyone deciding it should.

The speed is real, but the speed is not the point. The point is that you stop guessing. If you take one habit from this post, take the dry-run: run the read-only version, read the output, then run the identical command for real. That single pattern prevents the majority of the incidents I see caused by command-line work on WordPress.

Need a hand with this?

I work on WordPress infrastructure, and command-line automation is most of what makes that maintainable. Things I can help with directly:

• Domain and host migrations run properly, with staged dry-runs, correct table scoping and a verified rollback before anything is written.
• Replacing WP-Cron with real system cron across a fleet, including alerting when a job stops firing instead of finding out weeks later.
• Writing repeatable WP-CLI maintenance scripts, with proper exit codes and logging, so they can be scheduled rather than remembered.
• Diagnosing slow WordPress sites at the database layer, from autoloaded options and table bloat down to the query causing it.
• Incident triage on a site suspected of being compromised, starting with integrity checks and file-level evidence rather than a scanner plugin.
• Setting up multi-site management workflows with WP-CLI aliases, SSH keys and deployment pipelines that do not depend on anyone opening the dashboard.

If something specific is going wrong, send the actual output. A dry-run report, a wp cli info dump, a wp cron event list table, and I can usually tell you where the problem is before we talk about scope.

The post WP-CLI Commands That Save Hours (And the One That Bites Back) appeared first on John Nessime.