The same output showed roughly 11 GB in the available column. Nothing was wrong, and nothing had been wrong for months. The only thing broken was the alert that fired.

Low free memory on Linux is one of the most reliably misread signals in server operations. It looks like an emergency, it triggers scaling decisions, and it is almost always noise. This post covers what free is really telling you, which four signals actually indicate memory trouble, the one situation where a low free number genuinely matters, and what to put in your alert rules instead.

What the free command is actually reporting

free does not measure anything. It parses /proc/meminfo and prints a handful of those fields in columns. Every number it shows comes straight from the kernel and is readable directly if you want to skip the formatting.

A typical reading on a busy server looks roughly like this:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       3.4Gi       231Mi       128Mi        12Gi        11Gi
Swap:          2.0Gi          0B       2.0Gi

Read that left to right and the story is boring. Applications hold a few gigabytes. The kernel filled almost everything else with page cache and buffers, because every file that gets read or written passes through memory on the way, and directory metadata lands in reclaimable slab. What is left over, genuinely untouched, is the free column.

That last part is the bit people miss. Free memory is not a reserve. It is waste. A page holding a cached copy of a file your application reads every few seconds is doing useful work. A page sitting empty is doing nothing, and the kernel converts it into cache the moment it has a reason to. A server that has been up for a week and still shows plenty of free memory is a server that is barely touching its disk.

If you want the raw source rather than the formatted view:

$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SReclaimable|SwapTotal|SwapFree|Dirty):' /proc/meminfo

Two options are worth knowing. free -w switches to wide mode and splits buffers and cache into separate columns, which helps when attributing memory to a specific I/O pattern. free -h -s 5 reprints every five seconds, a crude but effective trend view while you reproduce a problem.

Signal one: available memory, not free memory

The available column maps to MemAvailable in /proc/meminfo, and it is the kernel’s own estimate of how much memory a new workload could allocate without pushing the system into swap. It has been present since kernel 3.14, so every distribution you are likely to be running exposes it.

What matters is that it is not free plus cache. That naive sum is what people calculated before MemAvailable existed, and it overstates the truth. The kernel starts from free pages, subtracts the reserves it refuses to dip below, then adds back the portion of file-backed page cache and reclaimable slab it believes it could actually release. Both additions are capped, because some cache is pinned and some reclaimable slab objects are in use.

The kernel documentation is explicit that this is an estimate whose accuracy varies from system to system, so treat it as a strong hint rather than an exact figure. It is still far better than anything you can compute yourself, and it is the number your dashboards should be built on.

If you change one thing after reading this: point your memory alerts at available, not at free. Everything else in this post is refinement on top of that.

Signal two: swap activity, not swap usage

The second most common false alarm is swap. Someone sees a few hundred megabytes of swap in use and concludes the machine is thrashing. It usually isn’t.

Swap usage is a stock: pages moved to disk at some point, possibly during a backup weeks ago, and never needed since. Leaving them there is correct, and pulling them back to satisfy a dashboard would evict something useful. Swap activity is a flow, and that is the part that hurts, because a page fault serviced from disk is orders of magnitude slower than one serviced from RAM.

vmstat gives you the flow. The si and so columns are swap-in and swap-out, in blocks per second over the sampling interval:

$ vmstat 5 6

Discard the first row. vmstat reports averages since boot on its first line, which is almost never what you want. Read the rows after that. If si and so are both consistently non-zero, the machine is simultaneously reading pages back to serve current work and writing pages out to make room for new allocations. That is thrashing, and it is a real signal with no ambiguity attached.

A single non-zero sample is not that. Brief swap activity during a log rotation or a backup window is normal. Sustained activity across several consecutive samples is what to escalate.

Signal three: memory pressure, which is the one that catches things early

Pressure Stall Information is the signal I reach for first now, and it is badly underused. It has been in the kernel since 4.20 and it answers a question none of the other tools answer directly: how much time are my processes losing because memory is contended?

$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

The some line is the share of wall-clock time in which at least one task was stalled waiting on memory. The full line is the share in which every non-idle task was stalled, meaning nothing useful happened at all. The averages cover ten seconds, one minute and five minutes, and total is a cumulative counter in microseconds that updates continuously, so it catches a two-second stall the running averages smooth away entirely.

Here is why this beats a percentage threshold. Utilization tells you how much of a resource is consumed; pressure tells you whether that consumption is costing you anything. A machine can sit at high utilization for months with zero pressure, which means the sizing is fine. It can also sit at moderate utilization while full climbs, which means reclaim is working hard and latency is already suffering with no threshold crossed.

With cgroup v2 mounted, the same data exists per cgroup in each directory’s memory.pressure file, so you can attribute pressure to a specific service rather than the whole machine. Prometheus node_exporter collects the system-level PSI counters, and if you already ship metrics to something like Grafana Cloud these graph well beside your latency panels.

Signal four: the OOM log, which is the only unambiguous one

When memory really does run out, the kernel writes a record. It does not do this quietly and it does not do it conditionally. If there is no OOM record, no process was killed for memory, full stop.

$ sudo journalctl -k --since "24 hours ago" | grep -i "oom"
$ sudo dmesg -T | grep -i "killed process"

The kill line names the victim and reports its memory breakdown at the moment of death: total virtual size, anonymous resident set, file-backed resident set, shared memory resident set, and the oom_score_adj that was in effect. The line above it, where a process “invoked oom-killer”, tells you which allocation triggered the whole event, and that process is frequently not the one that got killed.

That distinction is the most useful thing in the log. The kernel picks a victim by badness score, driven largely by how much memory a process holds relative to what is available to it, adjusted by oom_score_adj. A slow leak in a small helper can push the machine over the edge and get your database killed instead, because the database is simply the largest thing running. Read both lines before deciding who the culprit is.

If you have a service that should survive at almost any cost, systemd exposes OOMScoreAdjust= in the unit file, which persists across restarts in a way that writing to /proc/<pid>/oom_score_adj does not. Use it sparingly. Protecting a process that is itself leaking just moves the crash somewhere less convenient.

When low free memory on Linux really is the problem

Everything above assumes you are looking at a whole machine. Inside a container, the accounting changes, and this is the failure mode that is invisible until it isn’t.

The cgroup memory controller charges page cache to the cgroup that caused it. Clean, evictable, perfectly reclaimable cache still counts toward memory.current, and that is what gets compared against memory.max. A container reading a large file has just charged that file to its own budget.

$ cat /sys/fs/cgroup/memory.current
$ cat /sys/fs/cgroup/memory.max
$ grep -E '^(anon|file|slab_reclaimable) ' /sys/fs/cgroup/memory.stat

The split in memory.stat is what you actually want. anon is memory your process genuinely holds; file is page cache charged to you that the kernel can take back under pressure. A container at its limit on mostly file is in a very different position from one at its limit on mostly anon.

Two things follow from this that catch people out repeatedly:

• Reclaim is not instant. Under heavy pressure the reclaim path itself runs slowly, so a container can be OOM-killed while still holding clean page cache that was theoretically available.
• Running free inside a container lies to you. Unless something virtualizes /proc, you are reading the host’s /proc/meminfo, which knows nothing about your limit. The cgroup files are the only truth here.

cgroup v2 also has memory.high, a soft limit. Crossing it triggers reclaim and throttling rather than a kill, which turns a hard failure into a slow one. That is usually the better trade, and it is worth setting deliberately rather than leaving it at its default.

What to alert on

The alert on the ticket at the top of this post was watching the wrong number. Here is what a memory alert set looks like when it watches the right ones:

• Available memory below a percentage of total, sustained. Ten percent for ten minutes is a reasonable starting shape. A single thirty-second dip is noise and should never page anyone.
• Sustained swap-in and swap-out together. Rate of change on the swap counters, not the absolute amount of swap in use.
• Memory PSI rising. The some line moving off the floor is your early warning. The full line moving at all is your late one.
• Any OOM kill, ever. This one is binary and should always generate a ticket, even if the service recovered on its own.
• Per-container memory events. The cgroup memory.events file counts limit hits, and a rising count is a limit that needs revisiting.

Notice that free memory does not appear anywhere in that list. There is no threshold on the free column that produces a useful alert, because the number is a byproduct of the kernel’s caching behaviour rather than a measure of anything you care about.

Troubleshooting: which tool answers which question

When a memory question lands on you, the order you check things in matters more than the tools.

1. Did anything actually die? Check the kernel log for OOM records first. If there are none, this is a performance question and not a capacity failure, which changes everything about what you do next.
2. Is the machine stalling? Read /proc/pressure/memory. Zero pressure with low free memory means you are done. Go close the ticket.
3. Is it swapping right now? Run vmstat 5 for half a minute and watch si and so. Ignore the first row.
4. Who is holding the memory? ps aux --sort=-%mem | head -15 gets the shortlist. Resident set size double-counts shared pages, so summing the column overstates the total.
5. Is it the kernel rather than a process? If the process list does not account for the usage, check slabtop and SUnreclaim in /proc/meminfo. Unreclaimable slab growth is a different problem, usually a driver, filesystem or network workload creating enormous numbers of kernel objects.
6. Is it growing? A leak is a slope, not a level, and one reading tells you nothing. This is the whole argument for having the graph before you need it.

One more that saves time on crowded boxes: resident set size is the wrong measure of what a process really costs, because shared libraries are counted against every process that maps them. Proportional set size divides shared pages among the processes sharing them, and tools reporting PSS give a far more honest answer when you are deciding what to move off a busy VPS.

Common mistakes

• Dropping caches to make the number look better. Writing to /proc/sys/vm/drop_caches discards clean page cache and reclaimable slab. The kernel documentation says plainly that using it outside testing or debugging is not recommended, because everything dropped has to be re-read from disk. You trade a nicer-looking free output for an I/O spike and worse latency. It is a benchmarking tool, not an operations tool.
• Setting vm.swappiness to zero as a reflex. It does not disable swap and it does not make the machine faster. It makes the kernel strongly prefer evicting page cache over swapping anonymous pages, which on a database host means throwing away exactly the cache you wanted. Change it for a measured reason, not by default.
• Running with no swap at all. Swap is not just an overflow area. Without it the kernel loses the option of paging out genuinely cold anonymous memory, which means the only lever left under pressure is evicting cache or killing something. A modest swap file gives reclaim somewhere to go.
• Summing RSS across processes and comparing it to total RAM. Shared pages are counted once per process. The sum will exceed reality, sometimes dramatically.
• Resizing the instance before finding the slope. More RAM buys time against a leak. It does not fix one, and the bill arrives every month afterwards.

Best practices

• Graph available memory, swap rate and memory PSI on one panel. Together they read as a story; individually each one misleads.
• Keep the raw counters, not just derived percentages. A percentage of an instance size that has since changed cannot answer “was this happening last week”.
• Set container limits from observed anon usage plus headroom, not a round number someone picked, and put memory.high below memory.max so pressure shows up as slowness before it shows up as a kill.
• Alert on rate of change for anything that can leak. The steady climb is the signal; the level it reached is just when you happened to look.
• Right-size deliberately. On providers such as Contabo or InterServer, where RAM is a meaningful share of the monthly bill, the gap between a real requirement and a panicked upgrade is money you keep every month.
• Write the explanation down where your team will find it. Every organization rediscovers the free-versus-available confusion independently, usually at three in the morning.

Frequently asked questions

Why is free memory so low on my Linux server?

Because the kernel used it for page cache and buffers, which is the intended behaviour. Any file read or written passes through memory, and the kernel keeps those pages in case they are needed again. It reclaims them automatically the moment an application needs the space. Check the available column instead.

What is the difference between free and available in free -h?

Free is memory that is completely unused. Available is the kernel’s estimate of how much a new process could allocate without swapping: free memory plus the portion of cache and reclaimable slab it believes it could release. Available is almost always much larger, and it answers the question you are actually asking.

Should I clear the page cache to free up memory?

No. The kernel reclaims cache automatically under pressure, and more selectively than a blanket drop. Writing to drop_caches forces everything to be re-read from disk, and the kernel documentation advises against using it outside testing or debugging. Doing it on a schedule is a strong hint something else is misconfigured.

Is swap being used a bad sign?

Not by itself. Pages that were swapped out during a quiet period and never needed again are fine where they are. What matters is swap traffic: sustained non-zero si and so in vmstat at the same time means the machine is thrashing. Alert on the rate, not the amount.

How do I tell if a process was killed for running out of memory?

Search the kernel log. journalctl -k filtered for “oom”, or dmesg -T filtered for “killed process”, shows a record naming the process, its PID and its memory breakdown at the time. The kernel always logs this, so if there is no record the process died for some other reason.

Why does my container get OOM-killed when memory looks fine inside it?

Because free inside a container usually reads the host’s memory information, not the cgroup limit being enforced. Read memory.current, memory.max and memory.stat instead. Page cache charged to the container counts toward its limit, so a read-heavy workload can hit the ceiling without any process holding much anonymous memory.

What is a good memory alert threshold?

Available memory below roughly ten percent of total, sustained for ten minutes, is a defensible starting point to tune against your own workload. Pair it with memory PSI and an unconditional alert on any OOM kill. A threshold on the free column, at any value, will page you for nothing.

Conclusion

Low free memory on Linux is the expected steady state of a healthy, working server. The kernel fills memory with cache because empty memory earns nothing, and hands it back the instant an application asks. The free column is a byproduct of that behaviour, not a health metric, and alerting on it produces exactly the kind of urgent ticket that wastes an afternoon and ends in a change request nobody needed.

The one thing worth remembering: free tells you what is idle, available tells you what you can have, and pressure tells you whether any of it is costing you. Watch the second and third, log the OOM events, and read the cgroup files rather than free whenever a container is involved.

Need help reading your memory signals?

Most of the memory work I get asked for is not “add more RAM”. It is working out whether the number someone is worried about means anything. I can help with:

• Rebuilding memory alerts around available memory, swap rate and PSI so they stop paging on normal cache behaviour
• Diagnosing repeated OOM kills, including the cases where the process being killed is not the one causing the problem
• Setting container and pod memory limits from observed anonymous usage rather than guesses, with a soft limit so pressure surfaces before a kill does
• Separating a genuine leak from normal cache growth using retained metrics rather than a single reading
• Right-sizing servers and instances honestly, up or down, based on what the workload actually needs
• Building the dashboard that turns the next memory question into a five-minute conversation instead of an incident

If you have a free -h output, a vmstat capture or an OOM log line that is bothering you, send it over and I will tell you what it means.

The post Low Free Memory on Linux Isn’t a Problem. Here’s What Actually Is. appeared first on John Nessime.

The crontab entry is still there. The script still works when you run it by hand. Nothing in the monitoring went red, because nothing was watching whether the job ran at all. It was watching whether the server was up.

That is the real question behind cron vs systemd timers, and it has almost nothing to do with syntax. Both tools can run a command at 2 a.m. The difference is what happens on the night the command doesn’t run, and how long it takes anyone to find out.

This post compares the two by failure family rather than by feature list: missed runs, overlapping runs, environment surprises, output that goes nowhere, and jobs that eat the machine. Then a decision procedure, the same job written both ways, and how to debug each one when it goes quiet.

The failure that actually decides it

Scheduled jobs fail silently far more often than they fail loudly. A job that crashes with a stack trace gets fixed the same week. A job that stopped firing six weeks ago gets found during an incident.

Cron’s default answer to “what happened?” is to mail the job’s output to the owning user. On a server with no local mail transfer agent configured, which is most servers built in the last decade, that output is generated and then discarded. The daemon logs that it started the command. It does not log what the command printed, and in most implementations it does not log the exit status either.

So you get a log line saying cron ran your backup, every night, forever, including on the nights the backup wrote nothing. That line is the trap. It looks like evidence.

systemd timers hand this to journald instead. Standard output and standard error from the triggered service land in the journal, tagged with the unit name, alongside the exit status and the runtime. That is the single biggest practical difference between the two, and it is the reason most of my new scheduled work goes into timers.

Neither one alerts you. Both will happily not run for a month while you sleep well. Observability is not monitoring, and I’ll come back to that.

Where cron still wins

Cron is one line in one file. There is no unit to reload, no second file to keep in sync, no install section to get wrong. crontab -e, five fields, a command, done. For a job that trims a log directory on a box you own, that economy is not nothing.

It also runs where systemd does not. BSD hosts, Alpine images, containers, appliance firmware, embedded boxes, older enterprise systems still on a legacy init. If your configuration management has to cover a fleet that is not uniformly systemd, cron is the common denominator and it is not close.

Per-user scheduling is genuinely simpler too. A user runs crontab -e and owns their own jobs, with no root involvement and no unit files in /etc. On shared hosting and on control panels like DirectAdmin or cPanel, this is the only model available, and the panel writes the crontab for you.

Where cron loses

• No dependency ordering. Cron cannot wait for the network to be up or the database to be ready. It fires at the wall clock time and hopes.
• No overlap protection. If the 2 a.m. run is still going at 3 a.m., the 3 a.m. run starts anyway, on top of it.
• No resource containment. A runaway job competes with production for CPU, memory and disk with nothing standing between them.
• Minute resolution, and no catch-up if the machine was off.
• A syntax with sharp edges that fail quietly, which is the next section.

Where systemd timers win

A timer separates when from what. The .timer unit holds the schedule. The .service unit holds the job. That split feels like bureaucracy until the first time you need to run the job right now, out of band, without touching the schedule. Then it is just systemctl start myjob.service, and the run is logged exactly like a scheduled one.

Everything systemd knows how to do to a service, it will do to a scheduled job. Ordering after other units. Restart-on-failure. Runtime caps. Memory and CPU limits through cgroups. A dedicated user. A private /tmp. A failure handler unit that fires when the job exits non-zero.

You also get a schedule you can interrogate before it bites you. systemctl list-timers prints every timer on the box with its next and last elapse. systemd-analyze calendar takes an expression and tells you exactly when it would fire, which is the closest thing either scheduler has to a unit test.

Where systemd timers lose

• Two files, a daemon-reload and an enable for every job. That is real friction when you have forty small jobs.
• The calendar syntax is more expressive than cron’s and harder to eyeball. You will reach for systemd-analyze calendar more often than you expect.
• Portability is gone. These units run on systemd hosts and nowhere else.
• User timers need lingering enabled, or they stop when the user logs out and never start at boot. This surprises people once, memorably.
• The journal is not a log file. If your log shipping only tails /var/log, timer output is invisible to it until you point the shipper at journald.

Cron vs systemd timers, by failure family

The machine was off at 2 a.m.

Cron has no memory. If the box was down, rebooting, or suspended when the schedule came round, that run is gone. Nothing catches it up and nothing records that it was skipped.

The traditional fix is anacron, which tracks the last successful run per job in a timestamp file and runs the job shortly after boot if the interval has lapsed. It is why the scripts in /etc/cron.daily still run on a laptop that is closed every night. The catch is that anacron works in whole days and only covers the cron.daily, cron.weekly and cron.monthly directories by default. It does not help your custom job at 02:00.

systemd folds that behaviour into a single directive:

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

With Persistent=true, systemd writes a timestamp under /var/lib/systemd/timers/ each time the timer fires. On boot it compares that stamp against the schedule, and if a run was missed it triggers immediately. This matters more than it sounds on any VPS you reboot for kernel updates, and a great deal on hosts at providers like Contabo or InterServer where you take maintenance windows on their timetable rather than yours.

One caveat worth knowing: “immediately” means immediately. If you have twelve persistent timers and the box has been off for a week, all twelve fire within seconds of boot, on a machine that is still starting services. Pair Persistent=true with RandomizedDelaySec= so the catch-up is spread out.

Two copies of the same job

This is the one that corrupts data. A sync job normally takes four minutes and runs every five. One night the remote end is slow, the run takes eleven minutes, and cron starts two more copies while the first is still writing.

Cron will not stop this. You have to do it yourself, and the standard tool is flock from util-linux:

*/5 * * * * /usr/bin/flock -n /var/lock/sync.lock /usr/local/bin/sync.sh

-n means fail immediately rather than queue behind the running copy. Without it you build a backlog of processes all waiting for the same lock, which turns an overlap problem into a fork bomb with a slow fuse.

systemd gets this for free from its own model. A unit is either active or it isn’t. If sync.service is still running when the timer elapses, there is no second copy to start, because starting an already-active unit is a no-op. You do not have to remember anything.

One default is worth knowing. A calendar timer schedules its next elapse from the previous trigger time, so if a run overran, that next elapse is already in the past and the service fires again the moment it finishes. Newer systemd releases add a DeferReactivation= boolean that schedules from when the service went inactive instead, so an overrunning job waits for the next real slot. Check your distribution’s systemd version before relying on it.

The environment your job actually gets

“It works when I run it manually” is the single most common scheduled-job bug, and both schedulers cause it, for the same reason: neither one gives your job an interactive login shell.

Cron runs the command with a short, hardcoded environment. Your shell profile is not sourced. Your PATH is much shorter than the one you tested with, which is why a script that calls docker, wp, aws or a version-manager shim works from your terminal and fails from cron. The working directory is the user’s home, not wherever you happened to be standing.

Cron adds one hazard that is entirely its own. Inside a crontab, the percent sign is a metacharacter: it becomes a newline, and everything after the first one is fed to the command as standard input. So this looks fine and is broken:

# Truncated at the first %. The redirect never happens.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

# Escaped. This is what you meant.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

There is no warning. The line parses, cron runs the truncated fragment, and the log says the job started. Anything with a date + format string in a crontab deserves a second look.

systemd services get a minimal environment too, so the class of bug is identical. What differs is that the fix is declarative and lives with the unit rather than being hidden inside a wrapper script:

[Service]
Type=oneshot
User=deploy
WorkingDirectory=/srv/app
Environment=PATH=/usr/local/bin:/usr/bin:/bin
EnvironmentFile=/etc/myapp/backup.env
ExecStart=/usr/local/bin/backup.sh

The honest summary: absolute paths for every binary solve most of this on either scheduler. The rest is solved by never putting logic in the schedule. Put it in a script and schedule the script.

Everything firing at once

Write 0 0 * * * on fifty hosts and fifty hosts hit your backup target, your package mirror or your API at exactly midnight. The classic cron answer is to hash the hostname into a minute offset in configuration management, which works and which every team reinvents.

systemd has two directives here and they pull in opposite directions, which is the part people get wrong:

• RandomizedDelaySec= spreads timers apart. It adds a delay between zero and the value you set, so identical schedules on many hosts land at different moments.
• AccuracySec= pulls timers together. It lets systemd shift the firing time within a window to coalesce wakeups and save power. It defaults to one minute, which is why a timer set to 02:00:00 often fires a few seconds late.

If you want a timer to fire close to the stated second, lower AccuracySec=. If you want to stagger a fleet, raise RandomizedDelaySec=. Setting only AccuracySec= to a large value and expecting it to spread load is the mistake, because coalescing is the opposite of spreading.

A job that eats the box

Cron has no answer here beyond nice and ionice in front of the command, and a script that hangs forever will hang forever. A systemd service is a cgroup, so the limits are declarative and enforced:

[Service]
Type=oneshot
MemoryMax=512M
CPUQuota=40%
IOWeight=20
TimeoutStartSec=30min
OnFailure=notify-failure@%n.service
ExecStart=/usr/local/bin/reindex.sh

The two directives that earn their keep on a shared box are TimeoutStartSec=, which kills a oneshot job that has hung, and OnFailure=, which starts another unit when this one exits non-zero. That second one is how you get a scheduled job to page you without wiring an alerting call into every script.

The same job written both ways

A nightly database dump that must not overlap, must not run before the database is up, and must catch up if the host was rebooted. In cron:

0 2 * * * /usr/bin/flock -n /var/lock/db-backup.lock /usr/local/bin/db-backup.sh >> /var/log/db-backup.log 2>&1

That handles overlap and captures output. It does not handle “the database wasn’t ready” or “the host was down”, and the log file is now your problem to rotate.

The systemd version is two files. First the service, which is the job:

# /etc/systemd/system/db-backup.service
[Unit]
Description=Nightly application database dump
After=network-online.target mariadb.service
Wants=network-online.target

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/db-backup.sh
TimeoutStartSec=45min

Then the timer, which is only the schedule:

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run the nightly database dump

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=15min

[Install]
WantedBy=timers.target

Because the two units share a base name, you do not need a Unit= line. systemd pairs db-backup.timer with db-backup.service by convention. Add Unit= only when the names differ.

Then the part people skip, which is verifying it before walking away:

# Check the units parse before you enable anything
systemd-analyze verify /etc/systemd/system/db-backup.*

# Confirm the calendar expression means what you think
systemd-analyze calendar --iterations=5 "*-*-* 02:00:00"

systemctl daemon-reload
systemctl enable --now db-backup.timer

# Next and last elapse for this timer
systemctl list-timers db-backup.timer

# Run it now, out of band, without touching the schedule
systemctl start db-backup.service
journalctl -u db-backup.service -n 50 --no-pager

Note that you enable the timer, not the service. Enabling the service instead would try to run the job at every boot, which is a memorable way to learn the difference.

How I’d decide

Work down this list and stop at the first line that applies. It resolves most cases in under a minute.

1. The host might not be systemd, now or later. Use cron. Portability beats every other consideration on this list.
2. The job is inside a container. Use neither. Use your orchestrator’s scheduler, or a timer on the host that invokes the container.
3. A missed run matters, or the job must not overlap, or it needs to wait for another service. Use a timer. Every one of those is a directive rather than a wrapper script.
4. The job is heavy: a dump, a reindex, an import, anything that can pin a core or fill memory. Use a timer, for the cgroup limits and the runtime cap.
5. You need to see output after the fact without building a logging path first. Use a timer. The journal is already there.
6. You need sub-minute scheduling. Use a timer with a monotonic directive. Cron’s floor is one minute.
7. Otherwise, cron is fine, and one line beats two files.

The migration question answers itself from that. Do not convert forty working crontab lines in a weekend. Write new scheduled work as timers, convert the jobs that have actually burned you, and leave the rest. Mixing both on one host is normal and supported. Just never schedule the same job in both places, which is easier to do than it sounds when a configuration management run and a manual edit disagree.

Diagnosing a job that never fired

Cron

Establish what actually happened before you theorise. The daemon logs every job it starts, so the log tells you which of three situations you are in: cron never tried, cron tried and the command failed, or cron never saw the entry at all.

# Is the daemon running at all?
systemctl status crond      # RHEL, AlmaLinux, Rocky
systemctl status cron       # Debian, Ubuntu

# What did it start, and when?
journalctl -u crond --since "24 hours ago"
grep CRON /var/log/syslog | tail -20     # Debian family
tail -50 /var/log/cron                   # RHEL family

# Whose crontab are you actually looking at?
crontab -l -u deploy
ls -l /etc/cron.d/

Then work through the usual suspects, roughly in order of how often they turn out to be the cause:

• Wrong crontab. The job is in root’s crontab and you are reading the deploy user’s, or it’s in /etc/cron.d where there is an extra user field before the command.
• PATH, or an unescaped percent sign. Dump what the job really sees with a temporary entry like * * * * * env > /tmp/cron-env.txt 2>&1, then compare it against your shell.
• Permissions. The script is not executable, or the crontab owner cannot read it.
• The schedule is not what you think. When both day-of-month and day-of-week are restricted, cron runs the job when either matches, not both. 0 0 15 * 1 fires on the 15th and on every Monday.
• The command captured no output, so a real error was mailed into the void. Append >> /var/log/myjob.log 2>&1 and wait for the next run.

systemd

# Is the timer loaded and scheduled? --all shows inactive ones too
systemctl list-timers --all

# Timer state, last trigger, next elapse
systemctl status db-backup.timer

# What the job printed, and how it exited
journalctl -u db-backup.service --since "7 days ago"

# Prove the schedule independently of the unit
systemd-analyze calendar --iterations=3 "Mon..Fri *-*-* 06:30:00"

• The timer is not enabled. Creating the files does nothing. It needs daemon-reload and enable --now.
• You enabled the service instead of the timer. Check with systemctl is-enabled on both.
• A user timer with no lingering. Under systemctl --user, the manager stops when the session ends unless you run loginctl enable-linger for that user.
• The calendar expression is valid but wrong. Valid syntax that never matches will not error. systemd-analyze calendar is how you find out.
• The service failed before your script ran. A missing User=, an ExecStart path that does not exist, a WorkingDirectory that is not there. The journal names the reason.

Arguments that don’t survive contact

“Cron is deprecated.” It isn’t. It is packaged and supported on every mainstream distribution, and control panels, hosting providers and countless application installers write crontab entries as a matter of course. Nobody is removing it.

“Timers are too complicated for small jobs.” Two files is more typing, not more complexity. The complexity that matters is the wrapper script you write to get locking, logging and a runtime cap out of cron, and how well the next person understands it. Compare like for like.

“The journal means I have logging sorted.” The journal is a ring buffer with a size cap. If nobody set SystemMaxUse= deliberately and nobody ships journal entries anywhere else, your evidence has a retention window you never chose. Check it before you rely on it.

“Switching to timers will stop jobs failing silently.” This is the one I’d push back on hardest. Timers make failures visible. They do not make them noticed. Neither scheduler will ever tell you that a job did not run, because absence is not an event. That needs a dead man’s switch: the job pings a heartbeat endpoint on success, and something external alerts when the ping stops. Healthchecks.io, Cronitor and Better Stack all sell exactly this, and Healthchecks is open source if you would rather self-host it next to your own Grafana. If you already run Prometheus, writing a completion timestamp through the node exporter’s textfile collector and alerting on its age gets you the same property without another vendor.

“Just move everything to timers.” A mass migration of working jobs is a lot of change for no new capability, and every converted job is a chance to fat-finger a path. Convert on cause, not on principle.

Frequently asked questions

Are systemd timers better than cron?

For production jobs on a systemd host, generally yes: logging, dependency ordering, overlap protection, catch-up and resource limits are all built in rather than bolted on. For a one-line job on a box you own, cron’s simplicity is a real advantage and the timer buys you nothing you will use.

Is cron deprecated or being removed?

No. Cron implementations are still packaged and maintained across the major distributions. Some distributions do not install a cron daemon in minimal images by default, which is a packaging decision rather than a deprecation, so check with your package manager before assuming a fresh host has one.

Can cron and systemd timers coexist on the same server?

Yes, and most real servers run both. The cron daemon itself is usually managed as a systemd service. The only rule is that a given job should be scheduled in exactly one place. Double-scheduling is what produces the “why did this run twice” ticket.

How do I see when a systemd timer will next run?

systemctl list-timers shows next and last elapse for all active timers; add --all to include inactive ones. To test an expression without creating a unit, use systemd-analyze calendar "Mon..Fri *-*-* 06:30:00" with --iterations=N for more fire times.

How do I stop a cron job from overlapping itself?

Wrap it in flock -n with a dedicated lock file. The -n flag makes the new run exit straight away instead of queuing. Do not use a PID file you wrote yourself; it will leak a stale lock the first time the job is killed, and then the job stops running forever.

Do systemd timers work inside Docker containers?

Not usefully. A container normally runs a single process without systemd as PID 1, so there is no service manager to own a timer. Schedule from outside the container instead: a host timer that runs the container, or the scheduler your platform provides. Podman is the exception worth knowing, since it can generate systemd units for containers and the host’s systemd drives them.

How do I convert a crontab line to a systemd timer?

Move the command into a Type=oneshot service with an absolute ExecStart, then translate the five cron fields into an OnCalendar= expression of the form DayOfWeek Year-Month-Day Hour:Minute:Second. Verify the expression with systemd-analyze calendar, enable the timer, then remove the crontab line only after you have seen a successful run in the journal.

The one thing worth remembering

Cron vs systemd timers is not a contest between an old tool and a new one. Cron is a scheduler. A systemd timer is a scheduler attached to a supervisor, and the supervisor is what you are actually buying: dependency ordering, overlap protection, catch-up, resource limits, and output that lands somewhere you can read it.

Pick timers when a missed or duplicated run costs you something. Pick cron when it doesn’t. And whichever you pick, add a heartbeat, because the failure that hurts is not the job that crashes. It’s the job that stopped running while everything stayed green.

Need a second pair of eyes on your scheduled jobs?

Scheduling is one of those areas where the problem is rarely the syntax. It’s the job nobody has checked since the person who wrote it left. This is the kind of work I take on:

• Auditing every crontab and timer on a host and telling you which jobs have quietly stopped running
• Converting fragile cron entries to systemd timers with locking, runtime caps and failure handlers, without a big-bang migration
• Fixing the classic “works by hand, fails on schedule” environment and PATH problems, including the percent-sign trap
• Adding heartbeat monitoring and alerting so a missed run pages someone instead of sitting there
• Backup and dump jobs that verify their own output rather than exiting zero on an empty file
• Untangling duplicate schedules left behind by control panels, deploy scripts and configuration management

If something on your server isn’t running when it should, send me the crontab line or the unit file and the journal output around the time it should have fired. That’s usually enough to tell you what’s wrong before we talk about anything else.

The post Cron vs systemd Timers: The Failure Modes That Decide It appeared first on John Nessime.

Sometimes the disk really is dying. More often it isn’t, and the number you’re staring at is not measuring what you think it’s measuring.

This post covers troubleshooting high I/O wait the way I’d actually do it on a production box: what the metric means, why it lies in both directions, and how to work through the five failure families that produce it.

What high I/O wait actually measures

Here’s the definition that matters: iowait is a subcategory of idle time. For a given CPU, it’s the time that CPU spent doing nothing while at least one task scheduled on it had an outstanding block I/O request.

Two consequences fall out of that, and both will burn you. High iowait can mean nothing is wrong. If the box has spare CPU and one process is pulling a large file off disk, the CPU genuinely has nothing else to do. Copying a backup will push iowait toward 100 percent on an idle host, and that’s a working system, not a fault.

Low iowait can hide a severe problem. This is the one that costs people days. If the CPU has other runnable work, the kernel context switches to it and that time gets counted as user or system, not iowait. A busy application server can be stalling badly on storage while wa reads two percent, because the CPU is never actually idle.

Add the averaging problem on top. The wa figure in top is averaged across every core. One thread pinned to one core and blocked hard on disk shows up as roughly three percent on a 32-core machine. Press 1 in top to break out per-CPU lines before you conclude anything from the aggregate.

So iowait is a hint, not a verdict. Treat it as the thing that starts the investigation, never the thing that ends it.

Start with pressure, not percentages

Pressure Stall Information is the metric I reach for first, and it exists precisely because iowait is a poor proxy for pain. It answers the question you actually care about: how much time did tasks lose because they couldn’t get I/O?

cat /proc/pressure/io

You get two lines. some is the share of time at least one task was stalled waiting on I/O. full is the share of time every non-idle task was stalled at once, which is CPU capacity being burned for nothing. The avg10, avg60 and avg300 fields are percentages over the last 10, 60 and 300 seconds. total is a cumulative microsecond counter since boot, which is what you want for graphing because averages smooth away short spikes.

The practical rule I use: a rising full line on io is a real user-visible problem almost every time. A high some line with a flat full line usually means one noisy job is waiting while everything else gets on with its life.

If /proc/pressure doesn’t exist, the kernel was built without CONFIG_PSI, or it was built with PSI compiled in but disabled by default, in which case you need psi=1 on the kernel command line. On cgroup v2 systems you also get per-cgroup pressure, which is how you narrow from “the host is stalling” to “this container is stalling” in one command:

# Pressure for one systemd slice, useful on any modern distro
cat /sys/fs/cgroup/system.slice/io.pressure

# Pressure for every leaf cgroup that has it, sorted by nothing,
# just eyeball the full lines
find /sys/fs/cgroup -name io.pressure -exec sh -c 
  'printf "%s: " "$1"; sed -n 2p "$1"' _ {} ;

If you run Prometheus, node_exporter has exposed these since it gained a pressure collector. The counters are node_pressure_io_waiting_seconds_total for some and node_pressure_io_stalled_seconds_total for full. Rate those and you have a far better storage alert than any threshold on %iowait will ever be. Grafana Cloud and Netdata both chart PSI out of the box if you’d rather not build the panels yourself.

Failure family one: the device really is saturated

This is the case everyone assumes, so get it confirmed or eliminated quickly.

iostat -xy 1

The -x gives extended per-device stats. The -y matters more than people realise: without it, the first report is a since-boot average, and reading that first block is the single most common iostat mistake there is. It will tell you the disk has been fine all year, which is true and useless.

Now the columns worth your attention:

• r_await and w_await: average milliseconds a read or write took, queue time included. This is the closest thing to “how slow does storage feel”. Split reads from writes, because they fail very differently.
• aqu-sz: average queue depth. Requests either queued or in service. A depth that climbs while throughput stays flat is the signature of a genuinely saturated device.
• r/s, w/s, rkB/s, wkB/s: the actual workload. Compare against what the device or volume class is rated for.
• %util: treat with suspicion, see below.

The %util trap. That column counts the fraction of time at least one request was in flight. On a spinning disk that serviced one request at a time, it was a fair saturation signal. On NVMe and on cloud volumes that handle many requests in parallel, a device can sit at 100 percent while running at a fraction of its real capacity. Provisioning a bigger volume off the back of a 99 percent %util reading is a common and expensive way to change nothing. Use aqu-sz and the await columns instead. Similarly, svctm was deprecated and dropped from newer sysstat output because it was unreliable on parallel devices; ignore it if an old build still prints it.

Once you know the device is busy, find the culprit. pidstat -d 1 gives per-process read and write rates from the same sysstat package and needs nothing special enabled. iotop -o is friendlier, but on kernels from 5.14 onward per-task delay accounting is off by default, so the SWAPIN and IO> columns come back empty until you turn it on:

sudo sysctl kernel.task_delayacct=1
sudo iotop -oPa
# turn it back off when you're done, it costs a little scheduler overhead
sudo sysctl kernel.task_delayacct=0

And check the boring things before the clever ones. dmesg -T | grep -iE 'i/o error|reset|medium' catches a failing drive or a flapping controller. smartctl -a /dev/sda catches reallocated sectors on hardware you own. If either of those turns something up, stop reading and replace the disk.

Failure family two: memory pressure in an I/O costume

This is the failure family I’d check second, and it’s invisible if you only look at storage.

When RAM gets tight, the kernel starts evicting page cache and swapping. Both generate disk traffic. The disk gets busy, iowait climbs, and every graph points at storage while the actual constraint is memory. Give the box more RAM and the “disk problem” disappears.

# si and so are swap-in and swap-out in KB/s.
# Sustained non-zero values are your answer.
vmstat 1

# Are we thrashing writeback? Dirty pages waiting to be written.
grep -E 'Dirty|Writeback' /proc/meminfo

# The direct question
cat /proc/pressure/memory

If memory pressure and I/O pressure rise together, fix memory first. Everything else is downstream.

There’s a related trap on the write side. The kernel buffers writes in page cache and flushes them in the background, governed by vm.dirty_background_ratio and vm.dirty_ratio. Cross the background threshold and flusher threads start writing. Cross the hard vm.dirty_ratio threshold and writing processes are forced to do writeback themselves, synchronously. That transition is brutal, and on a box with plenty of RAM and a modest disk the defaults let you accumulate gigabytes of dirty pages and then pay for all of them at once. Lowering the ratios makes writeback more frequent and less spiky, trading away a little write batching. Measure before you commit to it.

Failure family three: small synchronous writes

Throughput can be trivial and the system still crawls. A workload doing thousands of tiny fsync calls a second is bound by round-trip latency, not bandwidth, and no amount of extra megabytes per second will help.

Databases are the usual source, because durability has a cost and that cost is a flush. MySQL and MariaDB expose it through innodb_flush_log_at_trx_commit; PostgreSQL through synchronous_commit. Loosening either buys real throughput and trades away durability on power loss or crash. That can be a perfectly reasonable trade for a cache or an analytics replica, and a terrible one for anything holding orders or payments. Decide it explicitly, write down what you decided, and don’t let it be a thing somebody changed once at 2am.

The other common source is logging. Application logs, access logs, container JSON logs and a database sharing one volume will starve each other. Splitting hot write paths onto separate devices is unglamorous and works.

Failure family four: throttled, not saturated

A throttled device and a saturated device look almost identical from inside the guest: high await, growing queue, miserable latency. The difference is that no amount of tuning inside the box will fix a throttle. Three flavours are worth knowing.

• Cloud volume limits. Managed block storage gives you a baseline of IOPS and throughput, sometimes with a burst allowance on top. Burn the burst and you drop to baseline, and everything that was fine in testing falls over in production. The tell is a latency cliff that arrives at a consistent point into a workload rather than tracking load smoothly. On AWS, watch the volume-level CloudWatch metrics such as BurstBalance on burstable volume types and VolumeQueueLength; other providers publish equivalents. The fix is a volume class change, not a sysctl.
• cgroup limits you set yourself. On cgroup v2, io.max caps a cgroup at a given bytes-per-second or IOPS figure, and io.weight shares bandwidth proportionally under contention. A limit that made sense when it was written can become the bottleneck a year later, and nothing in the container’s own metrics announces it.
• Neighbours on shared hardware. On oversubscribed VPS plans your storage is shared, and someone else’s backup job is your latency spike. This is where cheap and expensive hosting genuinely differ. Budget providers like Contabo and InterServer are fine value for a lot of workloads, but if storage latency is load-bearing for you, look for plans with dedicated NVMe rather than shared pools, and test before you migrate anything that matters.

# Per-device I/O accounting for a cgroup, including any throttling
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.stat

# Is a cap set? An empty file means no limit on that cgroup.
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.max

One more signal while you’re here. In top, the st column is steal time, meaning the hypervisor gave your vCPU’s slice to someone else. Steal and iowait rising together on a shared host is a strong hint that the problem is the neighbourhood, not your configuration.

Failure family five: the latency isn’t in the block layer

Here’s the failure mode that wastes the most time, because every storage metric you check will look healthy.

iostat measures the block layer. It cannot see anything above it. So if your latency is in NFS, in a network filesystem, in an overlay or FUSE mount, in filesystem lock contention, or in a journal that’s serialising writes, the device will report sub-millisecond service times while your application waits tens of milliseconds per operation.

The diagnosis is the gap. Measure latency at the application’s layer and at the block layer, and if the two numbers disagree wildly, the problem lives in between.

# Block layer latency as a histogram. Look for a bimodal shape
# or a long tail rather than the average.
sudo biolatency-bpfcc -m 10 1      # Debian/Ubuntu naming
sudo /usr/share/bcc/tools/biolatency -m 10 1   # RHEL family

# Filesystem operations slower than 10ms, with the process and file
sudo ext4slower-bpfcc 10
sudo xfsslower-bpfcc 10

# Everything at the VFS layer, higher overhead, use briefly
sudo fileslower-bpfcc 10

If ext4slower is full of hundred-millisecond operations and biolatency shows everything completing in microseconds, you have your answer and it is not the disk. For NFS specifically, nfsiostat splits out RTT from execution time, which separates network cost from server cost. Install the tooling with bpfcc-tools on Debian and Ubuntu, or bcc-tools on RHEL, AlmaLinux and Rocky.

A working order for troubleshooting high I/O wait

On an unfamiliar box, this is the sequence I work through. Each step either finds the problem or eliminates a family.

1. Read /proc/pressure/io. If full is near zero, the stall is not hurting the whole system and you should widen the search before you tune storage.
2. Check /proc/pressure/memory at the same time. If it’s rising too, fix memory first.
3. Run iostat -xy 1 for thirty seconds. Note r_await, w_await and aqu-sz per device. Ignore %util.
4. List processes stuck in uninterruptible sleep: ps -eo state,pid,comm | awk '$1 ~ /^D/'. That’s your blocked set, and it names the workload.
5. Attribute the traffic with pidstat -d 1. Confirm the process you suspect is the process actually doing the I/O.
6. If await is high, decide saturated versus throttled. Check cgroup io.max, check provider volume metrics, check steal time.
7. If await is low but the application is slow, jump to the layer gap. Run biolatency and ext4slower together and compare.
8. Only now consider tuning: dirty ratios, I/O scheduler, flush settings, splitting volumes.

Common mistakes

• Reading iostat’s first report. It’s a since-boot average. Use -y, or discard the first block.
• Trusting %util on NVMe or cloud volumes. Parallel devices break the assumption it was built on.
• Alerting on a fixed %iowait threshold. It fires on healthy backups and stays silent during real stalls on busy hosts.
• Changing the I/O scheduler first. Going from mq-deadline to none on NVMe is sometimes right, but it’s a tuning step, not a diagnosis, and it fixes nothing if you’re throttled.
• Adding swap to fix swapping. More swap on a memory-starved box means more disk traffic, not less.
• Stopping at the block layer. If every storage metric looks clean and the app is still slow, you’re measuring at the wrong layer, not measuring a healthy system.
• Testing with the wrong benchmark. A sequential dd tells you nothing about a workload doing random 4K synchronous writes. Use fio and match the real access pattern.

Best practices worth the effort

• Record PSI continuously. Storage incidents are short and bursty. Without history you’re guessing about something that already ended.
• Capture a baseline while things are healthy. Save an iostat -xy sample and a PSI reading from a normal day. “Is 8ms bad?” is unanswerable without one.
• Keep sar running. The sysstat package can archive metrics on a schedule, which means sar -d can show you what happened at 3am. Enable it before you need it.
• Separate hot write paths. Database data, database logs and application logs on separate volumes stops one from starving the others.
• Cap container logs. An uncapped JSON log driver quietly filling the root volume is a recurring cause of host-wide stalls.
• Alert on latency and pressure, not utilisation. Users feel w_await and PSI full. Nobody has ever felt a percentage of device busy time.

Frequently asked questions

What is a normal or acceptable I/O wait percentage?

There isn’t one, and any article that gives you a single number is guessing. Iowait scales with how idle your CPUs are, so the same workload produces very different figures on a busy server and a quiet one. Judge against your own baseline, and prefer PSI, which measures lost time directly rather than as a share of idle.

Can high I/O wait be caused by something other than the disk?

Regularly. Memory pressure driving swap and page cache eviction is the most common non-disk cause. Network filesystems, hypervisor contention on shared hosts, and cgroup or cloud volume throttling all produce the same symptom. The block device being genuinely slow is one possibility among several.

Why does iostat show 100% utilisation when the disk isn’t busy?

Because %util measures the fraction of time at least one request was in flight, which made sense for devices that handled one request at a time. NVMe drives and cloud volumes process many requests in parallel, so they can be continuously occupied while nowhere near capacity. Read aqu-sz and the await columns instead.

How do I find which process is causing I/O wait?

Start with pidstat -d 1, which needs no special kernel settings. List processes in uninterruptible sleep with ps -eo state,pid,comm and filter for state D to see who is actually blocked. iotop -o gives a live view but needs kernel.task_delayacct=1 on recent kernels for its percentage columns to populate.

Does adding more RAM reduce I/O wait?

Often, yes, and it’s underrated. More RAM means a larger page cache, fewer reads reaching the device, and less swapping. If vmstat shows sustained swap activity or memory pressure tracks I/O pressure, RAM is likely the cheapest fix available.

Should I change the I/O scheduler to fix high I/O wait?

Only after you know the device is the bottleneck. Check the current setting with cat /sys/block/sda/queue/scheduler. On NVMe, none is a common and sensible default because the hardware already does the queuing. On spinning disks, mq-deadline or bfq give better latency behaviour. It’s a tuning knob, not a diagnostic, and it will not help if you’re being throttled.

Why is my application slow when iostat says storage is fine?

Because iostat only sees the block layer. Latency in NFS, FUSE, overlay filesystems, journal serialisation or filesystem locking is entirely invisible to it. Run ext4slower or xfsslower alongside biolatency: if the filesystem is slow and the device is fast, the gap between them is your problem.

The one thing worth remembering

High I/O wait is a symptom that points in a general direction, not a diagnosis. It’s a slice of idle time, averaged across cores, measured at one specific layer of the stack. It goes up when nothing is wrong and stays flat when something is very wrong.

Read PSI to find out whether anyone is actually suffering. Read r_await, w_await and aqu-sz to find out whether the device is the constraint. Compare filesystem latency against block latency to find out whether you’re even looking at the right layer. Do those three things and the answer usually falls out in a few minutes, instead of after a week of arguing with your hosting provider about a volume that was fine all along.

Need someone to look at your storage stalls?

I work with teams on exactly this kind of problem, usually the ones that have already survived a round of “the host says the disk is fine”. Things I can help with:

• Diagnosing a live or recurring I/O stall and identifying which failure family it belongs to
• Setting up PSI, sar and per-cgroup I/O metrics in Prometheus and Grafana, with alerts that fire on latency rather than utilisation
• Separating cloud volume throttling from genuine device saturation, and sizing storage correctly instead of guessing upward
• Tuning writeback, dirty ratios and I/O schedulers for a specific workload, with before-and-after measurements
• Tracing container and Kubernetes I/O contention through cgroup v2 limits, overlay filesystems and log drivers
• Benchmarking with fio against your real access pattern so capacity decisions rest on something other than a dd run

If you’ve got an iostat capture, a PSI reading or just a graph that doesn’t make sense, send it over and I’ll tell you what I see in it.

The post Troubleshooting High I/O Wait Without Blaming the Disk appeared first on John Nessime.

That is the real problem with trying to bootstrap a VPS with Ansible in a single run. The failures that cost you are not the ones that turn the recap red. They are the tasks that report changed, do nothing useful, and leave you believing the server is hardened when it is not.

This post covers the ordering that keeps you connected while you dismantle your own access, the silent failure modes that only surface on reboot, and how to make the second run of the playbook as safe as the first. The examples are Debian and Ubuntu flavoured with notes for RHEL-family images, and they assume a plain provider VPS from somewhere like InterServer, Hetzner, DigitalOcean or Vultr, not a pre-baked golden image.

What a one-run bootstrap actually has to survive

A bootstrap playbook is unusual because it modifies the thing it is standing on. Every other playbook you write assumes a working connection. This one has to change the user it connects as, the port it connects to, and the firewall that lets it connect at all, without dropping the run.

Four things break it, and they break it in different ways:

• The chicken-and-egg problem. Ansible needs Python on the target to run almost any module, and a minimal image may not have it.
• Lockout. You remove your own access before proving the replacement access works.
• Non-idempotent tasks. The first run is fine. The second run is where the damage happens.
• Missing collections. The playbook that worked on your laptop fails on the CI runner because half the modules do not ship with ansible-core.

Everything below is organised around those four, because that is how they actually show up.

The chicken-and-egg problem: no Python, no modules

Most modern cloud images ship Python 3 and this never comes up. Minimal Debian netinstall images and some stripped provider templates do not, and then your very first task fails with a message about the interpreter before it has done anything at all.

The escape hatch is ansible.builtin.raw, which pipes a command straight down the SSH connection without needing Python on the other end. Pair it with gather_facts: false, because fact gathering is itself a Python module and will fail before your bootstrap task ever runs.

- name: Make the host manageable
  hosts: new_servers
  gather_facts: false
  become: true
  tasks:
    - name: Check whether a Python interpreter exists
      ansible.builtin.raw: test -e /usr/bin/python3
      register: python_present
      changed_when: false
      failed_when: false

    - name: Install Python if it is missing
      ansible.builtin.raw: apt-get update && apt-get install -y python3
      when: python_present.rc != 0
      changed_when: true

    - name: Gather facts now that we can
      ansible.builtin.setup:

Two details worth understanding rather than copying. failed_when: false on the check turns a non-zero exit into data instead of a failure, so you can branch on it. And ansible.builtin.setup is the fact-gathering module called explicitly, which means from that point on ansible_distribution and friends are available even though the play started with gathering disabled.

If you are provisioning through a provider that supports user-data, doing this in cloud-init instead is cleaner. Let cloud-init handle first-boot truths, then let Ansible own everything that will change later.

Failure family one: locking yourself out while the playbook says ok

This is the one that actually costs a rebuild, and it is entirely an ordering problem. The tasks are all correct in isolation. Run them in the wrong sequence and the playbook cheerfully removes your way back in before it has confirmed the replacement works.

The order that survives

The rule is simple: nothing that removes access happens until something has proven the new access works. In practice:

1. Connect as whatever the provider handed you, usually root or a sudo-capable default user.
2. Create the admin account, add it to the sudo or wheel group, install its authorized key.
3. Write the sudoers drop-in and validate it with visudo -cf.
4. Prove you can log in as that account and escalate. This is the step everyone skips.
5. Install the firewall and allow SSH before enabling it.
6. Write the sshd configuration, validate it, restart, reconnect on the new port and confirm.
7. Only now disable root login and password authentication.

Step four is a real task, not a comment. It runs from the control node against the target, using the credentials you are about to depend on:

- name: Prove the new admin account can log in and escalate
  ansible.builtin.command: id -u
  become: true
  changed_when: false
  vars:
    ansible_user: "{{ admin_user }}"
    ansible_ssh_private_key_file: "{{ admin_key_path }}"

Setting connection variables at the task level means this single task uses the new identity while the rest of the play carries on as root. If the key is wrong or sudo is misconfigured, the play stops here with root login still intact, which is exactly what you want.

The socket activation trap

This is the failure that passes green and bites on reboot. On Ubuntu images from 22.10 onward, sshd is started through systemd socket activation rather than as a persistent service. ssh.socket owns the listening port, and Port in sshd_config is not what decides where the daemon listens.

So your playbook writes Port 2222, restarts the service, reports changed, and the box carries on listening on 22. Your existing session still works. Your firewall rule for 2222 looks fine. Nothing indicates a problem until something forces a reconnect.

The fix is to write a socket override as well, and to check which mechanism the image actually uses rather than assuming:

# Is the socket unit in play on this host?
systemctl is-enabled ssh.socket

# What is actually listening, and under which unit?
ss -tlnp | grep -E 'ssh|:22'

If ssh.socket is enabled, the override needs an empty ListenStream= first to clear the inherited default, then the real value. Miss the empty line and the host listens on both ports, which is worse than either outcome alone because your hardening looks applied and is not.

Honestly, the simpler answer for most single-server setups is to leave SSH on 22 and lean on key-only authentication plus a firewall source restriction. Moving the port cuts log noise from opportunistic scanners. It does not stop anyone who is actually looking at you.

Firewall ordering

UFW defaults to denying inbound traffic when you enable it. Enable first, add the SSH rule second, and the run dies mid-task with an UNREACHABLE that will not recover. Rules go in first, always:

- name: Allow SSH before the firewall is armed
  community.general.ufw:
    rule: allow
    port: "{{ ssh_port }}"
    proto: tcp

- name: Set default inbound policy
  community.general.ufw:
    direction: incoming
    policy: deny

- name: Enable UFW
  community.general.ufw:
    state: enabled

Two extra things bite here. Provider-level firewalls, the ones configured in the control panel rather than on the host, are a second layer that Ansible has no visibility into. If you open a port on the host and traffic still does not arrive, check there before debugging iptables. And if you moved the SSH port, the host firewall rule and the provider firewall rule both need updating, in that order, before the daemon restarts.

Failure family two: the second run is the dangerous one

A bootstrap playbook you run once is a shell script with extra syntax. The value only appears when you can run it again on the same host, in six months, and have it either do nothing or fix drift. That is idempotency, and the usual killers are shell and command tasks with no guard.

An unguarded shell task reports changed every single time, which means any handler it notifies fires every single time. That is how a playbook you thought was a no-op ends up restarting sshd on a production box during a routine drift check.

Guard them properly:

• creates: or removes: makes the task skip when the target state already exists. Prefer this.
• changed_when: false for read-only commands so they stop lying about changing things.
• when: driven by a registered check, for anything conditional.

The other idempotency trap is lineinfile against sshd_config. It works, and it is fragile in a specific way: the regex has to match the shipped file, and different distributions ship different defaults, commented or not. Writing a drop-in file is more predictable:

- name: Write the sshd hardening drop-in
  ansible.builtin.template:
    src: 99-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/99-hardening.conf
    owner: root
    group: root
    mode: "0600"
  notify: Restart sshd

- name: Validate the full sshd configuration
  ansible.builtin.command: sshd -t
  changed_when: false

Note that the validation runs against the whole configuration rather than the drop-in in isolation. A drop-in is not a valid standalone config file, so validate: on the template task with sshd -t -f %s will fail on a temporary file that has no host keys and no context. Run sshd -t as its own task afterwards instead, and let the handler restart only if it passes.

Two prerequisites for the drop-in approach. The base sshd_config needs an Include /etc/ssh/sshd_config.d/*.conf line, which recent Debian, Ubuntu and RHEL 9 images have but older ones may not. And in sshd configuration the first value for a keyword wins, so the include has to appear near the top of the file for your drop-in to override anything below it. Check both before relying on the mechanism.

Failure family three: the collections are not where you think

ansible-core ships the ansible.builtin namespace and essentially nothing else. The full ansible package bundles a large set of community collections, which is why a playbook that runs on your laptop can fail on a lean CI runner with a module-not-found error.

Things people assume are built in and are not: ufw and timezone live in community.general. authorized_key, sysctl and mount live in ansible.posix. Declare them and pin them:

# requirements.yml
collections:
  - name: community.general
  - name: ansible.posix

# install before the run, and in CI
ansible-galaxy collection install -r requirements.yml

Use fully qualified names everywhere. Short names still resolve, but they resolve through a lookup that can pick a different module when two collections define the same name. ansible-lint flags this and it is worth listening to.

Failure family four: secrets in the repository

Bootstrap playbooks accumulate secrets faster than anything else you write: initial passwords, API tokens for monitoring agents, TLS material, backup credentials. They also tend to be the first repository someone shares with a client.

Ansible Vault is the low-friction option and it is genuinely fine for a small estate. Encrypt the variable file, not the playbook, so diffs stay readable:

ansible-vault encrypt group_vars/all/vault.yml
ansible-playbook site.yml --ask-vault-pass

The honest trade-off: Vault gives you one shared passphrase with no rotation story and no audit trail. Once more than two people run the playbook, or a client needs their own credentials kept separate from yours, move to something with real access control. That is more moving parts, so do not reach for it on day one.

Putting it together: how to bootstrap a VPS with Ansible end to end

Structure matters less than order, but a shape that works well is a single site.yml calling roles in dependency order, with the risky access changes fenced off at the end.

# inventory.yml
new_servers:
  hosts:
    web01:
      ansible_host: 203.0.113.10
      ansible_user: root
      ansible_port: 22
  vars:
    admin_user: deploy
    ssh_port: 22

Then the run itself, in this order:

1. Reachability. Python check, fact gathering, timezone, hostname, package cache refresh.
2. Identity. Admin user, group membership, authorized key, sudoers drop-in with visudo -cf validation.
3. Verification gate. The task shown earlier. Nothing destructive runs before this passes.
4. Firewall. Rules first, policy second, enable third.
5. Unattended updates. unattended-upgrades on Debian and Ubuntu, dnf-automatic with its systemd timer on RHEL-family hosts.
6. Brute-force protection. fail2ban with your own /etc/fail2ban/jail.local, never edits to jail.conf.
7. Workload. Docker, a web server, a monitoring agent, whatever the box is for.
8. Access teardown. Disable root SSH login, disable password authentication, restart sshd, reconnect, confirm.

When step eight changes the port or user, the play needs to actually reconnect rather than reusing its cached SSH connection. Ansible keeps a persistent connection per host, so a config change alone will not be picked up:

- name: Point subsequent tasks at the new identity
  ansible.builtin.set_fact:
    ansible_user: "{{ admin_user }}"
    ansible_port: "{{ ssh_port }}"

- name: Drop the cached connection
  ansible.builtin.meta: reset_connection

- name: Confirm the host answers on the new settings
  ansible.builtin.wait_for_connection:
    timeout: 60

meta: reset_connection does not accept a when conditional, so it runs on every pass. That is a small cost for a play that works on the first run and every run after.

Test it before you point it at anything real

The cheapest test is a disposable VPS. Spin one up, run the playbook, reboot it, run the playbook again, then destroy it. The reboot is the important part, because that is what exposes the socket activation problem and any service you enabled but never actually started.

# catch YAML and structural errors instantly
ansible-playbook site.yml --syntax-check

# style and correctness, including FQCN and idempotency smells
ansible-lint

# show what would change without changing it
ansible-playbook -i inventory.yml site.yml --check --diff

One caveat on --check: it is not fully reliable on a bootstrap playbook. Tasks that depend on earlier tasks having run will fail or report nonsense in check mode, because the earlier change did not really happen. Treat a clean check run as a weak signal, not proof.

Troubleshooting

• UNREACHABLE on the very first task. Usually host key verification against a rebuilt host reusing an old IP. Remove the stale entry with ssh-keygen -R and re-add the correct one with ssh-keyscan. Disabling host key checking globally makes this go away and removes a real protection, so do not.
• UNREACHABLE immediately after a firewall task. You armed the firewall before allowing SSH. Recover through the provider console, not SSH.
• “Missing sudo password”. The sudoers drop-in is not in effect or has bad syntax. Validate with visudo -cf on the file itself before trusting it.
• Module not found under an FQCN. The collection is not installed on the machine running the playbook. Confirm with ansible-galaxy collection list.
• New group membership ignored. Adding a user to docker mid-play does not affect the already-open connection. Insert meta: reset_connection after the group change.
• fail2ban installed but not banning. On images without rsyslog, /var/log/auth.log may not exist, so a jail configured to read it never starts. Check fail2ban-client status and switch the jail to the systemd backend if the log file is absent.
• Everything green, port unchanged. Socket activation. Check systemctl is-enabled ssh.socket and ss -tlnp.

Common mistakes

• Disabling root login in the same play that creates the replacement user, with no verification step between them.
• Enabling UFW before the SSH allow rule exists.
• Assuming Port in sshd_config is authoritative on every distribution.
• Unguarded shell tasks that notify handlers, causing service restarts on every run.
• Editing jail.conf instead of creating jail.local, so the next package update silently reverts it.
• Committing plaintext secrets because “it is only a bootstrap playbook”.
• Testing on a host you cannot afford to lose, with no console access configured.
• Relying on --check as a substitute for a real run on a disposable host.

Best practices

• Know your recovery path before the first run. Provider console, serial console or recovery mode, tested once so you are not learning it under pressure.
• Put a verification task before every irreversible one, and let the play fail there rather than after.
• Prefer drop-in files over line edits for sshd, sudoers, fail2ban and sysctl. They survive package upgrades and are trivially removable.
• Validate configuration before restarting the service that reads it. sshd -t, visudo -cf, nginx -t.
• Pin collections in requirements.yml and install them in CI, not just locally.
• Run the playbook twice on a fresh host and read the second recap. Anything still reporting changed is a bug.
• Reboot once during testing. Half of the silent failures only appear after a restart.
• Get a monitoring agent on the box in the same run. A server that is hardened but unobserved is a server you will hear about from a customer first.

Frequently asked questions

Can you really bootstrap a VPS with Ansible in a single playbook run?

Yes, including changing the SSH user and port mid-run. The mechanism is set_fact to update the connection variables, meta: reset_connection to drop the cached connection, and wait_for_connection to confirm the new settings work before continuing. What you cannot do safely is change access and skip the confirmation.

Should I use cloud-init or Ansible for initial server setup?

Both, split by lifetime. cloud-init handles things that are true exactly once at first boot: the admin user, the SSH key, a minimal package set. Ansible handles everything that will be tuned again later. Putting hardening rules in cloud-init means the only way to change them is to rebuild the host.

Do I need Terraform or OpenTofu as well?

Only if you are creating the infrastructure, not just configuring it. Terraform and OpenTofu create the VPS, the DNS records and the provider firewall. Ansible configures what runs inside. For a handful of long-lived servers ordered through a provider panel, Ansible alone is enough and adding a second tool costs more than it returns.

Why does my playbook work the first time and fail the second?

Almost always because the first run changed the connection parameters and the inventory still describes the original ones. Either update the inventory to the post-bootstrap values and make the play tolerate both, or split bootstrap and steady-state into separate playbooks with separate inventory entries.

Is changing the SSH port worth doing?

It reduces log noise from untargeted scanning, which makes real events easier to spot. It is not a security control on its own. Key-only authentication, a default-deny firewall and fail2ban do the actual work. If moving the port complicates your automation, the honest answer is to leave it on 22.

How do I recover if the playbook locks me out?

Through the provider’s console or recovery environment, which is why you should know where it is before you need it. Most VPS providers offer a web-based console, some offer a rescue boot with the disk mounted. From there, revert the offending drop-in file or firewall rule, restart the service, and reconnect. Then add the verification step the playbook was missing.

Can this run against several servers at once?

Yes, but use serial: 1 the first few times. If there is an ordering bug in the play, running it against one host at a time means you lock yourself out of one server instead of a fleet.

The one thing worth remembering

When you bootstrap a VPS with Ansible, the individual tasks are the easy part. Creating a user, writing a firewall rule and hardening sshd are all well-documented and hard to get wrong in isolation. What decides whether the run leaves you with a working server or a rebuild is the sequence, and specifically whether you verify new access before removing old access.

Add the verification gate. Run the playbook twice. Reboot the box once. Those three habits catch nearly every silent failure described here, and they cost about ten minutes on a disposable host.

Need a bootstrap playbook you can actually rerun?

Most of the provisioning work I take on starts with a playbook that worked once and has been too frightening to run since. Things I help with:

• Building a bootstrap role that takes a blank VPS to a hardened, monitored server in one run, with a verification gate before every irreversible step.
• Auditing an existing playbook for idempotency, so the second and hundredth runs are as safe as the first.
• Untangling SSH, firewall and socket activation interactions on mixed Debian, Ubuntu and RHEL-family estates.
• Splitting cloud-init and Ansible responsibilities cleanly so hardening stays changeable after first boot.
• Moving secrets out of a repository into Vault or a managed secret store without breaking existing runs.
• Wiring the playbook into CI with linting, collection pinning and a disposable-host test that runs on every change.

Send me the playbook, a recap from a second run, or the output of ss -tlnp from the host that is behaving oddly, and I will tell you what I would change first.

The post Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out appeared first on John Nessime.

The worse version of this problem is the mirror image. You reboot a server to clear something, come back to read what led up to it, and the journal starts at the reboot. Everything before is gone. Not rotated, not compressed, not archived somewhere clever. Gone, because the journal was living in RAM and nobody ever told it otherwise.

Both of those are the same misunderstanding wearing different clothes: not knowing where your logs actually live. This post walks through the journald vs rsyslog split on a modern Linux server, what each one really stores, the three failure modes that bite in production, and how to decide which of the two is the copy you keep.

What actually happens when a process logs a line

On any systemd-based distribution, systemd-journald is the first stop for almost everything. It is not one option among several. It sits underneath.

• Anything a unit writes to stdout or stderr, because systemd wires those to the journal by default.
• Kernel messages, read from the kernel ring buffer.
• Classic syslog(3) calls from libc, which land on the /dev/log socket that journald owns.
• Native journal API calls from anything linked against libsystemd, which carry structured key-value fields instead of a flat string.
• Audit records, if Audit=yes is in effect.

journald writes all of that into indexed binary journal files. Then, on most server distributions, rsyslog gets a second copy of the same events and writes them out as text into /var/log/messages, /var/log/secure, /var/log/maillog and friends.

That handoff happens one of two ways, and which one your box uses matters more than most people expect. More on that below. First, the part that costs you an outage.

Before changing anything, look at the effective journald configuration rather than the file you think is authoritative. Distributions ship drop-ins under /usr/lib/systemd/journald.conf.d/ that quietly override upstream defaults:

# Print the merged configuration, including every drop-in, in load order
systemd-analyze cat-config systemd/journald.conf

This is the single most useful command in this whole post. Half the arguments about “what the default is” evaporate once you run it, because upstream systemd and your distribution frequently disagree about ForwardToSyslog.

Failure one: the journal that was never on disk

journald’s Storage= setting defaults to auto. That word does more work than it looks like. Under auto, journald writes to /var/log/journal/ only if that directory already exists. If it does not, journald falls back to /run/log/journal/, which is tmpfs. Memory. Wiped on reboot.

Nothing warns you. journalctl works fine, colours are pretty, filters work. You just silently have no history.

The quickest test is boot history. If the journal is persistent you see multiple boots; if it is volatile you see exactly one:

journalctl --list-boots

To make it persistent, create the directory, let systemd-tmpfiles apply the correct ownership and ACLs, then restart the daemon and flush anything still held in RAM:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
journalctl --list-boots

The systemd-tmpfiles step is the one people skip. The journal directory needs specific group ownership and ACLs so that unprivileged users can read their own entries and members of the systemd-journal group can read everything. Creating the directory with a bare mkdir and walking away usually works, but it is the kind of “usually” that produces a confusing permissions ticket six months later.

If you want the behaviour to be explicit rather than inferred from a directory’s existence, set it in /etc/systemd/journald.conf.d/00-storage.conf:

[Journal]
Storage=persistent

A drop-in file is better than editing the main config, because package upgrades will not fight you over it.

Failure two: paying for the same log line twice

Once the journal is persistent, you are storing everything twice on a default server build: once as compressed binary in the journal, once as text under /var/log/. Two stores, two completely separate retention policies, neither of which knows the other exists.

The journal is capped by SystemMaxUse= and SystemKeepFree=. Left unset, they default to 10 percent and 15 percent of the filesystem respectively, and journald honours whichever is stricter. Recent systemd also caps those calculated defaults so they do not grow without bound on a large disk, which is worth knowing if you have ever wondered why a huge volume did not produce a proportionally huge journal.

The text copy is capped by logrotate, configured somewhere completely different, usually /etc/logrotate.d/rsyslog, on a weekly or daily schedule with its own rotate count.

Measure both before you tune either:

# What the journal is actually consuming on disk
journalctl --disk-usage

# What everything under /var/log costs, sorted, biggest last
sudo du -sh /var/log/* | sort -h

On a small VPS this stops being academic quickly. A modest instance from a provider like InterServer, Hetzner or DigitalOcean often has /var sharing a single root volume of 20 to 40 GB. A journal allowed to take 10 percent of that, plus four weeks of rotated text logs, plus a database, plus a container image cache, is how you end up paged at two in the morning for something that is not a real incident.

If you decide the text files are the copy you keep, put the journal on a short leash:

[Journal]
# Hard ceiling on total journal size
SystemMaxUse=200M
# Never let the journal push free space below this
SystemKeepFree=1G
# Discard entries older than this regardless of size
MaxRetentionSec=2day

Those three do different jobs and you generally want at least two of them. SystemMaxUse bounds the journal itself. SystemKeepFree protects the rest of the filesystem from the journal. MaxRetentionSec bounds it in time, which is what auditors and data-retention policies actually care about.

To reclaim space immediately without waiting for the next rotation:

# Trim archived journal files until the total falls below 500M
sudo journalctl --vacuum-size=500M

# Or trim by age instead
sudo journalctl --vacuum-time=7d

One caveat that surprises people: vacuuming only removes archived journal files. The currently active file is never deleted. If almost all your usage is in one large active file, run journalctl --rotate first, then vacuum.

Failure three: two rate limiters, both silent

This is the one that is invisible until it matters, and it is the real reason to understand the journald vs rsyslog relationship rather than treating them as interchangeable.

There are two independent rate limiters in the default pipeline, and both drop messages quietly.

journald’s limiter

journald applies RateLimitIntervalSec= and RateLimitBurst= per service, defaulting to 30 seconds and 10000 messages. Exceed the burst inside the interval and the rest of that service’s messages in that window are discarded. Not queued. Discarded.

The one piece of good news is that journald tells you, in the journal itself. Grep for it:

journalctl --grep="Suppressed" -n 50 --no-pager

A chatty web server or a container using the journald log driver will hit this. Rather than disabling rate limiting globally, override it for the one unit that needs it, using a drop-in:

# /etc/systemd/system/nginx.service.d/logging.conf
[Service]
LogRateLimitIntervalSec=30s
LogRateLimitBurst=50000

Per-unit values override journald.conf for that unit only, which is exactly what you want. Turning the global limiter off means one runaway process can fill your disk and take the box down. Turning it up for the single service you know is noisy is a bounded decision.

rsyslog’s limiter

If rsyslog is reading via the imjournal module, it applies its own rate limit on top, defaulting to 20000 messages per 600 seconds. A message can therefore survive journald’s limiter, land in the journal, and still never reach /var/log/messages or your remote log host.

That is the nasty case, because your local investigation finds the events but your centralised alerting never fired. Two people look at two sources and reach opposite conclusions about whether something happened.

rsyslog exposes counters for this through its statistics module, including how many messages it read from the journal, how many it submitted onward, and how many it discarded to rate limiting. Enable impstat if you run anything log-volume-sensitive; the discard counter is the number that tells you whether your pipeline is lying to you.

How rsyslog gets its copy: imjournal or imuxsock

There are two mechanisms, they behave differently under load, and running both at once is a classic source of duplicate lines.

imuxsock: journald pushes a copy to a socket

With ForwardToSyslog=yes in journald, journald writes a classic syslog-formatted copy of each message to a dedicated socket, and rsyslog reads it with imuxsock.

• Where it wins: simple, no state file, no second rate limiter, and no possibility of rsyslog reading back its own output and looping.
• Where it does not: you get the flat syslog view only. Structured journal fields such as the originating unit are not carried across.

imjournal: rsyslog pulls from the journal

imjournal reads journal files directly and keeps a state file so it can resume where it left off after a restart.

• Where it wins: structured fields survive the trip. If you want to route or template on the systemd unit name rather than parsing it out of a message string, this is the only option of the two.
• Where it does not: extra rate limiter to reason about, a state file that can drift, and a documented risk that a corrupted journal database causes rsyslog to re-read the same entries in a loop. rsyslog’s own documentation recommends using imuxsock instead unless you specifically need the structured data.

A representative imjournal load line, with the rate limit stated explicitly rather than left implicit:

module(load="imjournal"
       StateFile="imjournal.state"
       Ratelimit.Interval="600"
       Ratelimit.Burst="20000")

Writing the defaults out by hand is worth the two extra lines. The next person to debug missing messages will see the limiter exists instead of assuming there isn’t one.

To find out which mechanism your box is using right now:

grep -R "imjournal|imuxsock" /etc/rsyslog.conf /etc/rsyslog.d/ 2>/dev/null
grep -R "ForwardToSyslog" /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ /usr/lib/systemd/journald.conf.d/ 2>/dev/null

If the first command returns both modules and the second returns ForwardToSyslog=yes, you have a duplication problem. Pick one path and disable the other.

Only one of the two leaves the box

This is the argument that settles most journald vs rsyslog debates in practice. journald is a local store. In a stock server install it has no mechanism for shipping logs to another host. There are separate systemd components for journal upload and reception, and third-party projects that read the journal and forward it, but none of that is in the default path.

rsyslog forwards as a first-class feature, and it does the part that actually matters: it buffers when the destination is unreachable.

global(workDirectory="/var/spool/rsyslog")

action(type="omfwd"
       target="logs.example.net"
       port="514"
       protocol="tcp"
       action.resumeRetryCount="-1"
       queue.type="LinkedList"
       queue.filename="fwd_loghost"
       queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on")

Line by line, because these are not decoration:

• workDirectory is where spool files get written. Without it, the disk queue has nowhere to go.
• protocol="tcp" rather than UDP. UDP syslog silently drops under congestion, which defeats the point of forwarding at all.
• queue.type="LinkedList" makes the action asynchronous so a slow destination does not block local processing.
• queue.filename is what actually enables disk assistance. In-memory queue first, spilling to disk when it fills. The name must be unique across every action in the config.
• queue.maxDiskSpace caps that spool. Set it, or a long outage at the far end fills the disk you were trying to protect.
• queue.saveOnShutdown="on" persists whatever is still queued across a service restart.

One honest caveat on action.resumeRetryCount="-1". Infinite retry is the right default for “never lose a log line”, but there are reported cases where a relay under sustained back-pressure stops accepting new inbound messages while retrying forever. If your host is a relay rather than a leaf, test that behaviour deliberately with the destination firewalled off before you trust it.

rsyslog is not the only way off the box. Grafana Alloy into Loki, Vector, and Fluent Bit all read the journal directly and speak modern backends, and hosted platforms such as Better Stack, Papertrail or Datadog will take a plain syslog stream if you would rather not run the storage side. The mechanism differs; the decision does not. Something has to own the write path off the machine, and journald on its own is not it.

journald vs rsyslog: how I would decide

Stop asking which is better. Ask which one is the copy you are willing to be judged on, then make the other one cheap.

1. Do logs need to leave this machine? If yes, rsyslog or a dedicated shipper owns the outbound path. That is settled before anything else.
2. Do you troubleshoot mostly by unit? If journalctl -u something -b is your muscle memory, journald is your primary read path and should get the retention budget.
3. Do existing tools read text files? Fail2ban, logwatch, older SIEM collectors and a lot of home-grown scripts parse /var/log/*. Turning rsyslog off breaks them quietly.
4. How much disk do you actually have? Under about 40 GB, keeping two full copies is an unforced error. Cap one hard.
5. Is tamper-evidence a requirement? journald supports Forward Secure Sealing, which detects after-the-fact modification of journal files. Shipping to an append-only remote store is the more commonly accepted answer, and the two are not mutually exclusive.

The configuration I reach for first on a general-purpose server: journald persistent with a hard cap for a few days of fast local querying, rsyslog kept for the syslog files that other tools expect plus reliable forwarding off-box, imuxsock as the handoff unless something genuinely needs structured fields, and logrotate retention trimmed down because the remote copy is the one that matters for anything older than a week.

Troubleshooting

• Journal only shows the current boot. Volatile storage. journalctl --list-boots confirms it, and the fix is the persistent-storage sequence above.
• Events in journalctl but not in /var/log/messages. Either rsyslog is not reading from the journal at all, or the imjournal rate limiter is discarding. Check which input module is loaded first.
• Every line appears twice in the text logs. Both imjournal and imuxsock are active while ForwardToSyslog=yes. Disable one path.
• Bursts of messages disappear during incidents. That is a rate limiter, and incidents are exactly when services get loud. Search for suppression notices and raise the limit for that specific unit.
• journalctl reports corruption or behaves oddly. Run journalctl --verify. If files are damaged, journalctl --rotate starts a fresh active file so new writes are clean, then vacuum the bad archives.
• “Journal has been rotated since unit was started.” Not an error. Rotation happened mid-session, so journalctl -u cannot map the full range. Re-run the query without the unit filter or widen the time range.
• Disk still full after vacuuming. Vacuum skips the active journal file. Rotate first, then vacuum again.

Common mistakes

• Assuming the journal is persistent because journalctl returns results. It always returns results. The question is how far back.
• Disabling rsyslog on a box where fail2ban, logwatch or a scraper still reads /var/log/secure. Nothing errors. Detection just stops.
• Setting RateLimitBurst=0 globally to “stop losing logs”, then having a crash-looping service fill the disk in an afternoon.
• Editing /etc/systemd/journald.conf directly and being surprised when a vendor drop-in overrides it. Use a drop-in of your own with a name that sorts later.
• Forwarding over UDP because it is the one-line version. It drops silently under exactly the load that produced the logs you wanted.
• Enabling a disk-assisted queue without queue.maxDiskSpace, turning a remote outage into a local disk-full outage.
• Tuning journald limits on a machine whose real problem is that one application logs every health check at info level. Fix the source first.

Best practices

• Make storage explicit. Storage=persistent or Storage=volatile, never left to auto on a server you care about.
• Always set SystemKeepFree alongside SystemMaxUse. The first bounds the journal, the second protects everything else on the volume from it.
• Decide consciously which store is authoritative, and write it in the runbook. Two stores with no stated owner means neither gets maintained.
• Use exactly one journal-to-rsyslog path. Both is duplication; neither is silent data loss.
• Override rate limits per unit, not globally.
• Monitor journalctl --disk-usage as a metric, not as something you check after the alert.
• Ship off-box with TCP and a bounded disk-assisted queue, and test it with the destination firewalled off before you rely on it.
• Keep the local copy short and the remote copy long. Local disk is the expensive place to store history.

Frequently asked questions

Can I just disable rsyslog and use journald only?

On a self-contained box with no remote logging and nothing parsing text files, yes, and it saves real disk. Before you do it, grep your configuration management and cron jobs for /var/log/ paths, and check whether fail2ban or any monitoring agent reads those files. That is where this bites people.

Can I disable journald and use rsyslog only?

Not meaningfully. journald is how systemd captures unit stdout and stderr, so it stays in the path regardless. What you can do is set Storage=volatile with a small RuntimeMaxUse so it holds a short in-memory window and rsyslog owns everything durable.

Where are journald logs stored?

/var/log/journal/ when persistent, /run/log/journal/ when volatile. They are indexed binary files, not text, so grep and tail do not work on them directly. Use journalctl, or journalctl -o json if you want to pipe structured output into something else.

Why does journalctl show entries that never reached /var/log/messages?

Three usual causes: rsyslog is not reading the journal at all, the imjournal rate limiter dropped them, or a severity filter such as MaxLevelSyslog or an rsyslog rule excluded that facility or priority. Check in that order.

Is the binary journal format a problem for log analysis?

Locally, no. Structured fields and fast filtering are genuinely better than parsing text with regex. It becomes a problem when you need to move logs somewhere else, because almost every collector expects text or JSON. That conversion step is precisely the role rsyslog or a modern shipper plays.

How much disk should the journal be allowed to use?

Work backwards from how far back you actually query locally. If you rarely look past yesterday, a few hundred megabytes with MaxRetentionSec set to two or three days is plenty. Anything older belongs in a remote store where it is cheaper and survives the machine.

Does journald compress logs automatically?

Yes. Compress= defaults to enabled and applies to entries above a size threshold, which is one reason the journal is often smaller than the equivalent text files despite carrying more metadata. It does not remove the need for a size cap.

The one thing worth remembering

The journald vs rsyslog question is not a contest. On a stock server both are running, both are storing the same events, and the defaults were chosen by your distribution rather than by you. The failure modes all come from that: a journal in RAM that vanishes on reboot, two uncoordinated retention policies eating the same disk, and two rate limiters dropping messages without anyone noticing.

Run systemd-analyze cat-config systemd/journald.conf and journalctl --list-boots on your servers. Two commands, thirty seconds, and you will know whether the logs you would reach for during an incident are actually there.

Need help sorting out your logging pipeline?

Logging is one of those areas where the defaults work well enough to hide the problem until an incident makes it expensive. If any of the above sounded familiar, these are the things I take on:

• Auditing journald and rsyslog on existing servers and telling you exactly where each log line lives, how long it survives, and what is being silently dropped.
• Fixing volatile journals, runaway /var growth and duplicate text-and-binary storage on small VPS instances.
• Setting up reliable off-box forwarding with disk-assisted queues, TCP or RELP transport, and TLS where it is required.
• Migrating from text-file scraping to a structured pipeline into Loki, Elasticsearch or a hosted log platform, without losing the alerts you already depend on.
• Tuning rate limits and retention per service so noisy applications stop hiding the messages that matter.
• Writing the runbook that says which store is authoritative, so the next engineer does not have to reverse-engineer it during an outage.

Send me your journald.conf, your rsyslog.conf, or the output of journalctl --disk-usage and I will tell you what I see before we talk about scope.

The post journald vs rsyslog: Where Your Linux Logs Actually Live appeared first on John Nessime.

Either way you end up looking at Podman, reading a Podman vs Docker comparison table that says “daemonless” and “rootless by default”, and trying to work out whether those words are worth a migration.

Here is the short version, and I will spend the rest of the post defending it. The architectural difference between the two engines is real and it matters more than the feature tables suggest. The migration cost is also real, and it is almost entirely concentrated in four places that comparison posts tend to skip. Whether the switch is worth it depends on which of those four you actually depend on.

This post covers the one architectural difference everything else follows from, an honest profile of each engine with where it wins and where it does not, the four things that genuinely break when you switch, and a decision procedure you can run against your own setup rather than against a generic one.

The difference everything else follows from

Docker is a client and a daemon. The docker CLI is a thin thing that talks to dockerd, which runs as root and owns every container on the host. Podman has no daemon. When you run podman run, the container is started as a child of your own process and supervised by a small monitor, not handed off to a privileged background service.

Two consequences follow, and they are the whole argument.

Membership of the docker group is equivalent to root. Not “close to root”, not “root if misconfigured”. The daemon runs as root and will happily create a container that bind-mounts the host’s root filesystem, so anyone who can talk to the socket can read or write anything on the box. Docker’s own documentation says as much. This is the finding that shows up in access reviews, and it is not a bug you can configure away without switching to rootless mode.

Process supervision belongs to whoever started the container. With Docker, the daemon restarts your containers according to the restart policy, and if the daemon wedges you have a bad afternoon regardless of what the containers are doing. With Podman, containers are ordinary processes, so systemd supervises them the same way it supervises everything else on the machine, with the same logging, dependency ordering and failure handling.

Podman’s rootless mode builds on user namespaces: UID 0 inside the container maps to your unprivileged UID on the host. A break-out lands in a normal user account rather than on a root shell. Docker has a rootless mode too and it works, but it is an opt-in you configure rather than the path of least resistance, which in practice means far fewer installations run it.

Docker: where it wins and where it doesn’t

Where it wins

• The ecosystem assumes it. Testcontainers, VS Code Dev Containers, Traefik’s container discovery, Portainer, most CI runners, half the SDKs on your machine: all of them expect a Docker socket. Some work fine against Podman’s compatible socket. Some do not, and finding out which is your problem.
• Compose is first-party. Docker Compose is maintained by the same people who maintain the engine, tracks the Compose specification closely, and gets features like --wait and profiles as soon as they exist. Nothing in the Podman world matches that.
• Every answer to every question already exists. Fifteen years of Stack Overflow. That is not a technical merit but it is a real operational one at 2am.
• Cross-platform experience is smoother. Docker Desktop on macOS and Windows is a polished product, and the VM it manages is largely invisible.

Where it doesn’t

• The root daemon is a standing liability. One privileged process that everything flows through, plus a group that is root by another name.
• Docker Desktop is licensed, and the terms are specific. Docker’s own license page states Desktop is free for personal use, education, non-commercial open source, and small businesses defined as fewer than 250 employees and less than ten million dollars in annual revenue. Cross either threshold and commercial use needs a paid subscription. Government entities need one regardless. Docker Engine and the Moby project are not affected. This catches people out because the software installs without asking for a key.
• systemd integration is bolted on. You can wrap a Compose stack in a unit file, but systemd is supervising a wrapper, not the containers.
• Restarting the daemon touches everything. An engine upgrade is a whole-host event rather than a per-service one.

Podman: where it wins and where it doesn’t

Where it wins

• Rootless is the default, not a project. You get the safer posture by doing nothing special, which is the only kind of security control that survives contact with a busy team.
• Quadlet is genuinely better than restart policies. You write a declarative unit file, systemd generates a real service from it, and container lifecycle becomes indistinguishable from any other service on the box. This is the feature I would actually miss if I went back.
• No licensing question to answer. Podman is Apache 2.0 with no commercial-use restriction, and on Red Hat Enterprise Linux, Fedora, Rocky and AlmaLinux it is the packaged, supported option already sitting in the repos.
• Pods and Kubernetes YAML are first-class. podman kube play runs a Kubernetes manifest locally, and podman generate kube goes the other way. If your destination is Kubernetes, the mental model transfers.
• Per-user isolation comes free. Two developers on a shared build host cannot see or stop each other’s containers, because they are running in separate user namespaces rather than as siblings under one daemon.

Where it doesn’t

• Compose is a compatibility story, not a native one. More on this below, because it is the single biggest source of migration pain.
• Rootless networking has sharp edges. Traffic goes through a userspace network stack, and the behaviour is not identical to a root bridge. Source IPs, host-loopback access and privileged ports all behave differently, and you will meet at least one of those.
• Swarm and Docker plugins do not exist. Podman does not implement Swarm mode, and Docker volume and network plugins are not supported. If you use either, the conversation is over.
• Desktop platforms mean a VM you can see. Podman on macOS and Windows runs a Podman machine, and the seams show more than Docker Desktop’s do.
• Documentation assumes Docker. Every vendor quickstart, every README, every “run this to try it” snippet is written for docker run. Most translate directly. The ones that do not cost you an hour each.

The four things that actually break

This is the part worth reading twice. Command compatibility is not the problem; podman accepts almost every docker flag you know. The problems are elsewhere.

1. Compose

There are two routes and they are not equivalent.

podman-compose is a separate Python implementation. It covers the common cases and diverges on the uncommon ones. Container naming is the one that bites scripts first: it has historically used underscores where Docker Compose v2 uses hyphens, so anything referencing project-web-1 by name stops finding it.

The Podman socket plus real Docker Compose is the route I would take. Podman exposes a Docker-compatible REST API over a systemd socket, and the actual Compose binary talks to it:

# Start the Docker-compatible API socket for your user.
systemctl --user enable --now podman.socket

# Confirm it exists before pointing anything at it.
ls -l /run/user/$(id -u)/podman/podman.sock

# Point the real Docker Compose at Podman instead of dockerd.
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
docker compose up -d

You keep the tool that tracks the Compose specification and swap only the engine underneath. The catch is that the API is a compatibility layer, not the Docker daemon, so behaviour can differ at the edges. Run your real compose file through it before you commit to anything.

2. Anything that mounts the Docker socket

Traefik discovering containers by label, Portainer, Testcontainers in your test suite, any CI job that does Docker-in-Docker. All of these want /var/run/docker.sock. The Podman socket serves the same API and many of them work against it unchanged, but “many” is doing work in that sentence. Inventory what mounts the socket before you plan the migration, not after.

3. Ports below 1024

A rootless container cannot bind a privileged port, because the process genuinely is not privileged. If your compose file publishes 80 and 443, it will fail, and the error will not obviously say why.

Three ways out, in the order I would try them:

1. Publish a high port and put a host reverse proxy in front. Nginx or Caddy running as a normal system service, listening on 443, proxying to 8443. No kernel tuning, and you probably wanted TLS termination on the host anyway.
2. Lower the privileged port threshold. The kernel exposes net.ipv4.ip_unprivileged_port_start, and setting it to 0 lets any user bind any port. It works, and it is a host-wide change that weakens a defence for every process on the machine, not just yours. Understand that before you write it into a playbook.
3. Run that one container rootful. Podman supports it. Mixing modes is a legitimate choice as long as it is deliberate and documented rather than something that happened.

4. Restart on boot

This is the one that catches people who tested on a laptop and shipped to a server. Rootless containers belong to a user session, and a user’s services are stopped when their last session ends. Reboot the box and nothing comes back, no matter what restart policy you set.

The fix is two parts. Enable lingering so the user’s systemd instance runs without a login, and define the container as a Quadlet unit rather than relying on a restart policy:

# Let this user's services run without an active login session.
sudo loginctl enable-linger appuser

# Verify. Linger=yes is the line that matters.
loginctl show-user appuser --property=Linger

Then the unit itself. Quadlet files look like systemd units with a Podman-specific section, and a generator turns them into real services at boot and on daemon-reload. Rootless units live in ~/.config/containers/systemd/; system ones in /etc/containers/systemd/.

# ~/.config/containers/systemd/api.container

[Unit]
Description=Application API
After=network-online.target

[Container]
# Fully qualified. Podman does not assume Docker Hub.
Image=registry.example.com/api:1.4.2
PublishPort=8080:8080
Volume=api-data.volume:/var/lib/api
Environment=NODE_ENV=production

[Service]
Restart=always

[Install]
# Starts at boot once lingering is enabled for this user.
WantedBy=default.target

# Regenerate services from Quadlet files, then start it.
systemctl --user daemon-reload
systemctl --user start api

# From here it is an ordinary service.
systemctl --user status api
journalctl --user -u api -f

Note that podman generate systemd, which you will find in older tutorials, is deprecated in favour of Quadlet. Podman’s documentation says it receives urgent bug fixes only and no new features. Do not build anything new on it.

One smaller trap while you are here: Podman does not assume Docker Hub for unqualified image names. nginx:alpine may prompt you to choose a registry or fail outright depending on configuration. Write docker.io/library/nginx:alpine, or whatever your registry is, everywhere. This is arguably the correct behaviour and it is still an afternoon of edits.

Podman vs Docker: how I would decide

Run these in order and stop at the first one that answers.

1. Do you use Swarm, or Docker volume or network plugins? Stay on Docker. Podman does not implement them and no workaround changes that.
2. Is a compliance requirement or an access review driving this? Switch, or at minimum move to rootless. “The docker group is root” is a finding you cannot argue with, and rootless Podman is the shortest path to closing it.
3. Are you on RHEL, Fedora, Rocky or AlmaLinux? Lean Podman. It is the packaged, supported, tested-by-the-vendor option on those distributions, and you are swimming with the current instead of against it.
4. Is Docker Desktop licensing a real cost for your organisation? Check the thresholds against your actual headcount and revenue before assuming either way. If you are over, this is a budget conversation with a free technical answer, which is a rare shape of problem.
5. Does your test suite or CI depend on the Docker socket? Prototype that specific dependency against the Podman socket first. It is the highest-risk item and the cheapest to test. If it fails, everything else is academic.
6. None of the above, and Docker is working? Stay. “Podman has nicer architecture” is true and it is not a business case. Migrations cost weeks, introduce risk, and the containers you are already running do not care which engine started them.

The honest summary: Podman is the better-designed engine and Docker is the better-supported one. Design wins when you are building something new or when a policy requirement forces the question. Support wins when you have a working system and a limited number of weeks.

If you do switch, do it in this order

1. Inventory the socket mounts. Grep your compose files and CI config for docker.sock. That list is your actual migration scope.
2. Stand up one non-critical service on Podman. A staging box, an internal tool, something on a spare VPS. Run it for a fortnight through at least one reboot before you draw conclusions.
3. Fully qualify every image reference. Do this in the compose files while they still run under Docker. It changes nothing there and removes a whole class of failure later.
4. Switch the engine, keep Compose. Podman socket plus DOCKER_HOST. Change one variable at a time so you know what broke.
5. Convert to Quadlet last, and only where it earns its place. Long-lived services benefit. A three-container stack you rebuild constantly is fine staying on Compose.

Arguments that don’t survive contact

• “Podman is a drop-in replacement.” The CLI nearly is. The surrounding tooling is not, and that is where the weeks go.
• “Daemonless means faster.” The engine is not your bottleneck. Rootless networking has historically been slower than a root bridge, not faster. Pick Podman for the security model, not for benchmark numbers.
• “Rootless means secure.” It means a break-out lands somewhere less catastrophic. A container running rootless with the host filesystem bind-mounted read-write is still a bad idea.
• “Docker is dying.” It is not. Docker Engine, Compose and Buildx are actively developed and hosted CI runners still ship them by default.
• “Just alias docker to podman and you’re done.” The alias handles the commands. It does not handle the socket, the ports, the boot behaviour or the image names.

Frequently asked questions

Can Podman run docker-compose files?

Yes, by two routes. Install podman-compose, which is a separate implementation with good but incomplete coverage, or enable the Podman socket and point real Docker Compose at it with DOCKER_HOST. The second gives better specification coverage because you are running the actual Compose tool. Neither is a guarantee; test your own file.

Is Podman actually more secure than Docker?

Its default posture is better. Rootless out of the box means no root daemon and no group that is root by another name. Docker can be configured to a similar posture with rootless mode, so the difference is less “Podman is secure and Docker is not” and more “Podman’s safe path is the easy path”. Defaults determine what most installations actually run.

Do I need to pay for Docker?

Docker Engine on Linux is open source and free. Docker Desktop is the licensed product. Per Docker’s terms it is free for personal use, education, non-commercial open source, and businesses with fewer than 250 employees and under ten million dollars in annual revenue; cross either threshold, or be a government entity, and commercial use requires a subscription. If you only run Docker Engine on Linux servers, the licensing question does not apply to you.

Why do my Podman containers disappear after a reboot?

Because rootless containers run under your user’s systemd instance, which is torn down when your last session ends. Run loginctl enable-linger for the service account and define the containers as Quadlet units with WantedBy=default.target. A Podman restart policy alone will not survive a reboot.

What is Quadlet and do I have to use it?

Quadlet is a systemd generator that turns declarative container definitions into real systemd services. You do not have to use it, but the alternative is hand-writing unit files, and the older podman generate systemd command is deprecated in favour of it. For anything long-lived on a server, Quadlet is the current answer.

Can Podman and Docker coexist on the same host?

Yes. They keep separate image and container stores, so nothing collides at the storage layer. Watch for port conflicts if both are running services, and be deliberate about which one a given script is talking to, because DOCKER_HOST and an alias can quietly disagree with each other.

Does Podman work with Kubernetes?

Better than Docker does, in the sense that it speaks the format natively. podman kube play runs a Kubernetes manifest on a single host and podman generate kube produces one from running containers. It is not a substitute for a cluster, but as a local development story for a Kubernetes destination it is the closest thing available.

The one thing worth remembering

The Podman vs Docker decision is not really about container engines. Both run OCI images and both will start your workload. It is about whether a root daemon and a root-equivalent group are acceptable in your environment, and whether you have the weeks to spend on the four things that break.

If a policy requirement is driving the question, switch and budget properly for the compose and socket work. If you are on an enterprise Linux distribution and starting something new, start on Podman, because you will be swimming with the current. If Docker is working and nothing external is forcing the question, the correct engineering answer is to leave it alone and go fix something that is actually broken.

Thinking about a migration and want a second opinion?

I help teams decide this question and then execute it without a surprise outage. Things I am usually brought in for:

• Auditing what on your hosts actually depends on the Docker socket, and scoping the migration honestly before anyone commits to it
• Testing your real compose files against the Podman socket and reporting what breaks, rather than guessing from a feature table
• Converting long-lived services to Quadlet units with proper systemd ordering, health gating and journald logging
• Sorting out rootless networking: privileged ports, reverse proxy placement, and the source-IP behaviour that surprises people
• Closing the “docker group equals root” audit finding, whether that means Podman, rootless Docker, or removing the group entirely
• Writing the decision up so it survives the next person who asks why you chose what you chose

If you are weighing this up, send me a compose file and a list of what mounts the socket. That is usually enough to tell you whether the switch is a fortnight or a quarter.

The post Podman vs Docker: Is the Switch Actually Worth It? appeared first on John Nessime.

The thing that needed the memory was Redis, installed three weeks earlier to speed up the admin, running with the configuration it shipped with. Redis defaults to no memory limit and a policy of never evicting anything. Left alone it will happily grow until the machine runs out, and then the kernel picks a victim, and it will not necessarily pick Redis.

That is the framing for everything below. Redis object caching for WordPress is genuinely one of the highest-value changes you can make to a busy site, and Redis out of the box is configured as a durable datastore, not a cache. You are storing data you can regenerate at any time. The defaults assume the opposite.

Setup takes ten minutes. The gotchas are the post.

What it does, and what it doesn’t

WordPress has an object cache built in, and by default it lasts exactly one request. Every page load fetches the same options, terms, post meta and user meta from MySQL, uses them, and throws them away. A persistent object cache keeps that data between requests, so the second visitor does not repeat the first visitor’s queries.

Two expectations worth setting before you install anything.

It is not a page cache. Page caching serves a finished HTML file and never runs PHP. Object caching runs the whole application and makes the database part faster. They live in different drop-ins, do different jobs, and coexist happily.

Its biggest effect is where page caching does not reach. Logged-in requests, wp-admin, WooCommerce carts and checkouts, REST API calls. If your front end is already cached and fast while the dashboard crawls, an object cache is aimed exactly at your problem.

Setup

1. Install Redis from your distribution’s packages, and bind it to localhost unless something remote genuinely needs it.
2. Install a PHP Redis client. PhpRedis (the PECL extension) is faster than the pure-PHP Predis and is what you want on a real server. Relay is a newer option worth knowing about if you are pushing volume.
3. Install the Redis Object Cache plugin, then add the constants below to wp-config.php before enabling the drop-in.
4. Enable it, which copies object-cache.php into wp-content/. That file, not the plugin, is what WordPress actually loads.
5. Verify it end to end, which is step five for a reason.

// wp-config.php, above the "That's all, stop editing" line.

define( 'WP_REDIS_HOST',     '127.0.0.1' );
define( 'WP_REDIS_PORT',     6379 );

// Two different protections, and you want both. The database
// number scopes flushes; the prefix prevents key collisions.
define( 'WP_REDIS_DATABASE', 1 );
define( 'WP_REDIS_PREFIX',   'acme-prod:' );

// Ceiling for keys written with no expiry, so nothing lives forever.
define( 'WP_REDIS_MAXTTL',   86400 );

One note if you are following an older tutorial: the prefix constant used to be WP_CACHE_KEY_SALT and was renamed to WP_REDIS_PREFIX in version 2.0 of the plugin, to stop it being confused with a core constant of the same name. The old name is still read for compatibility, but if a guide is still using it, that guide predates a major version and its other advice may too.

Then prove it works rather than trusting a green indicator on a settings page:

# The plugin's own view.
wp redis status

# Write through WordPress, read it back through WordPress.
wp eval 'wp_cache_set("probe","ok","test",60); echo wp_cache_get("probe","test");'

# Then confirm the key landed in the database and prefix you expect.
redis-cli -n 1 --scan --pattern 'acme-prod:*' | head

Site Health under Tools will also tell you whether WordPress considers a persistent object cache to be in use. That is the check to trust, because it reflects what core sees rather than what the plugin believes.

Gotcha 1: the defaults are a datastore’s defaults

The one from the opening paragraph, and the only one on this list that can take a server down.

# /etc/redis/redis.conf

# Without a limit, Redis grows until the machine runs out and the
# kernel kills something. It will not necessarily kill Redis.
maxmemory 512mb

# The default policy is to refuse writes rather than evict. Correct
# for a datastore, wrong for a cache: you want the least recently
# used keys dropped, silently, forever.
maxmemory-policy allkeys-lru

# Cache contents are regenerable by definition. Persisting them buys
# nothing and costs you fork latency on save and slower restarts.
save ""
appendonly no

Size maxmemory against what the machine can spare after MySQL and PHP-FPM have what they need, not against what is currently free. Free memory on a healthy Linux box is mostly page cache, and it is doing useful work.

The honest trade-off with allkeys-lru: under memory pressure Redis will evict things you would rather it kept, and cache misses go up. That is the correct failure mode. The alternative is write errors surfacing inside WordPress, or an outage.

If you are on managed hosting, do not change any of this. Ask them what they have set, because they may have tuned it and may not appreciate you undoing it.

Gotcha 2: one Redis, several sites

Redis is a single key-value store. Two WordPress installs pointed at it with default settings both write post_123 to the same key and read each other’s data. On a server hosting several sites that is a correctness bug. Where it gets genuinely nasty is staging.

Staging is usually a clone of production, which means it inherits production’s wp-config.php, which means it inherits production’s Redis settings. Now both environments share a namespace, and a developer clearing the cache on staging clears production’s, or worse, staging serves production’s cached objects.

The two constants guard against different things and you want both:

• WP_REDIS_DATABASE selects a logical database, of which Redis provides sixteen by default. This scopes flushes, so one site clearing its cache does not clear another’s.
• WP_REDIS_PREFIX namespaces every key. This prevents collisions if two installs ever land in the same database, which is what happens the day somebody copies a config file.

Make the prefix readable and environment-specific: acme-prod: and acme-staging:, not a random string. It is a namespace, not a secret, and you will want to read it in redis-cli at some point.

If you use Relay as the client, a dedicated database and a unique prefix per install is a requirement rather than a suggestion.

Gotcha 3: some things are never cached, and one thing is enormous

Not every cache group is persistent. WordPress marks some as non-persistent, and plugins can mark their own, so those groups keep behaving exactly as they did before you installed anything. If you expected a specific thing to get faster and it did not, this is a likely reason and it is not a misconfiguration.

The more interesting one is alloptions. WordPress stores all autoloaded options as a single cache entry. On a site with a bloated options table, that is one very large key, fetched on every request, and invalidated whenever any autoloaded option is written. Under concurrency that produces a lot of simultaneous re-priming of the same large value.

Which means an object cache does not excuse a bloated options table; it changes where the cost lands. Check what your keys actually look like:

# Reports the largest key per type. If one key dwarfs everything
# else, it is probably alloptions and your options table needs work.
redis-cli -n 1 --bigkeys

Gotcha 4: flushing is not free

Clearing the object cache on a busy site means every subsequent request rebuilds what it needs from MySQL simultaneously. On a quiet site nobody notices. On a busy one you get a load spike at exactly the moment you were trying to fix something.

Know what your flush actually does. Some implementations issue a Redis-wide flush; others delete only keys matching your prefix. Those are very different operations when the instance is shared, and finding out during an incident is a bad time.

Also worth deciding deliberately: whether deploys flush the cache. Stale cached objects after a plugin update cause genuinely confusing bugs, so a flush in the deploy script is often right. Just do it knowing it costs a load spike, and put it at the start of a quiet window rather than at five o’clock on a Friday.

Troubleshooting

“Drop-in: invalid” or “not connected”. Usually the PHP extension is missing, or PHP cannot reach Redis on the host and port you gave it. Check with php -m | grep redis and redis-cli ping from the same machine PHP runs on, which is not always the machine you are logged into.

Everything looks connected but nothing is cached. Something replaced object-cache.php. Optimisation plugins install their own drop-in, and only one can win. Check the file’s contents, not just that it exists.

The site got slower. Latency to Redis. A remote instance across a network adds a round trip to every cache operation, and there are a lot of them per request. Localhost or a Unix socket beats a fast network every time.

Stale data after an update. Flush once, then work out which plugin is caching something it should be invalidating. Do not put a flush on every request to make the symptom go away.

Memory keeps climbing. Either no maxmemory, or keys written with no expiry and no WP_REDIS_MAXTTL ceiling. Check INFO memory and the eviction counters.

Staging and production interfering. They are sharing a database number, a prefix, or both. Fix the config on staging as part of your restore script so it cannot recur.

Common mistakes

• Running Redis with default memory settings on a shared box.
• Leaving persistence enabled for a pure cache workload.
• No prefix and no database number, so sites read each other’s objects.
• Cloning production to staging without changing the Redis settings.
• Trusting the plugin’s status page instead of Site Health and an actual read-back.
• Expecting it to speed up an already page-cached front end.
• Installing it and ignoring an options table full of autoloaded junk.
• Predis on a production site when PhpRedis is available.
• Putting Redis on a different host and adding a network hop to every cache call.
• Flushing the cache during peak traffic to fix a display bug.
• Two plugins competing for the object-cache.php drop-in.

Best practices

• Set maxmemory and allkeys-lru before you point any site at it.
• Disable persistence unless Redis is also doing a job that needs it.
• Unique database number and readable prefix per install and per environment.
• Bind to localhost, or require a password and TLS if it must be remote.
• PhpRedis over Predis, and Redis on the same host as PHP where you can.
• A WP_REDIS_MAXTTL ceiling so nothing lives forever by accident.
• Verify with Site Health plus a write-and-read test, not a status indicator.
• Clean up autoloaded options rather than caching the mess.
• Monitor memory used, evicted keys and hit rate; a collapsing hit rate is an early warning.
• Make staging’s Redis config part of the restore script, not something to remember.

FAQ

Do I need Redis if I already have a caching plugin?

Probably, because they solve different problems. Page caching handles anonymous visitors; object caching handles everything that cannot be page cached, which is the admin, logged-in users and anything transactional. Run both.

Redis or Memcached?

For WordPress, Redis, mostly because the tooling and plugin ecosystem around it is better and you get richer diagnostics. Memcached is perfectly capable and slightly simpler. This is not a decision worth agonising over.

How much memory should I give it?

Start modestly, watch memory used and evicted keys for a week, and raise it if evictions are constant while free memory remains. Sizing from an article is guessing; sizing from your own eviction counter is not.

Is it safe to lose everything in Redis?

Yes, for an object cache. Everything in it is regenerable from MySQL, which is exactly why persistence is unnecessary. Losing it costs a burst of database load, not data.

Why did my dashboard not get faster?

Check the cache is genuinely active first, then look at what else is slow: outbound HTTP calls during admin requests, a large autoloaded options blob, or the heartbeat. An object cache removes repeated database work and nothing else.

Should staging share the production Redis?

It can, with a different database number and a different prefix. It is cleaner not to. Either way, set it as part of the restore process so a clone never inherits production’s cache namespace.

The one thing to remember

Redis ships configured to protect data it assumes you cannot afford to lose. You are storing data you could regenerate in a second. Every default that follows from that assumption, unlimited memory, no eviction, persistence to disk, is wrong for your use case and one of them can take the server down.

So set maxmemory and an eviction policy before you enable the drop-in, give every install its own database and prefix, and verify with Site Health rather than a green dot. Then it is genuinely one of the best changes you can make to a WordPress site.

Want it set up properly?

Object caching is quick to install and easy to get subtly wrong in ways that show up weeks later. Work I take on:

• Installing and tuning Redis object caching on a VPS or dedicated server, including memory limits, eviction and persistence settings.
• Auditing an existing setup for shared namespaces, competing drop-ins and caches that are not actually active.
• Multi-site and multi-environment configuration so staging can never touch production’s cache.
• Cleaning up autoloaded options so the object cache is caching something sensible.
• Monitoring for Redis memory, evictions and hit rate, wired into Prometheus and Grafana.
• Diagnosing sites where caching was installed and nothing got faster.

Send me your Site Health info and redis-cli INFO memory, and I will tell you what is worth changing.

The post The OOM Killer Took MySQL: Redis Object Caching for WordPress appeared first on John Nessime.

The line you wrote was real. The file was saved. What you did not know is that your distro puts an Include /etc/ssh/sshd_config.d/*.conf near the top of sshd_config, that sshd uses the first value it finds for any given keyword, and that your cloud provider’s image dropped a 50-cloud-init.conf in that directory setting PasswordAuthentication yes. It sorts earlier. It wins. Your line at the bottom of the main file was never reached.

You never noticed because you log in with a key anyway. Nothing broke. The control just was not there.

That is the thing most guides get wrong, and it is the idea this VPS hardening checklist is built around: a hardening step is not done when you have edited the file. It is done when you have checked the effective configuration and seen the value you expected. Everything below comes with the command that proves it.

This covers the order to do things in so you do not lock yourself out, SSH keys and config, the firewall, automatic updates, fail2ban and whether it is worth it, the unglamorous things that matter more than any of it, and what I would skip.

Before you touch anything

Two minutes of preparation that saves an evening.

Find your provider’s console before you need it. Whether you are on InterServer, Hetzner, DigitalOcean or anything else, there is a web-based serial or VNC console that works when SSH does not. Open it now, log in as root, and confirm it works. This is your parachute. If you are going to lock yourself out, you want to already know where it is.

Never close the session you are working in. Every SSH change gets tested from a second terminal while the first one stays open. If the new settings are broken, the old session is still authenticated and you can undo them. Close it too early and your only route back in is the console.

Then update the system, because everything else assumes current packages:

sudo apt update && sudo apt upgrade -y
sudo reboot   # if a kernel update landed

1. A real user, and keys instead of passwords

Generate the key on your own machine, not on the server. A private key that has been on a server is not a private key any more.

# On your laptop. ed25519 is the sane default: short keys,
# fast verification, no parameter choices to get wrong.
ssh-keygen -t ed25519 -C "deploy@my-laptop"

# On the server, as root.
adduser deploy
usermod -aG sudo deploy        # 'wheel' instead of 'sudo' on RHEL-family

# Back on your laptop. ssh-copy-id appends to authorized_keys
# and sets the permissions correctly, which is the part people
# get wrong when they paste the key by hand.
ssh-copy-id deploy@203.0.113.10

Now open a second terminal and confirm ssh deploy@203.0.113.10 works, and that sudo -v works once you are in. Only then move on. If either fails, fix it before you disable anything, because in a minute you will be removing the fallback.

2. SSH config, and why your edit might do nothing

This is the section that actually matters. Two mechanisms silently override what you write.

Drop-in files and first-match-wins

Modern OpenSSH packages ship an Include line near the top of /etc/ssh/sshd_config, which pulls in every .conf file from /etc/ssh/sshd_config.d/ in alphabetical order, before the rest of the main file is read. Combine that with sshd’s rule that the first value found for a keyword wins, and the consequence is specific: your drop-in has to sort earlier than the vendor’s, not later.

Most guides tell you to name it 99-something.conf, which is the convention almost everywhere else in Linux and exactly backwards here.

# /etc/ssh/sshd_config.d/00-hardening.conf
# Sorts before 50-cloud-init.conf, so these values are seen first.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes

# Optional: restrict to the accounts that should have shell access.
AllowUsers deploy

A note on KbdInteractiveAuthentication: older guides and older OpenSSH call this ChallengeResponseAuthentication. Setting only PasswordAuthentication no can still leave a keyboard-interactive path open on some configurations, which is why both are here.

Now the part nobody does. Validate the syntax, then read the effective config:

# Syntax check. Prints nothing if the config is valid.
# Run this BEFORE restarting, always.
sudo sshd -t

# Dump the effective configuration with every include resolved.
# This is the only output that tells you the truth.
sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|kbdinteractive|pubkeyauth'

If passwordauthentication yes comes back, your file lost to something else in that directory. Look at what is in there with ls /etc/ssh/sshd_config.d/ and either rename yours to sort earlier or edit the offending file directly.

Socket activation, and the port that will not change

The second trap. On several current distro releases, SSH is started by systemd socket activation rather than as a persistent service: ssh.socket holds the listening port and starts sshd only when a connection arrives. On those systems the Port and ListenAddress directives in sshd_config may be ignored entirely, because systemd owns the socket.

People change the port, restart the service, see no error, and assume it worked. Check what is actually bound:

# -t TCP, -l listening, -n numeric, -p show the owning process
sudo ss -tlnp

If you genuinely want a different port on a socket-activated system, override the socket unit with a proper drop-in rather than editing files under /lib/systemd/system/, which a package update will overwrite:

sudo systemctl edit ssh.socket

# In the editor that opens:
[Socket]
# The empty assignment clears the inherited value. Without it
# you end up listening on both the old port and the new one.
ListenStream=
ListenStream=2222

sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -tlnp        # confirm before you trust it

Honest assessment of port changes: this is not security. Anyone scanning your address finds an SSH daemon on a non-standard port in seconds. What it genuinely does is cut the volume of automated login noise in your logs by a large margin, which makes real events easier to see. That is a maintenance benefit, not a defence, and it is worth doing for that reason alone. Just do not count it as one of your controls.

3. Firewall, in the right order

The ordering here is the whole trick. Set the default policy and add your rules before you enable the firewall. Enable first and you will drop your own connection with the policy that denies everything.

sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH BEFORE enabling. If you changed the port,
# allow that port instead of the OpenSSH profile.
sudo ufw allow OpenSSH
# sudo ufw allow 2222/tcp

sudo ufw allow 80,443/tcp

sudo ufw enable
sudo ufw status verbose

Read that status verbose output rather than assuming. It shows the default policies and every rule, and it is the proof that this step is done.

One thing worth knowing if you run containers: Docker writes its own rules directly into netfilter and bypasses ufw. A container published with -p 5432:5432 is reachable from the internet even though ufw says that port is denied. Bind container ports to 127.0.0.1 explicitly, or put the service behind a reverse proxy, and verify from outside the box with a port scan rather than trusting the firewall status.

A stronger option worth considering: do not expose SSH to the internet at all. Put the server on a private overlay network with something like Tailscale or a WireGuard tunnel, bind sshd to that interface, and close port 22 to the world entirely. More setup, and it removes an entire class of problem rather than filtering it.

4. Automatic security updates

Unpatched packages are a far more realistic route into your server than an SSH brute force. This is the highest-value item on the list and it takes two commands.

sudo apt install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Verify. Note the binary is singular, the package is plural.
# --dry-run makes no changes.
sudo unattended-upgrade --dry-run --debug

By default this applies security updates only, which is the right scope. Enabling automatic reboots is a real trade-off: it closes kernel vulnerabilities without you doing anything, and it will restart your server at whatever hour you configure. On a single web server with a proper service startup order, I turn it on. On anything holding state that does not recover cleanly from a hard restart, I leave it off and patch by hand.

5. fail2ban, and what it is actually for

Install it, but be clear about the role. Once password authentication is off and root login is disabled, a brute force against sshd cannot succeed. There is no password to guess. fail2ban is not what is keeping those attackers out; your key-only config is.

What it does earn its place for is reducing log noise and CPU spent on connection handling, and protecting the services where credentials do exist: a mail server, a web application login form, an FTP daemon.

sudo apt install fail2ban

# Do not edit jail.conf; it gets replaced on upgrade.
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

sudo systemctl enable --now fail2ban
sudo systemctl status fail2ban
sudo fail2ban-client status sshd

Check that status output properly. fail2ban has shipped in states where the service fails to start on a given distro release because the packaged version does not match the system Python, and it fails loudly in the journal and silently everywhere else. A stopped fail2ban looks exactly like a working one from the outside. If it is not running, journalctl -u fail2ban will tell you why.

Set ignoreip in jail.local to include your office or home address. Banning yourself is a rite of passage but it is avoidable.

6. The boring items that matter more than the exciting ones

Nobody writes blog posts about these, and they cause more real outages than any of the above.

Provider account security

Turn on two-factor authentication in your hosting control panel and your domain registrar. Someone with your provider login does not need to defeat sshd; they can reset the root password from the console or attach your disk to a machine they control. This is the weakest link on most setups and it takes two minutes.

Time and timezone

sudo timedatectl set-timezone UTC
timedatectl status       # look for "System clock synchronized: yes"

UTC on servers, always. Correlating logs across machines in different local times, one of which observes daylight saving, is a genuinely miserable way to spend an incident.

Swap on small instances

# A little swap stops the OOM killer choosing your database
# during a traffic spike. It is not a substitute for RAM.
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
free -h

Backups that leave the machine

Your provider’s snapshots live in the same account as the server. If that account is compromised, or you delete the wrong thing, they go with it. A snapshot is a rollback mechanism, not a backup.

Get database dumps and application data onto storage under a different set of credentials, on a schedule, and then do the part everyone skips: restore one. A backup you have never restored is a hypothesis. Point a dead man’s switch service at the backup job so you find out when it stops running, rather than the day you need it.

Knowing the box is alive

At minimum, an external uptime check that alerts you. Better, node metrics into Prometheus with Grafana dashboards, or a hosted equivalent. Disk filling up is the single most common way a small VPS falls over, and it is completely predictable if anything is watching the graph.

What I would skip

Not everything in the long hardening guides earns its cost on a single web server.

• Large sysctl hardening blocks pasted from a gist. Most of the values are already the default on a current kernel, several are cargo-culted from a decade ago, and one of them will break something in a way that takes hours to trace. Change kernel parameters when you understand what each one does.
• AIDE, auditd and file-integrity monitoring on a small server nobody is watching. These produce a large volume of output that needs a human to review. If nobody reads it, it is not a control, it is just disk usage.
• Rebuilding OpenSSH’s cipher list. Current defaults are good. Guides that hand you a Ciphers line are usually copying advice written for a version that shipped years ago, and you can pin yourself to something weaker than the default without realising.
• Disabling ping. It breaks your own monitoring and hides nothing.

Troubleshooting

Locked out after an SSH change

Provider console, log in as root, and revert. If you cannot remember what you changed, ls -lt /etc/ssh/sshd_config.d/ shows the most recently modified file. Run sshd -t to see whether the config is even valid, since a syntax error stops sshd starting at all.

Key auth rejected, password prompt appears instead

Almost always permissions. The home directory must not be group-writable, ~/.ssh needs 700 and authorized_keys 600, both owned by the user. sshd refuses keys from world-writable locations and does not explain itself on the client side. Run ssh -v to see how far it gets, and check journalctl -u ssh on the server for the actual reason.

The setting I applied is not in effect

sudo sshd -T and search for the keyword. If the effective value differs from your file, something in sshd_config.d/ sorts earlier, or a Match block later in the config is overriding it for your connection.

A service is reachable that the firewall says is blocked

Docker, or another daemon writing netfilter rules directly. Check with sudo iptables -S or sudo nft list ruleset, and always verify from a machine outside the server rather than trusting ufw status.

Nothing obviously wrong, but who has been logging in?

last              # successful logins
sudo lastb        # failed attempts, from /var/log/btmp
journalctl -u ssh --since "24 hours ago"

Common mistakes

• Closing your only SSH session before testing the new configuration from a second one.
• Enabling the firewall before adding the SSH rule.
• Naming your sshd drop-in 99-, so a vendor file that sorts earlier wins.
• Trusting the file you edited instead of running sshd -T.
• Generating the SSH key on the server.
• Changing the SSH port and counting it as a security control.
• Disabling password auth for SSH while leaving a web control panel with a weak password exposed.
• No two-factor authentication on the hosting account that can reset root.
• Treating provider snapshots as backups.
• Installing fail2ban and never checking that it started.
• Publishing container ports without binding them to localhost, then assuming ufw covers it.
• Pasting a sysctl block you cannot explain line by line.

Best practices

• Verify every control after applying it, with a command that reads the running state.
• Keep a second session open and the provider console tested before making SSH changes.
• Key-only authentication, root login disabled, and an explicit AllowUsers list.
• Default-deny inbound firewall, with rules added before the firewall is enabled.
• Automatic security updates on, with a deliberate decision about automatic reboots.
• Two-factor authentication on the hosting account and the registrar.
• Backups off the machine, on separate credentials, and restore-tested at least once.
• UTC everywhere, with clock sync confirmed.
• Monitoring that pages you on disk usage before it pages you on downtime.
• Script the whole build, so the next server is identical and you are not repeating this from memory.

FAQ

What is the single most important step?

Key-only SSH with root login disabled, verified with sshd -T. It removes the entire category of credential guessing. Automatic security updates is a close second, because unpatched software is the more realistic way in once passwords are off the table.

Should I change the SSH port?

For quieter logs, yes. As a security measure, no. A scanner finds the daemon regardless. The practical benefit is that failed-login noise drops enough that a genuine anomaly becomes visible, which has real value for anyone actually reading their logs.

Do I need fail2ban if password authentication is off?

Not for SSH specifically. Install it for the services that still accept credentials, and for the reduction in log volume and connection churn. Just do not let it occupy the slot in your head reserved for an actual defence.

Why did my sshd_config change get ignored?

Two likely causes. A file in /etc/ssh/sshd_config.d/ that sorts alphabetically earlier set the same keyword first, and sshd keeps the first value it sees. Or the setting is Port or ListenAddress on a socket-activated system, where systemd owns the listener. sshd -T answers the first case and ss -tlnp answers the second.

Is ufw enough, or should I use nftables directly?

ufw is enough for a single server with a handful of open ports, and it is a front end to the same machinery underneath. Write nftables rules directly when you need something ufw cannot express cleanly, such as rate limiting or complex source-based rules. Do not switch tools for its own sake; a firewall you understand beats an elegant one you do not.

How do I harden a server I did not build?

Audit before you change anything. sshd -T, ufw status verbose, ss -tlnp for unexpected listeners, systemctl list-units --type=service --state=running, and the contents of every authorized_keys on the box. Old keys belonging to people who left are extremely common and nobody ever removes them.

Should this be automated?

Yes, once you have done it manually enough times to understand each step. An Ansible playbook or a cloud-init script makes the next server identical and turns your hardening into something reviewable in version control. Automating a process you do not yet understand just makes the mistakes reproducible.

The one thing to remember

Editing a config file is not the same as changing a system’s behaviour. Between your file and the running daemon sit include directives, drop-in ordering, systemd units and packages that ship their own opinions, and any one of them can quietly discard what you wrote without an error message.

So treat this VPS hardening checklist as a set of claims to be tested rather than a list to be ticked. sshd -T for SSH, ufw status verbose for the firewall, ss -tlnp for what is listening, systemctl status for anything you installed. If you cannot show the control from the running system, you have not applied it. You have only intended to.

Want a second pair of eyes on your server?

Most of the servers I get handed were hardened once, by someone who has since moved on, and nobody has verified anything since. Work I take on:

• Auditing an existing VPS and reporting which controls are genuinely in effect versus configured but overridden.
• Building a new server from scratch: users, SSH, firewall, automatic updates, monitoring, backups, all verified.
• Turning a manual build into an Ansible playbook or cloud-init script so the next server matches this one.
• Moving SSH off the public internet onto a private overlay network or bastion.
• Backup strategy that survives the hosting account being compromised, including a tested restore.
• Firewall and container networking, especially where published Docker ports are bypassing the firewall.

Send me the output of sshd -T, ufw status verbose and ss -tlnp, and I will tell you what stands out.

The post You Probably Didn’t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself appeared first on John Nessime.

Your server is only sending the leaf certificate. The intermediate that links it to a trusted root is missing. Your browser papers over that: it either has the intermediate cached from another site, or it quietly fetches it using the AIA extension in the certificate. curl, Java, Go, Python and every other non-browser client do no such thing. They see a certificate signed by something they do not trust and stop.

This is the most common of all TLS certificate errors and it is close to invisible from a desktop. The padlock in your browser is not a test. It is a client with a large safety net doing you a favour.

This post covers the four things a client actually checks and how each one fails: the chain, the names, the clock, and the trust store. Then the automation half, which has its own quiet failure: a renewal that succeeds and never reaches the running server. Plus the two things about TLS that have genuinely changed recently and will break config you copied from an older guide.

Look at what the server is actually sending

Before theorising, get the facts. One command tells you almost everything.

# -servername sends SNI. Without it you get whatever the default
# vhost serves, which is a different certificate on most servers
# and the source of a lot of confusing results.
# -showcerts prints every certificate the server sent, in order.
openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null

Read the top of that output. There is a Certificate chain block listing what arrived, numbered from zero, with s: for subject and i: for issuer. Entry 0 is your certificate. Entry 1 should be the intermediate that issued it, and its subject should exactly match entry 0’s issuer.

If the chain block has only entry 0, that is your bug. Right at the bottom, Verify return code gives the verdict: 0 (ok) is what you want, 21 (unable to verify the first certificate) means the chain is incomplete, and 20 (unable to get local issuer certificate) means the issuer is not in the local trust store.

Run it from a machine that is not yours. A clean container is ideal, because your laptop has accumulated intermediates and corporate roots that a customer’s server has not.

Chain problems

Almost all of these come down to one file choice. ACME clients write out several files and two of them look interchangeable:

• cert.pem is the leaf alone. Using this is the mistake.
• chain.pem is the intermediates alone.
• fullchain.pem is leaf plus intermediates, in the right order. This is the one your server should point at.

# nginx
ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

# Apache. On current versions the chain goes in SSLCertificateFile.
# SSLCertificateChainFile is deprecated; guides that still use it
# are old enough that other advice in them is probably stale too.
SSLCertificateFile      /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile   /etc/letsencrypt/live/example.com/privkey.pem

Order matters as well as contents. The leaf must come first, then each intermediate that signs the one above it. A chain assembled in the wrong order will be rejected by strict clients even though every certificate in it is correct. You can check a chain offline before deploying it:

# Does this leaf chain to a trusted root using these intermediates?
# Exits non-zero and explains itself if not.
openssl verify -untrusted chain.pem cert.pem

Do not include the root certificate in the file you serve. The client either has it or does not, and sending it wastes bytes on every handshake without changing any outcome.

Name problems

The second family. A certificate can be perfectly valid and still be the wrong certificate for the name being requested.

# The hostname the client asked for must appear in here.
openssl x509 -in cert.pem -noout -text | grep -A1 "Subject Alternative Name"

Three things people get wrong here:

• The Common Name does not count. Modern clients ignore the CN field entirely for hostname matching. If the name is only in the CN and not in the SAN list, it will fail, and the error message will not explain that clearly.
• Wildcards are exactly one label deep. *.example.com covers www.example.com and api.example.com. It does not cover example.com itself, and it does not cover a.b.example.com. The bare domain has to be listed separately, and it usually is not.
• SNI decides which certificate you get. One IP serving many sites picks the certificate based on the name the client sends during the handshake. A client that does not send SNI, or an openssl command without -servername, gets the default vhost’s certificate and produces a name mismatch that has nothing to do with your actual configuration.

Time and trust problems

# Local file
openssl x509 -in cert.pem -noout -dates

# Live endpoint, subject and dates in one go
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null 
  | openssl x509 -noout -dates -subject

Expired is obvious. notBefore in the future is the interesting one, because it almost never means the certificate is wrong. It means the client’s clock is wrong. Containers with no NTP, a VM restored from a snapshot, an embedded device with a dead battery: all of them produce “certificate not yet valid” against a perfectly good certificate. Check the clock on the failing machine before you touch anything on the server.

Trust store problems look similar but come from a different place. The root that signed your chain has to be in the trust store of the machine making the request, and old systems have old trust stores. A device that has not received an OS update in years may simply not know about a certificate authority that appeared since. There is no server-side fix for that beyond choosing a CA with broader legacy coverage, which is one of the few genuine arguments for a paid certificate.

The renewal that succeeded and changed nothing

Here is the second invisible failure, and it catches experienced people. Your renewal timer runs, the ACME client obtains a new certificate, writes it to disk, logs success. The site goes down anyway.

Nginx read the certificate into memory when it started. It does not watch the file. Neither does Apache, HAProxy, Postfix, or almost anything else. Until the service is reloaded it keeps serving the certificate it loaded at boot, which is the one that just expired. Everything in your logs says the renewal worked, because it did. The delivery step was never wired up.

Certbot handles this with deploy hooks, which run only when a certificate was actually renewed. Put a script in the hooks directory and it applies to every certificate on the box:

#!/usr/bin/env bash
# /etc/letsencrypt/renewal-hooks/deploy/reload-services.sh
# Remember: chmod +x, or it silently never runs.
set -euo pipefail

# Validate the config BEFORE reloading. A reload with a broken
# config can take the site down at the exact moment you were
# trying to keep it up.
nginx -t
systemctl reload nginx

# Anything else that holds a copy of the certificate.
# systemctl reload postfix

Then prove the whole path works without waiting sixty days to find out:

# Exercises the full renewal against the staging environment.
# Installs nothing, counts against no rate limits, and will
# fail loudly if your challenge setup is broken.
certbot renew --dry-run

One caveat worth knowing: --dry-run does not run deploy hooks by default, precisely because nothing was deployed. So it validates the issuance path but not the reload. Test that separately by touching the service and confirming the served certificate changes.

Expiry automation, and why it is no longer optional

This part has a date attached, so it is worth stating plainly rather than dancing around it.

The CA/Browser Forum approved a phased reduction of maximum certificate lifetimes. The cap dropped from 398 days to 200 days on 15 March 2026, falls to 100 days on 15 March 2027, and reaches 47 days on 15 March 2029. Domain validation reuse periods shrink alongside it.

The practical consequence: any process that involves a person remembering to renew something once a year is already broken, and by the third phase you are renewing every six weeks. Whatever you were getting away with manually, stop. ACME with a timer is the baseline. Certbot is the default choice, acme.sh is lighter and easier to script around, and Caddy obtains and renews certificates with no configuration at all, which is a genuinely good reason to consider it for a simple reverse proxy.

Choosing a challenge type

• HTTP-01 proves control by serving a file over port 80. Simple, and it breaks whenever port 80 is closed, redirected wholesale to HTTPS in a way that mangles the challenge path, or sitting behind a proxy that does not forward it.
• DNS-01 proves control by writing a TXT record. More setup, needs an API token for your DNS provider, and it is the only option for wildcard certificates. It also works for hosts with no inbound internet access at all, which is why I reach for it on anything internal.

The trade-off with DNS-01 is a credential: your renewal process now holds a token that can edit your DNS zone. Scope it to the narrowest permission your provider offers, and if they support delegating just the _acme-challenge records to a separate zone, do that instead.

Monitor expiry externally

Do not rely on the renewal system reporting its own health. If it is broken, it is also probably not telling you. Let’s Encrypt has discontinued its expiration reminder emails, so if that was quietly acting as your safety net, it is not one any more.

Check from outside, against the live endpoint, the way a user would:

#!/usr/bin/env bash
set -euo pipefail

HOST="example.com"
DAYS=21

# -checkend takes a number of seconds and exits non-zero if the
# certificate expires within that window. No parsing dates by hand.
if ! openssl s_client -connect "$HOST:443" -servername "$HOST" </dev/null 2>/dev/null 
     | openssl x509 -noout -checkend $(( DAYS * 86400 )); then
  echo "TLS certificate for $HOST expires within $DAYS days" >&2
  exit 1
fi

If you already run Prometheus, the blackbox exporter gives you probe_ssl_earliest_cert_expiry for free, which is a timestamp you can alert on and graph in Grafana across every endpoint you own. That is the better answer at more than a handful of hosts, because it also catches the certificates nobody remembered existed.

Two pieces of config that are now wrong

OCSP stapling with Let’s Encrypt. Let’s Encrypt removed the OCSP responder URL from newly issued certificates in May 2025 and shut down its responders in August 2025. If your config still has this, it does nothing except print a warning on every reload:

# nginx: [warn] "ssl_stapling" ignored, no OCSP responder URL in the certificate
# Comment these out if your certificates come from Let's Encrypt.
# ssl_stapling on;
# ssl_stapling_verify on;

Nothing breaks by leaving them, but a warning that appears on every reload is a warning people learn to ignore, and then they ignore the next one too.

Hand-tuned cipher suite lists. If you pasted a ssl_ciphers line from a guide more than a couple of years old, re-check it against the current Mozilla SSL Configuration Generator. Lists that were considered hardened at the time now sometimes exclude things you want or include things you do not, and the defaults have improved considerably.

When there is a CDN in front

Behind Cloudflare or a similar edge, there are two separate TLS connections: browser to edge, and edge to your origin. They use different certificates and fail independently.

This creates a specific blind spot. Your origin certificate can be expired, self-signed or serving the wrong name, and visitors will see nothing wrong, because the edge presents its own valid certificate and, in the more permissive encryption modes, does not verify the origin. It works until you switch to strict origin validation, or until the edge is bypassed, and then everything fails at once for reasons that look unrelated to anything you changed.

Test the origin directly by name, bypassing DNS. If your origin is a VPS with a provider like InterServer and the public DNS points at the CDN, this is the only way to see what the origin is really serving:

# --resolve forces the hostname to a specific IP without touching
# /etc/hosts, so SNI and Host header stay correct.
curl -v --resolve example.com:443:203.0.113.10 https://example.com/

Troubleshooting by error message

unable to get local issuer certificate

Missing intermediate on the server, or a trust store on the client that lacks the root. Check the server first with -showcerts; if the chain is complete, the problem is on the client side.

certificate has expired, but I just renewed it

The service was never reloaded. Compare the dates on disk with the dates being served. If they differ, that is your answer, and the fix is a deploy hook rather than another manual renewal.

Hostname mismatch on a name that is definitely in the certificate

Either SNI is not reaching the right vhost, or the name is in the Common Name rather than the SAN list, or you are hitting a wildcard at the wrong depth. Check the SAN list and re-run s_client with an explicit -servername.

Works over IPv4, fails over IPv6 (or the reverse)

Two different machines are answering, and only one of them was updated. Force the family with curl -4 and curl -6 and compare. Load balancers with one member out of sync produce the same intermittent pattern.

Renewal fails with a challenge error

For HTTP-01, confirm that http://example.com/.well-known/acme-challenge/test is reachable from outside without being redirected or rewritten. A blanket redirect to HTTPS is the usual culprit. For DNS-01, check that the TXT record actually appears with dig, and give slow-propagating providers more time.

Everything looks right and it still fails

Run an external scan. SSL Labs or a local run of testssl.sh will check things you did not think to, including protocol support and chain issues from the perspective of clients you do not have installed.

Common mistakes

• Pointing the server at cert.pem instead of fullchain.pem.
• Treating a padlock in your own browser as proof the certificate is correctly deployed.
• Testing with openssl s_client without -servername and debugging the wrong certificate.
• Assuming *.example.com covers the bare domain.
• Putting the hostname only in the Common Name.
• Renewing without reloading the service.
• Forgetting chmod +x on a deploy hook, so it silently never runs.
• Relying on the renewal system to tell you when it is broken.
• Including the root certificate in the served chain.
• Leaving OCSP stapling enabled with certificates that no longer support it.
• Ignoring the origin certificate because the CDN hides its state.
• Debugging a “not yet valid” certificate on the server when the client’s clock is wrong.

Best practices

• Serve the full chain, leaf first, without the root.
• Verify from a clean machine or container, never from the laptop you browse on.
• Put every name you serve in the SAN list, including the bare domain alongside any wildcard.
• Automate renewal with ACME and a timer, and run certbot renew --dry-run after any change to the setup.
• Attach a deploy hook that validates the config before reloading, and make it executable.
• Monitor expiry externally against the live endpoint, and alert with weeks of headroom.
• Use DNS-01 for wildcards and internal hosts, with a narrowly scoped API token.
• Keep NTP running everywhere, including inside containers.
• Test the origin directly when there is a CDN in front, not just the public hostname.
• Inventory your certificates, because the one that takes you down is the one nobody remembered.

FAQ

Why does my certificate work in Chrome but fail in curl?

Your server is not sending the intermediate certificate. Browsers work around this by caching intermediates from other sites and by fetching them from the URL in the certificate’s AIA extension. Command-line tools and application HTTP libraries do neither. Switch the server to fullchain.pem and reload.

What is the difference between cert.pem, chain.pem and fullchain.pem?

Leaf only, intermediates only, and both concatenated in the correct order. Web servers want fullchain.pem. The private key stays in privkey.pem and never goes in any of them.

Does a wildcard certificate cover the root domain?

No. *.example.com matches exactly one label, so it covers www.example.com but not example.com and not a.b.example.com. Request the bare domain as an additional SAN, which most ACME clients will do if you pass both names.

How often should certificates renew now?

Whatever your CA issues, renew at roughly two-thirds of the lifetime so a failed attempt has room to retry before anything expires. Since maximum lifetimes are on a published downward schedule, build for automated renewal at any interval rather than tuning a number you will have to change again.

Should I still enable OCSP stapling?

Not with Let’s Encrypt certificates, which no longer carry a responder URL. If your CA still supports it, stapling remains worthwhile. Check whether your certificate actually contains an OCSP URL before enabling the directives, rather than assuming.

Is a paid certificate more secure than a free one?

Cryptographically, no. The encryption is identical. What money buys is broader compatibility with old trust stores, longer validation reuse, warranties, and human support when something goes wrong. For a normal public website, a free automated certificate is the right default.

How do I get a certificate for something with no public DNS or open ports?

DNS-01, on a public domain you control, pointed at whatever internal address you like. The CA only needs to see the TXT record; it never needs to reach the host. This is how internal services get publicly trusted certificates without being exposed.

The one thing to remember

Most TLS certificate errors are not about cryptography. They are about delivery: whether the server sends everything the client needs, whether the name it sends matches the name that was asked for, and whether the certificate sitting in memory is the same one sitting on disk.

So stop checking with your browser. Check with openssl s_client from a machine that owes you no favours, and make sure something outside your server is watching the expiry date. As lifetimes keep shrinking, the gap between a renewal process that works and one that merely appears to work gets narrower every year.

Certificates causing you trouble?

Certificate problems tend to arrive at bad moments and turn out to be one wrong file path. Things I help with:

• Diagnosing a certificate that works for some clients and not others, and telling you exactly which link in the chain is missing.
• Setting up ACME renewal with deploy hooks across nginx, Apache, HAProxy, mail servers and anything else holding a copy.
• Migrating to DNS-01 for wildcards and internal hosts, with properly scoped API credentials.
• Certificate inventory and expiry monitoring in Prometheus and Grafana, so nothing expires without warning.
• Preparing a certificate estate for shorter lifetimes, replacing anything that still depends on a person remembering.
• Origin and edge TLS configuration behind a CDN, including strict origin validation.

Send me the output of openssl s_client -connect yourhost:443 -servername yourhost -showcerts and the error your failing client reports, and I will tell you what is wrong.

The post It Works in My Browser: Debugging TLS Certificate Errors in Chains, SANs and Expiry appeared first on John Nessime.

That is not hypothetical caution. In March 2025 a widely used third-party action was compromised: the attacker got hold of a maintainer’s token, pushed a malicious commit, and then retroactively repointed every version tag at it. Repositories that referenced the action by tag, which is nearly all of them, picked up the payload automatically. It dumped the runner’s secrets into the build log. On public repositories, those logs are readable by anyone.

Nobody’s workflow file changed. Nobody clicked anything. The pipeline just quietly handed out its credentials one Friday.

A GitHub Actions VPS deployment is fundamentally a credential handoff, and most guides on the topic skip straight to the YAML. This post covers the YAML too, but the useful part is what surrounds it: how to narrow the key so it cannot do much, how to verify you are talking to the right server, how to stop an unreviewed commit reaching production, and which of the failures here are silent.

Who can actually reach that key

Worth being explicit, because it is a longer list than people expect:

• Anyone with write access to the repository. A workflow file is code. Push a branch with a step that prints the key somewhere, trigger it, done. Write access to the repo is effectively access to the server.
• Every action in the job. Secrets are exposed to the whole job, not just the step that references them. A compromised action anywhere in that job can read the environment.
• Anyone who can influence what runs. This is why pull_request_target combined with checking out the pull request’s own code is such a well-known foot-gun: it runs the fork’s code with the base repository’s secrets in scope.

The goal is not to make the key unreachable. It is to make a stolen key not worth much.

Decide how the runner authenticates

Three options, and the right one depends on what you already run.

1. A dedicated SSH key in repository secrets. The default. Simple, works everywhere, and it is a long-lived credential sitting in a place several people can reach. Acceptable when the key is narrowed properly, which is most of the rest of this post.
2. An overlay network. Put the runner and the server on a private network with Tailscale or WireGuard for the duration of the job, using an ephemeral auth key, and close SSH to the public internet entirely. More moving parts, and it removes port 22 from the internet rather than defending it. If you already run an overlay for other reasons, this is clearly better.
3. A self-hosted runner on the server. No inbound SSH at all, since the runner polls outward. The trade-off is real and often understated: you have just installed a general-purpose code execution service on your production box, and for a public repository that is close to indefensible. On a private repository with a trusted team it is reasonable.

One thing that sounds appealing and is not practical: allowlisting GitHub’s runner IP ranges at your firewall. The ranges are published through GitHub’s meta API, but they are enormous and they change, and every other GitHub customer’s job runs from inside them. It buys you very little for a lot of maintenance.

Narrow the key until it is boring

Assume the key leaks. What can the holder do? Work backwards from there.

Give it its own user. A deploy account that owns the application directory and nothing else. Not your admin account, not root, and definitely not a key you also use from your laptop.

Restrict the key in authorized_keys. SSH lets you attach options to an individual key, and they apply regardless of what the client asks for:

# ~deploy/.ssh/authorized_keys
#
# restrict = no port forwarding, no agent forwarding, no X11,
#            no PTY, no user rc file. Deny everything, then
#            re-enable only what you need.
# command  = ignore whatever the client asked to run and run
#            this instead. The client's command is available
#            to the script as $SSH_ORIGINAL_COMMAND.

restrict,command="/usr/local/bin/deploy-guard" ssh-ed25519 AAAA...  deploy@github-actions

The guard script decides what the key is allowed to do. Keep it short enough to read in one go:

#!/usr/bin/env bash
# /usr/local/bin/deploy-guard
set -euo pipefail

case "${SSH_ORIGINAL_COMMAND:-}" in
  # rsync sends its own server-side invocation. Match it loosely
  # enough to work, tightly enough to pin the destination.
  "rsync --server "*" /var/www/app/")
    exec $SSH_ORIGINAL_COMMAND
    ;;
  "deploy:reload")
    exec sudo /bin/systemctl reload nginx
    ;;
  *)
    echo "refused: $SSH_ORIGINAL_COMMAND" >&2
    exit 1
    ;;
esac

Now a stolen key can write to one directory and reload one service. It cannot open a shell, cannot forward a port into your private network, and cannot read anything else on the box.

Scope the sudo entry too. The deploy user needs to reload a service, not run arbitrary commands as root:

# visudo -f /etc/sudoers.d/deploy
# One command, no password, no wildcards. A wildcard here
# often turns back into a general-purpose root shell.
deploy ALL=(root) NOPASSWD: /bin/systemctl reload nginx

Be honest about the cost: the guard script is one more thing to maintain, and the day you add a new deploy step it will refuse it and you will spend ten minutes confused. That is the trade you are making, and on anything handling customer data it is worth it. On a hobby project, a plain unrestricted deploy user with no sudo is a reasonable stopping point.

Verify the server, not just the key

Almost every tutorial on this topic contains StrictHostKeyChecking=no, which turns off the check that stops you handing your credentials to whatever machine answered. It is there because the alternative is mildly annoying, and the alternative takes one command.

# Run this once, from a machine you trust, and compare the
# fingerprint against what the server reports locally.
ssh-keyscan -t ed25519 example.com

Store that output as a repository variable, not a secret. It is public information, and keeping it visible means you can actually see it when it changes, which matters because it will change the first time you rebuild the server.

The workflow

name: Deploy

on:
  push:
    tags: ['v*']          # production ships from tags, not branches

# This job reads code and talks to a server. It has no business
# writing to the repository, so take that away.
permissions:
  contents: read

concurrency:
  group: deploy-production
  # Deliberately NOT cancel-in-progress. Killing a deploy halfway
  # leaves the server in a state nobody designed.
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    environment:
      name: production      # scoped secrets and required reviewers live here
      url: https://example.com

    steps:
      # Pin every action to a full 40-character commit SHA, with the
      # version in a trailing comment so Dependabot can still bump it.
      # Get the SHA from the action's releases page.
      - uses: actions/checkout@PUT_THE_FULL_COMMIT_SHA_HERE  # pinned

      - name: Build
        run: |
          npm ci
          npm run build

      - name: Configure SSH
        # Pass secrets through env, never interpolate ${{ }} directly
        # into a run block. Interpolation happens before the shell
        # sees the script, so a value with the wrong characters
        # either breaks the script or becomes part of it.
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          KNOWN_HOSTS: ${{ vars.DEPLOY_KNOWN_HOSTS }}
        run: |
          install -m 700 -d ~/.ssh
          printf '%sn' "$DEPLOY_KEY" > ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          printf '%sn' "$KNOWN_HOSTS" > ~/.ssh/known_hosts

      - name: Ship
        run: |
          rsync -az --delete --exclude-from=.deployignore 
            -e "ssh -o BatchMode=yes" 
            ./dist/ deploy@example.com:/var/www/app/

      - name: Reload and verify
        run: |
          ssh -o BatchMode=yes deploy@example.com 'deploy:reload'
          curl --fail --silent --show-error https://example.com/healthz > /dev/null

Three details in there worth calling out, because they are the ones people leave out.

-o BatchMode=yes stops SSH prompting for anything. Without it, a missing key or an unknown host produces a prompt that nobody is there to answer, and the job hangs until the timeout instead of failing in three seconds.

--delete is what makes the target match the source, which is the point of a deploy, and also what will erase your uploads directory if .deployignore is wrong. Run it once with --dry-run against staging and read the list before you point it at production.

The final curl --fail is the difference between “the files copied” and “the site works”. Files landing on disk is not a successful deploy, and a job that goes green while the site returns 502 is worse than one that fails.

Third-party actions

There are convenient actions that wrap SSH deployment, and they save maybe eight lines. In the one job on your account that holds a production credential, eight lines is a bad trade for another maintainer in your trust chain. The workflow above uses plain ssh and rsync for that reason.

Where you do use third-party actions, pin them to a full commit SHA. A tag is a pointer and pointers can be moved, which is precisely what happened in the incident at the top of this post. A SHA is content-addressed: it cannot be repointed at different code.

Two things make this sustainable rather than miserable. Dependabot understands SHA pins with a version comment and will open pull requests to bump them. And you can restrict which actions are allowed to run at all, in the repository or organisation settings, which is worth doing once and forgetting about.

Gate production properly

The environment: key in that workflow is doing more work than it appears to. Environments give you three things you cannot get from repository secrets alone:

• Scoped secrets. The production key is only available to jobs that declare the production environment. A branch that adds a job without it gets nothing.
• Required reviewers. The job pauses and waits for a named human to approve it. This is the single control that turns “anyone with write access can deploy” into “anyone with write access can request a deploy”.
• Deployment branch and tag rules. Restrict the environment so only protected tags or a specific branch can use it, so a feature branch cannot reach production even with the right job definition.

Deploying from tags rather than branch pushes is worth the small friction. It makes a release a deliberate act, and it gives you an obvious rollback: redeploy the previous tag.

For anything with real traffic, put the files somewhere new and switch atomically rather than rsyncing over a live directory. Copy into a timestamped release directory, then move a symlink. The site changes version instantly instead of spending several seconds as a mix of old and new files, and rollback becomes a symlink change rather than a redeploy.

Troubleshooting

Permission denied (publickey)

The most common cause by far is a missing trailing newline. Private keys must end with one, and pasting into a web form frequently drops it. That is why the workflow uses printf '%sn' rather than echo: it guarantees the newline regardless of what got stored.

After that, check permissions on the server: ~/.ssh must be 700 and authorized_keys 600, owned by the deploy user, and the home directory must not be group-writable. Add -vvv to the ssh command to see how far the handshake gets, and read journalctl -u ssh on the server for the reason, since the client is deliberately vague.

Host key verification failed

Either the known_hosts variable is empty, or the server’s host key genuinely changed. If you rebuilt the server, regenerate it with ssh-keyscan and update the variable. If you did not rebuild the server, stop and find out why the key changed before you update anything.

The job hangs until the timeout

SSH is waiting on a prompt nobody will answer. Add -o BatchMode=yes and it fails immediately with a readable reason instead.

rsync works from my laptop, fails from the runner

If you set a forced command, it is rejecting the invocation. The refusal message goes to stderr on the server side, so add a log line to the guard script that records $SSH_ORIGINAL_COMMAND, run the job once, and read what rsync actually sent.

Works on push, fails on pull requests

Working as intended. Secrets are not exposed to workflows triggered by pull requests from forks. Do not reach for pull_request_target to work around it. Split the workflow so that pull requests build and test without secrets, and only pushes or tags deploy.

A secret appeared in the logs

Masking matches the exact stored string. Transform it in any way and the mask stops working: base64-encoded, JSON-escaped, split across lines, or with whitespace trimmed. Do not treat masking as a control. Rotate the key, delete the run logs, and remove whatever produced the output.

Common mistakes

• Using an existing personal SSH key instead of generating a dedicated one for the runner.
• Giving the deploy user full sudo, or a sudo rule with a wildcard in it.
• Disabling host key checking with StrictHostKeyChecking=no.
• Referencing third-party actions by tag in the job that holds your production key.
• Interpolating ${{ secrets.X }} directly into a run: script instead of passing it through env:.
• Leaving default write permissions on a job that only needs to read.
• Using cancel-in-progress: true on a deploy, so a second push interrupts the first mid-copy.
• Running rsync --delete at production without testing the exclude list.
• Treating “the files copied” as a successful deploy, with no health check.
• Deploying straight from branch pushes with no approval step.
• Relying on log masking to keep a leaked secret out of the output.
• Never rotating the deploy key, including after someone leaves the team.

Best practices

• Dedicated deploy user, dedicated key, used for nothing else.
• Restrict the key in authorized_keys with restrict and a forced command.
• Scope sudo to the exact commands the deploy needs.
• Pin every action to a full commit SHA and let Dependabot bump them.
• Keep the deploy job small: fewer steps means fewer things that can read the secret.
• Set permissions: explicitly, defaulting to read-only.
• Verify host keys from a stored known_hosts value.
• Use environments with required reviewers and deployment rules for production.
• Deploy from tags, and make rollback a redeploy of the previous tag or a symlink swap.
• End every deploy with a real HTTP request against the public URL.
• Rotate the deploy key on a schedule and whenever someone leaves.

FAQ

Is it safe to store an SSH private key in GitHub secrets?

Safe enough, if the key is worth little on its own. Secrets are encrypted at rest and masked in logs, but they are readable by any job that references them and by anyone who can modify a workflow. Assume it can leak and make sure a leaked key only grants write access to one directory and one service reload.

Should I use a ready-made SSH deploy action?

They work and they are convenient. My preference is plain ssh and rsync in the job that touches production, because it removes a maintainer from the trust chain for very little added effort. If you do use one, pin it to a SHA like anything else.

Can I use OIDC instead of a stored key?

Not directly against sshd, which has no concept of an OIDC token. OIDC removes long-lived secrets when the target is a cloud provider that can verify GitHub’s token. To get the same benefit for a plain VPS you need something in between that trades the token for short-lived access: a secrets manager, a bastion that supports it, or an overlay network with ephemeral auth keys.

Should I allowlist GitHub’s IP ranges on my firewall?

Generally not worth it. The published ranges are large and change over time, so you are signing up for ongoing maintenance, and every other GitHub customer’s runners share those ranges. Narrowing what the key can do achieves more.

Self-hosted runner or hosted runner?

Hosted for public repositories, without exception, because a self-hosted runner will execute code from pull requests. Self-hosted is reasonable on a private repository with a trusted team, and it removes inbound SSH entirely. Just be clear that you have moved the risk rather than removed it.

How do I roll back a bad deploy?

Redeploy the previous tag, or swap the symlink back if you use atomic releases. The part that does not roll back is the database, so if a release ran a migration, know what it did before you revert the files. Files going backwards while the schema stays forwards is often worse than the bug you were fixing.

How often should the deploy key be rotated?

On a schedule you will actually keep, and immediately whenever someone with repository access leaves or a run log looks suspicious. Rotation is two steps, adding the new public key to authorized_keys and updating the secret, so there is no good reason for it to be a project.

The one thing to remember

The YAML is the easy half. A working GitHub Actions VPS deployment takes an afternoon; the part that decides whether it is a good idea is what that key can do once it is out of your hands, and it will eventually be out of your hands.

So build it assuming the key leaks. Dedicated user, forced command, scoped sudo, pinned actions, an approval gate on production. Then the worst case is an inconvenient rotation rather than an incident, and you will not be reading a build log at midnight trying to work out what a stranger’s code did with your credentials.

Want this built or reviewed?

Most deploy pipelines I get asked to look at work fine and hand out far more access than they need. Work I take on:

• Building a GitHub Actions deploy to a VPS end to end: build, ship, reload, health check, rollback path.
• Auditing an existing pipeline for what a leaked runner secret would actually grant, and closing the gap.
• Locking down the server side: deploy user, forced commands, scoped sudo, atomic release directories.
• Moving SSH off the public internet onto an overlay network, or setting up a self-hosted runner safely.
• Pinning and auditing third-party actions, plus repository settings that limit what can run.
• Environments, required reviewers and tag-based releases for teams that currently deploy on every push to main.

Send me your workflow file and the output of ssh deploy@yourhost 'id', and I will tell you what a stolen key would get.

The post Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully appeared first on John Nessime.